A method and device for identifying, positioning and automatically picking nuts with green husks based on image recognition

Abstract

The present invention provides a method for identifying, positioning and automatically picking and controlling green nut fruits based on image recognition, belonging to the technical field of nut recognition and picking. The identification and positioning are carried out by a moving camera to take pictures of the nut tree from different angles, then the images taken at each angle are recognized, and then image fusion is performed, so that the fruits blocked by leaves can also be recognized. By taking pictures of the nut tree from different angles, the present invention can recognize the fruits blocked by some leaves. At the same time, through taking pictures from different angles, the reflective recognition of green nut fruits can be realized at different angles (the circular arc contour of the fruit is easier to be recognized at different angles), the position of the green nut fruits can be recognized more accurately, and through multi-image fusion, the fruits blocked by leaves can be displayed, avoiding omission in picking.

Description

Technical Field

[0001] The present invention relates to the technical field of nut recognition and picking, and particularly relates to a method for identifying, positioning and automatically picking and controlling green husk nuts of nuts based on image recognition and a picking device. Background Technique

[0002] The harvesting of macadamia nuts generally includes manual harvesting and mechanical vibration harvesting methods. Manual harvesting is to use a fruit hook to peel off the involucre one by one when the fruits are ripe. When using mechanical harvesting, ethephon is sprayed on the fruit trees two weeks before harvesting to ripen them, and then the branches are vibrated mechanically to make the ripe fruits fall. Using ethephon will cause the leaves to fall off prematurely, which will affect the growth of the fruit trees and the yield of the next year. At present, in China, the mechanical vibration method or the use of mechanical hands for assisted harvesting is mostly adopted. However, the mechanical hands for picking do not have the function of automatically positioning and identifying the green husk nuts of nuts, and the picking efficiency is low, the speed is slow, and the effect is not good.

[0003] Since the color of the green husk nuts of nuts is very similar to the color of the leaves, it is very difficult to identify them when using image recognition. Due to the similar color of the fruits and the leaves, the difficulty of recognition is increased, and the position of the green husk nuts of nuts cannot be accurately identified. At the same time, the position of the green husk nuts of nuts on the branches is easily blocked by the leaves, and it is easy to miss the recognition, which brings great difficulties to the use of mechanical hands for picking and results in low efficiency. Therefore, it is necessary to design a method for identifying, positioning and automatically picking and controlling green husk nuts of nuts based on image recognition and a picking device. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for identifying, positioning and automatically picking and controlling green husk nuts of nuts based on image recognition and a picking device, so as to solve the technical problems of difficult identification and positioning of existing green husk nuts of nuts and low picking efficiency. By moving the camera to take pictures of the nut trees from different angles, and then identifying the images taken at each angle and performing image fusion, the fruits blocked by the leaves can also be identified.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] A method for identifying and positioning green husk nuts of nuts based on image recognition, the method comprising the following steps:

[0007] Step 1: Install a camera mobile shooting device on the green husk nut picking trolley of nuts;

[0008] Step 2: Drive the green husk nut picking trolley of nuts in front of the nut tree, and the camera takes pictures of the nut tree at the initial position of the mobile shooting device to obtain an initial image;

[0009] Step 3: Perform object contour recognition on the initial image to obtain the initial recognized position points of several nut green-skinned fruit points, and mark several points as recognition points;

[0010] Step 4: Construct a two-dimensional coordinate plane, paste the initial image on the two-dimensional coordinate plane, obtain the coordinate position of the nut and the coordinates of the identification point, and set the center point;

[0011] Step 5: The camera moves on the mobile shooting device, takes pictures after each fixed distance to obtain different direction images, and then summarizes the direction images of all points to obtain a direction image set;

[0012] Step 6: Fit each image in the directional image set to the two-dimensional coordinate plane according to the identification points to obtain several layers of image sets, and then identify the fruit position coordinates of the green-skinned nuts in each layer of image;

[0013] Step 7: Summarize the fruit position coordinates of all images into a coordinate plane, and keep the position coordinates of each fruit unchanged, so as to realize the recognition and positioning of green-skinned nuts.

[0014] Furthermore, in step 1, the camera moving shooting device includes a camera, a sliding chassis and a sliding frame. The camera is set on the sliding chassis, the sliding chassis is clamped in the sliding frame and can be movably set. The sliding frame includes a vertical slide rail and a horizontal slide rail. The vertical slide rail and the horizontal slide rail are vertically arranged to cross each other. An initial positioning point is set inside the intersection. An initial positioning point sensing switch is set at the bottom of the sliding chassis. When the initial positioning point and the initial positioning point sensing switch are on the same straight line, it means that the camera is in the initial position. The sliding chassis is connected to an external control device and moves and pauses according to the signal of the control device. The camera is connected to the control device and transmits the collected pictures to the control device for processing. At the same time, the vertical slide rail and the horizontal slide rail are set to a retractable structure, which is extended or shortened according to the size of the picked nut tree.

[0015] Furthermore, in step 2, the nut green fruit picking cart comes to the front of the nut tree, and the camera recognizes the external wheel library of the nut tree image, identifies the height and width of the nut tree, and then moves the camera mobile shooting device up and down so that the center position of the camera mobile shooting device is aligned with the center position of the nut tree, and then the camera moves to the initial positioning point (5) of the mobile shooting device, and then shoots the nut tree to obtain the initial image of the nut tree.

[0016] Further, in step 3, the initial image is initialized, and then the objects inside the image are subjected to contour recognition, which includes leaf contours, round contours of the green husk fruits of nuts, and branch contours. When a round arc contour is recognized, it is considered as the fruit of the green husk fruit of the nut, and the positions of several point green husk fruits of nuts are obtained. At the same time, several points are selected as recognition points at the edge and the midline of the nut tree in the initial image. The recognition points are used for position coordinate correction during the later highlighting and fitting. Due to the change in the shooting angle, it is necessary to perform image position stretching or shrinking correction after shooting so that the recognition of all images coincides during fitting.

[0017] Further, in step 5, the camera moves along the sliding frame at the center of the camera moving shooting device. First, it moves to the upper part of the vertical slide rail. After moving a fixed position each time, it pauses, then notifies the camera to take a picture, then records the specific position of the taken image, then moves up a fixed position again and takes a picture until it reaches the top and then returns to the initial point. Then it moves down to take pictures, and then moves left and right to take pictures. All the taken direction images are collected to obtain a direction image set.

[0018] Further, in step 6, the positions of the recognition points of several layers of image sets coincide. Then, the round arc contour with the green husk fruit of the nut in each image is recognized as the fruit position of the green husk fruit of the nut, and the position of each green husk fruit of the nut displays detailed coordinate information;

[0019] In step 7, the specific process of summarizing the fruit position coordinates into a coordinate plane is as follows: several layers of images are fused together, and then the same or adjacent fruit position coordinates are superimposed. If there are fruits in the positions of some images and no fruits in other images, then the leaf occlusion of the images without fruits is erased, and the positions with fruits in all images are displayed to obtain the position of the green husk fruit of the nut.

[0020] An automatic picking control method for green husk fruits of nuts based on image recognition uses the nut green husk fruit recognition and positioning method of any one of claims 1-6 to position the nuts. Then, a picking temporary correction camera is installed at the front end of the manipulator of the picking trolley. According to the position of each green husk fruit of the nut, the control device calculates the rotation degrees of freedom required for the manipulator to pick the nut. When the manipulator comes to the green husk fruit of the nut, the temporary correction camera collects the best picking position in real time, and then the manipulator picks the nut. When the joint picks and leaves, the temporary correction camera collects whether there are still unpicked nuts at the just-picked place. If not, it is deleted in the fruit position plane. If so, the manipulator is continuously controlled for secondary picking until all are picked. The fruit coordinates in the fruit position plane are all deleted, and then the picking trolley moves to the other side to continue picking the nuts. Each nut tree can be picked after picking four sides.

[0021] Further, the specific process of calculating the rotational degrees of freedom required by the manipulator by the control device is as follows: Set H, L, and P to represent the height from the ground to the base of the robotic arm, the length of the long arm, and the length of the short arm respectively. Let m be the central height of the nut to be picked. The angle between the long arm and the short arm remains 90 degrees. Rotate the long arm so that the center of the camera is collinear with the target center. After collinearity, return the current base rotation angle θ and the long arm rotation angle θ1. Using the trigonometric theorem and the Pythagorean theorem, the length of the imaging ranging S can be obtained. The specific derivation formulas are shown in Equations (1), (2), (3), (4), and (5) as follows:

[0022] T = P·tanθ1 + L (1)

[0023]

[0024]

[0025] After obtaining S, through the rotation angle θ, decompose S into vectors. Using the sine and cosine function calculation formulas, obtain the x-axis coordinate X, y-axis coordinate Y, and z-axis coordinate Z with the robotic arm itself as the base coordinate. The specific formulas are shown in Equations (6), (7), and (8) as follows:

[0026] Z = 2·m (6)

[0027]

[0028] After the camera analyzes the relative true coordinates of the target picked nut and the manipulator, through the inverse solution algorithm, solve the servo rotation angles of each degree of freedom of the robotic arm. A three-degree-of-freedom robotic arm is used, and the trigonometric geometric method is used to solve the inverse kinematics formula of the robotic arm. Given that the true coordinates of the target nut are (X, Y, Z), the long arm of the robotic arm is L1, the short arm is L2, and the gripper connection part is L3. The three-axis rotation angles to be solved are respectively

[0029] From the inverse trigonometric function, in a right triangle, The formula is shown in Equation (9).

[0030]

[0031] Establish a geometric model of the robotic arm in the YOZ plane of the robotic arm. Using the cosine theorem in trigonometry, Equations (10), (11), and (12) can be obtained:

[0032]

[0033] Using the Pythagorean theorem of a right triangle, the expressions of T1 and T2 are shown in Equations (13) and (14).

[0034]

[0035] Using the inverse cosine formula in trigonometric functions, the respective angles can be obtained. The specific expressions are shown in Equations (15), (16), (17), and (18) as follows:

[0036]

[0037] After obtaining that the rotation angles of the three axes are respectively the manipulator is controlled to perform the picking action.

[0038] An automatic picking device for identifying and positioning nut green peel fruits based on image recognition, comprising a picking vehicle body, a mobile shooting device, an identification and positioning camera, a picking manipulator, a picking correction camera, and a data analysis and control device. The mobile shooting device, the picking manipulator, and the data analysis and control device are all arranged on the picking vehicle body. The mobile shooting device is arranged at the front end, and the identification and positioning camera is slidably arranged on the mobile shooting device. The picking vehicle body, the identification and positioning camera, the picking manipulator, and the picking correction camera are all connected to the data analysis and control device.

[0039] An application of a method for identifying and positioning nut green peel fruits based on image recognition, where the identification and positioning method is applied to automatic picking and identification and positioning of oranges, automatic picking and identification and positioning of walnuts, automatic picking and identification and positioning of snow pears, or automatic picking and identification and positioning of apples.

[0040] Due to the adoption of the above technical solution, the present invention has the following beneficial effects:

[0041] (1) By shooting the nut tree from different angles, the present invention can identify the fruits blocked by some leaves. At the same time, through shooting from different angles, different-angle specular reflection recognition can be realized for the nut green peel fruits (the circular arc contour of the fruits is easier to identify from different angles), the position of the nut green apples can be identified more accurately, and through multi-image fusion, the fruits blocked by leaves can be displayed to avoid missing picking;

[0042] (2) During the picking process, by adding a picking correction camera at the front end of the manipulator, the picking point can be corrected in real time, so that the picking clamping point is the optimal point, enabling more fruits to be clamped at one time. At the same time, at the moment when the picking is completed, the picking point is recognized for the second time to avoid missing any fruits during picking, realizing more thorough picking and avoiding omission;

[0043] (3) The identification and positioning method of the present application can also be used for identification and positioning in the automatic picking of other fruits, realizing more accurate identification and positioning. Description of the Drawings

[0044] Figure 1 is the flow chart of the identification and positioning method of the present invention;

[0045] Figure 2 is the image comparison diagram of the same point taken from different points in the direction of the tree branch of the present invention;

[0046] Figure 3 is the schematic structural diagram of the mobile shooting device of the present invention;

[0047] Figure 4 is the conversion model diagram of the actual distance parameter of the nut position of the present invention;

[0048] Figure 5 is the geometric model diagram of the robotic arm established in the XOY plane of the robotic arm of the present invention;

[0049] Figure 6 is the analysis diagram of the geometric model of the robotic arm established in the YOZ plane of the robotic arm of the present invention;

[0050] Figure 7 is the principle block diagram of the automatic picking device module of the present invention.

[0051] In the attached drawings, 1 - leaf occlusion area, 2 - fruit shape of the green husk nut, 3 - vertical slide rail, 4 - horizontal slide rail, 5 - initial positioning point. Detailed implementation manners

[0052] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the following provides preferred embodiments with reference to the attached drawings and further elaborates on the present invention in detail. However, it should be noted that many details listed in the specification are only for enabling the reader to have a thorough understanding of one or more aspects of the present invention, and these aspects of the present invention can be implemented even without these specific details.

[0053] As Figure 1 shown, a method for identifying and positioning green husk nuts based on image recognition, the method includes the following steps:

[0054] Step 1: Install a camera moving shooting device on the nut green husk fruit picking trolley. The camera moving shooting device includes a camera, a sliding chassis, and a sliding frame. The camera is set on the sliding chassis. The sliding chassis is clamped in the sliding frame and can be movably arranged. The sliding frame includes a vertical slide rail 3 and a horizontal slide rail 4. The vertical slide rail 3 and the horizontal slide rail 4 are arranged crosswise and perpendicularly. An initial positioning point 5 is arranged inside the intersection. An initial positioning point induction switch is arranged at the bottom of the sliding chassis. When the initial positioning point 5 and the initial positioning point induction switch are on the same straight line, it indicates that the camera is in the initial position. The sliding chassis is connected to an external control device and moves and pauses according to the signal of the control device. The camera is connected to the control device and transmits the collected pictures to the control device for processing. At the same time, the vertical slide rail 3 and the horizontal slide rail 4 are set as telescopic structures and are extended or shortened according to the size of the nut tree to be picked. The camera moving shooting device is as Figure 3 shown, set as a cross structure with a closed end structure, and then a telescopic guide rail structure is arranged in the middle to realize the telescopic setting according to the size of the tree.

[0055] Step 2: Drive the nut green husk fruit picking trolley in front of the nut tree, and the camera takes an initial image of the nut tree at the initial position of the moving shooting device. When the nut green husk fruit picking trolley comes in front of the nut tree, the camera identifies the external contour of the nut tree image, identifies the height and width of the nut tree, and then moves the camera moving shooting device up and down so that the center position of the camera moving shooting device is aligned with the center position of the nut tree. Then the camera moves to the initial positioning point 5 of the moving shooting device and then takes a picture of the nut tree to obtain the initial image of the nut tree. The initial image is a front-facing image. When taking pictures, one side of the whole tree is within the image range. Find the features of several relatively prominent points on the edge as identification points. After shooting at an angle, the positions of several of these points can basically be identified among these points.

[0056] Step 3: Conduct object contour recognition on the initial image to obtain the nut green husk fruit position points of several initially recognized points, and at the same time mark several points as identification points. Initialize the initial image, and then conduct contour recognition on the objects inside the image. The contour recognition includes leaf contours, nut green husk fruit circular contours, and branch contours. When a circular arc contour is recognized, it is considered to be the fruit of the nut green husk fruit, and the positions of several nut green husk fruits are obtained. At the same time, several points are selected at the edge and the midline of the nut tree in the initial image as identification points. The identification points are used for position coordinate correction when highlighting the fitting in the later stage. Due to the change in the shooting angle, image position stretching or shrinking correction is required after shooting, and the recognition of all images is coincident during fitting. The contour recognition uses the contour recognition method. The contours of fruits and leaves are obtained through edge recognition, and then the circular arc is identified as the fruit.

[0057] Step 4: Construct a two-dimensional coordinate plane, paste the initial image on the two-dimensional coordinate plane to obtain the coordinate positions of the green husk fruits of the nuts and the coordinates of the recognition points, and set the center point. The two-dimensional coordinate plane is the XY-axis plane. By setting the middle point position of the XY-axis plane as the origin of the plane, that is, the center point, and then identifying the characteristics of the initial image at the center point, the characteristics of the center point are also used as the feature points for later recognition and matching.

[0058] Step 5: The camera moves on the mobile shooting device. After moving a fixed distance each time, it takes pictures to obtain images in different directions, and then aggregates all the directional images of the points to obtain a set of directional images. The camera moves along the sliding frame at the center of the camera mobile shooting device. First, it moves to the upper half of the vertical slide rail 3. After moving a fixed position each time, it pauses, then notifies the camera to take pictures, and then records the specific position of the taken image. Then it moves up a fixed position and takes pictures again until it moves to the top and returns to the initial point. Then it moves down to take pictures, and then moves left and right to take pictures. Aggregate all the taken directional images to obtain a set of directional images. As Figure 2 shown, take pictures again at a position 10 cm each time. When the first picture is at the center, the fruit is on the back of the leaf and is blocked. When shooting only from the front, the fruit cannot be seen. However, when the camera moves up, the fruit starts to be gradually exposed. When moving up 20 cm, more of the exposed fruit is photographed. When these three images are fused together, the position of the fruit can be seen. However, the second and third pictures need to be stretched, then Figure 1 delete the blocked positions in it, and the image of the exposed fruit can be seen, realizing the wiping away of the fruit leaves that block, achieving more accurate recognition. At the same time, for the images taken from different angles, the direction of the fruit's reflection is different. When an image in one direction cannot be recognized. An image in another direction can be recognized.

[0059] Step 6: Fit each image in the set of directional images to the two-dimensional coordinate plane according to the recognition points to obtain several sets of images, and then identify the fruit position coordinates of the green husk fruits of the nuts in each set of images. The positions of the recognition points of several sets of images are all coincident. Then, identify the circular arc contour with the green husk fruits of the nuts in each image as the fruit position of the green husk fruits of the nuts, and detailed coordinate information of the position of each green husk fruit is displayed.

[0060] Step 7: Summarize the fruit position coordinates of all images into a coordinate plane, keeping the position coordinates of each fruit unchanged to achieve the recognition and positioning of the green husk nuts. The specific process of summarizing the fruit position coordinates into a coordinate plane is to fuse several layers of images together, and then superimpose the same or adjacent fruit position coordinates. If there are fruits in some image positions and no fruits in other images, then erase the leaf occlusion of the images without fruits to display the positions with fruits in all images, and obtain the positions of the green husk nuts.

[0061] An automatic picking control method for green husk nuts based on image recognition uses the nut green husk fruit recognition and positioning method of any one of claims 1-6 to position the nuts. Then, a picking temporary correction camera is installed at the front end of the manipulator of the picking trolley. According to the position of each green husk nut, the control device calculates the degree of freedom of rotation required for the manipulator to pick the nut. When the manipulator comes to the green husk nut, the temporary correction camera collects the optimal picking position in real time, and then the manipulator picks. When the joint picks and leaves, the temporary correction camera collects whether there are still unpicked nuts at the just-picked position. If not, it deletes them in the fruit position plane of the nut. If there are, it continues to control the manipulator for secondary picking until all are picked and the fruit coordinates in the fruit position plane are deleted. Then, the picking trolley moves to another side to continue picking the nuts, and each nut tree can be picked after four-sided picking.

[0062] In the embodiment of the present invention, the specific process of the control device calculating the degree of freedom of rotation required for the manipulator is as follows: As Figure 4 shown, let H, L, and P represent the height from the ground to the base of the robotic arm, the length of the long arm, and the length of the short arm respectively, m be the central height of the nut to be picked, the angle between the long arm and the short arm remains 90 degrees unchanged, the long arm rotates to make the camera center and the target center collinear. After collinearity, return the current base rotation angle θ and the long arm rotation angle θ1. Using the trigonometric theorem and the Pythagorean theorem, the imaging ranging S length can be obtained. The specific derivation formulas are shown in formulas (1), (2), (3), (4), and (5):

[0063] T = P·tanθ1 + L (1)

[0064]

[0065] After obtaining S, decompose S into vectors through the rotation angle θ, and use the sine and cosine function calculation formulas to obtain the x-axis coordinate X, y-axis coordinate Y, and z-axis coordinate Z with the robotic arm itself as the base coordinate. The specific formulas are shown in formulas (6), (7), and (8):

[0066] Z = 2·m (6)

[0067]

[0068] After the camera analyzes the relative true coordinates of the target picked nut and the manipulator, the rotation angles of the servos of each degree of freedom of the robotic arm are solved through the inverse solution algorithm. A three-degree-of-freedom robotic arm is used, and the trigonometric function geometric method is used to solve the inverse kinematics formula of the robotic arm. Given that the true coordinates of the target nut are (X, Y, Z), the long arm of the robotic arm is L1, the short arm is L2, and the gripper connection part is L3. The rotation angles of the three axes to be solved are respectively

[0069] As Figure 5 shown, from the inverse trigonometric function, in a right triangle, The formula of is shown in Equation (9).

[0070]

[0071] As Figure 6 shown, a geometric model of the robotic arm is established in the YOZ plane of the robotic arm. Using the cosine theorem in trigonometric functions, Equations (10), (11), and (12) can be obtained:

[0072]

[0073]

[0074] Using the Pythagorean theorem of a right triangle, the expressions of T1 and T2 are shown in Equations (13) and (14).

[0075]

[0076] Using the inverse cosine formula in trigonometric functions, can be obtained their respective angles. The specific expressions are shown in Equations (15), (16), (17), and (18):

[0077]

[0078] The rotation angles of the three axes are respectively After that, the manipulator is controlled to perform the picking action.

[0079] An automatic picking device for nut green fruit recognition and positioning based on image recognition, as Figure 7As shown in the figure, it includes a picking vehicle body, a mobile shooting device, an identification and positioning camera, a picking manipulator, a picking correction camera, and a data analysis and control device. The mobile shooting device, the picking manipulator, and the data analysis and control device are all arranged on the picking vehicle body. The mobile shooting device is arranged at the front end, and the identification and positioning camera is arranged on the mobile shooting device and can slide. The picking vehicle body, the identification and positioning camera, the picking manipulator, and the picking correction camera are all connected to the data analysis and control device. The picking vehicle body is mainly used to move among mountains, and at the same time carry the mobile shooting device, the identification and positioning camera, the picking manipulator, the picking correction camera, and the data analysis and control device to move. A fruit storage box is also arranged on the picking vehicle body to store the picked nuts. The mobile shooting device is as Figure 3 shown in the structure. The identification and positioning camera uses an OpenMV camera. The data analysis and control device mainly completes the contour recognition of internal leaves and fruits in the image, and at the same time controls the picking manipulator to pick by issuing commands. The picking correction camera realizes the positioning of the clamping point during the picking process, and at the same time realizes the secondary recognition of the clamping position after clamping to avoid missing picking.

[0080] An application of a nut green peel fruit identification and positioning method based on image recognition. The identification and positioning method is applied to the automatic picking and identification of oranges, the automatic picking and identification of walnuts, the automatic picking and identification of Sydney pears, or the automatic picking and identification of apples. In addition to the above-mentioned fruit automatic picking that can apply this method for identification and positioning, other fruits that can realize automatic picking can also use the identification and positioning method of this application.

[0081] Matters not covered in this invention are well-known technologies.

[0082] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for identifying and positioning green husk nuts based on image recognition, characterized in that: The method comprises the following steps: Step 1: Install a camera mobile shooting device on the nut and green fruit picking cart; Step 2: Drive the nut picking cart to the front of the nut tree, and the camera shoots the nut tree at the initial position of the mobile shooting device to obtain an initial image; Step 3: Perform object contour recognition on the initial image to obtain the initial recognized position points of several nut green-skinned fruit points, and mark several points as recognition points; Step 4: Construct a two-dimensional coordinate plane, paste the initial image on the two-dimensional coordinate plane, obtain the coordinate position of the nut green peel fruit and the coordinates of the identification point, and set the center point; Step 5: The camera moves on the mobile shooting device, takes pictures after each fixed distance to obtain different direction images, and then summarizes the direction images of all points to obtain a direction image set; Step 6: Fit each image in the directional image set to the two-dimensional coordinate plane according to the identification points to obtain several layers of image sets, and then identify the fruit position coordinates of the green-skinned nuts in each layer of image; The positions of the recognition points of several layers of image sets are all overlapped, and then the arc-shaped outline of the nut green peel fruit in each image is identified as the position of the nut green peel fruit, and the detailed coordinate information of each nut green peel fruit position is displayed; Step 7: Summarize the fruit position coordinates of all images into a coordinate plane, and keep the position coordinates of each fruit unchanged, so as to realize the recognition and positioning of green-skinned nuts.

2. The method for identifying and positioning the pericarp of nuts based on image recognition according to claim 1, wherein: In step 1, the camera moving shooting device includes a camera, a sliding chassis and a sliding frame, the camera is set on the sliding chassis, the sliding chassis is clamped in the sliding frame and can be movably set, the sliding frame includes a vertical slide rail (3) and a horizontal slide rail (4), the vertical slide rail (3) and the horizontal slide rail (4) are vertically arranged to cross, an initial positioning point (5) is arranged inside the intersection, an initial positioning point sensing switch is arranged at the bottom of the sliding chassis, when the initial positioning point (5) and the initial positioning point sensing switch are on the same straight line, it means that the camera is in the initial position, the sliding chassis is connected to an external control device, and moves and pauses according to the signal of the control device, the camera is connected to the control device, and transmits the collected picture to the control device for processing, and the vertical slide rail (3) and the horizontal slide rail (4) are set as a retractable structure, which is extended or shortened according to the size of the nut tree to be picked.

3. The method for identifying and positioning the green husk of nuts based on image recognition according to claim 1, characterized in that: In step 2, the nut green fruit picking cart comes to the front of the nut tree, and the camera recognizes the external wheel library of the nut tree image, identifies the height and width of the nut tree, and then moves the camera mobile shooting device up and down so that the center position of the camera mobile shooting device is aligned with the center position of the nut tree, and then the camera moves to the initial positioning point (5) of the mobile shooting device, and then shoots the nut tree to obtain the initial image of the nut tree.

4. A method for identifying and positioning the green husk of nuts based on image recognition according to claim 1, characterized in that: In step 3, the initial image is initialized, and then the objects inside the image are subjected to contour recognition, including leaf contours, round contours of the pericarp of nuts, and branch contours. When a circular arc contour is recognized, it is considered as the fruit of the pericarp of nuts, and the positions of several nuts with pericarp are obtained. At the same time, several points are selected as recognition points at the edge and the midline of the nut tree in the initial image. The recognition points are used for position coordinate correction during subsequent highlighting and fitting. Due to the change in the shooting angle, image position stretching or shrinking correction needs to be performed after shooting, and the recognition of all images is coincident during fitting.

5. A method for identifying and positioning the green husk of nuts based on image recognition according to claim 1, characterized in that: In step 5, the camera moves along the sliding frame at the center of the camera moving shooting device. First, it moves to the upper part of the vertical slide rail (3). After moving a fixed position each time, it pauses, then notifies the camera to take a picture, then records the specific position of the captured image, then moves up a fixed position and takes another picture until it reaches the top and returns to the initial point, then moves down to take pictures, and then moves left and right to take pictures. All the captured direction images are collected to obtain a direction image set.

6. The method for identifying and positioning the pericarp of nuts based on image recognition according to claim 1, wherein: In step 6, the positions of the recognition points of several layers of image sets are all coincident. Then, the circular arc contour with nuts in each image is recognized as the fruit position of the nuts with pericarp, and the position of each nut with pericarp shows detailed coordinate information. In step 7, the specific process of summarizing the fruit position coordinates into a coordinate plane is as follows: several layers of images are fused together, and then the same or adjacent fruit position coordinates are superimposed. If there are fruits in the positions of some images and no fruits in other images, the leaf occlusion of the images without fruits is erased, and the positions with fruits in all images are shown to obtain the positions of the nuts with pericarp.

7. An automatic picking control method for nut green husk fruits based on image recognition, characterized in that: Using the nut with pericarp recognition and positioning method of any one of claims 1-6 to position the nuts, and then a picking temporary correction camera is installed at the front end of the manipulator of the picking trolley. According to the position of each nut with pericarp, the control device calculates the degree of freedom of rotation required for the manipulator to pick the nut. When the manipulator comes to the nut with pericarp, the temporary correction camera collects the optimal picking position in real time, and then the manipulator picks the nut. When the manipulator picks and leaves, the temporary correction camera collects whether there are still unpicked nuts at the just-picked place. If not, they are deleted in the fruit position plane of the nuts. If so, the manipulator is continuously controlled for secondary picking until all are picked, and the fruit coordinates in the fruit position plane are all deleted. Then the picking trolley moves to the other side to continue picking the nuts, and each nut tree can be picked after picking the four sides.

8. The automatic picking control method of nut green husk fruits based on image recognition according to claim 7, characterized in that: The specific process of calculating the degrees of freedom of rotation required for the manipulator by the control device is as follows: Set H, L, and P to represent the height from the ground to the base of the robotic arm, the length of the long arm, and the length of the short arm respectively. Let m be the central height of the nut to be picked. The angle between the long arm and the short arm remains 90 degrees. Rotate the long arm so that the center of the camera is collinear with the target center. After collinearity, return the current base rotation angle θ and the long arm rotation angle θ1. Using the trigonometric theorem and the Pythagorean theorem, the length of the imaging ranging S can be obtained. The specific derivation formulas are shown in Equations (1), (2), (3), (4), and (5): T = P·tanθ1 + L (1) After obtaining S, by rotating the angle θ, decompose S into vectors. Using the sine and cosine function calculation formulas, obtain the x-axis coordinate X, y-axis coordinate Y, and z-axis coordinate Z with the robotic arm itself as the base coordinate. The specific formulas are shown in Equations (6), (7), and (8): Z = 2·m (6) After the camera analyzes the relative true coordinates of the target picked nut and the manipulator, the servo rotation angles of each degree of freedom of the robotic arm are solved through the inverse solution algorithm. A three-degree-of-freedom robotic arm is used, and the trigonometric function geometric method is used to solve the inverse kinematics formula of the robotic arm. Given that the true coordinates of the target nut are (X, Y, Z), the long arm of the robotic arm is L1, the short arm is L2, and the gripper connection part is L3. The rotation angles of the three axes to be solved are respectively As can be seen from the inverse trigonometric functions, in a right triangle, The formula is as shown in Equation (9): Establish a geometric model of the robotic arm in the YOZ plane of the robotic arm. Using the cosine theorem in trigonometry, Equations (10), (11), and (12) can be obtained: Using the Pythagorean theorem of a right triangle, the expressions of T1 and T2 are shown in Equations (13) and (14): Using the inverse cosine formula in trigonometric functions, the respective angles can be obtained. The specific expressions are shown in Equations (15), (16), (17), and (18) as follows: After obtaining the rotation angles of the three axes as respectively, control the manipulator to perform the picking action.

9. An automatic picking device for identifying and positioning nut green husk fruits based on image recognition, using the nut green husk fruit identification and positioning method according to any one of claims 1-6, characterized in that: It includes a picking vehicle body, a mobile shooting device, an identification and positioning camera, a picking manipulator, a picking correction camera, and a data analysis and control device. The mobile shooting device, the picking manipulator, and the data analysis and control device are all arranged on the picking vehicle body. The mobile shooting device is arranged at the front end. The identification and positioning camera is slidably arranged at the upper end of the mobile shooting device. The picking vehicle body, the identification and positioning camera, the picking manipulator, and the picking correction camera are all connected to the data analysis and control device.

10. Application of the nut green peel fruit recognition and positioning method based on image recognition according to any one of claims 1-6, characterized in that: The identification and positioning method is applied to automatic picking and identification and positioning of oranges, automatic picking and identification and positioning of walnuts, automatic picking and identification and positioning of snow pears, or automatic picking and identification and positioning of apples.

Images