Nut green peel fruit time positioning method based on multi-sensor fusion

Abstract

The invention provides a nut green-peel fruit positioning method based on multi-sensor fusion, and belongs to the technical field of nut green-peel fruit positioning, and the method comprises the following steps: carrying out the initial imaging positioning of a nut green-peel fruit through a laser radar, constructing a three-dimensional space position model of the nut green-peel fruit according to the position of the nut green-peel fruit, and carrying out the positioning of the three-dimensional space position model of the nut green-peel fruit; a distance sensor is installed on a picking manipulator and used for measuring the distance between the manipulator and an object at the front end in real time, and a picking three-dimensional space model is constructed according to the fixed position of the picking manipulator, the position of a nut collecting box and a nut green peel fruit three-dimensional space position model. A camera is used for observing the state of the mechanical arm, the state of the nut collecting box and picking environment condition data in the picking three-dimensional space model in real time, and the real-time position of the next nut needing to be picked is calculated according to the state and position of the mechanical arm. Firstly, a laser radar is used for carrying out multidirectional positioning on nut green peel fruits, so that the trouble of image recognition caused by green leaves is avoided.

Description

Technical Field

[0001] The invention relates to the technical field of locating green peel fruits of nuts, and in particular to a locating method of green peel fruits of nuts based on multi-sensor fusion. Background Art

[0002] Macadamia nuts are generally harvested manually or by mechanical vibration. Manual harvesting is when the fruit is ripe, using fruit hooks to peel off the involucres one by one. When using mechanical harvesting, the fruit trees are sprayed with ethephon two weeks before harvesting to accelerate ripening, and then the branches are vibrated mechanically to make the ripe fruits fall. The use of ethephon will cause the leaves to fall off early, which will affect the growth of the fruit trees and the yield of the following year. At present, most of the methods used in China are mechanical vibration, or manipulators are used for assisted harvesting. However, the manipulator picking does not have the function of automatically locating and identifying the green-skinned fruits of nuts, so the picking efficiency is low, the speed is slow, and the effect is not good.

[0003] Since the color of the green-skinned nuts is very similar to that of the leaves, it is difficult to identify them when using image recognition. Since the colors of the fruits and leaves are similar, the difficulty of identification is increased, and the position of the green-skinned nuts cannot be accurately identified, which makes it very difficult to use robotic arms to pick them and the efficiency is low.

[0004] At the same time, when picking, not only do we need to identify the green-skinned nuts in advance, but we also need to locate the green-skinned nuts that need to be picked in real time during the picking process. Only through real-time positioning can the efficiency of picking be improved, and at the same time, the picking can be carried out in an orderly manner without missing any fruits or repeated positioning. Summary of the invention

[0005] The purpose of the present invention is to provide a method for positioning green-skinned fruits of nuts based on multi-sensor fusion, so as to solve the technical problem that the existing green-skinned fruit robot picking process of nuts cannot perform real-time positioning of the nuts to be picked, resulting in lower picking efficiency.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A method for locating green-skinned fruits of nuts based on multi-sensor fusion, the method comprising the following steps:

[0008] Step 1: Use LiDAR to initially image and locate the green-skinned fruit of the nut;

[0009] Step 2: construct a three-dimensional spatial position model of the green-skinned fruit of the nut according to the position of the green-skinned fruit of the nut;

[0010] Step 3: Install a distance sensor on the picking robot to measure the distance between the robot and the front object in real time;

[0011] Step 4: Construct a picking three-dimensional space model according to the fixed position of the picking manipulator, the position of the nut collection box, and the three-dimensional space position model of the nut green peel fruit;

[0012] Step 5: Use the camera to observe the robot status, nut collection box status and picking environment status data in the picking three-dimensional space model in real time;

[0013] Step 6: Based on the state and position of the robot, calculate the real-time position of the next nut to be picked to facilitate picking by the robot.

[0014] Furthermore, the specific process of step 1 is:

[0015] Step 1.1: Set up the longitudinal and lateral movement devices of the laser radar;

[0016] Step 1.2: The laser radar is installed on the picker and brought to the nut tree. The laser radar first obtains the initial three-dimensional image of the nut range;

[0017] Step 1.3: Frame the initial 3D image and then mark several feature points as identification points;

[0018] Step 1.4: Construct a three-dimensional coordinate system for the framed area, set the identification center point, and then mark the coordinates of all points of the nut tree;

[0019] Step 1.5: The laser radar moves on the longitudinal and lateral moving devices, pausing after each point of movement. The laser radar scans to obtain a directional stereo image, and then obtains all the three-dimensional images and summarizes them to obtain a three-dimensional image set of the nut tree in different directions;

[0020] Step 1.6: All three-dimensional image sets are fused to obtain three-dimensional coordinate data sets of nut trees in different directions;

[0021] Step 1.7: Then find out the positions of all nut shapes from the nut tree three-dimensional coordinate data set, and output the three-dimensional position points of the nuts to achieve imaging positioning of the nuts.

[0022] Furthermore, in step 2, a three-dimensional position space is constructed based on the position of each nut green-skinned fruit, and the three-dimensional position coordinates of the nut are marked in the three-dimensional space. Then, the three-dimensional position coordinates of all the nut green-skinned fruits are summarized to obtain a three-dimensional space position model of the nut green-skinned fruit.

[0023] Furthermore, in step 3, three distance sensors are installed on the picking robot, one distance sensor is installed at the front end of the picking robot, and is used to detect the position of objects in front of the front end of the robot in real time, and the other two sensors are installed on both sides of the picking robot and are tilted to detect the position of objects at the front ends of both sides of the picking robot.

[0024] Furthermore, in step 4, the rear end of the nut collection box is taken as the rear edge, and then the lowest installation position of the picking robot is taken as the bottom edge, the horizontal plane of the highest nut position in the three-dimensional space position model of nuts and green peel fruits is taken as the top of the space, and the vertical plane at the front end of the three-dimensional space position model of nuts and green peel fruits is taken as the front end. A three-dimensional space box structure is formed according to the rear edge, bottom edge, top end and front end, and then the position of the three-dimensional space box structure in which the picking robot is located during movement is marked to form a three-dimensional space model for picking.

[0025] Furthermore, in step 5, the camera is used to collect whether the nut collection box is full, and then notify relevant personnel to replace the nut collection box. At the same time, the image is detected to see if there are people within the movement range of the picking robot to avoid accidental injury to people by the robot. The camera is based on the position of the picking robot in real time. When the picking robot comes in front of the nut bunch, it uses image recognition to identify the position of the nut bunch, calibrates the position of the picking robot's clamping, and improves the picking efficiency. At the same time, when the picking robot passes through the leaves and pushes the original leaves away, the camera identifies in real time whether there are nuts inside that have not been located and identified, and realizes supplementary identification during the picking process to avoid missing nuts.

[0026] Furthermore, in step 6, the camera is used to determine whether the picking robot is in an open or closed state, and to identify whether the nuts are picked successfully. When the picking is unsuccessful, the position just picked is picked again, and the straight-line distance between each nut and the nut collection box in the picking three-dimensional space model is calculated. The distances are then sorted to obtain the order in which the nuts are picked. When the picking robot picks the nuts and puts them back into the nut collection box, the radar distance detection transmits the position of the next nut to be picked to the robot control device, and at the same time generates the control rotation dimension of the robot, thereby realizing real-time positioning of the nuts and improving the picking efficiency.

[0027] The present invention has the following beneficial effects due to the adoption of the above technical solution:

[0028] The present invention first uses laser radar to perform multi-directional positioning of green-skinned nuts, avoiding the trouble caused by the green color of leaves on image recognition, and constructs the movement space of the picking robot and the position space of the green-skinned nuts into a three-dimensional structure space, realizing the distance between the collection storage point and each nut, and realizing the robot control program required to calculate each nut while picking, saving data processing time and improving picking efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the method of the present invention;

[0030] Figure 2 This is a flow chart of initial imaging and positioning of green-skinned nuts by the laser radar of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and preferred embodiments. However, it should be noted that many details listed in the specification are only for the purpose of enabling the reader to have a thorough understanding of one or more aspects of the present invention, and these aspects of the present invention can be implemented even without these specific details.

[0032] like Figure 1 As shown, a method for locating green-skinned fruits of nuts based on multi-sensor fusion comprises the following steps:

[0033] Step 1: Use LiDAR to initially image and locate the green-skinned fruit of nuts. LiDAR can effectively avoid the influence of color errors caused by green leaves. The specific process is as follows Figure 2 As shown:

[0034] Step 1.1: Set up the longitudinal and transverse movement devices of the laser radar. The longitudinal and transverse movement devices include transverse sliders and longitudinal sliders, which are cross-arranged. The laser radar is provided with a moving wheel device, which can switch between the transverse slider and the longitudinal slider. At the same time, the moving wheel device is provided with a timing movement control device, which is used to control the moving speed, moving position and dwelling time of the moving wheel device. The timing movement control device is connected to the laser radar and the external harvester respectively, and is used to receive the movement control signal and notify the laser radar to scan.

[0035] The moving wheel device is set as a square bottom plate, and four rotatable pulleys are set on the four sides of the square bottom plate. When longitudinal movement is required, the pulleys on both sides of the longitudinal direction rotate to the back of the transverse slide bar for sliding, while the two pulleys at the upper and lower ends are retracted. When transverse movement is required, the opposite is true, realizing longitudinal and transverse movement. The laser radar is a 192-line laser radar.

[0036] Step 1.2: The laser radar is installed on the picker and comes to the nut tree. The laser radar first obtains the initial three-dimensional image of the nut range. When the laser radar obtains the initial three-dimensional image, the picker uses the laser radar for height recognition, and then determines the height of the nut green-skin fruit tree. The picker aligns the center of the longitudinal and transverse moving device with the middle position of the nut green-skin fruit tree. The laser radar stops at the intersection of the transverse slide bar and the longitudinal slide bar of the longitudinal and transverse moving device. The laser radar starts to scan the initial three-dimensional stereo image data of the nut green-skin fruit tree, and uses the obtained initial three-dimensional stereo image as the nut green-skin fruit positioning position map.

[0037] When picking is needed, you can go to the direction in front of the nut tree, identify and locate it first, and then move the whole machine and use the robot to move and pick the fruits. Basically, to pick the fruits from a nut tree, you only need to identify the four directions of east, west, south and north, and then pick them four times respectively to complete the positioning of the nuts on a tree. After identifying one side, pick one side first.

[0038] Step 1.3: Frame the initial three-dimensional image, and then mark several feature points as identification points. When framing the initial three-dimensional image, use a square frame to frame it, and then the four sides of the square frame are close to the edge of the nut green-skinned fruit tree, and then obtain several edge points and three internal midpoints from the three-dimensional image of the nut green-skinned fruit tree as feature identification points. The feature identification points serve as reference points for the subsequent fusion of several images. During fusion, the points in each three-dimensional image that are the same or similar to the feature identification points are used as coincidence points to fit together, so that the corresponding coordinate points can be guaranteed to correspond when all three-dimensional images are fused. The feature identification points become the secondary calibration of the coordinate annotation of the subsequent image fusion to avoid too large deviations in the coordinate positions between images.

[0039] Step 1.4: Construct a three-dimensional coordinate system for the framed area, set the recognition center point, and then annotate the coordinates of all points on the nut tree. Establish a three-dimensional coordinate system, then use the lower left corner of the framed three-dimensional image as the origin, annotate the three-dimensional coordinate data for each point on the three-dimensional image, and use the internal center point of the frontmost X-axis and Y-axis planes as the recognition center point. The recognition center point is also used as the reference coincidence point for image fusion. The recognition center point is used as the secondary positioning and verification of the recognition, making the annotation of all images more accurate.

[0040] Step 1.5: The laser radar moves on the longitudinal and transverse moving devices, and pauses after each point is moved. The laser radar scans to obtain a directional stereo image, and then obtains all three-dimensional images to summarize to obtain a three-dimensional stereo image set of the nut tree in different directions. The laser radar first starts to move from the intersection of the transverse slider and the longitudinal slider, and first moves to the upper half of the longitudinal slider. After each fixed position is moved, it pauses, and then notifies the laser radar to scan a three-dimensional image of the nut green peel fruit tree. After scanning, it moves to a fixed position online, and then scans until it moves to the top and then returns to the intersection of the transverse slider and the longitudinal slider, and then moves down and scans to obtain several three-dimensional stereo images of vertical movement. The laser radar returns to the intersection of the transverse slider and the longitudinal slider, and then moves and scans to both sides in turn to obtain several three-dimensional stereo images of horizontal movement. The several three-dimensional stereo images of vertical movement and the several three-dimensional stereo images of horizontal movement are summarized to obtain a three-dimensional stereo image set.

[0041] When moving, generally after moving 3-5cm, scanning is performed to capture a three-dimensional image, first moving upward, then downward, then to the left, and then to the right.

[0042] Step 1.6: All three-dimensional image sets are fused to obtain three-dimensional coordinate data sets of nut trees in different directions. Each image in the three-dimensional image set is fitted and fused into the initial three-dimensional image. When fusion is performed, it is first used as several layers of fused images, and each image does not affect each other where it is occluded. Then, the three-dimensional coordinate data set is annotated for each layer of image. This fusion does not involve coordinate data fusion, but only superposition as layer fusion, which is a fusion of length and width directions, so that the three-dimensional coordinate data pairs of all images correspond to each other and there will not be too much deviation.

[0043] Step 1.7: Then find the positions of all nut shapes from the nut tree three-dimensional coordinate data set, and output the three-dimensional position points of the nuts to achieve imaging positioning of the nuts. Identify the nut green peel fruit model image in each layer of the three-dimensional image, and then collect the position coordinates of the nut green peel fruit model image, summarize the position coordinates of the nut green peel fruit model image in all layers of images to obtain the position set coordinates of the nut shape, and then compare each layer of image with the position set coordinates. When the image of a layer overlaps with the position set coordinates and the position does not appear in the nut green peel fruit model image, delete the image of the layer area, and process all layers of images in turn to remove the leaves and branches that block the nuts, and then fully fuse all images twice to obtain the position points of the nut model image, so as to filter out the blocked leaves or branches, and locate the nuts more accurately.

[0044] This time, the occluded part is deleted by fusing the coordinate data and image pixels, and the occluded object is filtered out. This is the "perspective" effect, which can better realize the positioning and identification of nuts and green fruits and avoid missed identification.

[0045] This method moves the position of the laser radar to image the green-skinned nuts in different directions, and recognizes the shape of the nuts at different angles. It can effectively avoid the inability to recognize the nuts due to the obstruction of leaves or branches in front, and realize more effective and accurate imaging and positioning. After recognition, the obstructed leaves or branches are deleted through image fusion, so that the nuts behind can be directly exposed, and the obstructions can be automatically filtered out. At the same time, the position of the nuts will not change, making it easier to locate when using a robot to pick them.

[0046] Step 2: Construct a three-dimensional spatial position model of nuts and green peel fruits according to the positions of nuts and green peel fruits. Construct a three-dimensional position space according to the position of each nut and green peel fruit, and mark the three-dimensional position coordinates of the nuts in the three-dimensional space. Then, summarize the three-dimensional position coordinates of all nuts and green peel fruits to obtain a three-dimensional spatial position model of nuts and green peel fruits.

[0047] Step 3: Install distance sensors on the picking manipulator to measure the distance between the manipulator and the object in front in real time. There are three distance sensors installed on the picking manipulator. One distance sensor is installed at the front end of the picking machine to detect the position of the object in front of the front end of the manipulator in real time. The other two sensors are installed on both sides of the picking manipulator and are tilted to detect the position of the objects in front of both sides of the picking manipulator.

[0048] Step 4: Construct a three-dimensional picking model based on the fixed position of the picking manipulator, the position of the nut collection box, and the three-dimensional spatial position model of the nut green peel fruit. Take the rear end of the nut collection box as the rear edge, and then take the lowest installation position of the picking manipulator as the bottom edge. Take the horizontal plane of the highest nut position in the three-dimensional spatial position model of the nut green peel fruit as the top of the space, and take the vertical plane at the front end of the three-dimensional spatial position model of the nut green peel fruit as the front end. The three-dimensional box structure is formed according to the rear edge, bottom edge, top and front end, and then mark the position of the three-dimensional box structure of the picking manipulator during the movement process to form a three-dimensional picking model.

[0049] Step 5: Use the camera to observe the robot status, nut collection box status and picking environment data in the picking three-dimensional space model in real time. The camera is used to collect whether the nut collection box is full, and then notify the relevant personnel to replace the nut collection box. At the same time, the image is used to detect whether there are people within the action range of the picking robot to avoid accidental injury by the robot. The camera is based on the position of the picking robot in real time. When the picking robot comes to the nut string, it uses image recognition to identify the position of the nut string, and calibrates the position of the picking robot to improve the efficiency of picking. At the same time, when the picking robot passes through the leaves and pushes the original leaves away, the camera recognizes in real time whether there are nuts inside that have not been located and identified, and realizes supplementary identification during the picking process to avoid missing nuts.

[0050] Step 6: According to the state and position of the manipulator, calculate the real-time position of the next nut to be picked, so that the manipulator can pick it conveniently. The camera determines whether the picking manipulator is in an open or closed state, and identifies whether the nut picking is successful. If the picking is unsuccessful, the position just picked is picked again. At the same time, the straight-line distance between each nut and the nut collection box in the picking three-dimensional space model is calculated, and then the distances are sorted to obtain the order of picking nuts. When the picking manipulator picks nuts and puts them back into the nut collection box, the radar distance detection transmits the position of the next nut to be picked to the manipulator control device, and generates the control rotation dimension of the manipulator at the same time, so as to realize the real-time positioning of the nuts and improve the efficiency of picking.

[0051] By placing all the nuts in a set space, the order and path of picking can be better planned to avoid missing unpicked nuts during the picking process. The camera and distance sensor are used in conjunction to make nut picking more efficient. At the same time, secondary internal supplementary recognition can be achieved, so that fruits that are tightly blocked by leaves can also be recognized for a second time, resulting in better picking effects.

[0052] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for locating green-skinned nuts and fruits based on multi-sensor fusion, characterized in that: The method comprises the following steps: Step 1: Use LiDAR to initially image and locate the green-skinned fruit of the nut; Step 2: construct a three-dimensional spatial position model of the green-skinned fruit of the nut according to the position of the green-skinned fruit of the nut; Step 3: Install a distance sensor on the picking robot to measure the distance between the robot and the front object in real time; Step 4: Construct a picking three-dimensional space model according to the fixed position of the picking manipulator, the position of the nut collection box, and the three-dimensional space position model of the nut green peel fruit; Step 5: Use the camera to observe the robot status, nut collection box status and picking environment status data in the picking three-dimensional space model in real time; Step 6: Based on the state and position of the robot, calculate the real-time position of the next nut to be picked to facilitate picking by the robot.

2. The method for locating green-skinned fruits of nuts based on multi-sensor fusion according to claim 1, characterized in that: The specific process of step 1 is: Step 1.1: Set up the longitudinal and lateral movement devices of the laser radar; Step 1.2: The laser radar is installed on the picker and brought to the nut tree. The laser radar first obtains the initial three-dimensional image of the nut range; Step 1.3: Frame the initial 3D image and then mark several feature points as identification points; Step 1.4: Construct a three-dimensional coordinate system for the framed area, set the identification center point, and then mark the coordinates of all points of the nut tree; Step 1.5: The laser radar moves on the longitudinal and lateral moving devices, pausing after each point of movement. The laser radar scans to obtain a directional stereo image, and then obtains all the three-dimensional images and summarizes them to obtain a three-dimensional image set of the nut tree in different directions; Step 1.6: All three-dimensional image sets are fused to obtain three-dimensional coordinate data sets of nut trees in different directions; Step 1.7: Then find out the positions of all nut shapes from the nut tree three-dimensional coordinate data set, and output the three-dimensional position points of the nuts to achieve imaging positioning of the nuts.

3. The method for locating green-skinned fruits of nuts based on multi-sensor fusion according to claim 1, characterized in that: In step 2, a three-dimensional position space is constructed based on the position of each nut green fruit, and the three-dimensional position coordinates of the nut are marked in the three-dimensional space. Then, the three-dimensional position coordinates of all the nuts and green fruits are summarized to obtain a three-dimensional space position model of the nuts and green fruits.

4. The method for locating green-skinned fruits of nuts based on multi-sensor fusion according to claim 1, characterized in that: In step 3, three distance sensors are installed on the picking robot. One distance sensor is installed at the front end of the picking robot to detect the position of objects in front of the front end of the robot in real time. The other two sensors are installed on both sides of the picking robot and are tilted to detect the position of objects at the front ends of both sides of the picking robot.

5. The method for locating green-skinned nuts and fruits based on multi-sensor fusion according to claim 1, characterized in that: In step 4, the rear end of the nut collection box is taken as the rear edge, and then the lowest installation position of the picking robot is taken as the bottom edge, the horizontal plane of the highest nut position in the three-dimensional space position model of nuts and green peel fruits is taken as the top of the space, and the vertical plane at the front end of the three-dimensional space position model of nuts and green peel fruits is taken as the front end. A three-dimensional space box structure is formed according to the rear edge, bottom edge, top end and front end, and then the position of the three-dimensional space box structure in which the picking robot is located during movement is marked to form a three-dimensional space model for picking.

6. The method for locating green-skinned fruits of nuts based on multi-sensor fusion according to claim 1, characterized in that: In step 5, the camera is used to collect whether the nut collection box is full, and then notify the relevant personnel to replace the nut collection box. At the same time, the image is detected to see if there are people within the action range of the picking robot to avoid accidental injury by the robot. The camera is based on the position of the picking robot in real time. When the picking robot comes in front of the nut bunch, it uses image recognition to identify the position of the nut bunch, calibrates the position of the picking robot's grip, and improves the picking efficiency. At the same time, when the picking robot passes through the leaves and pushes the original leaves away, the camera identifies in real time whether there are nuts inside that have not been located and identified, and realizes supplementary identification during the picking process to avoid missing nuts.

7. The method for locating green-skinned nuts and fruits based on multi-sensor fusion according to claim 1, characterized in that: In step 6, the camera is used to determine whether the picking robot is in an open or closed state, and to identify whether the nut picking is successful. If the picking is not successful, the position just picked is picked again. At the same time, the straight-line distance between each nut and the nut collection box in the picking three-dimensional space model is calculated, and then the distances are sorted to obtain the picking order of the nuts. When the picking robot picks the nuts and puts them back into the nut collection box, the radar distance detection transmits the position of the next nut to be picked to the robot control device, and at the same time generates the control rotation dimension of the robot, so as to realize real-time positioning of the nuts and improve the picking efficiency.

Images