Nut green peel fruit imaging positioning method based on laser radar

Abstract

The invention provides a nut green fruit imaging and positioning method based on a laser radar, and belongs to the technical field of nut green fruit imaging and positioning, and the method comprises the following steps: setting a longitudinal and transverse moving device of the laser radar, initially obtaining an initial three-dimensional image of a nut range by the laser radar, framing the initial three-dimensional image, and obtaining a final three-dimensional image of the nut range; then marking a plurality of feature points as identification points, constructing a three-dimensional coordinate system for the framed area, then performing coordinate marking on all points of the nut tree, moving the laser radar on the longitudinal and transverse moving devices, scanning by the laser radar to obtain three-dimensional images in a plurality of directions, and summarizing to obtain three-dimensional image sets of the nut tree in different directions; and fusing all the three-dimensional image sets to obtain a three-dimensional coordinate data set of the nut tree, finding out positions of all nut shapes, and outputting three-dimensional position points of the nuts. According to the method, the shielded nuts can be better displayed, and positioning is more accurate.

Description

Technical Field

[0001] The invention relates to the technical field of imaging and positioning of green-skinned nuts, and in particular to an imaging and positioning method of green-skinned nuts based on laser radar. Background Art

[0002] Macadamia nuts are generally harvested manually or by mechanical vibration. Manual harvesting is when the fruit is ripe, using fruit hooks to peel off the involucres one by one. When using mechanical harvesting, the fruit trees are sprayed with ethephon two weeks before harvesting to accelerate ripening, and then the branches are vibrated mechanically to make the ripe fruits fall. The use of ethephon will cause the leaves to fall off early, which will affect the growth of the fruit trees and the yield of the following year. At present, most of the methods used in China are mechanical vibration, or manipulators are used for assisted harvesting. However, the manipulator picking does not have the function of automatically locating and identifying the green-skinned fruits of nuts, so the picking efficiency is low, the speed is slow, and the effect is not good.

[0003] Since the color of nuts and green fruits is very similar to the color of leaves, it is difficult to identify them when using image recognition. Since the colors of fruits and leaves are similar, the difficulty of identification is increased, and the position of nuts and green fruits cannot be accurately identified, which makes it very difficult to use manipulators to pick them, and the efficiency is low. Therefore, it is necessary to design a nut green fruit imaging and positioning method based on laser radar. Summary of the invention

[0004] The purpose of the present invention is to provide a method for imaging and locating green-skinned fruits of nuts based on laser radar, so as to solve the technical problems that green-skinned fruits of nuts are difficult to identify and inaccurately positioned. The method can accurately and quickly locate green-skinned fruits of nuts, and provide green-skinned fruits identification and position positioning for picking with a robot.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A laser radar-based imaging and positioning method for green-skinned nuts, the method comprising the following steps:

[0007] Step 1: Set up the longitudinal and lateral movement devices of the laser radar;

[0008] Step 2: The laser radar is installed on the picker and brought to the nut tree. The laser radar first obtains the initial three-dimensional image of the nut range;

[0009] Step 3: Frame the initial 3D image and then mark several feature points as identification points;

[0010] Step 4: Construct a three-dimensional coordinate system for the framed area, set the identification center point, and then mark the coordinates of all points of the nut tree;

[0011] Step 5: The laser radar moves on the longitudinal and lateral moving devices, pausing after each point, and the laser radar scans to obtain a directional stereo image, and then obtains all the three-dimensional images and summarizes them to obtain a three-dimensional image set of the nut tree in different directions;

[0012] Step 6: All three-dimensional image sets are fused to obtain three-dimensional coordinate data sets of nut trees in different directions;

[0013] Step 7: Then find the positions of all nut shapes from the nut tree 3D coordinate data set, and output the 3D position points of the nuts to achieve imaging positioning of the nuts.

[0014] Furthermore, in step 1: the longitudinal and transverse moving devices include a transverse slide bar and a longitudinal slide bar, the transverse slide bar and the longitudinal slide bar are cross-arranged, a moving wheel device is arranged on the laser radar, the moving wheel device can switch and move between the transverse slide bar and the longitudinal slide bar, and a timing movement control device is arranged on the moving wheel device, the timing movement control device is used to control the moving speed, moving position and staying time of the moving wheel device, the timing movement control device is respectively connected to the laser radar and the external picking machine, and is used to receive the movement control signal and notify the laser radar to scan.

[0015] Furthermore, in step 2, when the laser radar obtains the initial three-dimensional image, the picker uses the laser radar for height recognition and then determines the height of the nut green-skinned fruit tree. The picker aligns the center of the longitudinal and transverse moving devices with the middle position of the nut green-skinned fruit tree, and the laser radar stops at the intersection of the transverse slide bar and the longitudinal slide bar of the longitudinal and transverse moving devices. The laser radar starts to scan the initial three-dimensional stereo image data of the nut green-skinned fruit tree, and uses the acquired initial three-dimensional stereo image as the nut green-skinned fruit positioning position map.

[0016] Furthermore, in step 3, when the initial three-dimensional image is framed, a square is used for framing, and then the four sides of the square are close to the edge of the nut green-skinned fruit tree, and then several edge points and three internal midpoints are obtained from the three-dimensional image of the nut green-skinned fruit tree as feature recognition points. The feature recognition points are used as reference points for the subsequent fusion of several images. During the fusion, the points in each three-dimensional image that are the same or similar to the feature recognition points are used as overlapping points to fit together, so that the corresponding coordinate points can be guaranteed to correspond when all three-dimensional images are fused.

[0017] Furthermore, in step 4, a three-dimensional coordinate system is established, and then the lower left corner of the framed three-dimensional image is taken as the origin, and the three-dimensional coordinate data is correspondingly annotated for each point on the three-dimensional image. The internal center point of the front-end X-axis and Y-axis planes is used as the recognition center point, and the recognition center point is also used as the reference coincidence point for image fusion.

[0018] Furthermore, in step 5, the laser radar first starts to move from the intersection of the horizontal slider and the vertical slider, and first moves to the upper part of the vertical slider. After each movement to a fixed position, it pauses, and then notifies the laser radar to scan a three-dimensional stereo image of the nut green-skin fruit tree. After scanning, it moves to a fixed position online, and then scans until it moves to the top and then returns to the intersection of the horizontal slider and the vertical slider, and then moves down and scans to obtain several three-dimensional stereo images of vertical movement. The laser radar returns to the intersection of the horizontal slider and the vertical slider, and then moves and scans to both sides in turn to obtain several three-dimensional stereo images of horizontal movement. The several three-dimensional stereo images of vertical movement and the several three-dimensional stereo images of horizontal movement are summarized to obtain a three-dimensional stereo image set.

[0019] Furthermore, in step 6, each image in the three-dimensional stereo image set is fitted and fused into the initial three-dimensional image. During the fusion, the images are first fused as several layers, and the occluded parts of each image do not affect each other. Then, the three-dimensional coordinate data set is annotated for each layer of the image.

[0020] Furthermore, in step 7, the nut green-skin fruit model image in each layer of the three-dimensional image is identified, and then the position coordinates of the nut green-skin fruit model image are collected, and the position coordinates of the nut green-skin fruit model image in all layers of images are summarized to obtain the position set coordinates of the nut shape, and then each layer of the image is compared with the position set coordinates. When the image of a layer coincides with the position set coordinates and the nut green-skin fruit model image does not appear at the position, the image of the layer area is deleted, and all layers of images are processed in turn to remove the leaves and branches that block the nuts, and then all images are completely fused for a second time to obtain the position points of the nut model image, thereby filtering out the blocked leaves or branches, and locating the nut position more accurately.

[0021] The present invention has the following beneficial effects due to the adoption of the above technical solution:

[0022] The present invention realizes imaging of green-skinned nuts in different directions by moving the position of the laser radar, and recognizes the shapes of nuts at different directions and angles, which can effectively avoid the inability to recognize nuts due to obstruction by leaves or branches directly in front, and realizes more effective and accurate imaging and positioning. At the same time, after recognition, the obstructed leaves or branches are deleted through image fusion, so that the nuts behind can be directly exposed, and the obstructions are automatically filtered out. At the same time, the position of the nuts will not change, making it easier to locate when using a robot arm to pick them. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and preferred embodiments. However, it should be noted that many details listed in the specification are only for the purpose of enabling the reader to have a thorough understanding of one or more aspects of the present invention, and these aspects of the present invention can be implemented even without these specific details.

[0025] like Figure 1 As shown, a method for imaging and locating green-skinned nuts based on laser radar comprises the following steps:

[0026] Step 1: Set up the longitudinal and transverse moving devices of the laser radar. The longitudinal and transverse moving devices include a transverse slider and a longitudinal slider, which are cross-arranged. The laser radar is provided with a moving wheel device, which can switch and move between the transverse slider and the longitudinal slider. At the same time, the moving wheel device is provided with a timing moving control device, which is used to control the moving speed, moving position and staying time of the moving wheel device. The timing moving control device is connected to the laser radar and the external harvester respectively, and is used to receive the moving control signal and notify the laser radar to scan.

[0027] The moving wheel device is set as a square bottom plate, and four rotatable pulleys are set on the four sides of the square bottom plate. When longitudinal movement is required, the pulleys on both sides of the longitudinal direction rotate to the back of the transverse slide bar for sliding, while the two pulleys at the upper and lower ends are retracted. When transverse movement is required, the opposite is true, realizing longitudinal and transverse movement. The laser radar is a 192-line laser radar.

[0028] Step 2: The laser radar is installed on the picker and comes to the nut tree. The laser radar first obtains the initial three-dimensional image of the nut range. When the laser radar obtains the initial three-dimensional image, the picker uses the laser radar for height recognition, and then determines the height of the nut green-skin fruit tree. The picker aligns the center of the longitudinal and transverse moving device with the middle position of the nut green-skin fruit tree. The laser radar stops at the intersection of the transverse slide bar and the longitudinal slide bar of the longitudinal and transverse moving device. The laser radar starts to scan the initial three-dimensional stereo image data of the nut green-skin fruit tree, and uses the obtained initial three-dimensional stereo image as the nut green-skin fruit positioning position map.

[0029] When picking is needed, you can go to the direction in front of the nut tree, identify and locate it first, and then move the whole machine and use the robot to move and pick the fruits. Basically, to pick the fruits from a nut tree, you only need to identify the four directions of east, west, south and north, and then pick them four times respectively to complete the positioning of the nuts on a tree. After identifying one side, pick one side first.

[0030] Step 3: Frame the initial three-dimensional image, and then mark several feature points as identification points. When framing the initial three-dimensional image, use a square frame to frame it, and then the four sides of the square frame are close to the edge of the nut green-skinned fruit tree, and then obtain several edge points and three internal midpoints from the three-dimensional image of the nut green-skinned fruit tree as feature identification points. The feature identification points serve as reference points for the subsequent fusion of several images. During fusion, the points in each three-dimensional image that are the same or similar to the feature identification points are used as coincidence points to fit together, so that the corresponding coordinate points can be guaranteed to correspond when all three-dimensional images are fused. The feature identification points become the secondary calibration of the coordinate annotations of the subsequent image fusion to avoid too large deviations in the coordinate positions between images.

[0031] Step 4: Construct a three-dimensional coordinate system for the framed area, set the recognition center point, and then annotate the coordinates of all points on the nut tree. Establish a three-dimensional coordinate system, then use the lower left corner of the framed three-dimensional image as the origin, annotate the three-dimensional coordinate data for each point on the three-dimensional image, and use the internal center point of the frontmost X-axis and Y-axis planes as the recognition center point. The recognition center point is also used as the reference coincidence point for image fusion. The recognition center point is used as the secondary positioning and verification of the recognition, making the annotation of all images more accurate.

[0032] Step 5: The laser radar moves on the longitudinal and transverse moving devices, and pauses after each point is moved. The laser radar scans to obtain a directional stereo image, and then obtains all the three-dimensional images to summarize to obtain a three-dimensional stereo image set of the nut tree in different directions. The laser radar first starts to move from the intersection of the transverse slider and the longitudinal slider, and first moves to the upper half of the longitudinal slider. After each fixed position is moved, it pauses, and then notifies the laser radar to scan a three-dimensional image of the nut green peel fruit tree. After scanning, it moves to a fixed position online, and then scans until it moves to the top and then returns to the intersection of the transverse slider and the longitudinal slider, and then moves down and scans to obtain several three-dimensional stereo images of vertical movement. The laser radar returns to the intersection of the transverse slider and the longitudinal slider, and then moves and scans to both sides in turn to obtain several three-dimensional stereo images of horizontal movement. The several three-dimensional stereo images of vertical movement and the several three-dimensional stereo images of horizontal movement are summarized to obtain a three-dimensional stereo image set.

[0033] When moving, generally after moving 3-5cm, scanning is performed to capture a three-dimensional image, first moving upward, then downward, then to the left, and then to the right.

[0034] Step 6: Fuse all the 3D image sets to obtain the 3D coordinate data sets of nut trees in different directions. Each image in the 3D image set is fitted and fused into the initial 3D image. When fusion is performed, it is first used as several layers of fused images, and each image does not affect each other where it is occluded. Then, the 3D coordinate data sets of each layer of images are annotated. This fusion does not involve coordinate data fusion, but only superposition as layer fusion, which is a fusion of length and width directions, so that the 3D coordinate data pairs of all images correspond to each other and there will not be too much deviation.

[0035] Step 7: Then find the positions of all nut shapes from the nut tree three-dimensional coordinate data set, and output the three-dimensional position points of the nuts to achieve imaging positioning of the nuts. Identify the nut green peel fruit model image in each layer of the three-dimensional image, and then collect the position coordinates of the nut green peel fruit model image, summarize the position coordinates of the nut green peel fruit model image in all layers of images to obtain the position set coordinates of the nut shape, and then compare each layer of image with the position set coordinates. When the image of a layer overlaps with the position set coordinates and the position is not a nut green peel fruit model image, delete the image of the layer area, and process all layers of images in turn to remove the leaves and branches that block the nuts, and then completely fuse all images twice to obtain the position points of the nut model image, so as to filter out the blocked leaves or branches, and locate the nuts more accurately.

[0036] This time, the occluded part is deleted by fusing the coordinate data and image pixels, and the occluded object is filtered out. This is the "perspective" effect, which can better realize the positioning and identification of nuts and green fruits and avoid missed identification.

[0037] This method moves the position of the laser radar to image the green-skinned nuts in different directions, and recognizes the shape of the nuts at different angles. It can effectively avoid the inability to recognize the nuts due to the obstruction of leaves or branches in front, and realize more effective and accurate imaging and positioning. After recognition, the obstructed leaves or branches are deleted through image fusion, so that the nuts behind can be directly exposed, and the obstructions can be automatically filtered out. At the same time, the position of the nuts will not change, making it easier to locate when using a robot to pick them.

[0038] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A laser radar-based imaging and positioning method for green-skinned nuts, characterized in that: The method comprises the following steps: Step 1: Set up the longitudinal and lateral movement devices of the laser radar; Step 2: The laser radar is installed on the picker and brought to the nut tree. The laser radar first obtains the initial three-dimensional image of the nut range; Step 3: Frame the initial 3D image and then mark several feature points as identification points; Step 4: Construct a three-dimensional coordinate system for the framed area, set the identification center point, and then mark the coordinates of all points of the nut tree; Step 5: The laser radar moves on the longitudinal and lateral moving devices, pausing after each point, and the laser radar scans to obtain a directional stereo image, and then obtains all the three-dimensional images and summarizes them to obtain a three-dimensional image set of the nut tree in different directions; Step 6: All three-dimensional image sets are fused to obtain three-dimensional coordinate data sets of nut trees in different directions; Step 7: Then find the positions of all nut shapes from the nut tree 3D coordinate data set, and output the 3D position points of the nuts to achieve imaging positioning of the nuts.

2. The laser radar-based imaging and positioning method for green-skinned nuts according to claim 1, characterized in that: In step 1: the longitudinal and transverse moving devices include a transverse slide bar and a longitudinal slide bar, the transverse slide bar and the longitudinal slide bar are cross-arranged, a moving wheel device is arranged on the laser radar, the moving wheel device can switch and move between the transverse slide bar and the longitudinal slide bar, and a timing movement control device is arranged on the moving wheel device, the timing movement control device is used to control the moving speed, moving position and staying time of the moving wheel device, the timing movement control device is respectively connected to the laser radar and the external picking machine, and is used to receive the movement control signal and notify the laser radar to scan.

3. The laser radar-based imaging and positioning method for green-skinned nuts according to claim 2, characterized in that: In step 2, when the laser radar obtains the initial three-dimensional image, the picker uses the laser radar for height recognition and then determines the height of the nut green-skinned fruit tree. The picker aligns the center of the longitudinal and transverse moving devices with the middle position of the nut green-skinned fruit tree, and the laser radar stops at the intersection of the transverse slide bar and the longitudinal slide bar of the longitudinal and transverse moving devices. The laser radar starts to scan the initial three-dimensional stereo image data of the nut green-skinned fruit tree, and uses the acquired initial three-dimensional stereo image as the nut green-skinned fruit positioning position map.

4. The laser radar-based imaging and positioning method for green-skinned nuts according to claim 1, characterized in that: In step 3, when framing the initial three-dimensional image, a square is used for framing, and the four sides of the square are close to the edge of the nut green-skinned fruit tree. Then, several edge points and three internal midpoints are obtained from the three-dimensional image of the nut green-skinned fruit tree as feature recognition points. The feature recognition points are used as reference points for the subsequent fusion of several images. During the fusion, the points in each three-dimensional image that are the same or similar to the feature recognition points are used as coincidence points to fit together, so that the corresponding coordinate points can be guaranteed to correspond when all three-dimensional images are fused.

5. The laser radar-based imaging and positioning method for green-skinned nuts according to claim 1, characterized in that: In step 4, a three-dimensional coordinate system is established, and then the lower left corner of the framed three-dimensional image is taken as the origin, and the three-dimensional coordinate data is correspondingly annotated for each point on the three-dimensional image. The internal center point of the front-end X-axis and Y-axis planes is used as the recognition center point, and the recognition center point is also used as the reference coincidence point for image fusion.

6. The laser radar-based imaging and positioning method for green-skinned nuts according to claim 1, characterized in that: In step 5, the laser radar starts to move from the intersection of the horizontal slider and the vertical slider, and moves to the upper part of the vertical slider first. After each fixed position is moved, it pauses, and then notifies the laser radar to scan a three-dimensional stereo image of the nut green-skin fruit tree. After scanning, it moves to a fixed position online, and then scans until it moves to the top and then returns to the intersection of the horizontal slider and the vertical slider, and then moves down and scans to obtain several three-dimensional stereo images of vertical movement. The laser radar returns to the intersection of the horizontal slider and the vertical slider, and then moves and scans to both sides in turn to obtain several three-dimensional stereo images of horizontal movement. The several three-dimensional stereo images of vertical movement and the several three-dimensional stereo images of horizontal movement are summarized to obtain a three-dimensional image set.

7. The laser radar-based imaging and positioning method for green-skinned nuts according to claim 1, characterized in that: In step 6, each image in the three-dimensional stereo image set is fitted and fused into the initial three-dimensional image. When fusion is performed, it is firstly used as several layers of fused images, and each image does not affect each other at the mutually occluded parts. Then, each layer of the image is annotated with a three-dimensional coordinate data set.

8. The laser radar-based imaging and positioning method for green-skinned nuts according to claim 1, characterized in that: In step 7, the nut green-skin fruit model image in each layer of the three-dimensional image is identified, and then the position coordinates of the nut green-skin fruit model image are collected, and the position coordinates of the nut green-skin fruit model image in all layers of images are summarized to obtain the position set coordinates of the nut shape, and then each layer of the image is compared with the position set coordinates. When the image of a layer coincides with the position set coordinates and the nut green-skin fruit model image does not appear, the image of the layer area is deleted, and all layers of images are processed in turn to remove the leaves and branches that block the nuts, and then all images are completely fused for a second time to obtain the position points of the nut model image, thereby filtering out the blocked leaves or branches, and locating the nut position more accurately.

Images