A method for locating lotus picking points based on multi-view observation

Abstract

The present invention discloses a method for locating lotus pod picking points based on multi-view observation, comprising the following steps: using multiple depth cameras to establish a multi-view observation system, collecting lotus pod RGB images and depth data from multiple viewpoints; obtaining detection bounding boxes and masks of lotus pod and lotus stem picking segments under each viewpoint; calculating the three-dimensional coordinates of lotus pod position feature points and lotus stem posture feature points in each viewpoint; performing subordinate pairing of lotus pod position feature points and lotus stem posture feature points in each single viewpoint, completing the calculation of picking information under a single viewpoint; performing subordinate matching of each lotus pod target and the picking information under each single viewpoint, and completing the calculation of lotus pod picking information through the synthesis of multi-view picking information. The present invention utilizes multi-view observation, deep vision and deep learning methods to realize the positioning of picking points and the directional calculation of picking segments facing lotus pods with different postures, effectively solving the problem that the lotus stem picking segment is blocked by the lotus pod itself or other objects under traditional single viewpoint observation and the picking point cannot be calculated.

Description

Technical Field

[0001] The invention belongs to the technical field of machine vision and image processing, and in particular to a lotus picking point positioning method based on multi-viewing angle observation. Background Art

[0002] For a long time, lotus pods have been picked mainly by hand, which is inefficient and faces a labor shortage problem. In recent years, the technology of automatic fruit and vegetable picking has developed rapidly, and the key to realizing automatic picking is to study fast and effective methods for locating fruit or fruit stalk picking points. Existing picking point positioning methods mainly include two methods based on traditional image processing and deep learning. Compared with traditional image processing methods, the picking point positioning method based on deep learning can realize end-to-end detection of fruit and fruit stalk targets, and can use the target detection frame and mask results to provide reliable support for picking point positioning calculation.

[0003] At present, there are few reports on the technology of locating the picking point of lotus pods. Traditional automatic fruit and vegetable picking mostly adopts single-view observation, and usually the recognition of fruit targets and the positioning of picking points can be completed through a single view. However, lotus pods have changeable postures and different directions. Under a single view, the lotus stem picking section is easily blocked by the lotus pod itself or other objects, making it difficult to observe it completely and effectively, and then it is impossible to calculate the picking point position and the lotus stem picking section vector information. Ultimately, it makes it difficult for the picking robot to perform the picking operation, affecting the economy and continuity of automatic picking. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a lotus pod picking point positioning method based on multi-perspective observation, which comprehensively utilizes multi-perspective observation, deep vision, deep learning and other means to realize the effective calculation of the picking point coordinates and picking vectors of the lotus stem picking section with different postures facing the lotus pod.

[0005] To achieve the above object, the present invention provides the following technical solution: a lotus picking point positioning method based on multi-view observation, comprising the following steps:

[0006] Step 1, using a plurality of depth cameras to establish a multi-view observation system, determining the positional relationship between the camera coordinate systems corresponding to each view through calibration; and establishing a main coordinate system in the multi-view observation system;

[0007] Step 2, collecting RGB images and depth data information of the lotus pod from multiple perspectives;

[0008] Step 3, using the target detection model to detect the RGB image, and obtain the detection bounding box and mask data of the lotus pod and lotus stem picking section under each viewing angle; based on the detection bounding box and mask data, establish lotus pod position feature points and lotus stem posture feature points to respectively characterize the lotus pod position and the lotus stem picking section position and posture in the RGB image; calculate the pixel coordinates of the lotus pod position feature points and the lotus stem posture feature points;

[0009] Step 4, combining the pixel coordinates of the lotus pod position feature points and the lotus stem posture feature points and the depth data information in step 2, to obtain the three-dimensional coordinates of each feature point in the camera coordinate system;

[0010] Step 5, performing subordinate pairing of lotus pod position feature points and lotus stem posture feature points in each single perspective, and completing the calculation of picking information in each single perspective, wherein the picking information includes lotus stem picking point coordinates and picking vectors;

[0011] Step 6, convert the picking information data under each single perspective into the main coordinate system; complete the subordinate matching of each lotus pod target and the corresponding picking information under each single perspective in the main coordinate system; perform data synthesis of multi-perspective picking information to form the final picking information and complete positioning.

[0012] Furthermore, in the above-mentioned lotus pod picking point positioning method based on multi-view observation, the lotus pod position feature point in step 3 is one of the centroid point of the lotus pod mask area and the center point of the lotus pod detection bounding box; the lotus stem picking segment is a short lotus stem segment adjacent to the lotus pod.

[0013] Furthermore, in the lotus picking point positioning method based on multi-view observation, the step of calculating the pixel coordinates of the lotus stem posture feature points in step 3 includes:

[0014] Step A1, extracting pixel data within the detection boundary box of the lotus stem picking segment; stepping and extracting pixel points in the mask of the lotus stem picking segment along a direction perpendicular to the short side of the detection boundary box, specifically, the stepping and extracting process is based on the rule of extracting columns in the horizontal direction and extracting rows in the vertical direction;

[0015] Step A2, calculating the centroid of the pixel points in the lotus stem picking segment mask extracted each time in the step extraction process;

[0016] Step A3, using a linear regression method to fit each centroid point into a straight line;

[0017] Step A4, calculating the intersection of the fitting line and the lotus stem picking segment detection boundary box, determining the midpoint O2 and two quartile points O1, O3 of the line segment connected by the two intersection points; and using the midpoint and two quartile points as lotus stem posture feature points.

[0018] Furthermore, in the above-mentioned lotus picking point positioning method based on multi-view observation, the target detection model in step 3 is an instance segmentation model.

[0019] Furthermore, in the lotus pod picking point positioning method based on multi-view observation, the step of performing subordinate pairing of lotus pod position feature points and lotus stem posture feature points in each single view in step 5 includes:

[0020] Step B1: record the position feature points of each lotus pod as K i (i = 1, 2...), and each lotus stem posture feature point is recorded as O 1j , O 2j , O 3j (j = 1, 2, ...);

[0021] Step B2, arbitrarily select a group of lotus stem posture feature points O 1j , O 2j , O 3j , respectively calculate the selected lotus stem posture feature points and all lotus pod position feature points K i The Euclidean distance d between 1ij , d 2ij , d 3ij ;

[0022] Step B3, perform the first pairing, when a set of Euclidean distance d 1ij , d 2ij , d 3ij When both are less than the preset distance threshold, the corresponding lotus position feature point K i and the selected lotus stem posture feature point O 1j , O 2j , O 3j Believed to belong to the same lotus pod;

[0023] Step B4, continuously executing step B2 and step B3 until all lotus pod position feature points or lotus stem posture feature points are paired.

[0024] Furthermore, in the lotus pod picking point positioning method based on multi-view observation, the calculation step of the lotus stalk picking point coordinates and picking vector under a single view in step 5 includes:

[0025] Step C1, extracting the paired lotus pod position feature points and lotus stem posture feature points, and extracting the three-dimensional coordinates of each feature point; the midpoint O2 of the lotus stem posture feature points is regarded as a picking point;

[0026] Step C2, respectively calculating the Euclidean distances d1 and d3 between O1 and O3 and the lotus position feature point K;

[0027] Step C3, taking the lotus stem posture feature point corresponding to the larger value of d1 and d3 as the starting point of the vector, and taking the lotus stem posture feature point corresponding to the smaller value as the end point of the vector, and calculating the picking vector.

[0028] Furthermore, in the above-mentioned lotus pod picking point positioning method based on multi-view observation, the subordinate matching step of each lotus pod target and the corresponding picking information under each single view in step 6 includes:

[0029] Step D1, select any picking point and construct a judgment ball with the point as the center of the ball;

[0030] Step D2, determining in turn whether the remaining picking points are located within the judgment sphere. If they are located within the judgment sphere and the distances between them are less than or equal to the radius of the judgment sphere, it is considered that these picking points belong to the same lotus pod, and the picking vectors corresponding to them also belong to the lotus pod.

[0031] Step D3, executing steps D1 and D2 again for the unmatched picking points until all picking points are matched.

[0032] Furthermore, in the lotus picking point positioning method based on multi-view observation, the method for synthesizing the data of multi-view picking information in step 6 is:

[0033] When the target lotus pod obtains picking information from N (N > 1) viewing angles, the centroid point between the picking points matched to the target lotus pod is calculated, and this point is used as the final picking point; the coordinate mean of each picking vector corresponding to each picking point is calculated, and the vector corresponding to the coordinate mean is used as the final picking vector; when the target lotus pod obtains picking information from only one viewing angle, the picking point coordinates and picking vector obtained from this viewing angle are used as the final picking information.

[0034] Furthermore, in the above-mentioned lotus picking point positioning method based on multi-view observation, the preset distance threshold is 100 mm.

[0035] Furthermore, in the above-mentioned lotus picking point positioning method based on multi-view observation, the radius of the judgment ball is 20 mm.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) The present invention realizes multi-angle observation of the target lotus pod at the same time by establishing a multi-view observation system, and obtains RGB images and depth data information under multiple view angles, which effectively improves the comprehensiveness of the observation of lotus pods in arbitrary postures and measurement points on the surface of obscured lotus pods.

[0038] (2) The present invention obtains the detection bounding box and mask data of the lotus pod and lotus stem picking segment through an instance segmentation model, and completes the effective calculation of the lotus pod picking information in a single perspective; on the basis of realizing the calibration of the camera coordinate system of each perspective, the subordinate matching of the picking information under each single perspective is completed, and the data synthesis of multi-perspective picking information is realized, and finally the picking information of the lotus pod target can be effectively obtained.

[0039] (3) The present invention can effectively observe the lotus pods in any posture and the lotus stem picking section of the obscured lotus pods, and can provide accurate picking point positioning and picking section orientation information for the lotus pod picking robot to perform the picking action. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a lotus picking point positioning method based on multi-view observation provided by the present invention;

[0041] Figure 2 is a schematic diagram of a multi-viewing observation system of the present invention;

[0042] Figure 3 It is a schematic diagram of the lotus pod position feature points and lotus stem posture feature points of the present invention;

[0043] Figure 4 1 is a schematic diagram of the calculation process of the pixel coordinates of the lotus stem posture feature points of the present invention;

[0044] Figure 5 It is a schematic diagram of a subordinate pairing process of a lotus pod position feature point and a lotus stem posture feature point of the present invention;

[0045] Figure 6 It is a schematic diagram of the calculation process of the lotus stem picking point coordinates and the picking vector under a single viewing angle of the present invention;

[0046] Figure 7 It is a schematic diagram of the subordinate matching process of the lotus pod target and the picking information under each single viewing angle of the present invention. DETAILED DESCRIPTION

[0047] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings.

[0048] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present invention. The specific implementation described is only used to explain the present invention and is not intended to limit the present invention.

[0049] The present invention provides a lotus pod picking point positioning method based on multi-view observation, which can accurately position the picking points of lotus pods with any posture and orientation.

[0050] like Figure 1 — Figure 7As shown, a lotus picking point positioning method based on multi-view observation is characterized in that it includes the following steps:

[0051] Step 1, using a plurality of depth cameras to establish a multi-view observation system, determining the positional relationship between the camera coordinate systems corresponding to each view through calibration; and establishing a main coordinate system in the multi-view observation system;

[0052] like Figure 2 As shown, in the specific implementation, three depth cameras are used and placed at the front view, left view and top view positions of the multi-view observation system, and the three view angles are perpendicular to each other and have been calibrated;

[0053] Step 2, collecting RGB images and depth data information of the lotus pod from multiple perspectives;

[0054] Step 3, using the target detection model to detect the RGB image, and obtain the detection bounding box and mask data of the lotus pod and lotus stem picking section under each viewing angle; based on the detection bounding box and mask data, establish lotus pod position feature points and lotus stem posture feature points to respectively characterize the lotus pod position and the lotus stem picking section position and posture in the RGB image; calculate the pixel coordinates of the lotus pod position feature points and the lotus stem posture feature points;

[0055] like Figure 3 As shown, the lotus stem picking segment is a short lotus stem segment adjacent to the lotus pod; as an optional implementation, the lotus pod position feature point is one of the centroid point of the lotus pod mask area and the center point of the lotus pod detection boundary box;

[0056] like Figure 4 As shown, in a specific implementation, the calculation process of the pixel coordinates of the lotus stem posture feature point includes: extracting pixel data within the detection boundary box of the lotus stem picking segment; stepping and extracting pixel points belonging to the lotus stem picking segment mask along a direction perpendicular to the short side of the detection boundary box, specifically, the stepping and extraction process is based on the rule of column-by-column extraction in the horizontal direction and row-by-row extraction in the vertical direction; calculating the centroid points of the pixel points belonging to the lotus stem picking segment mask extracted each time in the stepping and extraction process; using a linear regression method to fit the centroid points into a straight line; calculating the intersection of the fitted straight line and the lotus stem picking segment detection boundary box, and determining the midpoint O2 and two quartile points O1, O3 of the line segment connected by the two intersection points; and using the midpoint and the two quartile points as the lotus stem posture feature points.

[0057] Step 4, combining the pixel coordinates of the lotus pod position feature points and the lotus stem posture feature points and the depth data information in step 2, to obtain the three-dimensional coordinates of each feature point in the camera coordinate system;

[0058] In a specific implementation, after obtaining the pixel coordinates of the lotus pod position feature points and the lotus stem posture feature points on the RGB image, the pixel coordinates in the RGB image are converted into three-dimensional coordinates in the camera coordinate system according to the depth data information obtained by the depth camera.

[0059] Step 5, performing subordinate pairing of lotus pod position feature points and lotus stem posture feature points in each single perspective; completing the calculation of picking information in each single perspective, wherein the picking information includes lotus stem picking point coordinates and picking vectors;

[0060] like Figure 5 As shown, the process of performing subordinate pairing of the lotus pod position feature points and the lotus stem posture feature points in each single perspective includes: recording each lotus pod position feature point as K i (i = 1, 2...), and each lotus stem posture feature point is recorded as O 1j , O 2j , O 3j (j = 1, 2...), when j = 1, there is only a single lotus picking segment. 1j , O 2j , O 3j That is, O1, O2, O3; taking the case of two target lotus pods in the multi-view observation system, there are the following lotus pod position feature points and lotus stem posture feature point information, such as K1 and K2, O 11 , O 21 , O 31 and O 12 , O 22 , O 32 ; Randomly select a group of lotus stem posture feature points O 1j , O 2j , O 3j , respectively calculate the selected lotus stem posture feature points and all lotus pod position feature points K i The Euclidean distance d between 1ij , d 2ij , d 3ij ; Perform the first pairing, when a set of Euclidean distance d 1ij , d 2ij , d 3ij When both are less than the preset distance threshold, the corresponding lotus position feature point K i and the selected lotus stem posture feature point O 1j , O 2j , O 3j are considered to belong to the same lotus pod; for example, when a set of Euclidean distances d 111 , d 211 , d 311 When both are less than the preset distance threshold, the corresponding lotus pod position feature point K1 and the selected lotus stem posture feature point O 11 , O21 , O 31 are considered to belong to the same lotus pod; step B2 and step B3 are continuously performed until all lotus pod position feature points or lotus stem posture feature points are paired.

[0061] As an optional implementation, the preset distance threshold is 100 mm.

[0062] like Figure 6 As shown, in the specific implementation, the calculation process of the lotus stem picking point coordinates and the picking vector under each single viewing angle includes: extracting the paired lotus pod position feature points and lotus stalk posture feature points, and extracting the three-dimensional coordinates of each feature point; regarding the midpoint O2 among the lotus stalk posture feature points as the picking point; respectively calculating the Euclidean distances d1 and d3 between O1, O3 and the lotus pod position feature point K; taking the lotus stalk posture feature point corresponding to the larger value of d1 and d3 as the starting point of the vector, and taking the lotus stalk posture feature point corresponding to the smaller value as the end point of the vector, and calculating the picking vector.

[0063] Step 6, convert the picking information data under each single perspective into the main coordinate system; complete the subordinate matching of each lotus pod target and the corresponding picking information under each single perspective in the main coordinate system; perform data synthesis of multi-perspective picking information to form the final picking information and complete positioning.

[0064] like Figure 7 As shown, in the specific implementation, the subordinate matching process of each lotus pod target and the corresponding picking information under each single perspective includes: selecting an arbitrary picking point, such as P1, where P1 is the picking point of the target lotus pod under perspective No. 1, and constructing a judgment ball with this point as the center of the ball; judging in turn whether the remaining picking points are located in the judgment ball, such as P2 and P3, where P2 and P3 are the picking points of the target lotus pod under perspectives No. 2 and No. 3, if they are located in the judgment ball and the distances between them are less than or equal to the radius of the judgment ball, then it is considered that the picking points P1, P2, and P3 belong to the same lotus pod, and the picking vectors n1, n2, and n3 corresponding to them also belong to the target lotus pod; performing the above steps again for the unmatched picking points until all picking points are matched.

[0065] As an optional implementation, the radius of the judging ball is 20 mm.

[0066] In specific implementation, the method for data synthesis of multi-perspective picking information is as follows: when the target lotus pod obtains picking information from N (N>1) perspectives, the centroid point between the picking points matched to the target lotus pod is calculated, and this point is used as the final picking point; the coordinate mean of each picking vector corresponding to each picking point is calculated, and the vector corresponding to the coordinate mean is used as the final picking vector; when the target lotus pod obtains picking information from only one perspective, the picking point coordinates and picking vector obtained from this perspective are used as the final picking information.

[0067] The above is a description of the implementation steps of the present invention. Through the above description, professionals in the field can implement or use the present invention. The various modifications to these embodiments will be apparent to professionals in the field. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will conform to the widest range consistent with the principles and novelties disclosed herein.

Claims

1. A lotus picking point positioning method based on multi-view observation, characterized in that: The following steps are involved: Step 1, using a plurality of depth cameras to establish a multi-view observation system, determining the positional relationship between the camera coordinate systems corresponding to each view through calibration; and establishing a main coordinate system in the multi-view observation system; Step 2, collecting RGB images and depth data information of the lotus pod from multiple perspectives; Step 3, using the target detection model to detect the RGB image, and obtain the detection bounding box and mask data of the lotus pod and lotus stem picking section under each viewing angle; Based on the detection bounding box and mask data, a lotus pod position feature point and a lotus stem posture feature point are established to respectively represent the lotus pod position and the position and posture of the lotus stem picking section in the RGB image; Calculate the pixel coordinates of the lotus pod position feature points and the lotus stem posture feature points; Step 4, combining the pixel coordinates of the lotus pod position feature points and the lotus stem posture feature points and the depth data information in step 2, to obtain the three-dimensional coordinates of each feature point in the camera coordinate system; Step 5, performing subordinate pairing of lotus pod position feature points and lotus stem posture feature points in each single perspective, and completing the calculation of picking information in each single perspective, wherein the picking information includes lotus stem picking point coordinates and picking vectors; Step 6, convert the picking information data under each single perspective into the main coordinate system; complete the subordinate matching of each lotus pod target and the corresponding picking information under each single perspective in the main coordinate system; perform data synthesis of the multi-perspective picking information to form the final picking information and complete the positioning; The subordinate matching step of each lotus pod target and the corresponding picking information under each single viewing angle includes: Step D1, select any picking point and construct a judgment ball with the point as the center of the ball; Step D2, determining in turn whether the remaining picking points are located within the judgment sphere. If they are located within the judgment sphere and the distances between them are less than or equal to the radius of the judgment sphere, it is considered that these picking points belong to the same lotus pod, and the picking vectors corresponding to them also belong to the lotus pod. Step D3, executing steps D1 and D2 again for the unmatched picking points until all picking points are matched.

2. The lotus picking point positioning method based on multi-view observation according to claim 1 is characterized in that: The lotus pod position feature point in step 3 is one of the centroid point of the lotus pod mask area and the center point of the lotus pod detection boundary box; the lotus stem picking segment is a short lotus stem segment adjacent to the lotus pod.

3. The method for locating lotus picking points based on multi-view observation according to claim 1, characterized in that: The step of calculating the pixel coordinates of the lotus stem posture feature points in step 3 includes: Step A1, extracting pixel data within the detection boundary box of the lotus stem picking segment; stepping and extracting pixel points in the mask of the lotus stem picking segment along a direction perpendicular to the short side of the detection boundary box, specifically, the stepping and extracting process is based on the rule of extracting columns in the horizontal direction and extracting rows in the vertical direction; Step A2, calculating the centroid of the pixel points in the lotus stem picking segment mask extracted each time in the step extraction process; Step A3, using a linear regression method to fit each centroid point into a straight line; Step A4, calculating the intersection of the fitting line and the lotus stem picking segment detection boundary box, determining the midpoint O2 and two quartile points O1, O3 of the line segment connected by the two intersection points; and using the midpoint and two quartile points as lotus stem posture feature points.

4. The method for locating lotus picking points based on multi-view observation according to claim 1, characterized in that: The target detection model in step 3 is an instance segmentation model.

5. The method for locating lotus picking points based on multi-view observation according to claim 1, characterized in that: The step of performing subordinate pairing of the lotus pod position feature points and the lotus stem posture feature points in each single perspective in step 5 includes: Step B1: record the position feature points of each lotus pod as K i (i=1, 2...), and each lotus stem posture feature point is recorded as O 1j , O 2j , O 3j (j=1, 2, ...); Step B2, randomly select a group of lotus stem posture feature points O 1j , O 2j , O 3j , respectively calculate the selected lotus stem posture feature points and all lotus pod position feature points K i The Euclidean distance d between 1ij , d 2ij , d 3ij ; Step B3, perform the first pairing, when a set of Euclidean distance d 1ij , d 2ij , d 3ij When both are less than the preset distance threshold, the corresponding lotus position feature point K i and the selected lotus stem posture feature point O 1j , O 2j , O 3j Believed to belong to the same lotus pod; Step B4, continuously executing step B2 and step B3 until all lotus pod position feature points or lotus stem posture feature points are paired.

6. The method for locating lotus picking points based on multi-view observation according to claim 3, characterized in that: The calculation steps of the lotus stem picking point coordinates and picking vector under a single viewing angle in step 5 include: Step C1, extracting the paired lotus pod position feature points and lotus stem posture feature points, and extracting the three-dimensional coordinates of each feature point; the midpoint O2 of the lotus stem posture feature points is regarded as a picking point; Step C2, respectively calculating the Euclidean distances d1 and d3 between O1 and O3 and the lotus position feature point K; Step C3, taking the lotus stem posture feature point corresponding to the larger value of d1 and d3 as the starting point of the vector, and taking the lotus stem posture feature point corresponding to the smaller value as the end point of the vector, and calculating the picking vector.

7. The method for locating lotus picking points based on multi-view observation according to claim 1, characterized in that: The method for data synthesis of multi-view picking information in step 6 is: When the target lotus pod obtains picking information from N viewing angles, the centroid point between the picking points matched to the target lotus pod is calculated, and this point is used as the final picking point, where N>1; the coordinate mean of each picking vector corresponding to each picking point is calculated, and the vector corresponding to the coordinate mean is used as the final picking vector; When the target lotus pod has only one viewing angle to obtain picking information, the picking point coordinates and picking vector obtained from this viewing angle are used as the final picking information.

8. The method for locating lotus picking points based on multi-view observation according to claim 5, characterized in that: The preset distance threshold is 100 mm.

9. The method for locating lotus picking points based on multi-view observation according to claim 1, characterized in that: The radius of the judgment ball is 20 mm.

Images