Fruit contour correction method based on depth information

Abstract

A fruit contour correction method based on depth information comprises the following steps: S1, fruit identification and image segmentation: firstly, obtaining a fruit data set, preprocessing the data set, training a target detection model based on YOLOv8, extracting a fruit mask based on SAM2, and finally extracting an outer contour line of the fruit mask; s2, extracting a real contour of the fruit: firstly setting a noisy point judgment basis, determining a judgment radius R, and carrying out additional contour denoising processing on a sheltered fruit target to extract the real contour of the fruit; and S3, contour fitting and positioning of the fruits: firstly homogenizing the depth, then performing contour fitting, and realizing positioning and size estimation through space mapping. According to the method, the real contour of the fruit can be extracted under the condition that branches and leaves are shielded, contour fitting of the fruit is completed with high accuracy, and finally the center position and the size of the fruit are obtained.

Description

Technical Field

[0001] The invention belongs to the technical field of agricultural informatization, and in particular relates to a fruit contour correction method based on depth information. Background Art

[0002] Fruit and vegetable harvesting is the most time-consuming and labor-intensive stage in agricultural production. At the same time, it has become one of the most labor-intensive and expensive agricultural tasks due to its seasonality. The shortage of rural labor has led to an increase in the cost of manual harvesting, and the demand for automated harvesting has become increasingly urgent. Modern agriculture has moved towards an era of intelligence and refinement, and it is a development trend for intelligent robots to replace manual labor to complete automated harvesting operations. The realization of automated operations by harvesting robots requires the joint advancement of technologies in multiple fields such as machinery, control, and computer vision. As an important part of the harvesting robot, the visual system is used to identify and locate the harvesting target and detect the edge of the fruit. For fruits and vegetables with elliptical or circular fruit contours such as watermelon, melon, apple, and broccoli, the size and center of the fruit need to be fitted according to the contour, so that the picking end effector can move to the center of the circle and open the gripper to a size larger than the diameter of the fruit to complete the picking work. Therefore, fruit contour detection is one of the important tasks of the harvesting robot vision system. In the actual automated harvesting process, the fruit may be blocked by branches and leaves, which increases the difficulty of extracting the true contour of the fruit and affects the estimation accuracy of the fruit size and center of the circle. On the one hand, the large positioning error may lead to harvest failure, and on the other hand, the fruit size may be misjudged, resulting in the harvest of immature fruits that are too small, which will cause certain losses. Therefore, it is necessary to develop a method for extracting the true contour of the fruit under the condition of branch occlusion, complete the contour fitting of the fruit with high accuracy, and finally obtain the center position and size of the fruit. Summary of the invention

[0003] In order to overcome the shortcomings of the existing technology, the present invention provides a fruit contour correction method based on depth information, which can meet the requirements of fruit true contour extraction under branch and leaf occlusion, complete fruit contour fitting with high accuracy, and finally obtain the center position and size of the fruit.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A fruit contour correction method based on depth information comprises the following steps:

[0006] S1. Fruit recognition and image segmentation: First, the fruit dataset is acquired, the dataset is preprocessed, the target detection model based on YOLOv8 is trained, the fruit mask is extracted based on SAM2, and finally the outer contour of the fruit mask is extracted;

[0007] S2. Extraction of the true contour of the fruit: First, the noise point judgment basis is set, the judgment radius R is determined, and additional contour denoising is performed on the occluded fruit target to extract the true contour of the fruit;

[0008] S3. Fruit contour fitting and positioning: First, the depth is averaged, then contour fitting is performed, and positioning and size estimation are achieved through spatial mapping.

[0009] Further, the process of S1 is as follows:

[0010] S1.1 Acquisition of fruit dataset: Use a depth camera to obtain image data of fruits under different lighting conditions. The image data should include a dataset of fruits partially obscured by branches and leaves, which will be used for the subsequent training and verification of the target detection algorithm for fruits partially obscured by branches and leaves. The algorithm model may learn too many features of branch and leaf occlusion due to the introduction of too many fruit datasets that are severely obscured by branches and leaves, which may cause deviations in the training model. In the detection stage, targets that only contain branches and leaves may also be identified as fruits. Therefore, it is necessary to first perform statistics on the degree of fruit occlusion before annotating the training set, so as to clarify the standard for annotating the training set of fruits obscured by branches and leaves. Fruit targets that are obscured by branches and leaves by more than 70% will not be considered, because the overall proportion of such situations during actual fruit harvesting is very low.

[0011] S1.2 Dataset preprocessing: Use MakeSense software to label fruit targets that are blocked by branches and leaves. Fruits with an occlusion degree of less than 10% are divided into one category and labeled as fruit targets; and fruits with an occlusion degree between 10% and 70% are divided into another category and labeled as blocked fruit targets; then use geometric transformation (rotation, flipping, etc.), color transformation (brightness, saturation adjustment, etc.) and noise addition (Gaussian noise, blur, etc.) to complete the expansion of the labeled data set, which is conducive to improving the generalization ability of the deep learning model;

[0012] S1.3 Training of the target detection model based on YOLOv8: Based on the prepared data set, the two categories, namely, the fruit target and the occluded fruit target, are divided into a training set and a validation set according to the set ratio. Then, the same YOLOv8n basic target detection model is used to complete the training and validation of the target detection model of the fruit, and the optimal weight file is obtained when the average precision of the two categories is the largest.

[0013] S1.4 Fruit mask extraction based on SAM2: The original fruit image obtained by the depth camera D435i is processed using the trained and verified YOLOv8n optimal weight file to obtain the detection frame of the fruit target in the corresponding image, and this is used as a prompt to input into the SAM2 image segmentation algorithm model to complete the mask extraction of the target fruit in the corresponding detection frame;

[0014] S1.5 Fruit contour extraction: Use the cv2.cvtColor function of the OpenCV library to binarize the fruit color mask processed by SAM2, and then use the cv2.findContours function to extract the outer contour of the fruit mask.

[0015] Furthermore, the process of S2 is as follows:

[0016] S2.1 Noise point judgment basis: The contour pixel points on the boundary line between the fruit and the occluder area in the image are noise points. The contour will be processed as the contour of the fruit by the contour extraction after the image segmentation algorithm, but it is actually the contour of the occluder and needs to be excluded. Take the noise point as the center of the circle and R as the radius to obtain the depth information of all points (decision points) on the circumference. The decision point inside the outer contour line of the fruit mask is called the inner decision point, and its depth information is recorded as d i , the mean value d of the sum of the depth information of n internal decision points n That is The decision point outside the outer contour of the fruit mask is called the outer decision point, and its depth information is recorded as d e , the mean value d of the sum of the depth information of m external decision points m That is Because the occluder (external judgment point) is closer to the depth camera than the fruit (internal judgment point), the depth information of the external judgment point is smaller than that of the internal judgment point. Therefore, the judgment basis of the contour noise point on the boundary between the fruit and the occluder area is set as d m <d n ;

[0017] S2.2 Determine the judgment radius R: manually select the outer contour lines of the fruit mask where the fruit is blocked in multiple images, use the contour pixel points on the boundary between the fruit and the blocking area as noise points, and obtain the depth information of the corresponding image. The total number of noise point samples is counted as N t; Set different judgment radius R (1≤R≤10, and R is an integer) conditions, the recognition accuracy of each noise point is A, use the draw.ellipse() method in the ImageDraw module of PIL (Python Imaging Library) to draw a circle with a radius of R with the noise point as the center; use the function rs.rs2_deproject_pixel_to_point in the D435i camera SDK to obtain the actual distance to the camera in the space corresponding to each point on the circle with a specified pixel radius of R from the two-dimensional RGB image and the corresponding depth image data, that is, the depth information; use the polygon.contains function in the Shapely library to obtain the pixel points on the R circumference within the fruit outline pixel points, set as the inner judgment points; the remaining pixel points are set as the outer judgment points after excluding the points on the outline using the poly.touches function; the noise point recognition accuracy calculation formula A=N r / N, where N r Indicates that d is satisfied m <d n The number of noise points, N is the total number of noise points, and a suitable judgment radius R is comprehensively selected by comparing the recognition accuracy A and processing time of noise points under different judgment radius R conditions. a ;

[0018] S2.3 Noise removal: After completing S1.5, all the pixels of the outer contour line of the fruit mask of the occluded fruit target are selected as the center of the circle and the radius R is selected. a Draw the circle, combine the depth information value of the corresponding image circle position, and use the d m <d n The discriminant formula is used to judge the noise point. If the discriminant formula is satisfied, the point is a noise point and should be removed; otherwise, the point is the true contour point of the fruit and should be retained; after all points are judged, the pixel points that are finally retained are the true contour of the fruit, which will be used for subsequent fitting operations.

[0019] The S3 process is as follows:

[0020] S3.1 Average depth: Use the mask after SAM2 image segmentation in S1.4 as the limited range, use rs.rs2_deproject_pixel_to_point function to obtain the depth information corresponding to all pixels in the range, and calculate the average value H of the depth as the distance from the fruit to the depth camera;

[0021] S3.2 Contour fitting: If the fruit is identified as a fruit target (the degree of occlusion is less than 10%), the outer contour of the fruit mask after S1.5 is used as the data basis; if the fruit is identified as an occluded fruit target (the degree of occlusion is between 10% and 70%), the partial true contour of the fruit after S2.3 is used as the data basis, and the least squares method is used to complete the fitting of a circle (such as apple, broccoli, etc.) or an ellipse (such as watermelon, melon, etc.);

[0022] S3.3 Spatial mapping: If the fruit is nearly spherical, select the pixel coordinates of the center of the circle after fitting and any point on the circumference in the image; if the fruit is nearly ellipsoidal, select the image pixel coordinates of the center of the ellipse after fitting, the intersection of the ellipse circumference with the major axis and the minor axis (ellipse vertex), and use the rs.rs2_deproject_pixel_to_poin function to obtain the three-dimensional spatial position coordinates of the corresponding point in the depth camera coordinate system with a depth of H, where the center coordinates are the spatial positioning coordinates of the fruit in the depth camera coordinate system; use the distance formula between the two points to calculate the distance between the center of the circle and a point on the circumference, which is the estimated radius of the nearly circular fruit; calculate the distance between the center of the ellipse and the major and minor axes, which is the estimated major and minor axis lengths of the nearly ellipsoidal fruit; the estimated radius and major and minor axes of the fruit are used as the basis for judging whether the harvesting conditions are met. The center position of the fruit is subsequently converted to the robotic arm coordinate system through the coordinate system, which is the picking target point of the harvesting end effector.

[0023] The beneficial effects of the present invention are mainly manifested in: being able to extract the true contour of the fruit when it is blocked by branches and leaves, completing the contour fitting of the fruit with high precision, and finally obtaining the center position and size of the fruit. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a flow chart of a fruit contour correction method based on depth information.

[0025] Figure 2 This is a schematic diagram of the fruit outline.

[0026] Figure 3 It is a schematic diagram for determining the radius R.

[0027] Figure 4 It is a schematic diagram of internal and external judgment points. DETAILED DESCRIPTION

[0028] The present invention will be further described below in conjunction with the accompanying drawings.

[0029] Reference Figure 1 to Figure 4 , a fruit contour correction method based on depth information, comprising the following steps:

[0030] S1. Fruit recognition and image segmentation: First, the fruit dataset is acquired, the dataset is preprocessed, the target detection model based on YOLOv8 is trained, the fruit mask is extracted based on SAM2, and finally the outer contour of the fruit mask is extracted;

[0031] The trained and verified YOLOv8 algorithm is used to complete the recognition of fruit targets in the original fruit image. The fruit targets are divided into two categories: one is the fruit with an occlusion degree of less than 10%, which is defined as the fruit target; the other is the fruit with an occlusion degree between 10% and 70%, which is defined as the occluded fruit target. Then, SAM2 is used to perform image segmentation on the fruit targets in the identified detection frame to complete the extraction of the fruit mask. Then, cv2.findContours is used to extract the contour of the fruit.

[0032] The process of S1 is as follows:

[0033] S1.1 Acquisition of fruit dataset: Use a depth camera to obtain image data of fruits under different lighting conditions. The image data should include a dataset of fruits partially obscured by branches and leaves, which will be used for the subsequent training and verification of the target detection algorithm for fruits partially obscured by branches and leaves. The algorithm model may learn too many features of branch and leaf occlusion due to the introduction of too many fruit datasets that are severely obscured by branches and leaves, which may cause deviations in the training model. In the detection stage, targets that only contain branches and leaves may also be identified as fruits. Therefore, it is necessary to perform statistics on the degree of occlusion of the fruit before labeling the training set, in order to clarify the standards for labeling the training set of fruits obscured by branches and leaves. Fruit targets that are obscured by branches and leaves by more than 70% will not be considered, because the overall proportion of this situation during actual fruit harvesting is very low.

[0034] S1.2 Dataset preprocessing: Use MakeSense software to label fruit targets that are blocked by branches and leaves. Fruits with an occlusion degree of less than 10% are divided into one category and labeled as fruit targets; and fruits with an occlusion degree between 10% and 70% are divided into another category and labeled as blocked fruit targets. Then use geometric transformation (rotation, flipping, etc.), color transformation (brightness, saturation adjustment, etc.) and noise addition (Gaussian noise, blur, etc.) to expand the labeled dataset, which is conducive to improving the generalization ability of the deep learning model;

[0035] S1.3 Training of the target detection model based on YOLOv8: Based on the prepared data set, the two categories are divided into training set and verification set in a ratio of 7 to 3, and then the same YOLOv8n basic target detection model is used to complete the training and verification of the target detection model of the fruit, and the optimal weight file with the maximum average precision is obtained;

[0036] S1.4 Fruit mask extraction based on SAM2: The original fruit image obtained by the depth camera D435i is processed using the trained and verified YOLOv8n optimal weight file to obtain the detection frame of the fruit target in the corresponding image, and this is used as a prompt to input into the SAM2 image segmentation algorithm model to complete the mask extraction of the target fruit in the corresponding detection frame;

[0037] S1.5, fruit contour extraction: Use the cv2.cvtColor function of the OpenCV library to binarize the fruit color mask processed by SAM2, and then use the cv2.findContours function to extract the outer contour of the fruit mask;

[0038] S2. Extraction of the true contour of the fruit: First, the noise point judgment basis is set, the judgment radius R is determined, and additional contour denoising is performed on the occluded fruit target to extract the true contour of the fruit;

[0039] Use the appropriate determination radius R a And the discriminant formula d m <d n The fruit contour pixels extracted from the occluded fruit target are identified with noise points one by one, and part of the real contour of the fruit is retained;

[0040] For unobstructed fruits, the outer contour of the fruit mask is the true contour of the fruit. However, for obstructed fruits, part of the outer contour of the fruit mask is the true contour of the fruit, and part of it is the contour of the obstruction (regarded as noise). If this part of the noise cannot be excluded before contour fitting, the subsequent fitted fruit size will be smaller and the positioning will also be biased, such as Figure 2 Therefore, it is necessary to perform additional contour denoising on the occluded fruit target to extract the true contour of the fruit.

[0041] The process of S2 is as follows:

[0042] S2.1 Noise point judgment basis: The contour pixel points on the boundary between the fruit and the occluder area in the image are noise points, such as Figure 2 As shown in (B), the contour will be processed as the contour of the fruit by the contour extraction after the image segmentation algorithm, but it is actually the contour of the occluder, so it needs to be excluded. With the noise point as the center of the circle and R as the radius, the depth information of all points (decision points) on the circumference is obtained. The decision point inside the outer contour line of the fruit mask is called the inner decision point, and its depth information is recorded as d i , the mean value d of the sum of the depth information of n internal decision points n That is The decision point outside the outer contour of the fruit mask is called the outer decision point, and its depth information is recorded as d e, the mean value d of the sum of the depth information of m external decision points m That is Because the occluder (external judgment point) is closer to the depth camera than the fruit (internal judgment point), the depth information of the external judgment point is smaller than that of the internal judgment point. Therefore, the judgment basis of the contour noise point on the boundary between the fruit and the occluder area is set as d m <d n .

[0043] S2.2 Determination of radius R: By manually selecting the outer contour lines of the fruit mask where the fruit is blocked in multiple images, the contour pixel points on the boundary between the fruit and the blocking area are used as noise points, and the depth information of the corresponding image is obtained. The total number of noise point samples is counted as N t . Under different judgment radius R (1≤R≤10, and R is an integer), the recognition accuracy of each noise point is A. Use the draw.ellipse() method in the ImageDraw module of PIL (Python Imaging Library) to draw a circle with a radius of R with the noise point as the center, as shown in Figure 3 The figure shows a circle with a certain noise point as a schematic point, and a radius R of 1, 3, 5, and 7 pixels from the inside to the outside. Use the function rs.rs2_deproject_pixel_to_point in the D435i camera SDK to obtain the actual distance to the camera in the space corresponding to each point on the circle with a specified pixel radius of R from the two-dimensional RGB image and the corresponding depth image data, that is, the depth information. Use the polygon.contains function in the Shapely library to obtain the pixel points on the R circumference within the fruit outline pixel point and set them as the inner judgment point; the remaining pixel points are set as the outer judgment points after excluding the points on the outline using the poly.touches function, as shown in Figure 4 The schematic diagram of the inner and outer judgment points when the radius R is 5 pixels. The noise point recognition accuracy calculation formula A = N r / N, where N r Indicates that d is satisfied m <d n The number of noise points, N is the total number of noise points, and the recognition accuracy A and processing time of noise points under different judgment radius R are compared to comprehensively select the judgment radius R. a ;

[0044] S2.3 Noise removal: After completing S1.5, all the pixels of the outer contour line of the fruit mask of the occluded fruit target are selected as the center of the circle and the radius R is selected. a Draw the circle, combine the depth information value of the corresponding image circle position, and use the d m <d nThe discriminant formula is used to judge the noise points. If the discriminant formula is satisfied, the point is a noise point and should be removed; otherwise, the point is the true contour point of the fruit and should be retained; after all points are judged, the pixel points that are finally retained are the true contour of part of the fruit, which will be used for subsequent fitting operations;

[0045] S3, fruit contour fitting and positioning: first, the depth is averaged, then contour fitting is performed, and positioning is achieved through spatial mapping;

[0046] According to the fruit mask range obtained by SAM2, calculate the average value of the depth information value of the corresponding range. It is defined as the distance of the fruit from the depth camera. Based on the fruit contour extracted by the fruit target after step S1 and the real contour of the fruit part retained by the occluded fruit target after step S2, respectively, use the least squares method to complete the circular or elliptical fitting, obtain the center of the circle and a point on any circumference, or the intersection of the center of the ellipse and its circumference with the major and minor axes, and project it to a plane with a distance equal to the distance from the fruit to the depth camera. Use the rs.rs2_deproject_pixel_to_poin function to solve and obtain the center of the circle and the point on its circumference on the projection plane, and then use the distance formula between the two points to solve the radius of the circle or the length of the major and minor axes. The center of the circle is the position of the fruit in the depth camera coordinate system, and the radius or the length of the major and minor axes can be used to estimate the size of the fruit;

[0047] The S3 process is as follows:

[0048] S3.1 Averaged depth: Using the mask after SAM2 image segmentation in S1.4 as the limited range, use the rs.rs2_deproject_pixel_to_point function to obtain the depth information corresponding to all pixels in the range, and calculate the average depth H as the distance from the fruit to the depth camera.

[0049] S3.2 Contour fitting: If the fruit is identified as a fruit target (the degree of occlusion is less than 10%), the outer contour line of the fruit mask after S1.5 is used as the data basis; if the fruit is identified as an occluded fruit target (the degree of occlusion is between 10% and 70%), the partial true contour of the fruit after S2.3 is used as the data basis, and the least squares method is used to complete the fitting of a circle (such as apple, broccoli, etc.) or an ellipse (such as watermelon, melon, etc.).

[0050] S3.3 Spatial mapping: If the fruit is nearly spherical, the pixel coordinates of the center of the circle after fitting and any point on the circumference are selected; if the fruit is nearly ellipsoidal, the pixel coordinates of the center of the ellipse after fitting, the intersection of the ellipse circumference with the major axis and the minor axis (ellipse vertex) are selected, and the rs.rs2_deproject_pixel_to_poin function is used to obtain the three-dimensional spatial position coordinates of the corresponding point in the depth camera coordinate system with a depth of H, where the center coordinates are the spatial positioning coordinates of the fruit in the depth camera coordinate system. Using the distance formula between two points, the distance between the center of the circle and a point on the circumference is calculated, which is the estimated radius of the nearly circular fruit; the distance between the center of the ellipse and the circumference and the major and minor axes is calculated, which is the estimated major and minor axis length of the nearly ellipsoidal fruit. The estimated radius and major and minor axes of the fruit are used as the basis for judging whether the harvesting conditions are met. The center position of the fruit is subsequently converted to the robotic arm coordinate system through the coordinate system, which is the picking target point of the harvesting end effector.

[0051] Figure 2 It is a schematic diagram of the outline of the fruit and the occluder. Figure A shows the true outline of the fruit and the outline of the fruit fitted by the outline extracted based on the image segmentation algorithm after being occluded. There are large errors in the center positioning and radius size between the two. Figure B shows that for the occluded fruit target, the outline extracted based on the image segmentation algorithm is actually composed of two parts. One part is noise points, which are actually the outline of the occluder, and the other part is the true outline pixel points of the fruit.

[0052] Figure 3 This is a schematic diagram of the determination radius R in the denoising algorithm. The determination circle is based on the pixel point on the contour extracted by the image segmentation algorithm as the center of the circle, and R (1≤R≤10, and R is an integer) is the circumference of the radius. The figure only shows the pixel circumferences with radii of 1, 3, 5, and 7 from the inside to the outside.

[0053] Figure 4 This is a schematic diagram of internal and external judgment points. The figure shows that when a circle with a certain fruit outline schematic point as the center and a judgment radius of 5 pixels is used, the points on the circumference that are inside the fruit outline segmented by SAM are set as internal judgment points, and the points that are outside the fruit outline segmented by SAM are set as external judgment points.

[0054] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A fruit contour correction method based on depth information, characterized in that: The method comprises the following steps: S1. Fruit recognition and image segmentation: First, the fruit dataset is acquired, the dataset is preprocessed, the target detection model based on YOLOv8 is trained, the fruit mask is extracted based on SAM2, and finally the outer contour of the fruit mask is extracted; S2. Extraction of the true contour of the fruit: First, the noise point judgment basis is set, the judgment radius R is determined, and additional contour denoising is performed on the occluded fruit target to extract the true contour of the fruit; S3. Fruit contour fitting and positioning: First, the depth is averaged, then contour fitting is performed, and positioning and size estimation are achieved through spatial mapping.

2. A fruit contour correction method based on depth information as claimed in claim 1, characterized in that: The process of S1 is as follows: S1.1 Acquisition of fruit dataset: Use a depth camera to obtain image data of fruits under different lighting conditions. The image data includes a dataset of fruits partially obscured by branches and leaves, which is used for the subsequent training and verification of the target detection algorithm for fruits partially obscured by branches and leaves. Before labeling the training set, the degree of occlusion of the fruit is counted to clarify the standard for labeling the training set of fruits occluded by branches and leaves. Fruit targets that are occluded by branches and leaves by more than 70% are not considered. S1.2 Dataset preprocessing: Use MakeSense software to label fruit targets that are blocked by branches and leaves. Fruits with an occlusion degree of less than 10% are divided into one category and labeled as fruit targets; and fruits with an occlusion degree between 10% and 70% are divided into another category and labeled as blocked fruit targets. Then, geometric transformation, color transformation and noise addition are used to expand the label dataset. S1.3 Training of the target detection model based on YOLOv8: Based on the prepared data set, the two categories, namely, the fruit target and the occluded fruit target, are divided into a training set and a validation set according to a set ratio. Then, the same YOLOv8n basic target detection model is used to complete the training and validation of the target detection model of the fruit, and the optimal weight file with the maximum average precision is obtained. S1.4 Fruit mask extraction based on SAM2: The original fruit image obtained by the depth camera D435i is processed using the trained and verified YOLOv8n optimal weight file to obtain the detection frame of the fruit target in the corresponding image, and this is used as a prompt to input into the SAM2 image segmentation algorithm model to complete the mask extraction of the target fruit in the corresponding detection frame; S1.5 Fruit contour extraction: Use the cv2.cvtColor function of the OpenCV library to binarize the fruit color mask processed by SAM2, and then use the cv2.findContours function to extract the outer contour of the fruit mask.

3. A fruit contour correction method based on depth information as claimed in claim 2, characterized in that: The process of S2 is as follows: S2.1 Noise point judgment basis: The contour pixel point on the boundary line between the fruit and the occluder area in the image is the noise point. With the noise point as the center and R as the radius, the depth information of all points on the circumference, i.e., the judgment point, is obtained. The judgment point inside the outer contour line of the fruit mask is called the inner judgment point, and its depth information is recorded as d i , the mean value d of the sum of the depth information of n internal decision points n That is The decision point outside the outer contour of the fruit mask is called the outer decision point, and its depth information is recorded as d e , the mean value d of the sum of the depth information of m external decision points m That is The judgment basis of the contour noise point on the boundary between the fruit and the occluder area is set to d m <d n ; S2.2 Determine the judgment radius R: manually select the outer contour lines of the fruit mask where the fruit is blocked in multiple images, use the contour pixel points on the boundary between the fruit and the blocking area as noise points, and obtain the depth information of the corresponding image. The total number of noise point samples is counted as N t ; Under different judgment radius R conditions, the recognition accuracy of each noise point is A, and the noise point recognition accuracy calculation formula is A=N r / N, where N r Indicates that d is satisfied m <d n The number of noise points, N is the total number of noise points, and the recognition accuracy A and processing time of noise points under different judgment radius R are compared to comprehensively select the judgment radius R. a ; S2.3 Noise removal: After completing step S1.5, all the pixels of the outer contour line of the fruit mask of the occluded fruit target are selected as the center of the circle and the radius R is selected. a Draw the circle, combine the depth information value of the corresponding image circle position, and use the d m <d n The discriminant formula is used to judge the noise point; if the discriminant formula is satisfied, the point is a noise point and should be removed; Otherwise, the point is the true contour point of the fruit and should be retained; After all points are identified, the pixels that are finally retained are the actual outline of the fruit and will be used for subsequent fitting operations.

4. A fruit contour correction method based on depth information as claimed in claim 3, characterized in that: The S3 process is as follows: S3.1 Average depth: Use the mask after SAM2 image segmentation in step S1.4 as the limited range, use rs.rs2_deproject_pixel_to_point function to obtain the depth information corresponding to all pixels in the range, and calculate the average value H of the depth as the distance from the fruit to the depth camera; S3.2 Contour fitting: If the fruit is identified as a fruit target, the outer contour line of the fruit mask after step S1.5 is used as the data basis; if the fruit is identified as an obscured fruit target, the partial true contour of the fruit after step S2.3 is used as the data basis, and the least squares method is used to complete the circle or ellipse fitting; S3.3 Spatial mapping: If the fruit is nearly spherical, select the pixel coordinates of the center of the circle after fitting and any point on the circumference in the image; if the fruit is nearly ellipsoidal, select the image pixel coordinates of the center of the ellipse after fitting, the intersection of the ellipse circumference with the major axis and the minor axis, and use the rs.rs2_deproject_pixel_to_poin function to obtain the three-dimensional spatial position coordinates of the corresponding point in the depth camera coordinate system with a depth of H, where the center coordinates are the spatial positioning coordinates of the fruit in the depth camera coordinate system; use the distance formula between the two points to calculate the distance between the center of the circle and a point on the circumference, which is the estimated radius of the nearly circular fruit; calculate the distance between the center of the ellipse and the circumference and the major and minor axes, which is the estimated major and minor axis lengths of the nearly ellipsoidal fruit; the estimated radius and major and minor axes of the fruit are used as the basis for judging whether the harvesting conditions are met. The center position of the fruit is subsequently converted to the robotic arm coordinate system through the coordinate system, which is the picking target point of the harvesting end effector.

Images