Adhesive peanut two-dimensional grayscale image segmentation method based on depth information three-dimensional morphology

Abstract

The invention discloses a depth information three-dimensional morphology-based adhered peanut two-dimensional grayscale image segmentation method, and relates to the technical field of image adhesion segmentation, and the method comprises the following steps: building a line structured light imaging system based on a triangulation principle, collecting the point clouds of a peanut in operation line by line through the system, obtaining a three-dimensional morphology through splicing, and carrying out the segmentation of the three-dimensional morphology of the peanut. The method comprises the following steps: calculating a point cloud normal vector and curvature by combining a local surface fitting algorithm, carrying out three-dimensional segmentation by taking the normal vector and curvature as features and utilizing a region growing segmentation algorithm, then extracting a two-dimensional contour of a single peanut model by adopting a rolling ball method, establishing a search template, and implementing multi-scale feature matching on an adhered image to realize region positioning. And finally, optimizing the segmentation region by using an opening operation. According to the method, the problem that the adhesion image is difficult to segment due to the characteristics of non-convexity of the peanut pod appearance, complex surface texture and the like is effectively solved, the information of peanut height, spherical degree and the like is increased, and the accuracy and rationality of later quality detection are facilitated.

Description

Technical Field

[0001] The invention relates to the technical field of image adhesion segmentation, and in particular to a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information. Background Art

[0002] The rapid development of smart agricultural technology has made post-harvest processing the key to improving agricultural efficiency and competitiveness. Quality inspection and sorting, as key steps in the selection and breeding of agricultural products, directly affect the overall quality of crops of different grades. Peanuts must be sorted for quality before planting and post-harvest processing. However, peanuts may touch and stick to each other, and the collected images need to be segmented and pre-processed, otherwise it will affect the subsequent quality evaluation.

[0003] The existing image segmentation algorithms (such as concave point detection combined with improved watershed segmentation algorithm) have been fully applied to round objects such as rice grains, potatoes, and wolfberries, and have fast running speed, high precision, and high efficiency. However, the segmentation effect is not ideal when faced with the adhesion images caused by the non-fully convex shape of peanut pods, the presence of a waist, complex surface texture, and most non-circular structures.

[0004] Therefore, in view of this, the existing structure and defects are studied and improved, and a two-dimensional grayscale image segmentation method of adhered peanuts based on three-dimensional morphology of depth information is proposed. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information, which solves the problems raised in the above-mentioned background technology.

[0006] To achieve the above objectives, the present invention is implemented by the following technical solutions: a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information, comprising the following steps:

[0007] Step 1: Based on the principle of triangulation, build a line structured light imaging system, calibrate the camera and light plane of the system, and establish the transformation relationship from the camera coordinate system to the world coordinate system;

[0008] Step 2: Use a line structured light imaging system to collect three-dimensional point cloud data of peanuts placed on the guide rails line by line, and stitch them together to obtain the three-dimensional morphology of the peanuts. In addition, collect images of adhered peanuts to be segmented;

[0009] Step 3: Use normal vector and curvature as features of region growing algorithm to achieve 3D segmentation of point cloud;

[0010] Step 4: Use the rolling ball method to extract the two-dimensional contours of the individual peanut models after segmentation, and establish a search template;

[0011] Step 5: Based on the template, multi-scale feature matching is performed on the adhesion image to realize region positioning, and finally the segmentation region is optimized by opening operation to realize two-dimensional segmentation of the adhesion image;

[0012] Step 6: Use model evaluation indicators to evaluate model performance.

[0013] Furthermore, in step 1, the line structured light imaging system includes a laser, an industrial camera and a one-dimensional motion guide rail, wherein the camera coordinate system (X c , Y c , Z c ) to the world coordinate system (X w , Y w , Z w ) is:

[0014]

[0015] Furthermore, R is a rotation matrix, and T is a translation matrix. These two parameters are also called external parameters of the camera and are determined by calibration.

[0016] Furthermore, in step 2, the three-dimensional morphology reconstruction specifically includes: considering that the camera and the laser remain fixed in the system, after obtaining the three-dimensional point cloud coordinates of each frame of the line structured light image, the distance d of the object relative to the initial scanning position when the image is acquired is calculated according to the order in which each picture is taken, so that the reconstructed multiple laser center lines are spliced ​​in the same direction in sequence according to the direction of movement of the guide rail to obtain the complete object contour. The specific calculation formula is: d=vt.

[0017] Furthermore, v is the movement speed of the peanut, in units of mm / s; t is the time from the start of the camera shooting to the shooting of the line structured light image, in units of s.

[0018] Furthermore, in step 3, the step of calculating the normal vector and the curvature includes: calculating the point cloud normal vector using a local surface fitting method, and calculating the curvature of the sampling point according to the eigenvalues ​​of the covariance matrix of the constructed domain point cloud set.

[0019] Furthermore, in step 3, the step of using the region growing algorithm to screen and segment the point cloud includes: constructing a KD-Tree structure of the point cloud, and screening neighborhood points in this structure based on the normal angle and curvature threshold, and gradually expanding to complete the point cloud segmentation of individual peanuts.

[0020] Furthermore, in step 4, the specific steps of establishing the search template include: graying the image, threshold segmentation, connecting adjacent areas, reducing the image area, obtaining the shape template outline, and creating a shape scalable template.

[0021] Furthermore, in step 5, the multi-scale feature matching specifically includes: for the adhesion image to be matched, searching for the optimal matching position of the peanut template from top to bottom at the appropriate image pyramid level, and at the same time, setting the minimum distance accuracy threshold in the search process by the least squares method, and then minimizing the distance between the model point and its corresponding point in the image, obtaining the posture parameters of the template with sub-pixel accuracy including rotation angle, scaling ratio, translation distance, etc., to achieve accurate matching.

[0022] Furthermore, in step 6, the model evaluation index is the segmentation accuracy, which is specifically calculated as follows:

[0023] Among them, N is the number of pixels of a single peanut after matching the segmentation algorithm, and M is the number of complete pixels of a single peanut.

[0024] The present invention provides a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information, which has the following beneficial effects:

[0025] 1. This method of segmenting the two-dimensional grayscale image of adhered peanuts based on the three-dimensional morphology of depth information is combined with the three-dimensional peanut model obtained by the line structured light system to realize the two-dimensional adhesion image segmentation of peanuts. Compared with the existing adhesion segmentation algorithm, it has higher segmentation accuracy and lower computational complexity.

[0026] 2. This method of segmenting the two-dimensional grayscale image of adhered peanuts based on the three-dimensional morphology of depth information can not only obtain the surface features of the peanut shell, but also extract the height, sphericity and other features of the peanuts, which is conducive to the accuracy and rationality of the later quality inspection. It can also be applied to the segmentation of adhesion of other objects with irregular shapes, which has strong theoretical significance and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flowchart of peanut adhesion image segmentation based on three-dimensional morphology of a method for segmenting two-dimensional grayscale images of adhered peanuts based on three-dimensional morphology of depth information of the present invention;

[0028] Figure 2 It is a line structured light three-dimensional principle diagram of a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information of the present invention;

[0029] Figure 3 It is a schematic diagram of the three-dimensional reconstruction process of a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to the present invention;

[0030] Figure 4 This is a peanut three-dimensional reconstruction effect diagram of a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to the present invention;

[0031] Figure 5 It is a regional growth flow chart of a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to the present invention;

[0032] Figure 6 It is a three-dimensional segmentation diagram of a region growing algorithm of a two-dimensional grayscale image segmentation method of adhered peanuts based on three-dimensional morphology of depth information of the present invention;

[0033] Figure 7 It is a multi-scale feature matching segmentation flow chart of a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information of the present invention;

[0034] Figure 8 This is a peanut matching segmentation effect diagram of a two-dimensional grayscale image segmentation method of adhered peanuts based on three-dimensional morphology of depth information of the present invention;

[0035] Fig. 9 This is a comparison diagram of the open operation processing of a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to the present invention;

[0036] Fig.10 This is a peanut adhesion segmentation result diagram of a two-dimensional grayscale image segmentation method of peanut adhesion based on three-dimensional morphology of depth information of the present invention;

[0037] Fig.11 This is a comparison diagram of segmentation results of different algorithms of a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to the present invention;

[0038] Fig.12 This is a comparison table of segmentation results of different algorithms of a method for segmenting two-dimensional grayscale images of adhered peanuts based on three-dimensional morphology of depth information according to the present invention. DETAILED DESCRIPTION

[0039] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0040] like Figure 1-Figure 12 As shown, the present invention provides a technical solution: a method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information, comprising the following steps:

[0041] Step 1: Based on the principle of triangulation, build a line structured light imaging system, calibrate the camera and light plane of the system, and establish the transformation relationship from the camera coordinate system to the world coordinate system;

[0042] The line structured light imaging system includes a laser, an industrial camera and a one-dimensional motion guide rail, where the camera coordinate system (X c , Yc , Z c ) to the world coordinate system (X w , Y w , Z w ) is:

[0043]

[0044] Among them, R is the rotation matrix and T is the translation matrix. These two parameters are also called the external parameters of the camera and are determined by calibration;

[0045] Step 2: Use a line structured light imaging system to collect three-dimensional point cloud data of peanuts placed on the guide rails line by line, and stitch them together to obtain the three-dimensional morphology of the peanuts. In addition, collect images of adhered peanuts to be segmented;

[0046] The three-dimensional shape reconstruction specifically includes: considering that the camera and laser remain fixed in the system, after obtaining the three-dimensional point cloud coordinates of each frame of the line structured light image, the distance d of the object relative to the initial scanning position when the image is collected is calculated according to the order in which each picture is taken, so that the reconstructed multiple laser center lines are spliced ​​in the same direction in sequence according to the direction of movement of the guide rail to obtain the complete object contour. The specific calculation formula is: d = vt;

[0047] Among them, v is the movement speed of the peanut, in mm / s; t is the time from the camera starting to shoot to the shooting of the line structured light image, in s;

[0048] The 3D reconstruction process is described in detail as follows: Figure 3 As shown, firstly, image acquisition is performed, the acquired line structured light image is preprocessed, and the center line of the structured light stripe is extracted. According to the conversion relationship from the camera coordinate system to the world coordinate system in step 1, the three-dimensional coordinates of each independent laser stripe are calculated. According to the order in which each picture is taken, the distance d of the object relative to the initial scanning position when the image is acquired is calculated, so that the reconstructed multiple laser center lines are sequentially spliced ​​in the same direction according to the direction of movement of the guide rail to obtain the complete object contour. The specific calculation formula is: d = vt;

[0049] The three-dimensional reconstruction effect of adhesion peanuts is as follows Figure 4 As shown;

[0050] Step 3: Use normal vector and curvature as features of region growing algorithm to achieve 3D segmentation of point cloud;

[0051] The steps of calculating the normal vector and the curvature include: calculating the point cloud normal vector by using a local surface fitting method, and calculating the curvature of the sampling point according to the eigenvalue of the covariance matrix of the constructed domain point cloud set;

[0052] The steps of screening and segmenting the point cloud using the region growing algorithm include: constructing a KD-Tree structure of the point cloud, screening neighborhood points in the structure based on the normal angle and the curvature threshold, and gradually expanding to complete the point cloud segmentation of the peanut individual;

[0053] Moreover, the region growing algorithm is applied to the field of 3D segmentation. According to the non-convexity and gourd-like characteristics of peanut pods, the normal vector and curvature are selected as the features of the region growing algorithm to achieve accurate segmentation of the 3D model of the adhesion peanuts, laying a good foundation for auxiliary 2D adhesion image segmentation.

[0054] Specifically, the point cloud normal vector is calculated by the local surface fitting method, which is obtained by fitting the least squares plane to the target point and the sampling points in its surrounding area. The normal vector of the fitted plane is basically perpendicular to the direction of the line connecting the target point and each of its neighboring points. Assuming that the number of neighboring points is k, each spatial point is represented by [X1, Y2, Z3] T , all neighbor point clouds are represented as P i =[X 1i ,X 2i ,X 3i ](i=1,2,…,k), then the objective function expression with the target point normal vector n as the parameter to be determined is as follows:

[0055]

[0056] Let f(n)=min(n T Sn), where The constructed matrix S is the covariance matrix. By finding the minimum value of f(n), it can be seen that the normal vector n is an eigenvector of the covariance matrix S. The point cloud normal vector can be obtained by principal component analysis.

[0057] After the covariance matrix of the domain point cloud set is constructed, the curvature of the sampling point can be calculated according to the eigenvalues ​​of the matrix. Assuming that the three eigenvalues ​​of the matrix are λ1, λ2, and λ3, the calculation formula of the curvature ρ of the point is as follows:

[0058]

[0059] Then, regional growing segmentation is performed according to the required normal vector and curvature value. First, the point cloud data is sorted according to the curvature size, and the point with the minimum value is selected as the starting seed point to be added to the seed set. The neighborhood of the seed point is searched through the KD-Tree algorithm, and the normal angle and curvature of the domain point are calculated one by one. If both are lower than the preset threshold, the domain point is included in the seed set and removed from the original point cloud. This process continues to iterate until all point cloud data are segmented into independent regions or the seed set no longer grows. When the number of remaining point clouds is lower than the set minimum number of segmentation points, the algorithm automatically stops, thereby achieving accurate segmentation of the peanut model. Figure 6It is the three-dimensional segmentation map of the region growing algorithm;

[0060] Step 4: Use the rolling ball method to extract the two-dimensional contours of the individual peanut models after segmentation, and establish a search template;

[0061] Specifically, the rolling ball method is used to extract the edge contour of each peanut for matching and segmenting the two-dimensional adhesion image. The specific steps of establishing the search template include: image grayscale, threshold segmentation, connecting adjacent areas, reducing the image area, obtaining the shape template contour, and creating a shape scalable template;

[0062] Step 5: Based on the template, multi-scale feature matching is performed on the adhesion image to realize region positioning, and finally the segmentation region is optimized by opening operation to realize two-dimensional segmentation of the adhesion image;

[0063] The multi-scale feature matching specifically includes: for the adhesion image to be matched, the optimal matching position of the peanut template is explored from top to bottom at the appropriate image pyramid level. At the same time, the minimum distance accuracy threshold in the search process is set by the least squares method, and then the posture parameters of the template with sub-pixel accuracy, including rotation angle, scaling ratio, translation distance, etc., are obtained by minimizing the distance between the model point and its corresponding point in the image, so as to achieve accurate matching;

[0064] The details are as follows: Figure 7 As shown in FIG. 1 , the specific steps include: after the template is created, the optimal matching position of the peanut template is explored from top to bottom at the appropriate image pyramid level for the adhesion image to be matched. At the same time, the minimum distance accuracy threshold in the search process is set by the least squares method, and then the posture parameters of the template with sub-pixel accuracy, including rotation angle, scaling ratio, translation distance, etc., are obtained by minimizing the distance between the model point and its corresponding point in the image, so as to achieve accurate matching. The matching segmentation effect is shown in FIG. Figure 8 As shown;

[0065] Since the peanut point cloud may be missing some of the peanut edges during the 3D segmentation and subsequent edge contour extraction process, such as concave edges, uneven contours and sharp corners, in order to ensure the integrity of the peanut shape and facilitate the effective extraction of peanut features for subsequent quality inspection, the matching peanut target area is repaired using an open operation to make the peanut contour smoother and more complete, and to eliminate fine burrs. Fig. 9 shown.

[0066] Fig.10 This is the segmentation result of the algorithm of the embodiment of the present invention for the adhered peanuts. The edges of the regions after the segmentation of the six peanuts are relatively smooth, with high integrity and clear surface texture, which is conducive to the subsequent extraction of peanut phenotypic parameters and easy classification;

[0067] Step 6: Use model evaluation indicators to evaluate model performance;

[0068] The model evaluation index is segmentation accuracy, which is calculated as follows:

[0069] Among them, N is the number of pixels of a single peanut after matching the segmentation algorithm, and M is the number of complete pixels of a single peanut;

[0070] In the embodiment, the watershed algorithm based on distance transformation, the watershed algorithm based on gradient transformation and the algorithm in the embodiment of the present invention are respectively used to segment the peanut adhesion image. Figure 11-Figure 12 As shown, the following conclusions can be drawn: the present invention combines the three-dimensional peanut model obtained by the line structured light system to realize the two-dimensional adhesion image segmentation of peanuts, and the segmentation accuracy is significantly improved. Not only can the surface features of the peanut shell be obtained, but also the height, sphericity and other features of the peanut can be extracted, which is beneficial to the accuracy and rationality of the subsequent quality inspection.

[0071] Based on the above description, the present invention improves the traditional two-dimensional adhesion segmentation method, uses the three-dimensional point cloud morphology representing height changes to assist the two-dimensional adhesion segmentation, and uses the characteristics of different peanuts with height mutations to solve the problem of difficult segmentation of objects such as peanut pods with non-fully convex appearance, complex surface texture, diverse structure, and similar colors.

[0072] The peanuts that are separated by adhesion can not only obtain two-dimensional color, texture, geometric structure and other information, but also obtain peanut height, sphericity and other information based on the obtained three-dimensional peanut morphology, thereby realizing accurate quality detection.

[0073] The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information, characterized in that: The steps include: Step 1: Based on the principle of triangulation, build a line structured light imaging system, calibrate the camera and light plane of the system, and establish the transformation relationship from the camera coordinate system to the world coordinate system; Step 2: Use a line structured light imaging system to collect three-dimensional point cloud data of peanuts placed on the guide rails line by line, and stitch them together to obtain the three-dimensional morphology of the peanuts. In addition, collect images of adhered peanuts to be segmented; Step 3: Use normal vector and curvature as features of region growing algorithm to achieve 3D segmentation of point cloud; Step 4: Use the rolling ball method to extract the two-dimensional contours of the individual peanut models after segmentation, and establish a search template; Step 5: Based on the template, multi-scale feature matching is performed on the adhesion image to realize region positioning, and finally the segmentation region is optimized by opening operation to realize two-dimensional segmentation of the adhesion image; Step 6: Use model evaluation indicators to evaluate model performance.

2. The method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to claim 1, characterized in that: In step 1, the line structured light imaging system includes a laser, an industrial camera and a one-dimensional motion guide rail, wherein the camera coordinate system (X c , Y c , Z c ) to the world coordinate system (X w , Y w , Z w ) is:

3. The method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to claim 2, characterized in that: R is the rotation matrix, and T is the translation matrix. These two parameters are also called the external parameters of the camera and are determined by calibration.

4. The method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to claim 1, characterized in that: In step 2, the three-dimensional morphology reconstruction specifically includes: considering that the camera and the laser remain fixed in the system, after obtaining the three-dimensional point cloud coordinates of each frame of the line structured light image, the distance d of the object relative to the initial scanning position when the image is acquired is calculated according to the order in which each picture is taken, so that the reconstructed multiple laser center lines are spliced ​​in the same direction in sequence according to the direction of movement of the guide rail to obtain the complete object contour. The specific calculation formula is: d=vt.

5. The method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to claim 4, characterized in that: The v is the movement speed of the peanut, in mm / s; t is the time from the start of the camera shooting to the shooting of the line structured light image, in s.

6. The method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to claim 1, characterized in that: In step 3, the steps of calculating the normal vector and the curvature include: calculating the point cloud normal vector using a local surface fitting method, and calculating the curvature of the sampling point according to the eigenvalues ​​of the covariance matrix of the constructed domain point cloud set.

7. The method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to claim 1, characterized in that: In step 3, the step of using the region growing algorithm to screen and segment the point cloud includes: constructing a KD-Tree structure of the point cloud, and screening neighborhood points in this structure based on the normal angle and curvature threshold, and gradually expanding to complete the point cloud segmentation of individual peanuts.

8. The method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to claim 1, characterized in that: In step 4, the specific steps of establishing the search template include: graying the image, threshold segmentation, connecting adjacent areas, reducing the image area, obtaining the shape template outline, and creating a shape scalable template.

9. The method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to claim 1, characterized in that: In step 5, the multi-scale feature matching specifically includes: for the adhesion image to be matched, the optimal matching position of the peanut template is explored from top to bottom at the appropriate image pyramid level, and at the same time, the minimum distance accuracy threshold in the search process is set by the least squares method, and then the posture parameters of the template with sub-pixel accuracy, including rotation angle, scaling ratio, translation distance, etc., are obtained by minimizing the distance between the model point and its corresponding point in the image, so as to achieve accurate matching.

10. The method for segmenting a two-dimensional grayscale image of adhered peanuts based on three-dimensional morphology of depth information according to claim 1, characterized in that: In step 6, the model evaluation index is the segmentation accuracy, which is specifically calculated as: Among them, N is the number of pixels of a single peanut after matching the segmentation algorithm, and M is the number of complete pixels of a single peanut.

Images