A method for detecting and locating cracks in macadamia nuts

Abstract

The present invention relates to a method for detecting and locating cracks in macadamia nuts, and belongs to the technical field of computer vision target detection and measurement. A crack detection method comprises the following steps: firstly performing image processing and crack extraction on a picture captured by a CCD camera; and then obtaining a predicted surface crack image of macadamia nuts through deep learning. A crack positioning method comprises the following steps: (I) calibrating a binocular camera to obtain internal and external parameters of the camera, and using left and right cameras to capture macadamia nut pictures; (II) using a feature extractor to extract feature points of the left and right images captured by the binocular camera; (III) using a matcher to match the feature points extracted in step (II); (IV) for the internal and external parameters of the camera obtained in step (I), calculating the three-dimensional coordinates of the feature points successfully matched in step (III) based on the optical center position of the left camera as the origin of the spatial coordinates; and (V) performing plane fitting by obtaining the three-dimensional coordinates of three or more matching points on the crack.

Description

Technical Field

[0001] The present invention relates to a macadamia nut crack detection and positioning method, belonging to the technical field of computer vision target detection and measurement. Background Art

[0002] As people's living standards improve, nut foods are becoming increasingly popular. Nutritious macadamia nuts are particularly popular. However, handling the hard shell of macadamia nuts during processing poses a significant challenge. Due to their hard shell, they often require a machine to crack open so that consumers can easily consume them. Research and investigations have found that macadamia shells naturally develop tiny cracks to ensure the seeds can fall and grow naturally. Traditional nut-opening machines simply cut the shell surface, failing to fully exploit this natural process or consider the size and shape of the macadamia nut to determine the opening position. This results in nut products on the market often experiencing problems such as difficulty opening the shell, low kernel integrity, and kernel sticking. This negatively impacts the consumer experience. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method for detecting and locating cracks in macadamia nuts, which can effectively solve the problems existing in the existing technology and fill the gaps in this technology.

[0004] To achieve the above objectives, the present invention provides a crack detection method for detecting and locating cracks in macadamia nuts, comprising the following steps:

[0005] Step 1: Crack detection:

[0006] Step 1: Perform image processing and crack extraction on the images captured by the CCD camera;

[0007] Step 2: Obtain the predicted macadamia nut surface crack image through deep learning;

[0008] Step 2: Locate the cracks on the macadamia nuts whose shells are detected to have cracks.

[0009] Specifically, the step 1 is as follows:

[0010] Step 1.1: The image captured by the CCD camera is grayscaled and then sharpened to enhance the crack features on the surface of the macadamia nut;

[0011] In step 1.2, perform mean shift calculation on the image processed in step 1.1 to smooth out unnecessary image details and eliminate small color areas. The mean shift formula is as follows:

[0012]

[0013] Among them, K(x i -x) is a given kernel function, x is a set unknown number, typically a Gaussian kernel, and N(x) is x i The set of m(x) is the density-weighted average of x.

[0014] Step 1.3, perform median filtering on the image processed in step 1.2 to eliminate image noise;

[0015] Step 1.4: Use the Canny algorithm to extract edges from the noise-removed image obtained in step 1.3. The extracted contour information includes the crack contour and the external contour of the macadamia nut.

[0016] Step 1.5, perform image dilation processing on the edge extracted in step 1.4 to fill the crack outline and the outer contour;

[0017] In step 1.6, the image captured by the CCD camera is processed using a median filter with a large kernel size to blur the image information on the surface of the macadamia nuts.

[0018] Step 1.7: Use the Canny algorithm to extract the edge of the image obtained in step 1.6. The extracted contour information is the outer contour of the macadamia nut.

[0019] Step 1.8, dilate the edge extracted in step 1.7 and fill the extracted outer contour;

[0020] In step 1.9, perform image subtraction between the image processed in step 1.5 and the image processed in step 1.8. The image subtraction formula is as follows:

[0021] g(x0,y0)=f1(x0,y0)-f2(x0,y0)

[0022] g(x0,y0) represents the output image pixel value, f1(x0,y0) represents the pixel value of the first input image at the coordinate (x0,y0), and f2(x0,y0) represents the pixel value of the second input image at the same coordinate;

[0023] Step 1.10: For the image obtained in step 1.9, perform maximum connected region extraction to extract the crack features in the image separately.

[0024] Step 1.11, refine the image obtained in step 1.10, and peel off the light stripe boundaries by continuously corroding the image contour to obtain light stripe connected lines with a single pixel width.

[0025] Specifically, the step 2 is as follows:

[0026] Step 2.1, extracting surface cracks from a large number of macadamia nut images using the method of step 1, to obtain a large number of different macadamia nut crack images;

[0027] In step 2.2, the large number of different macadamia nut crack images obtained in step 2.1 are used as a training set and input into the neural network for training, so that the neural network has excellent crack detection capabilities.

[0028] Specifically, the step 2 includes the following steps:

[0029] (I) Calibrate the binocular camera to obtain the camera's internal and external parameters, and use the left and right cameras to collect images of macadamia nut cracks respectively;

[0030] (II) extracting feature points of the left and right images captured by the binocular camera using a feature extractor;

[0031] (III) using a matcher to match the feature points extracted in step (II) and screen out the correct matching points;

[0032] (IV) For the camera internal and external parameters obtained in step (I), calculate the three-dimensional coordinates of the feature points successfully matched in step (III) based on the optical center position of the left camera as the spatial coordinate origin;

[0033] (V) Obtain the three-dimensional coordinates of three or more matching points on the crack for cutting plane fitting.

[0034] Specifically, the step (I) is as follows:

[0035] Multiple checkerboard photos were acquired through the left and right cameras of the binocular camera. The MATLAB calibration toolbox was then used to calibrate the binocular camera. The internal parameter matrices K1 and K2 of the left and right cameras, the distortion matrices D1 and D2 of the left and right cameras, as well as the R rotation matrix and T translation matrix between the two cameras were obtained. The left and right cameras were used to collect images of macadamia nut cracks, respectively.

[0036] Specifically, the step (II) is as follows:

[0037] For the macadamia nut crack image obtained in step (I), SIFT feature extraction is performed to extract the feature points of the left and right images.

[0038] Specifically, the step (III) is as follows:

[0039] For the feature points of the left and right images extracted in step (II), the FLANN matcher is used to match the feature points of the left and right images to obtain the initial matching results. Then, the Lowe's ratio test is used to screen out the correct matching points and the matching results are plotted.

[0040] Specifically, the step (IV) is as follows:

[0041] Calculate the projection matrices P1 and P2 of the left and right cameras. The projection matrices P1 and P2 are obtained by obtaining the internal parameter matrices K1 and K2 of the left and right cameras and the R rotation matrix and T translation matrix between the two cameras through step (I). Then use the cv2.triangulatePoints function to calculate the three-dimensional homogeneous coordinates points4D of the matching points. Divide the homogeneous coordinates points4D by the fourth component to obtain the three-dimensional coordinates of the homogeneous coordinate points. The principle is as follows:

[0042] Two cameras, their projection matrices are P1 and P2 respectively. For a feature point, the coordinates in the first image are (x1, y1), and the coordinates in the second image are (x2, y2);

[0043] First, convert the homogeneous coordinates ([x1, y1, 1]) to normalized plane coordinates ([X1, Y1, Z1]) in the world coordinate system using the following formula:

[0044]

[0045] Where w is the vector inner product between (x1, y1) and the last column of P1;

[0046] Similarly, ([x2,y2,1]) is converted to normalized plane coordinates ([X2,Y2,Z2]) using the following formula:

[0047]

[0048] Next, according to the P1 and P2 projection matrices, we get the equations:

[0049] P1*[X1,Y1,Z1,1]T=0P2*[X2,Y2,Z2,1]T=0

[0050] The above equations represent the projection relationship between the two cameras. The cv2.triangulatePoints function calculates the three-dimensional coordinates of the feature points by solving this equation.

[0051] Finally, the result in homogeneous coordinate form is normalized to obtain the three-dimensional coordinates (X, Y, Z):

[0052]

[0053] Among them, Xc, Yc, Zc are the first three rows of the homogeneous coordinate points4D matrix, and W is the fourth row of the homogeneous coordinate points4D matrix.

[0054] Specifically, the step (V) is as follows:

[0055] Import the three-dimensional coordinates obtained in step (IV) into MATLAB, and calculate its SVD decomposition based on the covariance matrix of the input three-dimensional coordinate points. In the SVD decomposition result, the right singular vector corresponding to the minimum singular value is the normal vector of the plane where the point cloud is located, and store it in the created vector [a, b, c]. At the same time, find the constants in the plane fitting equation. After obtaining these parameters, realize the fitting plane, that is, plane fitting can be achieved by obtaining the plane's normal vector and a point on the normal vector.

[0056] The beneficial effects of the present invention are as follows: the present invention addresses the current problems and combines natural laws to detect and locate naturally formed cracks on the surface of macadamia nuts, so as to facilitate subsequent cutting and opening of the macadamia nuts, thereby improving the quality of macadamia nut products and making the consumer eating experience more perfect. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of the present invention;

[0058] Figure 2 This is the original image of macadamia nuts;

[0059] Figure 3 This is the sharpening processing image;

[0060] Figure 4 This is the mean shift method processing diagram;

[0061] Figure 5 This is the overall outline diagram;

[0062] Figure 6 is the large kernel median fuzzy image;

[0063] Figure 7 is the edge contour map;

[0064] Figure 8 This is a subtraction processing diagram;

[0065] Figure 9 Extract graph for maximum connected region;

[0066] Figure 10 is the crack refinement diagram;

[0067] Figure 11 It is the binocular feature matching map;

[0068] Figure 12 Plane fitting plot. DETAILED DESCRIPTION

[0069] The following will be combined with specific embodiments and drawings to clearly and in detail describe the technical solutions in the embodiments of the present invention. The described embodiments are only used to explain the present invention and are not used to limit the present invention.

[0070] Example 1: A method for detecting and locating cracks in macadamia nuts, such as Figure 1 As shown, the following steps are included:

[0071] Step 1: Crack detection:

[0072] Step 1: Perform image processing and crack extraction on the images captured by the CCD camera;

[0073] Step 2: Obtain the predicted macadamia nut surface crack image through deep learning;

[0074] Step 2: Locate the cracks on the macadamia nuts whose shells are detected to have cracks.

[0075] Furthermore, the step 1 is specifically as follows:

[0076] Step 1.1, the CCD camera captures Figure 2 The original image of macadamia nuts is grayscaled and then sharpened. Figure 3 The sharpening convolution kernel size is [[-1,-1,-1],[-1,9,-1],[-1,-1,-1]]. After sharpening, the crack features on the surface of macadamia nuts are significantly enhanced. After processing, Figure 3 The crack characteristics of the macadamia nut surface are significantly enhanced;

[0077] Step 1.2: Since the surface of macadamia nuts not only has the main crack features that need to be extracted, but also has small color blocks, small holes, secondary cracks and other noises, after processing in step 1.1, the noise features will also be enhanced. Figure 3 Perform mean shift calculation, set the filter color radius (color_radius) to 30, and the spatial radius (spatial_______radius) to 6,

[0078] The mean shift method formula is as follows:

[0079]

[0080] Among them, K(x i -x) is a given kernel function, x is a set unknown number, N(x) is x i The set of , m(x) is the density-weighted average of x;

[0081] The mean shift method was used to calculate the Figure 4, after processing, the loss of crack features in the image is not obvious, and the noise points on the image are much smaller than the original image. It can be observed Figure 4 Unwanted image details on the surface of the macadamia nut shell are smoothed out, and some small areas of color are eliminated;

[0082] Step 1.3, after processing in step 1.2 Figure 4 Perform median filtering to eliminate image noise;

[0083] Step 1.4: Use the Canny algorithm to perform edge extraction on the noise-removed image obtained in step 1.3. At this time, the overall outline of the macadamia nut is extracted, including the crack outline and external outline of the macadamia nut;

[0084] Step 1.5: Perform image expansion on the edge extracted in step 1.4 and fill the crack outline and the outer contour to obtain Figure 5 ;

[0085] Step 1.6: Use a median filter with a large kernel value of 13 to process the image 2 captured by the CCD camera to blur the image information on the surface of the macadamia nut. Figure 6 ;

[0086] Step 1.7, for the data obtained in step 1.6 Figure 6 ,Use the Canny algorithm to extract the edge. The extracted contour information is the outer contour of the macadamia nut;

[0087] Step 1.8: dilate the edge extracted in step 1.7 and fill the extracted outer contour to obtain Figure 7 ;

[0088] Step 1.9, after step 1.5 Figure 5 , after processing in step 1.8 Figure 7 Perform image subtraction to obtain image 8 containing only the crack outline and the unsubtracted outline information; the image subtraction formula is as follows:

[0089] g(x0,y0)=f1(x0,y0)-f2(x0,y0)

[0090] g(x0,y0) represents the output image pixel value, f1(x0,y0) represents the pixel value of the first input image at the coordinate (x0,y0), and f2(x0,y0) represents the pixel value of the second input image at the same coordinate;

[0091] Step 1.10, for the data obtained in step 1.9 Figure 8 , extract the maximum connected area and obtain the image containing only crack information Figure 9 ;

[0092] Step 1.11, refine the image obtained in step 1.10, and peel off the light stripe boundary by continuously corroding the image contour to obtain the light stripe connection line (also known as the skeleton) with a single pixel width. Figure 10 .

[0093] Furthermore, the step 2 is specifically as follows:

[0094] Step 2.1: extracting surface cracks from a large number of macadamia nut images using the method of step 1 to obtain a large number of different macadamia nut crack images;

[0095] Step 2.2 uses the large number of different macadamia nut crack images obtained in step 2.1 as a training set and inputs them into the neural network for training, so that the neural network has excellent crack detection capabilities.

[0096] Furthermore, the step 2 includes the following steps:

[0097] (I) Calibrate the binocular camera to obtain the camera's internal and external parameters, and use the left and right cameras to collect images of macadamia nut cracks respectively;

[0098] (II) extracting feature points of the left and right images captured by the binocular camera using a feature extractor;

[0099] (III) using a matcher to match the feature points extracted in step (II) and screen out the correct matching points;

[0100] (IV) For the camera internal and external parameters obtained in step (I), calculate the three-dimensional coordinates of the feature points successfully matched in step (III) based on the optical center position of the left camera as the spatial coordinate origin;

[0101] (V) Because the macadamia nut opening process does not require excessive precision, the processing speed should not be too slow, and the thickness of the cutting tool far exceeds the positioning accuracy, the cutting plane fitting can be performed by only obtaining the three-dimensional coordinates of three or more matching points on the crack.

[0102] Furthermore, step (I) is specifically as follows: a plurality of checkerboard photos are acquired by the left and right cameras of the binocular camera, and then the binocular camera is calibrated using the MATLAB calibration toolbox to obtain the internal parameter matrices K1 and K2 of the left and right cameras and the distortion matrices D1 and D2 of the left and right cameras, as well as the R rotation matrix and T translation matrix between the two cameras, and the left and right cameras are used to respectively capture images of macadamia nut cracks;

[0103] Furthermore, step (II) is specifically as follows: performing SIFT feature extraction on the macadamia crack image obtained in step (I) to extract feature points of the left and right images;

[0104] Furthermore, step (III) is specifically as follows: for the feature points of the left and right images extracted in step (II), the feature points of the left and right images are matched using the FLANN matcher to obtain the initial matching results, and then the correct matching points are screened using the Lowe's ratio test, and the matching results are plotted. Figure 11 ;

[0105] Furthermore, step (IV) is specifically as follows: calculate the projection matrices P1 and P2 of the left and right cameras. The projection matrices P1 and P2 are obtained by obtaining the internal parameter matrices K1 and K2 of the left and right cameras and the R rotation matrix and T translation matrix between the two cameras in step (I). Then, the cv2.triangulatePoints function is used to calculate the three-dimensional homogeneous coordinates points4D of the matching points. The homogeneous coordinates points4D are divided by the fourth component to obtain the three-dimensional coordinates of the homogeneous coordinate points. The principle is as follows:

[0106] Two cameras, their projection matrices are P1 and P2 respectively. For a feature point, the coordinates in the first image are (x1, y1), and the coordinates in the second image are (x2, y2);

[0107] First, convert the homogeneous coordinates ([x1, y1, 1]) to normalized plane coordinates ([X1, Y1, Z1]) in the world coordinate system using the following formula:

[0108]

[0109] Where w is the vector inner product between (x1, y1) and the last column of P1;

[0110] Similarly, ([x2,y2,1]) is converted to normalized plane coordinates ([X2,Y2,Z2]) using the following formula:

[0111]

[0112] Next, according to the P1 and P2 projection matrices, we get the equations:

[0113] P1*[X1,Y1,Z1,1]T=0P2*[X2,Y2,Z2,1]T=0

[0114] The above equations represent the projection relationship between the two cameras. The cv2.triangulatePoints function calculates the three-dimensional coordinates of the feature points by solving this equation.

[0115] Finally, the result in homogeneous coordinate form is normalized to obtain the three-dimensional coordinates (X, Y, Z):

[0116]

[0117] Among them, Xc, Yc, Zc are the first three rows of the homogeneous coordinate points4D matrix, and W is the fourth row of the homogeneous coordinate points4D matrix.

[0118] Furthermore, step (V) is specifically as follows: click Figure 11 Matching points on the cracks of macadamia nuts, import the three-dimensional coordinates obtained in step IV into MATLAB, and calculate its SVD decomposition based on the covariance matrix of the input three-dimensional coordinate points. In the SVD decomposition result, the right singular vector corresponding to the minimum singular value is the normal vector of the plane where the point cloud is located, and store it in the created vector [a, b, c]. At the same time, find the constants in the plane fitting equation. After obtaining these parameters, the plane is fitted, that is, the plane fitting can be achieved by obtaining the normal vector of the plane and a point on the normal vector, and we get Figure 12 .

[0119] The above embodiments should be understood as merely illustrating the present invention and not as limiting the scope of protection of the present invention. After reading the contents of the present invention, technicians may make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A method for detecting and locating cracks in macadamia nuts, characterized by: The following steps are involved: Step 1: Crack detection: Step 1: Perform image processing and crack extraction on the images captured by the CCD camera; Step 2: Obtain the predicted macadamia nut surface crack image through deep learning; Step 2: locating cracks on macadamia nuts with cracks detected on the shell surface; The step 2 comprises the following steps: (I) Calibrate the binocular camera to obtain the camera's internal and external parameters, and use the left and right cameras to collect images of macadamia nut cracks respectively; (II) extracting feature points of the left and right images captured by the binocular camera using a feature extractor; (III) using a matcher to match the feature points extracted in step (II) and screen out the correct matching points; (IV) For the camera internal and external parameters obtained in step (I), calculate the three-dimensional coordinates of the feature points successfully matched in step (III) based on the optical center position of the left camera as the spatial coordinate origin; (V) obtaining the three-dimensional coordinates of three or more matching points on the crack for cutting plane fitting; The step (I) is specifically: Multiple checkerboard images were acquired using the left and right cameras of the binocular camera. The cameras were then calibrated using the MATLAB calibration toolbox to obtain the internal parameter matrices K1 and K2 of the left and right cameras, the distortion matrices D1 and D2 of the left and right cameras, and the R rotation matrix and T translation matrix between the two cameras. Images of macadamia nut cracks were then collected using the left and right cameras. The step (IV) is specifically: Calculate the projection matrices P1 and P2 of the left and right cameras. The projection matrices P1 and P2 are obtained by obtaining the internal parameter matrices K1 and K2 of the left and right cameras and the R rotation matrix and T translation matrix between the two cameras through step (I). Then use the cv2.triangulatePoints function to calculate the three-dimensional homogeneous coordinates points4D of the matching points. Divide the homogeneous coordinates points4D by the fourth component to obtain the three-dimensional coordinates of the homogeneous coordinate points. The principle is as follows: Two cameras, their projection matrices are P1 and P2 respectively. For a feature point, the coordinates in the first image are (x1, y1), and the coordinates in the second image are (x2, y2); First, convert the homogeneous coordinates ([x1, y1, 1]) to normalized plane coordinates ([X1, Y1, Z1]) in the world coordinate system using the following formula: Where w is the vector inner product between (x1, y1) and the last column of P1; Similarly, ([x2,y2,1]) is converted to normalized plane coordinates ([X2,Y2,Z2]) using the following formula: Next, according to the P1 and P2 projection matrices, we get the equations: The above equations represent the projection relationship between the two cameras. The cv2.triangulatePoints function calculates the three-dimensional coordinates of the feature points by solving this equation. Finally, the result in homogeneous coordinate form is normalized to obtain the three-dimensional coordinates (X, Y, Z): Among them, Xc, Yc, Zc are the first three rows of the homogeneous coordinate points4D matrix, and W is the fourth row of the homogeneous coordinate points4D matrix.

2. The method for detecting and locating cracks in macadamia nuts according to claim 1, wherein: The step 1 is specifically as follows: Step 1.1: The image captured by the CCD camera is grayscaled and then sharpened to enhance the crack features on the surface of the macadamia nut; Step 1.2: Perform mean shift calculation on the image processed in step 1.1 to smooth out unnecessary image details and eliminate small color areas. The mean shift formula is as follows: Among them, K(x i -x) is a given kernel function, x is a set unknown number, N(x) is x i The set of , m(x) is the density-weighted average of x; Step 1.3, perform median filtering on the image processed in step 1.2 to eliminate image noise; Step 1.4: Use the Canny algorithm to extract edges from the noise-removed image obtained in step 1.

3. The extracted contour information includes the crack contour and the external contour of the macadamia nut. Step 1.5, perform image dilation processing on the edge extracted in step 1.4 to fill the crack outline and the outer contour; In step 1.6, the image captured by the CCD camera is processed using a median filter with a large kernel size to blur the image information on the surface of the macadamia nuts. Step 1.7: Use the Canny algorithm to extract the edge of the image obtained in step 1.

6. The extracted contour information is the outer contour of the macadamia nut. Step 1.8, dilate the edge extracted in step 1.7 and fill the extracted outer contour; In step 1.9, perform image subtraction between the image processed in step 1.5 and the image processed in step 1.

8. The image subtraction formula is as follows: g(x0,y0)=f1(x0,y0)-f2(x0,y0) g(x0,y0) represents the output image pixel value, f1(x0,y0) represents the pixel value of the first input image at the coordinate (x0,y0), and f2(x0,y0) represents the pixel value of the second input image at the same coordinate; Step 1.10: For the image obtained in step 1.9, perform maximum connected region extraction to extract the crack features in the image separately. Step 1.11, refine the image obtained in step 1.10, and peel off the light stripe boundaries by continuously corroding the image contour to obtain light stripe connected lines with a single pixel width.

3. The method for detecting and locating cracks in macadamia nuts according to claim 1, wherein: The step 2 is specifically as follows: Step 2.1, extracting surface cracks from a large number of macadamia nut images using the method of step 1, to obtain a large number of different macadamia nut crack images; In step 2.2, the large number of different macadamia nut crack images obtained in step 2.1 are used as a training set and input into the neural network for training, so that the neural network has excellent crack detection capabilities.

4. The method for detecting and locating cracks in macadamia nuts according to claim 1, wherein: The step (II) is specifically: For the macadamia nut crack image obtained in step (I), SIFT feature extraction is performed to extract the feature points of the left and right images.

5. The method for detecting and locating cracks in macadamia nuts according to claim 1, wherein: The step (III) is specifically: For the feature points of the left and right images extracted in step (II), the FLANN matcher is used to match the feature points of the left and right images to obtain the initial matching results. Then, the Lowe's ratio test is used to screen out the correct matching points and the matching results are plotted.

6. The method for detecting and locating cracks in macadamia nuts according to claim 1, wherein: The step (V) is specifically: Import the three-dimensional coordinates obtained in step (IV) into MATLAB, and calculate its SVD decomposition based on the covariance matrix of the input three-dimensional coordinate points. In the SVD decomposition result, the right singular vector corresponding to the minimum singular value is the normal vector of the plane where the point cloud is located, and store it in the created vector [a, b, c]. At the same time, find the constants in the plane fitting equation. After obtaining these parameters, realize the fitting plane, that is, plane fitting can be achieved by obtaining the plane's normal vector and a point on the normal vector.

Images