Citrus defect detection method based on pedicel influence

Abstract

The invention discloses a citrus defect detection method based on pedicel influence, which comprises the following steps of: calibrating through a binocular horizontal moving camera, acquiring accurate geometric parameters of the camera, processing a binocular image by adopting an image registration and fusion technology, and extracting a pedicel contour through means such as color space conversion, graying and morphological processing. Depth information is obtained by using a three-dimensional reconstruction technology, and an accurate pedicel image is obtained by applying Otus binarization filtering and morphological closed operation; according to the method, a self-adaptive Gamma correction algorithm based on orange multi-scale texture fusion and asymmetric brightness diffusion characteristics is improved, a defect and pedicel mixed region is corrected, a pedicel image is removed through a mask method, a defect region is highlighted, finally, an active contour model and a Green formula are utilized, the orange area and the defect area are calculated respectively, and finally, the defect area is calculated. And comparing with a set value to realize accurate citrus defect detection.

Description

Technical Field

[0001] The present invention relates to a method for detecting citrus defects, and particularly to a method for detecting citrus defects under the influence of fruit stalks. Background Art

[0002] In recent years, diseases of citrus crops have posed a great threat to agricultural production. Traditional manual visual inspection has low efficiency, poor accuracy, and is easily affected by subjective factors. To improve the efficiency and accuracy of citrus defect detection, using machine vision technology for defect recognition has become a research hotspot. Existing recognition technologies mainly use the watershed algorithm to segment citrus fruit images, and identify them by calculating the shape, texture, and color features of the segmented regions.

[0003] However, existing recognition technologies have obvious defects. On the one hand, when the watershed algorithm processes the fruit stalk region, it is easy to misjudge it as a defect, resulting in inaccurate recognition results and unable to restore the actual area of citrus defects. On the other hand, the watershed algorithm is difficult to accurately distinguish some minor defects with insignificant differences from normal regions, resulting in inaccurate detection results. Summary of the Invention

[0004] Object of the Invention: The object of the present invention is to provide a method for detecting citrus defects under the influence of fruit stalks to improve the detection accuracy and reliability, thereby effectively guaranteeing the quality of citrus and reducing economic losses caused by diseases.

[0005] Technical Solution: A method for detecting citrus defects under the influence of fruit stalks according to the present invention includes the following steps:

[0006] (1) Calibration of binocular horizontally moving cameras: Install and fix binocular horizontally moving cameras, take target images of the calibration board, establish a model, and obtain parameters.

[0007] (2) Image preprocessing: Fuse the collected images using the chromaticity H in the HSV color space with three-dimensional reconstruction to obtain an accurate fruit stalk image, then use an adaptive Gamma algorithm based on citrus multi-scale texture fusion and asymmetric luminance diffusion characteristics for correction, and use the masking method to remove the fruit stalk image to obtain an accurate defect region.

[0008] (3) Object detection of images: Perform numerical integration operations on the areas of irregular citrus elliptical contours and complex evaluations of Gaussian integrals, and compare with the set values.

[0009] Preferably, the parameters in step 1 include internal parameters, external parameters, and motion parameters of the horizontally moving camera.

[0010] The internal parameters include the horizontal magnification coefficient of the camera, the vertical magnification coefficient of the camera, the coupling magnification coefficient in the horizontal direction, the coupling magnification coefficient in the vertical direction, the abscissa of the center of the image captured by the camera, the ordinate of the center of the image captured by the camera, the tangential distortion coefficient, and the radial distortion coefficient;

[0011] The external parameters include the rotation matrix and the translation matrix between the cameras.

[0012] Preferably, the specific steps of the image preprocessing in step 2 are as follows:

[0013] (21) The binocular cameras synchronously transmit the captured images by using the frame synchronization method;

[0014] (22) After synchronous transmission, image registration is performed, and the registered pictures are fused by using the direct average fusion method;

[0015] (23) Perform color space conversion from RGB to HSV on the target image, extract the H component image, and then grayscale it;

[0016] (24) Use the Canny operator to remove the citrus background and obtain a blurred image of the fruit stalk;

[0017] (25) Perform three-dimensional reconstruction on the blurred image to obtain a depth image, and use the Otus method for binaryzation and morphological closing operation to obtain an accurate fruit stalk image;

[0018] (26) Use the adaptive Gamma correction algorithm to correct the grayscale image to obtain a mixed area of defects and fruit stalks, and finally use the masking method to remove the fruit stalk image to obtain an accurate defect area.

[0019] Preferably, the specific process of the image registration in step 22 is as follows: Rotate the target image to make it parallel to the corresponding reference image plane, rotate the image according to the calculated HomMat2D matrix, then crop a part of the image to cut off the unnecessary background area, and finally calculate the initial translation amount in the image according to the calculated projective transformation matrix to register the target image with the base image.

[0020] Preferably, when removing the citrus background in step 24, the Gaussian filter used by the Canny operator has a specific standard deviation, the gradient amplitude and direction are obtained by calculating the partial derivatives of the image pixel points in the horizontal and vertical directions, and edge detection and threshold processing are performed according to the set high threshold and low threshold, and finally the citrus image is obtained through morphological closing operation.

[0021] Preferably, in the process of obtaining the accurate calyx image in step 25, first perform median filtering on the hue H and take the difference to detect the approximate position of the calyx, then perform morphological closing operation to make the calyx region more continuous and complete, and then perform three-dimensional reconstruction on the blurred region, determine the appropriate threshold using the Otus method for binary processing, and fill the holes through morphological closing operation to obtain the accurate calyx image.

[0022] Preferably, the formula of the adaptive Gamma algorithm in step 2 is as follows:

[0023]

[0024] Contrast l (θ,d) = ∑ i,j |i - j| 2 P l (θ,d)(i,j);

[0025]

[0026] Energy l (θ,d) = ∑ i,j P l (θ,d)(i,j) 2 ;

[0027] Entropy l (θ,d) = -∑ i,j P l (θ,d)(i,j)logP l (θ,d)(i,j);

[0028]

[0029]

[0030] E(L0,σ x ,σ y ) = ∑ x,y [L(x,y) - I real (x,y)] 2 ;

[0031] I'(x,y) = I(x,y) γ(x,y) ;

[0032] Among them, i and j represent pixel coordinates, l represents the number of pyramid layers, {G0, G1 ··· G l} represents the image sequence of different scales, G0 represents the original image; P l (θ,d) represents the gray-level co-occurrence matrix, representing the occurrence probability that the gray values of two pixel points separated by a distance d in the direction θ are i and j respectively; Contrastl (θ, d) represents extracting contrast from each P l (θ, d), and Correlation l (θ, d) represents correlation, μ i and μ j represent the mean gray value, σ i and σ j represent the standard deviation of gray value; Energy l (θ, d) represents energy, Entropy l (θ, d) represents entropy, and T represents the multi-scale texture feature vector; I highlight represents the average gray value of the highlight area, n i represents the normal vector, represents the gray gradient of the highlight area, I shadow represents the average gray value, represents the gray gradient of the shadow area, (x i , y i ) represents the coordinates of the citrus contour points, n i represents the normal vector, λ represents the regularization parameter, h i represents the gray value of the pixel points in the highlight area, g i represents the gray value of the pixel points in the shadow area;

[0033] s represents the light source direction, and s is obtained by solving ;

[0034] (x, y) represents the pixel coordinates, (x0, y0) represents the image center or relative center coordinates, L0 represents the central luminance, σ x and σ y represent the diffusion coefficients in the horizontal and vertical directions, and σ x ≠σ y , and I(x, y) represents the image pixel luminance value.

[0035] Preferably, the adaptive Gamma correction algorithm described in step 26 includes: constructing a neural network, training the neural network, and correcting the image pixel luminance value according to the trained neural network to obtain the defective and peduncle mixed area.

[0036] Preferably, the specific steps of the object detection of the image described in step 3 are as follows:

[0037] (34) For the case where the contours of citrus and defects are irregular, use the active contour model to determine their contours, manually initialize the contours near the approximate areas of citrus and defects, and define the energy function of the active contour model;

[0038] (35) Solve the minimum value of the energy function using numerical calculation methods, evolve the contour towards the defect boundary by iteratively updating the points on the contour, and set termination conditions.

[0039] (36) After the contour converges, use Green's formula to calculate the area enclosed by the contour, obtain the citrus area and the defect area respectively, divide the two to get the proportion of the defective part and compare it with the set value to complete the target recognition operation.

[0040] Preferably, the energy function described in step 31 includes an internal energy term and an external energy term. The internal energy term consists of continuity constraints and smoothness constraints. The external energy term is realized by numerical methods such as the finite difference method based on the calculation method of the image gradient. When iteratively updating the contour points, the iteration stops according to termination conditions such as the set maximum number of iterations or the contour change threshold.

[0041] Beneficial effects: Compared with the prior art, the present invention has the following remarkable advantages: 1. By adopting a unique method of combining the hue H component map and the three-dimensional reconstruction depth map, the fruit stalk area can be accurately determined, effectively avoiding interference with defect detection caused by misjudgment of the fruit stalk, and greatly improving the detection accuracy; 2. By adopting an improved adaptive Gamma correction algorithm, the contrast between defects and normal areas in the image is significantly enhanced, enabling the originally blurred defect details to be clearly presented, reducing false detections caused by light and texture interference, and further improving the detection accuracy; 3. Using the active contour model combined with Green's formula to calculate the citrus and defect areas can more accurately quantify the defect degree, effectively reduce errors, and enhance the reliability of the detection results. Brief Description of the Drawings

[0042] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0043] Figure 2 It is a schematic diagram of the binocular camera calibration process of the present invention;

[0044] Figure 3 It is a schematic diagram of the image processing process of the present invention. Detailed Embodiments

[0045] The technical solutions of the present invention will be described in detail below with reference to the accompanying drawings.

[0046] (1) Calibration of the binocular camera.

[0047] 1. Install and fix the binocular horizontally moving camera. The cameras are sequentially set as camera 1 and camera 2. Place the calibration board in advance, select camera 1, adjust the focal length f, and use camera 1 to directly shoot the calibration board to obtain an image of the target array n×m.

[0048] 2. Establish an internal parameter model for the camera. There are five internal parameters, including the horizontal magnification factor \(f\) of the camera x , the vertical magnification factor \(f\) of the camera y , the coupling magnification factor \(s\) in the horizontal and vertical directions x , the abscissa \(c\) of the center of the image captured by the camera x , the abscissa \(c\) of the center of the image captured by the camera y , the tangential distortion coefficient and the radial distortion coefficient;

[0049] The internal parameter model is:

[0050]

[0051] 3. Establish an external parameter model for the camera. There are two external parameters, including the rotation matrix \(R\) and the translation matrix \(T\);

[0052] The external parameter model is:

[0053]

[0054] Combining equations (1) and (2), using the properties of the rotation matrix and the real symmetric matrix, find five internal parameters, two external parameters, and the projection matrix.

[0055] 4. According to the five internal parameters obtained, find the image point coordinates of the image point on the focal length normalized imaging plane. Based on the radial distortion model and the tangential distortion model, find the radial distortion and tangential distortion coefficients;

[0056] The distortion model is:

[0057]

[0058] Among them, \((u, v)\) represents the actual image coordinates, \(k1\) represents the second-order radial distortion coefficient, \(k2\) represents the fourth-order radial distortion coefficient, \(k3\) represents the sixth-order radial distortion coefficient, \(p1\) represents the second-order tangential distortion coefficient, \(p2\) represents the fourth-order tangential distortion coefficient, and \(r\) represents the distance from the image point \((u', v')\) to \((0, 0)\),

[0059] 5. Considering a point \(p\) in space, its coordinates in the world coordinate system are \((X w , Y w , Z w ). The coordinates in the camera coordinate systems of camera 1 and camera 2 are \((x1, y1)\) and \((x2, y2)\) respectively, and the corresponding depth information is \(z1\) and \(z2\). The projection matrices of the cameras are \(M1\) and \(M2\) respectively. The relationship between the two coordinate systems is:

[0060]

[0061] The relationship from camera 1 to camera 2 is as follows:

[0062]

[0063] Among them, M1 = R1T1, M2 = R2T2, and the projection matrix parameters between the cameras are obtained.

[0064] 6. Suppose there are n feature points in space on the calibration board. For each feature point p i , its coordinates in the world coordinate system are (X w , Y w , Z w ). The image point coordinates in camera 1 without horizontal displacement are p 1i , and the image point coordinates taken at different positions after horizontal displacement are p 2i (corresponding to shooting at different horizontal positions), p 1i and p 2i are all homogeneous coordinates;

[0065] Suppose the pose of camera 1 is (R1, T1). Assume that camera 1 only undergoes horizontal displacement relative to the initial position. Let the horizontal displacement vector be According to the imaging model of the camera and the known internal parameter matrix K, the following relationship is established:

[0066]

[0067] Based on the above relationship, using the linear line transformation method, the initial value d x of the horizontal displacement motion parameter is obtained.

[0068] 7. Establish an error equation, and the error function is as follows:

[0069]

[0070] This error function represents the sum of the squares of the errors between the actual image point coordinates of all feature points taken at different horizontal positions and the image point coordinates calculated according to the displacement vector . Take the derivative of the error function with respect to the horizontal displacement vector . Let

[0071]

[0072] Let A equation about d x is obtained. Solving this equation can obtain the value of d in the horizontal displacement vector x , which is the optimized value of the horizontal displacement motion parameter.

[0073] (2) Image preprocessing.

[0074] 1. When the binocular cameras are set to the same frequency for shooting, the captured images are synchronously transmitted by using the frame synchronization method. The frame synchronization selects the falling edge signal as the start of the transmission of a frame of image data, and this sequence is called the "frame header". The host computer determines the start of image transmission by verifying the "frame header" data. Before the next "frame header" arrives, if the amount of received data meets the data amount corresponding to a frame of image, the transmission is regarded as successful; otherwise, it is regarded as failed and re-transmitted. After the synchronous transmission of the images is completed, in order to accurately analyze the citrus images captured from different perspectives or at different times in the same coordinate system, then the image registration operation is performed.

[0075] 2. In the process of stitching multiple pictures taken at the same frequency, the method of calculating the transformation domain of the image to be registered using a calibration board is selected for image registration. The hom_mat2d operator in Halcon is used for calculation. By manually selecting, 4 points in the calibration board image corresponding to the optimal solution satisfy the homogeneous projective transformation matrix HomMat2D of the following equation. The equation is as follows:

[0076]

[0077] where, (P x , P y ) and (Q x , Q y ) represent the points of the reference image and the target image manually selected.

[0078] 3. Rotate the target image to make it parallel to the corresponding reference image plane. Rotate the image according to the calculated HomMat2D matrix, then crop a part of the image to cut off the unnecessary background area. Finally, calculate the initial translation amount in the image according to the calculated projective transformation matrix to register the target image with the reference image.

[0079] 4. After image registration, the registered pictures are fused. The direct average fusion method is adopted, and the gray values of the overlapping area between the target image and the reference image are all set to the average value of the overlapping area. The specific formula is expressed as:

[0080]

[0081] where, I1 represents the reference image, I2 represents the target image, and I represents the fused image.

[0082] 5. The target image undergoes a color space conversion from RGB to HSV, and the values of H, S, and V are calculated. The specific formulas are as follows:

[0083]

[0084] V = max(R, G, B);

[0085]

[0086] After the conversion is completed, extract the image of component H. To reduce the amount of calculation, perform dimensionality reduction on the RGB image, that is, grayscale the image.

[0087] 6. The citrus image is divided into two parts: the citrus itself and the background. To reduce the interference of the background on citrus defect detection and improve the detection accuracy, use the Canny operator to remove the background. The Canny operator detects edges by calculating the gradients of the pixel points in the image. First, use a Gaussian filter to smooth the image to reduce the influence of noise on edge detection. Let the original image be I(x, y) and the Gaussian filter be G(x, y);

[0088] The smoothed image S(x, y) is S(x, y) = I(x, y) * G(x, y);

[0089] Among them, σ represents the standard deviation of the Gaussian filter. Then calculate the partial derivatives of the smoothed image in the horizontal direction G x and the vertical direction G y to obtain the gradient magnitude and direction. M(x, y) represents the gradient magnitude, and θ(x, y) represents the gradient direction;

[0090]

[0091] 7. Edge detection and threshold processing. By setting appropriate thresholds, the citrus can be separated from the background. For the pixels on the edge, those with gradient values greater than the high threshold are considered strong edges, those less than the low threshold are considered non-edges, and those between the two are determined whether to be edges according to their connection with strong edges.

[0092] Let the high threshold be T h , and the low threshold be T l . If M(x, y) > T h , then this pixel is a strong edge pixel; if M(x, y) < T l , then this pixel is a non-edge pixel; if T l ≤M(x, y)≤T h , then whether this pixel is an edge pixel needs to be determined according to its connection with strong edge pixels.

[0093] In this way, the general outline of the citrus can be obtained. After threshold segmentation, some darker areas are mistaken for the background, resulting in holes. The citrus image is obtained through morphological closing operation.

[0094] 8. After removing the background, focus on the extraction part of the fruit stem area. Since the fruit stem is green, its hue value H is higher than the citrus base. By performing median filtering on the hue value H and taking the difference, the contrast between the fruit stem area and other parts of the citrus can be enhanced, so as to more effectively detect the approximate position of the fruit stem. The median filter reduces the impact of noise on the hue value and takes the difference to further highlight the characteristics of the fruit stem area. The formula is as follows:

[0095]

[0096] After median filtering, difference taking and binarization processing, the obtained fruit stem area may have problems such as discontinuity or holes. In order to obtain a more complete and accurate fruit stem area, a morphological closing operation is required to make the fruit stem area more continuous and complete. The blurred fruit stem area P is obtained, where the structure element D is the original image just obtained and S is a 5×5 structure element.

[0097] 9. Perform three-dimensional reconstruction on the blurred area. The formula for obtaining the depth of three-dimensional reconstruction is:

[0098]

[0099] Wherein, f represents the focal length, d represents the distance between the two cameras, and a three-dimensional reconstructed depth image of the blurred fruit stem area P is obtained.

[0100] 10. After three-dimensional reconstruction of the blurred fruit stem area P, its depth image is obtained. The depth information can provide an important basis for more accurate extraction of the fruit stem area. In order to accurately segment the fruit stem area from the depth image, it is necessary to determine a suitable threshold for binarization processing. The Otus method is used to find the optimal threshold. After filling the holes through morphological closing operations, the precise fruit stem area P1 is finally obtained. This area will be masked with the main citrus image in the subsequent process to eliminate the influence of the fruit stem on defect detection.

[0101] 11. Based on the citrus image with the background removed and the influence of the fruit stem eliminated, in order to further improve the quality of the citrus image and enhance the contrast between the defective area and the normal area, and to facilitate subsequent defect detection, it is necessary to correct the image brightness. Here, an adaptive Gamma correction algorithm based on the multi-scale texture fusion and asymmetric light diffusion characteristics of citrus is used to adaptively adjust the brightness according to the local features of the image, making the details in the image clearer, and providing better conditions for accurately detecting citrus defects.

[0102] The process of the constructed adaptive Gamma algorithm is as follows:

[0103] Read the citrus image: Denoise the grayscale image, and use median filtering to remove noise points to avoid interfering with texture feature extraction and luminance analysis. Perform multi-scale texture feature extraction: Use the Gaussian pyramid algorithm to perform multi-scale decomposition on the grayscale image, construct an image pyramid, calculate the gray-level co-occurrence matrix (GLCM) to extract texture feature parameters such as contrast, correlation, energy, and entropy. These multi-scale texture features can reflect the texture changes on the citrus surface, providing an important basis for accurately estimating the light source direction and constructing an asymmetric luminance diffusion model in the subsequent steps, thereby achieving more accurate luminance correction.

[0104] Let the original image be G0, and the image at the l-th layer be G l Through downsampling of G l-1 obtained:

[0105]

[0106] where i and j represent pixel coordinates, l represents the pyramid layer number, and {G0, G1 ··· G l} represents a sequence of images at different scales. The large scale reflects the overall texture trend, and the small scale highlights the local detailed texture;

[0107] Calculate the gray-level co-occurrence matrix: For each layer of image G l Calculate GLCM P l (θ, d), whose elements represent the occurrence probability that the gray values of two pixel points separated by a distance d in the direction θ are i and j respectively;

[0108] Extract texture feature parameters: Extract contrast Contrast l (θ, d), correlation Correlation l (θ, d), energy Energy l (θ, d), entropy Entropy l (θ, d) from each P l (θ, d), and combine them into a multi-scale texture feature vector T;

[0109] Contrast l (θ, d) = ∑ i,j |i - j| 2 P l (θ, d)(i, j) (17);

[0110]

[0111] Energy l (θ, d) = ∑ i,j P l(θ,d)(i,j) 2 (19);

[0112] Entropy l (θ,d) = -∑ i,j P l (θ,d)(i,j)logP l (θ,d)(i,j) (20);

[0113]

[0114] Then, construct the asymmetric luminance diffusion model. By analyzing the shadow area (average gray value I shadow ), highlight area (average gray value I highlight ), citrus contour point coordinates (x i , j i ), and their normal vectors (n i ) in the image, calculate the gray gradients of the shadow and highlight areas and According to the law of light reflection, by optimizing the objective function:

[0115]

[0116] I highlight represents the average gray value of the highlight area. According to the characteristics of the image, set the gray threshold T highligh to 200, and screen out the pixel points with gray values greater than T highlight as highlight candidate points. Check the pixel gray values in the 3×3 neighborhood around each candidate point. If it is the maximum value in the neighborhood, keep it as a highlight point; otherwise, exclude it. After determining the highlight area, set the gray values of the pixel points in the highlight area as h1, h2,..., h m , then the average gray value n i represents the normal vector, represents the gray gradient of the highlight area, I shadow represents the average gray value. Set the gray threshold T shadow to 50, and mark the pixel points with gray values less than T shadow as possible shadow pixels, and perform morphological operations of erosion and dilation on the marked shadow pixel area. Then, screen according to the characteristics such as the area and shape of the area. A minimum area threshold A min can be set. If the area of a certain area is less than A min , then exclude it from the shadow area. For the finally determined shadow area, set the gray values of the pixel points in the shadow area as g1, g2,..., g n , then Denote the gray - level gradient of the shadow region, (x i , y i ) denote the coordinates of the citrus contour points, n i denote the normal vector, and λ denote the regularization parameter;

[0117] λ denote the regularization parameter, and s denote the light source direction;

[0118] s is obtained by solving to obtain the description of the light - intensity diffusion using the elliptic diffusion function:

[0119]

[0120] where, (x, y) denote the pixel coordinates, (x0, y0) denote the image center or relative center coordinates, L0 denote the central light intensity, σ x and σ y denote the horizontal and vertical diffusion coefficients, and σ x ≠σ y .

[0121] According to the image brightness distribution, texture orientation, and light source direction, assume the actual brightness value of the pixel is, and optimize the objective function by the least - squares method:

[0122] E(L0, σ x , σ y ) = ∑ x,y [L(x, y)-I real (x, y)] 2 (23);

[0123] Solve and to obtain the parameters L0, σ x and σ y .

[0124] Then calculate and correct the Gamma factor:

[0125] Construct a neural network: The number of neurons in the input layer is the dimension of the multi - scale texture feature vector plus the number of parameters of the asymmetric light - intensity diffusion model. The output layer has 1 neuron (Gamma factor). The hidden layer has multiple layers and the number of neurons is determined as needed. Assume the weight matrix W, the bias vector b, the input vector x = [T T , L0, σ x , σ y , the output of the hidden layer h = f(W1x + b1), and the output of the output layer γ(x, y) = f(W2h + b2);

[0126] Train the neural network: Use a large number of citrus images under different textures and lighting conditions and their ideal Gamma - factor training samples to optimize W and b through the back - propagation algorithm;

[0127] Finally, perform Gamma correction: According to γ(x, y) calculated by the trained neural network, correct the pixel brightness value of the image to obtain I'(x, y), and obtain the mixed region of defects and stalks;

[0128] I'(x, y) = I(x, y) γ(x,y) (24).

[0129] Mask the obtained target object with the complete stalk region P1 obtained previously to get the citrus defect part excluding the influence of the stalk.

[0130] (3) Image target detection.

[0131] 1. Initialization and optimization of the active contour model: Since the contours of citrus and defects are irregular, the active contour model (Snake model) is used to determine their contours.

[0132] First, a contour needs to be manually initialized near the approximate regions of citrus and defects. Let the points on the initialized contour be v(s) = (x(s), y(s)) (where s is the parameter of the contour), and then define the energy function of the active contour model, including the internal energy term and the external energy term. The internal energy term consists of the continuity constraint and the smoothness constraint;

[0133] The continuity constraint ensures that the contour remains continuous during the evolution process, and its energy term:

[0134]

[0135] where α(s) represents the weight function, which can be set as a constant α0 first;

[0136] The smoothness constraint makes the contour as smooth as possible, and its energy term:

[0137]

[0138] Second derivative In the discrete case, it can be approximately calculated by finite differences. The external energy term guides the contour to move towards the defect boundary based on the image gradient:

[0139]

[0140] where represents the image gradient, I x and I y are the partial derivatives of the image in the and directions respectively, which can be calculated by the finite difference method;

[0141] such as

[0142] Total energy function: E(s) = Econt (v) + E smooth (v) + E image (v) (28);

[0143] Use numerical calculation methods, such as the finite difference method and the variational method, to solve the minimum value of the energy function. Update the points on the contour iteratively to make the contour evolve towards the defect boundary. Set termination conditions during the iteration process, such as reaching the maximum number of iterations or the change in the contour being less than a certain threshold.

[0144] 2. Calculate the contour area: When the contour converges, use Green's formula to calculate the area enclosed by the contour. For a closed contour represented by a series of points (x i , y i )(i = 0, 1, …, n - 1, (x n , y n ) = (x0, y0)), its area:

[0145]

[0146] Calculate the citrus area S1 and the defect area S2 respectively. Divide the obtained citrus area S1 by the defect area S2 to obtain the proportion of the defective part, and compare it with the set value to complete the target recognition operation.

Claims

1. A method for detecting citrus defects based on the influence of fruit pedicles, characterized in that: The following steps are involved: (1) Calibration of binocular horizontal mobile camera: Install a fixed binocular horizontal mobile camera, shoot the calibration plate to obtain the target image, build a model, and calculate the parameters; (2) Image preprocessing: The collected image is fused with the chromaticity H in the HSV color space and the three-dimensional reconstruction to obtain an accurate fruit stem image. Then, the adaptive Gamma algorithm based on the multi-scale texture fusion and asymmetric brightness diffusion characteristics of citrus is used for correction. The mask method is used to remove the fruit stem image to obtain the accurate defect area. (3) Target detection in images: Perform numerical integration of the area of ​​the irregular elliptical outline of the orange and perform complex Gaussian integral calculations, and compare them with the set values.

2. The citrus defect detection method based on the influence of the fruit pedicle according to claim 1 is characterized in that: The parameters in step 1 include internal parameters, external parameters and motion parameters of the horizontal motion camera; The internal parameters include the horizontal magnification factor of the camera, the vertical magnification factor of the camera, the horizontal coupling magnification factor, the vertical coupling magnification factor, the abscissa of the center of the image captured by the camera, the ordinate of the center of the image captured by the camera, the tangential distortion coefficient and the radial distortion coefficient; The external parameters include the rotation matrix and the translation matrix between the cameras.

3. The citrus defect detection method based on the influence of the fruit pedicle according to claim 1 is characterized in that: The specific steps of image preprocessing in step 2 are as follows: (21) The binocular camera synchronously transmits the captured images using a frame synchronization method; (22) After synchronous transmission, image registration is performed, and the registered images are fused using a direct average fusion method; (23) Convert the target image from RGB to HSV color space and extract the H component image and then grayscale it; (24) Use the Canny operator to remove the citrus background and obtain a blurred image of the fruit stem; (25) The blurred image is reconstructed into three dimensions to obtain a depth image, and the accurate fruit pedicle image is obtained by using the Otus method for binarization and morphological closing operations; (26) An adaptive gamma correction algorithm is used to correct the grayscale image to obtain the mixed area of ​​defects and fruit stems. Finally, the mask method is used to remove the fruit stem image to obtain the accurate defect area.

4. The citrus defect detection method based on the influence of the fruit pedicle according to claim 3 is characterized in that: The specific process of image registration described in step 22 is: rotate the target image to make it parallel to the corresponding reference image plane, rotate the image according to the calculated HomMat2D matrix, then crop part of the image area to cut off the unnecessary background area, and finally calculate the initial translation in the image according to the calculated projection transformation matrix to align the target image with the base image.

5. The citrus defect detection method based on the influence of the fruit pedicle according to claim 3 is characterized in that: When removing the citrus background as described in step 24, the Gaussian filter used by the Canny operator has a specific standard deviation. The gradient amplitude and direction are obtained by calculating the partial derivatives of the image pixels in the horizontal and vertical directions, and edge detection and threshold processing are performed based on the set high and low thresholds. Finally, the citrus image is obtained through a morphological closing operation.

6. The citrus defect detection method based on the influence of the fruit pedicle according to claim 3 is characterized in that: In the process of obtaining the accurate fruit stem image described in step 25, the hue H is first subjected to median filtering to obtain the difference, the approximate position of the fruit stem is detected, and then a morphological closing operation is performed to make the fruit stem area more continuous and complete. The blurred area is then three-dimensionally reconstructed, and the Otus method is used to determine the appropriate threshold for binarization processing and the holes are filled through morphological closing operations to obtain an accurate fruit stem image.

7. The citrus defect detection method based on the influence of the fruit pedicle according to claim 1, characterized in that: The adaptive Gamma algorithm formula described in step 2 is as follows: Contrast l (θ,d)=∑ i,j |ij| 2 P l (θ,d)(i,j); Energy l (θ,d)=∑ i,j P l (θ,d)(i,j) 2 ; Entropy l (θ,d)6-∑ i,j P.S l (θ,d)(i,j)log P l (θ,d)(i,j) E(L0,σ x ,s y )=∑ x,y [L(x,y)-I real (x,y)] 2 ; I'(x,y)=I(x,y) γ(x,y) ; Among them, i and j represent pixel coordinates, l represents the number of pyramid levels, {G0, G1…G l } represents image sequences of different scales, G0 represents the original image; P l (θ, d) represents the gray level co-occurrence matrix, which represents the probability of occurrence of two pixels with gray values ​​i and j in the direction θ and a distance d apart; Contrast l (θ,d) represents the l (θ, d) Extract contrast, Correlation l (θ,d) represents the correlation, μ i and μ j represents the grayscale mean, σ i and σ j Indicates grayscale standard deviation; Energy l (θ, d) represents energy, Entropy l (θ, d) represents entropy, T represents multi-scale texture feature vector; I highlight Indicates the average gray value of the highlight area, n i represents the normal vector, Represents the grayscale gradient of the highlight area, I shadow represents the average gray value, Represents the grayscale gradient of the shadow area, (x i ,y i ) represents the coordinates of the citrus contour points, n i represents the normal vector, λ represents the regularization parameter, and h i Indicates the gray value of the pixel in the highlight area, g i Represents the grayscale value of the pixel in the shadow area; s represents the direction of the light source, and s is solved by get; (x, y) represents the pixel coordinates, (x0, y0) represents the image center or relative center coordinates, L0 represents the central brightness, σ x and σ y represents the horizontal and vertical diffusion coefficients, and σ x ≠σ y , I(x,y) represents the brightness value of the image pixel.

8. The citrus defect detection method based on the influence of the fruit pedicle according to claim 3 is characterized in that: The adaptive gamma correction algorithm described in step 26 includes: constructing a neural network, training the neural network, correcting the image pixel brightness value according to the trained neural network, and obtaining the defect and fruit stem mixed area.

9. The citrus defect detection method based on the influence of the fruit pedicle according to claim 1, characterized in that: The specific steps of target detection of the image in step 3 are as follows: (31) In view of the irregular contours of citrus and defects, the active contour model is used to determine their contours, the contours are manually initialized near the approximate areas of citrus and defects, and the energy function of the active contour model is defined; (32) Using numerical calculation methods to solve the minimum value of the energy function, the points on the contour are iteratively updated to make the contour evolve toward the defect boundary, and the termination condition is set; (33) When the contour converges, the Green’s formula is used to calculate the area enclosed by the contour, and the citrus area and defect area are obtained respectively. The two are divided to obtain the proportion of the defective part and compared with the set value to complete the target recognition operation.

10. The citrus defect detection method based on the influence of the fruit pedicle according to claim 9, characterized in that: The energy function in step 31 includes internal energy terms and external energy terms. The internal energy terms are composed of continuity constraints and smoothness constraints. The external energy terms are implemented based on the calculation method of image gradients through numerical methods such as the finite difference method. When the contour points are iteratively updated, the iteration is stopped according to the termination conditions such as the set maximum number of iterations or the contour change threshold.

Images