Fruit surface damage detection method based on structured light

Abstract

The present invention particularly relates to a method for detecting fruit surface damage by integrating structured light, comprising the following steps: using a projector to project a sinusoidal fringe pattern onto a reference plane and a surface of a fruit to be tested, and using a camera to collect a modulated deformation fringe pattern; using a structured light four-step phase shift method to calculate the three-dimensional structure of the fruit surface; extracting binary features of the three-dimensional structure of the fruit surface; using a feature fusion algorithm based on Bayesian theory to fuse the features of a uniform illumination image with binary features; using a support vector machine method to classify the fused features to obtain the fruit surface damage. The entire detection process is automatic, which reduces labor costs and increases detection efficiency; non-destructive detection technology is used to avoid damage caused by detection; the entire device only needs a projector, a camera and a computer to achieve low-cost industrial requirements; the method of integrating structured light and uniform illumination is used to improve the accuracy of damage detection.

Description

Technical Field

[0001] The present invention relates to the technical field of fruit quality detection, and in particular to a method for detecting fruit surface damage by integrating structured light. Background Art

[0002] Apple is one of the most popular fruits in the world. With economic and material prosperity, people begin to pursue a healthy life, and the demand for fresh and high-quality fruits continues to increase. Damage first affects the appearance of apples, and the nutrients in the damaged area are exposed, which will become an incubator for foodborne pathogens, and then accumulate harmful substances, causing apple lesions. Therefore, damage detection can ensure the external quality of apples and avoid further harm caused by lesions.

[0003] At present, apple damage detection methods include manual inspection, mechanical inspection, multispectral imaging, hyperspectral imaging and RGB imaging, etc. Manual inspection has the disadvantages of large labor, low accuracy and efficiency, and excessive reliance on human subjective factors; mechanical inspection will cause increased fruit damage; multispectral imaging and hyperspectral imaging face the disadvantages of large data volume, slow analysis speed and high cost; RGB imaging has difficulty in identifying specific types of damage, such as friction and tissue bruises.

[0004] In addition to apples, other fruits such as pears and peaches also have these problems. Therefore, a more advanced method is urgently needed to detect fruits. Summary of the invention

[0005] The object of the present invention is to provide a method for detecting fruit surface damage by integrating structured light with low cost, high efficiency and high accuracy.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is: a method for detecting fruit surface damage by integrating structured light, comprising the following steps: using a projector to project a sinusoidal fringe pattern onto a reference plane and a surface of a fruit to be tested, and using a camera to collect the modulated deformation fringe pattern; using a structured light four-step phase shift method to calculate the three-dimensional structure of the fruit surface; extracting the binary features of the three-dimensional structure of the fruit surface; using a feature fusion algorithm based on Bayesian theory to fuse the features of a uniformly illuminated image with the binary features; and using a support vector machine method to classify the fused features to obtain the damage status of the fruit surface.

[0007] Compared with the prior art, the present invention has the following technical effects: the entire detection process is automatic, which reduces labor costs and increases detection efficiency; non-destructive testing technology is used to avoid damage caused by detection; the entire device only requires a projector, a camera and a computer to achieve low-cost industrial requirements; a method of fusion of structured light and uniform illumination is used to detect damage such as friction and tissue bruises that are difficult to identify under uniform illumination, thereby improving the accuracy of damage detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic diagram of the process of the present invention;

[0009] Figure 2 It is a schematic diagram of the structure of the present invention;

[0010] Figure 3 It is a schematic diagram of triangulation principle calculation;

[0011] Figure 4 It is a schematic diagram of feature point selection;

[0012] Figure 5 It is a schematic diagram of the numbering of surrounding points when calculating binary features;

[0013] Figure 6 This is the structural diagram of the lightweight network Inception ResNet v2. DETAILED DESCRIPTION

[0014] Combine the following Figures 1 to 6 , the present invention is further described in detail.

[0015] See also Figure 1 and Figure 2 The present invention discloses a method for detecting fruit surface damage by integrating structured light, comprising the following steps: using a projector to project a sinusoidal fringe pattern onto a reference plane and the surface of the fruit to be tested, and using a camera to collect the modulated deformation fringe pattern; using a structured light four-step phase shift method to calculate the three-dimensional structure of the fruit surface; extracting the binary features of the three-dimensional structure of the fruit surface; using a feature fusion algorithm based on Bayesian theory to fuse the features of the uniform illumination image with the binary features; using a support vector machine method to classify the fused features to obtain the fruit surface damage. The entire detection process is automatic, which reduces labor costs and increases detection efficiency; non-destructive testing technology is used to avoid damage caused by detection; the entire device only needs a projector, a camera and a computer to achieve low-cost industrial requirements; the method of integrating structured light and uniform illumination can detect damage such as friction and tissue bruises that are difficult to identify under uniform illumination, thereby improving the accuracy of damage detection.

[0016] Furthermore, the optical axes of the camera and the projector are in the same plane, and this plane is perpendicular to the plane where the object to be measured is located. The projector is a DLP4500 projector, and the projector projects a light intensity of I to the reference plane and the surface of the apple to be measured. b (x, y) is uniformly illuminated, and the camera collects pictures under uniform illumination; the projector projects a vibration with an amplitude intensity of I onto the reference plane and the surface of the apple being measured. a(x, y), four phase-shifted sinusoidal fringe images with a phase shift of π / 2, and the camera synchronously collects the modulated deformation fringe image. There are also four deformation fringe images. The light intensity of the i-th deformation fringe image at the pixel point (x, y) is I i (x, y), where i = 1, 2, 3, 4. The method of using structured light four-step phase shift method to calculate the three-dimensional structure of the fruit surface comprises the following steps: according to the light intensity I at the pixel point (x, y) in the four deformation fringe images i (x, y), calculate the phase difference caused by the object modulating the fringe at point (x, y) according to the following formula:

[0017]

[0018] The phase difference was calculated by four-way shearing least square method. Perform phase unwrapping to obtain the unwrapped phase According to the phase expansion The depth of the fruit surface is calculated to obtain the three-dimensional structural information of the fruit surface. The three-dimensional structural information of the fruit surface can be obtained quickly and conveniently using the above steps.

[0019] See also Figure 3 , further, the present invention calculates the depth d according to the triangulation principle, Figure 3 Where P is a projector, C is a camera, the projector and the camera are located at the same level and the distance from the reference plane is H, the distance between the projector and the camera is W, the period of the sinusoidal fringe pattern is T, S is a point on the surface of the fruit, d is the depth of point S, and AB / W=d / (Hd) is obtained by the similarity of the two triangles. AB can be obtained by the phase difference express, From the above two formulas, we can know that the depth d of the point on the fruit surface corresponding to the pixel point (x, y) is calculated by the following formula:

[0020]

[0021] Furthermore, the binary feature extraction of the three-dimensional structure of the fruit surface comprises the following steps: according to the three-dimensional structure of the fruit surface obtained by calculation, rays are made at equal intervals with the coordinate origin as the endpoints, and the points where the rays intersect with the fruit surface are taken as feature points, and any three adjacent feature points constitute the three vertices of an equilateral triangle, such as Figure 4 As shown. For any point V C , and V C The eigenvalues ​​of the six adjacent points are related to V C The point with the smallest difference is set as V1, and V C With V1 as the starting point and V2, V3, V4, V5, and V6 as the center, the remaining points are set in a counterclockwise direction. Figure 5 As shown, calculate point V according to the following formulaC The binary features of:

[0022]

[0023] Where, d i Point V i The depth, L c Point V C According to the above steps, the binary features of all points are calculated and normalized. Through the above steps, the binary features of each point can be quickly calculated.

[0024] Furthermore, the feature fusion algorithm based on Bayesian theory is used to fuse the features of the uniform illumination image with the binary features, which includes the following steps: the pattern space Ω contains c patterns, which are denoted as Ω = {ω1, ω2, ..., ω c}, where c is the number of damage types; sample x is composed of four-dimensional real-valued features, denoted as x = [x1, x2, x3, x4], where x1, x2, x3 are uniform illumination image features, and x4 is a binary feature of a three-dimensional image; x is regarded as the output result of the classifier, there are M classifiers, and each classifier will output a result yj. At this time, the feature is: y = [y1, y2, ..., y M ], according to Bayesian theory, we can get: if Then y→ω j , where y is the fused feature. Through the above steps, the features of the uniform illumination image and the binary features can be easily fused.

[0025] See also Figure 6 Furthermore, the method of using support vector machine to classify the fused features includes the following steps: constructing a lightweight network Inception ResNet v2, the network structure of which is as follows Figure 6 As shown; make samples and train the network structure; substitute the fused features into the trained network to get the classification results. Here, the inception-resnet network is established based on tensorflow. The filter connection in the inception architecture is replaced by residual connection, and the calculation between sub-networks is balanced by adjusting the layer size. Use Inception blocks that are lighter than the initial inception network, and each Inception block is followed by a filter expansion layer (1×1 convolution, no activation). Batch normalization is used in the top traditional layer, and batch normalization is not used in the top residual. Using this network can achieve classification more accurately.

[0026] The present invention also discloses a computer-readable storage medium and an electronic device. A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for detecting fruit surface damage by integrating structured light as described above is implemented. An electronic device includes a memory, a processor, and a computer program stored in the memory, and when the processor executes the computer program, the method for detecting fruit surface damage by integrating structured light as described above is implemented.

Claims

1. A method for detecting fruit surface damage by integrating structured light, characterized in that: The steps include: A projector is used to project a sinusoidal fringe pattern onto a reference plane and a surface of the fruit to be tested, and a camera is used to collect the modulated deformation fringe pattern; The three-dimensional structure of the fruit surface was calculated using the structured light four-step phase shift method; Extract binary features of the three-dimensional structure of the fruit surface; The feature fusion algorithm based on Bayesian theory is used to fuse the features of uniform illumination images with binary features. The support vector machine method is used to classify the fused features to obtain the surface damage of the fruit; The method of fusing the features of the uniform illumination image with the binary features using the feature fusion algorithm based on Bayesian theory includes the following steps: The pattern space Ω contains c modes, denoted as Ω = {ω1,ω2,…,ω c }, where c is the number of damage types; Sample x consists of four-dimensional real-valued features, denoted as x = [x1, x2, x3, x4], where x1, x2, x3 are uniform illumination image features, and x4 is a binary feature of a three-dimensional image; Consider x as the output of the classifier. There are M classifiers, and each classifier will output a result yj. The feature at this time is: y=[y1,y2,…,y M ], according to Bayesian theory: if Then y→ω j , where y is the fused feature.

2. The method for detecting fruit surface damage by integrating structured light according to claim 1, characterized in that: There are four sinusoidal fringe images, the four sinusoidal fringe images have the same amplitude intensity and a phase shift of π / 2. There are also four deformation fringe images, and the light intensity of the i-th deformation fringe image at the pixel point (x, y) is I i (x,y), where i=1,2,3,4.

3. The method for detecting fruit surface damage by integrating structured light according to claim 2, characterized in that: The method of calculating the three-dimensional structure of the fruit surface using the structured light four-step phase shift method comprises the following steps: According to the illumination intensity I at the pixel point (x, y) in the four deformed fringe images i (x, y), calculate the phase difference caused by the object modulating the fringe at point (x, y) according to the following formula: The phase difference was calculated by four-way shearing least square method. Perform phase unwrapping to obtain the unwrapped phase According to the phase expansion The depth of the fruit surface is calculated to obtain the three-dimensional structural information of the fruit surface.

4. The method for detecting fruit surface damage by integrating structured light according to claim 3, characterized in that: The height of the projector and the camera are located on the same horizontal plane and the distance from the reference plane is H, the distance between the projector and the camera is W, the period of the sinusoidal fringe pattern is T, and the depth d of the point on the fruit surface corresponding to the pixel point (x, y) is calculated by the following formula:

5. The method for detecting fruit surface damage by integrating structured light according to claim 1, characterized in that: The binary feature extraction of the three-dimensional structure of the fruit surface comprises the following steps: According to the calculated three-dimensional structure of the fruit surface, a ray is drawn with equal interval angles with the coordinate origin as the endpoint. The point where the ray intersects with the fruit surface is taken as the feature point. Any three adjacent feature points constitute the three vertices of an equilateral triangle. For any point V C , and V C The eigenvalues ​​of the six adjacent points are related to V C The point with the smallest difference is set as V1, and V C With V1 as the starting point and V2, V3, V4, V5, and V6 as the center, go counterclockwise. Calculate point V as follows C The binary features of: Where, d i Point V i The depth, L c Point V C Binary features of According to the above steps, the binary features of all points are calculated and normalized.

6. The method for detecting fruit surface damage by integrating structured light according to claim 1, characterized in that: The method of using support vector machine to classify the fused features includes the following steps: Build a lightweight network Inception ResNet v2; Create samples and train the network structure; Substitute the fused features into the trained network to obtain the classification results.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for detecting surface damage of fruit by integrating structured light as described in any one of claims 1 to 6 is implemented.

8. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory. When the processor executes the computer program, the method for detecting surface damage of fruit by integrating structured light as described in any one of claims 1 to 6 is implemented.

Images