Pistachio head and tail recognition device and recognition method based on multi-channel information fusion

Abstract

The invention discloses a pistachio head and tail recognition device based on multi-channel information fusion and a recognition method thereof. The device includes a stage, a left imaging device, a right imaging device and a vertical imaging device. The imaging device includes a computer, an industrial camera, a lens, a spherical integral light source, a fixed adjustment bracket, etc. The recognition method of the horizontal left and right imaging devices includes: collecting color images, converting to grayscale images, image median filtering, binarization, Sobel edge detection, image morphological transformation, extracting pistachio target areas, constructing a deep learning network structure, training networks, and identifying the pistachio tail and tail. The recognition method of the vertical imaging device includes: image acquisition, converting to grayscale images, image segmentation, BLOB analysis, feature extraction, target screening, target left area calculation, target right area calculation, head and tail recognition based on area difference, etc. The invention can realize multi-channel image information fusion and comprehensive evaluation, and improve the accuracy of pistachio head and tail recognition.

Description

Technical Field

[0001] The invention belongs to the technical field of automatic optical detection, and in particular relates to a device for identifying the head and tail of pistachios by multi-channel fusion of horizontal imaging and vertical imaging, and also relates to a recognition algorithm of the device. Background Art

[0002] Pistachios, also known as pistachios, are rich in vitamins, minerals and antioxidants, and are low in fat, low in calories and high in fiber. They have medical and health benefits for cardiovascular and cerebrovascular diseases, senile retinopathy, and anti-aging. They are a very popular health and leisure food in the world's nut market and have become the world's fifth largest nut crop. Shelling is an indispensable part of their production.

[0003] At present, the most common and traditional method for identifying the head and tail of pistachios is manual visual inspection, which uses human eyes to check the head and tail of pistachios. However, the visual method is time-consuming and labor-intensive, with low efficiency and a large influence of subjective factors. Moreover, as the inspection time increases, it is easy to cause visual fatigue of the inspector, resulting in a higher identification error rate. In the process of shelling pistachios, manual identification alone is not only time-consuming and labor-intensive, but also prone to identification errors. Therefore, the visual method can no longer meet the requirements of high efficiency and high quality of the production line.

[0004] The Chinese patent with the publication number [CN 107303566] proposes an open-and-closed mouth sorting device based on the impact sound characteristics of the physical structure of pistachios. The pistachio shells are separated by an impact cavity, an acoustic sensor holding device, an impacted device, a support frame, a control and analysis device, and a sorting execution device. However, this method does not have accurate positioning and correction, and it is easy to cause errors due to noise.

[0005] The Chinese patent with the publication number [CN 109410209] proposes a method for detecting foreign matter in nuts based on deep learning classification. A deep learning classification network is designed and the detection model training is completed, and the trained network connection weight matrix is ​​saved; the image of the object area to be detected is sent to the trained deep learning classification network for image classification detection to complete the detection of foreign matter in nuts. However, this method requires a large number of training samples, and the amount of training and detection and recognition calculations is large; the accuracy of identifying the head and tail of pistachios using this method is not high, and overfitting is prone to occur.

[0006] In summary, there is an urgent need in the art for a device and method for quickly and effectively identifying the head and tail of pistachios, so as to improve work efficiency, reduce labor costs, liberate labor, and realize intelligent and automated pistachio processing and production. Summary of the invention

[0007] The purpose of the present invention is to provide a pistachio head and tail recognition device and a recognition method thereof with multi-channel fusion of horizontal imaging and vertical imaging, which solves the problem of time-consuming and low efficiency of manual detection.

[0008] The present invention also aims to provide a method for detecting the head and tail of pistachios.

[0009] A technical solution adopted by the present invention is: a device and method for identifying the head and tail of pistachios by photographing the middle part of pistachios from a vertical direction, comprising a stage on an assembly line, a computer and an industrial camera connected with signals, a spherical integral light source, and a coaxial light source are respectively arranged on the vertical direction of the stage, and pistachios placed horizontally are arranged on the stage support on the assembly line: the computer collects color images through the industrial camera, the coaxial light source and the spherical integral light source. The recognition algorithm flow of the imaging units on the left and right sides of the horizontal side includes: collecting color images, converting to grayscale images, image median filtering, binarization, Sobel edge detection, image morphological transformation, extracting pistachio target areas, constructing a deep learning network structure, training networks, identifying the tail and tail of pistachios, etc.; the recognition algorithm flow of the vertical imaging unit includes: image acquisition, converting to grayscale images, image segmentation, BLOB analysis, feature extraction, target screening, target left area calculation, target right area calculation, head and tail recognition based on area difference, etc. The three imaging devices and their recognition algorithms can realize multi-channel image information fusion, comprehensive evaluation, and output the optimal recognition decision of the head and tail of pistachios.

[0010] Furthermore, the industrial camera of the vertical imaging unit is facing the middle part of the pistachio to shoot.

[0011] Furthermore, the industrial camera of the horizontal imaging unit is facing the two ends of the pistachio to shoot.

[0012] Furthermore, the detection system also includes a light source controller connected to the industrial camera signal, the light source controller is connected to a spherical integrating light source, the spherical integrating light source is located above the pistachio, and uses the curved surface of the pistachio shell for uniform illumination.

[0013] Furthermore, the spherical integrating light source is arranged on the assembly line stage and is perpendicular to the horizontal plane of the stage.

[0014] Another technical solution adopted by the present invention is: a method for identifying the head and tail of pistachios by multi-channel fusion of horizontal imaging and vertical imaging. The head and tail of pistachios are identified in the horizontal direction, including the following steps:

[0015] Step 1: Obtain a color image of the pistachio head through an industrial camera 102 and a spherical integrating light source 103;

[0016] Step 2, converting the color image into a grayscale image by binarization and preprocessing, and using the Sobel operator to calculate and obtain the x and y direction gradients of the pistachio;

[0017] Step 3, use low-pass filter median filter to smooth the image;

[0018] Step 4: Use image morphological processing to fill the black background of the image and remove the white noise outside the pistachio.

[0019] Step 5: Use the minimum rectangle to cut out the effective area of ​​the entire pistachio;

[0020] Step 6: Construct a deep learning network structure, which can be one of (MobileNet, VGG16, LeNet5, RseNet, etc.);

[0021] Step 7: Use the deep learning network to identify the probability value η that the pistachio head is on the left side. L , and the probability that the head is on the right side η R Among them, if η L is greater than 50%, the head is on the left; if η R If it is greater than 50%, the head is on the right side.

[0022] Furthermore, the deep learning framework in step 6 may adopt a variety of image classification networks and transfer learning methods.

[0023] A method for identifying the head and tail of pistachios by multi-channel fusion of horizontal imaging and vertical imaging. The method for identifying the head and tail of pistachios in the vertical direction includes the following steps:

[0024] Step 1, obtaining a color image of the pistachio head through the industrial camera 105 via the coaxial light source 106 and the spherical integral light source;

[0025] Step 2: convert the color image into a grayscale image and perform preprocessing to obtain it;

[0026] Step 3: Segment the image;

[0027] Step 4: Extract the pistachio target through Blob analysis;

[0028] Step 5, using the Sobel operator to calculate the x and y direction gradients of the pistachio to obtain the edge image of the pistachio;

[0029] Step 6: Extract the pistachio features by analyzing the edge contour and gradient of the pistachio.

[0030] Step 7: Figure 5As shown, get the straight line L with the longest distance between the two ends, make a perpendicular bisector A of the straight line L, and divide the pistachio image into left and right halves;

[0031] Step 8: Based on the perpendicular bisector A, calculate the area A1 of the left side contour of the target;

[0032] Step 9: Based on the perpendicular bisector A, calculate the area A2 of the right side contour of the target;

[0033] Step 10: Identify the head and tail by area difference, and output the probability value η that the head is on the left M , see formula (1) for details;

[0034]

[0035] Taking the pistachio head as the probability criterion, since η M There will be a negative probability if η M If η is greater than 0, it is the head. M If it is less than 0, it is judged as the tail. M The distribution interval is η M ∈(-1, 1), then η M Normalization is required, so the distribution of probability data from three different perspectives is (0, 1), see format (2) for details.

[0036]

[0037] The pistachio head and tail recognition method based on multi-channel fusion of horizontal imaging and vertical imaging can realize the multi-channel information fusion of the left imaging unit, the middle top imaging unit and the right imaging unit, and obtain the probability value η of the optimal decision End , see formula (3)

[0038]

[0039] If the probability value of the optimal decision is η End If η is greater than 50%, the head of the pistachio is on the left; otherwise, η End If it is less than 50%, the head of the pistachio is on the right side.

[0040] The beneficial effects of the present invention are as follows: the present invention is a device and method for identifying the head and tail of pistachios based on multi-channel information fusion of horizontal imaging and vertical imaging, which introduces machine vision and deep learning into the detection of pistachios, solves the problem of time-consuming and low efficiency of manual detection, and adopts spherical integral light source and multi-view imaging to achieve rapid imaging of pistachios, improve effective information content, accurately extract targets, and adopt the decision-making mode of optimal probability to complete high-performance recognition of the head and tail of pistachios, with an accuracy rate of up to 98%, effectively replacing manual labor and greatly improving the production efficiency of pistachio processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a mechanical assembly of a pistachio head and tail identification device based on multi-channel fusion of horizontal imaging and vertical imaging provided by the present invention;

[0042] Figure 2 A flow chart of the pistachio head and tail recognition algorithm based on horizontal imaging and vertical imaging multi-channel information provided by the present invention;

[0043] Figure 3 A schematic diagram of a pistachio head provided by the present invention;

[0044] Figure 4 A schematic diagram of the pistachio tail provided by the present invention;

[0045] Figure 5 A schematic diagram of a top plan view of pistachios provided by the present invention.

[0046] Figure 1 In, 101. Computer, 102. X industrial camera, 103. Spherical integrating light source (3), 104. Pistachio, 105. Z industrial camera, 106. Coaxial light source, 107. -X industrial camera, 108. Stage on the assembly line.

[0047] Figure 2 In the process, S100 starts to detect pistachios. Then it is divided into three branches for parallel processing, as follows:

[0048] The first branch, for left-side imaging, S401 collects a pistachio image from the left side to determine whether it is a head, S402 obtains a target area of ​​the pistachio on the left side, S403 pre-processes the left-side image using the Sobel operator, binarization, image morphology, etc., S404 constructs a left-side deep learning network structure (such as: MobileNet, VGG16, LeNet5, RseNet, any one of them is fine), S405 trains the left-side network structure, S406 recognizes the left-side image of the pistachio, and S407 outputs the probability value η of recognizing the head and tail of the pistachio L .

[0049] The second branch, for right-side imaging, S201 collects a pistachio image from the right side to determine whether it is the tail, S202 obtains the target area of ​​the pistachio on the right side, S203 pre-processes the right-side image using the Sobel operator, binarization, image morphology, etc., S204 constructs a deep learning network structure (such as: MobileNet, VGG16, LeNet5, RseNet, any one of them is fine), S205 trains the right-side network structure, S206 recognizes the right-side image of the pistachio, and S207 outputs the probability value η of recognizing the head and tail of the pistachio R .

[0050] The third branch, for the middle (top) imaging, S301 collects the middle part of the pistachio image, S302 obtains the middle part target area, S303 uses the Sobel operator, binarization, etc. to pre-process the middle part image and divide the image into two left and right areas, S304 extracts the area A1 of the left area, S305 extracts the right area A2, S306 compares the difference between the left and right areas, and S307 outputs the probability value η of identifying the head and tail of the pistachio M .

[0051] Taking the pistachio head as the probability criterion, since η M There will be a negative probability if η M If η is greater than 0, it is the head. M If it is less than 0, it is judged as the tail. M The distribution interval is η M ∈(-1, 1), then η M Normalization is required, so the distribution of probability data from three different perspectives is (0, 1), see format (2) for details.

[0052]

[0053] The pistachio head and tail recognition method based on multi-channel fusion of horizontal imaging and vertical imaging can realize the multi-channel information fusion of the left imaging unit, the middle top imaging unit and the right imaging unit, and obtain the probability value η of the optimal decision End , see formula (3)

[0054]

[0055] If the probability value of the optimal decision is η End If η is greater than 50%, the head of the pistachio is on the left; otherwise, η End If it is less than 50%, the head of the pistachio is on the right side.

[0056] Figure 3 110. The pistachio head has irregular round features, and the most notable feature is the presence of a small round area in the middle;

[0057] Figure 4 120. The tail of a pistachio. The most notable feature is the presence of a diamond-shaped skeleton extending from the left to the right; DETAILED DESCRIPTION

[0058] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] The present invention provides a schematic diagram of a mechanical assembly of a pistachio head and tail identification device with multi-channel fusion of horizontal imaging and vertical imaging, as shown in FIG. Figure 1 As shown. It includes: a stage on the assembly line, on which pistachios to be tested are placed, and three imaging processing and recognition units of different viewing angles. Among them, the horizontal imaging processing units on the left and right sides of the pistachios, the signals of the industrial cameras 107 and 102 of the processing units, are transmitted to the computer 101 via TCP / IP Ethernet. Similarly, the vertical imaging processing unit at the top of the pistachios, the signals of the industrial camera 105 of the processing unit are also transmitted to the computer 101 via TCP / IP Ethernet. These three units respectively use a spherical integrating light source 103, and in particular, the top imaging unit also uses a coaxial light source 106, which is located above the spherical integrating light source 103 and below the industrial camera 105. All the industrial cameras and light sources are fixed on a designed bracket 109.

[0060] The key parameters of the machine vision system imaging are: 1) the distance between the industrial camera 105 and the surface of the pistachio 104 is 40cm-50cm; 2) the distance between the spherical integrating light source and the pistachio 104 is 15cm-25cm; 3) the distance between the coaxial light source 106 and the pistachio 104 is 30cm-40cm; 4) there are many grooves with a width of 1.5cm-2.cm on the stage and it is coordinated with the assembly line to operate quickly.

[0061] See also Figure 2 Another technical solution adopted by the present invention is: a method for identifying the head and tail of pistachios by multi-channel fusion of horizontal imaging and vertical imaging. Among them, the horizontal direction identification of the head and tail of pistachios includes the following steps:

[0062] Step 1: Obtain a color image of the pistachio head through an industrial camera 102 and a spherical integrating light source 103;

[0063] Step 2, converting the color image into a grayscale image by binarization and preprocessing, and using the Sobel operator to calculate and obtain the x and y direction gradients of the pistachio;

[0064] Step 3, use low-pass filter median filter to smooth the image;

[0065] Step 4: Use image morphological processing to fill the black background of the image and remove the white noise outside the pistachio.

[0066] Step 5: Use the minimum rectangle to cut out the effective area of ​​the entire pistachio;

[0067] Step 6: Construct a deep learning network structure, which can be one of (MobileNet, VGG16, LeNet5, RseNet, etc.);

[0068] Step 7: Use the deep learning network to identify the probability value η that the pistachio head is on the left side. L , and the probability that the head is on the right side η R ;

[0069] Furthermore, the deep learning framework in step 6 may adopt a variety of image classification networks and transfer learning methods.

[0070] In addition, the vertical identification of the head and tail of the pistachio includes the following steps:

[0071] Step 1, obtaining a color image of the pistachio head through the industrial camera 105 via the coaxial light source 106 and the spherical integral light source;

[0072] Step 2: convert the color image into a grayscale image and perform preprocessing to obtain it;

[0073] Step 3: Segment the image;

[0074] Step 4: Extract the pistachio target through Blob analysis;

[0075] Step 5, using the Sobel operator to calculate the x and y direction gradients of the pistachio to obtain the edge image of the pistachio;

[0076] Step 6: Extract the pistachio features by analyzing the edge contour and gradient of the pistachio.

[0077] Step 7: Figure 5 As shown, get the straight line L with the longest distance between the two ends, make a perpendicular bisector A of the straight line L, and divide the pistachio image into left and right halves;

[0078] Step 8: Based on the perpendicular bisector A, calculate the area A1 of the left side contour of the target;

[0079] Step 9: Based on the perpendicular bisector A, calculate the area A2 of the right side contour of the target;

[0080] Step 10: Identify the head and tail by area difference, and output the probability value η that the head is on the left M , see formula (1) for details;

[0081]

[0082] Taking the pistachio head as the probability criterion, since η M There will be a negative probability if η M If η is greater than 0, it is the head. M If it is less than 0, it is judged as the tail. M The distribution interval is η M∈(-1, 1), then η M Normalization is required, so the distribution of probability data from three different perspectives is (0, 1), see format (2) for details.

[0083]

[0084] The pistachio head and tail recognition method based on multi-channel fusion of horizontal imaging and vertical imaging can realize the multi-channel information fusion of the left imaging unit, the middle top imaging unit and the right imaging unit, and obtain the probability value η of the optimal decision End , see formula (3)

[0085]

[0086] If the probability value of the optimal decision is η End If η is greater than 50%, the head of the pistachio is on the left; otherwise, η End If it is less than 50%, the head of the pistachio is on the right side.

[0087] The above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A pistachio head and tail recognition device based on multi-channel information fusion, characterized in that: The device comprises: a stage arranged on the production line, on which pistachios to be inspected are placed, and imaging processing and identification units of three viewing angles; wherein, the horizontal imaging processing units on the left and right sides of the pistachios, the signals of the industrial cameras 107 and 102 of the processing units are transmitted to the computer 101 via TCP / IP Ethernet; similarly, the vertical imaging processing unit at the top of the pistachios, the signals of the industrial camera 105 of the processing unit are also transmitted to the computer 101 via TCP / IP Ethernet; the three units respectively adopt a spherical integrating light source 103, in particular, the vertical imaging processing unit at the top also adopts a coaxial light source 106, which is located above the spherical integrating light source 103 and below the industrial camera 105; all industrial cameras and light sources are fixed on a designed bracket 109; Among them, the horizontal imaging processing unit and the vertical imaging processing unit also include a pistachio head and tail recognition method of the horizontal imaging processing unit and the vertical imaging processing unit, specifically: the horizontal imaging processing unit recognizes the left side of the pistachio and outputs a probability value, the horizontal analysis imaging processing unit recognizes the right side of the pistachio and outputs a probability value, and the vertical imaging processing unit recognizes the middle part of the pistachio and outputs a probability value; the probability values ​​of the above three units are normalized, and the optimal decision probability of the pistachio head and tail is output.

2. The pistachio head and tail identification device based on multi-channel information fusion as claimed in claim 1, characterized in that: The distance between the industrial camera 105 and the surface of the pistachio 104 is 40cm-50cm, the distance between the spherical integrating light source and the pistachio 104 is 15cm-25cm, the distance between the coaxial light source 106 and the pistachio 104 is 30cm-40cm, and a plurality of grooves with a width of 1.5cm-2.0cm are arranged on the stage and operate rapidly in coordination with the assembly line.

3. The pistachio head and tail identification device based on multi-channel information fusion as claimed in claim 1, characterized in that: The recognition device also includes a computer processing system connected to the industrial camera signal, and the computer processing system includes an image data analysis unit and an image data acquisition unit.

4. The pistachio head and tail identification device based on multi-channel information fusion as claimed in claim 1, characterized in that: The process of identifying the head and tail of pistachios based on the horizontal imaging processing unit is also included, specifically: Collect color images, convert to grayscale images, perform image median filtering, binarization, Sobel edge detection, image morphological transformation, extract pistachio target area, construct deep learning network structure, train the network, identify the head and tail of pistachios, and output the probability value of identifying the head and tail.

5. The pistachio head and tail identification device based on multi-channel information fusion as claimed in claim 1, characterized in that: The method also includes a process for identifying the middle part of a pistachio based on the vertical imaging processing unit, specifically: Image acquisition, conversion to grayscale, image segmentation, BLOB analysis, feature extraction, Sobel operator edge detection, edge contour and gradient analysis, target left area calculation A1, target right area calculation A2, head and tail recognition based on area difference, output head and tail probability value.

6. A method for identifying the head and tail of pistachios based on multi-channel information fusion, characterized in that: The device for identifying the head and tail of pistachios based on multi-channel information fusion as claimed in any one of claims 1 to 5 specifically comprises the following steps: Step S11, obtaining a color image of the pistachio head by using the industrial camera 105, the coaxial light source 106 and the spherical integrating light source; Step S12, converting the color image into a grayscale image and performing preprocessing; Step S13, segmenting the image; Step S14, extracting effective targets of pistachios through Blob analysis; Step S15, using the Sobel operator to calculate and obtain the x and y direction gradients of the pistachio to obtain the edge image of the pistachio; Step S16, extracting pistachio features through edge contour and gradient analysis of pistachios; Step S17, obtain a straight line L with the longest distance between the two ends, and make a perpendicular bisector A of the straight line L to divide the pistachio image into left and right halves; Step S18, based on the perpendicular bisector A, calculating the area A1 of the left side contour of the target; Step S19, based on the perpendicular bisector A, calculating the area A2 of the right side contour of the target; Step S20: Identify the head and tail by area difference, and output the probability value η that the head is on the left side. M , the specific formula is as follows:

7. The method for identifying the head and tail of pistachios based on multi-channel information fusion as claimed in claim 6, characterized in that: Taking the pistachio head as the probability criterion, since η M There will be a negative probability if η M If η is greater than 0, it is the head. M If it is less than 0, it is judged as the tail; that is, η M The distribution interval is η M ∈(-1, 1), then η M Normalization is required, where the distribution of probability data from three different perspectives is (0, 1), specifically:

8. The method for identifying the head and tail of pistachios based on multi-channel information fusion as claimed in claim 6, characterized in that: Also includes: The process of identifying the head and tail of pistachios based on the horizontal imaging processing unit specifically includes the following steps: Step S21, obtaining a color image of the pistachio head through the industrial camera 102 and the spherical integrating light source 103; Step S22, converting the color image into a grayscale image based on binarization and performing preprocessing, and using the Sobel operator to calculate and obtain the x and y direction gradients of the pistachio; Step S23, smoothing the image using a low-pass filter and median filtering; Step S24, using image morphological processing to fill the black background of the image and remove the white noise points other than the pistachio; Step S25, using the minimum rectangle to cut out the effective area of ​​the entire pistachio; Step S26, constructing a deep learning network structure; Step S27: Use the deep learning network to output the probability value η that the pistachio head is on the left side L , and the probability that the head is on the right side η R .

9. The method for identifying the head and tail of pistachios based on multi-channel information fusion as claimed in claim 6, characterized in that: Also includes: Based on the probability value output of the three methods for the optimal decision of the head and tail of the pistachio, the probability value η of the optimal decision can be obtained by using multi-channel information fusion of the left imaging unit, the middle top imaging unit and the right imaging unit. End , the calculation formula is as follows: Among them, η L is the probability value of the pistachio head being on the left; η R is the probability that the head of the pistachio is on the right side; is the normalized probability value of the head of the pistachio head and tail being on the left; η End is the probability value of the optimal decision.

Images