Pleurotus eryngii surface defect detection method, device and storage medium

Abstract

The present invention discloses a method, device and storage medium for detecting surface defects of Pleurotus eryngii. The method comprises the following steps: S1: performing video acquisition around Pleurotus eryngii for at least one week, parsing the video into a sequence of Pleurotus eryngii image frames; S2: extracting foreground targets in the obtained Pleurotus eryngii image frames; S3: constructing a conditional generative adversarial network including a generator and a discriminator, inputting the extracted foreground targets into the generator and the discriminator respectively to generate a standard Pleurotus eryngii contour image; S4: extracting the original Pleurotus eryngii contour image from the foreground target; S5: performing sum and difference operations on the standard Pleurotus eryngii contour image and the original Pleurotus eryngii contour image to obtain the defect position of Pleurotus eryngii. The above method improves the detection efficiency of Pleurotus eryngii by replacing manual detection with machine vision, and adopts a generative adversarial network to generate a standard Pleurotus eryngii contour, so as to realize the defect position detection of Pleurotus eryngii with various shapes and defect shapes.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision, and in particular relates to a method, a device and a storage medium for detecting surface defects of Pleurotus eryngii. Background Art

[0002] Intelligent detection of surface defects in agricultural products has gradually replaced previous manual inspection and has become a trend. This is also a technology that traditional agricultural production companies need to break through in the process of moving towards intelligent production.

[0003] At present, the surface defect detection method of edible mushrooms such as Pleurotus eryngii uses machine vision algorithms in addition to manual detection. However, Pleurotus eryngii has different shapes and defect shapes. It is difficult for manually designed algorithms to adapt to the above-mentioned variable characteristics of Pleurotus eryngii. At the same time, traditional machine vision detection algorithms do not have generalization and low recognition accuracy. Therefore, it is necessary to find an intelligent detection method for Pleurotus eryngii surface defects based on deep learning, which can quickly, flexibly and accurately identify the location of defects on the surface of Pleurotus eryngii. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a method for detecting surface defects of Pleurotus eryngii, which can quickly, flexibly and with high precision identify the location of surface defects of edible mushrooms such as Pleurotus eryngii.

[0005] Another object of the present invention is to provide a device that can implement the above-mentioned method for detecting surface defects of Pleurotus eryngii, and a storage medium storing a computer program that instantiates the above-mentioned method for detecting surface defects of Pleurotus eryngii.

[0006] Technical solution: The method for detecting surface defects of Pleurotus eryngii of the present invention comprises the following steps:

[0007] S1: collect video around Pleurotus eryngii for at least one week, and parse the video into a sequence of Pleurotus eryngii image frames;

[0008] S2: extracting the foreground target in the Pleurotus eryngii image frame;

[0009] S3: Build a conditional generative adversarial network including a generator and a discriminator, and input the extracted foreground target into the generator and the discriminator respectively to generate a standard Pleurotus eryngii outline image;

[0010] S4: extracting the original Pleurotus eryngii outline image from the foreground object;

[0011] S5: Perform sum and difference operations on the standard Pleurotus eryngii contour image and the original Pleurotus eryngii contour image to obtain the Pleurotus eryngii defect position.

[0012] Furthermore, the step S2 comprises the steps of:

[0013] S2.1: Build a target detection backbone feature network to extract feature maps from Pleurotus eryngii image frames;

[0014] S2.2: using the attention algorithm to obtain the attention feature map of the feature map;

[0015] S2.3: Use the obtained attention feature map as node input and use the multi-scale feature fusion algorithm to obtain the fusion feature;

[0016] S2.4: Input the fused features into the global pooling layer, and output the region and category of Pleurotus eryngii in the image;

[0017] S2.5: Use image segmentation algorithm to cut out the Pleurotus eryngii in the Pleurotus eryngii image according to the area of ​​the Pleurotus eryngii in the image, and output the foreground target.

[0018] Furthermore, in step S2.3, the multi-scale feature fusion algorithm adopts a top-down and bottom-up bidirectional fusion method.

[0019] Furthermore, the target detection backbone feature network in step S2.1 is a MobileNet-SSD network.

[0020] The surface defect detection device of Pleurotus eryngii of the present invention comprises a clamp, a driving mechanism, an image acquisition device and a processor, wherein the clamp is used to clamp the Pleurotus eryngii, the driving mechanism is used to drive the clamp to rotate, the image acquisition device is used to perform video acquisition of the rotating Pleurotus eryngii, the processor is used to calculate the defect position, and the processor comprises a preprocessing module, a foreground target extraction module, a conditional generation confrontation module and a defect position calculation module, the preprocessing module is used to parse the video into a Pleurotus eryngii image frame sequence, the foreground target extraction module is used to extract the Pleurotus eryngii foreground target without background from the Pleurotus eryngii image frame, the conditional generation confrontation module is used to generate a standard Pleurotus eryngii contour image and an original Pleurotus eryngii contour image, and the defect position calculation module is used to perform sum and difference calculation on the Pleurotus eryngii contour image and the original Pleurotus eryngii contour image to obtain the Pleurotus eryngii defect position.

[0021] Furthermore, it also includes a background plate, and the clamp and the driving mechanism are arranged between the background plate and the image acquisition device.

[0022] Furthermore, the foreground target extraction module includes a backbone feature extraction network, a multi-channel attention sub-module and a multi-scale feature fusion module. The backbone feature extraction network is used to extract feature maps in the Pleurotus eryngii image frame. The multi-channel attention sub-module is embedded in the 6th, 8th, 10th, 12th and 14th layers of the backbone feature extraction network to obtain attention feature maps from the feature maps. The multi-scale feature fusion module is used to obtain fusion features from the attention feature maps.

[0023] Furthermore, the target detection backbone feature network is a MobileNet-SSD network.

[0024] Furthermore, the multi-scale feature fusion module adopts a top-down and bottom-up bidirectional fusion approach.

[0025] The storage medium of the present invention stores a computer program, and the computer program is configured to implement the above-mentioned method for detecting surface defects of Pleurotus eryngii when running.

[0026] Beneficial effects: Compared with the prior art, the present invention has the following advantages: 1. By replacing manual detection with machine vision, the detection efficiency of edible mushrooms such as Pleurotus eryngii is improved; 2. A standard Pleurotus eryngii image is directly generated from the original Pleurotus eryngii image using a generative adversarial network, which can realize the defect position monitoring of Pleurotus eryngii with various shapes and defect shapes; 3. A lightweight backbone network structure is used, and a lightweight multi-channel attention module and a bidirectional fusion multi-scale feature fusion module are added to the network to improve the detection accuracy and monitoring speed while reducing the amount of calculation. The network is easy to build and has low deployment difficulty, which is suitable for actual processing sites. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a method for detecting surface defects of Pleurotus eryngii according to an embodiment of the present invention;

[0028] Figure 2 A mechanical structure diagram of a device for detecting surface defects of Pleurotus eryngii according to an embodiment of the present invention;

[0029] Figure 3 A structural diagram of a multi-channel attention module according to an embodiment of the present invention;

[0030] Figure 4 is a structural diagram of a multi-scale feature fusion module according to an embodiment of the present invention;

[0031] Figure 5 4 is a structural diagram of a target detection backbone feature network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0033] Reference Figure 1 According to an embodiment of the present invention, a method for detecting surface defects of Pleurotus eryngii comprises the following steps:

[0034] S1: collect video around Pleurotus eryngii for at least one week, and parse the video into a sequence of Pleurotus eryngii image frames;

[0035] S2: extracting the foreground target in the Pleurotus eryngii image frame;

[0036] S3: Build a conditional generative adversarial network including a generator and a discriminator, and input the extracted foreground target into the generator and the discriminator respectively to generate a standard Pleurotus eryngii outline image;

[0037] S4: extracting the original Pleurotus eryngii outline image from the foreground object;

[0038] S5: Perform sum and difference operations on the standard Pleurotus eryngii contour image and the original Pleurotus eryngii contour image to obtain the Pleurotus eryngii defect position.

[0039] According to the above technical solution, the method for detecting surface defects of Pleurotus eryngii improves the detection efficiency of edible Pleurotus eryngii and other edible Pleurotus eryngii by replacing manual detection with machine vision. In addition, a generative adversarial network is used. The generator in the generative adversarial network generates a standard Pleurotus eryngii contour image based on the collected Pleurotus eryngii image frame for identification by the discriminator. The two are optimized alternately, so that the generator can generate a standard Pleurotus eryngii contour image, avoiding the time, manpower and material resources consumed by artificially making a standard Pleurotus eryngii contour image, and also improving the accuracy of the original Pleurotus eryngii contour image extracted by the discriminator. Finally, by performing sum and difference operations on the obtained standard Pleurotus eryngii contour and the original Pleurotus eryngii contour, the surface defect position of the Pleurotus eryngii can be obtained, and the defect position detection of Pleurotus eryngii and other Pleurotus eryngii with different shapes and defect shapes can be realized.

[0040] Among them, in step S1, the King Oyster Mushroom is clamped on the clamp 100, and the clamp 100 is driven to rotate by the driving mechanism 200, and the image acquisition device 300 takes a video of the rotating King Oyster Mushroom, and an image is captured once at an interval of 15 frames to obtain a group of image frame sequences of the King Oyster Mushroom, and the contour of each King Oyster Mushroom in each image frame sequence is calculated, and the 12 image frames with the largest contour diameter are taken to build a target detection backbone feature network for foreground target extraction.

[0041] Reference Figure 4 In this embodiment, the convolution kernel sizes of the network convolution layer of the backbone feature extraction network are 3×3 and 5×5 respectively, and a lightweight multi-channel attention module is embedded after the 6th, 8th, 10th, 12th and 14th layers of the backbone network, and a multi-channel feature fusion module is set between the multi-channel attention module and the global average pooling layer of the backbone feature extraction network. The specific foreground target extraction process is as follows: Figure 3 As shown:

[0042] Step 2.1: Input the 12 image frames of Pleurotus eryngii selected in step 1 into the backbone feature extraction network. The feature maps X output by the 6th, 8th, 10th, 12th and 14th layers of the backbone feature extraction network are i Input to the global average pooling layer of the backbone network for feature aggregation to obtain the aggregated feature y i , where the feature map X iThe sizes are C×H i ×W i , aggregate feature y i The size is C×1×1, and the formula is:

[0043]

[0044] Here, ε is a weight coefficient, which is 0.4 in this embodiment.

[0045] Step 2.2: The aggregated feature y obtained in step 2.1 i Input into the 1×1 convolution kernel of the multi-channel attention module for local cross-channel information interaction to obtain the interaction feature map ω i , after the activation function σ is applied, the output is:

[0046] ω i =σ(C1D k (y i ))(i=6, 8, 10, 12, 14) (2)

[0047] Step 2.3: Transform the feature map X i With the interaction feature map ω i Perform element-wise product to obtain cross-channel attention feature map Cross-channel attention feature map The size is C×H i ×W i , the formula is:

[0048]

[0049] Step 2.4: Input the cross-channel attention feature map output in step 2.3 as 5 feature nodes into the multi-scale feature fusion algorithm module, input the global average pooling layer after feature fusion, and output the area and category of Pleurotus eryngii in the image frame;

[0050] Step 2.5: The image segmentation algorithm cuts out the King Oyster Mushroom foreground target according to the area of ​​the King Oyster Mushroom in the image, and outputs the King Oyster Mushroom image without the background.

[0051] In order to reduce the amount of calculation, lighten the backbone network structure, reduce the difficulty of deployment, and make it more suitable for actual processing site operations, in this embodiment, the target detection backbone network adopts the MobileNet-SSD network, and uses deep separable convolution to decompose and calculate the standard convolution kernel to reduce the amount of calculation. At the same time, the multi-channel feature fusion module adopts a top-down and bottom-up bidirectional fusion method, adding a shortcut between the bottom and top layers, shortening the path between layers. The specific synthesis method can be described as:

[0052] Step 2.4.1: Feature nodes 6 and 8 are merged to obtain node A, A = 6 + 8;

[0053] Step 2.4.2: Feature nodes A and 10 are merged to obtain node B, B = A + 10;

[0054] Step 2.4.3: The feature nodes B and 12 are merged to obtain node C, C = B + 12;

[0055] Step 2.4.4: The feature nodes C and 14 are merged to obtain node D, D = C + 14;

[0056] Step 2.4.5: The feature points D, C and 12 are fused to obtain the node E, E = D + C + 12;

[0057] Step 2.4.6: The feature points E, B and 10 are fused to obtain the node F, F = E + B + 10;

[0058] Step 2.4.7: The feature point F is fused with A and 8 to obtain the node G, G = F + A + 8;

[0059] Step 2.4.8: The feature points G and 6 are fused to obtain the node H, G = H + 6;

[0060] Step 2.4.9: Input feature nodes D, E, F, G, and H into the global pooling layer, and output the region and category of Pleurotus eryngii in the image.

[0061] In this embodiment, the GrabCut algorithm is used to model the foreground and background according to the position of the Pleurotus eryngii in the image frame predicted by the input target detection backbone network, and the minCut algorithm is used to achieve segmentation, and the foreground target image with the background removed and only the Pleurotus eryngii foreground retained is output.

[0062] In this embodiment, the generator network uses the UNet network, and the discriminator network uses the patchGAN network. The generator generates a standard Pleurotus eryngii image based on the Pleurotus eryngii foreground target image, and then inputs the standard Pleurotus eryngii image into the discriminator to extract the features of the standard Pleurotus eryngii image and calculate the probability distribution P of the image features. norm To identify, when the probability distribution P norm Close to the true standard Pleurotus eryngii probability distribution P Ture When , the generator outputs the contour image of the standard Pleurotus eryngii image, and performs sum and difference operations with the contour image of the Pleurotus eryngii foreground target image, that is, the original Pleurotus eryngii contour image, to obtain the defect position of the Pleurotus eryngii.

[0063] In the process of extracting the contour of Pleurotus eryngii, the foreground target image of Pleurotus eryngii is first converted into a grayscale image and denoised using a Gaussian filter algorithm. Finally, the Canny algorithm is used to extract the contour of Pleurotus eryngii in the image and output the original Pleurotus eryngii contour image.

[0064] In this embodiment, the defective pixel Pixel i The calculation method is as follows:

[0065] I SUMi =I i +P i (4)

[0066] Pixel i =I SUMi -P i (5)

[0067] Among them I SUMi The original Pleurotus eryngii outline image I i (i=1…12) and the standard Pleurotus eryngii outline image P i (i=1…12) The sum of the sums according to the image operation rules. Extract the defective pixel Pixel i The coordinates of [(u1,v1),…,(u n ,v n )], that is, the position coordinates of the Pleurotus eryngii defect are obtained.

[0068] It is understandable that the above method can also be applied to edible mushrooms with similar structures such as Agaricus bisporus and Boletus.

[0069] Reference Figure 2 According to an embodiment of the present invention, the surface defect detection device of Pleurotus eryngii includes a fixture 100, a driving mechanism 200, an image acquisition device 300 and a processor 400. The fixture 100 is used to clamp the Pleurotus eryngii, the driving mechanism 200 is used to drive the fixture 100 to rotate, the image acquisition device 300 is used to capture the video of the rotating Pleurotus eryngii, and the processor 400 is used to calculate the defect position. The processor 400 includes a preprocessing module, a foreground target extraction module, a conditional generation adversarial module and a defect position calculation module. The preprocessing module is used to parse the video into a sequence of Pleurotus eryngii image frames, the foreground target extraction module is used to extract the Pleurotus eryngii foreground target without background from the Pleurotus eryngii image frame, the conditional generation adversarial module is used to generate a standard Pleurotus eryngii contour image and an original Pleurotus eryngii contour image, and the defect position calculation module is used to perform sum and difference calculation on the Pleurotus eryngii contour image and the original Pleurotus eryngii contour image to obtain the Pleurotus eryngii defect position.

[0070] In this embodiment, in order to further reduce the amount of calculation, a monochrome background plate 500 is also provided, and the clamp 100 and the driving mechanism 200 are arranged between the background plate 500 and the image acquisition device 300. At the same time, in order to ensure that the Pleurotus eryngii clamped by the clamp 100 rotates at a uniform speed, the driving mechanism 200 includes a servo motor and a gear set, and the servo motor drives the clamp 100 to rotate by a gear set composed of large and small gears. The storage medium of the embodiment of the present invention stores a computer program that instantiates the above-mentioned Pleurotus eryngii surface defect detection method.

Claims

1. A method for detecting surface defects of Pleurotus eryngii, characterized in that: The steps include: S1: The clamp is used to clamp the Pleurotus eryngii, the driving mechanism is used to drive the clamp to rotate, and the image acquisition device is used to capture the video of the rotating Pleurotus eryngii; the video is parsed into a Pleurotus eryngii image frame sequence; S2: extracting foreground targets in the obtained Pleurotus eryngii image frame; the foreground target extraction process includes a backbone feature extraction network, a multi-channel attention submodule and a multi-scale feature fusion module, wherein the backbone feature extraction network is used to extract feature maps in the Pleurotus eryngii image frame, the multi-channel attention submodule is embedded in the 6th, 8th, 10th, 12th and 14th layers of the backbone feature extraction network, and is used to obtain attention feature maps from the feature maps, and the multi-scale feature fusion module is used to obtain fusion features from the attention feature maps; S3: Build a conditional generative adversarial network including a generator and a discriminator, and input the extracted foreground target into the generator and the discriminator respectively to generate a standard Pleurotus eryngii outline image; The generator generates a standard King Pleurotus eryngii image according to the King Pleurotus eryngii foreground target, and then inputs the standard King Pleurotus eryngii image into the discriminator, extracts the features of the standard King Pleurotus eryngii image, and calculates the probability distribution Pnorm of the image features for identification. When the probability distribution Pnorm is close to the probability distribution PTure of the real standard King Pleurotus eryngii, the generator outputs the contour image of the standard King Pleurotus eryngii image. S4: extracting the original Pleurotus eryngii outline image from the foreground object; S5: performing sum and difference operations on the standard Pleurotus eryngii contour image and the original Pleurotus eryngii contour image to obtain the defect position of the Pleurotus eryngii; The step S2 comprises the steps of: S2.1: Build a target detection backbone feature network to extract feature maps from Pleurotus eryngii image frames; S2.2: using the attention algorithm to obtain the attention feature map of the feature map; S2.3: Use the obtained attention feature map as node input and use the multi-scale feature fusion algorithm to obtain the fusion feature; S2.4: Input the fused features into the global pooling layer, and output the region and category of Pleurotus eryngii in the image; S2.5: using an image segmentation algorithm to cut out the Pleurotus eryngii in the Pleurotus eryngii image according to the area of ​​the Pleurotus eryngii in the image, and outputting the foreground target; Step S2.4 inputs the cross-channel attention feature map output in step 2.3 as a feature node into the multi-scale feature fusion algorithm module, inputs the global average pooling layer after feature fusion, and outputs the area and category of Pleurotus eryngii in the image frame, specifically including: Step 2.4.1: Feature nodes 6 and 8 are merged to obtain node A, A = 6 + 8; Step 2.4.2: Feature nodes A and 10 are merged to obtain node B, B = A + 10; Step 2.4.3: The feature nodes B and 12 are merged to obtain node C, C = B + 12; Step 2.4.4: The feature nodes C and 14 are merged to obtain node D, D = C + 14; Step 2.4.5: The feature points D, C and 12 are fused to obtain the node E, E = D + C + 12; Step 2.4.6: The feature points E, B and 10 are fused to obtain the node F, F = E + B + 10; Step 2.4.7: The feature point F is fused with A and 8 to obtain the node G, G = F + A + 8; Step 2.4.8: The feature points G and 6 are fused to obtain the node H, G = H + 6; Step 2.4.9: Input feature nodes D, E, F, G, and H into the global pooling layer, and output the region and category of Pleurotus eryngii in the image.

2. The method for detecting surface defects of Pleurotus eryngii according to claim 1, characterized in that: In step S2.3, the multi-scale feature fusion algorithm adopts a top-down and bottom-up bidirectional fusion method.

3. The method for detecting surface defects of Pleurotus eryngii according to claim 1, characterized in that: The target detection backbone feature network in step S2.1 is a MobileNet-SSD network.

4. A device for detecting surface defects of Pleurotus eryngii, characterized in that: The invention comprises a clamp, a driving mechanism, an image acquisition device and a processor, wherein the clamp is used to clamp the Pleurotus eryngii, the driving mechanism is used to drive the clamp to rotate, the image acquisition device is used to perform video acquisition on the rotating Pleurotus eryngii, the processor is used to calculate the defect position, the processor comprises a preprocessing module, a foreground target extraction module, a conditional generation confrontation module and a defect position calculation module, the preprocessing module is used to parse the video into a Pleurotus eryngii image frame sequence, the foreground target extraction module is used to extract the Pleurotus eryngii foreground target without background from the Pleurotus eryngii image frame, the conditional generation confrontation module is used to generate a standard Pleurotus eryngii contour image and a Pleurotus eryngii contour image. The original Pleurotus eryngii contour image, the defect position calculation module is used to perform sum and difference calculation on the Pleurotus eryngii contour image and the original Pleurotus eryngii contour image to obtain the Pleurotus eryngii defect position, the foreground target extraction module includes a backbone feature extraction network, a multi-channel attention submodule and a multi-scale feature fusion module, the backbone feature extraction network is used to extract the feature map in the Pleurotus eryngii image frame, the multi-channel attention submodule is embedded in the 6th, 8th, 10th, 12th and 14th layers of the backbone feature extraction network, and is used to obtain the attention feature map from the feature map, and the multi-scale feature fusion module is used to obtain the fusion feature from the attention feature map; The foreground target extraction module builds a target detection backbone feature network to extract feature maps in the Pleurotus eryngii image frame; An attention algorithm is used to obtain an attention feature map of the feature map; The obtained attention feature map is used as the node input, and the fusion feature is obtained by using the multi-scale feature fusion algorithm; The fused features are input into the global pooling layer, and the region and category of the Pleurotus eryngii in the image are output; the image segmentation algorithm is used to cut out the Pleurotus eryngii in the Pleurotus eryngii image according to the region of the Pleurotus eryngii in the image, and the foreground target is output; The multi-scale feature fusion module specifically includes: Step 2.4.1: Feature nodes 6 and 8 are merged to obtain node A, A = 6 + 8; Step 2.4.2: Feature nodes A and 10 are merged to obtain node B, B = A + 10; Step 2.4.3: The feature nodes B and 12 are merged to obtain node C, C = B + 12; Step 2.4.4: The feature nodes C and 14 are merged to obtain node D, D = C + 14; Step 2.4.5: The feature points D, C and 12 are fused to obtain the node E, E = D + C + 12; Step 2.4.6: The feature points E, B and 10 are fused to obtain the node F, F = E + B + 10; Step 2.4.7: The feature point F is fused with A and 8 to obtain the node G, G = F + A + 8; Step 2.4.8: The feature points G and 6 are fused to obtain the node H, G = H + 6; Step 2.4.9: Input feature nodes D, E, F, G, and H into the global pooling layer, and output the region and category of Pleurotus eryngii in the image.

5. The Pleurotus eryngii surface defect detection device according to claim 4, characterized in that: It also includes a background plate, and the clamp and the driving mechanism are arranged between the background plate and the image acquisition device.

6. The Pleurotus eryngii surface defect detection device according to claim 4, characterized in that: The target detection backbone feature network is a MobileNet-SSD network.

7. The Pleurotus eryngii surface defect detection device according to claim 4, characterized in that: The multi-scale feature fusion module adopts a top-down and bottom-up bidirectional fusion approach.

8. A storage medium storing a computer program, characterized in that: The computer program is configured to implement the method for detecting defective surfaces of Pleurotus eryngii according to any one of claims 1 to 3 when executed.

Images