Apple Fungal Infection Species Identification Method Based on Multi-Type Images and Recombinant Datasets of U-Net

Abstract

The present invention relates to a method for identifying apple fungal infection types based on multi-type images and a U-Net reorganized dataset, including obtaining an HSI image of an apple sample through an image acquisition device, obtaining a denoised HSI image; converting it into a pseudo-RGB image; expanding the dataset and training a U-Net semantic segmentation model; obtaining a second binary image; combining the second binary image with the denoised HSI image, extracting spectral features of the fungal infection area, and selecting characteristic wavelengths and corresponding monochromatic images; inputting the pseudo-RGB image and the monochromatic image into a VGG16 convolutional neural network to obtain fused features; inputting the fused features of the pseudo-RGB image and the monochromatic image into a classification model, and the classification model outputs the types of apple fungal infections. The present invention fully optimizes the U-Net semantic segmentation model, accurately segments the rotten area infected by apple fungi; the features are more diverse, and can effectively identify the types of fungi; it realizes automatic and non-destructive detection, saving time and labor costs.

Description

Technical Field

[0001] The present invention relates to the technical field of apple fungal infection detection, and in particular to a method for identifying apple fungal infection types based on multi-type images and a U-Net recombined dataset. Background Art

[0002] Apples are extremely vulnerable to fungal infections during harvesting, transportation, and storage, resulting in apple rot and causing huge post-harvest losses. Identifying the types of fungi and formulating targeted prevention and solution strategies can help improve apple quality and ensure food safety.

[0003] Hyperspectral imaging (HSI) has obvious advantages in fruit detection due to its rich spectral information. Currently, traditional manual fruit detection methods require a large amount of manpower and material resources, with high time costs and low accuracy; biochemical detection methods, such as gas chromatography-mass spectrometry, enzyme-linked immunosorbent assay, polymerase chain reaction, etc., although the accuracy has been improved, the experimental steps are complex; most of the current spectral map detection methods use only the characteristic wavelengths of hyperspectral or RGB images as data sources. The characteristic wavelengths of hyperspectral describe the narrowband information of a single band but do not have the two-dimensional characteristics of an image, while RGB images express broadband information but it is difficult to perceive the internal changes of the target object, and the analysis accuracy may be limited. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for identifying apple fungal infection types based on multi-type images and a U-Net recombined dataset, which can segment the rotten area of apples infected by fungi and identify the types of infected fungi.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions: A method for identifying apple fungal infection types based on multi-type images and a U-Net recombined dataset, the method includes the following steps in sequence:

[0006] (1) Obtain the HSI image of the apple sample through an image acquisition device, perform black-and-white version correction on the HSI image, and remove its noise to obtain the denoised HSI image;

[0007] (2) Use the CIE 1931 color matching function to convert the denoised HSI image into a pseudo-RGB image, and the pseudo-RGB images form a dataset;

[0008] (3) Expand the dataset, and use the expanded dataset to train the U-Net semantic segmentation model to obtain the trained U-Net semantic segmentation model;

[0009] (4) Segment the fungal infection area of the pseudo RGB image using the trained U-Net semantic segmentation model. The obtained segmentation result is a label marked with different colors. Then, binarize the label and denoise it to obtain the second binary image;

[0010] (5) Combine the second binary image with the denoised HSI image to extract the spectral features of the fungal infection area, and select the characteristic wavelengths and corresponding monochromatic images;

[0011] (6) Input the pseudo RGB image and the monochromatic image into the VGG16 convolutional neural network to obtain the convolutional network features of the pseudo RGB image and the monochromatic image. Then, fuse the convolutional network features of the pseudo RGB image and the monochromatic image to obtain the fused features;

[0012] (7) Build a classification model, input the fused features of the pseudo RGB image and the monochromatic image into the classification model, and the classification model outputs the types of apple fungal infections to achieve the identification of apple fungal infection types.

[0013] In step (1), the image acquisition device includes a dark box. At the central position on the top of the dark box, a hyperspectral imager for capturing HSI images is installed. On the bottom plate of the dark box, a lifting platform for controlling the distance between the apple sample and the hyperspectral imager is installed. At the upper end of the lifting platform, a placement table for placing the apple sample is provided. Four halogen lamps are respectively installed at the four corners of the top of the dark box to provide light sources. The computer is connected to the hyperspectral imager by wire, and the SRANL710 software is loaded on the computer.

[0014] In step (1), the black and white version correction of the HSI image is performed, and the correction formula is as follows:

[0015]

[0016] where I is the denoised HSI image, I o is the HSI image of the apple sample, I w is the reference image on the white board, and I d is the blackboard reference image obtained by covering the lens of the hyperspectral instrument with an opaque cover.

[0017] In step (2), the formula of the CIE 1931 color matching function is as follows:

[0018]

[0019]

[0020]

[0021] where, They are all color matching functions, which are numerical descriptions of the chromaticity response of an observer and are given fixed values; L(λ) is the spectral reflectance of the HSI image, and X, Y, and Z are the tristimulus values of the color; after obtaining the tristimulus values, the R, G, and B channels of all pixel points are obtained through matrix transformation:

[0022]

[0023] The superposition of the R, G, and B channels gives the pseudo-RGB image.

[0024] In step (3), the augmented dataset refers to: using the open-source image annotation tool LabelMe to segment the rotten area of the apple infected by fungi, that is, continuously taking points at the edge of the rotten area of the apple, and finally these points form a closed area. In LabelMe, set the rotten area of the apple to green and the normal area of the apple to red to obtain a colored label image. Then extract the image of the G channel of the colored label image to obtain the first binary image. In the first binary image, the rotten area of the apple is white, and the other areas except the rotten area of the apple are black. The pixel value of the rotten area of the apple is 1, and the pixel value of other areas is 0; multiply the first binary image by the corresponding apple image infected by fungi to obtain a lesion image with only the rotten area of the apple. Then perform an inversion operation on the first binary image to obtain the inverted binary image. In the inverted binary image, the rotten area of the apple is black, and the other areas except the rotten area are white. The pixel value of the rotten area of the apple is 0, and the pixel value of other areas is 1; multiply the inverted binary image by the normal apple image and then add it to the lesion image to achieve image stitching and obtain the augmented dataset.

[0025] The specific content of step (4) is: input the pseudo-RGB image into the trained U-Net semantic segmentation model, and the obtained segmentation result is a label marked with different colors. The rotten area of the apple is green, the normal area of the apple is red, and the background outside the apple is white. Then extract the single-channel image and binarize the label, and use morphological operations of dilation and erosion to remove tiny noise to obtain the second binary image. The rotten area of the apple and the background outside the apple become white, and the normal area of the apple is black.

[0026] The specific content of step (5) is: multiply the second binary image by the denoised HSI image to automatically obtain the spectral characteristics of the fungal infection area, and use the random frog leaping of the variable selection algorithm to select the characteristic wavelengths, and extract the monochromatic images corresponding to the characteristic wavelengths.

[0027] The specific content of step (6) is as follows: Use the VGG16 convolutional neural network to extract the convolutional network features of the image, and perform pre-training using the ImageNet dataset. Input the pseudo-RGB image and the monochromatic image into the trained VGG16 convolutional neural network respectively. The VGG16 convolutional neural network consists of 13 convolutional layers, 5 pooling layers, and 3 fully connected layers. The last convolutional layer outputs the convolutional network features of the image, which are output in the form of a one-dimensional vector, obtaining the convolutional network features of the pseudo-RGB image and the monochromatic image, and then fusing the convolutional network features of the pseudo-RGB image and the monochromatic image to obtain the fused features.

[0028] The specific content of step (7) is as follows: The classification model uses an SVM support vector machine. Input the pseudo-RGB image, the monochromatic image, and their fused features into the SVM support vector machine. The SVM support vector machine outputs the types of apple fungal infections, realizing the identification of apple fungal infection types.

[0029] As can be seen from the above technical solutions, the beneficial effects of the present invention are as follows: First, the present invention realizes the effect of dataset expansion by reorganizing the existing dataset, fully optimizes the U-Net semantic segmentation model, and can accurately segment the rotten area of apple fungal infection; Second, the present invention can fully combine the information of broadband and narrowband by using the pseudo-RGB image and the monochromatic image corresponding to the characteristic wavelength, and the features are more diverse, which can effectively identify the fungal species; Third, the present invention realizes automatic and non-destructive detection, saving time and labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is the flowchart of the method of the present invention;

[0031] Figure 2 is the flowchart of dataset expansion;

[0032] Figure 3 is the flowchart of fungal infection area segmentation;

[0033] Figure 4 is the structural schematic diagram of the image acquisition device in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0034] As Figure 1 shown, a method for identifying apple fungal infection types based on multi-type images and a reorganized dataset of U-Net, the method includes the following steps in sequence:

[0035] (1) Obtain the HSI image of the apple sample through the image acquisition device, perform black and white version correction on the HSI image, and remove its noise to obtain the denoised HSI image;

[0036] (2) Using the CIE 1931 color matching function, convert the denoised HSI image into a pseudo-RGB image, and the pseudo-RGB images form a dataset;

[0037] (3) Expand the dataset, and use the expanded dataset to train the U-Net semantic segmentation model to obtain a trained U-Net semantic segmentation model;

[0038] (4) Use the trained U-Net semantic segmentation model to segment the fungal infection area of the pseudo-RGB image. The obtained segmentation result is a label marked with different colors, and then binarize and denoise the label to obtain a second binary image;

[0039] (5) Combine the second binary image with the denoised HSI image, extract the spectral features of the fungal infection area, and select the characteristic wavelengths and corresponding monochromatic images;

[0040] (6) Input the pseudo-RGB image and the monochromatic image into the VGG16 convolutional neural network to obtain the convolutional network features of the pseudo-RGB image and the monochromatic image, and then fuse the convolutional network features of the pseudo-RGB image and the monochromatic image to obtain fused features;

[0041] (7) Construct a classification model, input the fused features of the pseudo-RGB image and the monochromatic image into the classification model, and the classification model outputs the types of apple fungal infections to realize the identification of apple fungal infection types.

[0042] In step (1), as Figure 4 shown, the image acquisition device includes a dark box 2. At the central position on the top of the dark box 2, a hyperspectral imager 1 for capturing HSI images is installed. On the bottom plate of the dark box 2, a lifting platform 5 for controlling the distance between the apple sample and the hyperspectral imager 1 is installed. At the upper end of the lifting platform 5, a placement table 4 for placing the apple sample is provided. Four halogen lamps 6 are respectively installed at the four corners of the top of the dark box 2 to provide light sources. The computer 3 is connected to the hyperspectral imager 1 by wire, and the SRANL710 software is loaded on the computer 3.

[0043] In step (1), the black-and-white version correction of the HSI image is performed, and the correction formula is as follows:

[0044]

[0045] where I is the denoised HSI image, I o is the HSI image of the apple sample, I w is the reference image on the white board, and I d is the blackboard reference image obtained by covering the lens of the hyperspectral instrument with an opaque cover. The opaque cover is a small lid that can completely cover the lens, similar to the lens cap of a camera.

[0046] In step (2), the formula of the CIE 1931 color matching function is as follows:

[0047]

[0048]

[0049]

[0050] where are all color matching functions, which are numerical descriptions of the observer chromaticity response and are given fixed values; L(λ) is the spectral reflectance of the HSI image, and X, Y, and Z are the three color tristimulus values; after obtaining the tristimulus values, the R, G, and B channels of all pixel points are obtained through matrix transformation:

[0051]

[0052] The superposition of the R, G, and B channels results in a pseudo-RGB image. λ is discrete, and the range of λ is the spectral range of the HSI image, that is, λ ∈ [374, 1031]. In this way, the integral formula for X, Y, and Z here is transformed into a discrete sum operation.

[0053] As Figure 2 shown, in step (3), the augmented dataset refers to: using the open-source image annotation tool LabelMe to segment the rotten area of the apple infected by fungi, that is, continuously taking points at the edge of the rotten area of the apple, and finally these points form a closed area. In LabelMe, set the rotten area of the apple to green and the normal area of the apple to red to obtain a colored label image, and then extract the image of the G channel of the colored label image to obtain a first binary image. In the first binary image, the rotten area of the apple is white, and other areas except the rotten area of the apple are black. The pixel value of the rotten area of the apple is 1, and the pixel value of other areas is 0; multiply the first binary image by the corresponding image of the apple infected by fungi to obtain a lesion image with only the rotten area of the apple, and then perform an inversion operation on the first binary image to obtain an inverted binary image. In the inverted binary image, the rotten area of the apple is black, and other areas except the rotten area are white. The pixel value of the rotten area of the apple is 0, and the pixel value of other areas is 1; multiply the inverted binary image by the image of the normal apple and then add it to the lesion image to achieve image stitching and obtain the augmented dataset. Suppose there are 20 rotten apples and 10 normal apples, then it can be expanded to 20 * 10 = 200 samples.

[0054] As Figure 3As shown in the figure, step (4) specifically refers to: inputting the pseudo RGB image into the trained U-Net semantic segmentation model, and the obtained segmentation result is a label marked with different colors. The rotten area of the apple is green, the normal area of the apple is red, and the background outside the apple is white. Then, a single-channel image is extracted to binarize the label, and morphological operations of dilation and erosion are used to remove tiny noise points, obtaining a second binary image. The rotten area of the apple and the background outside the apple become white, and the normal area of the apple is black.

[0055] The U-Net semantic segmentation model is an open-source classic semantic segmentation network, and a large amount of data is required to train this network. Therefore, an operation of data augmentation is performed. The augmented dataset is used as the training set of the U-Net semantic segmentation model, and the training label is the label made by LabelMe as described above. The trained model is saved, and then a sample of a fungus-infected apple is input into this model for testing, and the output is the segmented label marked with different colors.

[0056] Step (5) specifically refers to: multiplying the second binary image with the denoised HSI image to automatically obtain the spectral characteristics of the fungal infection area, and using the random frog leaping of the variable selection algorithm to select the characteristic wavelengths, and extracting the monochromatic images corresponding to the characteristic wavelengths.

[0057] Random frog leaping is specifically divided into three steps: (1) randomly initialize a variable subset V0 containing Q variables; (2) based on V0, propose a candidate variable subset V* containing Q* variables, accept V* with a certain probability as V1, and replace V0 with V1. This step is cycled until N iterations are completed; (3) finally, calculate the selection probability of each variable, and this probability can be used as a measure of variable importance, and the variables to be selected are sorted according to the variable importance. The hyperspectral image HSI can be regarded as an image with 260 channels, that is, there are 260 bands. In addition, each pixel point of each channel image of HSI also contains a spectral value, that is, spectral characteristics.

[0058] Step (6) specifically refers to: using the VGG16 convolutional neural network to extract the convolutional network features of the image, and using the dataset ImageNet for pre-training. The pseudo RGB image and the monochromatic image are respectively input into the trained VGG16 convolutional neural network. The VGG16 convolutional neural network consists of 13 convolutional layers, 5 pooling layers and 3 fully connected layers. The last convolutional layer outputs the convolutional network features of the image in the form of a one-dimensional vector, obtaining the convolutional network features of the pseudo RGB image and the convolutional network features of the monochromatic image, and then fusing the convolutional network features of the pseudo RGB image and the convolutional network features of the monochromatic image to obtain the fusion features.

[0059] The specific content of step (7) is as follows: The classification model adopts an SVM (Support Vector Machine). The pseudo-RGB image, the monochromatic image, and their fusion features are input into the SVM. The SVM outputs the types of apple fungal infections, realizing the identification of apple fungal infection types.

[0060] In summary, through the reorganization of the existing dataset, the present invention achieves the effect of dataset expansion, fully optimizes the U-Net semantic segmentation model, and can accurately segment the rotten areas infected by apple fungi; by using the pseudo-RGB image and the monochromatic image corresponding to the characteristic wavelength, the present invention can fully combine the information of broadband and narrowband, with more diverse features, and can effectively identify the types of fungi; the present invention realizes automatic and non-destructive detection, saving time and labor costs.

Claims

1. A method for identifying apple fungal infection species based on multi-type images and a recombined U-Net dataset, characterized in that: The method includes the following steps in sequence: (1) Obtain the HSI image of the apple sample through an image acquisition device, perform black-and-white calibration on the HSI image, remove its noise, and obtain the denoised HSI image; (2) Use the CIE 1931 color matching function to convert the denoised HSI image into a pseudo RGB image, and the pseudo RGB images form a data set; (3) Expand the data set, and use the expanded data set to train the U-Net semantic segmentation model to obtain the trained U-Net semantic segmentation model; (4) Use the trained U-Net semantic segmentation model to segment the fungal infection area of the pseudo RGB image. The obtained segmentation result is a label marked with different colors, and then the label is binarized and denoised to obtain the second binary image; (5) Combine the second binary image with the denoised HSI image, extract the spectral features of the fungal infection area, and select the characteristic wavelengths and corresponding monochromatic images; (6) Input the pseudo RGB image and the monochromatic image into the VGG16 convolutional neural network to obtain the convolutional network features of the pseudo RGB image and the monochromatic image, and then fuse the convolutional network features of the pseudo RGB image and the monochromatic image to obtain the fused features; (7) Construct a classification model, input the fused features of the pseudo RGB image and the monochromatic image into the classification model, and the classification model outputs the types of apple fungal infections to realize the identification of the types of apple fungal infections.

2. The method for identifying apple fungal infection types based on multi-type images and a U-Net reorganized dataset according to claim 1, wherein: In step (1), the image acquisition device includes a dark box. At the central position on the top of the dark box, a hyperspectral imager for capturing the HSI image is installed. On the bottom plate of the dark box, a lifting platform for controlling the distance between the apple sample and the hyperspectral imager is installed. At the upper end of the lifting platform, a placement table for placing the apple sample is provided. Four halogen lamps are respectively installed at the four corners of the top of the dark box to provide light sources. The computer is connected to the hyperspectral imager by wire, and the SRANL710 software is loaded on the computer.

3. The apple fungal infection species identification method based on multi-type images and U-Net recombined datasets according to claim 1, wherein: In step (1), the black-and-white calibration of the HSI image is performed, and the calibration formula is as follows: Among them, I is the denoised HSI image, and I o is the HSI image of the apple sample, and I w is the reference image on the whiteboard, and I d is the reference image of the blackboard obtained after covering the lens of the hyperspectral spectrometer with an opaque cover.

4. The method for identifying apple fungal infection types based on multi-type images and a U-Net recombined dataset according to claim 1, wherein: In step (2), the formula of the CIE 1931 color matching function is as follows: wherein, are all color matching functions, which are numerical descriptions of the chromaticity response of an observer and are given fixed values; L(λ) is the spectral reflectance of the HSI image, and X, Y, and Z are the tristimulus values of the color; after obtaining the tristimulus values, the R, G, and B channels of all pixel points are obtained through matrix transformation: The R, G, and B channels are superimposed together to obtain the pseudo RGB image.

5. The method for identifying apple fungal infection types based on multi-type images and U-Net recombined datasets according to claim 1, characterized in that: In step (3), the augmented dataset refers to: using the open-source image annotation tool LabelMe to segment the rotten area of the apple infected by fungi, that is, continuously taking points at the edge of the rotten area of the apple, and finally these points form a closed area. In LabelMe, set the rotten area of the apple to green and the normal area of the apple to red to obtain a colored label image. Then extract the image of the G channel of the colored label image to obtain a first binary image. In the first binary image, the rotten area of the apple is white, and the other areas except the rotten area of the apple are black. The pixel value of the rotten area of the apple is 1, and the pixel value of the other areas is 0; multiply the first binary image by the corresponding apple image infected by fungi to obtain a lesion image with only the rotten area of the apple. Then perform an inversion operation on the first binary image to obtain an inverted binary image. In the inverted binary image, the rotten area of the apple is black, and the other areas except the rotten area are white. The pixel value of the rotten area of the apple is 0, and the pixel value of the other areas is 1; multiply the inverted binary image by the normal apple image and then add it to the lesion image to achieve image stitching and obtain the augmented dataset.

6. The apple fungal infection species identification method based on multi-type images and U-Net recombined datasets according to claim 1, characterized in that: Step (4) specifically refers to: inputting the pseudo-RGB image into the trained U-Net semantic segmentation model, and the obtained segmentation result is a label marked with different colors. The rotten area of the apple is green, the normal area of the apple is red, and the background outside the apple is white. Then extract a single-channel image to binarize the label, and use morphological operations of dilation and erosion to remove small noise points to obtain a second binary image. The rotten area of the apple and the background outside the apple become white, and the normal area of the apple is black.

7. The method for identifying apple fungal infection types based on multi-type images and a U-Net reconstructed dataset according to claim 1, characterized in that: Step (5) specifically refers to: multiplying the second binary image by the denoised HSI image to automatically obtain the spectral characteristics of the fungal infection area, and using the random frog leaping of the variable selection algorithm to select the characteristic wavelengths, and extracting the monochromatic images corresponding to the characteristic wavelengths.

8. The method for identifying apple fungal infection types based on multi-type images and U-Net recombined datasets according to claim 1, wherein: Step (6) specifically refers to: using the VGG16 convolutional neural network to extract the convolutional network features of the image and using the dataset ImageNet for pre-training. Input the pseudo-RGB image and the monochromatic image into the trained VGG16 convolutional neural network respectively. The VGG16 convolutional neural network consists of 13 convolutional layers, 5 pooling layers and 3 fully connected layers. The last convolutional layer outputs the convolutional network features of the image in the form of a one-dimensional vector to obtain the convolutional network features of the pseudo-RGB image and the convolutional network features of the monochromatic image, and then fuse the convolutional network features of the pseudo-RGB image and the convolutional network features of the monochromatic image to obtain the fusion features.

9. The apple fungal infection species identification method based on multi-type images and U-Net recombined datasets according to claim 1, wherein: Step (7) specifically refers to: the classification model uses the SVM support vector machine. Input the pseudo-RGB image, the monochromatic image and their fusion features into the SVM support vector machine, and the SVM support vector machine outputs the types of apple fungal infections to achieve the identification of the types of apple fungal infections.

Images