Edible mushroom miscellaneous bacteria detection method based on attention multi-source feature fusion

Abstract

The application discloses a kind of based on attention multi-source feature fusion edible mushroom miscellaneous bacteria detection method, comprising:1. collect temperature, humidity, CH4, H2S, NH3 and image data, normalize data set to improve convergence speed;2. construct main network ResNet50, utilize the feature extraction capability of residual structure to extract visual feature;3. construct MLP, utilize the network to the effective processing of continuous number, extract sensor feature;4. design feature fusion network, self-attention mechanism calculates the attention of image and sensor feature respectively, cross-attention mechanism learns the correlation between image and sensor feature, effectively fuses two kinds of features;5. using Softmax classification, process the feature information after fusion and mapping classification;6. using cross-entropy loss function to evaluate model;7. using transfer learning, load the pre-training weight of model and carry out fine-tuning, to speed up the training speed of model and improve the accuracy of model;8. use the trained model to detect miscellaneous bacteria.

Description

Technical Field

[0001] This invention relates to the fields of computer vision image recognition technology, data fusion technology, and Internet of Things, specifically to combining deep learning models with sensors to detect contamination by miscellaneous bacteria during the growth of edible fungi, including a multifunctional smart box, a cloud platform for monitoring and storing sensor data, and a deep learning model. Background Technology

[0002] Edible fungi are a type of edible fungus, rich in nutrients, and are a high-protein, low-fat health food. my country is the world's largest producer of edible fungi. The edible fungi industry has developed rapidly and is now the fifth largest industry in China's planting sector, surpassing cotton, tea, and oilseeds. However, in edible fungi production, factors such as the rich nutrients in the culture medium, the relatively sealed and humid environment, excessively high temperatures, and the lack of strict sterilization in the early stages can lead to contamination of the mushroom logs with other microorganisms, resulting in decreased quality and yield. To ensure the yield and quality of edible fungi, timely detection and control of major competing diseases in edible fungi production are of research significance and application value. Currently, the identification of contamination by other microorganisms in my country mainly relies on the experience of agricultural experts and plant protection technicians, which is time-consuming, labor-intensive, and has significant limitations.

[0003] Existing research on the detection of contaminants in edible fungi based on deep learning models, such as the technology based on the Ghost-YOLOV4 deep learning model to identify shiitake mushroom logs contaminated with contaminants, not only ensures the accuracy of the identification and detection of contaminant contamination in shiitake mushroom logs, but also has good performance in terms of detection speed and real-time performance. It provides an effective solution for the lightweight deployment of target detection models on embedded equipment in aquaculture greenhouses, with an accuracy of 94.5%.

[0004] Deep learning models have achieved good results in the identification of contamination by various fungi. However, given the wide variety of edible fungi, it is necessary to propose a model applicable to contamination by more types of edible fungi. Furthermore, in the early stages of contamination, images alone are insufficient for accurate identification, while temperature and humidity are related to contamination, and odor and color antagonism lines can provide more accurate identification. Therefore, this paper proposes a multi-source feature fusion-based method for detecting contamination in edible fungi. This method uses methane (CH4), hydrogen sulfide (H2S), and ammonia (NH3) gas sensors to collect gases produced by fungi, temperature and humidity sensors to obtain real-time environmental information, and an imaging device to capture image data of the fungal substrate. Data fusion technology is used to achieve multi-sensory contamination detection, improving the model's accuracy and enabling the detection of contamination by more types of edible fungi. Summary of the Invention

[0005] In response to the aforementioned practical problems and to address the shortcomings of existing technologies, this invention proposes a method for detecting miscellaneous bacteria in edible fungi based on attention-based multi-source feature fusion, which can improve the accuracy of miscellaneous bacteria detection in edible fungi.

[0006] The technical solution of the present invention is as follows;

[0007] A method for detecting contaminating microorganisms in edible fungi based on attention-based multi-source feature fusion includes the following steps:

[0008] Step 1: Preprocess the acquired image dataset and the sensor dataset from the same time period respectively;

[0009] Step 2: Construct the backbone network of the image feature extraction module and output the image feature vector;

[0010] Step 3: Construct a multilayer perceptron (MLP) for sensor feature extraction and output sensor feature vectors;

[0011] Step 4: Construct a network that fuses image features and sensor features to generate new feature maps;

[0012] Step 5: Classify the data according to the feature information using the Softmax classifier, and select the classification result with the highest confidence rate;

[0013] Step 6: Define the loss function for model training;

[0014] Step 7: Train the model;

[0015] Step 8: Use the trained model to detect miscellaneous bacteria on the edible mushroom sticks.

[0016] Further, step 1 specifically includes: placing edible mushroom spawn in a multifunctional intelligent box for growth, where the volatile substances produced by the spawn will fill the entire space. CH4, H2S, and NH3 sensors continuously collect gas concentration values. The gas sensors convert their responses into electrical signals, and then convert the electrical signals into gas concentration values. A temperature sensor acquires the temperature value within the space, a humidity sensor acquires the humidity value within the space, and an imaging device acquires image data of the spawn during the same time period. Then, the sensor data and image data are preprocessed separately. Noise in the image data is manually removed, including faded, blurred, and duplicate images, and data augmentation methods such as scaling, rotation, movement, and random rotation cropping are used to add more usable new image data from the existing image data. The gas sensors collect concentration values ​​over a continuous time period and take the average as the final value. Due to the heterogeneity of the sensor data, normalization is performed to unify all data into a roughly the same numerical range.

[0017]

[0018] Further, step 2 specifically includes: constructing a backbone network ResNet50, consisting of 5 layers. The first layer C1 consists of a 7×7 kernel convolution with a stride of 2, a BN normalization layer, and a ReLU activation function. C2 consists of a 3×3 kernel pooling layer and 3 residual blocks. C3 to C5 consist of 4, 6, and 3 residual blocks, respectively. The residual blocks consist of a 1×1 kernel convolution, a 3×3 kernel convolution, and a 1×1 kernel convolution. After the 5th layer, an average pooling layer and a fully connected layer are constructed to utilize the strong feature extraction capability of the residual structure to extract visual features.

[0019] Furthermore, step 3 specifically includes: constructing a multilayer perceptron (MLP), consisting of three layers: an input layer, a hidden layer, and an output layer. The first layer, C1, consists of 5 neurons, corresponding to the values ​​from the three gas, temperature, and humidity sensors. The second hidden layer, C2, consists of multiple network layers with varying numbers of neurons. The third layer, C3, consists of 64 neurons and uses the sigmoid activation function. Based on the MLP's effective processing of continuous numbers, sensor features are extracted.

[0020] Furthermore, the network described in step 4 is a feature fusion network based on a cross-attention mechanism. The visual feature vector S extracted by the backbone network and the sensor feature vector I extracted by the multilayer perceptron are used as inputs to the feature fusion network. The self-attention mechanism is similar to a retrieval system: the query vector is used to search for information, and then the search engine attempts to find the key in its database and match it with the query vector. Finally, the value vector corresponding to the key is used as the output. S and I are input into the self-attention module respectively to obtain the query vector Q, the key vector K, and the value vector V. In the self-attention mechanism, the input sequence is mapped to the query vector, key vector, and value vector through a linear layer:

[0021] Q = Input * W q (2)

[0022] K = Input * W k (3)

[0023] V = Input * W v (4)

[0024] Among them W x Let x represent the weights, taking values ​​q, k, and v. The cross-attention mechanism is an asymmetric combination of Q, K, and V. Then, an attention score is calculated, which determines the degree of importance attached to each element.

[0025]

[0026] s(q i ,k jThe value is the score after dot product and scaling. Attempt to find the optimized mapping matrix (weights of these linear layers) during backpropagation. Attention weight calculation:

[0027]

[0028] Where d k It is the dimension of the key vector. It is a scalar used to avoid the dominant term when computing the softmax function. Finally, the attention weights are multiplied by the value vector to obtain the final weighted mapping:

[0029]

[0030] This yields the output vector O1, and similarly, Q... S K I V I The input is fed into the cross-attention module to obtain the output vector O2. Connecting O1 and O2 forms a new feature map O. The cross-attention mechanism can effectively fuse sensor features and image features.

[0031] Furthermore, the Softmax classifier described in step 5 is constructed by inputting O into a fully connected layer. Since the proposed model is a multi-class classification problem, the Softmax activation function is used for classification. The Softmax activation function generates a vector with 6 values, representing the 5 types of edible fungi and their health status. Each value in the vector represents the confidence score for each class, and the highest confidence score is selected as the classification result.

[0032] Furthermore, step 6 specifically includes: the loss function, used to estimate the degree of inconsistency between the model's predicted value f(X) and the true value Y, is a non-negative real-valued function, usually denoted by L(Y,f(X)). The smaller the loss function, the better the robustness of the model. The cross-entropy loss function is a commonly used objective loss function in image multi-class classification. Let X be the feature space, and Y = {1,…,c} be the label space. The classifier maps the input feature space to the label space f:x→R. c The function of cross-entropy loss in multi-class classification:

[0033]

[0034] Where θ is the parameter set of the classifier, y ij Corresponding sample x i The j-th element of the one-hot encoded tag.

[0035] Furthermore, step 7 specifically includes: To accelerate model training and improve model accuracy, the pre-trained weights of the model are loaded and fine-tuned based on transfer learning. The model input includes the original image, sensor readings of C2H5OH, H2S, and NH3, as well as temperature and humidity values. The image data and sensor data are normalized before being input into the network. Adam is used for optimization, and the weight updates are adaptively adjusted.

[0036] Step 8 specifically includes: the multifunctional intelligent box collects image data and sensor data, and preprocesses them before inputting them into the model. The trained model is then used to detect the input data, combining chemical sensor, physical sensor, and image features to detect miscellaneous bacteria on the edible mushroom substrate.

[0037] The innovative point of this invention is:

[0038] 1. This invention introduces features from chemical and physical sensors into image classification. Classification based on a single data source suffers from low recognition accuracy. Features from multiple data sources can more comprehensively characterize diseases, mimicking human multi-sensory disease identification, and can improve the accuracy of disease identification to a certain extent. Furthermore, the initial characteristics of edible fungi and other microorganisms are not obvious, so introducing sensor values ​​can improve the model's ability to identify early-stage microbial diseases.

[0039] 2. This invention proposes a feature fusion network based on a cross-attention mechanism. The cross-attention mechanism possesses global learning capabilities and can effectively fuse key information from different sensors. Visual features and sensor features are respectively processed by a self-attention module to obtain their respective Q, K, and V vectors. These three vectors are then asymmetrically combined, and the attention distribution is calculated. The feature fusion method based on the cross-attention mechanism can learn the correlation between different sensors, better fuse cross-domain sensor features, thereby improving the model's accuracy and robustness.

[0040] The advantages of this invention are: during the growth of edible fungi, competitive pathogenic bacteria may arise due to improper operation or environmental factors in the early stages of production. This network model is designed to address this issue. The multi-sensor feature fusion strategy based on the cross-attention mechanism described above can effectively improve the detection of pathogenic bacteria and provide a foundation for disease prevention and timely treatment during the growth of edible fungi, thereby contributing to improved yield and quality. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the neural network model of the present invention.

[0042] Figure 2 This is a schematic diagram of the neural network model construction process of the present invention. Detailed Implementation

[0043] The embodiments of the present invention will now be described in more detail, and schematic diagrams of the embodiments are provided in the appendix. Figure 1 It is displayed in the middle.

[0044] Reference Figure 1 A method for detecting contaminating microorganisms in edible fungi based on attention-based multi-source feature fusion includes the following steps:

[0045] Step 1: Clear the gas from the space. Place the mushroom spawn in the multi-functional smart box and record the growth process of the mushroom spawn. The volatile substances produced by the spawn will fill the entire space. The smart box can be set with temperature and humidity. CH4, H2S, and NH3 sensors continuously measure gas concentrations, selecting concentration values ​​over a continuous 45 seconds. The gas sensors convert their responses into electrical signals, which are then converted into gas concentration values. A temperature sensor acquires the temperature value within the space, and a humidity sensor acquires the humidity value. Simultaneously, an imaging device collects image data of the mushroom spawn. Then, preprocessing operations are performed on the sensor data and image data. Noise in the image data is manually removed, including faded, blurred, and duplicate images. Data augmentation methods such as scaling, rotation, movement, and random rotation cropping are used to add more usable new image data from the existing image data. The gas concentration values ​​collected by the three gas sensors are taken as the average over a 45-second time period as the final value. Due to the heterogeneity of the data from different sensors, the sensor values ​​are normalized to unify all data into a roughly similar numerical range.

[0046]

[0047] Step 2: Construct the backbone network ResNet50, consisting of 5 layers. The first layer, C1, comprises a 7×7 kernel convolution with a stride of 2, a BN normalization layer, and a ReLU activation function. C2 consists of a 3×3 kernel pooling layer and 3 residual blocks. C3 to C5 consist of 4, 6, and 3 residual blocks, respectively. The residual blocks are constructed sequentially using a 1×1 kernel convolution, a 3×3 kernel convolution, and a 1×1 kernel convolution. After the 5th layer, average pooling layers and fully connected layers are constructed to leverage the strong feature extraction capabilities of the residual structure to extract visual features.

[0048] Step 3: Construct a Multilayer Perceptron (MLP), consisting of three layers: an input layer, a hidden layer, and an output layer. The first layer, C1, comprises 5 neurons, corresponding to the values ​​from three sensors: a gas sensor, a temperature sensor, and a humidity sensor. The second hidden layer, C2, consists of network layers with 16, 48, 32, and 64 neurons respectively. The third layer, C3, consists of 64 neurons and uses the sigmoid activation function. Based on the MLP's efficient processing of continuous numbers, sensor features are extracted.

[0049] Step 4: Construct a feature fusion network based on a cross-attention mechanism. The visual feature vector S extracted by the backbone network and the sensor feature vector I extracted by the multilayer perceptron are used as inputs to the feature fusion network. The self-attention mechanism is similar to a retrieval system: the query vector is used to search for information, then the search engine attempts to find the key in its database and match it with the query vector, and finally the value vector corresponding to the key is used as the output. S and I are input into their respective self-attention modules to obtain the query vector Q, the key vector K, and the value vector V. In the self-attention mechanism, the input sequence is mapped to the query vector, key vector, and value vector through a linear layer:

[0050] Q = Input * W q (2)

[0051] K = Input * W k (3)

[0052] V = Input * W v (4)

[0053] Among them W x Let x represent the weights, taking values ​​q, k, and v. Cross-attention is an asymmetric combination of different input sequences Q, K, and V. An attention score is calculated, which determines the degree of importance given to each element.

[0054]

[0055] s(q i ,k j The value is the score after dot product and scaling. Attempt to find the optimized mapping matrix (weights of these linear layers) during backpropagation. Attention weight calculation:

[0056]

[0057] Where d k It is the dimension of the key vector. It is a scalar used to avoid the dominant term when computing the softmax function. Finally, the attention weights are multiplied by the value vector to obtain the final weighted mapping:

[0058]

[0059] This yields the output vector O1, and similarly, Q... S K I V I The input is fed into the cross-attention module to obtain the output vector O2. O1 and O2 are then concatenated to form a new feature map O. The cross-attention mechanism can further and more effectively fuse sensor features and image features while suppressing unwanted noise.

[0060] Step 5: Construct the Softmax classifier. The output O from the previous step is input into the fully connected layer. The model proposed in this invention is a multi-class classification problem involving various fungi; therefore, the Softmax activation function is used for classification. The Softmax activation function generates a vector with 6 values, containing the 5 types of edible fungi and their health status. Each value in the vector represents the confidence score for each classification, and the highest confidence score is selected as the classification result.

[0061] Step 6: Define the loss function for model training. The loss function estimates the degree of inconsistency between the model's predicted value f(X) and the true value Y. It is a non-negative real-valued function, usually denoted by L(Y,f(X)). The smaller the loss function, the better the robustness of the model. The cross-entropy loss function is a commonly used objective loss function in image classification. Let X be the feature space, and Y = {1,…,c} be the label space. The classifier maps the input feature space to the label space f:x→R. c The function of cross-entropy loss in multi-class classification:

[0062]

[0063] Where θ is the parameter set of the classifier, y ij Corresponding sample x i The j-th element of the one-hot encoded tag.

[0064] Step 7: Train the network model. To accelerate the training speed and improve the accuracy of the model, a transfer learning method is adopted, loading the pre-trained weights of the model and fine-tuning them. The input of the proposed model includes a 1280×720 original RGB image, gas sensor response values ​​for CH4, H2S, and NH3, as well as temperature and humidity values. The image data and sensor data are normalized before being input into the network. Adam is used for optimization, and the weight updates are adaptively adjusted.

[0065] Step 8: Use the trained model to detect miscellaneous bacteria on the edible mushroom substrate. The multifunctional intelligent box collects image data and sensor data, and preprocesses them before inputting them into the model. The trained model is used to detect the input data, combining chemical sensor, physical sensor, and image features to detect miscellaneous bacteria on the edible mushroom substrate.

Claims

1. An edible mushroom miscellaneous fungus detection method based on attention multi-source feature fusion, characterized by, Comprise the following steps: Step 1, the collected image data set and the same time period of sensor data set are pretreated respectively; the sensor data set includes temperature value, humidity value and CH4, H2S, NH3 gas concentration value information; Step 2, the image feature extraction module is constructed, and the image feature vector is output; Step 3, the multi-layer perception MLP of sensor feature extraction is constructed, and the sensor feature vector is output; Step 4, a network for fusing image features and sensor features is constructed to generate a new feature vector; the network is a feature fusion network based on cross attention mechanism; The visual feature vector S extracted by the image feature extraction module and the sensor feature vector I extracted by the multilayer perception are taken as inputs of the feature fusion network; the self-attention mechanism is similar to a retrieval system: a query vector is used to search information, then the search engine will try to find the key in its database and pair the query vector, and finally the value vector corresponding to the key is taken as the output; S and I are respectively input into the self-attention module to obtain a query vector , a key vector and a value vector ; in the self-attention mechanism, the input sequence is mapped to the query vector, the key vector and the value vector through a linear layer: （2） （3） （4） wherein represents a weight, takes a value ; the cross-attention mechanism is an asymmetric combination of Q, K, V; then the attention score is calculated, and the score determines the importance of the element: （5） is a scaled score value after dot product; Try to find the optimized mapping matrix, i.e. the weight of linear layer, in the back propagation process, and calculate the attention weight: （6） where is the dimension of the key vector, is a scalar used to avoid the dominant term when computing the function; finally the attention weight is multiplied by the value vector and the final weighted mapping is obtained: （7） Thus, the output vector O1 is obtained, and similarly, the output vector O2 is obtained by inputting the vector O1 into the cross attention module, and the new feature vector O is obtained by connecting O1 and O2. Thus, the output vector O1 is obtained, and similarly, the output vector O2 is obtained by inputting the vector O1 into the cross attention module, and the new feature vector O is obtained by connecting O1 and The cross attention mechanism can effectively fuse sensor features and image features; Step 5, classify according to the feature information through the Softmax classifier, and select the highest confidence rate as the classification result; Step 6, define the loss function of model training; Step 7, train the model; Step 8, use the trained model to detect the miscellaneous bacteria of edible fungus stick.

2. The edible mushroom weed detection method based on attention multi-source feature fusion according to claim 1, characterized in that, Step 1 specifically includes: placing the edible fungus stick in the multifunctional intelligent box for growth, and the volatile substances generated by the stick will fill the whole space; the CH4, H2S and NH3 sensors continuously collect the gas concentration values; the gas sensor converts the response into an electrical signal, and then converts the electrical signal into a gas concentration value; the temperature sensor obtains the temperature value in the space, the humidity sensor obtains the humidity value in the space, and the image data of the stick is collected by the shooting device at the same time period; then the sensor data and the image data are pretreated respectively; the noise in the image data is removed artificially, including faded, blurred and repeated images, and more new image data is added from the existing image data through scaling, rotation, moving, random rotation and cropping data enhancement methods; the gas sensor collects the concentration values in the continuous time period, and takes the average value as the final value; due to the heterogeneity of the sensor data, normalization is performed; （1）。 3. The edible mushroom weed detection method based on attention multi-source feature fusion according to claim 1, characterized in that, Step 2 specifically includes: constructing a backbone network ResNet50, which is divided into 5 layers; the first layer C1 is composed of a convolution with a kernel size of 7x7 and a step of 2, a BN normalization layer and a ReLU activation function; C2 is composed of a pooling layer with a kernel size of 3x3 and 3 residual blocks, C3 to C5 are composed of 4, 6 and 3 residual blocks respectively; the residual block is composed of a kernel size of 1x1 convolution, a kernel size of 3x3 convolution and a kernel size of 1x1 convolution; an average pooling layer and a fully connected layer are constructed after the 5 layers, and the visual features are extracted by using the feature extraction capability of the residual structure.

4. The edible mushroom weed detection method based on attention multi-source feature fusion according to claim 1, characterized in that, Step 3 specifically includes: constructing a multi-layer perception MLP, which is divided into an input layer, a hidden layer and an output layer; the first layer C1 is composed of 5 neurons, corresponding to the values of 3 gas sensors, temperature sensor and humidity sensor, the second layer hidden layer C2 is composed of multiple network layers with different number of neurons, and the third layer C3 is composed of 64 neurons, using Sigmoid activation function; based on the effective processing of continuous number by MLP, the sensor features are extracted.

5. The edible mushroom weed detection method based on attention multi-source feature fusion according to claim 1, characterized in that, The Softmax classifier described in step 5 is constructed as follows: O is input to a fully connected layer; since the proposed model is a multi-classification problem, a Softmax activation function is used for classification; the Softmax activation function generates a vector with 6 values, including 5 edible mushroom species and a healthy state; each value in the vector is the confidence of each classification, and the one with the highest confidence is selected as the classification result.

6. The edible mushroom weed detection method based on attention multi-source feature fusion according to claim 1, characterized in that, Step 6 specifically includes: the loss function is used to estimate the prediction value of the model The degree of inconsistency between the true value and the predicted value is a non-negative real value function, denoted by ; the smaller the loss function, the better the robustness of the model; let be the feature space, be the label space; the classifier is a function that maps the input feature space to the label space ; cross-entropy loss function in multi-classification: （8） wherein is a parameter set of the classifier, corresponding sample the first element of the one-hot encoded label.

7. The edible mushroom weed detection method based on attention multi-source feature fusion according to claim 1, characterized in that, Step 7 specifically includes: in order to speed up the training speed of the model and improve the accuracy of the model, based on the transfer learning method, the pre-trained weights of the model are loaded and fine-tuned; the input of the model includes the original image, the sensor reading values of CH4, H2S, NH3, and the temperature and humidity values; the image data and sensor data are normalized before inputting into the network; Adam is used for optimization, and the update of the weight value is adaptively adjusted.

8. The edible mushroom weed detection method based on attention multi-source feature fusion according to claim 1, characterized in that, Step 8 specifically includes: the multifunctional intelligent box collects image data and sensor data, and performs preprocessing operations before inputting into the model; the trained model is used to detect the input data, and the three features of chemical sensors, physical sensors and images are combined to detect the miscellaneous bacteria of the edible mushroom stick.

9. The edible mushroom weed detection method based on attention multi-source feature fusion according to claim 1, characterized in that, In the step 1, the intelligent box simultaneously collects temperature, humidity, CH4, H2S, NH3 and image data, compared with single information of single source data, the common representation of multi-data source can effectively improve the accuracy and robustness of the model.

Images