A method for segmenting microbial colonies based on the attention mechanism

Abstract

The present invention discloses a method for segmenting microbial colonies based on an attention mechanism. 1: After culturing the colonies, capture the colony images to obtain a colony dataset D , and divide it into a training set, a validation set, and a test set; 2: Use data augmentation methods to process the pre-input images; 3: Design an attention mechanism module CSA and embed it into a high-resolution convolutional neural network to form a deep convolutional neural network model; 4: Initialize the parameters of the convolutional neural network by loading the pre-trained network model parameters, and each layer of neurons updates the parameters in the network structure according to the error; 5: After each training, use the validation set to verify the network model and save the model parameters with the highest accuracy as the optimal model; 6: Test the optimal model on the test set and output the colony semantic segmentation result map. The colony boundaries of the method of the present invention are closer to the real situation and the accuracy is also higher.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning of attention mechanism and convolutional neural network, and particularly relates to a method for segmenting microbial colonies based on attention mechanism. Background Art

[0002] In recent years, food safety problems have occurred frequently, and all food safety incidents are related to colonies. How to determine the types of colonies in various special occasions has also become a common concern. To a certain extent, the number of colonies can directly or indirectly reflect the quality of food safety and environmental hygiene. In recent years, as China has strengthened the supervision in the fields of food safety and environmental protection, the demand for work such as colony observation and analysis has been increasing. However, the traditional method of dealing with colonies relying on manual labor is not only cumbersome and time-consuming, but also prone to subjective errors.

[0003] A convolutional neural network is composed of multiple convolutional layers and a fully connected layer at the top (corresponding to a classical neural network), and also includes an activation layer and a pooling layer. Compared with other deep learning structures, the convolutional neural network can give better results in image segmentation and recognition.

[0004] The attention mechanism draws on human visual attention. When humans observe external things or an image, there is a selectivity, that is, they focus on the part of the region of interest and ignore other irrelevant details, which is the so-called attention focus. This mechanism can help humans quickly and accurately screen out the interesting and valuable information from a large amount of redundant information, thus greatly improving the efficiency of obtaining information. Applying the attention mechanism to the neural network can capture the long-range dependencies in the feature map and more accurately locate the position and shape of a certain type of colony.

[0005] The existing methods for dealing with colonies either have better classification effects on colonies or are more ideal for segmenting the boundary shapes and details of colonies. However, the problem is that the existing methods cannot achieve high-precision segmentation of complex colony boundaries while obtaining good colony classification results. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for segmenting microbial colonies based on attention mechanism to solve the technical problem that the existing methods cannot achieve high-precision segmentation of complex colony boundaries while obtaining good colony classification results.

[0007] To solve the above technical problem, the specific technical solution of a method for segmenting microbial colonies based on attention mechanism of the present invention is as follows:

[0008] A method for segmenting microbial colonies based on an attention mechanism, comprising the following steps:

[0009] Step 1: After culturing the colonies, use an industrial camera to capture colony images, obtain the colony dataset D, and then divide all n colony image data in D into a training set, a validation set, and a test set according to a ratio of 40:7:7;

[0010] Step 2: Use data augmentation methods to process the pre-input images;

[0011] Step 3: Design an attention mechanism module CSA and embed it into a high-resolution convolutional neural network to form a deep convolutional neural network model;

[0012] Step 4: Initialize the parameters of the convolutional neural network by loading pre-trained network model parameters. When the input data passes through the forward propagation in the convolutional neural network, an expected output will be obtained. If this expected output is different from the actual class label of the data, the error will be propagated layer by layer in the reverse direction to the input layer, and each layer of neurons will update the parameters in the network structure according to this error;

[0013] Step 5: After each training ends, use the validation set to verify the network model, and save the model parameters with the highest accuracy as the optimal model;

[0014] Step 6: Test the optimal model on the test set and output the colony semantic segmentation result map. Different gray-level regions in the result map represent different types of colonies respectively.

[0015] Further, the step 1 includes the data input of the deep convolutional neural network. Each input data includes the original image and the marked image, where the marked image marks different microbial colony types with different colors.

[0016] Further, the data augmentation methods in the step 2 include random horizontal flipping, random vertical flipping, 360-degree random rotation, and random cropping.

[0017] Further, the attention mechanism module CSA in the step 3 first performs horizontal average pooling and vertical average pooling on the feature image with the input dimension of C×H×W respectively to obtain intermediate feature maps with dimensions of C×H×1 and C×1×W respectively; use c to represent the c-th channel, h to represent the h-th row in the feature map, and w to represent the w-th column in the feature map. The calculation formulas for this process are respectively:

[0018] Horizontal average pooling:

[0019] Vertical average pooling:

[0020] After that, the obtained intermediate feature maps are processed using three different 1×1 convolutional kernels respectively; for row pooling, the feature maps obtained from the three 1×1 convolutions are Q h , K h and V h ; correspondingly, for column pooling, the obtained ones are Q w , K w and V w ; among them, the first two 1×1 convolutional kernels will change the dimension of the intermediate feature map to C'×H×1 or C'×1×W, where C' is the number of channels after dimension reduction; while the third 1×1 convolutional kernel will not change its dimension, which remains C×H×1 or C×1×W; immediately afterwards, the feature maps processed by the first two 1×1 convolutional kernels are subjected to matrix multiplication, and this process will perform dimension transformation on the feature maps, such that the form of matrix multiplication is: (1×H×C')*(1×C'×H) or (1×W×C')*(1×C'×W), so the dimension of the obtained attention map is 1×H×H or 1×W×W; then, after processing it with the Softmax function, matrix multiplication is performed with the result of the third 1×1 convolution. Similarly, it is necessary to transform the dimension of the 1*1 convolution result to 1×C×H or 1×C×W before calculation. The specific calculation formula can be expressed as:

[0021] Horizontal direction:

[0022] Vertical direction:

[0023] In the above formula, Q h i , Q w i represent the i-th position of the feature maps Q h and Q w ; K h j , K w j represent the j-th position of the feature maps K h and K w ; while s ij and t ij represent the influence of the i-th position on the j-th position in the attention map. The larger s ij or t ij , the higher the correlation between the two positions; the dimension of the obtained attention feature map is C×H×1 and C×1×W; then, the results obtained in the horizontal and vertical directions are both expanded to C×H×W, and finally added to the original input feature map pixel by pixel to output a feature map processed by one CSA.

[0024] Further, after the attention mechanism module CSA is embedded into each convolutional module of the main network in step 4, forward propagation is performed to calculate the network parameters, and the obtained output vector O i is passed through the Softmax function to obtain the correlation weight vector W of a certain pixel with all other pixels i ; then, the product of the weight value in the vector W i and the pixel at the corresponding position is multiplied and accumulated, which is the final result y predicted by the network for each pixel i ; the obtained classification result y i and the current correct label value y' i are respectively used as the two inputs of the CrossEntropy loss function to calculate the loss value; the error signal is passed to the output of each layer, and then the derivative of the function of each layer with respect to the parameters can be used to obtain the gradient of the parameters; then, the network parameters affecting model training and model output are updated and calculated through the Stochastic Gradient Descent (SGD) optimizer to approximate or reach the optimal value, so as to minimize the loss function. Further, in step 5: after each training ends, the Intersection over Union (IOU), the most commonly used evaluation criterion in image segmentation, is used to verify the effectiveness of the model proposed in the present invention on the validation set. The specific calculation formula is: IOU = (y' ∩ y) / (y' ∪ y), where y' is the true value and y is the predicted value; at the same time, the model parameters with the largest IOU are saved, and training stops when the number of training times reaches a certain number; finally, the model parameters with the largest IOU are loaded to obtain the final model of the trained convolutional neural network. Further, in step 6, an untreated colony image is input into the convolutional neural network. First, two convolutional layers with a convolutional kernel size of 3×3 and a convolutional stride of 2 are used to downsample the input image to 1 / 4 of the original size, and batch normalization (BN) is used after each convolutional layer to reduce errors; then, the processing result X of the convolutional layer is input into the attention mechanism module CSA, so that the further processed feature map captures long-range dependencies or dense context information while retaining the accurate position information of the colony; then, the feature map processed by CSA is pixel-wise added to the input map X to obtain the final result X' processed by the attention mechanism module.

[0025] Further, after being processed by the attention mechanism module multiple times, the convolutional neural network further includes a Segmentation layer. First, a convolutional layer with a convolutional kernel size of 1×1 and a stride of 1 is used to process the previously obtained feature map, which keeps the channel dimension of the feature map unchanged; then, batch normalization and the ReLU activation function are used to ensure the accuracy of the feature data; finally, a convolutional layer with a convolutional kernel size of 1×1 and a stride of 1 is used to change the channel dimension of the feature map to the number of colony types; thus, the final colony segmentation effect diagram can be obtained.

[0026] A method for segmenting microbial colonies based on an attention mechanism of the present invention has the following advantages:

[0027] (1) The present invention proposes a novel method for segmenting microbial colonies, proposes an attention mechanism module CSA, and for the first time adopts a method combining a convolutional neural network for semantic segmentation with an attention mechanism to segment and classify microbial colonies.

[0028] (2) The colony segmentation method proposed by the present invention produces colony boundaries that are closer to the real situation and have higher accuracy, which is conducive to subsequent analysis operations on the colonies and is of great significance in industries such as food safety, medical and health, and environmental protection. Description of the Drawings

[0029] Figure 1 is the overall flowchart of the present invention.

[0030] Figure 2 is the detailed processing flowchart of the attention mechanism module of the present invention.

[0031] Figure 3 is the structural diagram of the main body of the convolutional neural network of the present invention.

[0032] Figure 4 is the overall processing flowchart of the model of the present invention. Detailed Embodiments

[0033] In order to better understand the purpose, structure and function of the present invention, the following further describes in detail a method for segmenting microbial colonies based on an attention mechanism of the present invention with reference to the drawings.

[0034] As Figure 1 shown, a method for segmenting microbial colonies based on an attention mechanism of the present invention includes the following specific steps:

[0035] Step 1: After culturing the colonies, use an industrial camera to capture colony images to obtain a colony dataset D. Then, divide all n colony image data in D into a training set, a validation set, and a test set according to a ratio of 40:7:7. For example, among 216 colony images, the numbers of colony images in the training set, validation set, and test set are 160, 28, and 28 respectively. The data input to the deep convolutional neural network each time includes the original image and the marked image, where the marked image is marked with different colors for different microbial colony types.

[0036] Step 2: When training the network model, use data augmentation methods (random horizontal flipping, random vertical flipping, 360-degree random rotation, random cropping) to process the pre-input images on the training set. Among the 160 colony images in the training set, 80 were randomly horizontally flipped, 80 were randomly vertically flipped, and all 160 images utilized random rotation and random cropping. Finally, convert the image data into vector form;

[0037] Step 3: As Figure 2 shown, design an attention mechanism module CSA (Cross Strip Attention) and embed it into the high-resolution convolutional neural network to form the deep convolutional neural network model proposed in the present invention. Different from the commonly used global average pooling in traditional neural networks, the attention mechanism module CSA proposed in the present invention first performs horizontal average pooling and vertical average pooling on the feature image with an input dimension of C×H×W respectively to obtain intermediate feature maps with dimensions of C×H×1 and C×1×W respectively. Let c represent the c-th channel, h represent the h-th row in the feature map, and w represent the w-th column in the feature map. The calculation formulas for this process are respectively:

[0038] Horizontal average pooling:

[0039] Vertical average pooling:

[0040] After that, we process the obtained intermediate feature maps with three different 1×1 convolutional kernels respectively. For row pooling, the feature maps obtained by the three 1×1 convolutions are Q h , K h and V h ; correspondingly, for column pooling, the obtained ones are Q w , K w and V w . Among them, the first two 1×1 convolutional kernels will change the dimension of the intermediate feature map to C'×H×1 or C'×1×W, where C' is the number of channels after dimension reduction; while the third 1×1 convolutional kernel will not change its dimension, which is still C×H×1 or C×1×W. The purpose of this approach is to reduce the computational complexity. Immediately afterwards, perform matrix multiplication on the feature maps processed by the first two 1×1 convolutional kernels. This process will perform dimension transformation on the feature maps, making the form of matrix multiplication: (1×H×C')*(1×C'×H) or (1×W×C')*(1×C'×W). Therefore, the dimension of the obtained attention map is 1×H×H or 1×W×W. Then process it with the Softmax function and perform matrix multiplication with the result of the third 1×1 convolution. Similarly, it is necessary to transform the dimension of the 1*1 convolution result to 1×C×H or 1×C×W before calculation. The specific calculation formula can be expressed as:

[0041] Horizontal direction:

[0042] Vertical direction:

[0043] In the above formula, Qh i and Qw i represent the i-th position of the feature maps Q h and Q w Similarly, Kh j and Kw j represent the j-th position of the feature maps K h and K w ; while s ij and t ij represent the influence of the i-th position on the j-th position in the attention map. The larger s ij or t ij , the higher the correlation between the two positions. The dimensions of the obtained attention feature maps are C×H×1 and C×1×W. The traditional convolutional neural network method can only obtain the context information equal to the size of the convolutional kernel, and the amount of information that can be used for classification or segmentation is less, so the accuracy of classification or segmentation is not high. The attention map obtained through the CSA module designed by the present invention can obtain the context information of the original input feature map in the horizontal or vertical direction, increasing the amount of information that can be utilized in the processing process, which is beneficial to the segmentation of the overall shape and boundary details of the colonies and the classification of different colonies. Then, the results obtained in the horizontal and vertical directions are both expanded to C×H×W, and finally added to the original input feature map pixel by pixel to output a feature map processed by one CSA; due to the fusion of horizontal attention and vertical attention, this process further enhances the feature representation ability of the network proposed by the present invention, which is beneficial to the segmentation and classification of microbial colonies of the same type but in different positions.

[0044] Step 4: Initialize the parameters of the convolutional neural network designed by the present invention by loading the pre-trained network model parameters. When the input data passes through the forward propagation in the convolutional neural network, an expected output will be obtained. If this expected output is different from the actual class label of the data, the error will be propagated layer by layer in the reverse direction to the input layer, and each layer of neurons will update the parameters in the network structure according to this error. As Figure 3 shown, initialize the neural network by loading the parameters of the pre-trained model, and this network is the main body of the convolutional neural network designed by the present invention. After embedding the attention mechanism module CSA designed by the present invention into each convolutional module of the main network, perform forward propagation to calculate the network parameters, and obtain the output vector O i through the Softmax function to obtain the correlation weight vector W i of a certain pixel point with all other pixels; then follow the vector W iThe weighted value in it is multiplied by the pixel at the corresponding position and accumulated, which is the final result y predicted by the network for each pixel point. i ; The obtained classification result y i and the current correct label value y' i are respectively used as the two inputs of the CrossEntropy loss function to calculate the loss value. The error signal is passed to the output of each layer, and then through the derivative of each layer's function with respect to the parameters, the gradient of the parameters can be obtained. Then, the network parameters that affect model training and model output are updated and calculated through the Stochastic Gradient Descent (SGD) optimizer to make them approach or reach the optimal value, thereby minimizing the loss function.

[0045] Step 5: After each training is completed, use the most commonly used evaluation criterion in image segmentation, the Intersection over Union (IOU), to verify the effectiveness of the model proposed in the present invention on the validation set. Its specific calculation formula is: IOU = (y' ∩ y) / (y' ∪ y), where y' is the true value and y is the predicted value. At the same time, save the model parameters with the largest IOU. When the number of training times reaches a certain number, stop training. Finally, load the model parameters with the largest IOU to obtain the final model of the trained convolutional neural network.

[0046] Step 6: Use the deep convolutional neural network model proposed in the present invention to test the optimal model on the test set and output the semantic segmentation result map of microbial colonies. Different gray-level regions in the result map represent different types of colonies respectively. This process is as Figure 4 shown. Input an untreated colony image into the convolutional neural network proposed in the present invention. First, use two convolutional layers with a convolutional kernel size of 3×3 and a convolutional stride of 2 to downsample the input image to 1 / 4 of the original size, and use batch normalization (BN) after each convolutional layer to reduce errors. Then, input the processing result X of the convolutional layer into the attention mechanism module CSA proposed in the present invention, so that the further processed feature map can capture long-range dependencies or dense context information while retaining the precise position information of the colonies. Next, perform pixel-by-pixel camera on the feature map processed by CSA and the input map X to obtain the final result X' processed by the attention mechanism module. After being processed by the attention mechanism module multiple times, the network proposed in the present invention also has a Segmentation layer. First, use a convolutional layer with a convolutional kernel size of 1×1 and a stride of 1 to process the previously obtained feature map, which keeps the channel dimension of the feature map unchanged; then use batch normalization and the ReLU activation function to ensure the accuracy of the feature data; finally, change the channel dimension of the feature map to the number of colony types ( Figure 4 is 4 in it, including the background) through a convolutional layer with a convolutional kernel size of 1×1 and a stride of 1. Thus, the final colony segmentation effect diagram can be obtained, and different gray-level regions in the effect diagram represent different types of colonies respectively.

[0047] It will be understood that the present invention is described by way of some embodiments, and those skilled in the art will be aware that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Additionally, under the teaching of the present invention, these features and embodiments can be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the scope protected by the present invention.

Claims

1. A method for segmenting microbial colonies based on an attention mechanism, characterized in that, It includes the following steps: Step 1: After culturing the colonies, use an industrial camera to capture colony images to obtain a colony dataset D. Then, divide all n colony image data in D into a training set, a validation set, and a test set according to the ratio of 40:7:7; Step 2: Use data augmentation methods to process the pre-input images; Step 3: Design an attention mechanism module CSA and embed it into a high-resolution convolutional neural network to form a deep convolutional neural network model; In the attention mechanism module CSA of Step 3, first perform horizontal average pooling and vertical average pooling on the feature image with the input dimension of C×H×W respectively to obtain intermediate feature maps with dimensions of C×H×1 and C×1×W respectively. Let c represent the c-th channel, h represent the h-th row in the feature map, and w represent the w-th column in the feature map. The calculation formulas for this process are respectively: Horizontal average pooling: Vertical average pooling: After that, the obtained intermediate feature maps are processed with three different 1×1 convolutional kernels respectively; for row pooling, the feature maps obtained by the three 1×1 convolutions are Q h , K h and V h ; correspondingly, for column pooling, the obtained ones are Q w , K w and V w ; among which the first two 1×1 convolutional kernels will change the dimension of the intermediate feature map to C'×H×1 or C'×1×W, where C' is the number of channels after dimension reduction; The third 1×1 convolutional kernel will not change its dimension, which is still C×H×1 or C×1×W; immediately afterwards, perform matrix multiplication on the feature maps processed by the first two 1×1 convolutional kernels. This process will perform dimension transformation on the feature maps, making the form of matrix multiplication: (1×H×C')*(1×C'×H) or (1×W×C')*(1×C'×W). Therefore, the dimension of the obtained attention map is 1×H×H or 1×W×W; Then, after processing it with the Softmax function, perform matrix multiplication with the result of the third 1×1 convolution. Similarly, it is necessary to transform the dimension of the 1*1 convolution result to 1×C×H or 1×C ×W before calculation. The specific calculation formula can be expressed as: Horizontal direction: Vertical direction: In the above formula represents the feature map Q h and Q w at the i-th position, Denote the feature map K h and K w at the j-th position; while s ij and t ij represent the influence of the i-th position on the j-th position in the attention map. The larger s ij or t ij is, the higher the correlation between the two positions. The dimension of the obtained attention feature map is C×H×1 and C×1×W. Then, the results obtained in the horizontal and vertical directions are both expanded to C×H×W, and finally added to the original input feature map pixel by pixel to output a feature map processed by a CSA; Step 4: Initialize the parameters of the convolutional neural network by loading the pre-trained network model parameters. When the input data passes through the forward propagation in the convolutional neural network, an expected output will be obtained. If this expected output is different from the actual class label of the data, the error will be propagated layer by layer back to the input layer, and each neuron will update the parameters in the network structure according to this error; Step 5: After each training ends, use the validation set to verify the network model and save the model parameters with the highest accuracy as the optimal model; Step 6: Test the optimal model on the test set and output the colony semantic segmentation result map. In the result map, regions with different gray levels represent different types of colonies respectively.

2. The method for segmenting microbial colonies based on the attention mechanism according to claim 1, wherein Step 1 includes the data input of the deep convolutional neural network. Each input data contains the original image and the marked image, where the marked image marks different microbial colony types with different colors.

3. The method for segmenting microbial colonies based on the attention mechanism according to claim 1, wherein The data augmentation methods in Step 2 include random horizontal flipping, random vertical flipping, 360-degree random rotation, and random cropping.

4. The method for segmenting microbial colonies based on the attention mechanism according to claim 1, wherein After embedding the attention mechanism module CSA into each convolutional module of the main network in step 4, forward propagation is performed to calculate the network parameters, and the obtained output vector O i is passed through the Softmax function to obtain the correlation weight vector W of a certain pixel with all other pixels i ; then, the weights in the vector W i are multiplied by the pixels at the corresponding positions and accumulated, which is the final result y predicted by the network for each pixel i ; The obtained classification result y i and the current correct label value y' i are respectively used as the two inputs of the CrossEntropy loss function to calculate the loss value; the error signal is passed to the output of each layer, and then the derivative of the function of each layer with respect to the parameters can be used to obtain the gradient of the parameters; then the network parameters that affect model training and model output are updated and calculated through the Stochastic Gradient Descent (SGD) optimizer to approximate or reach the optimal value, thereby minimizing the loss function.

5. The method for segmenting microbial colonies based on an attention mechanism according to claim 1, wherein Step 5: After each training session, use the Intersection over Union (IOU), the most commonly used evaluation criterion in image segmentation, to verify the effectiveness of the model proposed by the present invention on the validation set. The specific calculation formula is: IOU = (y' ∩ y) / (y' ∪ y), where y' is the ground truth and y is the prediction. At the same time, save the model parameters with the largest IOU. When the number of training times reaches a certain number, stop training. Finally, load the model parameters with the largest IOU to obtain the final model of the trained convolutional neural network.

6. The method for segmenting microbial colonies based on an attention mechanism according to claim 1, characterized in that Step 6: Input an unprocessed colony image into the convolutional neural network. First, use two convolutional layers with a convolutional kernel size of 3×3 and a convolutional stride of 2 to downsample the input image to 1 / 4 of its original size, and use batch normalization (BN) after each convolutional layer to reduce errors. Then, input the processed result X of the convolutional layer into the Channel-Spatial Attention (CSA) module so that the further processed feature map can capture long-range dependencies or dense context information while retaining the precise location information of the colonies. Next, perform pixel-wise addition of the feature map processed by the CSA and the input map X to obtain the final result X' processed by the attention mechanism module.

7. The method for segmenting microbial colonies based on the attention mechanism according to claim 6, wherein After being processed by the attention mechanism module multiple times, the convolutional neural network also includes a Segmentation layer. First, use a convolutional layer with a convolutional kernel size of 1×1 and a stride of 1 to process the previously obtained feature map, which keeps the channel dimension of the feature map unchanged. Then, use batch normalization and the ReLU activation function to ensure the accuracy of the feature data. Finally, change the channel dimension of the feature map to the number of colony types through a convolutional layer with a convolutional kernel size of 1×1 and a stride of 1. Thus, the final colony segmentation effect diagram can be obtained.

Images