Microorganism image recognition method and system based on double-input convolutional neural network

Abstract

The invention discloses a microbial image recognition method and system based on a dual-input convolutional neural network, and relates to the technical field of microbial image recognition, and the method comprises the steps: 1, collecting a microbial sample image, and carrying out the preprocessing of the sample image; 2, constructing a data set according to the preprocessed sample images; 3, inputting the data set into a backbone network of the double-input convolutional neural network for feature extraction, and forming an independent feature vector; 4, fusing the feature vectors, and introducing an attention mechanism module for optimization to obtain an optimized fused feature vector; and 5, performing classification decision according to the fusion feature vector, and outputting a microbial image recognition classification result. According to the invention, two types of images are used as independent inputs for deep fusion, and morphological features of different levels are complementarily extracted; and the distinguishing capability can be obviously improved in distinguishing morphological similar species (such as penicillium and aspergillus).

Description

Technical Field

[0001] The present invention relates to the technical field of microbial image recognition, and more particularly to a microbial image recognition method and system based on a dual-input convolutional neural network. Background Art

[0002] Microorganisms are widespread in nature, some of which are pathogenic and can cause serious infections in immunocompromised patients. Their morphological classification primarily relies on microscopic examination and bacterial colony culture observation. However, due to the complex morphologies and subtle differences between bacterial species, traditional morphological analysis is prone to misidentification and omission.

[0003] The morphological classification of microorganisms mainly relies on manual judgment of microscopic features, which poses many technical problems: manual identification relies on the experience of senior inspectors, has a long training cycle, is highly subjective, and is difficult to replicate at the grassroots level; existing image recognition models are mostly single-input and cannot utilize the complementary information between macroscopic colonies and microscope images, affecting accuracy; fungal culture and image acquisition are costly, training data is scarce and the categories are unbalanced, and existing solutions are insufficient to cope with this; some models have complex structures and poor interpretability.

[0004] In recent years, deep learning has made significant breakthroughs in morphological image recognition, but challenges still exist in microbial image recognition: microscope and colony image modalities are different, and most existing models only perform single-path processing and cannot fuse across modalities; data collection and annotation costs are high, and the scarcity of rare samples leads to category imbalance, and classifiers ignore minority classes; most models lack attention mechanisms, making it difficult to focus on key morphological areas.

[0005] Therefore, designing a microbial image recognition method and system based on a dual-input convolutional neural network for automatic image recognition of microorganisms and significantly improving the recognition accuracy is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a microbial image recognition method and system based on a dual-input convolutional neural network, which can automatically recognize microbial images and significantly improve the recognition accuracy.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a microbial image recognition method based on a dual-input convolutional neural network, comprising: Step 1: Collect microbial sample images and preprocess the sample images; Step 2: Construct a dataset based on the preprocessed sample images; Step 3: Input the dataset into the backbone network of the dual-input convolutional neural network for feature extraction and form an independent feature vector; Step 4: Fuse the feature vectors and introduce the attention mechanism module for optimization to obtain the optimized fused feature vector; Step 5: Make classification decisions based on the fused feature vector and output the microbial image recognition classification results.

[0008] Preferably, the microbial sample image in step 1 includes a microscopic microscope image and a macroscopic colony image, and the process of preprocessing the sample image includes adjusting the microscopic microscope image and the macroscopic colony image to a uniform resolution, and performing standardization and data enhancement to obtain the preprocessed microscopic microscope image and macroscopic colony image.

[0009] Preferably, constructing the data set includes: performing category balancing processing on the preprocessed microscopic microscope images and macroscopic colony images to obtain balanced dual-path sample pairs, each sample pair containing a microscopic microscope image and a macroscopic colony image, and dividing the balanced dual-path sample pairs into a training set, a validation set, and a test set.

[0010] Preferably, step 3 includes: inputting the dual-path sample pair into a dual-input convolutional neural network structure, the dual-input convolutional neural network structure includes two input channels, and performing feature extraction on the microscopic microscope image and macroscopic colony image of the dual-path sample pair through the backbone feature extraction network to obtain two independent feature vectors.

[0011] Preferably, the step 4 includes: concatenating two independent feature vectors into a fused vector, and reconstructing the obtained fused vector into a four-dimensional tensor; The attention mechanism module is introduced to optimize the four-dimensional tensor. The specific process includes: inputting the four-dimensional tensor into the channel attention module, performing global average pooling and global maximum pooling respectively to obtain the description vectors of the two channels; inputting the two channel description vectors into a shared fully connected network to extract channel weight information; Apply the Sigmoid activation function to the channel weight information to generate the channel attention weight, and multiply it with the four-dimensional tensor to complete the attention weighting of the channel dimension and output the feature map; The feature map is input into the spatial attention module, and maximum pooling and average pooling are performed on the channel dimension to obtain two single-channel feature maps; The two single-channel feature maps are concatenated into a 2-channel feature map along the channel dimension, and then a single-channel attention map is generated through convolution operation; After the single-channel attention map is normalized by the Sigmoid function, it is multiplied with the input feature map to complete the attention weighting of the spatial dimension and obtain the optimized fusion feature vector.

[0012] Preferably, the classification decision process includes: inputting the optimized fusion feature vector into a fully connected layer, and outputting it to a softmax classification layer after passing through a Dropout and activation layer to generate a predicted probability distribution of the microbial category.

[0013] Preferably, a microbial image recognition system based on a dual-input convolutional neural network comprises: Image acquisition module: collects microbial sample images and preprocesses the sample images; Dataset construction module: constructs a dataset based on preprocessed sample images; Feature extraction module: Input the data set into the backbone network of the dual-input convolutional neural network for feature extraction and form an independent feature vector; Feature fusion module: fuses the feature vectors and introduces the attention mechanism module for optimization to obtain the optimized fused feature vector; Classification module: Make classification decisions based on the fused feature vector and output the microbial image recognition classification results.

[0014] From the above technical solution, it can be seen that compared with the existing technology, the present invention discloses a microbial image recognition method and system based on a dual-input convolutional neural network, which has the following significant advantages: 1. Multimodal image fusion recognition to improve accuracy and robustness: Most existing technologies only process colony images or microscope images. The present invention uses the two types of images as independent inputs for deep fusion, and complementarily extracts morphological features at different levels. It can significantly improve the discrimination ability in distinguishing morphologically similar species (such as Penicillium and Aspergillus).

[0015] 2. Introducing the attention mechanism to improve the model's feature interpretation ability: The present invention innovatively introduces the CBAM module into the fused feature tensor, enabling the model to automatically focus on key morphological areas (such as spore structure, branch points, spore stalk direction, etc.), improving the model's ability to "focus on the key points"; it is significantly better than the existing coarse-grained strategy of "global average pooling" direct classification.

[0016] 3. Engineering mechanisms to address small sample and class imbalance issues: The system of the present invention designs a default colony map completion mechanism to solve the problem of inconsistent image acquisition volume of different modalities; the sample alignment mechanism ensures that each training sample has a microscope + colony map pair, greatly enhancing the stability and learnability of the model.

[0017] 4. Integrated training, evaluation, and deployment, suitable for grassroots or scientific research scenarios: Most existing research uses "experimental" models, lacking complete logging, model export, and evaluation report mechanisms. This invention provides a closed-loop process from training to model preservation, confusion matrix output, and classification report generation, making it more suitable for direct use or deployment in scientific research institutions.

[0018] 5. Multi-backbone network compatible design to adapt to different computing resource scenarios: The model architecture of the present invention can flexibly switch backbone networks, such as lightweight MobileNetV2 or high-performance DenseNet; it supports running in multiple environments such as Colab and local GPU, facilitating promotion and implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0020] Figure 1 The present invention provides a flow chart of the method. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, the embodiment of the present invention discloses a microbial image recognition method based on a dual-input convolutional neural network, comprising: Step 1: Collect microbial sample images and preprocess the sample images; Step 2: Construct a dataset based on the preprocessed sample images; Step 3: Input the dataset into the backbone network of the dual-input convolutional neural network for feature extraction and form an independent feature vector; Step 4: Fuse the feature vectors and introduce the attention mechanism module for optimization to obtain the optimized fused feature vector; Step 5: Make classification decisions based on the fused feature vector and output the microbial image recognition classification results.

[0023] Specifically, the microbial sample image in step 1 includes a microscopic microscope image and a macroscopic colony image. The process of preprocessing the sample image includes adjusting the microscopic microscope image and the macroscopic colony image to a uniform resolution, and performing standardization and data enhancement to obtain the preprocessed microscopic microscope image and the macroscopic colony image.

[0024] In a specific embodiment of the present invention, two types of images are collected for each microbial sample: one type is a high-magnification microscope image under a microscope (microscopic microscope image), and the other type is a macroscopic colony image in a culture dish (macroscopic colony image); all images are uniformly adjusted to a fixed resolution (such as 160×160) and standardized; and the data augmentation module of TensorFlow is used to perform image perturbations (rotation, flipping, brightness, contrast, adding noise, etc.) to enhance the generalization ability of the model.

[0025] Specifically, constructing the dataset includes: performing category balancing processing on the preprocessed microscopic microscope images and macroscopic colony images to obtain balanced dual-path sample pairs, each sample pair containing a microscopic microscope image and a macroscopic colony image, and dividing the balanced dual-path sample pairs into training set, validation set and test set.

[0026] In a specific embodiment of the present invention, for each classification class, if its colony images are insufficient, the system automatically supplements the preset “default reference images” to match the number of colony images; After achieving equilibrium, the dual-path sample pair: Each sample contains a microscope image and a colony image, each corresponding to a same label. The label can be understood as the name of the strain. In this example, a two-digit number is used instead to represent a one-to-one correspondence between the strain name and the number. All samples are divided into training set, validation set and test set. Images of each category are selected in proportion and included in the training set, validation set and test set to keep the category distribution consistent.

[0027] Specifically, step 3 includes: inputting the dual-path sample pair into a dual-input convolutional neural network structure, the dual-input convolutional neural network structure includes two input channels, and performing feature extraction on the microscopic microscope image and macroscopic colony image of the dual-path sample pair through the backbone feature extraction network to obtain two independent feature vectors.

[0028] In a specific embodiment of the present invention, a dual-input convolutional neural network structure uses two independent input channels to receive microscope images and colony images respectively; each input channel uses the same or different backbone feature extraction networks (such as EfficientNet, ResNet, etc.); and supports automatic alignment and balancing processing mechanisms for input data.

[0029] A dual-input convolutional neural network (CNN) is a deep learning architecture that can simultaneously process two different modalities of image or feature input. The architecture can be divided into the following main parts as shown in Table 1: Table 1 Dual-input convolutional neural network module table

[0030] Advantages include: modal complementarity, combining macro- and microstructure, texture, and other information. Easily scalable to support three-input, multi-input, and even multi-modal inputs such as image and text.

[0031] The balance processing mechanism is that when the model has two inputs (such as colony image + micrograph), it is necessary to ensure that the two input images of the same strain are correctly paired and the corresponding labels are sent at the same time. The effect is shown in Table 2 Table 2 Balance processing mechanism usage

[0032] The specific implementation method is that for samples with missing colony photos, white noise images are used instead in this embodiment to ensure that the lengths of the two-way input data are consistent.

[0033] Specifically, the step 4 includes: concatenating two independent feature vectors into a fusion vector, and reconstructing the obtained fusion vector into a four-dimensional tensor; An attention mechanism module (CBAM module) is introduced to optimize the four-dimensional tensor. The specific process includes: inputting the four-dimensional tensor into the channel attention module, performing global average pooling and global maximum pooling respectively to obtain the description vectors of the two channels; inputting the two channel description vectors into a shared fully connected network to extract channel weight information; Apply the Sigmoid activation function to the channel weight information to generate the channel attention weight, and multiply it with the four-dimensional tensor to complete the attention weighting of the channel dimension and output the feature map; The feature map is input into the spatial attention module, and maximum pooling and average pooling are performed on the channel dimension to obtain two single-channel feature maps; The two single-channel feature maps are concatenated into a 2-channel feature map along the channel dimension, and then a single-channel attention map is generated through convolution operation; After the single-channel attention map is normalized by the Sigmoid function, it is multiplied with the input feature map to complete the attention weighting of the spatial dimension and obtain the optimized fusion feature vector.

[0034] In a specific embodiment of the present invention, the features extracted from the two channels are concatenated into a fused vector; reconstructed into a four-dimensional tensor and then fed into the Convolutional Block Attention Module (CBAM); first, channel attention is calculated to identify which channels are more important; then spatial attention is calculated to identify which areas in the image are more critical; and finally, an optimized fused feature vector is obtained.

[0035] Specifically, the classification decision process includes: inputting the optimized fusion feature vector into the fully connected layer, passing through the Dropout and activation layers, and outputting it to the softmax classification layer to generate a predicted probability distribution of the microbial category.

[0036] In a specific embodiment of the present invention, the fusion vector output by the CBAM module is sent to the fully connected layer; after passing through the Dropout and activation layers, it is output to the softmax classification layer to generate a predicted probability distribution for each category.

[0037] Specific methods: Mathematical definition of the Softmax classification layer: Assume that the output of the last layer is a vector:

[0038] where z i is the “logit” (unnormalized score) of the neural network for the ith category, and K is the total number of categories.

[0039] The Softmax function is defined as:

[0040] Output , is the probability of belonging to the i-th category; The sum of the probabilities of all classes is 1: .

[0041] 1. Dropout layer (effective during training) Training phase: randomly block some neurons with a certain probability (set weights to 0) to prevent overfitting; Inference phase: Dropout is not activated, but all connections are retained.

[0042] During inference, units are not randomly dropped, but all trained weights are used for standard forward propagation.

[0043] 2. Activation layer (such as ReLU) Add nonlinearity to help the model learn complex mappings; The activated features (non-negative or other transformations) enter the fully connected layer to produce logit ziz_izi.

[0044] 3.Fully-connected (FC) layer Usually the main transformation of the last layer, outputting the same number of logits as the number of categories; Example: If there are 10 categories, the FC layer output vector length is 10.

[0045] 4. Softmax layer (core during inference) Convert logits into an interpretable probability distribution; The model predicts which category has the highest probability.

[0046] In another specific embodiment of the present invention, a model training and evaluation process of a dual-input convolutional neural network is also included: Use a training strategy with early stopping, learning rate decay, etc. Leverage GPU training acceleration and support both Colab and local deployment; The model training process saves training logs, model weights, and intermediate image results; Finally, a classification evaluation is performed on the test set, and a classification report and confusion matrix image are output.

[0047] In another specific embodiment of the present invention, it also includes visualization output and model export: Automatically generate confusion matrix heatmaps and classification metric text reports (such as precision, recall, F1-score); The model is saved in .keras format for easy deployment to local systems or web services; Output prediction results in JSON or image format and connect to the database.

[0048] The specific embodiments provided by the present invention are as follows: The overall goal is to implement an end-to-end training process for a dual-input image classification model that can automatically complete data pairing, category balancing, model training, validation and evaluation, and provide a stable and available model deployment foundation for routine morphological identification and scientific research scenarios.

[0049] The integrated training process includes: 1. Data preparation and automatic matching Perform number verification and path extraction on image data from different modalities (such as colony photos and microscope images); Construct each pair of input image samples to form a unified data structure (image A, image B, category label); Ensure one-to-one correspondence between sample pairs to avoid information leakage or label mismatch.

[0050] 2. Category Balance and Enhancement Processing Count the number of samples of all categories and identify the minority and majority classes; Oversampling of minority class samples using image enhancement methods (such as rotation, mirroring, and scaling); Construct a dataset with a roughly balanced number of category samples for model training to ensure that all types of features are effectively learned during the training process.

[0051] 3. Data partitioning and standardization Divide the balanced dataset into training set, validation set and test set (e.g. 7:2:1 ratio); All images are uniformly preprocessed, including resizing, pixel normalization, and format standardization, to ensure input compatibility and computational efficiency.

[0052] 4. Model Building and Training Construct a dual-input convolutional neural network structure to process two types of image inputs respectively; Introducing an attention mechanism (such as CBAM) in the fusion layer to enhance important channel and key area features; Configure training parameters (optimizer, loss function, learning rate decay strategy, early stopping, etc.); Start the training process and continuously record the training loss, accuracy, and validation set performance.

[0053] 5. Automatic monitoring and model saving Real-time monitoring of performance indicators (such as loss, accuracy, AUC, etc.) on the validation set during training; If the verification performance does not improve for more than the preset number of rounds, the training will be automatically stopped; Automatically save the model weights with the best validation performance for subsequent inference or testing.

[0054] The model validation process includes: 1. Performance evaluation index calculation Evaluate the final model using the validation set / test set; Automatically output key performance indicators, including accuracy, recall, specificity, F1-score, ROC curve, AUC value, etc. Supports multi-category confusion matrix output to identify the bias or weaknesses of the model on various types of samples.

[0055] 2. Error Analysis and Visualization Provides automatic error sample analysis and outputs pairs of sample prediction errors and their true labels; Visualize attention maps (via Grad-CAM, CBAM output, etc.) to help researchers understand the model's attention areas; Providing high and low confidence sample sorting helps to correct annotations and identify model improvement directions.

[0056] 3. Model export and deployment preparation Unified packaging of optimal model structure and weights, supporting multiple deployment formats (such as TensorFlow SavedModel, ONNX, and PyTorch models); Automatically generate model documentation and input and output specifications to facilitate deployment to local research platforms or web front-end interfaces; An API calling interface is reserved to support subsequent integration with data management systems and image databases.

[0057] The process summary and local scientific research adaptation advantages are shown in Table 3.

[0058] Table 3 Stage and local scientific research adaptation advantages

[0059] Take a bacterial species identification task as an example (such as identifying Penicillium): A total of 40 microscope images and 20 colony images were collected; The system automatically replicates the colony images to 40 images in a balanced manner and pairs them with the microscope images one by one; Input to a dual-input model configured as a ResNet50 backbone; After 30 rounds of training, the accuracy rate reached over 96%; The system outputs the recognition results in the form of a confusion matrix, showing the value with the highest probability and other possible identifications.

[0060] Specifically, a microbial image recognition system based on a dual-input convolutional neural network includes: Image acquisition module: collects microbial sample images and preprocesses the sample images; Dataset construction module: constructs a dataset based on preprocessed sample images; Feature extraction module: Input the data set into the backbone network of the dual-input convolutional neural network for feature extraction and form an independent feature vector; Feature fusion module: fuses the feature vectors and introduces the attention mechanism module for optimization to obtain the optimized fused feature vector; Classification module: Make classification decisions based on the fused feature vector and output the microbial image recognition classification results.

[0061] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0062] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A microbial image recognition method based on a dual-input convolutional neural network, characterized in that: include: Step 1: Collect microbial sample images and preprocess the sample images; Step 2: Construct a dataset based on the preprocessed sample images; Step 3: Input the dataset into the backbone network of the dual-input convolutional neural network for feature extraction and form an independent feature vector; Step 4: Fuse the feature vectors and introduce the attention mechanism module for optimization to obtain the optimized fused feature vector; Step 5: Make classification decisions based on the fused feature vector and output the microbial image recognition classification results.

2. A microbial image recognition method based on a dual-input convolutional neural network according to claim 1, characterized in that: The microbial sample image in step 1 includes a microscopic image and a macroscopic colony image. The process of preprocessing the sample image includes adjusting the microscopic image and the macroscopic colony image to a uniform resolution, and performing standardization and data enhancement to obtain the preprocessed microscopic image and the macroscopic colony image.

3. A microbial image recognition method based on a dual-input convolutional neural network according to claim 2, characterized in that: Constructing the dataset includes: performing category balancing on the preprocessed microscopic microscope images and macroscopic colony images to obtain balanced dual-path sample pairs, each sample pair containing a microscopic microscope image and a macroscopic colony image, and dividing the balanced dual-path sample pairs into training set, validation set and test set.

4. A microbial image recognition method based on a dual-input convolutional neural network according to claim 3, characterized in that: The step 3 includes: inputting the dual-path sample pair into a dual-input convolutional neural network structure, the dual-input convolutional neural network structure includes two input channels, and extracting features from the microscopic microscope image and macroscopic colony image of the dual-path sample pair through a backbone feature extraction network to obtain two independent feature vectors.

5. A microbial image recognition method based on a dual-input convolutional neural network according to claim 4, characterized in that: The step 4 includes: concatenating two independent feature vectors into a fused vector, and reconstructing the obtained fused vector into a four-dimensional tensor; The attention mechanism module is introduced to optimize the four-dimensional tensor. The specific process includes: inputting the four-dimensional tensor into the channel attention module, performing global average pooling and global maximum pooling respectively to obtain the description vectors of the two channels; inputting the two channel description vectors into a shared fully connected network to extract channel weight information; Apply the Sigmoid activation function to the channel weight information to generate the channel attention weight, and multiply it with the four-dimensional tensor to complete the attention weighting of the channel dimension and output the feature map; The feature map is input into the spatial attention module, and maximum pooling and average pooling are performed on the channel dimension to obtain two single-channel feature maps; The two single-channel feature maps are concatenated into a 2-channel feature map along the channel dimension, and then a single-channel attention map is generated through convolution operation; After the single-channel attention map is normalized by the Sigmoid function, it is multiplied with the input feature map to complete the attention weighting of the spatial dimension and obtain the optimized fusion feature vector.

6. A microbial image recognition method based on a dual-input convolutional neural network according to claim 5, characterized in that: The classification decision process includes: inputting the optimized fusion feature vector into a fully connected layer, passing through a dropout and activation layer, and outputting it to a softmax classification layer to generate a predicted probability distribution of the microbial category.

7. A microbial image recognition system based on a dual-input convolutional neural network, characterized in that: include: Image acquisition module: collects microbial sample images and preprocesses the sample images; Dataset construction module: constructs a dataset based on preprocessed sample images; Feature extraction module: Input the data set into the backbone network of the dual-input convolutional neural network for feature extraction and form an independent feature vector; Feature fusion module: fuses the feature vectors and introduces the attention mechanism module for optimization to obtain the optimized fused feature vector; Classification module: Make classification decisions based on the fused feature vector and output the microbial image recognition classification results.

Images