Multi-modal colony sample classification and identification method and system based on diffusion model

Abstract

The application discloses a kind of multi-modal bacterial colony sample autonomous learning classification identification method and system based on diffusion model, it is related to computer vision technical field, by being input into multi-modal bacterial colony sample classification model to multi-modal bacterial colony sample, obtain classification result;First, the application is by the feature fusion of multi-modal sample input, the correlation between different modalities is in-depth mined, and the semantic information of multi-modal is fully utilized.Secondly, the application combines diffusion model, and the multi-modal bacterial colony sample is classified after being denoised using diffusion model, which is conducive to eliminating inherent noise in multi-modal data samples and interference introduced by human operation errors, and is more conducive to learning.Finally, the application combines autonomous learning, and the model can complete the automatic labeling of new multi-modal samples, can select the most suitable multi-modal sample for training and learning, and continuously improve the robustness of multi-modal bacterial colony sample classification model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly to a multi-modal bacterial colony sample autonomous learning classification and identification method and system based on a diffusion model. BACKGROUND

[0002] At present, infectious diseases are a major category of clinical diseases affecting human health, including respiratory tract infections, digestive tract infections, abdominal infections, bloodstream infections, skin and soft tissue infections, and various types of infections, which are the causes of high morbidity and mortality in various countries. With the emergence of various new infectious diseases and the continuous rise of pathogenic bacteria drug resistance, microbiological laboratory detection plays an increasingly important role in the diagnosis and treatment of infectious diseases.

[0003] Traditional microbial identification mainly includes culture and isolation, biochemical identification, which is often based on traditional culture methods and morphology, biochemistry, physiology (phenotype) of microorganisms. With the progress of science and technology, the method of microbial identification is also more and more. But the traditional culture method is irreplaceable, and it is still the basis for drug sensitivity phenotype detection, molecular epidemiology detection, and genome analysis. In the clinical microbiology laboratory, the identification of microorganisms in patient samples has long been based on the preliminary judgment of the morphological characteristics exhibited by the isolated pathogens. Among them, the classification of microorganisms according to the colony morphology is an essential part of the pre-screening process, because it can roughly classify the pathogens, making subsequent identification more specific and improving the work efficiency of the clinical microbiology laboratory. In the current work of the clinical laboratory, the pre-screening process is often completed by clinical microbiology staff, but the staff needs to have sufficient knowledge of colony morphology and accumulation, which is difficult for beginners, and manual review of the culture medium plate or image often has problems such as large workload, long time-consuming, and results easily affected by subjective factors.

[0004] In recent years, with the continuous development of artificial intelligence and deep learning, classification and recognition technology has begun to be applied to various aspects such as industrial production, medical care, smart cities, etc. Because the multi-modal sample images of the microorganism pipeline will increase over time, and the morphology and distribution of microorganisms will be different between different modalities and different culture cycles, the previous classification and recognition algorithm directly applies the trained classification model for inference to obtain the classification result, and the model is fixed and cannot learn the knowledge in the new sample. At the same time, the existing classification model is mostly single modal, and cannot process multi-modal colony samples. Only one modality is used for training, which may cause misjudgment and misjudgment due to the lack of information between modalities, resulting in low accuracy of classification and recognition and poor robustness of the model. At the same time, during the cultivation of microorganisms, the staff inevitably makes operation mistakes, resulting in noise in the multi-modal colony sample. If the existing classification method is used for training, it may cause accumulation of noise and poor model fitting, thereby greatly increasing the probability of misidentification.

[0005] Therefore, how to improve the accuracy of multi-modal colony sample recognition is a problem that those skilled in the art need to solve. SUMMARY

[0006] Therefore, the present application provides a multi-modal colony sample autonomous learning classification and recognition method and system based on a diffusion model to solve the problems in the background art.

[0007] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] On the one hand, a multi-modal colony sample autonomous learning classification and recognition method based on a diffusion model is provided, comprising:

[0009] Obtaining a multi-modal colony sample;

[0010] Inputting the multi-modal colony sample into a multi-modal colony sample classification model to obtain a classification result;

[0011] The classification process of the multi-modal colony sample classification model is as follows:

[0012] Generating a multi-modal global prior and a multi-modal local prior through multi-modal dual-granularity conditional guidance of the multi-modal colony sample;

[0013] Applying a diffusion process to the multi-modal global prior, the multi-modal local prior, and the one-hot encoding of the sample label to generate three noise variables;

[0014] Performing feature fusion on the multi-modal colony sample through multi-modal feature fusion encoding to obtain a multi-modal feature embedding;

[0015] After the three noise variables are connected and projected into the latent space, they are input into a denoising network together with the multi-modal feature embedding for integration, and the noise distribution of the three noise variables sampled in the forward process is learned while parameterizing the reverse diffusion process, and the final classification result is generated in the reverse diffusion process.

[0016] Optionally, it further comprises:

[0017] The trained multi-modal colony sample classification model is used as an initial teacher network and student network; the teacher network is used to generate pseudo-labels, and according to the difficulty and information two active sampling indicators, top-N data are selected and used as pseudo-labels for annotation and added to the training data set;

[0018] The teacher network is frozen, the student network is trained, and the parameters of the teacher network are updated through EMA to update the pseudo-labels, thereby completing the autonomous learning of the multi-modal colony sample classification model for new multi-modal colony samples.

[0019] Optionally, the multi-modal dual-granularity conditional guidance is divided into two branches, namely a multi-modal global branch and a multi-modal local branch, wherein the multi-modal global branch uses a global encoder with a ViT backbone network to extract and fuse the features of the multi-modal sample, and then generates a multi-modal saliency map through a convolution layer The average response is used to predict the multi-modal global prior from the saliency map; the multi-modal local branch crops the response salient RoI by combining the saliency map generated by the multi-modal global branch, and then obtains the multi-modal local prior through a feature extraction network and a gated attention mechanism.

[0020] Optionally, the multi-modal feature fusion encoding adopts a ready-to-use pre-trained variational autoencoder to fuse and encode the features of the multi-modal colony sample into a latent space, then splices the multi-modal features to obtain a multi-modal joint feature and introduces an attention mechanism to predict the importance distribution probability of different modalities, and then performs dot product on the distribution probability and the multi-modal fusion feature to obtain a new multi-modal fusion feature with the importance of different modal features reweighted.

[0021] Optionally, the denoising network adopts DiTs, and the input of the denoising network is the noisy multi-modal colony sample and additional condition information: noise time step And sample label Splicing generates an additional embedding, which, together with the noisy image, enters the latent diffusion to parameterize the reverse diffusion process and learn the noise distribution in the forward process, and in the reverse diffusion process, the noise variable distribution is restored to

[0022] Optionally, the pseudo-label generation process is as follows: a batch of unlabeled multimodal new samples are inferred through a teacher network to obtain category predictions, the difficulty of obtaining the samples and the information content of the images are calculated respectively, and the top-N new samples are selected, and their category predictions are used as their pseudo-labels.

[0023] Optionally, unlabeled multimodal new samples are added to the training set after strong data augmentation, the teacher network is frozen, and the student network is retrained. The parameters of the teacher network are gradually updated from the student network through exponential moving average. At the same time, unlabeled multimodal new samples are entered into the teacher network for inference after weak data augmentation to update pseudo-labels. Through continuous iteration, the quality of pseudo-labels is improved.

[0024] Optionally, the formulas for calculating the difficulty of the sample and the information content of the image are as follows:

[0025]

[0026]

[0027] in, For the number of categories, Indicates the teacher network's... The predicted probabilities of each category, These are the parameters for the teacher network.

[0028] On the other hand, a multimodal colony sample autonomous learning classification and identification system based on a diffusion model is provided, including:

[0029] The acquisition module acquires multimodal bacterial colony samples;

[0030] The classification module takes multimodal colony samples and inputs them into the multimodal colony sample classification model to obtain the classification results.

[0031] The classification module includes:

[0032] The multimodal dual-granularity condition guidance module generates multimodal global priors and multimodal local priors from the multimodal colony samples through multimodal dual-granularity condition guidance.

[0033] The noise variable module applies a diffusion process to the multimodal global prior, the multimodal local prior, and the one-hot encoding of the sample labels to generate three noise variables.

[0034] The fusion module performs feature fusion on multimodal colony samples through multimodal feature fusion encoding to obtain multimodal feature embeddings;

[0035] The output module is connected to the three noise variables, and after being projected into the latent space, is input into a denoising network together with a multi-modal feature embedding for integration, and learns the noise distribution of the three noise variables sampled in the forward process while parameterizing the backward diffusion process, and generates a final classification result in the backward diffusion process.

[0036] According to the technical solution, the application discloses a multi-modal bacterial colony sample autonomous learning classification and identification method and system based on a diffusion model, which has the following beneficial effects compared with the prior art.

[0037] 1. The application can fuse features of multi-modal sample input, deeply excavate the correlation between different modalities, and fully utilize semantic information of multi-modalities.

[0038] 2. The application combines a diffusion model, uses the diffusion model to classify the multi-modal bacterial colony sample after noise reduction, is beneficial to eliminating inherent noise in the multi-modal data sample and interference introduced by human operation errors, and is more beneficial to learning.

[0039] 3. The application combines autonomous learning, and the model can complete automatic labeling of new multi-modal samples, can select the most suitable multi-modal sample for training and learning, and continuously improves the robustness of the model.

[0040] 4. High precision, accurate to 100% for judging whether a culture dish has microbial growth, accurate to 98% for judging single colony or multiple colonies, accurate to 95% for identifying the category of the colony in the case of a single colony. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0042] Figure 1 A multi-modal bacterial colony classification and identification method flowchart is provided for the embodiments of the application.

[0043] Figure 2 A model optimization and automatic labeling flowchart based on autonomous learning is provided for the embodiments of the application.

[0044] Figure 3 A multi-modal double-granularity conditional guidance principle diagram is provided for the embodiments of the application.

[0045] Figure 4A multi-modal colony sample autonomous learning classification and identification system structure diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0047] An embodiment of the present application discloses a multi-modal colony sample autonomous learning classification and identification method based on a diffusion model, as shown in Figure 1 , which comprises the following steps.

[0048] Obtaining a multi-modal colony sample;

[0049] Inputting the multi-modal colony sample into a multi-modal colony sample classification model to obtain a classification result;

[0050] The classification process of the multi-modal colony sample classification model is as follows.

[0051] S1, generating a multi-modal global prior and a multi-modal local prior through multi-modal dual-granularity conditional guidance for the multi-modal colony sample;

[0052] After the colony samples of different modalities are preprocessed through data enhancement such as flipping, rotating and scaling, multi-modal dual-granularity conditional guidance is used to realize feature extraction. As shown in Figure 3 , for a multi-modal input sample , the sample is processed in two branches of a global flow and a local flow. The global flow uses a global encoder with a ViT backbone network to realize feature extraction and fusion of the multi-modal sample, and then the output is generated into a multi-modal saliency map through a convolution layer . The multi-modal global prior is predicted from the saliency map by using an average response. The local flow multiplies the multi-modal input after splicing with , and crops out the response saliency RoI. Each RoI is fed into a local encoder with a ViT backbone to obtain a multi-modal RoI feature vector, and then a gating attention mechanism is applied to fuse all the feature vectors from the RoI to obtain a multi-modal weighted vector. Finally, a linear layer is used to calculate the multi-modal local prior . The calculation formulas of the global and local multi-modal local priors are as follows.

[0053]

[0054]

[0055] wherein, denotes the average response, is a multi-modal ViT feature extractor, is a gating attention operation, denotes a concatenation operation, denotes a multiplication.

[0056] S2, a diffusion process is applied to the multi-modal global prior, multi-modal local prior and one-hot encoding of the sample label to generate three noise variables;

[0057] The label of the multi-modal colony sample is one-hot encoded to obtain a variable , and a Gaussian noise is added through a diffusion process under the condition of a time step sampled from a uniform distribution , and the noise variable is denoted as , and the noise variable is sampled according to the multi-modal global prior and the multi-modal local prior in the diffusion process, and the sampling formula is as follows:

[0058]

[0059] wherein, , ， , is a fixed value sequence ， 。

[0060] S3, the multi-modal colony sample is fused by multi-modal feature fusion coding to obtain a multi-modal feature embedding;

[0061] The original multi-modal sample is input into a multi-modal feature coding module, which can extract the features of different modal samples and realize multi-modal feature fusion.

[0062] Among them, the feature encoder adopts the pre-training variational auto-encoder (Variational Auto-Encoder, VAE) of Stable Diffusion, specifically, the down-sampling factor of the VAE encoder is 8, and the multi-modal colony sample with an input shape of is encoded to The multi-modal feature is spliced to obtain a multi-modal joint feature, which is input into an attention module to predict the importance distribution probability of different modalities, and then the distribution probability is dot multiplied with the multi-modal fusion feature to obtain a new multi-modal fusion feature with the importance of different modalities being reweighted The process of multi-modal feature fusion is as follows:

[0063]

[0064] wherein represents a VAE feature encoder, is a multi-modal feature splicing operation, represents an attention operation performed on the preliminary multi-modal feature.

[0065] S4, after the three noise variables are connected and projected into the latent space, together with the multi-modal feature embedding are input into the DiTs denoising network for integration, and the noise distribution in the forward process is learned while the parameterized reverse diffusion process is performed. ， and the sampled noise distribution generates the final classification result in the reverse diffusion process.

[0066] DiTs is adopted as the denoising network. The input of the denoising network is the noisy multi-modal colony sample and additional condition information: noise time step embedding and sample label embedding of the additional embedding generated after splicing . and the noise image enters the latent diffusion Transformer to parameterize the reverse diffusion process and learn the noise distribution in the forward process, as follows:

[0067]

[0068]

[0069]

[0070] wherein represents a fully connected layer, is a projection layer to the latent space, is a connection operation, represents performing Hadamard dot product, is reshape operation, is a multi-head self-attention operation, This is a multi-head cross-attention operation. For layer normalization, This is the output of the hidden layer. Finally, a fully connected layer with an output dimension of 1 is connected to it as noise prediction.

[0071] During the back diffusion process, trained DiTs By distributing noise variables Restore to the true label distribution To generate the final classification prediction. .

[0072]

[0073]

[0074] in for DiTs Parameters of the denoising network Indicates a Gaussian distribution. It is an identity matrix.

[0075] Predicting the first Class probability and final classification prediction Represented as:

[0076]

[0077]

[0078] in It is the temperature coefficient. For the number of categories, A C-dimensional vector consisting entirely of 1s. express and The first element-wise squared error vector between them k Wei, that is .

[0079] In a specific embodiment, such as Figure 2 As shown, it also includes:

[0080] The trained multimodal colony sample classification model is used as the initial teacher network and student network. The teacher network is used to generate pseudo-labels for inference. Based on two active sampling indicators, difficulty and information, the top-N data are selected and labeled with pseudo-labels and added to the training dataset.

[0081] A batch of unlabeled multimodal new samples is inferred through a teacher network to obtain category predictions. The difficulty of obtaining the samples and the information content of the images are calculated respectively. The top-N new samples are selected, and their category predictions are used as their pseudo-labels. The formula is as follows:

[0082]

[0083]

[0084] in, For the number of categories, Indicates the teacher network's... The predicted probabilities of each category, These are the parameters for the teacher network.

[0085] The teacher network is frozen, the student network is trained, and the parameters of the teacher network are updated via EMA, thereby updating the pseudo-labels and completing the autonomous learning of the multimodal colony sample classification model for new multimodal colony samples.

[0086] New samples are selected, heavily augmented, and added to the training set. The teacher network is then frozen, and the student network is retrained. The parameters of the teacher network are gradually updated from the student network using an exponential moving average (EMA). Simultaneously, new multimodal samples are introduced into the teacher network after weak data augmentation to update pseudo-labels. Through continuous iteration, the quality of the pseudo-labels is improved, enabling the model to label new samples and autonomously learn semantic information from them, thereby continuously improving the model's robustness and generalization. The formula is as follows:

[0087]

[0088] in and These are the parameters for the teacher network and the student network, respectively. For the first Step training, It is a hyperparameter that determines the transmission speed of parameters, and is generally close to 1.

[0089] In one specific embodiment, unlabeled multimodal new samples are added to the training set after strong data augmentation, the teacher network is frozen, and the student network is retrained. The parameters of the teacher network are gradually updated from the student network through exponential moving average. At the same time, unlabeled multimodal new samples are introduced into the teacher network for inference after weak data augmentation to update pseudo-labels. Through continuous iteration, the quality of pseudo-labels is improved.

[0090] On the other hand, a multimodal colony sample autonomous learning classification and recognition system based on a diffusion model is provided, such as... Figure 4 As shown, it includes:

[0091] The acquisition module acquires multimodal bacterial colony samples;

[0092] The classification module inputs the multi-modal colony sample into a multi-modal colony sample classification model to obtain a classification result.

[0093] The classification module comprises:

[0094] The multi-modal dual-granularity conditional guidance module generates a multi-modal global prior and a multi-modal local prior through multi-modal dual-granularity conditional guidance of the multi-modal colony sample.

[0095] The noise variable module generates three noise variables by applying a diffusion process to the multi-modal global prior, the multi-modal local prior, and a one-hot encoding of a sample label.

[0096] The fusion module performs feature fusion on the multi-modal colony sample through multi-modal feature fusion encoding to obtain a multi-modal feature embedding.

[0097] The output module inputs the three noise variables after being connected and projected into a latent space, together with the multi-modal feature embedding, into a denoising network for integration, and learns the noise distribution of the three noise variables sampled in the forward process while parameterizing the backward diffusion process, to generate a final classification result in the backward diffusion process.

[0098] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0099] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-modal colony sample autonomous learning classification and identification method based on a diffusion model, characterized in that, The method comprises the following steps: obtaining a multi-modal colony sample; inputting the multi-modal colony sample into a multi-modal colony sample classification model to obtain a classification result; the classification process of the multi-modal colony sample classification model is as follows: the multi-modal colony sample is guided by a multi-modal dual-granularity conditional guidance to generate a multi-modal global prior and a multi-modal local prior; a diffusion process is applied to the multi-modal global prior, the multi-modal local prior and the one-hot encoding of the sample label to generate three noise variables; the multi-modal colony sample is fused by multi-modal feature fusion coding to obtain a multi-modal feature embedding; after the three noise variables are connected and projected into a latent space, they are input into a denoising network together with the multi-modal feature embedding for integration, and the noise distribution of the three noise variables sampled in the forward process is learned while the parameterized back diffusion process is performed, and a final classification result is generated in the back diffusion process; the method further comprises the following steps: the trained multi-modal colony sample classification model is used as an initial teacher network and student network; the teacher network is used to generate pseudo labels, and top-N data are selected according to the difficulty and information two active sampling indicators and used as the labels to be added to the training data set; the teacher network is frozen, the student network is trained, and the parameters of the teacher network are updated through EMA to update the pseudo labels, thereby completing the autonomous learning of the multi-modal colony sample classification model for new multi-modal colony samples.

2. The method according to claim 1, wherein, The multi-modal dual-granularity conditional guidance is divided into two branches, namely a multi-modal global branch and a multi-modal local branch, wherein the multi-modal global branch uses a global encoder with a ViT backbone network to extract and fuse the features of the multi-modal sample, and then outputs the features through a convolutional layer to generate a multi-modal saliency map, and a multi-modal global prior is predicted from the saliency map by using an average response; the multi-modal local branch crops out a response salient RoI by combining the saliency map generated by the multi-modal global branch, and then obtains a multi-modal local prior through a feature extraction network and a gated attention. The multi-modal dual-granularity conditional guidance is divided into two branches, namely a multi-modal global branch and a multi-modal local branch, wherein the multi-modal global branch uses a global encoder with a ViT backbone network to extract and fuse the features of the multi-modal sample, and then outputs the features through a convolutional layer to generate a multi-modal saliency map, and a multi-modal global prior is predicted from the saliency map by using an average response; the multi-modal local branch crops out a response salient RoI by combining the saliency map generated by the multi-modal global branch, and then obtains a multi-modal local prior through a feature extraction network and a gated attention.

3. The method of claim 1, wherein the method is characterized by, The multi-modal feature fusion coding adopts a ready-to-use pre-trained variational autoencoder to fuse and encode the features of the multi-modal colony sample into a latent space, then splices the multi-modal features to obtain multi-modal joint features and introduces an attention mechanism to predict the importance distribution probability of different modalities, and then performs dot product operation on the distribution probability and the multi-modal fusion features to obtain new multi-modal fusion features with the importance of different modal features being reweighted.

4. The method of claim 1, wherein the method is characterized by, The denoising network adopts DiTs, and the input of the denoising network is a noisy multi-modal colony sample and additional condition information: noise time steps and sample labels After splicing, an additional embedding is generated, and a noise image enters a latent diffusion to parameterize a reverse diffusion process and learn a noise distribution in a forward process. In the reverse diffusion process, the noise variable distribution is restored into and a classification result is generated.

5. The method of claim 1, wherein the method is characterized by, The generation process of the pseudo labels is as follows: a batch of unlabeled multi-modal new samples are input into the teacher network to obtain class predictions, the difficulty of the samples and the information amount of the images are calculated respectively, and top-N new samples are selected as the pseudo labels.

6. The method of claim 5, wherein the method is characterized by, After the unlabeled multi-modal new samples are subjected to strong data enhancement and added to the training set, the teacher network is frozen, the student network is retrained, the parameters of the teacher network are gradually updated from the student network through exponential moving average, and at the same time, the unlabeled multi-modal new samples are input into the teacher network after being subjected to weak data enhancement to update the pseudo labels, and through continuous iteration, the quality of the pseudo labels is improved.

7. The method of claim 5, wherein the method is characterized by, The calculation formula of the difficulty of the samples and the information amount of the images is as follows: wherein, is the number of classes, denotes the predicted probability of the teacher network for the th class, are the parameters of the teacher network.

8. A multi-modal colony sample autonomous learning classification and identification system based on a diffusion model, characterized in that, The method comprises the following steps: an acquisition module is configured to acquire a multi-modal colony sample; a classification module is configured to input the multi-modal colony sample into a multi-modal colony sample classification model to obtain a classification result; the classification module comprises: a multi-modal dual-granularity conditional guidance module configured to guide the multi-modal colony sample by a multi-modal dual-granularity conditional guidance to generate a multi-modal global prior and a multi-modal local prior; a noise variable module configured to apply a diffusion process to the multi-modal global prior, the multi-modal local prior and the one-hot encoding of the sample label to generate three noise variables; A fusion module is configured to fuse features of the multi-modal colony sample through multi-modal feature fusion coding to obtain multi-modal feature embedding; An output module is configured to input the three noise variables connected after being projected into a latent space and the multi-modal feature embedding into a denoising network for integration, and learn noise distribution of sampling of the three noise variables in the forward process while parameterizing the backward diffusion process, to generate a final classification result in the backward diffusion process. Further comprising: The trained multi-modal colony sample classification model is used as an initial teacher network and student network; the teacher network is used to generate pseudo labels, and according to two active sampling indicators of difficulty and information, top-N data are selected and used as labels to be added to the training data set; The teacher network is frozen, the student network is trained, and the parameters of the teacher network are updated through EMA to update the pseudo labels, thereby completing the autonomous learning of the multi-modal colony sample classification model for new multi-modal colony samples.

Images