Multimodal colony sample fine-grained classification method and system based on prompt increment

Abstract

The present invention discloses a multimodal colony sample fine-grained classification method and system based on prompt increment, which relates to the technical field of computer vision, including: obtaining a microbial plate image as a sample, establishing a sample fine-grained classification model, and the operation process is as follows: processing the sample, dividing the processed sample into blocks and converting it into a sequence and splicing it with the learned category representation parameters into a new feature sequence; sending it to a first ViT encoder to obtain a discriminative fusion feature and a coarse category representation; inputting the coarse category representation into a linear classifier to obtain a strain classification result, and querying the corresponding prompt feature in the subtype prompt space; splicing the discriminative fusion feature and the prompt feature, inputting the second ViT encoder, and performing average pooling on the output as a fine category representation; inputting the fine category representation into a linear classifier to obtain a subtype classification result. The present invention solves the problems of high colony classification training cost and insufficient accuracy in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and more particularly to a method and system for fine-grained classification of multimodal colony samples based on prompt increment. Background Art

[0002] Microbial resistance is a major challenge facing global public health and a global issue of widespread concern to governments and societies worldwide. With the emergence of various new infectious diseases and the continued rise in antibiotic resistance among pathogens, microbiology laboratory testing is playing an increasingly important role in the diagnosis and treatment of infectious diseases. Timely and accurate clinical identification of microbial species can provide a basis for early diagnosis of infectious diseases, aid in targeted medication, and reduce antibiotic abuse. In clinical microbiology laboratories, the identification of microorganisms in patient samples has long been based primarily on preliminary assessments of the morphological characteristics of isolated pathogens.

[0003] Microbial classification based on colony morphology is a crucial step in pre-screening. It allows for a broad classification of pathogens, enabling more specific subsequent identification. The meticulousness and accuracy of pre-screening impact the efficiency of clinical microbiology laboratories. Currently, pre-screening is typically performed by clinical microbiologists, who often only roughly identify the broad categories of colonies but are unable to identify specific subtypes.

[0004] However, in recent years, with the continuous development of artificial intelligence and computer vision technologies, automatic classification technology has been widely used in the medical field and has become an important tool for assisting clinical diagnosis. For bacterial species classification tasks, previous automatic classification technologies have the following major problems: First, bacterial colony samples obtained in clinical practice are of different modalities, and traditional single-modality algorithms have poor robustness. Second, due to the subtle morphological differences between different bacterial species and subtypes, direct classification using traditional image classification algorithms is very difficult. Third, the number of bacterial colony samples collected in clinical practice is constantly increasing, and the number of subtypes and species encountered is also increasing. To address the problem of knowledge forgetting after each new data is obtained, traditional deep learning algorithms often need to retrain the entire model by mixing new and old data. This leads to an exponential increase in training costs and reduces the efficiency of the model in learning new knowledge.

[0005] Therefore, how to provide a multimodal colony sample fine-grained classification method that can continuously learn, has low training cost and high accuracy is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0006] In view of this, the present invention provides a multimodal colony sample fine-grained classification method and system based on prompt increment, which can continuously learn colony sample classification with low training cost and high accuracy, while avoiding the waste of human resources.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A multimodal colony sample fine-grained classification method based on hint increment, including:

[0009] Acquire multimodal microbial plate images, use them as samples, and establish a fine-grained sample classification model;

[0010] The sample fine-grained classification model runs as follows:

[0011] Acquire multimodal microbial plate images, use them as samples, and establish a fine-grained sample classification model;

[0012] The sample fine-grained classification model runs as follows:

[0013] Step 1: Scale and normalize the sample, then divide the processed sample into blocks and convert it into a sequence, concatenate it with the learned category representation parameters, and add the position code as a new feature sequence;

[0014] Step 2: Send the new feature sequence to the first ViT encoder to obtain the discriminative fusion features and coarse category representation;

[0015] Step 3: Input the coarse category representation into a linear classifier to obtain the bacterial species classification result;

[0016] Step 4: According to the bacterial species classification result, query and obtain corresponding prompt features in the subtype prompt space;

[0017] Step 5: The discriminative fusion feature is concatenated with the hint feature as a new input, which is fed into the second ViT encoder. The output hint feature is average-pooled and then represented as a fine category.

[0018] Step 6: Input the sub-category representation into the linear classifier to obtain the sub-type classification result.

[0019] Preferably, it also includes: calculating the loss value between the subtype classification result and the sample true value label, fixing the backbone model parameters of the second ViT encoder, and updating the prompt features corresponding to the subtype prompt space and the linear classifier according to the loss value.

[0020] Preferably, the first ViT encoder specifically includes: a pre-trained ViT encoder, and the ViT encoder performs region selection based on the total attention value before the last layer of the network.

[0021] Preferably, the calculation formula of each layer of the pre-trained ViT encoder is as follows:

[0022]

[0023]

[0024] in, It is a multi-head self-attention mechanism. is a multi-layer perceptron, is layer normalization, is the total number of modules of the ViT encoder, Represents the input features of each layer of the network, Indicates passing Layer characteristics, Indicates passing Characteristics of the layer.

[0025] Preferably, the calculation formula for region selection based on the total attention value is as follows:

[0026]

[0027]

[0028] in, The attention value of each layer corresponding to each feature block The product of is the number of feature blocks, , is the number of image blocks, is the sample mode type, is the attention value of each layer corresponding to each feature block, is the total attention value, select the top The features corresponding to the feature blocks As the fusion feature input of the last layer of the network.

[0029] Preferably, the subtype prompts space storage Group prompt features , is the number of bacterial species; each bacterial species is assigned a set of prompt features , to indicate features As training parameters, Is the length, the dimension is equal to the dimension of the input fusion feature , the subtype prompt space is trained.

[0030] A multimodal colony sample fine-grained classification system based on hint increment, including:

[0031] A sample acquisition module acquires multimodal microbial plate images and uses them as samples;

[0032] A sequence generation module scales and normalizes the samples, then blocks and converts the processed samples into sequences, concatenates them with the learned category representation parameters, and adds position encoding as a new feature sequence;

[0033] A feature category acquisition module feeds the new feature sequence into the first ViT encoder to obtain discriminative fusion features and coarse category representation;

[0034] a strain classification acquisition module, which inputs the coarse category representation into a linear classifier to obtain a strain classification result;

[0035] A prompt feature acquisition module queries and acquires corresponding prompt features in the subtype prompt space according to the bacterial species classification result;

[0036] The category acquisition module concatenates the discriminative fusion features with the hint features as new input, inputs the new input to the second ViT encoder, performs average pooling on the output hint features, and then represents them as fine categories;

[0037] The classification acquisition module inputs the sub-category representation into the linear classifier to obtain a sub-type classification result.

[0038] Preferably, it also includes: an update module, which calculates the loss value of the subtype classification result and the sample true value label during fine-tuning training, fixes the backbone model parameters of the second ViT encoder, and updates the prompt features corresponding to the subtype prompt space and the linear classifier according to the loss value.

[0039] It can be seen from the above technical solution that compared with the existing technology, the present invention discloses a fine-grained classification method and system for multimodal colony samples based on prompt increments, which can be deployed on the clinical microbiology pipeline to complete the automatic classification and pre-screening of multimodal colony samples in real time; it has strong robustness and fully integrates multimodal data to make the comprehensive ability of the model stronger; it has strong plasticity and combines the prompt increment method to enable the model to train new data at any time and integrate old knowledge while absorbing new knowledge; the training cost is low and a prompt mechanism is introduced. Only about 1% of the parameters need to be updated during fine-tuning training of the model; it has high accuracy and can accurately classify the specific subtypes of each type of bacteria, which is more than 10% higher than the accuracy of traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0041] Figure 1 The accompanying drawing is a flow chart of the method provided by the present invention.

[0042] Figure 2 The accompanying drawing is a structural diagram of the model provided by the present invention.

[0043] Figure 3 The accompanying drawing is a schematic diagram of task decomposition provided by the present invention.

[0044] Figure 4 The accompanying figure is a schematic diagram of incremental learning based on the prompt mechanism provided by the present invention. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] The embodiment of the present invention discloses a multimodal colony sample fine-grained classification method based on prompt increment, such as Figure 1 Shown, including:

[0047] Acquire multimodal microbial plate images, use them as samples, and establish a fine-grained sample classification model;

[0048] The sample fine-grained classification model runs as follows;

[0049] Step 1: Scale and normalize the samples, then divide the processed samples into blocks and convert them into sequences, concatenate them with the learned category representation parameters, and add position encoding as a new feature sequence;

[0050] Step 2: Feed the new feature sequence into the first ViT encoder to obtain discriminative fusion features and coarse category representation;

[0051] Step 3: Input the coarse category representation into the linear classifier to obtain the bacterial species classification result;

[0052] Step 4: Based on the bacterial species classification results, query the subtype prompt space to obtain the corresponding prompt features;

[0053] Step 5: The discriminative fusion features are concatenated with the hint features as new inputs, which are fed into the second ViT encoder. The output hint features are average-pooled and then represented as fine categories.

[0054] Step 6: Input the fine category representation into the linear classifier to obtain the subtype classification result.

[0055] In a specific embodiment, it also includes: calculating the loss value between the subtype classification result and the sample true value label, fixing the backbone model parameters of the second ViT encoder, and updating the prompt features and linear classifier corresponding to the subtype prompt space according to the loss value.

[0056] In a specific embodiment, the first ViT encoder specifically includes: a pre-trained ViT encoder, and the region selection is performed based on the total attention value before the last layer of the ViT encoder.

[0057] In a specific embodiment, the calculation formula for each layer of the pre-trained ViT encoder is as follows:

[0058]

[0059]

[0060] in, It is a multi-head self-attention mechanism. is a multi-layer perceptron, is layer normalization, is the total number of modules of the ViT encoder, Represents the input features of each layer of the network, Indicates passing Layer characteristics, Indicates passing Characteristics of the layer.

[0061] In a specific embodiment, the calculation formula for region selection based on the total attention value is as follows:

[0062]

[0063]

[0064] in, The attention value of each layer corresponding to each feature block The product of is the number of feature blocks, , is the number of image blocks, is the sample mode type, is the attention value of each layer corresponding to each feature block, is the total attention value, select the top The features corresponding to the feature blocks As the fusion feature input of the last layer of the network.

[0065] In one embodiment, the subtype hint space stores Group prompt features , is the number of bacterial species; each bacterial species is assigned a set of prompt features , to indicate features As training parameters, Is the length, the dimension is equal to the dimension of the input fusion feature , train the subtype cue space.

[0066] A multimodal colony sample fine-grained classification system based on hint increment, including:

[0067] A sample acquisition module acquires multimodal microbial plate images and uses them as samples;

[0068] The sequence generation module scales and normalizes the samples, then blocks the processed samples and converts them into sequences, concatenates them with the learned category representation parameters, and adds position encoding as a new feature sequence;

[0069] The feature category acquisition module feeds the new feature sequence into the first ViT encoder to obtain the discriminative fusion features and coarse category representation;

[0070] The bacterial species classification acquisition module inputs the coarse category representation into the linear classifier to obtain the bacterial species classification results;

[0071] The prompt feature acquisition module queries and obtains the corresponding prompt features in the subtype prompt space according to the bacterial species classification results;

[0072] The category acquisition module concatenates the discriminative fusion features with the hint features as new input, inputs the second ViT encoder, performs average pooling on the output hint features, and then represents them as fine categories;

[0073] The classification acquisition module inputs the fine category representation into the linear classifier to obtain the subtype classification results.

[0074] In a specific embodiment, it also includes: an update module, which calculates the loss value between the subtype classification result and the sample true value label during fine-tuning training, fixes the backbone model parameters of the second ViT encoder, and updates the prompt features and linear classifier corresponding to the subtype prompt space according to the loss value.

[0075] In specific embodiment 1, a multimodal colony sample fine-grained classification method based on prompt increment, such as Figure 2As shown, the following steps are included:

[0076] Multimodal microbial plate images were automatically collected through the clinical microbiology pipeline, and each group of samples had modalities, and the image size of each modality is , Indicates the width of the image. Represents the height of the image, which is used as a sample to establish a sample fine-grained classification model.

[0077] The sample fine-grained classification model runs as follows:

[0078] Step 1: Scale the obtained images of each modality to , and normalize it, and then divide it into The size is image blocks, where . Then a linear embedding layer is used to transform the image into The characteristic sequence of is the number of embedded feature channels, and finally The feature sequence of each modality is concatenated with a learnable category representation parameter and positional encoding is added as the input of the encoder. The calculation formula is as follows:

[0079]

[0080] in For category representation, For the The input sequence of the modalities, is a linear embedding, is the position code, Indicates size The vector space of .

[0081] Step 2: Feed the input into the ViT encoder that has been pre-trained using a large amount of existing data. The encoder contains layer network, the network calculation formula for each layer is as follows:

[0082]

[0083]

[0084] in, It is a multi-head self-attention mechanism. is a multi-layer perceptron, is layer normalization, is the total number of modules of the ViT encoder, Represents the input features of each layer of the network, Indicates passing Layer characteristics, Indicates passing Characteristics of the layer.

[0085] In order to enable the model to better capture local discriminative features, this scheme performs region selection based on the total attention value before the last layer of the ViT encoder. First, calculate the attention value of each layer corresponding to each feature block. The product of , and then based on these total attention values Size pair feature blocks are sorted, where . To select the top The features corresponding to the feature blocks

[0086] As the fusion feature input of the last layer of the network, the unimportant areas are discarded. The calculation formula is as follows:

[0087]

[0088]

[0089] in, The attention value of each layer corresponding to each feature block The product of is the number of feature blocks, , is the number of image blocks, is the sample mode type, is the attention value of each layer corresponding to each feature block, is the total attention value, select the top The features corresponding to the feature blocks As the fusion feature input of the last layer of the network.

[0090] Finally, these features pass through another layer of network to obtain discriminative fusion features and coarse category representation.

[0091] Step 3: Input the coarse category representation into the linear classifier to obtain the bacterial species classification result;

[0092] Step 4: The strain classification results are queried through the subtype prompt space to obtain the corresponding prompt features for further subtype classification, such as Figure 3 As shown, the task is decomposed into a bacterial species classification task and The subtype classification task is grouped, so the subtype prompt space stores a total of Group prompt features These suggestive features are all trainable parameters, length Set flexibly according to actual conditions, and the dimension is equal to the dimension of the input fusion feature .

[0093] Step 5: Fusion of the discriminative features obtained in the previous stage and prompt features Splice it into a new input, use another pre-trained ViT encoder to extract features, and output the corresponding prompt features , and then after an average pooling, we get the final fine category representation:

[0094]

[0095] Step 6: Pass the fine category representation through a linear classifier to obtain the subtype classification result.

[0096] In a specific embodiment, it also includes:

[0097] The subtype classification results and the true value labels are calculated through the loss function. The loss function uses FocalLoss. The calculation formula is as follows:

[0098]

[0099] in, Represents the prediction result, represents the true value of the label, represents the label value of the i-th category, represents the predicted value of the i-th category, represents the frequency of occurrence of the i-th category, is a modulation parameter. Using this loss function can make the model more focused on learning difficult samples.

[0100] During fine-tuning training, Figure 4 Freeze the network backbone as shown 、 , by minimizing the predicted results With label Calculate the loss value to optimize separately Group prompt features , represents the input feature space, Represents the length of the feature space:

[0101]

[0102] The present invention discloses a fine-grained classification method and system for multimodal colony samples based on prompt increments, which can be deployed on the clinical microbiology pipeline to complete the automatic classification and pre-screening of multimodal colony samples in real time; it has strong robustness and fully integrates multimodal data, making the comprehensive ability of the model stronger; it has strong plasticity and combines the prompt increment method to enable the model to train new data at any time, integrating old knowledge while absorbing new knowledge; the training cost is low and the prompt mechanism is introduced, and only about 1% of the parameters need to be updated during fine-tuning training of the model; it has high accuracy and can accurately classify the specific subtypes of each type of bacteria, which is more than 10% higher than the accuracy of traditional methods.

[0103] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0104] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A fine-grained classification method for multimodal colony samples based on hint increment, characterized by: include: Acquire multimodal microbial plate images, use them as samples, and establish a fine-grained sample classification model; The sample fine-grained classification model runs as follows: Step 1: Scale and normalize the sample, then divide the processed sample into blocks and convert it into a sequence, concatenate it with the learned category representation parameters, and add the position code as a new feature sequence; Step 2: Send the new feature sequence to the first ViT encoder to obtain the discriminative fusion features and coarse category representation; Step 3: Input the coarse category representation into a linear classifier to obtain the bacterial species classification result; Step 4: According to the bacterial species classification result, query and obtain corresponding prompt features in the subtype prompt space; Step 5: The discriminative fusion feature is concatenated with the hint feature as a new input, which is fed into the second ViT encoder. The output hint feature is average-pooled and then represented as a fine category. Step 6: Input the sub-category representation into the linear classifier to obtain the sub-type classification result.

2. A multimodal colony sample fine-grained classification method based on prompt increment according to claim 1, characterized in that: Also includes: Calculate the loss value between the subtype classification result and the true value label of the sample, fix the backbone model parameters of the second ViT encoder, and update the prompt features corresponding to the subtype prompt space and the linear classifier according to the loss value.

3. A multimodal colony sample fine-grained classification method based on prompt increment according to claim 1, characterized in that: The first ViT encoder specifically includes: a pre-trained ViT encoder, and the ViT encoder performs region selection based on the total attention value before the last layer of the network.

4. A multimodal colony sample fine-grained classification method based on prompt increment according to claim 3, characterized in that: The calculation formula for each layer of the pre-trained ViT encoder is as follows: in, It is a multi-head self-attention mechanism. is a multi-layer perceptron, is layer normalization, is the total number of modules of the ViT encoder, Represents the input features of each layer of the network, Indicates passing Layer characteristics, Indicates passing Characteristics of the layer.

5. The multimodal colony sample fine-grained classification method based on prompt increment according to claim 3 is characterized in that: The calculation formula for region selection based on the total attention value is as follows: in, The attention value of each layer corresponding to each feature block The product of is the number of feature blocks, , is the number of image blocks, is the sample mode type, is the attention value of each layer corresponding to each feature block, is the total attention value, select the top The features corresponding to the feature blocks As the fusion feature input of the last layer of the network.

6. A multimodal colony sample fine-grained classification method based on prompt increment according to claim 1, characterized in that: The subtypes suggest spatial storage Group prompt features , is the number of bacterial species; each bacterial species is assigned a set of prompt features , to indicate features As training parameters, Is the length, the dimension is equal to the dimension of the input fusion feature , the subtype prompt space is trained.

7. A multimodal colony sample fine-grained classification system based on prompt increment, characterized by: include: A sample acquisition module acquires multimodal microbial plate images and uses them as samples; A sequence generation module scales and normalizes the samples, then blocks and converts the processed samples into sequences, concatenates them with the learned category representation parameters, and adds position encoding as a new feature sequence; A feature category acquisition module feeds the new feature sequence into the first ViT encoder to obtain discriminative fusion features and coarse category representation; a strain classification acquisition module, which inputs the coarse category representation into a linear classifier to obtain a strain classification result; A prompt feature acquisition module queries and acquires corresponding prompt features in the subtype prompt space according to the bacterial species classification result; The category acquisition module concatenates the discriminative fusion features with the hint features as new input, inputs the new input to the second ViT encoder, performs average pooling on the output hint features, and then represents them as fine categories; The classification acquisition module inputs the sub-category representation into the linear classifier to obtain a sub-type classification result.

8. A multimodal colony sample fine-grained classification system based on hint increment according to claim 7, characterized in that: Also includes: The update module calculates the loss value between the subtype classification result and the sample true value label during fine-tuning training, fixes the backbone model parameters of the second ViT encoder, and updates the prompt features corresponding to the subtype prompt space and the linear classifier according to the loss value.

Images