Automatic colony image counting method based on U2-Net and Resnet50

Abstract

The present invention discloses a method for automatically counting colony images based on U2‑Net and Resnet50, and the steps of identification and counting are as follows: (1) culture dish edge segmentation, i.e., using the U2net model to extract the culture dish edge, overlapping the extracted edge with the original image to separate the area where the culture medium in the center of the culture dish is located, and performing the next step of processing as ROI; (2) colony unit segmentation, i.e., using the U2net model to extract colonies from the ROI area, with the colony as the foreground and the culture medium area as the background, segmenting the output threshold of U2net to obtain a colony mask, and then searching through the connected area, each connected area is used as a colony unit, which is rotated and input into subsequent steps after standardization; (3) colony unit counting, i.e., using ResNet50 to extract features from the colony unit, and outputting the number of colonies contained in the colony unit according to the extracted features. The present invention has high versatility without adjusting hyperparameters, i.e., it has lower dependence on operating personnel, and the whole process requires almost no human intervention.

Description

Technical Field

[0001] The present invention relates to an automatic colony image counting method based on U2-Net and Resnet50, and belongs to the technical field of biological testing. Background Art

[0002] Colony counting and bacterial species identification are the basis for determining the microbial content or contamination level in the environment or product being inspected. They are widely used in food quality and hygiene testing, water and air quality analysis, yogurt product testing, hospital clinical testing, cosmetics testing, drug quality testing, and microbiology laboratory teaching in universities. Currently, domestic disease control and food and drug supervision inspection and quarantine agencies, as well as undergraduate teaching and research laboratories, still choose to count colonies after cultivation manually. This method is inefficient, prone to errors and omissions, and difficult to preserve original records. Based on the latest advances in image recognition algorithm research, it has become possible to apply it to the field of colony image analysis. The development of colony counting software can assist manual counting and significantly advance the field of microbial testing. At present, there are still deficiencies in the processing of colony images after cultivation on microbial plate culture media at home and abroad.

[0003] In the implementation of automatic colony counting software, colony image segmentation is a key technology. Image segmentation not only includes the segmentation of colonies and background, but also the segmentation of adhesion colonies. The segmentation of colonies and background is generally based on thresholds, edges, clustering, regions, morphology, graph theory, specific theories, etc., and can be directly segmented through edge detection algorithms (Sobel, Canny, Prewitt, etc.), region growing methods and threshold segmentation methods, but these algorithms cannot be directly applied to the counting of adhesion colonies. There are currently two main methods for identifying adhesion colonies. One is based on image processing algorithms, mainly including watershed algorithms, and algorithms based on pit detection. However, traditional algorithms are not suitable for situations with irregular shapes, large differences in target size, and a large number of adhesions. The other type of artificial intelligence based on deep learning of convolutional neural networks is more adaptive and self-learning, and can extract more deep information. The deep learning architecture based on convolutional neural networks is a relatively advanced means to solve the problem of target counting. Unlike traditional algorithms that require the use of complex algorithms, directly sending images into the neural network for feature extraction can greatly reduce the loss information rate, thereby learning more features than traditional algorithms, which is of great significance for improving accuracy. Summary of the Invention

[0004] The purpose of the present invention is to provide an automatic colony image counting method based on U2-Net and Resnet50. This method first uses the light intensity of the culture medium area to correct the light intensity value of the colony area, and uses the U2-Net network to find the edge of the culture dish and segment the colonies and background. Then, the Resnet50 network is used to realize the image processing technology and process of colony feature extraction and recognition counting.

[0005] The technical solution adopted by the present invention is: a method for automatically counting bacterial colony images based on U2-Net and Resnet50, which comprises the following steps:

[0006] Step 1: Image acquisition: using an imaging system to capture real-time images of the culture dish;

[0007] Step 2: Image pre-processing: The captured culture dish data image is subjected to fuzzy denoising, differential transformation, and threshold segmentation steps to readjust the image absorbance;

[0008] Step 3: Colony identification and counting. The process of this step is as follows:

[0009] The first step is to segment the edge of the culture dish. This involves using the U2net model to extract the edge of the culture dish. The extracted edge is then overlapped with the original image to separate the center of the culture dish where the culture medium is located, which is used as the ROI for the next step of processing.

[0010] The second step is colony unit segmentation. This involves using the U2net model to extract colonies from the ROI area, with the colonies as the foreground and the culture medium area as the background. The U2net output threshold is segmented to obtain a colony mask, which is then searched for connected regions. Each connected region is considered a colony unit and rotated to a uniform size of 128*128*3 for input into subsequent steps.

[0011] The third step is colony unit counting. That is, for the colony units in the second step, ResNet50 is used to extract features, and the number of colonies contained in the colony units is output based on the extracted features.

[0012] Furthermore, in step one, the imaging system includes a black box divided into two parts, and the top is transparent. The upper transparent area is used to place a high-definition CMOS camera, a Raspberry Pi, and a data cable. The high-definition CMOS camera is connected to the Raspberry Pi, and the Raspberry Pi is used to process the data collected by the camera.

[0013] Furthermore, in the first step of step three, the preprocessed colony image is manually segmented to obtain the edge of the culture dish as a label, and two approximate circles are used to approximate the inner and outer edges of the culture dish; the inner and outer edges are selected and the value between the two circles is filled; these labels are used for training and verification respectively.

[0014] Furthermore, during training, the optimizer uses AdamW and the weight-decay parameter is set to 1e-4;

[0015] The learning rate decay strategy of the model is:

[0016]

[0017] where η t is the learning rate of the tth iteration, η max is the maximum learning rate, set to 1, η min is the minimum learning rate, set to 1e-6, and T is the total number of iterations;

[0018] The loss function uses weighted binary cross entropy:

[0019] L c ={l 1,c ,…,l n,c},l n,c =-w n,c [p c y n,c ·logσ(x n,c )+(1-y n,c )·log(1-σ(x n,c ))]

[0020] Where n is batch size, y n is the true value, σ is sigmoid, x n is the predicted value, c is the category of the image, p c is the weight of category c;

[0021] Use MAE and F1 scores as evaluation indicators of the model to evaluate the segmentation effect on the validation set:

[0022]

[0023]

[0024]

[0025]

[0026] Where n is batch size, h(x i ) is the predicted value, y i is the true value, TP, FN, and FP are true positive (TruePositive), false negative (FalseNegative), and false positive (FalsePositive), respectively.

[0027] Furthermore, in the second step of step 3, the colony mask extracted by U2net is first segmented by threshold to obtain a grayscale image with a value of 0 or 255. This grayscale image is then searched for connected regions to obtain multiple connected regions. Each connected region is defined as a colony unit, and the colony unit is rotated as the input of the colony counting network.

[0028] Furthermore, in the third step of step three, all colony unit images are marked from 0 to 9 according to the number of colonies, wherein a mark of 1 indicates that there is one colony, but a mark of 9 indicates 9 or more colonies.

[0029] Furthermore, in the third step of the step three, the colony unit image after U2net segmentation in the previous step is subjected to ellipse fitting to find the rotation center point O and the rotation angle θ; the image is rotated by angle θ with O as the center point.

[0030] Furthermore, in the third step of step three, a CNN network is used to count the adhered colonies, that is, CNN is used to extract features, and then a counting operation is performed based on the features.

[0031] Furthermore, a ResNet network was trained to count the number of colonies in each colony unit, with separate training and validation sets for each category. Adam was used as the optimizer and weighted binary cross entropy as the loss function.

[0032] Furthermore, weights are introduced when calculating loss, and the weight of each category is calculated as:

[0033]

[0034] where w i is the weight of label i, n i is the amount of data with label i;

[0035] The model is evaluated on the validation set using the micro F1 score and macro F1 score as evaluation indicators:

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042] Where n represents the total number of categories, TPi represents the TP (true positive) of the i-th category, FNi, and FPi represent the FN (false negative) and FP (false positive) of the i-th category respectively.

[0043] Compared with existing technologies, the beneficial effects of the present invention are: compared with traditional algorithms and some current CNN processing methods, the present method has improved the recognition rate of adherent colonies, and the colony counting of the present method does not require the adjustment of hyperparameters to have high versatility, that is, it is less dependent on the operator, and the entire process requires almost no human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of obtaining absorbance data of colony area;

[0045] Figure 2 This is the colony counting flow chart (from the upper left corner in the order of arrows: original image, u2net, culture dish edge image, culture dish center ROI, u2net, colony mask, colony unit, ResNet, statistical results);

[0046] Figure 3 is an example of a Petri dish edge segmentation input / label;

[0047] Figure 4 is a sample colony segmentation input / label;

[0048] Figure 5 It is a sample of colony unit;

[0049] Figure 6 Is the proportion of colony area to the entire map

[0050] Figure 7 It is a schematic diagram of the rotation of the colony unit;

[0051] Figure 8 Evaluation results on the validation set for petri dish edge segmentation. The left y-axis indicates the F1 score, and the right y-axis indicates the MAE score.

[0052] Figure 9 This is the loss reduction curve in the segmentation of the petri dish edge, and the horizontal axis is the number of iterations;

[0053] Figure 10 The ratio of the culture dish edge area to the whole image in the culture dish edge segmentation;

[0054] Figure 11 Evaluation results on the validation set for colony unit segmentation. The left y-axis indicates the F1 score, and the right y-axis indicates the MAE score.

[0055] Figure 12 It is the loss decline curve in the segmentation of colony units, and the horizontal axis is epoch;

[0056] Figure 13 This is a heat map of colony counts on the validation set;

[0057] Figure 14 Heatmap of colony counts in the training set. DETAILED DESCRIPTION

[0058] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] 1. Materials and Methods

[0060] 1.1 Colony culture

[0061] Experimental Materials: Seven laboratory-preserved standard strains of common foodborne pathogens were used: Escherichia coli (ATCC 25922), Listeria ivanovii (ATCC 19119), Listeria monocytogenes (ATCC 19115), Staphylococcus aureus (ATCC 6538), Staphylococcus epidermidis (ATCC 12228), Salmonella typhimurium (ATCC 14028), and Shigella sp. (ATCC 12038). Culture Method: These strains were activated from culture collection tubes and single colonies were obtained. Single colonies were collected and suspended in saline. The concentration of the suspension was estimated using a McFarland turbidimeter tube. The suspension was then diluted with sterile saline. Use the plate spreading separation method, take a certain amount of bacterial suspension and spread it on the PCA agar counting plate, spread it with a spreading stick until the bacterial liquid is dry, and then place it in a 37℃ incubator for 48 hours, trying to ensure that the number of colonies on each plate is between 10-300 CFU.

[0062] 1.2 Image Acquisition

[0063] Imaging system: A high-pixel CMOS camera, a 2.8-12mm, 1 / 2 manual zoom wide-angle lens, and a 12V DC frequency-modulated white backlight (LED light source) are assembled in a self-designed two-layer fully enclosed box and connected to a computer with a photo-taking program installed.

[0064] The imaging system hardware consists of a two-part black box with a transparent top. The upper transparent area primarily houses the high-definition CMOS camera, a Raspberry Pi, and the necessary data cables connecting the camera to the Pi. This module integrates wireless transmission and data processing capabilities. The wireless transmission, powered by the Raspberry Pi, is used to transmit data over long distances to data centers and other computers, while the Raspberry Pi processes the data collected by the camera. A central hole at the junction of the transparent section and the black darkroom allows the lens to penetrate the darkroom. The lower section is the darkroom, with a backlight panel installed at the bottom to provide LED light for dark field viewing. On the front of the darkroom is a sliding door with a handle that can be opened to place or replace culture dishes for testing.

[0065] Image Acquisition: Using the imaging system constructed above, place the plate to be tested on a white backlit stage. In a closed light box, fix the viewing angle, focus, and background. Once the light source and instrument are stable, adjust the focus to achieve optimal clarity. Using a computer camera program, capture 100 frames and select the best one to obtain a real-time image of the culture dish.

[0066] 1.3 Training Environment

[0067] This method was developed using PyCharm on Windows 10. The recommended runtime environment is Python 3.9, OpenCV-Python 4.5.5.64, NumPy 1.23.3, Torch 1.12.1, and TorchVision 0.13.1. The processor used in this method is an Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz (4 processors) with 72GB of RAM.

[0068] 1.4 Image Preprocessing

[0069] Acquisition of colony area absorbance data; Figure 1 As shown in the figure, the culture dish data image obtained by shooting is segmented from the image through fuzzy denoising, differential transformation, threshold segmentation and other steps. The edge search is performed on the culture dish to separate the colony area and the culture medium area. The average brightness of all pixels in the culture medium area is represented by I0, and the average brightness of all pixels in each colony area on the culture dish is represented by I. The actual absorbance value of each colony area is then expressed as lgI0-lgI.

[0070] 1.5 Colony identification and counting

[0071] The colony recognition and counting process is divided into three steps: the first step is the segmentation of the culture dish edge, that is, the use of the U2net model to extract the culture dish edge, and the extracted edge is overlapped with the original image to separate the area where the culture medium is located in the center of the culture dish, which is used as the ROI for the next step of processing; the second step is the segmentation of the colony unit, that is, the use of the U2net model to extract the colonies in the ROI area, with the colony as the foreground and the culture medium area as the background. The output threshold of U2net is segmented to obtain the colony mask, and then the connected area is searched. Each connected area is regarded as a colony unit, which is rotated and unified into a 128*128*3 image for input into the subsequent steps; the third step is the counting of colony units, that is, for the colony units in the second step, ResNet50 is used for feature extraction, and the number of colonies contained in the colony unit is output based on the extracted features.

[0072] 1.5.1. Petri dish edge segmentation

[0073] 1.5.1.1. Dataset preparation and network selection

[0074] The edge of the culture dish often affects colony segmentation, so the center area of ​​the culture dish is generally used as the ROI for colony identification, which helps reduce the false positive rate of colonies. Traditional ROI area identification generally uses threshold segmentation and k-means clustering algorithms. We use the u2net network to extract the edge of the culture dish to obtain the ROI, thereby further realizing colony identification. We manually segment the colony image after preprocessing (absorbance processing) to obtain the edge of the culture dish as a label. Here, we use two approximate circles to approximate the inner and outer edges of the culture dish. After selecting the inner and outer edges, the value between the two circles is filled with 255. We prepared 255 colony images, and these colony images were manually segmented to obtain the edge of the culture dish as a label. 219 of them were used for training and 36 for validation.

[0075] U2Net is a network structure based on the Unet architecture, widely used for foreground recognition in images. Based on the encoder-decoder architecture, it draws on the FPN and Unet architectures. A new module, RSU (Residual U-blocks), was proposed. After testing, a new module, called RSU, was proposed. Each RSU is a small Unet, and all RSUs are connected together using a structure similar to FPN, similar to down-top top-down. This approach increases multi-scale capabilities and achieves excellent segmentation results. The U2Net model used in this method is the complete version officially provided by PyTorch.

[0076] 1.5.1.2 Detailed training

[0077] All images were resized to half their original dimensions (1250 x 961 x 3) to reduce video memory usage during training. We rescaled the absorbance of the images during image preprocessing, so no mean and standard deviation normalization was performed. Instead, the images were divided by 255 and their RBG channels were scaled to between 0 and 1 for network input.

[0078] U2net does not use any pre-trained weights. During training, the optimizer uses AdamW (lr = 1e-3, betas = (0.9, 0.999), eps = 1e-8), and the weight-decay parameter is set to 1e-4. The learning rate decay strategy of the model is Cosine learning rate decay (Formula (6), η t is the learning rate of the tth iteration, η max is the maximum learning rate, set to 1, η min The loss function uses the weighted binary cross entropy (BCEWithLogitsLoss, formula (5), n is the batch size, y n is the true value, σ is sigmoid, x n is the predicted value, c is the category of the image, p c is the weight of category c;). According to statistics, the area of ​​the edge of the culture dish in an image is about 1 / 9 of the entire image. Therefore, in order to balance the area of ​​the foreground and background, we set the weight of the foreground and background in BCEWithLogitsLoss to 8:1. To evaluate the prediction effect of the model, we use mae (Formula (4), n is the batch size, h (x i ) is the predicted value, y i is the true value) and f1 score (Formula (3), precision, recall are Formulas (2) and (1), TP, FN, FP are true positive, false negative and false positive respectively) as the evaluation indicators of the model, and the segmentation effect is evaluated on the validation set.

[0079]

[0080]

[0081]

[0082]

[0083] L c ={l 1,c ,…,l n,c},l n,c =-wn,c [p c y n,c ·logσ(x n,c )+(1-y n,c )·log(1-σ(x n,c ))](5)

[0084]

[0085] 1.5.2 Colony Unit Separation

[0086] 1.5.2.1, Colony segments dataset

[0087] 1.5.2.1.1 Segmentation

[0088] We manually annotated all colonies in the 255 images and used them to train a U2Net network for colony extraction. All contiguous colonies were treated as a single entity, with only the outer edges labeled, not the colony boundaries. Furthermore, because the input images for this U2Net were segmented along the edges of the culture dish, colonies that overlapped with the edges were not labeled.

[0089] 1.5.2.1.2、Dataset creation

[0090] We use the original image as input and the binary classification mask as the label to generate the dataset. This splits the colony mask extraction into two steps: first removing the edge of the culture dish and then extracting the colonies to reduce false positives in the extracted colonies.

[0091] 1.5.2.1.3 Dataprocess

[0092] We use horizontal flipping on the images (flip ratio 0.5) to increase the amount of data.

[0093] All images are resized to 1 / 2 of their original size (1250 x 961).

[0094] The 3 channels of the image are scaled down to 0-1 (divided by 255).

[0095] 1.5.2.2 Detailed training

[0096] To segment the colony units, we first trained a U2Net to extract the colonies for subsequent segmentation. We used the center region of the Petri dish, obtained by segmenting the edges, as input and manually segmented colony masks as labels to train the U2Net network. 221 images were used for training and 34 for validation.

[0097] In the evaluation of U2Net, we still use MAE and F1 scores as indicators and evaluate them on the validation set.

[0098] The colony mask extracted by u2net is first segmented by threshold (thresh = 0.99, and the effects of different thresholds on the segmentation results are compared in the results section) to obtain a grayscale image with a value of 0 or 255. This grayscale image is then searched for connected areas to obtain multiple connected areas. Each connected area is defined as a colony unit, and each colony unit has 0 to multiple colonies (most colony units have only 1 colony).

[0099] We also calculated the colony area ratio and gave different weights (10:1) to the foreground and background (colonies and background) when calculating the loss to balance the imbalance in the number of samples. Figure 5 is a colony unit example, where two colonies are divided into the same colony unit, and the colony unit is rotated and used as the input of the colony counting network.

[0100] 1.5.3 Colony unit counting

[0101] 1.5.3.1, Colony counting dataset

[0102] 1.5.3.1.1 Segmentation

[0103] Each preprocessed image is first segmented using the Petri dish edge segmentation algorithm to extract the Petri dish edge. After threshold segmentation (thresh = 0.1), the original image and the Petri dish edge map are overlapped to separate the Petri dish center region map. For colony unit segmentation, the colony mask is first extracted using U2Net. The colony mask is then thresholded (thresh = 0.99) and a connected component search algorithm is used to obtain several independent connected regions, namely the colony unit map.

[0104] 1.5.3.1.2 Data creation

[0105] All colony unit images were manually labeled from 0 to 9 based on the number of colonies, with a label of 1 indicating one colony and a label of 9 indicating nine or more colonies. The number of samples in each category is shown in the figure. Although U2net was used to segment the colony region foreground, some non-foreground areas were still misidentified and the misidentified areas were marked as 0. The colony unit images were then rotated. To ensure consistency in the dataset input, spatial normalization was employed instead of traditional dataset augmentation through rotation and flipping. After the U2net segmentation in the previous step, the colony unit images were fitted with an ellipse to find the rotation center point O and the rotation angle θ. The image was rotated by angle θ with O as the center point. Spatial normalization helps reduce the influence of colony position on ResNet.

[0106] Each image is resized to 128x128 in both length and width. Images smaller than 128x128 are padded to 128x128 by copying the edge pixels. Images larger than 128x128 are first resized to 128 on their long side, scaled to the same ratio on their short side, and then padded to 128 on both sides. All three channels of the image are scaled to 0-1 (divided by 255).

[0107] 1.5.3.2 Network preparation

[0108] Res is the abbreviation of Residual. Its purpose is to make the neural network deeper and more accurate based on residual learning. First of all, ResNet has 5 different depth structures in the official code of PyTorch, with depths of 18, 34, 50, 101, and 152 respectively (the depth of various networks refers to the number of layers that "need to update parameters through training", such as convolutional layers, fully connected layers, etc.). Due to the differences in running speed and classification accuracy among various versions, we finally chose ResNet50. The output channel of the last layer of the Resnet we selected was modified to 10, which is used to predict 10 categories of 0-9 in each colony unit.

[0109] 1.5.3.3 Detailed training

[0110] The colony units obtained after colony unit segmentation already achieve good segmentation results for most colonies, but a small number of these colony units contain multiple, overlapping colonies. Instead of using the traditional watershed algorithm for further image segmentation, we employed a CNN network to count overlapping colonies. The basic idea is to use a CNN for feature extraction and then perform counting based on these features. To further improve counting accuracy, we trained a ResNet network to count the number of colonies in each colony unit. 255 colony images obtained after colony unit segmentation were used for training and validation. Each image contained zero to multiple colonies, and each image was manually annotated with the number of colonies as labels for training the network. 219 images (70%) were used for training, and 36 (30%) were used for prediction. Separate training and validation sets were used for each class. Adam (lr = 0.0001) was used as the optimizer. Weighted binary cross entropy (BCEWithLogitsLoss) was used as the loss function.

[0111] Due to the imbalance between categories, we also introduce weights (7) when calculating the loss. The weight of each category is calculated as (w i is the weight of label i, n i is the amount of data with label i):

[0112]

[0113] The model is evaluated on the validation set using the micro f1 score (10) and macro f1 score (13). (n represents the total number of categories. Precision micro and Recall micro in formula (10) are derived from formulas (8) and (9), respectively. Precision macro and Recall macro in formula (13) are derived from formulas (11) and (12), respectively.)

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] 2. Results:

[0121] 2.1 Colony culture and image acquisition

[0122] All seven bacterial species were successfully cultured, yielding 127 culture plates containing between 10 and 300 CFU. During photography, the bottom-lit light within our light box prevented reflections often seen with mobile phones or cameras. A total of 255 high-quality, original images of the cultured plates were captured, encompassing all seven bacterial species.

[0123] 2.2. Petri dish edge segmentation

[0124] The loss curve F1 score curve of each epoch on the validation set is as follows Figure 8 and 9 The figure shows the training curve of U2Net for culture medium edge segmentation. The model has an F1 score of 0.995 and a MAE of 0.0033.

[0125] The colony mask is then filtered with a threshold (threshold = 0.1) to obtain a binary (0-255) mask. The edge mask of each dish is extracted. This edge mask serves as training and validation data for the U2Net edge segmentation algorithm. The goal is to segment the colony image using the dish edge as the foreground to extract the culture medium region, eliminating edge blurring caused by the dish region. This means that colonies close to the dish edge will have a noticeably incomplete colony mask, inferior to other colonies.

[0126] The percentage of the edge area in each image to the entire image is as follows Figure 11 , we select the weight ratio of each category in BCEWithweight accordingly to balance the category imbalance.

[0127] Different thresholds can have a significant impact on the generated mask. When the threshold is set to 0.1, the edges of the culture dish can be almost completely removed, while when the threshold is set to 0.9, the edges are not well covered. These unmasked edges affect the subsequent colony segmentation and counting.

[0128] 2.3 Colony Unit Separation

[0129] The training loss curve and F1 curve of each epoch on the validation set are as follows Figure 12 and 11 The figure shows the training curve of U2Net in colony unit segmentation. The model has an F1 score of 0.965 and a MAE of 0.005 on the validation set.

[0130] In addition, the threshold selected during threshold segmentation has a significant impact on the results of colony unit segmentation. We statistically analyzed the results of colony unit segmentation using OpenCV's connected region search algorithm under different segmentation thresholds. This statistics was performed on the validation set (n=48). It can be seen that as the threshold increases, the number of segments decreases, and invalid segments are filtered out (because U2Net is trained to segment colony foregrounds, the larger the value, the more likely it is a colony). This step reduces the pressure on ResNet to a certain extent, because the better segmentation effect can better segment cohesive colonies, and the number of colonies in the colony unit is smaller.

[0131] 2.4 Colony Recogonition and Counting

[0132] The confusion matrix of ResNet trained and validated on the colony count dataset is as follows Figure 13 and 14 The matrix shows that even though clusters of 3-5 colonies sometimes correspond to classes that are misclassified, the incorrectly chosen labels remain close to the main diagonal. For example, clusters of 4 colonies are confused with clusters of 3 or 5 colonies. It is also noted that counting these clusters is often quite difficult, even for trained technicians. Due to insufficient validation set images, some classes were not assigned to colonies. Identification results were manually annotated.

[0133] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the scope of protection of the present invention in any form. All technical solutions obtained by equivalent substitution, etc., fall within the scope of protection of the present invention. Parts not covered by the present invention are the same as the existing technology or can be implemented using existing technology.

Claims

1. A method for automatic colony image counting based on U2-Net and Resnet50, characterized in that The steps include: Step 1: Image acquisition: using an imaging system to capture real-time images of the culture dish; Step 2: Image pre-processing: The captured culture dish data image is subjected to fuzzy denoising, differential transformation, and threshold segmentation steps to readjust the image absorbance; Step 3: Colony identification and counting. The process of this step is as follows: The first step is to segment the edge of the culture dish. This involves using the U2net model to extract the edge of the culture dish. The extracted edge is then overlapped with the original image to separate the center of the culture dish where the culture medium is located, which is used as the ROI for the next step of processing. The second step is colony unit segmentation. This involves using the U2net model to extract colonies from the ROI area, with the colonies as the foreground and the culture medium area as the background. The U2net output threshold is segmented to obtain a colony mask, which is then searched for connected regions. Each connected region is considered a colony unit and rotated to a uniform size of 128*128*3 for input into subsequent steps. The third step is colony unit counting. For the colony units in the second step, ResNet50 is used to extract features, and the number of colonies contained in the colony units is output based on the extracted features. In the second step of step 3, the colony mask extracted by U2net is first segmented by threshold to obtain a grayscale image with a value of 0 or 255. This grayscale image is then searched for connected regions to obtain multiple connected regions. Each connected region is defined as a colony unit, and the colony unit is rotated and used as the input of the colony counting network. In the third step of step three, all colony unit images are labeled from 0 to 9 according to the number of colonies, where 1 indicates one colony and 9 indicates nine or more colonies; after the colony unit image is segmented by U2net in the previous step, the center point O and the rotation angle θ are found by fitting an ellipse; the image is rotated around O by an angle θ; and the adhering colonies are counted using a CNN network, i.e., CNN is used to extract features and then counting is performed based on the features; A resnet network was trained to calculate the number of colonies in each colony unit. Each category was divided into training and validation sets separately. The optimizer used Adam and the loss function used weighted binary cross entropy.

2. The automatic colony image counting method based on U2-Net and Resnet50 according to claim 1 is characterized in that: In step 1, the imaging system includes a black box divided into two parts, and the top is transparent. The upper transparent area is used to place a high-definition CMOS camera, a Raspberry Pi, and a data cable. The high-definition CMOS camera is connected to the Raspberry Pi, and the Raspberry Pi is used to process the data collected by the camera.

3. The automatic colony image counting method based on U2-Net and Resnet50 according to claim 1 is characterized in that In the first step of step three, the pre-processed colony image is manually segmented to obtain the edge of the culture dish as a label, and two approximate circles are used to approximate the inner and outer edges of the culture dish; Select the inner and outer edges and fill the value in the middle of the two circles; use these labels for training and validation respectively.

4. The automatic colony image counting method based on U2-Net and Resnet50 according to claim 3 is characterized in that During training, the optimizer uses AdamW and the weight-decay parameter is set to 1e-4; The learning rate decay strategy of the model is: where η t is the learning rate of the tth iteration, η max is the maximum learning rate, set to 1, η min is the minimum learning rate, set to 1e-6, and T is the total number of iterations; The loss function uses weighted binary cross entropy: L c ={l 1,c ,…,l n,c },l n,c =-w n,c [p c y n,c ·logσ(x n,c )+(1-y n,c )·log(1-σ(x n,c ))] Where n is batch size, y n is the true value, σ is sigmoid, x n is the predicted value, c is the category of the image, p c is the weight of category c; Use MAE and F1 scores as evaluation indicators of the model to evaluate the segmentation effect on the validation set: Where n is batch size, h(x i ) is the predicted value, y i is the true value, TP, FN, and FP are true positive, false negative, and false positive, respectively.

5. The automatic colony image counting method based on U2-Net and Resnet50 according to claim 1 is characterized in that: Weights are introduced when calculating loss, and the weight of each category is calculated as: where w i is the weight of label i, n i is the amount of data with label i; The model is evaluated on the validation set using the micro F1 score and macro F1 score as evaluation indicators: Where n represents the total number of categories, TPi represents the true positive of the i-th category, FNi, and FPi represent the false negative and false positive of the i-th category respectively.

Images