A method, device and platform for counting colony numbers based on artificial intelligence

Through the combination of artificial intelligence and neural network model CBAM-YOLOV5, the automation and high-precision recognition of bacterial counting are achieved, solving the problem of large bacterial counting errors in the existing technology, and improving work efficiency and recognition accuracy.

CN114119580BActive Publication Date: 2024-11-26SHANGHAI OCEAN UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202111454152.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-01
Publication Date
2024-11-26
Estimated Expiration
2041-12-01

AI Technical Summary

Technical Problem

In the prior art, bacterial counting methods are limited to colony counting, making it difficult to analyze and maintain data, and the error rate of manual counting is large, so the number of bacteria cannot be effectively identified.

Method used

An automatic colony counting device based on artificial intelligence is adopted, and bacterial image recognition and counting is used to use high-performance GPU and neural network model CBAM-YOLOV5, and the recognition results are corrected through manual intervention to improve detection accuracy.

Benefits of technology

It realizes automation of bacteria counting, improves work efficiency, reduces labor costs, significantly improves recognition accuracy, and improves the accuracy of the data set through continuous correction and training.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114119580B_ABST
    Figure CN114119580B_ABST
Patent Text Reader

Abstract

A colony counting method includes obtaining an original bacterial image of a colony to be counted; preprocessing the original bacterial image to obtain a first bacterial image; inputting the first bacterial image into a trained colony counting model to obtain the number of bacteria after counting. The first bacterial image is input into the colony counting model to obtain a second bacterial image, and a predicted confidence is obtained. According to the confidence, a bacterial detection target is set to a different color, and the bacterial target is verified. The colony counting model is a hybrid domain attention module-target detection model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and microbial science and technology, and in particular to a colony counting method, device and platform based on artificial intelligence. Background Art

[0002] Bacteria are small and numerous organisms. In microbiological experiments, it is often necessary to count bacteria and understand their bacterial density. At present, the counting of colonies in the test is generally done by manual counting, that is, combining a colony counter and an amplifier to record the number of pressure pulses generated when the counting pen hits the colony. This method is limited to the counting of colonies, which is not conducive to the subsequent data analysis and maintenance, and the error rate of manual counting is relatively large. Summary of the invention

[0003] One of the embodiments of the present invention discloses an automatic colony counting device using artificial intelligence, including a microscope, a computer with a high-performance GPU, and a neural network model CBAM-YOLOV5 suitable for detecting small targets such as bacteria. The colonies are identified and marked through the neural network, and human intervention is added to correct the identification results. The detection accuracy is continuously improved through continuous correction and training, thereby achieving high-precision colony identification and statistics.

[0004] The colony counting device of the present invention realizes automatic counting of colonies with the help of artificial intelligence and image processing, which not only improves work efficiency but also realizes timely adjustment of recognition results, and has the advantages of reducing labor costs and improving recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0006] Figure 1 This is a schematic diagram of the structure of a neural network model for counting colony numbers according to one embodiment of the present invention.

[0007] Figure 2 This is a flow chart of automatically counting the number of bacterial colonies using artificial intelligence according to one embodiment of the present invention.

[0008] Figure 3 -a, 3-b, 3-c, and 3-d are schematic diagrams of examples of detection results using artificial intelligence to automatically count the number of colonies according to an embodiment of the present invention. DETAILED DESCRIPTION

[0009] In view of the problems of existing colony statistics, the present invention designs a device suitable for automatic colony counting, which can realize automatic counting of colonies, reduce work intensity with the help of artificial intelligence and image processing, add manual intervention, and realize timely adjustment of recognition results, which has the characteristics of reducing labor costs and improving recognition accuracy.

[0010] The method for automatically counting the number of bacterial colonies of the present invention, according to the model after neural network training, the collected bacterial colony image is automatically generated on the PC end. Editable identification images to be modified, wherein if the recognition confidence is greater than 80, the identification frame is red (its color can be changed, with convenient identification as the selection principle); otherwise it is yellow, and the PC end presents the recognition result, and the detection personnel can use the mouse to manually edit the identification frame. If it is considered to be the required bacteria after the detection personnel check, the color of its identification frame is manually changed to red; if it is considered not to be the required bacteria after the detection personnel check, the identification frame is manually deleted to obtain the final bacterial identification map, and the annotation file of the identification frame is generated at the same time, which is in txt format, and the system will send the modified image and txt annotation file to the optimized yolov5 model for training. Finally, the red pixel value in the bacterial colony picture is used to count the number of identification frames to obtain the number of bacteria. With continuous correction and continuous training, the data set can be made more accurate, and the training result can be more precise. The method can save manpower to a great extent and continuously improve the recognition accuracy.

[0011] According to one or more embodiments, a bacterial image recognition and bacterial quantity counting method based on YOLOV5 includes the following steps:

[0012] Step 1: Obtain the original bacterial data through the microscope camera and preprocess the original data. The preprocessing process mainly includes: data segmentation and cell separation.

[0013] Step 2: Select a portion of the original data and use the manual labeling method as the sample library.

[0014] Step 3: Randomly select data from the sample library in a ratio of 9:1, where 9 is the training set and 1 is the test set.

[0015] Step 4: Use the training set to train and extract features of the improved hybrid domain attention module-object detection model (CBAM-YOLOV5). In this process, the weights of the extracted features are adjusted through the hybrid domain attention module CBAM-Block, and the trained YOLOV5 model is tested through the test set.

[0016] Step 5: Perform target detection on bacterial images through the trained and tested hybrid domain attention module-target detection model (CBAM-YOLOV5).

[0017] Step 6: Send the image to be detected to the hybrid domain attention module-target detection model (CBAM-YOLOV5) for prediction, and obtain an editable recognition image to be modified on the PC.

[0018] Step 7: The model assigns different colors to the prediction boxes based on the confidence level of the recognized image.

[0019] Step 8: Professionals manually determine whether it is the desired target based on the color of the prediction box.

[0020] Step 9: Send the modified data to the hybrid domain attention module-target detection model (CBAM-YOLOV5) for training to improve training accuracy.

[0021] Step 10: Count the number of modified prediction boxes to get the number of bacteria.

[0022] YOLO is an end-to-end network that converts the prediction problem into a regression problem, directly regresses the target's coordinates and category, and avoids the step of extracting redundant candidate boxes. It has a fast detection speed and is suitable for embedded devices, but its accuracy is not as good as that of the SSD network. Therefore, the yolov5 network is improved to ensure the detection speed while improving the detection accuracy of small targets.

[0023] As a preferred solution, the hybrid domain attention module-target detection model (CBAM-YOLOV5) is composed of Backbone, Neck and Detect, Backbone includes Focus slice structure, the first convolution layer + BN + Leaky_ReLU activation function (CBL_1), the first hybrid domain attention module CBAM_1, the first layer CSP1, the second convolution layer + BN + Leaky_ReLU activation function (CBL_2), the second hybrid domain attention module CBAM_2, the second layer CSP2, the third convolution layer + BN + Leaky_ReLU activation function (CBL_3), the third hybrid domain attention module CBAM_3, the third layer CSP3, and the pooling layer SPP; Neck includes CSP layer, upsampling layer Upsampling and convolution layer + BN + Leaky_ReLU activation function (CBL_*);

[0024] Furthermore, the specific operations of step one include:

[0025] Step (1) constructs a k*k sliding window, where k can divide the original data.

[0026] Step (2) slides on the original data with a step size of k. Assuming that the size of the original data is b*b, the original data is divided into b^2 / k^2 data blocks.

[0027] Step (3) performs erosion-dilation operations on b^2 / k^2 data blocks in batches using the OpenCV image processing algorithm, setting the erosion kernel to (5*5), the dilation kernel to (3*3), and the number of iterations to 3, which can separate connected cells and facilitate subsequent calibration and counting.

[0028] The specific operations of step 4 include:

[0029] Step (1) inputs the training set into the Focus structure of the pre-established YOLOV5 network for slicing to reduce the amount of calculation.

[0030] Step (2) The feature map obtained by slicing and concat of the Focus structure passes through the first convolution layer group (CBL_1); then the obtained feature map is inferred along the spatial and channel dimensions in the first layer of mixed domain attention module (CBAM_1) in turn to obtain the attention weights, and multiplied with the features after the first layer of convolution to adaptively adjust the features, completing the recalibration of the features after the first layer of convolution. Then the output feature map is obtained through the CSP1 residual structure (CSP1_1).

[0031] Step (3) inputs the feature map output by step (2) into the second convolutional layer group (CBL_2) for processing.

[0032] Step (4) The second mixed domain attention module (CBAM_2) infers the attention weights of the feature map obtained by processing the second convolutional layer group (CBL_2) along the spatial and channel dimensions within the second mixed domain attention module (CBAM_2) in sequence, and multiplies it with the feature map obtained by the second convolutional layer group (CBL_2) to adaptively adjust the feature map, thereby completing the recalibration of the feature map obtained by the second convolutional layer group (CBL_2); then the recalibrated feature map is processed using the CSP2 residual structure (CSP1_3) to obtain the output feature map.

[0033] Step (5) Input the feature map outputted from step (4) into the third convolutional layer group (CBL_3) for processing. Step (6) The third mixed domain attention module (CBAM_3) sequentially infers the attention weights along the spatial and channel dimensions of the feature map processed by the third convolutional layer group (CBL_3), and multiplies the feature map obtained by the third convolutional layer group (CBL_3) to adaptively adjust the feature map, thereby completing the recalibration of the feature map outputted from the third convolutional layer group (CBL_3); then the recalibrated feature map is processed through the CSP3 residual structure (CSP1_3) to obtain the output feature map. Step (7) Input the features adjusted by the third mixed domain attention module CBAM_3 and the CSP3 residual structure (CSP1_3) in step (6) into the fourth convolutional layer group (CBL_4), the fifth convolutional layer group (CBL_5) and the maximum pooling layer SPP.

[0034] Step (8) passes the feature map obtained in step (7) through the sixth convolution layer group (CBL_6), the CSP4 residual structure (CSP2_1) and the seventh convolution layer group (CBL_7) in sequence, and then, after upsampling, the feature map is expanded and then feature fused (Concat) with the feature map output by the CSP3 residual structure (CSP1_3).

[0035] Step (9) Concat the feature map obtained in step (8) through the CSP5 residual structure (CSP2_1) and the eighth convolution layer group (CBL_8) in sequence, and then, after upsampling to expand the feature map, perform feature fusion with the second layer CSP2 residual structure (CSP1_3).

[0036] Step (10) passes the features obtained in step (9) through the CSP6 residual structure (CSP2_1) and the ninth convolution layer group (CBL_9) in sequence, and then, after upsampling, the feature map is expanded and fused with the feature map output by the first layer CSP1 residual structure (CSP1_1).

[0037] Step (11) passes the fused feature map of step (10) through the seventh residual structure (CSP2_1) of CSP7 and the convolution layer Conv to obtain the prediction grid 1 (152*152*256).

[0038] Step (12) performs feature fusion on the feature map of step (11) after passing through the seventh-layer residual structure of CSP7 through the tenth-layer convolution layer group (CBL_10) and the ninth-layer convolution layer group (CBL_9).

[0039] Step (13) passes the fused feature map of step (12) through the eighth residual structure (CSP2_1) of CSP8 and the convolution layer Conv to obtain the prediction grid 2 (76*76*255).

[0040] Step (14) performs feature fusion on the feature map output by the eleventh convolutional layer group (CBL_11) in step (13) and the eighth convolutional layer group (CBL_8).

[0041] Step (15) passes the fused feature map of step (14) through the ninth residual structure (CSP2_1) of CSP9 and the convolution layer Conv to obtain the prediction grid 3. (38*38*255)

[0042] Step (16) performs feature fusion on the feature map output by the twelfth convolutional layer group (CBL_12) in step (15) and the seventh convolutional layer group (CBL_7).

[0043] Step (17) The feature map fused in step (16) is passed through the tenth layer residual structure of CSP10 to obtain the prediction grid 4 (19*19*255). Use the generated 4 prediction networks to detect bacteria. Since bacteria are small targets, small target detection has always been a difficult problem in the recognition of neural networks. This paper adds a mixed domain attention module* (CBAM) and increases the original three detection layers to four detection layers. This operation can better identify small targets, which is conducive to the detection of small targets such as bacteria. Step (18) Input the test set into the trained network to obtain the predicted label.

[0044] Step (19) trains each batch of samples in turn, repeating steps (1)-(17) until the network converges and the network training process is completed; an editable prediction box image with confidence is obtained.

[0045] Furthermore, in step (2), the features obtained by the first convolution layer are used to infer the attention weights in the spatial and channel dimensions of the first mixed domain attention module, and are multiplied with the features after the first convolution layer to adaptively adjust the features; the formula is: Where F represents the feature map, Mc and Ms represent channel-based and space-based attention respectively. represents element-by-element multiplication, F' and F" represent the output feature maps after channel attention and spatial attention, respectively.

[0046] According to one or more embodiments, a device for automatically counting the number of colonies using artificial intelligence includes a neural network. The device is provided with a microscope camera and a computer. The microscope collects images of colonies to be counted, and data is annotated on the images on the PC. The computer has a high-performance GPU. The neural network is optimized based on yolov5, and a hybrid domain attention module-target detection model (CBAM-YOLOV5) is designed, which has a more accurate detection effect on small targets such as bacteria.

[0047] According to one or more embodiments, Figure 1 , a neural network structure in a device for automatically counting the number of bacterial colonies using artificial intelligence includes the following steps:

[0048] Step (1) inputs the training set into the Focus structure of the pre-established YOLOV5 network for slicing to reduce the amount of calculation.

[0049] Step (2) The feature map obtained by slicing and concat of the Focus structure passes through the first convolution layer group (CBL_1); then the obtained feature map is inferred along the spatial and channel dimensions in the first layer of mixed domain attention module (CBAM_1) in turn to obtain the attention weights, and multiplied with the features after the first layer of convolution to adaptively adjust the features, completing the recalibration of the features after the first layer of convolution. Then the output feature map is obtained through the CSP1 residual structure (CSP1_1).

[0050] Step (3) inputs the feature map output by step (2) into the second convolutional layer group (CBL_2) for processing.

[0051] Step (4) The second mixed domain attention module (CBAM_2) infers the attention weights of the feature map obtained by processing the second convolutional layer group (CBL_2) along the spatial and channel dimensions within the second mixed domain attention module (CBAM_2) in sequence, and multiplies it with the feature map obtained by the second convolutional layer group (CBL_2) to adaptively adjust the feature map, thereby completing the recalibration of the feature map obtained by the second convolutional layer group (CBL_2); then the recalibrated feature map is processed using the CSP2 residual structure (CSP1_3) to obtain the output feature map.

[0052] Step (5) inputs the feature map outputted from step (4) into the third convolutional layer group (CBL_3) for processing.

[0053] Step (6) The third mixed domain attention module (CBAM_3) infers the attention weights of the feature map processed by the third convolutional layer group (CBL_3) along the spatial and channel dimensions within the third mixed domain attention module (CBAM_3) in turn, and multiplies it with the feature map obtained by the third convolutional layer group (CBL_3) to adaptively adjust the feature map, thereby completing the recalibration of the feature map output by the third convolutional layer group (CBL_3); and then processes the recalibrated feature map through the CSP3 residual structure (CSP1_3) to obtain the output feature map.

[0054] Step (7) inputs the features adjusted by the third mixed domain attention module CBAM_3 and the CSP3 residual structure (CSP1_3) in step (6) into the fourth convolutional layer group (CBL_4), the fifth convolutional layer group (CBL_5) and the maximum pooling layer SPP.

[0055] Step (8) passes the feature map obtained in step (7) through the sixth convolution layer group (CBL_6), the CSP4 residual structure (CSP2_1) and the seventh convolution layer group (CBL_7) in sequence, and then, after upsampling, the feature map is expanded and then feature fused (Concat) with the feature map output by the CSP3 residual structure (CSP1_3).

[0056] Step (9) Concat the feature map obtained in step (8) through the CSP5 residual structure (CSP2_1) and the eighth convolution layer group (CBL_8) in sequence, and then, after upsampling to expand the feature map, perform feature fusion with the second layer CSP2 residual structure (CSP1_3).

[0057] Step (10) passes the features obtained in step (9) through the CSP6 residual structure (CSP2_1) and the ninth convolution layer group (CBL_9) in sequence, and then, after upsampling, the feature map is expanded and fused with the feature map output by the first layer CSP1 residual structure (CSP1_1).

[0058] Step (11) passes the fused feature map of step (10) through the seventh residual structure (CSP2_1) of CSP7 and the convolution layer Conv to obtain the prediction grid 1 (152*152*256).

[0059] Step (12) performs feature fusion on the feature map of step (11) after passing through the seventh-layer residual structure of CSP7 through the tenth-layer convolution layer group (CBL_10) and the ninth-layer convolution layer group (CBL_9).

[0060] Step (13) passes the fused feature map of step (12) through the eighth residual structure (CSP2_1) of CSP8 and the convolution layer Conv to obtain the prediction grid 2 (76*76*255).

[0061] Step (14) performs feature fusion on the feature map output by the eleventh convolutional layer group (CBL_11) in step (13) and the eighth convolutional layer group (CBL_8).

[0062] Step (15) passes the fused feature map of step (14) through the ninth residual structure (CSP2_1) of CSP9 and the convolution layer Conv to obtain the prediction grid 3. (38*38*255)

[0063] Step (16) performs feature fusion on the feature map output by the twelfth convolutional layer group (CBL_12) in step (15) and the seventh convolutional layer group (CBL_7).

[0064] Step (17) passes the fused feature map of step (16) through the tenth layer residual structure of CSP10 to obtain prediction grid 4 (19*19*255). Use the generated four prediction networks to detect bacteria.

[0065] According to one or more embodiments, Figure 2 As shown in the figure, a working flow diagram of a device for automatically counting the number of colonies using artificial intelligence. The specific working process is:

[0066] The bacteria in the culture dish were image sampled using a microscope, and the initial image annotation was performed manually;

[0067] Send it to the PC for training using the optimized yolov5 model;

[0068] The .pt weight file obtained after training is used to predict the bacterial graph to be detected;

[0069] If the prediction confidence is greater than 80, the prediction box is red; otherwise, it is a yellow prediction box. At this time, the PC displays a bacterial image with a prediction box. The inspector checks the target in the yellow box. If it is confirmed to be the required bacteria, the yellow box is corrected to a red box with the mouse.

[0070] If it is confirmed that it is not the desired bacteria, delete the yellow box;

[0071] After correction, the accurate annotation map and label of bacteria are saved in the database and sent to the yolov5 model as a training set;

[0072] The purpose of automatically counting the number of bacteria is achieved by counting the red pixel values ​​of the corrected bacterial image;

[0073] Through continuous correction and the continuous addition of precise annotation maps, the training accuracy will be continuously improved, which will greatly save manpower and continuously improve the accuracy of detection.

[0074] like Figure 3 As shown, the mAP value, accuracy Precision and recall rate Recall of the neural network structure YOLOV5-CBAM in the device for automatically counting the number of colonies of the present invention all have good performance, and have a good effect on bacteria detection.

[0075] Table 1 is a comparison of the results of the model algorithm of the present invention using artificial intelligence to automatically count the number of colonies and the algorithm before optimization.

[0076] Table 1

[0077]

[0078] In Table 1, YOLOV5 is the original network, YOLOV5-4Predict is a network with four detection layers, YOLOV5-ECA is a network with an added ECA attention module, and YOLOV5-CBAM is an improved network in this paper.

[0079] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0080] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0081] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0082] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for counting colony numbers, It is characterized in that include, Obtain the original bacterial image of the colony to be counted; Preprocessing the original bacteria image to obtain a first bacteria image; The first bacterial image is input into the trained colony statistical model to obtain the statistical number of bacteria. The colony statistics model is a hybrid domain attention module-target detection model, which is composed of Backbone, Neck and Detect. Backbone includes Focus slice structure, the first convolution layer + BN + Leaky_ReLU activation function CBL_1, the first mixed domain attention module CBAM_1, the first layer CSP1, the second convolution layer + BN + Leaky_ReLU activation function CBL_2, the second mixed domain attention module CBAM_2, the second layer CSP2, the third convolution layer + BN + Leaky_ReLU activation function CBL_3, the third mixed domain attention module CBAM_3, the third layer CSP3, the fourth and fifth convolution layers + BN + Leaky_ReLU activation functions CBL_4 and CBL_5, pooling layer SPP, the sixth convolution layer + BN + Leaky_ReLU activation function CBL_6; Neck includes the CSP2_1 layer, the upsampling layer Upsampling and the convolution layer + BN + Leaky_ReLU activation function CBL_*, Detect includes the convolution layer Conv and four prediction grids, namely prediction grid 152*152*255, prediction grid 76*76*255, prediction grid 38*38*255, and prediction grid 19*19*255.

2. The method for counting colony numbers according to claim 1, It is characterized in that The first bacterial image is input into the colony statistical model to obtain a second bacterial image and a predicted confidence. According to the confidence, the bacterial detection target is set to different colors and the bacterial target is verified.

3. The method for counting colony numbers according to claim 2, It is characterized in that The training process of the hybrid domain attention module-target detection model includes: The labeled bacterial image training set is used to train the target detection model and extract features. Then the extracted features are weighted by the hybrid domain attention module CBAM-Block, and the trained target detection model is tested by the bacterial image test set.

4. The method for counting colony numbers according to claim 3, It is characterized in that The target detection model is based on the YOLOV5 network.

5. A device for counting the number of colonies, It is characterized in that The device includes a memory; and A processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method according to any one of claims 1 to 4.

6. The colony counting device according to claim 5, It is characterized in that The device also includes a microscope camera to obtain raw bacterial images.

7. A colony counting system, It is characterized in that The system includes a server having a memory; as well as A processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Finite pixel infrared unmanned aerial vehicle target detection method based on mixed domain attention

    CN112560907A

  • Fungus target detection method based on feature fusion and attention

    CN113095265A

  • SAR image target detection method in complex scene

    CN113536929A

  • Automatic pest counting method based on combination of multi-scale feature fusion network and positioning model

    WO2020047738A1