Visual inspection flora identification and counting method based on deep optimization

Abstract

The invention discloses a visual inspection flora identification and counting method based on deep optimization. The method comprises the following steps: S1, image acquisition: acquiring a flora image through an image acquisition system; s2, image preprocessing: using an OpenCV library of Python to carry out preprocessing on the acquired flora image; s3, data annotation: annotating the preprocessed flora image by using LabelImg software, and dividing an identification target into a plurality of categories; s4, data set expansion: expanding the training data set through a data enhancement method; s5, model optimization: optimizing the original YOLO-v7 model, including newly adding a small target detection layer and optimizing an anchoring frame; s6, model training: training the training data set by using the optimized YOLO-v7 model; s7, result verification: utilizing an independent test set evaluation model to calculate F1-score and mAP indexes; and S8, practical application: providing a visual user interface and a data storage function. According to the method, the YOLO-v7 deep learning model is optimized by newly adding the small target detection layer, so that the flora is automatically identified and counted.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and small target recognition, and in particular to a visual detection bacterial colony recognition and counting method based on deep optimization. Background Art

[0002] Food inspection and testing is a key link in ensuring food safety and hygiene, which covers the quality and safety assessment of each link of food production, processing, storage, transportation and sales. This process is not only directly related to the health and life safety of consumers, but also plays a fundamental role in food quality management and the sustainable development of the food industry. Given its importance, food inspection and testing assessments are widely implemented around the world and will continue to be used.

[0003] However, the current food inspection and testing system usually requires the total number of bacteria cultured in standard culture media to be counted, which limits the efficiency and accuracy of the test to a certain extent. Therefore, it is necessary to improve existing technologies and practices, and any innovation that can improve the accuracy and efficiency of detection should be encouraged and appreciated.

[0004] At present, there are two main bacterial counting methods used in the field of food inspection and testing. The first is the plate count method, which involves diluting the sample to be tested and inoculating it into a culture medium plate, and culturing it for a certain period of time to form colony-forming units (CFU). CFU refers to each colony formed on an agar plate after culturing at a specific temperature and time. It is widely used as a unit of measurement for the number of bacteria or molds. The second method is the membrane filtration counting method, which filters the sample to be tested onto a membrane, then transfers the membrane to a culture medium for culturing, and also counts CFU. Although the number of live bacteria can be estimated by the number of colonies on a culture dish or membrane, the manual counting process is not only time-consuming and labor-intensive, but also has a high error rate. In addition, the accuracy of CFU counting is limited by multiple factors such as the characteristics of the culture medium, the culture conditions, and the physiological state of the microorganisms, resulting in differences in the counting results obtained by different operators even under the same conditions.

[0005] There are two commonly used methods: manual counting, where experienced inspectors count the bacterial colonies in the culture dish according to rules and experience; and equipment counting, which uses the automatic counters available on the market. However, the manual method has the problems of high labor intensity, strong subjectivity, and high error rate, while the equipment method is limited by its narrow application range, poor adaptability, and insufficient compatibility.

[0006] Research work to solve this problem has been introduced in the literature and is mainly carried out under the following three technical routes.

[0007] The first is based on classical image processing methods. For example, the Otsu and Canny methods are widely used in bacterial image segmentation due to their computational efficiency. This is satisfactory from the perspective of image processing effects alone, but it cannot meet the requirements in terms of bacterial identification and counting processing. The reason is that this method cannot achieve batch processing, the threshold required for each image is different, the recognition efficiency is low, and it is impossible to output stably and perform standardized evaluation.

[0008] The second is free open source tools. OpenCFU and CellProfiler are both popular tools for detecting, counting, and quantifying colony forming units (CFU) in this field. However, although OpenCFU has simple parameter settings, its strict CFU selection criteria limit its scope of application, and it is more suitable for round targets with high separation. Although CellProfiler integrates a variety of image processing algorithms, its core methodology is still limited to the image technology processing level, and it has many parameters and cumbersome operations, which is not suitable for industrial application scenarios.

[0009] The third is to combine machine learning algorithms and deep learning models. For example, Welikala et al. proposed a deep learning model based on a UNet-like architecture, which locates bacterial colonies through binary segmentation and improves the recognition of crowded cell boundaries by combining weight maps. Anantharaman et al. demonstrated how to use open source deep learning technology to analyze bacterial microscope images, including tasks such as image segmentation, object detection, and denoising.

[0010] The above methods have been well explored in their respective fields, but they are all based on specific objects, such as cells, and do not provide solutions to the complex bacterial flora morphology in the field of food testing.

[0011] Therefore, how to provide a visual detection bacterial colony identification and counting method based on deep optimization is an urgent problem that technicians in this field need to solve. Summary of the invention

[0012] One object of the present invention is to propose a visual detection bacterial flora identification and counting method based on deep optimization. The present invention utilizes an optimized YOLO-v7 deep learning model to achieve efficient and accurate bacterial flora identification and counting through automated image processing and data enhancement, and has the advantages of high accuracy, low error rate and wide adaptability.

[0013] A method for visual detection of bacterial flora identification and counting based on deep optimization according to an embodiment of the present invention comprises the following steps:

[0014] S1. Image acquisition: Obtain bacterial flora images through the image acquisition system;

[0015] S2. Image preprocessing: Use Python's OpenCV library to preprocess the collected bacterial flora images;

[0016] S3. Data labeling: LabelImg software is used to label the preprocessed bacterial flora images and classify the identification targets into multiple categories;

[0017] S4, Dataset expansion: Expand the training dataset through data augmentation methods;

[0018] S5, model optimization: optimize the original YOLO-v7 model, including adding a small target detection layer and optimizing the anchor box;

[0019] S6. Model training: Use the optimized YOLO-v7 model to train the training data set;

[0020] S7. Result verification: Use an independent test set to evaluate the model, calculate the F1-score and mAP indicators, and verify the accuracy and stability of the model;

[0021] S8. Practical application: Provide visual user interface and data storage functions.

[0022] Furthermore, the image acquisition system includes a rack, an industrial camera, an LED ring light source, a Mini industrial computer, an acquisition card and a display; the industrial camera is a Hikvision color area array model with a 1.2-inch target surface and a resolution of 12 million pixels; the LED ring light source has a power of 10 watts and a color temperature set to 6500K.

[0023] Furthermore, the S2 specifically includes:

[0024] S21, taking the center of the culture dish as the origin, cropping the original image from 1920×1080 pixels to 960×960 pixels. The reason for cropping is that there are too many interference factors in the non-effective area of ​​the image;

[0025] S22, purify the ineffective areas of the image and set the four corners of the ineffective areas to all white pixels [255, 255, 255] to reduce the interference on the feature extraction of the convolutional neural network;

[0026] S23. Use a 3×3 Gaussian kernel to sharpen the image to highlight the edges and retain image details.

[0027] Furthermore, the data annotation categories include categories divided by flora morphology and categories divided by flora subject. The purpose of this division is guided by the research purpose of the project. The research purpose of the project is to count flora as the primary goal, and the identification of the flora subject is a secondary goal, because the flora subject requires a rigorous experimental identification process, and cannot be confirmed only by morphological similarity.

[0028] Furthermore, the data enhancement includes rotation, adding Gaussian noise and brightness change, and the image is processed automatically and with high precision by combining the Python image processing module with the LabelImg program; the interchange formula of the data enhancement is:

[0029] When rotating 90° counterclockwise: x′=1-y; y′=x; w′=h; h′=1-w;

[0030] When rotating 90° clockwise: x′=y; y′=1-x; w′=h; h′=1-w;

[0031] Among them, x, y, w and h represent the horizontal coordinate, vertical coordinate, width and height before conversion respectively; x′, y′, w′ and h′ represent the horizontal coordinate, vertical coordinate, width and height after conversion respectively.

[0032] Furthermore, the original YOLO-v7 model optimized by the model includes three detection layers and corresponding three sets of initialized anchor box values. The 75th detection layer is used to detect objects larger than 8×8 pixels, the 88th layer is used to detect objects larger than 16×16 pixels, and the 101st layer is used to detect objects larger than 32×32 pixels.

[0033] The novel network structure of the newly added small target detection layer includes:

[0034] Perform upsampling on the feature map at the 75th layer to expand the size of the feature map;

[0035] At the 77th layer, the 160×160 pixel feature map is fused with the 11th layer feature map in the backbone network to obtain a larger feature map for small object detection;

[0036] A 160×160 pixel detection layer is added at the 87th layer to detect objects larger than 4×4 pixels.

[0037] Furthermore, the optimizing the anchor boxes includes recalculating the sizes and proportions of the anchor boxes using a k-means clustering algorithm with 1000 iterations.

[0038] Going a step further, we developed an application with a visual user interface that supports cloud and local deployment, and implements the entire process from image acquisition, preprocessing, uploading to automatic detection technology, and then to manual review, revision, and warehousing.

[0039] Furthermore, the program has the functions of storing, classifying, retrieving, and visualizing bacterial flora images, as well as performing simple analysis and early warning on the number of bacterial flora categories.

[0040] The beneficial effects of the present invention are:

[0041] (1) Improve the accuracy of bacterial identification and counting: By using deep learning technology, especially the optimized YOLO-v7 model, this method can significantly improve the accuracy of bacterial identification and counting. Compared with traditional manual counting methods and existing automated equipment, the present invention can reduce the error rate and improve the consistency of detection results.

[0042] (2) Improved detection efficiency: This method significantly reduces the time and labor intensity of manual operations by automating steps such as image acquisition, preprocessing, and data annotation. Data augmentation technology and optimized model training methods further improve detection efficiency and can process a large number of samples in a shorter time.

[0043] (3) Adaptability to various bacterial morphologies: The present invention can handle various bacterial morphologies, including monoploid bacteria, diploid bacteria, chain bacteria and other different forms of bacterial morphologies, and has wide applicability. By optimizing the anchor frame and adding a small target detection layer, the model can accurately identify and count bacterial morphologies of different forms, and has strong adaptability.

[0044] (4) Reduce missed detection and false detection: The optimized YOLO-v7 model can effectively reduce missed detection and false detection during the detection process, especially for the detection of small target bacterial colonies. The newly added small target detection layer and the reconfigured anchor box enable the model to better handle complex backgrounds and diverse bacterial colony morphologies.

[0045] (5) Improve comprehensive detection solutions: This method not only includes image acquisition and preprocessing, but also provides a full-process solution for data annotation, model training, result verification, and practical application. By developing an application with a user-friendly interface, supporting cloud deployment and local deployment, a fully automated process from image acquisition to result output is achieved, which is convenient for users.

[0046] (6) Enhanced data storage and analysis capabilities: The developed application has the functions of storing, classifying, retrieving, visualizing, and performing simple analysis and early warning of bacterial flora images. Users can conveniently manage and analyze test data, further improving the quality and efficiency of food inspection and testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 This is a flow chart of a method for visual detection of bacterial flora identification and counting based on deep optimization proposed by the present invention;

[0049] Figure 2 This is a schematic diagram of an image acquisition system for a method for visually detecting bacterial colony identification and counting based on deep optimization proposed by the present invention;

[0050] Figure 3 This is a schematic diagram of the image preprocessing process of the visual detection bacterial colony recognition and counting method based on deep optimization proposed by the present invention;

[0051] Figure 4 This is a schematic diagram of the original architecture of the YOLO-v7 model for a visual detection bacterial colony recognition and counting method based on deep optimization proposed in the present invention;

[0052] Figure 5 This is a comparison diagram of the IoU threshold and F1 value of a visual detection bacterial community recognition and counting method based on deep optimization proposed in the present invention;

[0053] Figure 6 This is an example diagram of training detection of a visual detection bacterial colony recognition and counting method based on deep optimization proposed by the present invention;

[0054] Figure 7 This is a model training and application flow chart of a visual detection bacterial colony identification and counting method based on deep optimization proposed in the present invention. DETAILED DESCRIPTION

[0055] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0056] refer to Figure 1-7 , a visual detection bacterial colony identification and counting method based on deep optimization, comprising the following steps:

[0057] S1. Image acquisition: Obtain bacterial flora images through the image acquisition system;

[0058] S2. Image preprocessing: Use Python's OpenCV library to preprocess the collected bacterial flora images;

[0059] S3. Data labeling: LabelImg software is used to label the preprocessed bacterial flora images and classify the identification targets into multiple categories;

[0060] S4, Dataset expansion: Expand the training dataset through data augmentation methods;

[0061] S5, model optimization: optimize the original YOLO-v7 model, including adding a small target detection layer and optimizing the anchor box;

[0062] S6. Model training: Use the optimized YOLO-v7 model to train the training data set;

[0063] S7. Result verification: Use an independent test set to evaluate the model, calculate the F1-score and mAP indicators, and verify the accuracy and stability of the model;

[0064] S8. Practical application: Provide visual user interface and data storage functions.

[0065] In this embodiment, the image acquisition system includes a rack, an industrial camera, an LED ring light source, a Mini industrial computer, an acquisition card and a display. Figure 2 As shown; the industrial camera is a Hikvision color area array model with a 1.2-inch target surface and a resolution of 12 million pixels; the LED ring light source has a power of 10 watts and a color temperature setting of 6500K.

[0066] In this implementation, S2 specifically includes:

[0067] S21, taking the center of the culture dish as the origin, crop the original image from 1920×1080 pixels to 960×960 pixels (see Figure 3 B), the reason for cropping is that there are too many interference factors in the non-effective area of ​​the image ( Figure 3 As indicated by the arrow in A);

[0068] S22, purify the ineffective area of ​​the image (see Figure 3 C), set the four corner non-valid areas to all white pixels [255, 255, 255] to reduce the interference on the feature extraction of convolutional neural network;

[0069] S23, sharpen the image using a 3×3 Gaussian kernel (see Figure 3 D), highlighting edges and preserving image details (see Figure 3 E).

[0070] In this embodiment, the data annotation categories include categories divided according to the morphology of the bacterial flora and categories divided according to the subject to which the bacterial flora belongs.

[0071] Since YOLO-v7 is a visual-based supervised learning algorithm that focuses on real-time target detection tasks, accurate labeling of targets is a key step in training deep learning models. This study divides the identified targets into 21 categories based on the bacterial morphology displayed in the image, as shown in Table 1. The typical illustrations listed in the table are a small part of the many bacterial morphologies. During the actual labeling, the recruited students were trained to label the various bacterial communities in the image one by one. Therefore, the actual bacterial morphology includes but is not limited to the legend shown in Table 1.

[0072] Table 1 Typical legends of bacterial flora annotation

[0073]

[0074]

[0075] It should be noted that of the 21 labeled categories above, the 11 on the left are divided according to the morphology of the flora, and the 10 on the right are divided according to the category to which the flora belongs. They are specially cultivated based on pathogens. The purpose of this division is guided by the research purpose of the project. The research purpose of the project is to count the flora as the primary goal, and the identification of the category to which the flora belongs is a secondary goal, because the category to which the flora belongs requires a rigorous experimental identification process and cannot be confirmed solely by morphological similarity. Therefore, when labeling, typical bacterial species are separately labeled according to the last 10 categories, and mixed flora samples are labeled according to the first 11 categories. All the labeled images are input into the training model, and the total number of identifications is added to the multiplier of polymicrobial bacteria, and the number of food residues and impurities is subtracted, which is the total number of flora identified by the visual system.

[0076] In this embodiment, the data enhancement includes rotation, adding Gaussian noise and brightness change, and the image is processed automatically and with high precision by combining the Python image processing module and the LabelImg program; the interchange formula of the data enhancement is:

[0077] When rotating 90° counterclockwise: x′=1-y; y′=x; w′=h; h′=1-w;

[0078] When rotating 90° clockwise: x′=y; y′=1-x; w′=h; h′=1-w;

[0079] Among them, x, y, w and h represent the horizontal coordinate, vertical coordinate, width and height before conversion respectively; x′, y′, w′ and h′ represent the horizontal coordinate, vertical coordinate, width and height after conversion respectively.

[0080] In this implementation, the model-optimized original YOLO-v7 model includes three detection layers and corresponding three sets of initialized anchor box values, the 75th detection layer is used to detect targets larger than 8×8 pixels, the 88th layer is used to detect targets larger than 16×16 pixels, and the 101st layer is used to detect targets larger than 32×32 pixels;

[0081] The novel network structure of the newly added small target detection layer includes:

[0082] Perform upsampling on the feature map at the 75th layer to expand the size of the feature map;

[0083] At the 77th layer, the 160×160 pixel feature map is fused with the 11th layer feature map in the backbone network to obtain a larger feature map for small object detection;

[0084] A 160×160 pixel detection layer is added at the 87th layer to detect targets larger than 4×4 pixels, meeting the requirements of small target detection. In this embodiment, the optimization of the anchor frame includes recalculating the size and proportion of the anchor frame using a k-means clustering algorithm with 1000 iterations.

[0085] In this embodiment, the result verification selected 104 pictures that did not participate in the training, and they were manually annotated by experienced inspectors to form an independent test data set. Subsequently, we applied the optimal parameters of the models in three different training stages - the pre-training model (Pre_test), the full data set training model (Alldata_test), and the optimized full data set training model (Update_test) - to the same test program (test.py). We use F1-score as the evaluation indicator, which is the harmonic mean of precision and recall, widely used in machine learning competitions for multi-classification problems, and its value ranges from 0 to 1.

[0086] The evaluation results show that the optimized full-dataset training model surpasses the previous two models in F1-score. The specific results are shown in Table 2. It is worth noting that according to historical data, SBac flora is the largest in all categories, usually accounting for more than 98% of the flora in all categories. Therefore, we specifically extracted the F1-score value of SBac (F1-SBac) and the average F1-score value of all categories (F1-All), both of which are listed in Table 2. The results show that the optimized model (Update_train) shows superior performance.

[0087] Table 2 F1-score comparison of three models

[0088]

[0089] Secondly, we focus on the improvement of feature capture and framing capabilities before and after model optimization, which is reflected by precision and recall. Both models are trained using the same dataset. We use mAP (Mean Average Precision) as the performance evaluation indicator, which is the average accuracy under the Precision-Recall curve, which is used to measure the balance between the accuracy and recall of the model at different thresholds. We selected two evaluation indicators, mAP@.5 and mAP@.95, where mAP@.5 represents the average mAP when the threshold is greater than 0.5, and mAP@.5:.95 represents the average mAP under the threshold range of 0.5 to 0.95 (step size is 0.05). See Table 3 for specific indicator values.

[0090] Table 3 Comparison of mAP values ​​before and after model optimization

[0091]

[0092] In this implementation, an application with a visual user interface is developed to support cloud deployment and local deployment, realizing the whole process operation from image acquisition, preprocessing, uploading to automatic detection technology, and then to manual review, revision and warehousing.

[0093] In this embodiment, the program has the functions of storing, classifying, retrieving, and visualizing flora images, as well as performing simple analysis and early warning on the number of flora categories.

[0094] Embodiment 1:

[0095] The first stage is to establish the initial model; this study uses the seventh generation YOLO model (YOLO-v7) to build a model framework for bacterial flora identification and counting. The model architecture diagram is as follows Figure 4As shown. The input image of the model is first resized to a uniform size of 640×640 pixels and then input into the Backbone network, which is responsible for extracting key features from the image. The extracted features are then fed into the Head layer network, which aggregates the image features and calculates and outputs three layers of feature maps of different scales. The output of the model is designed to identify 21 predefined categories, and the output of each category contains five parameters: x and y represent the coordinates of the center point of the detected object in the horizontal and vertical directions of the image, respectively; w and h represent the horizontal width and vertical height of the object, respectively; o represents the confidence score of the object, reflecting the model's certainty of the detection result. Since three anchor boxes are set, the output size of each layer of the feature map is (21+5)×3 multiplied by the pixel size of the feature map. The software platform used in the experiment is PyCharm 2021.3.1, and the toolbox is based on the PyTorch deep learning framework, using the official toolbox YOLOv7-main of YOLO-v7.

[0096] During the initial model building process, we did not make any structural modifications or optimizations to the YOLO-v7 model architecture. This decision was made to run the program on a small-scale dataset first to lay the foundation for subsequent parameter tuning.

[0097] The second stage is parameter tuning and small data set training; since the YOLO-v7 model involves many parameters and a complete model training process requires a lot of computing power and time, this study adopted a strategy of using a small data set to tune key parameters before conducting full data set training. This method not only provides sufficient preparation for the training of the full data set, but also takes economic considerations, because small-scale training can be performed on local computers, while large-scale training may require renting cloud servers, which will bring higher economic and time costs.

[0098] The small dataset for this study consists of 500 images, which are divided into training set (70%), validation set (20%) and test set (10%) by automated methods. The YOLO-v7 model is deployed locally, and the configuration includes Core TM i5-10505 processor, 8GB memory and 64-bit operating system.

[0099] In this stage, 500 images are used as the training set, and no data augmentation technology is applied. The purpose is to run the model with a smaller amount of data to determine the appropriate training cycle (epoch), so as to provide cost-effectiveness analysis for subsequent operation with a larger amount of data. From the experimental data of the training set, it can be seen that after 50 to 100 epochs, the loss value tends to be stable. The experimental data of the validation set shows that after 50 epochs, the target box position error and target detection loss increase. Therefore, setting the training cycle to 50 epochs is a reasonable choice, which can not only ensure the training effect of the model, but also control the training cost.

[0100] In the field of object detection, the accuracy of the bounding box is crucial to the overall performance of the model. YOLO-v7 uses a bounding box loss function design based on Intersection over Union (IoU). IoU is an indicator that measures the degree of overlap between the target box and the prediction box, and is defined as the ratio of the intersection area of ​​the target box and the prediction box to the union area. The calculation formula of this loss function is as follows:

[0101] The bounding box loss function is: L IoU =1-IoU.

[0102] Among them, L IoU represents the bounding box loss, AI B represents the area of ​​the intersection of the target box and the predicted box, and AU B represents the area of ​​the union of the target box and the predicted box. The loss function is designed to optimize the performance of the detection model by minimizing the difference between the predicted bounding box and the true bounding box.

[0103] The bounding box loss function is a component of the overall loss function, so choosing a reasonable IoU threshold provides parameter guarantee for subsequent full data training. We selected 8 IoU thresholds that increase in proportion, trained on the same dataset, and used the F1 value as a comparison indicator. The training results are shown in Table 4 and Figure 5 .

[0104] Table 4 Comparison of IoU threshold and F1 value

[0105]

[0106] As Figure 5As shown in the figure, when the confidence threshold is set to 0.45, the detection performance of the model is optimal. The choice of this threshold is based on the performance of the model on the validation set, where the threshold determines the minimum confidence in the model output that is considered a true detection. In the object detection task, the appropriate selection of the confidence threshold is crucial to balance the precision and recall of the model. Therefore, when training on the full dataset, we configure the confidence threshold to 0.45 in order to maximize the detection ability of the model while maintaining detection accuracy.

[0107] The third stage is full dataset training; after determining the basic parameters of the model, we uniformly preprocessed and annotated the 1546 images, and then completed the division into training set (70%), validation set (20%) and test set (10%) in a random and automated manner. During the full dataset training phase, in order to ensure the compatibility of the image size with the YOLO-v7 model and improve execution efficiency, all images were resized to 960×960 without losing the effective area. This size selection was based on two considerations: first, 960×960 is an integer multiple of 32 pixels, which ensures smooth execution during the training and inference phases of the model; second, the unified image size helps to improve computational efficiency at all levels of the network, including pooling and convolution operations.

[0108] Since the training of the full dataset requires high computing resources and the local machine cannot meet this demand, we rented an Alibaba Cloud server for training. The server configuration is as follows: CPU is 48 cores (vCPU), memory is 186GB, equipped with two NVIDIAT4 GPUs, operating system is Ubuntu 18.04, 64-bit. During the training process, we used yolov7.pt as the initial weight parameter, and the batch size was set to 8. The loss function of the model training adopts the mean square error (MSE), that is, the loss function of the three parts of the target confidence loss (confidence loss, defined as obj_loss in YOLO-v7), category confidence loss (classification loss, denoted as cls_loss) and coordinate regression loss (localization loss, defined as box_loss in YOLO-v7) is weighted summed, and then the average is taken to obtain the final loss value. The brief calculation formula of the loss function is as follows:

[0109]

[0110] Among them, λ 1 =0.05,λ 2 =0.1,λ 3=0.125, x, y, w, h are the coordinate values, width values ​​and height values ​​of the real frame; and are the coordinate values, width values ​​and height values ​​of the prediction box, is the predicted value of whether there is an object in the bounding box; c is the true value of whether the bounding box contains the object, and p(c) is the true confidence of a certain type of object. The confidence level of a certain type of target predicted by the model.

[0111] The training started with a learning rate of 0.001. This choice showed better stability in the Stochastic Gradient Descent (SGD) algorithm in subsequent experiments. The model converged within nearly 50 epochs. After the training was completed, we obtained the best performing model parameter file best.pt. This parameter file was applied to the detection program (detect.py) and achieved satisfactory results for the partner company in the actual detection task. In order to more intuitively demonstrate the detection effect, we divided the detected image samples into four categories: low separation ( Figure 6 A) Higher ( Figure 6 B) High Figure 6 C) and mixed ( Figure 6 D) These classifications reflect the complexity and detection difficulty of different bacterial flora images, thereby more comprehensively evaluating the performance of the model.

[0112] At present, the research on the detection and counting of bacterial colonies using multi-scale convolution technology (MCV) based on deep learning is relatively limited. Krzysztof et al. adopted the R-CNN framework and proposed a self-normalized density map (SNDM) method for the separation and counting of microbial targets. Guo et al. combined the U-Net network structure and the self-attention mechanism to propose a deep learning model and verified its effectiveness in four public cell databases. Although these studies have made progress in the recognition and counting of cell image targets, they are not completely suitable for the recognition and counting of bacterial colonies. The reason is that the cell morphology is relatively simple, the separation is high, the boundaries are clear, and the density function of the cell image is easy to construct, which is suitable for analysis by applying the attention mechanism. However, the morphology of bacterial colonies is varied and the separation is different. It is challenging to directly apply these technologies to identify and count bacterial colonies. This is one of the reasons why the universality of automatic counting instruments on the market is not high, and it also highlights the development value of this study.

[0113] To improve practicality, we developed a user-friendly application. The application supports cloud deployment and local deployment. It is developed using Java and MySQL databases and can implement the entire process from image acquisition, preprocessing, uploading to automatic detection and counting, and then to manual review, revision, and storage. In addition, the application is also a management information system that has the functions of storing, classifying, retrieving, and visualizing flora images, as well as simple analysis and early warning of flora category numbers.

[0114] The findings of this study show that the optimized YOLO-v7 small target convolutional neural network model can effectively identify the morphological characteristics of a single bacterial colony, and can make high-quality judgments on impurities and impurities, and has the potential to classify the identified bacterial colonies. Based on this, we have made the following two assumptions on how to transform the results of this project into practical applications. The application flow chart is as follows: Figure 7 As shown. (1) The model training part of this study performs two steps. The first step is the model training step, which includes the training of the parameter adjustment model and the training of the frozen model. The so-called parameter adjustment model refers to the optimal parameter model generated by the training process described in this article. The data set it relies on is the data collected before the start of this study, which we call the native data set. The so-called frozen model refers to the new data set generated after manual deletion, addition, and modification in actual application, which we call the regenerated data set. The regenerated data set is sent back to the model for training, but the training process of the entire network structure is no longer carried out. Instead, the weight value of the Backbone network is kept unchanged and not updated. Only the Head layer network is trained, that is, training starts from the 50th layer of the optimized model. The purpose of this is to adopt the thinking of transfer learning, that is, to keep the underlying feature extractor of the pre-trained model unchanged and only train the top-level classifier of the model to maintain the stability and generalization ability of the model. (2) The purpose of the model application part of this study is to provide enterprises with a new set of bacterial colony counting methods. First, the optimal parameter values ​​obtained in the training part are implanted into the detection program. After obtaining the new data set, the detection program is executed. After the results are obtained, they are confirmed by experienced inspectors before they can be stored for future reference. It is worth mentioning that an important function of the application part is to collect the regenerated data set, which is an important link in strengthening the performance of the model.

[0115] In summary, this study successfully built and optimized a model based on the YOLO-v7 convolutional neural network, aiming to realize the automatic identification and counting of bacterial flora in the field of food inspection and detection. At the same time, this study also built a corresponding bacterial flora image data acquisition system, and conducted an in-depth study on the detection effect of the YOLO-v7 model on different forms of bacterial flora. After a series of experiments and analysis, the following conclusions were drawn:

[0116] The YOLO-v7 model established in this study performed well in the recognition of food bacterial flora images. In a set of independent real sample experiments, the model achieved an accuracy rate of 88.5% for the recognition of intra-category colonies, verifying the effectiveness of the model.

[0117] This study significantly improved the performance of the model by optimizing the model structure, improving parameter settings, and introducing automated data enhancement methods. In particular, the newly added small object detection layer increased the mean average precision (mAP) of the optimized model by 4.5% after training on the same dataset.

[0118] This study provides a complete set of solutions for bacterial colony counting in the field of food inspection and testing. From the construction of the image acquisition system to the training of the data set, and then to the output of automated results, a closed and complete process has been formed, which significantly improves the efficiency of research results transformation.

[0119] The research value of this project lies in the authenticity of its data and the richness of the database. With long-term application practice, it is expected that a large-scale food testing database will be established, providing valuable data resources for deep learning and future research.

[0120] The results of this study not only improve the automation level of food inspection and testing, but also provide new ideas and methods for the application of deep learning in the field of microbial detection. Future work will further explore the generalization ability of the model and consider applying it to a wider range of food inspection scenarios.

[0121] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for visual detection of bacterial flora identification and counting based on deep optimization, It is characterized in that The steps include: S1. Image acquisition: Obtain bacterial flora images through the image acquisition system; S2. Image preprocessing: Use Python's OpenCV library to preprocess the collected bacterial flora images; S3. Data labeling: LabelImg software is used to label the preprocessed bacterial flora images and classify the identification targets into multiple categories; S4, Dataset expansion: Expand the training dataset through data augmentation methods; S5, model optimization: optimize the original YOLO-v7 model, including adding a small target detection layer and optimizing the anchor box; S6. Model training: Use the optimized YOLO-v7 model to train the training data set; S7. Result verification: Use an independent test set to evaluate the model, calculate the F1-score and mAP indicators, and verify the accuracy and stability of the model; S8. Practical application: Provide visual user interface and data storage functions.

2. According to claim 1, a method for visual detection of bacterial flora identification and counting based on deep optimization, It is characterized in that The image acquisition system includes a rack, an industrial camera, an LED ring light source, a Mini industrial computer, an acquisition card and a display; the industrial camera is a Hikvision color area array model with a 1.2-inch target surface and a resolution of 12 million pixels; the LED ring light source has a power of 10 watts and a color temperature set to 6500K.

3. According to claim 1, a method for visual detection of bacterial flora identification and counting based on deep optimization, It is characterized in that The S2 specifically includes: S21, taking the center of the culture dish as the origin, crop the original image from 1920×1080 pixels to 960×960 pixels; S22, purify the ineffective area of ​​the image and set the four corners of the ineffective area to all white pixels [255,255,255]; S23. Use a 3×3 Gaussian kernel to sharpen the image to highlight the edges and retain image details.

4. According to claim 1, a method for visual detection of bacterial flora identification and counting based on deep optimization, It is characterized in that The data annotation categories include categories divided according to the morphology of the bacterial flora and categories divided according to the subject to which the bacterial flora belongs.

5. According to claim 1, a method for visual detection of bacterial flora identification and counting based on deep optimization, It is characterized in that The data enhancement includes rotation, adding Gaussian noise and brightness change, and the image is processed automatically and with high precision by combining the Python image processing module with the LabelImg program; the interchange formula of the data enhancement is: When rotating 90° counterclockwise: x ′ =1-y;y ′ =x;w ′ =h;h ′ =1-w; When rotating 90° clockwise: x ′ =y;y ′ =1-x;w ′ =h;h ′ =1-w; Among them, x, y, w and h represent the horizontal coordinate, vertical coordinate, width and height before conversion respectively; x ′ ,y ′ 、w ′ and h ′ Respectively represent the converted horizontal coordinate, vertical coordinate, width and height.

6. According to claim 1, a method for visual detection of bacterial flora identification and counting based on deep optimization, It is characterized in that The original YOLO-v7 model optimized by the model includes three detection layers and corresponding three sets of initialized anchor box values. The 75th detection layer is used to detect targets larger than 8×8 pixels, the 88th layer is used to detect targets larger than 16×16 pixels, and the 101st layer is used to detect targets larger than 32×32 pixels. The novel network structure of the newly added small target detection layer includes: Perform upsampling on the feature map at the 75th layer to expand the size of the feature map; At the 77th layer, the 160×160 pixel feature map is fused with the 11th layer feature map in the backbone network to obtain a larger feature map for small object detection; A 160×160 pixel detection layer is added at the 87th layer to detect objects larger than 4×4 pixels.

7. The method for visual detection of bacterial flora identification and counting based on deep optimization according to claim 1, It is characterized in that The optimization of the anchor box includes recalculating the size and proportion of the anchor box using a k-means clustering algorithm with 1000 iterations.

8. According to claim 1, a method for visual detection of bacterial flora identification and counting based on deep optimization, It is characterized in that Develop applications with visual user interfaces that support cloud and local deployment, and implement the entire process from image acquisition, preprocessing, uploading to automatic detection technology, and then to manual review, revision, and warehousing.

9. The method for visual detection of bacterial flora identification and counting based on deep optimization according to claim 8, It is characterized in that The program has the functions of storing, classifying, retrieving, and visualizing bacterial flora images, as well as performing simple analysis and early warning on the number of bacterial flora categories.

Images