A method for classification and identification of common bacterial colonies in food based on improved YOLOX

Abstract

The present invention provides a method for classifying and identifying common bacterial colonies in food based on improved YOLOX, comprising the following steps: step S1: collecting images of common microbial colonies in food; step S2: annotating the images of common microbial colonies in food collected in step S1; step S3: data enhancement of the colony images collected in step S1; step S4: constructing an improved YOLOX target detection model; step S5: performing model lightweight processing on the improved YOLOX target detection model in step S4; step S6: deploying the lightweight processed model in step S5 to a terminal device; step S7: detecting the colony features in the image through the colony identification system constructed in step S6, and outputting the detection results. The application of this technical solution can effectively improve the work efficiency of bacterial species identification in food testing laboratories.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning target detection, and in particular to a method for classifying and identifying common bacterial colonies in food based on an improved YOLOX. Background Art

[0002] With the advancement of science and technology and the development of interdisciplinary studies, establishing effective and rapid classification, identification and detection methods for microorganisms is an important technical means to ensure food safety and human health. Group morphology is one of the effective traditional methods for identifying microorganisms, but the actual identification process is very cumbersome, with low work efficiency and low accuracy. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a method for classifying and identifying common bacterial colonies in food based on the improved YOLOX, to achieve the classification and identification of bacterial flora, and to develop an intelligent identification system for bacterial genus. The system has the characteristics of simple operation, high collection efficiency, high accuracy in identifying microbial species, and fast analysis speed, and can effectively improve the work efficiency of bacterial species identification in food testing laboratories.

[0004] To achieve the above object, the present invention adopts the following technical solution: a method for classifying and identifying common bacterial colonies in food based on improved YOLOX, comprising the following steps:

[0005] Step S1: Collecting images of common microbial colonies in food;

[0006] Step S2: annotating the common microbial colony images in food collected in step S1, annotating the species information, appearance information and colony location information of the colonies, and establishing a database of common microbial colony images in food;

[0007] Step S3: performing data enhancement on the bacterial colony image collected in step S1;

[0008] Step S4: construct an improved YOLOX target detection model to identify common microbial colonies in food;

[0009] Step S5: Performing model lightweight processing on the YOLOX target detection model improved in step S4;

[0010] Step S6: deploying the lightweight model of step S5 to the terminal device, and building a colony identification and analysis system for common microorganisms in food;

[0011] Step S7: The colony image preprocessed in step S3 is used by the colony identification system constructed in step S6 to detect the colony features in the image and output the detection result.

[0012] In a preferred embodiment, the colony image acquisition method in step S1 is to acquire a colony image of common food bacteria in a culture dish through a camera.

[0013] In a preferred embodiment, the labeling method in step S2 is: labeling the collected colony images by using the labelme tool to label the species information, appearance information and location information of the colonies respectively.

[0014] In a preferred embodiment, the data enhancement operations in step S3 include: random cropping, random rotation, random horizontal flipping, random brightness transformation, random saturation transformation, random blurring, random noise, random deformation, mosaic enhancement, and copy enhancement of the colony image. The above data enhancement is used as a training data set for the detection model to enhance the generalization performance and robustness of the detection model.

[0015] In a preferred embodiment, the YOLOX model in step S4 is improved as follows:

[0016] Step S41: YOLOX uses MBConv and Fused-MBConv modules to replace CSP modules for backbone feature extraction, and uses lightweight neural network modules to improve detection speed and ensure real-time detection.

[0017] Step S42: MBConv and Fused-MBConv modules integrate CA coordinate attention mechanism, ECA channel attention mechanism and DFC long-term spatial attention mechanism to form lightweight Attention-MBConv module and Attention-Fused-MBConv module. Attention-Fused-MBConv module is used in the shallow area of ​​the backbone network, and Attention-MBConv module with deep convolution and point-by-point convolution is used in the deep area of ​​the backbone network. Without greatly increasing network parameters and computational complexity, the feature extraction capability of the backbone network is enhanced, and the channel information and spatial information of the feature map are mined.

[0018] Step S43: adding an E-SPP structure to the tail of the backbone feature extraction network to obtain multi-scale image information of a single feature map, increase the model receptive field, and enable the detection model to adapt to input images of different sizes;

[0019] Step S44: The feature extraction layer uses two sets of weighted bidirectional feature pyramid network structures BiFPN to replace the original FPN structure, so that the detection model pays more attention to important levels and realizes a fast and efficient multi-scale fusion method;

[0020] Step S45: Add two detection heads for small targets to the original three detection heads of YOLOX to improve the sensitivity of the detection model to small colonies; use the anchor-free mechanism without setting the anchor box; and add an auxiliary head to improve the recall rate of the model. At the same time, it is only used during model training and does not increase the detection time in the deployment phase.

[0021] In a preferred embodiment, the model lightweight processing method in step S5 is: through structural re-parameterization, the convolution layer and the BN layer in the detection model in S4 are merged to reduce the reasoning time of the detection model; and then the model is further compressed through pruning and quantization so that the detection model can be deployed in a lightweight terminal.

[0022] In a preferred embodiment, the structure of the common bacterial colony classification and identification system in step S6 is as follows:

[0023] Step S61: Use Qt to build the host computer of the detection system to realize the classification and identification functions of the bacterial colonies; and provide the function of adding other bacterial colony categories;

[0024] Step S62: deploy the lightweight model in S5 to NVIDIA's Jetson Nano B01 terminal in TensorRT format;

[0025] Step S63: Use the RER-USB48MP02 camera to collect colony images.

[0026] Compared with the prior art, the present invention has the following beneficial effects: the present invention applies the improved YOLOX target detection model to the colony classification and identification of common bacteria in food, can accurately classify 33 types of colonies, and can accurately identify microbial species. The parameters and calculation amount of the YOLOX model before the improvement are greatly reduced, while ensuring the high accuracy of the model detection. It has certain application and research value in terms of colony classification and identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A method flow chart of a common bacterial colony classification and identification system in food based on improved YOLOX according to a preferred embodiment of the present invention

[0028] Figure 2 These are colony images of some Penicillium species;

[0029] Figure 3 These are the colony images of Escherichia coli on different culture media;

[0030] Figure 4The original MBConv structure and Fused-MBConv structure diagram of EfficientNetV2;

[0031] Figure 5 The structure diagrams of the Attention-MBConv and Attention-Fused-MBConv in the preferred embodiments of the present invention are as follows;

[0032] Figure 6 The structure diagram of the CA coordinate attention mechanism used in the improved YOLOX backbone network of the present invention;

[0033] Figure 7 The structure diagram of the ECA channel attention mechanism used in the improved YOLOX backbone network of the present invention;

[0034] Figure 8 This is a structural diagram of the DFC long-term spatial attention mechanism used in the improved YOLOX backbone network of the present invention;

[0035] Fig. 9 The structure diagram of E-SPP used in the improved YOLOX backbone network of the present invention;

[0036] Fig.10 This is a BiFPN structure diagram used in the feature extraction network of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0039] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0040] The present invention provides Figure 1-Figure 10 A system for classifying and identifying common bacteria in food based on an improved YOLOX is shown, referring to Figure 1 , the system design method comprises the following steps:

[0041] Step 1: Professional food researchers culture common bacteria in food and use image acquisition equipment to collect images of colonies in the culture dish. Figure 2 and Figure 3 Examples of colony images of different species of the same genus and colony images of the same species on different culture media are given, among which: Figure 2 These are colony images of some Penicillium species. Figure 3 These are images of Escherichia coli colonies on different culture media.

[0042] Step 2: Use the labelme tool to annotate the common bacterial colony images in food collected in step 1 in the coco format. The main annotation information includes: the species of the colonies in the image, the size of the colonies, and the spatial coordinates of the location. 33 types of common bacterial genera in food were obtained through annotation, including Pseudomonas, Bacillus, Salmonella, Vibrio, Aeromonas, Campylobacter, Paenibacillus, Moraxella, Trichoderma, Citrobacter, Carnobacterium, Corynebacterium, and Interactobacillus. Tautomonas, Hafnia, Kochella, Leuconostoc, Proteus, Enterococcus, Enterobacter, Shewanella, Erwinia, Escherichia, Micrococcus, Pantoea, Lactococcus, Pediococcus, Psychrobacter, Staphylococcus, Rhizopus, Trichoderma, Geotrichum, Fusarium, Yeast (Brettanomyces, Candida, Debaryomyces, Hansenula, Kluyveromyces, Pichia). There are about 500 images per species on average.

[0043] Step 3: Perform data enhancement operations on the collected images of common bacterial colonies in food. The enhancement methods are: random cropping, random rotation, random horizontal flipping, random brightness change, random saturation change, random blur, random noise, random deformation, Mosaic enhancement (i.e., splicing and mixing four colony images), and copy enhancement (i.e., copying and pasting the target of the food colony image).

[0044] Step 4. Select the YOLOX detection model as the original food common bacteria colony detection model, use Pytorch to build the YOLOX detection model, optimize the network model based on the original model, and make targeted structural adjustments according to the detection task.

[0045] The backbone network structure of YOLOX is adjusted as shown in Table 1. The original YOLOX backbone network is replaced by the EfficientNetV2 structure, and the MBConv and Fused-MBConv structures in EfficientNetV2 are fused with coordinate attention CA, channel attention ECA, and long-term spatial attention DFC to form new structural blocks: Attention-MBConv and Attention-Fused-MBConv.

[0046]

[0047] The MBConv and Fused-MBConv structures in EfficientNetV2 are as follows Figure 4 As shown, depthwise convolution and point-by-point convolution are used to reduce the amount of computation and parameters of the model.

[0048] The coordinate attention CA structure is as follows Figure 6 As shown in the figure, through horizontal pooling and vertical pooling, the spatial information in the horizontal and vertical directions is obtained. After the convolution is used for information fusion, the spatial relationship is used to weight the original feature map to realize the coordinate attention mechanism.

[0049] The channel attention ECA structure is as follows Figure 7 As shown in the figure, the compression information of each channel is obtained through global average pooling. After the channel information is fused by convolution, the weight information of each channel is obtained and weighted with the original feature map to realize the channel attention mechanism.

[0050] The long-term spatial attention DFC structure is as follows Figure 8 As shown in the figure, the attention mechanism is executed in parallel with the backbone network structure to achieve the effect of long-term information retention; the spatial information of the feature map is extracted and weightedly fused with the parallel feature map to realize the long-term spatial attention mechanism.

[0051] The structures of Attention-MBConv and Attention-Fused-MBConv that integrate multiple attention mechanisms are as follows: Figure 5 As shown in the figure, the Attention-MBConv and Attention-Fused-MBConv structures introduce an attention mechanism with a small number of parameters while maintaining lightweight, thus ensuring the detection accuracy of the model.

[0052] Furthermore, the SPP in the original YOLOX network backbone is replaced by the E-SPP structure. The E-SPP structure is as follows: Fig. 9As shown in the figure, cascade structure pooling is used to achieve efficient pooling; the residual structure is used to solve the degradation problem that is prone to occur in neural networks; the E-SPP structure is used to increase the receptive field of the model, so that the detection model can have better detection performance for small targets and multi-scale targets, and at the same time enable the detection network to accept input images of different scales.

[0053] Furthermore, the original YOLOX feature extraction structure is replaced by the BiFPN structure. The BiFPN structure is as follows: Fig.10 As shown in the figure, a residual structure is added to PAFPN to enhance the feature expression ability; nodes with single input edges are removed to enhance the efficiency of the network structure; weight fusion is used, Fast-softmax is used to improve the detection speed, and the idea of ​​attention mechanism is integrated.

[0054] Furthermore, the original YOLOX detection head adds two detection heads to address the small target problem of common bacterial colonies in food, thereby improving the sensitivity of the detection model to small colonies; using an anchor-free mechanism to reduce the design of anchor hyperparameters, while reducing the amount of calculation and parameters generated by multiple anchors, and enhancing the generalization performance of the detection model; in addition, an auxhead is added to improve the recall rate of the detection model, and it is only used during model training and does not increase the detection time during the deployment phase.

[0055] Step 5. The constructed improved YOLOX detection model is re-parameterized through the structure, and the convolutional layer and BN layer in the model backbone network are merged. During model training, they are expressed as convolutional layers and BN layers, and during model reasoning, they are expressed as convolutional layers, reducing the number of model parameters and the amount of calculation. The constructed YOLOX detection model is further compressed through pruning and quantization, so that the detection model can be deployed on lightweight terminals.

[0056] Step 6: Deploy the lightweight and improved YOLOX detection model to NVIDIA Jetson Nano B01 in TensorRT format. This terminal is designed for AI and has more powerful performance than Raspberry Pi. It is equipped with a quad-core Cortex-A57 processor, a 128-core Maxwell GPU and 4GB LPDDR memory, which can provide sufficient AI computing power for robot terminals and industrial vision terminals.

[0057] Step 7: Collect colony images through the RER-USB48MP02 camera, process the collected colony images through the JetsonNano B01 terminal, use QT to build the host computer of the detection system, and display the detection results.

[0058] Specific functions of the detection system: Based on the differences in colony characteristics, including shape, size, gloss, viscosity, transparency, edge, protrusions, front and back colors, whether water-soluble pigments are secreted, etc., the improved YOLOX detection model is used to effectively identify and classify bacterial species.

Claims

1. A method for classifying and identifying common bacterial colonies in food based on improved YOLOX, characterized in that: The following steps are involved: Step S1: Collecting images of common microbial colonies in food; Step S2: annotating the common microbial colony images in food collected in step S1, annotating the species information, appearance information and colony location information of the colonies, and establishing a database of common microbial colony images in food; Step S3: performing data enhancement on the bacterial colony image collected in step S1; Step S4: construct an improved YOLOX target detection model to identify common microbial colonies in food; Step S5: Performing model lightweight processing on the YOLOX target detection model improved in step S4; Step S6: deploying the lightweight model of step S5 to the terminal device, and building a colony identification and analysis system for common microorganisms in food; Step S7: The colony image preprocessed in step S3 is used by the colony identification system constructed in step S6 to detect the colony characteristics in the image, wherein the colony characteristics include: shape, size, gloss, viscosity, transparency, edge, ridge, front and back color, and whether water-soluble pigments are secreted, and the detection result is output; The YOLOX model in step S4 is improved as follows: Step S41: YOLOX uses MBConv and Fused-MBConv modules to replace CSP modules for backbone feature extraction, and uses lightweight neural network modules to improve detection speed and ensure real-time detection. Step S42: MBConv and Fused-MBConv modules integrate CA coordinate attention mechanism, ECA channel attention mechanism and DFC long-term spatial attention mechanism to form lightweight Attention-MBConv module and Attention-Fused-MBConv module. Attention-Fused-MBConv module is used in the shallow area of ​​the backbone network, and Attention-MBConv module with depthwise convolution and pointwise convolution is used in the deep area of ​​the backbone network. Step S43: adding an E-SPP structure to the tail of the backbone feature extraction network to obtain multi-scale image information of a single feature map, increase the model receptive field, and make the detection model adapt to the input of images of different sizes; Step S44: The feature extraction layer uses two sets of weighted bidirectional feature pyramid network structures BiFPN to replace the original FPN structure; Step S45: Add two detection heads for small targets to the three detection heads of the original YOLOX to improve the sensitivity of the detection model to small colonies; use the anchor-free mechanism without setting the anchor box; and add an auxiliary head to improve the recall rate of the model. At the same time, it is only used during model training and does not increase the detection time in the deployment phase. The structure of the common bacterial colony classification and identification system in step S6 is as follows: Step S61: Use Qt to build the host computer of the detection system to realize the classification and identification functions of the bacterial colonies; and provide the function of adding other bacterial colony categories; Step S62: deploy the lightweight model in S5 to NVIDIA's Jetson Nano B01 terminal in TensorRT format; Step S63: Use the RER-USB48MP02 camera to collect colony images.

2. The method for classifying and identifying common bacterial colonies in food based on improved YOLOX according to claim 1, characterized in that: The colony image acquisition method in step S1 is to acquire a colony image of common food bacteria in a culture dish through a camera.

3. The method for classifying and identifying common bacteria colonies in food based on improved YOLOX according to claim 1, characterized in that: The labeling method in step S2 is: labeling the collected colony images by using the labelme tool, and labeling the species information, appearance information and location information of the colonies respectively.

4. The method for classifying and identifying common bacterial colonies in food based on improved YOLOX according to claim 1, characterized in that: The data enhancement operations in step S3 include random cropping, random rotation, random horizontal flipping, random brightness transformation, random saturation transformation, random blurring, random noise, random deformation, mosaic enhancement, and copy enhancement of the colony image.

5. The method for classifying and identifying common bacterial colonies in food based on improved YOLOX according to claim 1, characterized in that: The model lightweight processing method in step S5 is: through structural re-parameterization, the convolution layer and the BN layer in the detection model in S4 are merged to reduce the reasoning time of the detection model; and then the model is further compressed through pruning and quantization, so that the detection model can be deployed in a lightweight terminal.

Images