Abnormal cut tobacco detection method and system based on deep learning, computer equipment and storage medium

Abstract

An abnormal cut tobacco detection method based on deep learning comprises the following steps that cut tobacco is continuously shot in the cut tobacco conveying process, and original cut tobacco images are obtained; the original tobacco shred image is input into a pre-trained tobacco shred detection model for detection, a tobacco shred abnormal result is output, and the tobacco shred detection model is based on a YOLOv5s network model and comprises a backbone network, a bottleneck layer network and a prediction layer network; the method for detecting the original tobacco shred image through the tobacco shred detection model comprises the steps that feature extraction is conducted on the original tobacco shred image through a plurality of feature extraction modules of a backbone network to output a plurality of basic feature maps, and the basic feature map output by the last feature extraction module is processed through an SPPF module; performing depth feature enhancement on the basic feature map by using a bottleneck layer network to obtain an enhanced feature map; and outputting a prediction result based on the enhanced feature map by using the prediction layer network. The tobacco shreds can be rapidly and accurately detected in the tobacco shred conveying process, and the abnormal condition of the tobacco shreds is determined.

Description

Technical Field

[0001] The present invention relates to the technical field of tobacco production management, and in particular to an abnormal tobacco detection method, system, computer equipment and storage medium based on deep learning. Background Art

[0002] In the process of tobacco processing and production, the quality of tobacco plays a vital role in the quality and production efficiency of the final product. Existing methods for detecting abnormal tobacco mainly rely on manual detection or automatic detection systems based on simple image processing technology. Manual detection is easily affected by subjective factors, has low detection efficiency and insufficient accuracy, and cannot meet the needs of large-scale production. In addition, manual detection methods are inefficient, highly subjective, and have slow information updates. If the status and quantity information of abnormal tobacco cannot be monitored in real time, the abnormality of the silk-making equipment cannot be discovered in time, and the pre-silk-making process cannot be adjusted in time.

[0003] Although the traditional automatic detection system has improved efficiency to a certain extent, its detection accuracy is limited by the algorithm and hardware conditions, especially in complex backgrounds and different types of detection.

[0004] With the rapid development of deep learning technology, some online tobacco detection technologies based on deep learning technology have gradually emerged. The main challenges of existing abnormal tobacco detection based on deep learning algorithms are the accuracy and computational efficiency of the algorithms. Since there are a large number of difficult targets such as small targets, occluded targets, and multi-scale targets in densely stacked tobacco images, and the tobacco is transported at a high speed in the production line, the time available for deep learning models to analyze and predict is very short. This results in a significant difference between the detection performance of abnormal tobacco in a densely stacked state and the detection performance of natural images if the existing deep learning model is used directly without specific design. It is easy for tobacco to enter the subsequent production line, but the detection results are not generated. Summary of the invention

[0005] In order to solve the shortcomings of the abnormal tobacco detection algorithm in the prior art, such as low accuracy, large number of network parameters, slow running speed and difficult deployment, the present invention provides an abnormal tobacco detection method, system, computer equipment and storage medium based on deep learning, which can quickly and accurately detect tobacco during tobacco transportation and determine tobacco abnormalities.

[0006] In order to achieve the above object, the specific scheme adopted by the present invention is: a method for detecting abnormal tobacco based on deep learning, comprising the following steps: The tobacco is continuously photographed during the tobacco conveying process to obtain the original tobacco image; The original tobacco image is input into the pre-trained tobacco detection model for detection and output of tobacco abnormality results. The tobacco detection model is based on the YOLOv5s network model, including a backbone network, a bottleneck layer network, and a prediction layer network. The tobacco detection model detects the original tobacco image in the following ways: The multiple feature extraction modules of the backbone network are used to extract features from the original tobacco image and output multiple basic feature maps, and the basic feature map output by the last feature extraction module is processed by the SPPF module. The feature extraction module includes a Conv submodule and a C3faster submodule which are arranged in sequence. The bottleneck layer network is used to perform deep feature enhancement on the basic feature map to obtain an enhanced feature map, in which the basic feature map processed by the SPPF module is optimized using the BiFormer attention mechanism; The prediction layer network is used to output the prediction results based on the enhanced feature map, and the Shape-IoU loss function is used to perform prediction constraints during the prediction process.

[0007] As a further optimization of the above-mentioned abnormal tobacco detection method based on deep learning: the C3faster submodule includes three high-efficiency hybrid convolution units, a BottleNeck unit and a Concat unit, wherein two of the high-efficiency hybrid convolution units are arranged in parallel and are used to process the input data to generate intermediate data, the BottleNeck unit is used to further process one set of data to generate transition data, and the Concat unit is used to splice the transition data and another intermediate data and input them into the third high-efficiency hybrid convolution unit.

[0008] As a further optimization of the above-mentioned abnormal tobacco detection method based on deep learning: the efficient hybrid convolution unit includes a partial convolution subunit, a point convolution subunit and an activation function subunit.

[0009] As a further optimization of the above-mentioned abnormal tobacco detection method based on deep learning: in the feature extraction module, the Conv submodule is a PConv submodule or a PWConv submodule.

[0010] As a further optimization of the above abnormal tobacco detection method based on deep learning: Shape-IoU loss function is: L Shape-IoU =1-IoU+distance shape +0.5×Ω shape ; Among them, Ω shapeis the shape loss, θ is the degree of attention to the shape loss, ww and hh are the weight coefficients in the horizontal and vertical directions respectively, c is the diagonal distance between the minimum box in the real box and the predicted box, ω t is the width / height loss, and Scale is the scaling factor.

[0011] As a further optimization of the above abnormal tobacco detection method based on deep learning: the method for training the tobacco detection model includes: During the tobacco conveying process, an industrial camera is used to continuously shoot tobacco from a fixed position above to obtain multiple original images; The original image is annotated, and image optimization and data enhancement are performed to obtain multiple sample images, and the number of sample images is more than the original image; All sample images are divided into training set, validation set and test set in a ratio of 7:2:1; Initialize the tobacco detection model and set the learning parameters; Use the training set to train the tobacco detection model; When the tobacco detection model meets the training parameters, the trained tobacco detection model is tested using the validation set and the test set in turn, and the training ends when the test passes.

[0012] As a further optimization of the above-mentioned abnormal tobacco detection method based on deep learning: methods for image optimization of the original image include noise suppression and size cropping, and methods for data enhancement of the original image include random brightness, random rotation and / or random flipping.

[0013] An abnormal tobacco detection system based on deep learning, comprising: A data acquisition device, used for continuously photographing the tobacco shreds during the tobacco conveying process to obtain original tobacco shreds images; A data processing device, used to detect the original tobacco image using a pre-trained tobacco detection model and output a tobacco abnormality result; The data upload module is used to upload tobacco abnormality results to the upper management device.

[0014] Computer equipment, including: Memory for storing computer programs; A processor is used to read and execute the computer program to implement the above-mentioned abnormal tobacco detection method based on deep learning.

[0015] A storage medium is used to store a computer program, which, when executed, implements the above-mentioned abnormal tobacco detection method based on deep learning.

[0016] Beneficial effects:

[0017] 1. Aiming at abnormal tobacco of different scales under complex background, the present invention replaces the C3 module in the YOLOv5s backbone network with the C3-Faster module by introducing partial convolution and point-by-point convolution, effectively reducing the computing and storage requirements while maintaining the performance, and is suitable for abnormal detection in complex environments; 2. The present invention effectively captures local and global dependencies by introducing the BiFormer attention mechanism after SPPF, and introduces a two-layer routing process, combining the standard self-attention mechanism with the global level routing attention mechanism, so that the model can flexibly capture contextual information, better learn the ability to distinguish difficult objects from the background, and improve the detection accuracy of abnormal tobacco; 3. The present invention adopts an improved loss function, which focuses on the shape and scale of the border itself, thereby improving the accuracy of border regression; 4. The present invention can overcome the problems of low accuracy, large number of network parameters, slow operation speed and difficult deployment of existing abnormal tobacco detection algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a structural schematic diagram of the tobacco detection model in the present invention; Figure 2 This is a schematic diagram of the structure of the C3faster module; Figure 3 This is a schematic diagram of the Biformer attention mechanism; Figure 4 is a flow chart of the present invention; Figure 5 It is a flow chart of the tobacco detection model detecting the original tobacco image. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] like Figure 4 and 5 As shown, a method for detecting abnormal tobacco based on deep learning includes S1 to S2.

[0021] S1. Continuously shoot the tobacco during the tobacco conveying process to obtain the original tobacco image. When shooting the tobacco continuously, fix the industrial camera for shooting above the tobacco conveying line. The shooting distance can be changed during the shooting process to ensure that the tobacco can be completely captured by the industrial camera. After obtaining the original tobacco image, the original tobacco image is preprocessed. The preprocessing method mainly includes denoising and resizing. Denoising is used to improve the quality of the original tobacco image, thereby improving the accuracy of the abnormal tobacco detection result. Resizing is used to change the size of the original tobacco image to facilitate input into the tobacco detection model for detection. In this embodiment, the size of the original tobacco image is adjusted to 4096×3000 and uniformly exported to jpg format.

[0022] S2. Input the original tobacco image into the pre-trained tobacco detection model for detection and output the tobacco abnormality result. Figure 1 As shown, the tobacco detection model is based on the YOLOv5s network model, including a backbone network, a bottleneck layer network and a prediction layer network. The backbone network is used to extract features from the input original tobacco image, the bottleneck layer network is used to enhance the features extracted by the backbone network, and the prediction layer network is used to output the prediction results according to the features enhanced by the bottleneck layer network, and the prediction results are reflected in the annotation box in the image. The method for the tobacco detection model to detect the original tobacco image includes S21 to S23.

[0023] S21, using multiple feature extraction modules of the backbone network to extract features from the original tobacco image and output multiple basic feature maps, and the basic feature map output by the last feature extraction module is processed by the SPPF module, the feature extraction module includes a Conv submodule and a C3faster submodule arranged in sequence, wherein the Conv submodule is a PConv submodule or a PWConv submodule. Figure 2 As shown, the C3faster submodule includes three efficient hybrid convolution units DPCBS, a BottleNeck unit and a Concat unit, wherein two efficient hybrid convolution units DPCBS are arranged in parallel and used to process the input data to generate intermediate data, the efficient hybrid convolution unit DPCBS includes a partial convolution subunit, a point convolution subunit and an activation function subunit, the BottleNeck unit is used to further process one set of data to generate transition data, and the Concat unit is used to splice the transition data and another intermediate data and input them into the third efficient hybrid convolution unit DPCBS.

[0024] More specifically, after the original tobacco image is input into the tobacco detection model, it is firstly convolved to generate a basic image of size 640×640×3, and then a Conv submodule is used for preliminary extraction, and then four feature extraction modules are used in sequence for feature extraction, and the second, third and fourth feature extraction modules respectively output a basic feature map to the bottleneck layer network. When the feature extraction module performs feature extraction, it first uses the Conv submodule to perform a convolution process, and then uses the two efficient hybrid convolution units DPCBS in the C3faster submodule to perform efficient convolution processing to generate two sets of intermediate data, one set of intermediate data is directly transmitted to the Concat unit, and the other set of intermediate data is transmitted to the BottleNeck unit for further processing to generate transition data and transmitted to the Concat unit. After that, the Concat unit splices the received transition data and intermediate data, and then processes them through the third efficient hybrid convolution unit DPCBS, and finally outputs the basic feature map. In the efficient hybrid convolution unit DPCBS, by introducing point convolution and partial convolution, the data processing process can be simplified and the operation and detection efficiency of the tobacco detection model can be improved. In the feature extraction module of the present invention, by adopting the C3faster submodule, compared with the traditional YOLOv5s network, while reducing the amount of network parameters, the feature extraction capability is enhanced and the network is lightweight.

[0025] S22, using the bottleneck layer network to perform deep feature enhancement on the basic feature map to obtain an enhanced feature map, wherein the basic feature map processed by the SPPF module is optimized using the BiFormer attention mechanism, that is, the basic feature map output by the fourth feature extraction module is optimized by the BiFormer attention mechanism after being processed by the SPPF module. Figure 3 As shown in the figure, the BiFormer attention mechanism effectively captures local and global dependencies through the BRA module, and introduces a two-layer routing process, combining the standard self-attention mechanism with the global level routing attention mechanism, so that the model can flexibly capture contextual information, better learn the ability to distinguish difficult objects from the background, and improve the detection accuracy of abnormal tobacco.

[0026] S23. Use the prediction layer network to output the prediction result based on the enhanced feature map, and use the Shape-IoU loss function to perform prediction constraints during the prediction process.

[0027] The Shape-IoU loss function is: L Shape-IoU =1-IoU+distanceshape +0.5×Ω shape ; Among them, Ω shape is the shape loss, θ is the degree of attention to the shape loss, ww and hh are the weight coefficients in the horizontal and vertical directions respectively, c is the diagonal distance between the minimum box in the real box and the predicted box, ω t is the width / height loss, Scale is the scaling factor, and is related to the scale of the data set used by the tobacco detection model during training.

[0028] The method for training a tobacco detection model includes the following steps.

[0029] The first step is to use an industrial camera to continuously shoot tobacco from a fixed position above during the tobacco conveying process to obtain multiple original images. After the original images are collected, they need to be filtered first. The filtered original images need to ensure that they basically contain different postures, sizes and occlusions.

[0030] The second step is to annotate the original image, and perform image optimization and data enhancement to obtain multiple sample images, and the sample images are more than the original image. The labeling tool can use the Labelme tool, and the annotated marks can include "running film", "drawing strips" and "stem tags", etc., corresponding to the abnormal conditions of tobacco shreds, which can be selected according to actual needs. The image optimization method for the original image includes noise suppression and size cropping. The noise suppression method can use a conventional Gaussian filtering algorithm. The data enhancement method for the original image includes random brightness, random rotation and / or random flipping.

[0031] In the third step, all sample images are divided into training set, validation set and test set in the ratio of 7:2:1.

[0032] The fourth step is to initialize the tobacco detection model and set the learning parameters, which include image input size, optimizer, momentum parameter, weight decay, learning rate, batch size, and maximum number of iterations. In this embodiment, the specific setting results of the learning parameters are: image input size 640×640, optimizer SGD, momentum parameter 0.937, weight decay 0, training iterations 300, batch size 16, and learning rate 0.0001.

[0033] The fifth step is to use the training set to train the tobacco detection model.

[0034] In the sixth step, when the tobacco detection model meets the training parameters, the trained tobacco detection model is tested using the validation set and the test set in turn. If the test passes, the training ends.

[0035] The present invention further provides an abnormal tobacco detection system based on deep learning, comprising a data acquisition device, a data processing device and a data uploading module.

[0036] A data acquisition device is used to continuously shoot tobacco during the transportation process to obtain the original tobacco image. The data acquisition device can be an industrial camera, which is set above the tobacco conveying line and can be supported by a bracket that can adjust the distance between the industrial camera and the assembly line. The bracket may include a guide rail that is perpendicular to the conveying direction of the assembly line, and a slider is slidably set on the guide rail. The slider is driven by a linear driver such as a cylinder. The industrial camera is set on the slider. By driving the slider to move, the position of the industrial camera can be changed, thereby changing the distance between the industrial camera and the assembly line. By adjusting the position of the industrial camera, it can be ensured that the tobacco is completely captured by the industrial camera during the transportation process. In the process of collecting original images to construct a training set, the position of the industrial camera can also be changed to generate more different images to improve the richness of the training set.

[0037] The data processing device is used to detect the original tobacco image using the pre-trained tobacco detection model and output the tobacco abnormality result. The data processing device can use an ordinary computer. In order to improve the operating efficiency of the tobacco detection model, an NVIDIA GPU product can be installed in the data processing device.

[0038] The data upload module is used to upload the abnormal results of tobacco shreds to the upper management device. After the abnormal results of tobacco shreds are uploaded to the upper management device, the staff can view the statistical data of the abnormal results of tobacco shreds in real time through the upper management device, and then adjust the pre-process of tobacco making or repair the equipment according to the abnormal results of tobacco shreds. Furthermore, when the number of abnormal tobacco shreds detected exceeds a preset threshold, that is, when the number of abnormal results of tobacco shreds exceeds the threshold, the upper management device automatically triggers an alarm signal, and automatically suspends the production line, notifying the operator to repair the production equipment.

[0039] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0040] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0041] The present invention also provides a computer device, including a memory and a processor.

[0042] Memory, used to store computer programs.

[0043] A processor is used to read and execute the computer program to implement the above-mentioned abnormal tobacco detection method based on deep learning.

[0044] The present invention finally provides a storage medium for storing a computer program, which, when executed, implements the above-mentioned abnormal tobacco detection method based on deep learning.

[0045] The memory, as a carrier for storing resources, may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system, a computer program, etc., and the storage method may be temporary storage or permanent storage. The operating system is used to manage and control various hardware devices and computer programs on the electronic device, which may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the adaptive emotion regulation method based on personalized reconfigurable music performed by the electronic device disclosed in any of the aforementioned embodiments, the computer program may further include a computer program that can be used to complete other specific tasks. The processor may adopt a general-purpose processor product based on X86, IA64, RISC, MIPS, ARM and other architectures.

[0046] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting abnormal tobacco based on deep learning, characterized in that: The steps include: The tobacco is continuously photographed during the tobacco conveying process to obtain the original tobacco image; The original tobacco image is input into the pre-trained tobacco detection model for detection and output of tobacco abnormality results. The tobacco detection model is based on the YOLOv5s network model, including a backbone network, a bottleneck layer network, and a prediction layer network. The tobacco detection model detects the original tobacco image in the following ways: The multiple feature extraction modules of the backbone network are used to extract features from the original tobacco image and output multiple basic feature maps, and the basic feature map output by the last feature extraction module is processed by the SPPF module. The feature extraction module includes a Conv submodule and a C3faster submodule which are arranged in sequence. The bottleneck layer network is used to perform deep feature enhancement on the basic feature map to obtain an enhanced feature map, in which the basic feature map processed by the SPPF module is optimized using the BiFormer attention mechanism; The prediction layer network is used to output the prediction results based on the enhanced feature map, and the Shape-IoU loss function is used to perform prediction constraints during the prediction process.

2. The abnormal tobacco detection method based on deep learning according to claim 1, characterized in that: The C3faster submodule includes three efficient hybrid convolution units, a BottleNeck unit and a Concat unit. Two of the efficient hybrid convolution units are arranged in parallel and are used to process the input data to generate intermediate data. The BottleNeck unit is used to further process one set of data to generate transition data. The Concat unit is used to splice the transition data and another intermediate data and input them into the third efficient hybrid convolution unit.

3. The abnormal tobacco detection method based on deep learning according to claim 2, characterized in that: The efficient hybrid convolution unit includes a partial convolution subunit, a point convolution subunit, and an activation function subunit.

4. The abnormal tobacco detection method based on deep learning according to claim 1, characterized in that: In the feature extraction module, the Conv submodule is a PConv submodule or a PWConv submodule.

5. The method for detecting abnormal tobacco based on deep learning according to claim 1, characterized in that: The Shape-IoU loss function is: L Shape-IoU =1-IoU+distance shape +0.5×Ω shape ; Among them, Ω shape is the shape loss, θ is the degree of attention to the shape loss, ww and hh are the weight coefficients in the horizontal and vertical directions respectively, c is the diagonal distance between the minimum box in the real box and the predicted box, ω t is the width / height loss, and Scale is the scaling factor.

6. The abnormal tobacco detection method based on deep learning according to claim 1, characterized in that: Methods for training tobacco detection models include: During the tobacco conveying process, an industrial camera is used to continuously shoot tobacco from a fixed position above to obtain multiple original images; The original image is annotated, and image optimization and data enhancement are performed to obtain multiple sample images, and the number of sample images is more than the original image; All sample images are divided into training set, validation set and test set in a ratio of 7:2:1; Initialize the tobacco detection model and set the learning parameters; Use the training set to train the tobacco detection model; When the tobacco detection model meets the training parameters, the trained tobacco detection model is tested using the validation set and the test set in turn, and the training ends when the test passes.

7. The method for detecting abnormal tobacco based on deep learning according to claim 6, characterized in that: Methods for performing image optimization on the original image include noise suppression and size cropping, and methods for performing data enhancement on the original image include random brightness, random rotation and / or random flipping.

8. An abnormal tobacco detection system based on deep learning, characterized in that: include: A data acquisition device, used for continuously photographing the tobacco shreds during the tobacco conveying process to obtain original tobacco shreds images; A data processing device, used to detect the original tobacco image using a pre-trained tobacco detection model and output a tobacco abnormality result; The data upload module is used to upload tobacco abnormality results to the upper management device.

9. Computer device, characterized in that include: Memory for storing computer programs; A processor, configured to read and execute the computer program to implement a method for detecting abnormal tobacco based on deep learning as described in any one of claims 1 to 7.

10. A storage medium, characterized in that Used to store a computer program, which, when executed, implements an abnormal tobacco detection method based on deep learning as described in any one of claims 1 to 7.

Images