A method, device and system for automatically determining the maintenance quality of packaging equipment based on deep learning

Abstract

The present invention provides a method, device and system for automatically distinguishing the maintenance quality of a wrapping device based on deep learning. The method respectively constructs a recognition model for identifying maintenance parts and a detection model for detecting dirt in maintenance parts. The recognition model adopts a ResNet50 network model embedded with an SE module to improve the accuracy of recognition. The detection model adopts a Faster R-CNN model, adds an RPN network, generates candidate frames based on an Anchor mechanism, integrates feature extraction, candidate frame selection, frame regression and classification into one network, and effectively improves detection accuracy and efficiency. Combined with the information of maintenance part recognition and the results of dirt detection, the automatic distinction of maintenance quality can be achieved, and the accuracy of distinction can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of maintenance identification of rolling and packaging equipment in the tobacco industry, and in particular to a method, device and system for automatically identifying the maintenance quality of rolling and packaging equipment based on deep learning. Background Art

[0002] In cigarette production enterprises, packing equipment is an indispensable core element. Among them, the maintenance of packing equipment is an important part of the cigarette production process, and the judgment of maintenance quality is the key to the entire maintenance. At present, the judgment of the maintenance quality of packing equipment is generally carried out by professional and technical personnel with rich experience. This work not only requires high human resource costs, but also takes a long time, which greatly affects the production efficiency of the factory. With the expansion of production scale, the types and quantity of packing equipment will increase, and the problem of manual maintenance quality judgment will be further aggravated. Therefore, it is necessary to design and develop an automatic, independent system for judging the maintenance quality of packing equipment.

[0003] Traditional image-based object detection and recognition technologies rely heavily on carefully designed features. The features must be invariant to translation, scale, rotation, and illumination, such as the LBP operator used to describe local texture features of images, the HOG algorithm of the histogram of oriented gradients, and the DPM algorithm, an extension of the HOG algorithm. However, due to the limitations of human understanding of visual mechanisms, manually designed features are difficult to accurately describe the target objects to be identified and detected. Secondly, in the task of object detection, in order to detect objects of various scales in an image, a sliding window method is often required, which is a very time-consuming process. The above two shortcomings have greatly limited the application of traditional object recognition and detection technologies in the industry.

[0004] The inventors of this application discovered the following technical problems in the prior art during the implementation of the present invention:

[0005] Traditional image-based target detection and recognition technologies rely heavily on carefully designed features, which must be invariant to translation, scale, rotation, and illumination, such as the LBP operator used to describe local texture features of images, the HOG algorithm of the histogram of directional gradients, and the DPM algorithm, an extension of the HOG algorithm. However, due to the limitations of human understanding of visual mechanisms, manually designed features are difficult to accurately describe the target objects to be identified and detected. Secondly, in target detection tasks, in order to detect targets of various scales in an image, a sliding window method is often required, which is a very time-consuming process. The above two shortcomings have greatly limited the application of traditional target recognition and detection technologies in the industry. In addition, due to the large number of maintenance parts of the packaging equipment and the complexity of automatic judgment of maintenance quality, there is currently no better way to achieve automatic judgment of maintenance quality in the tobacco industry. Summary of the invention

[0006] The present invention proposes a method, device and system for automatically determining the maintenance quality of wrapping equipment based on deep learning, which are used to solve or at least partially solve the technical problem that the methods in the prior art cannot automatically determine the maintenance quality of wrapping equipment or the determination effect is poor.

[0007] In order to solve the above technical problems, the first aspect of the present invention provides a method for automatically determining the maintenance quality of a wrapping device based on deep learning, comprising:

[0008] S1: Construct a recognition model for maintenance part recognition and a detection model for maintenance part dirt detection, wherein the recognition model uses a ResNet50 network model embedded with a compression excitation module: SE module, and the SE module is used to obtain a channel-weighted feature with the same dimension as the output feature after a series of transformation operations on the output feature of the convolution layer; the detection model uses a Faster R-CNN model, including a feature extraction submodule, a region proposal submodule and a classification submodule. The feature extraction submodule is used to extract the features of the maintenance part image input at that time, the region proposal submodule is used to generate a candidate box that may contain the target according to the extracted image features, and the classification submodule is used to classify the dirt and predict the position of the generated candidate area;

[0009] S2: Obtain a maintenance part image dataset by manually taking photos, construct a first training dataset after labeling the images with equipment names, construct a second training dataset after labeling the dirt categories in the images with rectangular boxes, train the recognition model using the first training dataset to obtain a trained recognition model, and train the detection model using the second training dataset to obtain a trained detection model;

[0010] S3: receiving the identification instruction and the corresponding image sent by the control module, wherein the corresponding image is obtained by format conversion according to the original image collected by the image collection module;

[0011] S4: Based on the discrimination instruction, the trained recognition model is loaded to identify the maintenance part of the corresponding image. If the recognition result is the maintenance part, the trained detection model is loaded to detect dirt on the corresponding image;

[0012] S5: Based on the dirt detection results and maintenance part recognition results, and incorporating professional prior knowledge, determine whether the maintenance quality of the maintenance part corresponding to the image is qualified.

[0013] Based on the same inventive concept, the second aspect of the present invention provides a device for automatically determining the maintenance quality of a wrapping device based on deep learning, wherein the device is a server and includes:

[0014] A model building module is used to build a recognition model for maintenance part recognition and a detection model for maintenance part dirt detection. The recognition model uses a ResNet50 network model with an embedded compression excitation module: the SE module. The SE module is used to obtain a channel-weighted feature with the same dimension as the output feature after a series of transformation operations on the output feature of the convolution layer. The detection model uses a Faster R-CNN model, including a feature extraction submodule, a region proposal submodule and a classification submodule. The feature extraction submodule is used to extract the features of the maintenance part image input at that time. The region proposal submodule is used to generate a candidate box that may contain the target based on the extracted image features. The classification submodule is used to classify dirt and predict the position of the generated candidate area.

[0015] A model training module is used to obtain a maintenance part image data set by manually taking photos, construct a first training data set after annotating the image with the equipment name, construct a second training data set after annotating the dirt category in the image with a rectangular frame, train the recognition model using the first training data set to obtain a trained recognition model, and train the detection model using the second training set to obtain a trained detection model;

[0016] An instruction receiving module is used to receive the identification instruction sent by the control module and the corresponding image, wherein the corresponding image is obtained by format conversion according to the original image collected by the image collection module;

[0017] The recognition and detection module is used to load the trained recognition model to identify the maintenance part of the corresponding image based on the discrimination instruction. If the recognition result is the maintenance part, the trained detection model is loaded to detect dirt on the corresponding image;

[0018] The maintenance quality determination module is used to determine whether the maintenance quality of the maintenance part corresponding to the image is qualified based on the dirt detection result and the maintenance part recognition result.

[0019] Based on the same inventive concept, the third aspect of the present invention provides a system for automatically distinguishing the maintenance quality of wrapping equipment based on deep learning, comprising the automatic distinguishing device for the maintenance quality of wrapping equipment based on deep learning, an image acquisition module and a control module as described in the second aspect, wherein the control module is used to send an image acquisition instruction to the image acquisition module, and to convert the format of the image acquired by the image acquisition module, and to send the format-converted image and the parameters related to the distinction instruction to the automatic distinguishing device for the maintenance quality of wrapping equipment, and to receive the distinction result of the automatic distinguishing device for the maintenance quality of wrapping equipment, and the image acquisition module is used to execute the collection of images of various maintenance parts of the current wrapping equipment according to the image acquisition, and transmit them to the control module.

[0020] In one implementation, the image acquisition module is a pair of smart glasses, the control module is a single-board computer, and is also used to feed back the identification results to the smart glasses.

[0021] The above one or more technical solutions in the embodiments of the present application have at least one or more of the following technical effects:

[0022] The present invention provides a method for automatically distinguishing the maintenance quality of a wrapping device based on deep learning, which respectively constructs a recognition model for identifying maintenance parts and a detection model for detecting dirt in maintenance parts, wherein the recognition model adopts a ResNet50 network model embedded with an SE module, which can obtain the importance of each feature channel by learning, and enhance useful features and suppress features that are not very useful for the current task according to the importance, thereby improving the accuracy of recognition; the detection model adopts a Faster R-CNN model, by adding an RPN network (region proposal submodule), generating candidate frames based on the Anchor mechanism, integrating feature extraction, candidate frame selection, frame regression and classification into one network, effectively improving the detection accuracy and efficiency. Furthermore, combined with the information of maintenance part identification and the dirt detection results, the maintenance quality can be automatically distinguished, and the accuracy of the distinction can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1It is a flow chart of a method for automatically determining the maintenance quality of a wrapping device based on deep learning in an embodiment of the present invention;

[0025] Figure 2 It is a structural schematic diagram of the SE module in the recognition model of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of the ResNet50 network model according to an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of a network structure based on the Faster R-CNN framework in an embodiment of the present invention;

[0028] Figure 5 Schematic diagram of the network structure of FPN based on ResNet50 in an embodiment of the present invention;

[0029] Figure 6 It is a structural block diagram of a device for automatically distinguishing the maintenance quality of a wrapping device based on deep learning provided in an embodiment of the present invention;

[0030] Figure 7 This is a structural block diagram of a system for automatically determining the maintenance quality of wrapping equipment based on deep learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The inventor of this application has found through extensive research and practice that the introduction of deep learning technology can overcome the shortcomings of traditional methods. Through the deep model, it can accurately describe the features of the image targets to be detected and identified; at the same time, through parallel computing technology, it overcomes the time-consuming shortcomings of the sliding window method, so that image-based target detection and recognition technology has been increasingly applied in practice, including the tobacco industry. However, due to the large number of maintenance parts of the packaging equipment and the complexity of automatic maintenance quality judgment, there is currently no good way to achieve automatic maintenance quality judgment in the tobacco industry.

[0032] The main inventive concepts of the present invention are as follows:

[0033] First, a recognition model for maintenance part recognition and a detection model for maintenance part dirt detection are constructed respectively. Then, a maintenance part image data set is obtained by manual photography. After annotation, training sets are constructed for training respectively. The discrimination instruction and the corresponding image sent by the control module are received. Then, based on the discrimination instruction, the trained recognition model is loaded to perform maintenance part recognition on the corresponding image. If the recognition result is a maintenance part, the trained detection model is loaded to perform dirt detection on the corresponding image. Finally, based on the dirt detection result and the maintenance part recognition result, and incorporating professional prior knowledge, it is judged whether the maintenance quality of the maintenance part corresponding to the image is qualified.

[0034] The present invention realizes automatic judgment of the maintenance quality of PG equipment (wrapping equipment) through a deep learning method, and improves the accuracy of the deep learning method for judging dirt through model optimization (including building a ResNet50 network model embedded with an SE module and a dirt detection model of Faster R-CNN), and automatically judges the maintenance quality in combination with the dirt detection results and the maintenance part identification results, thereby improving the accuracy of the judgment. In the specific application process, the present invention improves the work efficiency of equipment operation and maintenance personnel, and at the same time has good scalability.

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Embodiment 1

[0037] The embodiment of the present invention provides a method for automatically determining the maintenance quality of a wrapping device based on deep learning, comprising:

[0038] S1: Construct a recognition model for maintenance part recognition and a detection model for maintenance part dirt detection, wherein the recognition model uses a ResNet50 network model embedded with a compression excitation module: the SE module, the SE module is used to obtain a channel-weighted feature with the same dimension as the output feature after a series of transformation operations on the output feature of the convolution layer; the detection model uses a Faster R-CNN model, including a feature extraction submodule, a region proposal submodule and a classification submodule, the feature extraction submodule is used to extract the features of the maintenance part image input at that time, the region proposal submodule is used to generate a candidate box that may contain the target according to the extracted image features, and the classification submodule is used to classify the dirt and predict the position of the generated candidate area;

[0039] S2: Obtain a maintenance part image dataset by manually taking photos, construct a first training dataset after labeling the images with equipment names, construct a second training dataset after labeling the dirt categories in the images with rectangular boxes, train the recognition model using the first training dataset to obtain a trained recognition model, and train the detection model using the second training dataset to obtain a trained detection model;

[0040] S3: receiving the identification instruction and the corresponding image sent by the control module, wherein the corresponding image is obtained by format conversion according to the original image collected by the image collection module;

[0041] S4: Based on the discrimination instruction, the trained recognition model is loaded to identify the maintenance part of the corresponding image. If the recognition result is the maintenance part, the trained detection model is loaded to detect dirt on the corresponding image;

[0042] S5: Based on the dirt detection results and maintenance part recognition results, and incorporating professional prior knowledge, determine whether the maintenance quality of the maintenance part corresponding to the image is qualified.

[0043] Specifically, see Figure 1 , which is a flow chart of a method for automatically distinguishing the maintenance quality of wrapping equipment based on deep learning in an embodiment of the present invention; steps S1 and S2 are performed in advance before automatic quality distinction, that is, the trained model is obtained in advance and does not need to be retrained in each distinction process.

[0044] In one embodiment, the transformation operations included in the SE module in step S1 include a compression operation Fsq(.), an excitation operation Fex(.) and a normalization operation Fscale(.,.), wherein the compression operation is used to compress the input features in the spatial dimension, and the excitation operation Fex(.) includes two fully connected layers, a ReLU function and a Sigmoid function. The first fully connected layer FC1 and the ReLU function are used to extract the correlation between different feature channels, and the second fully connected layer FC2 and the Sigmoid function are used to extract the importance of each feature channel; the normalization operation multiplies the channel weight calculated by the SE module by the original feature, so as to introduce the importance of different channels into the weighted feature map.

[0045] Specifically, see Figure 2 , which is a structural diagram of the SE module in the recognition model of the present invention.

[0046] Given a convolutional layer output feature X, a channel-weighted feature Xs with the same dimension as X can be obtained through a series of transformation operations of the SE module. The SE module transforms the input feature X through three operations: First, the Squeeze operation Fsq(.) performs feature compression in the spatial dimension, converting the H×W×C feature map (H and W represent the height and width of the feature map, respectively, and C is the number of channels of the feature map) into 1×1×C. This operation is often implemented by global average pooling to convert each two-dimensional feature map into a real number ( Figure 2The first one is the Excitation operation Fex(.), which uses two fully connected layers to implement the first fully connected layer FC1 and the ReLU function to extract the correlation between different feature channels, and the second fully connected layer FC2 and the Sigmoid function to extract the importance of each feature channel; the last one is the Scale operation Fscale(.,.), which multiplies the channel weights calculated by the SE module by the original features, thereby introducing the importance of different channels into the weighted feature map.

[0047] See also Figure 3 , is a schematic diagram of the structure of the ResNet50 network model (i.e., recognition module) according to an embodiment of the present invention.

[0048] In one embodiment, the feature extraction submodule of the Faster R-CNN model in step S1 includes a residual network ResNet and a feature pyramid network FPN. ResNet constructs a bottom-up path to extract features from the input image. The processing of FPN includes: the high-level feature map is first upsampled by 2 times to make the scale of the high-level feature map and the low-level feature map Figure 1 The low-level feature map is then convolved with 1x1 to obtain the same number of channels as the high-level feature map. The two feature maps are then added together to obtain a new low-level feature map that incorporates high-level semantic information.

[0049] Specifically, the Feature Pyramid Networks (FPN) based on ResNet50. FPN aggregates information from different layers by means of feature fusion, so that the features of the lower layers can also have rich semantic information, thus better detecting small dirt. The structure of FPN based on ResNet50 is as follows: Figure 5 As shown in the figure, it includes two parts, ResNet and FPN. During the forward propagation of the ResNet network, the input maintenance image will gradually become smaller due to the pooling effect. These feature maps of different scales form a pyramid structure, in which the upper feature map contains the semantic information of the dirt, while the lower feature map has rich detail information of the dirt. In contrast to the bottom-up path of ResNet, by constructing a top-down path, FPN can pass the semantic information of the high-level feature map to the low-level feature map, so that the low-level feature map has both detail information and semantic information, so it can better detect small dirt such as sparse soot and glue scale. The specific process of FPN is: the high-level feature map is first upsampled by 2 times to make the scale of the high-level feature map and the low-level feature map consistent. Figure 1Then the low-level feature map is convolved by 1x1 to obtain the same number of channels as the high-level feature map; finally, the above two feature maps (i.e. the high-level feature map obtained by 2 times upsampling and the feature map with the same number of channels as the high-level feature map obtained by 1x1 convolution) are added to obtain a new low-level feature map that integrates the high-level semantic information. These new feature maps will be used for the detection of dirt later.

[0050] In one implementation, the region proposal submodule RPN in step S1 includes a convolution layer, and the convolution layer uses deformable convolution. A new convolution branch is created in the standard convolution process to specifically generate the offset position of the convolution kernel sampling point.

[0051] Specifically, the deformable convolution adds an offset value to the position of each sampling point, so that the convolution kernel can adaptively sample near the standard position, no longer limited to the regular rectangular grid position in the standard convolution kernel, so as to better match the shape of the target object. The implementation of deformable convolution is to create a new convolution branch in the standard convolution process to specifically generate the offset position of the convolution kernel sampling point. For a B×C×H×W feature map (B represents the number of images in a batch, C represents the number of channels, H and W represent the height and width of the feature map respectively), assuming that a 3×3 convolution kernel is used, because there are 9 sampling points, each sampling point has two directions of x and y, so for each position, 18 values ​​need to be predicted. In this way, sending the feature map to this newly added convolution layer will get an output of (b, 18, h, w) as the sampling offset value of each position. Then, the position of the standard convolution kernel is calibrated according to this offset value, and the convolution operation is performed at the corresponding position of the calibrated feature map, so that the deformable convolution has the ability to learn the spatial geometric deformation of the target object.

[0052] Figure 4 In the Faster R-CNN structure, the Head submodule is the classification submodule, in which ROI Align does not perform any floating-point rounding operations, but obtains the feature values ​​of the floating-point positions through bilinear interpolation, thus avoiding the errors caused by rounding. The specific operation process of ROI Align is to assume that the size of the fixed output feature is K×K. First, the candidate box is divided into K×K sub-regions. Multiple sampling points are set in each sub-region, and the values ​​at the sampling points are usually obtained by bilinear interpolation. In many cases, the maximum pooling method is used, and the maximum value of the sampling points is used as the output of each sub-region. Repeating the above operations for all sub-regions can generate a feature map of size K×K.

[0053] In one embodiment, when constructing the first training data set in step S2, the maintenance parts are cigarette making machines and packaging machines.

[0054] The cigarette making machine includes the tobacco supply part (VE) side door, the bottom protective door of the tobacco supply part, the leveler, the tobacco supply part magnetic separation, the rear protective cover door of the tobacco supply part, the cigarette gun part, the printing system, the tobacco rod rolling and forming part (SE) cloth belt box and front and rear cloth belt rollers, the spider hand protective cover and the V-groove bottom cover, the back protective cover door of the tobacco rod rolling and forming part, the filter tip forming part (MAX) body, the filter tip forming part glue supply and various drum wheels, 23 parts including the drum air hole, the side protective cover of the large wall panel at the rear of the filter tip connection and forming part, the filter tip receiving device, the wheel guide rail, the rubber barrel of the filter tip connection and forming part, the paper cutter holder, the paper cutter, the paper cutter wheel, the rubbing wheel, the rubbing board and the crescent board, the valve island protective cover door of the filter tip connection and forming part, the full tray drive motor of the tray unloader (HCF), the full tray lower tray channel of the tray loader, the inside of the solenoid valve cabinet on the right side of the tray turning machine (MAGOMAT), the feet, the surface of the integrated cabinet and the inside of the surface of the electronic scale box;

[0055] The packaging machine includes 13 parts, including the cigarette lowering channel and surrounding wall panels, the No. 1 wheel and surrounding dead corners, the trademark paper conveying channel, the second and third wheel mold boxes, the sealing components, the inside of the main motor case and the lifting holes, the small box transparent paper packaging machine (CH) entrance part, the cigarette pack conveying part, the small transparent conveying and forming channel, the cigarette pack output part, the strip box glue cylinder and glue barrel part, the strip box conveying folding and forming part, the transparent unfolding conveying part, the strip pack lifting channel and folding and forming.

[0056] In one embodiment, in step S2, when constructing the first training data set, the categories of dirt marked with rectangular frames include cigarette ash, gum scale, cigarette sticks, film, waste paper and iron filings.

[0057] Specifically, the labelimg tool can be used to annotate six types of dirt in artificially captured images, including cigarette ash, glue stains, cigarette sticks, films, waste paper, and iron filings. In addition, data can be enhanced by image flipping, image rotation, and fixed-size random cropping to obtain more data for training dirt detection models. The model can be developed using the mmdetection deep model platform based on the pytorch framework.

[0058] In the specific implementation process, the dirt is collected from the on-site images, 3,000 pictures before and after maintenance, and 509 test pictures are calibrated with image target detection data. By annotating these pictures, the following labels are formulated:

[0059] Table 1 Dirt category labels

[0060]

[0061] The training parameters of the dirt detection model are shown in Table 2:

[0062] Table 2 Training parameters

[0063]

[0064]

[0065] In one embodiment, the dirt detection result is a predicted score of dirt, and step S5 includes: presetting a threshold value, and when the predicted score is higher than the set threshold value, it is determined that the maintenance is unqualified, otherwise it is qualified.

[0066] Specifically, the key to determining whether the maintenance quality is qualified is the determination of the dirt detection results. Therefore, in this embodiment, the predicted score of each dirt output by the detection module is set to a suitable threshold. When it is higher than this threshold, it is determined that the maintenance is unqualified, otherwise it is qualified. In the production process of cigarettes, due to the manufacturing process, only specific dirt will appear in some parts. For example, after the cigarettes are manufactured, the parts responsible for transportation and packaging are mainly composed of paper scraps and seals. Since the cigarettes are already packaged in boxes, dirt such as cigarette ash will basically not appear in this part. Therefore, a lower threshold can be given for paper scraps, and a higher threshold can be given for cigarette ash. Similarly, for the 36 maintenance parts of the PG equipment, thresholds for various dirt can be designed in a targeted manner.

[0067] During the specific implementation process, the present invention tests each module.

[0068] When testing the maintenance part recognition module, the test set is denoted as I1, with a total of 2856 images, including 36 different parts of the machine. Each image contains one part, and the corresponding part name must be labeled before testing. Since the task of the maintenance part recognition module is to determine whether the input image is a part that is undergoing maintenance, the one vs rest strategy is first used to calculate the accuracy of each category, and then the average accuracy of all 36 categories is calculated as the final test indicator. For example, if the test image is a "cigarette gun part", only the identification of the "cigarette gun part" is correct, and identification of other parts is incorrect. The formula for calculating the average accuracy is as follows:

[0069]

[0070] Among them, TP k To identify the correct number, N k is the total number of test images of the kth category.

[0071] When testing the dirt detection module, the test set is denoted as I2, and its images are consistent with those of the test set I1. Unlike the test set I1, the test set I2 has 1628 images containing dirt annotated with rectangular frames using the labelimg tool. Since the method of the present invention pays special attention to whether the type of dirt detected is correct, the mean average precision mAP (Mean Average Precision) is used for evaluation, which is also the most important test indicator for target detection. To calculate mAP, the AP of ash, gum, cigarettes, films, waste paper and iron filings are first calculated separately. The theoretical formula of AP is as follows:

[0072]

[0073] Among them, P is the precision, R is the recall, and P(R) is a function with R as a parameter. As can be seen from the formula, AP is equal to the area under the PR curve. In actual calculation, it is necessary to divide R into N segments in the interval [0,1]. The specific formula is as follows:

[0074]

[0075] in, Here, the definitions of P and R are as follows:

[0076]

[0077]

[0078] In the above definition, TP is the number of correctly detected target boxes, FP is the number of incorrectly predicted prediction boxes, and FN is the number of target boxes that are not predicted. As can be seen from the above, P refers to the ratio of the number of correctly detected target boxes to the number of all predicted target boxes (all_detections), and R refers to the ratio of the number of correctly detected target boxes to the number of all labeled target boxes (all_ground_truths).

[0079] Maintenance quality discrimination module test. The test set is denoted as I3, which is expanded to 9748 images based on the test set I1. Since the goal of the entire system is to automatically judge whether the maintenance quality is qualified or not, the test of this module is the most important test link, so nearly 10,000 test images were collected. Each image is marked as qualified or unqualified. This task is a simple binary classification problem, and its test indicator is the accuracy of the binary classification, which is calculated as follows:

[0080]

[0081] Here, we assume that there are only two types of samples, positive and negative. TP refers to positive sample images that are identified as positive, and TN refers to negative sample images that are identified as negative. We assume that images that fail to meet maintenance standards are positive samples.

[0082] The image test results after training are shown in Table 3:

[0083] Table 3 Image test results

[0084] category mAP / % Maintenance area identification module test 95.7 Dirt detection module test 63 Maintenance quality judgment module test 86.3

[0085] Embodiment 2

[0086] Based on the same inventive concept, this embodiment provides a device for automatically determining the maintenance quality of a wrapping device based on deep learning. The device is a server. Figure 6 , the server includes:

[0087] The model building module 201 is used to respectively build a recognition model for maintenance part recognition and a detection model for maintenance part dirt detection, wherein the recognition model adopts a ResNet50 network model embedded with a compression excitation module: an SE module, and the SE module is used to obtain a channel-weighted feature with the same dimension as the output feature after a series of transformation operations on the output feature of the convolution layer; the detection model adopts a Faster R-CNN model, including a feature extraction submodule, a region proposal submodule and a classification submodule, the feature extraction submodule is used to extract the features of the maintenance part image input at that time, the region proposal submodule is used to generate a candidate box that may contain the target according to the extracted image features, and the classification submodule is used to classify the dirt and predict the position of the generated candidate area;

[0088] The model training module 202 is used to obtain a maintenance part image dataset by manually taking photos, construct a first training dataset after labeling the images with equipment names, construct a second training dataset after labeling the dirt categories in the images with rectangular frames, train the recognition model using the first training dataset to obtain a trained recognition model, and train the detection model using the second training dataset to obtain a trained detection model;

[0089] The instruction receiving module 203 is used to receive the identification instruction sent by the control module and the corresponding image, wherein the corresponding image is obtained by format conversion according to the original image collected by the image collection module;

[0090] The recognition and detection module 204 is used to load the trained recognition model to perform maintenance part recognition on the corresponding image based on the discrimination instruction, and if the recognition result is a maintenance part, load the trained detection model to perform dirt detection on the corresponding image;

[0091] The maintenance quality determination module 205 is used to determine whether the maintenance quality of the maintenance part corresponding to the image is qualified according to the dirt detection result and the maintenance part identification result.

[0092] Since the device introduced in the second embodiment of the present invention is a device used to implement the method for automatically distinguishing the maintenance quality of a wrapping device based on deep learning in the first embodiment of the present invention, a person skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it will not be described in detail here. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0093] Embodiment 3

[0094] Based on the same inventive concept, this embodiment provides a system for automatically determining the maintenance quality of a wrapping device based on deep learning. Figure 7 , including the automatic determination device 303 for maintenance quality of wrapping equipment based on deep learning described in Example 2, namely, a server, an image acquisition module 301 and a control module 302, wherein the control module is used to send an image acquisition instruction to the image acquisition module, and convert the format of the image acquired by the image acquisition module, send the format-converted image and the determination instruction related parameters to the automatic determination device for maintenance quality of wrapping equipment, and receive the determination result of the automatic determination device for maintenance quality of wrapping equipment, and the image acquisition module is used to execute the acquisition of images of various maintenance parts of the current wrapping equipment according to the image acquisition, and transmit them to the control module.

[0095] In one implementation, the image acquisition module is a pair of smart glasses, the control module is a single-board computer, and is also used to feed back the identification results to the smart glasses.

[0096] During the specific implementation process, the smart glasses, the single-board computer, and the server are connected in sequence through the network;

[0097] The smart glasses are used to collect images of various maintenance parts of the packaging equipment according to the image acquisition instructions sent by the single-board computer, and transmit them to the single-board computer;

[0098] The single-board computer is used to send instructions to the smart glasses and the server, and receive feedback data from the smart glasses and the server. Specifically, the single-board computer sends an image acquisition instruction to the smart glasses; after the smart glasses acquire the image of the maintenance part of the wrapping equipment, they upload it to the single-board computer; the single-board computer converts the format of the received image, packages it with the relevant instructions and their parameters, and sends it to the server; the server determines the maintenance quality of the uploaded image, and feeds back the result to the single-board computer, which then feeds back the determination result to the smart glasses.

[0099] The server is used to identify the maintenance parts, detect dirt and judge the maintenance quality of the images of each maintenance point of the wrapping equipment;

[0100] During the specific implementation process, the smart glasses selected are: Rokid Glass 2 foldable AR glasses.

[0101] The server is an HP DL388 Gen10 server with two Intel Xeon-Silver 4110 (2.1GHz / 8-core / 85W) processors; 32GB DDR4-2666 MT / s memory, built-in HPE Smart Array P408i-a / 2GB SRGenlO 12Gb SAS array controller, 1.8T hard drive; 3 PCI-E3.0 slots; embedded 4-port Gigabit Ethernet card, and 1 FlexibleLOM slot.

[0102] The single-board computer selected is LattePanda, a Win 10 development board that runs a full version of Windows 10. It has an onboard Intel quad-core processor, built-in Bluetooth and WiFi modules, and supports USB 3.0 interface, HDMI video output, 3.5mm audio interface, 100Mbps Ethernet port, and MicroSD expansion card slot.

[0103] The technical solution provided in this application is to solve the problems of low efficiency and inconsistent standards of manual inspection of equipment maintenance in the tobacco industry, and designs a roll maintenance quality judgment method, device and system based on deep learning. The maintenance part recognition model based on deep learning judges whether the collected image is the correct maintenance part; the dirt detection model based on deep learning detects unqualified dirt from the collected maintenance image of the time; the maintenance quality judgment module integrated with knowledge judges whether the maintenance of the time is qualified according to the detected dirt and maintenance part information. Through the evaluation effect of the deep learning image test, it can be known that the comprehensive judgment accuracy of the method of the present invention for whether the maintenance quality is qualified or not reaches 86.3%. It is a judgment method with good performance and stable practicality, which can initially meet the introduction of roll maintenance judgment parts in actual production. Only fine-tuning of the maintenance part recognition module is required to complete the system update.

[0104] Since the system introduced in the third embodiment of the present invention is a system used to implement the method for automatically distinguishing the maintenance quality of a wrapping device based on deep learning in the first embodiment of the present invention, a person skilled in the art can understand the specific structure and deformation of the system based on the method introduced in the first embodiment of the present invention, so it will not be described in detail here. All systems used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0105] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically distinguishing the maintenance quality of wrapping equipment based on deep learning, characterized in that: include: S1: Construct a recognition model for maintenance part recognition and a detection model for maintenance part dirt detection, wherein the recognition model uses a ResNet50 network model embedded with a compression excitation module: SE module, and the SE module is used to obtain a channel-weighted feature with the same dimension as the output feature after a series of transformation operations on the output feature of the convolution layer; the detection model uses a Faster R-CNN model, including a feature extraction submodule, a region proposal submodule and a classification submodule. The feature extraction submodule is used to extract the features of the maintenance part image input at that time, the region proposal submodule is used to generate a candidate box that may contain the target according to the extracted image features, and the classification submodule is used to classify the dirt and predict the position of the generated candidate area; S2: Obtain a maintenance part image dataset by manually taking photos, construct a first training dataset after labeling the images with equipment names, construct a second training dataset after labeling the dirt categories in the images with rectangular boxes, train the recognition model using the first training dataset to obtain a trained recognition model, and train the detection model using the second training dataset to obtain a trained detection model; S3: receiving the identification instruction and the corresponding image sent by the control module, wherein the corresponding image is obtained by format conversion according to the original image collected by the image collection module; S4: Based on the discrimination instruction, the trained recognition model is loaded to identify the maintenance part of the corresponding image. If the recognition result is the maintenance part, the trained detection model is loaded to detect dirt on the corresponding image; S5: Based on the dirt detection results and maintenance part recognition results, and incorporating professional prior knowledge, determine whether the maintenance quality of the maintenance part corresponding to the image is qualified.

2. The method for automatically determining the maintenance quality of a wrapping device according to claim 1, characterized in that: The transformation operations included in the SE module in step S1 include compression operation Fsq(.), excitation operation Fex(.) and normalization operation Fscale(.,.), wherein the compression operation is used to compress the input features in the spatial dimension, the excitation operation Fex(.) includes two fully connected layers, a ReLU function and a Sigmoid function, the first fully connected layer FC1 and the ReLU function are used to extract the correlation between different feature channels, the second fully connected layer FC2 and the Sigmoid function are used to extract the importance of each feature channel; the normalization operation multiplies the channel weight calculated by the SE module by the original feature, and is used to introduce the importance of different channels into the weighted feature map.

3. The method for automatically determining the maintenance quality of a wrapping device according to claim 1, characterized in that: The feature extraction submodule of the Faster R-CNN model in step S1 includes a residual network ResNet and a feature pyramid network FPN. ResNet constructs a bottom-up path to extract features from the input image. The processing process of FPN includes: the high-level feature map is first upsampled by 2 times to make the scale of the high-level feature map consistent with the low-level feature map; then the low-level feature map is convolved by 1x1 to obtain the same number of channels as the high-level feature map; then the two feature maps are added to obtain a new low-level feature map that integrates high-level semantic information.

4. The method for automatically determining the maintenance quality of a wrapping device according to claim 1, characterized in that: The region proposal submodule RPN in step S1 includes a convolution layer, which uses deformable convolution. In the standard convolution process, a new convolution branch is created to specifically generate the offset position of the convolution kernel sampling point.

5. The method for automatically determining the maintenance quality of a wrapping device according to claim 1, characterized in that: Step S2, when constructing the first training data set, sets the maintenance parts as cigarette making machines and packaging machines. The cigarette making machine includes the side door of the tobacco supply part, the bottom protective door of the tobacco supply part, the leveler, the magnetic separation of the tobacco supply part, the rear protective cover door of the tobacco supply part, the cigarette gun part, the printing system, the cloth belt box and the front and rear cloth belt rollers of the tobacco rod rolling and forming part, the spider hand protective cover and the bottom cover of the V-groove, the back protective cover door of the tobacco rod rolling and forming part, the body of the filter tip connecting and forming part, the glue supply and each drum wheel of the filter tip connecting and forming part, the drum wheel air hole, the side protective cover of the large wall panel of the rear body of the filter tip connecting and forming part, the filter tip receiving device, the wheel guide rail, the glue barrel of the filter tip connecting and forming part, the paper cutter frame, the paper cutter, the paper cutter wheel, the rubbing wheel, the rubbing board and the crescent board, the valve island protective cover door of the filter tip connecting and forming part, the full tray drive motor of the unloader, the full tray lower tray channel of the loader, the inside of the electromagnetic valve cabinet on the right side of the turner, the foot, the surface of the integrated cabinet and the inside of the surface of the electronic scale box; The packaging machine includes the cigarette lowering channel and surrounding wall panels, the No. 1 wheel and surrounding dead corners, the trademark paper conveying channel, the second and third wheel mold boxes, the sealing components, the inside of the main motor case and the lifting holes, the small box transparent paper packaging machine entrance part, the cigarette pack conveying part, the small transparent conveying and forming channel, the cigarette pack output part, the strip box glue cylinder and glue barrel part, the strip box conveying folding and forming part, the transparent unfolding conveying part, the strip pack lifting channel and folding and forming.

6. The method for automatically determining the maintenance quality of a wrapping device according to claim 1, characterized in that: In step S2, when constructing the second training data set, the categories of dirt marked with rectangular frames include cigarette ash, gum scale, cigarette sticks, film, waste paper and iron filings.

7. The method for automatically determining the maintenance quality of a wrapping device according to claim 1, characterized in that: The dirt detection result is a predicted score of dirt. Step S5 includes: presetting a threshold value. When the predicted score is higher than the set threshold value, it is determined that the maintenance is unqualified, otherwise it is qualified.

8. A device for automatically determining the maintenance quality of a wrapping device based on deep learning, characterized in that: The device is a server, comprising: A model building module is used to build a recognition model for maintenance part recognition and a detection model for maintenance part dirt detection. The recognition model uses a ResNet50 network model with an embedded compression excitation module: the SE module. The SE module is used to obtain a channel-weighted feature with the same dimension as the output feature after a series of transformation operations on the output feature of the convolution layer. The detection model uses a Faster R-CNN model, including a feature extraction submodule, a region proposal submodule and a classification submodule. The feature extraction submodule is used to extract the features of the maintenance part image input at that time. The region proposal submodule is used to generate a candidate box that may contain the target based on the extracted image features. The classification submodule is used to classify dirt and predict the position of the generated candidate area. A model training module is used to obtain a maintenance part image data set by manually taking photos, construct a first training data set after annotating the image with the equipment name, construct a second training data set after annotating the dirt category in the image with a rectangular frame, train the recognition model using the first training data set to obtain a trained recognition model, and train the detection model using the second training set to obtain a trained detection model; An instruction receiving module is used to receive the identification instruction sent by the control module and the corresponding image, wherein the corresponding image is obtained by format conversion according to the original image collected by the image collection module; The recognition and detection module is used to load the trained recognition model to identify the maintenance part of the corresponding image based on the discrimination instruction. If the recognition result is the maintenance part, the trained detection model is loaded to detect dirt on the corresponding image; The maintenance quality determination module is used to determine whether the maintenance quality of the maintenance part corresponding to the image is qualified based on the dirt detection result and the maintenance part recognition result.

9. A deep learning-based automatic identification system for the maintenance quality of packaging equipment, characterized in that: It includes the automatic determination device for the maintenance quality of wrapping equipment based on deep learning, an image acquisition module and a control module as described in claim 8, wherein the control module is used to send an image acquisition instruction to the image acquisition module, and convert the format of the image acquired by the image acquisition module, send the format-converted image and the related parameters of the determination instruction to the automatic determination device for the maintenance quality of wrapping equipment, and receive the determination result of the automatic determination device for the maintenance quality of wrapping equipment, and the image acquisition module is used to execute the acquisition of images of various maintenance parts of the current wrapping equipment according to the image acquisition, and transmit them to the control module.

10. The automatic determination system for the maintenance quality of the wrapping equipment according to claim 9, characterized in that: The image acquisition module is the smart glasses, and the control module is a single-board computer, which is also used to feed back the identification results to the smart glasses.

Images