Detection method of sandwich paperboard in tobacco packaging box based on neural network
Through the neural network-based detection method, the attention mechanism and HSV color space conversion are used to solve the problem of recognition error of the interlayer cardboard in the tobacco leaf packaging box, achieving high-precision recognition effect, and the detection accuracy is 99%.
Patent Information
- Application Number
- CN202210602092.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-30
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2042-05-30
AI Technical Summary
The prior art has large detection errors when identifying the mezzanine cardboard in the tobacco leaf packaging box. Especially in complex environments, it is difficult for traditional machine learning methods to judge the mezzanine cardboard through nuanced area segmentation.
Using a neural network-based detection method, the original image is collected, histogram enhancement processing is performed and converted to the HSV color space, and a preset residual neural network based on attention mechanism is used for classification recognition, combining space and channel attention mechanism extraction Features, whether there is a mezzanine board in the output.
The accuracy of identification and judgment of easily confusing targets in the tobacco leaf box is improved, and the accurate extraction and identification of yellow cardboard areas is achieved, with the detection accuracy reaching 99%, far exceeding the traditional visual detection methods.
Smart Images
Figure CN115019090B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of cigarette manufacturing, and in particular to a method for detecting sandwich paperboard in a tobacco packaging box based on a neural network. Background Art
[0002] When a robot (such as KUKA) is unable to identify and judge the interlayer conditions in tobacco packaging boxes on the production line, it is easy to cause misoperation. Specifically, yellow cardboard is mostly used as the interlayer inside tobacco packaging boxes. When extracting the features of yellow cardboard, the yellow cardboard is generally yellowish and the tone is not fixed. The shape of the cardboard is generally rectangular. In the tobacco packaging box, parts of the yellow cardboard are usually covered by tobacco leaves with similar features. At the same time, tobacco leaves come from different origins, and their surface textures, colors and other features are relatively complex, resulting in undetectable situations during the detection process. For example, the robot determines that there is no interlayer board in the tobacco packaging box, causing the robot to misjudge the interlayer conditions in the packaging box on the production line.
[0003] Tobacco technology practitioners have always attached great importance to the research on image feature extraction in complex environments. However, for the internal interlayer detection of tobacco packaging boxes, it has been found through practice that if traditional machine learning methods are used, there will be large detection errors, especially in the tobacco industry. The research and application of image feature recognition mainly focuses on extracting shallow features such as leaf color and contour. However, it is difficult to directly use the existing image detection strategies in the tobacco industry to segment and judge the interlayer paperboard of tobacco materials by slightly different areas under relatively similar colors. Summary of the invention
[0004] In view of the above, the present invention aims to provide a method for detecting interlayer cardboard in tobacco packaging boxes based on a neural network to solve the technical problems mentioned above.
[0005] The technical solution adopted by the present invention is as follows:
[0006] The present invention provides a method for detecting sandwich paperboard in a tobacco packaging box based on a neural network, which includes:
[0007] Collecting original images related to the sandwich panels in tobacco packaging boxes;
[0008] Performing enhancement processing on the original image through a histogram;
[0009] Convert the enhanced image from RGB color space to HSV color space;
[0010] The tobacco box images converted to the HSV color space are classified and recognized through the preset residual neural network based on the attention mechanism;
[0011] Based on the classification and recognition results, output whether there is a sandwich panel in the tobacco packaging box.
[0012] In at least one possible implementation manner, the original image includes an image of a tobacco packaging box with yellow cardboard or an image of only tobacco leaves without the yellow cardboard.
[0013] In at least one possible implementation manner, the enhancement processing includes: mapping the pixel range of the original image to a range of [0, 255] based on a fixed histogram normalization coefficient.
[0014] In at least one possible implementation, the classification and identification process includes:
[0015] The spatial attention mechanism and the channel attention mechanism are linearly combined, and different attention mechanisms are used to extract features of the area to be identified in the tobacco box image converted to the HSV color space.
[0016] In at least one possible implementation, the feature extraction method includes:
[0017] The image features of tobacco packaging boxes in HSV color space are input into different channel attention modules, and the weight value of each channel is obtained;
[0018] Performing matrix multiplication of the weight value and the image feature to extract the weight distribution of the image feature in different channels;
[0019] Obtaining local area feature distribution from the weight distribution via a spatial attention module;
[0020] The target features are extracted by combining the local area feature distribution and the weight distribution.
[0021] In at least one possible implementation, the channel attention module includes an average pooling layer, a maximum pooling layer, four connection layers, a ReLU activation layer, and a Sigmoid activation layer.
[0022] In at least one possible implementation, the spatial attention module includes an average pooling layer and a maximum pooling layer.
[0023] In at least one possible implementation, the structure of the residual neural network based on the attention mechanism includes: connecting a comprehensive attention module composed of a channel attention module and a spatial attention module after the first convolutional layer of the residual neural network; and connecting an attention module after the entire residual neural network.
[0024] The main design concept of the present invention is to use an improved neural network model to extract features of specific areas in a cigarette box through the distribution characteristics of different colors, thereby achieving accurate identification of tobacco leaves and inner interlayer cardboard in different scenarios. Specifically, the residual neural network is added to the attention mechanism, and the HSV color space conversion mechanism is introduced based on the enhanced image to be processed to meet the image recognition needs of the interlayer board in the tobacco packaging box on the tobacco factory assembly line, solving the problem that the tobacco factory robot cannot accurately judge the processing time before operation and the result after operation cannot be feedback verified. The present invention is applied in the actual production process, and can reliably eliminate tobacco image features while retaining the yellow cardboard area and performing corresponding feature extraction and recognition detection, thereby effectively improving the recognition and judgment accuracy of easily confused targets in the tobacco box. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described below with reference to the accompanying drawings, in which:
[0026] Figure 1 A flowchart of a method for detecting interlayer cardboard in a tobacco packaging box based on a neural network provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0027] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.
[0028] The present invention proposes an embodiment of a method for detecting sandwich paperboard in a tobacco packaging box based on a neural network. Specifically, Figure 1 shown, including:
[0029] Step S1, collecting original images related to the sandwich panels in the tobacco packaging box.
[0030] During the algorithm training stage involved in the present method, original image samples of the sandwich cardboard of the tobacco packaging boxes can be collected in advance from the silk production line using industrial cameras and other equipment. For example, in a certain implementation process, 520 original images are obtained, including 320 images of yellow cardboard in the tobacco packaging boxes and 200 images of only tobacco leaves without yellow cardboard.
[0031] Step S2: Enhance the original image through histogram to make the details such as brightness information, texture information, color information, etc. in the original image clearer. Specifically, the pixel range of the original image can be mapped to the range of [0, 255] on the basis of a fixed histogram normalization coefficient. In actual operation, the grayscale range of the tobacco leaf can be set to [I min ,I max ], and the output tobacco leaf gray value range is [O min ,O max ] (usually [0,255]), the gray value of the pixel at the current position is set to I r,c , and the gray value of the corresponding position is 0 r,c , then the mapping relationship between them is:
[0032]
[0033] Step S3: convert the enhanced image from the RGB color space to the HSV color space.
[0034] In the HSV color space, color parameters are composed of hue (H), saturation (S), and brightness (V). The H parameter represents color information; the S parameter represents the ratio between the purity of the selected color and the maximum purity of the color; and V represents the brightness of the color. The specific conversion of the aforementioned enhanced image from RGB color space to HSV color space can be referred to as follows:
[0035] V←max(R,G,B)
[0036]
[0037]
[0038] Step S4: Classify and identify the tobacco box image converted to the HSV color space through a preset residual neural network (Resnet 34 network model) based on the attention mechanism.
[0039] Specifically, it includes linearly combining the spatial attention mechanism and the channel attention mechanism, and extracting the laminated cardboard (yellow cardboard) features in the tobacco box image by different attention mechanisms. For example, first input the tobacco box image feature x into different channel attention modules, and obtain the weight value W1 of each channel, and use the weight value to perform matrix multiplication with the tobacco box image feature, so as to extract the weight distribution s of the tobacco box image feature in different channels. The formula is:
[0040] s=F1(x,W1)=xW1
[0041] Then, the spatial attention module obtains the local area feature distribution from the weight distribution s. The local area feature distribution and the weight distribution s (may be obtained by matrix multiplication) are combined to obtain the final output feature. The formula is:
[0042] y=F2(s,W2)=sW2
[0043] Regarding the channel attention module mentioned above, it mainly includes an average pooling, a maximum pooling, four connection layers, a ReLU activation layer, and a Sigmoid activation layer. In the data processing process, the global average pooling operation and the global maximum pooling operation are first performed on the input matrix W*H*C. The purpose is to compress the features of each channel and output global features of size 1*1*C respectively. Then, the channel is scaled through a 1*1 convolution layer to reduce the complexity of the calculation. The nonlinear ability is improved through a ReLU activation layer, and then the number of channels is restored through a 1*1 convolution layer. The features calculated separately are added, and finally the results are mapped to the range of 0 to 1 through a Sigmoid function to obtain the weight ratio of different channels. The mature algorithms can be referred to as follows:
[0044] Avg_out=Conv2(ReLU(Conv1(Avg_pool(x))))
[0045] Max_out=Conv2(ReLU(Conv1(Max_pool(x))))
[0046] Out=Sigmoid[Avg_out+Max_out]
[0047] The spatial attention module mentioned above includes an average pooling and a maximum pooling. In the data processing process, a W*H*C feature matrix is first input. After average pooling and maximum pooling operations, the average pooling takes the average weight of the features in the vertical direction of the feature matrix. The purpose of maximum pooling is to obtain the maximum weight of the features in the vertical direction of the feature matrix, that is, the obtained matrix size is W*H*1. After that, the matrix splicing operation is performed and a 7*7 convolution kernel operation is performed to change the channel parameters. Finally, as with the channel attention module, Sigmoid activation processing will be performed, and the result will be mapped to the range of 0 to 1 to obtain the weight ratio of different areas. The mature algorithms can be referred to as follows:
[0048] Out1=[avg_pool(x),max_pool(x)]
[0049] Out2 = Sigmoid(Conv(Out1))
[0050] In some preferred embodiments of the present invention, the proposed global residual network based on the attention mechanism is based on the ResNet 34 basic network, combined with the attention module and the global residual structure for improvement. Specifically, in the basic network, the comprehensive attention module composed of the channel attention module and the spatial attention module is set after the Conv1 layer of the ResNet 34 network, the purpose is to first extract the features of interest in the initial image, that is, the channel weights and spatial distribution weights; then the features are entered into the residual network, and the processing results are fused and sent to the ReLU layer; and after the residual network (after the ReLU layer), another attention module is set, the purpose is to use the attention mechanism in the subsequent global average pooling process to make the pooling ability more effective, and the feature heat map obtained by the last attention module is averaged and fully connected to output the final result.
[0051] In the residual network, the residual blocks Conv2_x, Conv3_x, Conv4_x, and Conv5_x are shortcut-connected as a whole to form a global residual network, which will further improve the convergence speed of the network and the retention of information, and overcome the problem of feature information loss caused by the ReLU activation function. The improved Resnet 34 network architecture can be found in Table 1.
[0052] Table 1 Improved Resnet 34 network architecture
[0053]
[0054] In terms of loss function, the cross entropy loss function can be selected as the loss function of the model. For the training of the present invention, the main goal is to distinguish and identify tobacco leaves and yellow cardboard. During implementation, existing related algorithms can be used for reference, such as the Adam optimization algorithm. The Adam optimization algorithm combined with the momentum gradient descent algorithm and the RMSprop algorithm can speed up the network training speed, which will not be elaborated or limited in the present invention.
[0055] Step S5: Based on the classification and recognition results, output whether there is a sandwich panel in the tobacco packaging box.
[0056] It has been verified in practice that the improved method provided by the present invention can achieve an accuracy rate of 99% in identifying the yellow cardboard interlayer, far exceeding other existing visual detection methods in the tobacco industry.
[0057] Corresponding to the above-mentioned detection method embodiments, the present invention also provides an embodiment of a neural network-based interlayer cardboard detection device in a tobacco packaging box, which specifically includes an image acquisition module, an image enhancement module, a color space conversion module, a feature extraction and analysis module, and a result output module.
[0058] Corresponding to the above step S1, the image acquisition module is divided into offline or online acquisition to ensure that the original image that meets the requirements is collected. The image acquisition module includes a system composed of an industrial computer, an industrial camera, a touch screen, a cold light source, etc. The acquisition system can realize real-time image acquisition in the tobacco packaging box. In the image acquisition environment, the configuration of a cold light source can play a fill light role, and the industrial camera is fixed on a mounting frame about 3m high vertically from the ground, and the camera can be installed at an angle of about 60° to the horizontal plane. Preferably, the detection accuracy of the industrial camera reaches 0.5mm, so that high-quality tobacco leaves and inner interlayer images can be obtained.
[0059] Corresponding to the above step S2, the image enhancement module solves the problem of low contrast caused by the small grayscale range of the image through histogram normalization, so that the details of the brightness information, texture information, and color information in the tobacco leaf image are clearer. By performing histogram homogenization processing on the collected original image, brighter tobacco leaf and inner layer images are obtained, which is convenient for subsequent processing.
[0060] Corresponding to the above step S3, the color space conversion module converts the enhanced image from the RGB color space to the HSV color space to facilitate subsequent image classification and recognition.
[0061] Corresponding to the above step S4, the feature extraction and analysis module is used to input the image after the above preprocessing into the improved residual network model to realize accurate classification and recognition of the image in the HSV color space. Specifically, the tobacco leaves and inner layers in different scenes are identified through a global residual network model that integrates multiple attention mechanisms.
[0062] Corresponding to the above step S5, the result output module is used to send the judgment result information whether there is a sandwich board in the tobacco packaging box.
[0063] In summary, the main design concept of the present invention is to use an improved neural network model to extract features of specific areas in a cigarette box through the distribution characteristics of different colors, thereby achieving accurate identification of tobacco leaves and inner interlayer cardboard in different scenarios. Specifically, the residual neural network is added to the attention mechanism, and the HSV color space conversion mechanism is introduced based on the enhanced image to be processed, so as to meet the image recognition requirements of the interlayer board in the tobacco packaging box on the tobacco factory assembly line, and solve the problem that the tobacco factory robot cannot accurately judge the processing time before operation and the result after operation cannot be feedback verified. The present invention is applied in the actual production process, and can reliably eliminate the tobacco image features while retaining the yellow cardboard area and performing corresponding feature extraction and identification detection, thereby effectively improving the recognition and judgment accuracy of easily confused targets in the tobacco box.
[0064] In the embodiments of the present invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c may represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c may be single or multiple.
[0065] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings, but the above is only a preferred embodiment of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched into a variety of equivalent schemes by those skilled in the art without departing from or changing the design ideas and technical effects of the present invention; therefore, the present invention is not limited to the scope of implementation shown in the drawings, and all changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the specification and drawings, should be within the protection scope of the present invention.
Claims
1. A method for detecting interlayer cardboard in tobacco packaging boxes based on neural network. It is characterized in that include: Collecting original images related to the sandwich panels in tobacco packaging boxes; Performing enhancement processing on the original image through a histogram; Convert the enhanced image from RGB color space to HSV color space; The tobacco box image converted to the HSV color space is classified and identified by a preset residual neural network based on the attention mechanism, including: linearly combining the spatial attention mechanism and the channel attention mechanism, and extracting features of the to-be-identified area in the tobacco box image converted to the HSV color space by different attention mechanisms; wherein the feature extraction method includes: inputting the image features of the tobacco packaging box in the HSV color space into different channel attention modules, and obtaining the weight value of each channel; performing matrix multiplication of the weight value and the image feature to extract the weight distribution of the image feature in different channels; obtaining the local area feature distribution from the weight distribution through the spatial attention module; and extracting the target feature by combining the local area feature distribution and the weight distribution; Based on the classification and recognition results, output whether there is a sandwich panel in the tobacco packaging box.
2. The neural network-based interlayer cardboard detection method for tobacco packaging boxes according to claim 1, It is characterized in that The original image includes an image in which yellow cardboard exists in a tobacco packaging box or an image in which yellow cardboard does not exist but only tobacco leaves exist.
3. The neural network-based interlayer cardboard detection method for tobacco packaging boxes according to claim 1, It is characterized in that The enhancement process includes: mapping the pixel range of the original image to the range of [0, 255] on the basis of a fixed histogram normalization coefficient.
4. The method for detecting interlayer cardboard in tobacco packaging box based on neural network according to claim 1, It is characterized in that The channel attention module includes an average pooling layer, a maximum pooling layer, four connection layers, a ReLU activation layer, and a Sigmoid activation layer.
5. The method for detecting sandwich paperboard in tobacco packaging box based on neural network according to claim 1, It is characterized in that The spatial attention module includes an average pooling layer and a maximum pooling layer.
6. The method for detecting sandwich paperboard in tobacco packaging box based on neural network according to any one of claims 1 to 5, It is characterized in that The structure of the residual neural network based on the attention mechanism includes: connecting a comprehensive attention module composed of a channel attention module and a spatial attention module after the first convolutional layer of the residual neural network; and connecting an attention module after the entire residual neural network.