Cigarette packet texture identification method and system based on binocular camera and convolutional neural network

Abstract

The invention provides a cigarette packet texture recognition method and system based on a binocular camera and a convolutional neural network. The method comprises the following steps: S1, quickly capturing a cigarette packet texture image based on a high-speed binocular camera; s2, selecting a CNN as a main model, and accurately identifying surface texture features of the cigarette packet; and S3, according to the monitoring result, generating an identification report at least comprising the monitoring result and the monitoring time information. Through the provided focusing strategy, the binocular camera system can reach the required image resolution and quality under various production conditions, and the binocular camera is ensured to focus quickly and accurately and capture the fine characteristics of the cigarette packet, so that the strict requirement of an automatic monitoring system on the image quality is met, and the production efficiency is improved. And the monitoring accuracy and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of tobacco product monitoring, and in particular to a cigarette package texture recognition method and system based on a binocular camera and a convolutional neural network. Background Art

[0002] With the development of science and technology, image processing technology and machine vision technology have been gradually applied to the field of tobacco product monitoring technology. This is because traditional tobacco product monitoring methods can no longer meet modern needs. Existing tobacco product monitoring methods are mainly carried out through manual monitoring or simple machine vision technology.

[0003] For production lines and large-scale cigarette / tobacco factories, although manual monitoring can more accurately monitor the quality of tobacco products, it is inefficient and cannot meet the needs of large-scale production. Although simple machine vision technology can improve detection efficiency, it is not accurate and is easily affected by environmental factors.

[0004] Moreover, in practice, considering the space limitations of the production line and the need for continuous operation, third-party monitoring equipment must be quickly installed in the gaps of the production line while ensuring that it does not affect the operation of the entire production line. However, due to the wide variety of tobacco products, the image characteristics of different types of tobacco products are different, which leads to the fact that traditional machine vision technology is often unable to accurately identify different types of tobacco products. Often, the production line of a tobacco factory is designed to be compact to maximize the use of production space. In such an environment, the introduction of third-party monitoring equipment, such as high-speed cameras and machine vision systems, needs to be quickly installed in the limited space of different production lines without interfering with the production process. The task of these devices is to monitor the quality of cigarette packs in real time, including monitoring defects such as loose transparent paper packs, printing errors, and sealing integrity. Due to the wide variety of tobacco products, from classic hard boxes to soft packs, to various special holiday packaging, the image characteristics of each cigarette pack are different, and the positions of the linkage mechanism / transmission mechanism on the production line are different. As a result, traditional machine vision technology performs poorly when monitoring the loose sealing of hard boxes / soft packs and various types of cigarette packs.

[0005] Based on this, it is urgent to construct a focusing strategy to limit the constraint relationship so that the cigarette package or detection device can achieve fast focusing on any line, ensuring that the camera focuses quickly and accurately and captures the subtle features of the cigarette package, thereby meeting the strict requirements of the automated monitoring system for the cigarette package image quality. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a cigarette package texture recognition method and system based on a binocular camera and a convolutional neural network to solve the problems raised in the above background technology.

[0007] In order to achieve the above object, the present invention is implemented by the following technical solution: a cigarette package texture recognition method based on a binocular camera and a convolutional neural network, comprising the steps of:

[0008] S1. Rapidly capture the texture image of the cigarette package based on a high-speed binocular camera. The operation is as follows:

[0009] S1-1, based on the width W, height H, focal length f of the binocular camera, and detail size D on the target cigarette package, determine and construct a geometric constraint relationship between the binocular camera and the target cigarette package to ensure that the binocular camera can clearly and fully capture all details of the target cigarette package;

[0010] S1-2. Construct a focusing strategy to determine the minimum focal length f that meets the basic viewing angle θ and resolution R required for binocular camera detection min , and use this minimum focal length f min Calculate the viewing angle θ required to actually cover the target cigarette pack width and height on the tobacco production line w and θ h , performing a focusing operation so that the cigarette pack texture image generates a binary cigarette pack texture image representing the surface texture features of the cigarette pack in the clearest focus state;

[0011] S2. Select CNN as the main model and build a convolutional neural network model with a network architecture including multiple convolutional layers, pooling layers, and fully connected layers to accurately identify the surface texture features of cigarette packages and ensure the monitoring of target cigarette packages;

[0012] S3. Generate an identification report including at least the monitoring results and monitoring time information according to the monitoring results.

[0013] As a second aspect of the present invention, a cigarette package texture recognition system based on a binocular camera and a convolutional neural network, a memory and a processor are proposed, wherein the memory includes a cigarette package texture recognition program based on a binocular camera and a convolutional neural network, and the cigarette package texture recognition program based on a binocular camera and a convolutional neural network implements the cigarette package texture recognition method when executed by the processor.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] 1. Considering the space limitation of the production line and the requirement of continuous operation, the present invention proposes a focusing strategy so that the binocular camera system can achieve the required image resolution and quality under various production conditions, including different cigarette pack models, sizes and production speeds. This strategy ensures that the camera focuses quickly and accurately and captures the subtle features of the cigarette pack through an automated focusing process, including the acquisition, processing and feature extraction of cigarette pack texture analysis images, as well as the precise calculation of the defocus distance and the direction of cigarette pack texture change, thereby meeting the strict requirements of the automated monitoring system for image quality, improving the accuracy and efficiency of monitoring, and ensuring the continuity and stability of the production line, thereby achieving strict control of product quality;

[0016] 2. In order to determine whether the packaging of the cigarette package meets the production standards, whether there are defects, or whether it is damaged during transportation, the present invention processes the image texture of each partition by using the Sobel operator to extract the surface texture features of the cigarette package image, and improves the accuracy of focusing by accurately identifying these edges. At the same time, it also ensures that every detail of the cigarette package is clearly captured, so as to timely discover problems during the quality control process. For example, if the edge of a certain area of ​​the cigarette package is blurred, it may indicate that the area has unclear printing or material defects, which may be difficult to detect without edge monitoring. The present invention can focus on these key areas more accurately through edge monitoring, thereby improving the reliability and efficiency of monitoring;

[0017] 3. Since the change of lighting conditions of cigarette packs on the production line will affect the gray value and lead to inaccurate segmentation, the present invention constructs a dynamic adjustment threshold of image content under the background of cigarette pack monitoring by comprehensively considering the factors that generate noise, such as strong light and shadow, so as to enhance the visibility of important features in the image, and also effectively suppress noise and improve image quality;

[0018] 4. Since the image types of each cigarette pack are different, the present invention determines to use CNN as the main model for image classification by adopting visual algorithms and large model technology, and constructs a CNN model that can accurately identify the image features of different types of tobacco products, and quickly monitor the characteristics of cigarette packs, such as packaging defects and printing errors, among tens of thousands of cigarette packs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The disclosure of the present invention is described with reference to the accompanying drawings. It should be understood that the drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In the drawings, the same reference numerals are used to refer to the same components. Among them:

[0020] Figure 1 This is a schematic diagram of the overall processing flow of a cigarette package texture recognition method based on a binocular camera and a convolutional neural network proposed in one embodiment of the present invention;

[0021] Figure 2 A schematic diagram of the principle of performing gradient calculation on pixel points of a cigarette pack texture image in four directions (horizontal H, vertical V, diagonal D1, and diagonal D2) to identify changes in cigarette pack texture, as proposed in one embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the principle of the optical path for collecting the texture image of a cigarette package proposed in one embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a focus strategy construction process proposed in an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of a process for training a CNN model using a binary cigarette pack texture image that has been preprocessed in a focusing operation and / or a binarization generation step, as proposed in one embodiment of the present invention. DETAILED DESCRIPTION

[0025] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can propose a variety of interchangeable structural modes and implementation modes. Therefore, the following specific implementation modes and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as the whole of the present invention or as a limitation or restriction to the technical solution of the present invention.

[0026] The present invention is further described in detail below with reference to the accompanying drawings, but is not intended to limit the present invention.

[0027] As an understanding of the technical concept and implementation principle of the present invention, the main purpose of the cigarette pack texture recognition method and system based on binocular camera and convolutional neural network proposed in the present invention is: due to the wide variety of tobacco products, from classic hard boxes to soft packs, to various special holiday packaging, the image features of each cigarette pack are different, and the positions of the linkage mechanism / transmission mechanism on the production line are different. As a result, the traditional machine vision technology cannot focus quickly and accurately when monitoring the loose sealing of hard boxes / soft packs and various cigarette packs, and cannot capture the subtle features of the cigarette pack. Therefore, the present invention, through the proposed focusing strategy, enables the binocular camera system to achieve the required image resolution and quality under various production conditions, including different cigarette pack models, sizes and production speeds. It can be understood that this focusing strategy is to limit the constraint relationship so that the cigarette pack or the device can achieve fast focusing on any line. This focus is obtained by or by obtaining image defects. As long as the focus is completed, the image obtained in the focusing process can be directly used, without the need to repeatedly collect data, to achieve fast processing.

[0028] As a first aspect of the present invention, Figure 1-Figure 5As shown in the figure, a cigarette pack texture recognition method based on a binocular camera and a convolutional neural network is proposed, comprising the following steps:

[0029] S1. Build a high-speed binocular camera detection system to capture images of cigarette packages at a frequency of 1,000 times per minute. It should be noted that the system needs to be equipped with a high-speed binocular camera. The camera model is preferably such as Hikvision's RGB-D smart stereo camera, Obbec's Gemini 2 binocular structured light 3D camera, equipped with a depth engine chip, or a DUO MLX depth camera based on the active infrared binocular principle, which can quickly capture the texture image of cigarette packages to meet the high-speed requirements of the production line. It is understandable that in cigarette package detection technology, the recognition method based on binocular cameras has significant advantages in recognition accuracy and reliability compared with traditional monocular camera methods because it can provide rich spatial information. Often the texture and pattern of cigarette packages have complex three-dimensional features, and binocular cameras can capture this depth information, thus playing a key role in cigarette package detection. The principle is that the binocular camera uses two cameras to capture images of the same object from different angles by simulating the stereoscopic vision principle of the human visual system. The difference between these images, i.e., parallax, is used to calculate the depth information of the object. In cigarette package detection, the depth information of the texture and pattern on the surface of the cigarette package is crucial for identification and classification. Therefore, by adaptively adjusting the focus, the use of a binocular camera can ensure that the key features of the cigarette package, such as texture and pattern, are clearly captured. Since the use of binocular cameras in cigarette package detection also needs to consider the focal length matching problem, that is, the focal length difference between the two images will cause the images to be mismatched or blurred, which will seriously affect the accuracy of the cigarette package detection results. Therefore, by precisely controlling and adjusting the focus and focal length of the binocular camera, this difference can be minimized, thereby improving the accuracy and reliability of cigarette package detection. The specific implementation steps are as follows:

[0030] S1-0. According to the actual needs of the production line of the tobacco factory, the hardware configuration is first completed, such as using a high-performance GPU to process a large amount of image data and execute complex visual algorithms, and using the CPU for logic control. The logic control in this example is at least: linkage control of hardware configuration, that is, synchronization of the target cigarette package conveyor belt with the binocular camera: the conveyor belt needs to be synchronized with the shooting speed of the binocular camera to ensure that each cigarette package can be correctly photographed, light source control: the use of LED light source or laser light source needs to be synchronized with the binocular camera shooting to ensure that the image is captured in the best state. It is understandable that in order to adapt to the monitoring of cigarette packages of different brands and models, the hardware configuration system needs to achieve high-speed and high-precision cigarette package monitoring through high-performance hardware and advanced visual algorithms, and the linkage in structure needs to ensure the synchronization of components including conveyor belts, cameras, and light sources to ensure the efficient operation of the system.

[0031] S1-1. Based on the width W, height H, focal length f, sensor and detail size D of the target cigarette package, determine the geometric constraint relationship between the binocular camera and the target cigarette package to ensure that the binocular camera can capture all details of the target cigarette package clearly and fully.

[0032] Based on the above technical concept, it should be noted that the purpose of determining the detailed geometric constraint relationship between the binocular camera and the target cigarette pack is to ensure that the binocular camera can clearly capture all details of the target cigarette pack. In the actual production line of the tobacco factory, in order to build a binocular camera detection system to monitor the cigarette pack, it is necessary to integrate the following key parameters: the size of the cigarette pack (width W, height H), the focal length f of the camera, the size of the camera sensor (width l), and the detail size D on the cigarette pack. Among them, the preceding detail size D can be understood as when designing the binocular camera detection system, it is necessary to ensure that the resolution and focal length of the binocular camera can capture these detail sizes D for accurate monitoring and analysis. This usually involves accurate calculation of the pixel size of the binocular camera sensor, the focal length of the camera, and the distance between the cigarette pack and the camera, so as to ensure that the three-dimensional image texture information and detail features of the cigarette pack can be clearly captured. All details of the target cigarette package include at least the clarity and integrity of the pattern, text or barcode on the cigarette package, and the detail size D is the minimum detectable size when checking printing defects, such as blur, missing printing or ghosting, or the minimum size when checking seals, such as loose seals or uneven seams at the seals, seams or folds of the cigarette package.

[0033] In one embodiment of the present invention, the constructed geometric constraint relationship includes: ensuring that the viewing angle of the binocular camera satisfies the viewing angle constraint required to cover the width and height of the target cigarette pack:

[0034]

[0035] The focal length f constraint required to ensure that the binocular camera's field of view can cover the maximum size of the target cigarette pack is:

[0036]

[0037] Ensure that the resolution R of the binocular camera meets the resolution constraint required to capture the detail size D on the target cigarette package: R = D / d, where d is the corresponding pixel size on the binocular camera sensor; and

[0038] The distance constraint SD between the binocular camera and the target cigarette pack is:

[0039] As an embodiment of the present invention, the proposed cigarette pack texture recognition method further includes:

[0040] S1-2. Construct a focusing strategy to determine the minimum focal length f that meets the basic viewing angle θ and resolution R required for binocular camera detection min , and use this minimum focal length f min Calculate the viewing angle θ required to actually cover the target cigarette pack width and height on the tobacco production line w and θ h , a focusing operation is performed to generate a binary cigarette pack texture image representing the surface texture features of the cigarette pack in the clearest focus state.

[0041] Based on the above technical concept, it can be understood that the focal length f determines the viewing angle and field of view of the binocular camera, which directly affects the target cigarette pack size range that the binocular camera can cover. The appropriate focal length can ensure that the width and height of the cigarette pack are within the camera's viewing angle, so that they can be clearly captured. The resolution R determines the minimum detail size that the binocular camera can capture. A high-resolution camera can capture smaller details, which is crucial for monitoring subtle features on cigarette packs. Based on this, it is necessary to use the resolution R size as the end condition based on the constructed focus strategy, and obtain the minimum focal length f through iterative solution. min , and in practice, after adjustment and verification, the focal length that enables the camera system to achieve the best imaging effect.

[0042] The above calculation of the minimum focal length f min In the specific implementation process, first, the required resolution R of the camera is determined according to the detail size D that needs to be monitored on the target cigarette package and the corresponding pixel size d on the camera sensor. It can be understood that this resolution is the end condition of the focusing strategy, that is, the focal length f sought should make the actual resolution of the binocular camera meet or exceed this requirement.

[0043] Example: The specific size of cigarette packs varies depending on the type of packaging. The domestic hard and soft packs of cigarette packs mainly have the following general models, as shown in the following table:

[0044]

[0045] Secondly, select an initial estimated focal length f based on experience or the recommended settings of the binocular camera 0 , calculate the angle θ required for the binocular camera to cover the width and height of the target cigarette pack w and θ h It can be understood that calculating the viewing angle θ can ensure that the field of view (FOV) of the binocular camera is large enough to cover the size of the entire cigarette pack. If the viewing angle is too small, the entire target cigarette pack cannot be captured, resulting in incomplete monitoring. Therefore, an appropriate viewing angle θ can avoid image distortion, such as perspective distortion. In this example,

[0046] Again, according to the calculated viewing angle θ w and θ h , determine the maximum viewing angle θ max ,θ max =max(31.36°,47.12°)=47.12°; calculate the distance SD between the binocular camera and the target cigarette pack through the focal length f constraint relationship, Based on this, the actual resolution R required is determined when the distance between the binocular camera and the target cigarette pack is appropriate. act ;

[0047] Finally, by comparing the actual resolution R act The resolution target value R is set according to the detail recognition requirements of the cigarette package monitoring system. tar , perform the fine-tuning step, where if R act <R tar , you need to fine-tune the focus: Repeat the above steps until R act >R tar , end, otherwise, directly record the final focal length end.

[0048] As an embodiment of the present invention, it should be noted that the purpose of constructing the focusing strategy is to achieve efficient integration and precise operation of the camera system in the production line that has been built in the cigarette factory. Considering the space limitations of the production line and the need for continuous operation, the equipment must be quickly built in the gaps of the production line while ensuring that it does not affect the operation of the entire production line. Therefore, by proposing a focusing strategy, the binocular camera system can achieve the required image resolution and quality under various production conditions, including different cigarette pack models, sizes and production speeds. In specific implementation, the focusing strategy uses an automated focusing process, including the acquisition, processing, and feature extraction of cigarette pack texture analysis images, as well as the precise calculation of the defocus distance and the direction of cigarette pack texture change, to ensure that the camera focuses quickly and accurately, captures the subtle features of the cigarette pack, and thus meets the strict image quality requirements of the automated monitoring system.

[0049] In S1-2, the specific process of constructing the focus strategy includes:

[0050] S1-21, such as Figure 2 As shown, the cigarette pack texture image acquisition and preprocessing are performed, and the operations are as follows:

[0051] S1-211. Determine the focus position of the binocular camera to ensure that the acquisition area framed by the width W and height H of the target cigarette package can be clearly captured. Use the binocular camera to capture the laser reflection texture image on the target cigarette package. Within the acquisition area, perform gradient calculation on the pixel points of the cigarette package texture image in the four directions of horizontal H, vertical V, diagonal D1, and diagonal D2 to identify changes in the cigarette package texture.

[0052] Based on the above technical concept, in the process of obtaining and preprocessing the cigarette pack texture image, the gradient calculation is performed on the pixel points of the cigarette pack texture image in four directions. The specific operation of identifying the cigarette pack texture change is as follows: For the horizontal gradient of the cigarette pack texture image The cigarette pack texture changes are identified by comparing the brightness difference between each pixel (p, q) in the cigarette pack texture image and the pixels in the adjacent columns in the horizontal direction: Vertical gradient for cigarette pack texture image Identifying cigarette pack texture changes is done by comparing the brightness differences of adjacent rows in the same row: Diagonal gradient for cigarette pack texture image and The texture changes of cigarette packs are identified by comparing the brightness differences of diagonally adjacent pixels (p, q) from the upper left to the lower right and from the upper right to the lower left: In the formula, h p,q is the pixel value at the pixel position (p, q) in the cigarette pack texture image.

[0053] S1-212, select a pixel point (p, q) in the cigarette pack texture image as the starting point for analysis, and define the diffusion radius r. By calculating the distance between each pixel point and the starting point (p 0 ,q 0 ), In the formula, p 0 ,q 0 The diffusion coefficient is used for weight adjustment to further determine the precise area for analyzing the cigarette pack texture in the acquisition area, ensure that the precise area for analysis covers the key texture features of the target cigarette pack, and apply a filter (such as a Gaussian filter) to the captured cigarette pack texture image in the precise area to reduce image noise and improve image smoothness, and enhance the image contrast through histogram equalization or adaptive histogram equalization to improve image quality.

[0055] Example: Taking the cigarette pack type (hard pack) in the above data table as an example, an example of the above steps in the cigarette pack texture area diffusion and weight adjustment is given. It is known that the size parameters of the hard pack are as follows: width W = 56mm, height H = 87mm, thickness T = 22mm, detail size D = 1mm, sensor pixel size d = 0.01mm.

[0056] In the specific implementation, a pixel point h(p 0 ,q 0 ) is used as the starting point for the analysis. This point can be an area with obvious texture features in the cigarette pack texture image, such as the edge of the cigarette pack or the center of a specific pattern. At this time, a diffusion radius r is defined. This radius can be set according to the size of the cigarette pack and the expected analysis area size or the cigarette factory production line area. Assume that the starting pixel point (p 0 ,q 0 ) is located at the center of the cigarette pack texture image, the pixel coordinates are (W / 2, H / 2) = (56 / 2, 87 / 2), and the diffusion radius r is set to 1 / 10 of the width or height of the target cigarette pack, that is, 5.6 or 8.7 mm, to ensure that the analysis area covers the key texture features of the target cigarette pack. The calculated Euclidean distance

[0057] Based on the above technical concept, it should be noted that in order to further determine the precise area for analyzing the cigarette pack texture in the acquisition area and ensure that the precise area for analysis covers the key texture features of the target cigarette pack, in practice, the Gaussian function can also be used to calculate the weight W of each pixel point. p,q , According to the calculated weights, the pixels in the cigarette pack texture image are weighted to highlight the texture features around the starting point. For example, cigarette packs usually have brand names, patterns, and other decorative designs. The edges and contrast of these printed features are important features in texture analysis. The barcode on the cigarette pack is another important texture feature. The straight edges and black-white contrast of the barcode are crucial for identifying and tracking products. The texture features at the seals and seams of the cigarette packs, such as loose seals or uneven seams, can reflect the packaging quality of the cigarette packs. The cigarette packs may have scratches, stains, or other surface defects during the production process. This can be achieved by adjusting the brightness or color value of the pixels to make the texture features more obvious. Through the above method, the precise area for cigarette pack texture analysis in the hard pack type can be determined, and the characteristics of the cigarette pack texture can be made more prominent by adjusting the weights, thereby improving the accuracy of subsequent analysis.

[0058] S1-22, perform cigarette pack texture image processing and feature extraction, the operation is as follows:

[0059] S1-221. Convert the acquired cigarette package texture image into a grayscale image to simplify the data and reduce computational complexity, and divide the grayscale image into three partitions: highlight, brighter, and darker according to the grayscale value. The custom highlight partition corresponds to the high-reflection area of ​​the target cigarette package, the brighter partition corresponds to the medium-reflection area of ​​the target cigarette package, and the darker partition corresponds to the low-reflection and / or shadow area of ​​the target cigarette package.

[0060] Based on the above technical concept, in the process of cigarette package texture image processing and feature extraction, the specific operations of implementing grayscale post-partitioning corresponding to the reflection area are as follows:

[0061] First, a grayscale image G(p,q) with a single grayscale value is generated by weighted averaging the red, green, and blue channels of the obtained cigarette pack texture image. The common formula is: G p,q =0.299R+0.587G+0.114B, where R, G, and B are the pixel values ​​of the red, green, and blue channels respectively. p,q is the pixel value of the converted grayscale image at position (p,q);

[0062] Secondly, the grayscale threshold T is determined according to the grayscale cigarette packet texture image histogram or empirical value. 1 and T 2 , where the custom grayscale threshold T 1 Used to distinguish darker and lighter partitions, the grayscale threshold T 2 Used to distinguish between brighter and highlighted partitions, the grayscale image G(p,q) is divided into three partitions according to the grayscale threshold:

[0063]

[0064] In the formula, R(p,q) is the partition result, a, b, and c are the highlighted, brighter, and darker partitions respectively;

[0065] Again, apply the partitioning results to the top, bottom, left, right, front, and back of the target cigarette pack to obtain the face partitioning results. In the formula, F i is the partition result of the i-th surface of the target cigarette package, i=1,2,...,6.

[0066] It can be understood that by partitioning the grayscale image, the texture features on different surfaces of the target cigarette package can be identified, and the highlighted area is set to correspond to the high-reflection area of ​​the cigarette package, the brighter area corresponds to the medium-reflection area, and the darker area corresponds to the low-reflection or shadow area. This partitioning helps to analyze the texture features of the cigarette package more carefully and provide support for subsequent texture analysis and quality control.

[0067] S1-222, apply Sobel operators in four directions, horizontal H, vertical V, diagonal D1, and diagonal D2, to each partition to extract the surface texture features of the cigarette pack image in each partition, and obtain the texture sensitivity coefficient T that characterizes the texture change of the cigarette pack sf , based on the texture sensitivity coefficient T sf Generate a binary cigarette pack texture image.

[0068] Based on the above technical concept, it can be understood that in the process of texture extraction and average gray value calculation, by applying the Sobel operator (edge ​​monitoring algorithm) to process each area of ​​the cigarette package texture image, the texture features of the cigarette package surface can be quickly identified and extracted. These features are crucial for the quality control and automated monitoring system of cigarette packages. By analyzing these texture features, it can be determined whether the packaging of the cigarette package meets the production standards, whether there are defects, or whether it is damaged during transportation. By accurately identifying these edges, the accuracy of focus can be improved to ensure that every detail of the cigarette package is clearly captured, so that problems can be discovered in time during the quality control process. If the edges of a certain area of ​​the cigarette package are blurred, it may indicate that there is unclear printing or material defects in the area, which may be difficult to detect without edge monitoring. Through edge monitoring, these key areas can be focused more accurately, improving the reliability and efficiency of monitoring.

[0069] In one embodiment of the present invention, in the process of processing and extracting features of cigarette pack texture images, when using the Sobel operator to extract the surface texture features of the cigarette pack image in each face partition, it is based on respectively calculating the grayscale change rate of the cigarette pack texture image in each partition in the four directions of horizontal H, vertical V, diagonal D1, and diagonal D2. The specific operation is as follows: construct a 3×3 Sobel operator matrix in the four directions of horizontal H, vertical V, diagonal D1, and diagonal D2, respectively, wherein the middle value of the 3×3 Sobel operator matrix is ​​set to 0 to represent The Sobel operator is insensitive to changes in the four directions of horizontal H, vertical V, diagonal D1, and diagonal D2, and the positive and negative values ​​in the Sobel operator are set to represent the brightness changes of the cigarette pack texture image in each partition in the four directions. When it is a positive value, it means that the brightness changes from left to right in the horizontal H direction, from top to bottom in the vertical V direction, from upper left to lower right in the diagonal D1 direction, and from upper right to lower left in the diagonal D2 direction. When it is a negative value, the brightness changes in the four directions decrease.

[0070] In specific implementation, the constructed 3×3 Sobel operator matrix can be: Used to monitor the horizontal edges of cigarette pack texture images. Used to monitor the vertical edges of cigarette pack texture images. Used to monitor the edge of the cigarette pack texture image from the upper left to the lower right direction. It is used to monitor the edge of the cigarette pack texture image from the upper right to the lower left, and then calculate the grayscale change rate in the four directions of horizontal H, vertical V, diagonal D1, and diagonal D2 through convolution operation.

[0071] In one embodiment of the present invention, the texture sensitivity coefficient T representing the texture change of the cigarette pack is obtained. sf The specific operations are as follows:

[0072] First, by calculating the pixel points (p j ,q j ) is weighted according to its gradient amplitude in four directions, and the average texture grayscale value of the cigarette pack texture image in each partition is calculated

[0073]

[0074] In the formula, h i (p j ,q j ) is the pixel point in the i-th partition (p j ,q j ) gray value, N i is the pixel point in the i-th partition (p j ,q j ) quantity, W ij is the pixel point (p j ,q j ) is calculated based on the gradient magnitude,

[0075] W ij =1+α(G H (p j ,q j )+G V (p j ,q j )+G D1 (p j ,q j )+G D2 (p j ,q j )) (14);

[0076] Secondly, according to the average texture gray value The brightness B and contrast C of the cigarette pack texture image in the partition are used to calculate the texture sensitivity coefficient T sf :

[0077]

[0078] Where a and b are the texture sensitivity coefficients T sf The minimum and maximum values ​​of G(p j ,q j ) is the pixel point (p j ,q j ) is the comprehensive gradient amplitude in four directions, β is the adjustment coefficient, which is used to balance the brightness B and contrast C to the texture sensitivity coefficient T sf The influence of the texture sensitivity coefficient T sf The value range is

[0079] In one embodiment of the present invention, the grayscale value may be affected by the change of the lighting conditions of the target cigarette pack on the production line, resulting in inaccurate segmentation. For example, when the target cigarette pack is photographed under strong light, the shadow part may be mistakenly identified as a noise defect. In an image, noise is usually manifested as randomly distributed high or low grayscale value pixels. Therefore, it is necessary to select an appropriate dynamic adjustment threshold P t , Effectively suppress noise and improve image quality.

[0080] The steps to generate a binary cigarette pack texture image are as follows: for each partition i, use the calculated threshold P t To determine the binary result of the pixel; traverse each pixel in the cigarette pack texture image and compare its gray value with the corresponding threshold P t For each pixel, if the gray value is greater than or equal to the dynamic adjustment threshold P t , the grayscale value of the pixel in the binary image is set to the maximum value (usually 255, representing white), indicating that the area is the highlight or brighter part of the target cigarette pack and contains important texture information. If the grayscale value is less than the dynamic adjustment threshold P t , the grayscale is set to the minimum value (usually 0, representing black), indicating that the area is the darker part or background of the cigarette package. After completing the classification of all pixels, the generated binary image will clearly distinguish the different areas of the cigarette package, providing intuitive visual information for subsequent quality control and defect monitoring.

[0081] S1-23, execute the calculation of focus error, the operation is as follows: for each feature pixel point monitored, calculate the deviation between its position in the current binary cigarette package texture image and the ideal position, that is, for each feature pixel point monitored and identified, use Hough transform to locate its actual position in the binary cigarette package texture image, calculate the ideal position of each feature pixel point, and obtain the focus error E used to evaluate the focus accuracy.

[0082] Based on the above technical concept, it should be noted that in the cigarette factory's cigarette pack inspection process, after generating a binary cigarette pack texture image, it is necessary to further verify the focus error by changing the position of several pixel points in the image. If this focus error and the actual positioning of the characteristic pixel points are ignored, a series of serious problems will occur: a. Inaccurate focus will blur the cigarette pack texture image, thereby losing key texture information, which will not only affect the accurate extraction of cigarette pack texture features, but also cause the automated defect monitoring system in the subsequent production line to be unable to identify subtle defects such as scratches or uneven colors. b. In addition, blurred cigarette pack texture images will also directly lead to incorrect classification of cigarette packs, increase classification uncertainty in the production process, increase additional image acquisition requirements, and increase storage and processing costs. Therefore, the ideal position of the characteristic pixel point can be calculated through the Hough transform. If the deviation between the actual position and the ideal position (focus error E) is within an acceptable range, then the focus can be considered accurate. If the deviation exceeds the preset threshold, the system will automatically adjust the focus to ensure that the image quality meets the monitoring requirements.

[0083] In the specific implementation, the first step is to construct a grayscale value based on the average texture The matching algorithm evaluates the actual binary cigarette pack texture image F in each partition i. S Compared with the ideal binary cigarette pack texture image F T The similarity between them is calculated by calculating the average texture gray value difference ΔD of each feature pixel p,q , quantifies the degree of texture matching between two images. The smaller the difference, the more similar the two images are in this area:

[0084]

[0085] In the formula, U and V represent the length and width of the area to be detected, that is, the number of pixels in the horizontal and vertical directions. It can be understood that the area to be detected is the small area divided from the cigarette pack texture image. This area is the basic unit for image processing and analysis. By partitioning the entire cigarette pack texture image, the image processing process can be more finely controlled, and the accuracy of focus error calculation and binarization processing can be improved. S (fs,ft) is the actual binary cigarette pack texture image F S Gray value at position (fs,ft), F T (fs,ft) is the ideal binary cigarette pack texture image F T The gray value at the position (fs, ft). It should be noted that in image processing, the gray value is a numerical value describing the brightness of a pixel, usually ranging from 0 (black) to 255 (white). In this embodiment, the position (fs, ft) is specifically a pixel position in the binary cigarette pack texture image, and the gray value of the position is the color information of the pixel point (which is the same as the gray value of the pixel point (pj ,q j )) are different applications of the same concept. It is a double summation of fs from 0 to U-1 and ft from 0 to V-1 when traversing each pixel in the entire area to be detected.

[0086] Secondly, compare the average texture gray value difference ΔD p,q , find the pixel position that minimizes the difference as the actual binary cigarette pack texture image F representing the actual focus error S Compared with the ideal binary cigarette pack texture image F T The best matching position (x, y) is used to obtain the focus error E that needs to be adjusted: E H =F S (x,y)-F T (x Hi ,y) (18),E V =F S (x,y)-F T (x,y Vi ) (19), In the formula, x Hi ,y Vi 、x D1i and D2i They are the ideal binary cigarette pack texture images F T At the positions of the horizontal H, vertical V, diagonal D1, and diagonal D2, x, y, x D1 and D2 are the actual binary cigarette pack texture images F S At this point, corresponding correction and adjustment measures can be taken to ensure the quality and accuracy of the image.

[0087] S1-24, adjust the focus of the binocular camera and verify, the operation is as follows: judge and adjust the focal length according to the obtained focus error, after adjustment, recapture the binary cigarette package texture image and repeat the feature pixel monitoring and focus error calculation process until the current focus error is reduced to the system preset range, verify the focus effect, ensure that the feature pixels are clear and / or the edges are sharp, and the focus error is minimized. It should be noted that in cigarette package quality inspection, the minimum acceptable clarity of key features (such as cigarette package edges and textures) is 0.05 mm, and the resolution of the binocular camera and sensor is 0.1 mm / pixel. Based on this information, the system preset range can be set to 0.1 mm to ensure that in most cases, the focus error will not affect the recognition of key features. If the monitoring system needs to identify very small defects or texture changes, then the system preset range should be correspondingly smaller.

[0088] Based on the above technical concept, it is understandable that after focus adjustment, the focus error has been reduced to an acceptable range, and the image quality meets the monitoring requirements, so no additional image acquisition is required, reducing computing power. However, since the image type of each cigarette package is different, powerful data processing capabilities are required to store and process the classification and identification of numerous cigarette package image data.

[0089] Based on this, as an embodiment of the present invention, the proposed cigarette pack texture recognition method further includes:

[0090] S2. CNN is selected as the main model. A convolutional neural network model is constructed with a network architecture including multiple convolutional layers to extract features, pooling layers to reduce feature dimensions and computational complexity, and fully connected layers for classification to accurately identify the surface texture features of cigarette packages. A ReLU activation function is added to the network architecture to increase nonlinearity and prevent overfitting. The operation is as follows:

[0091] S2-1, in the cigarette pack texture image preprocessing stage, confirm whether the input cigarette pack texture image is a binary cigarette pack texture image, wherein, if it is a binary image, skip the normalization process and directly proceed to the subsequent steps; otherwise, perform the normalization process first and then proceed to the subsequent steps;

[0092] S2-2. For the binary image, since it only contains two gray levels, the preprocessing step of subtracting the mean RGB is not applicable. However, if the binary image is converted from a color image, or needs to be further processed to adapt to a specific process, the noise in the image can be reduced by subtracting the mean, ensuring that the image size and format match the requirements of the network input layer. Then determine the input layer of the network architecture, which is usually a convolutional layer to receive the preprocessed cigarette pack texture image, and set the input layer size to match the size of the preprocessed cigarette pack texture image;

[0093] S2-3. After the network architecture weights are initialized, the CNN model is trained using the binarized cigarette pack texture images that have been preprocessed in the focusing operation and / or binarization generation step. It is understandable that these images have been prepared through the focusing and binarization steps. During the training process, CNN automatically learns to extract useful features from the images. The prediction results are calculated through forward propagation, and then the loss function, such as cross entropy loss, is minimized by adjusting the weights through back propagation. Then, the network weights are updated using optimization algorithms such as Adam or SGD, and cross validation is used to evaluate the performance of the model on unseen data. The hyperparameters are adjusted accordingly to complete the evaluation and optimization of the CNN model to ensure the monitoring of the target cigarette packs.

[0094] In one embodiment of the present invention, during the CNN model evaluation and optimization stage, it is first necessary to evaluate the model performance on an independent test set, using indicators such as accuracy, precision, and recall to analyze images that are misclassified by the model. Then, based on the evaluation results and error analysis, the network architecture or hyperparameters are adjusted, such as adding more convolutional layers, adjusting the learning rate, or using data enhancement techniques to improve the recognition accuracy of the model. Through the above steps, an efficient CNN model can be constructed and optimized to monitor features in tens of thousands of cigarette packages, such as packaging defects, printing errors, etc., to ensure high-quality monitoring of tobacco products.

[0095] As an embodiment of the present invention, the proposed cigarette package texture recognition method further includes: S3, generating an identification report including at least the monitoring result and monitoring time information according to the monitoring result.

[0096] As a second aspect of the present invention, a cigarette package texture recognition system based on a binocular camera and a convolutional neural network, a memory and a processor are proposed, wherein the memory includes a cigarette package texture recognition program based on a binocular camera and a convolutional neural network, and the cigarette package texture recognition method is implemented when the cigarette package texture recognition program based on the binocular camera and the convolutional neural network is executed by the processor.

[0097] The technical scope of the present invention is not limited to the contents in the above description. Those skilled in the art can make various deformations and modifications to the above embodiments without departing from the technical idea of ​​the present invention, and these deformations and modifications should all fall within the protection scope of the present invention.

Claims

1. A cigarette pack texture recognition method based on a binocular camera and a convolutional neural network, characterized in that: Includes steps: S1. Rapidly capture the texture image of the cigarette package based on a high-speed binocular camera. The operation is as follows: S1-1, based on the width W, height H, focal length f of the binocular camera, and detail size D on the target cigarette package, determine and construct a geometric constraint relationship between the binocular camera and the target cigarette package to ensure that the binocular camera can clearly and fully capture all details of the target cigarette package; S1-2. Construct a focusing strategy to determine the minimum focal length f that meets the basic viewing angle θ and resolution R required for binocular camera detection min , and use this minimum focal length f min Calculate the viewing angle θ required to actually cover the target cigarette pack width and height on the tobacco production line w and θ h , performing a focusing operation so that the cigarette pack texture image generates a binary cigarette pack texture image representing the surface texture features of the cigarette pack in the clearest focus state; S2. Select CNN as the main model and build a convolutional neural network model with a network architecture including multiple convolutional layers, pooling layers, and fully connected layers to accurately identify the surface texture features of cigarette packages and ensure the monitoring of target cigarette packages; S3. Generate an identification report including at least the monitoring results and monitoring time information according to the monitoring results.

2. The cigarette package texture recognition method based on binocular camera and convolutional neural network according to claim 1 is characterized in that: The specific process of constructing the focusing strategy is as follows: Cigarette Packet Texture Image Acquisition and Preprocessing Determine the focus position of the binocular camera to ensure that the acquisition area selected by the width W and height H of the target cigarette pack can be clearly captured, use the binocular camera to capture the cigarette pack texture image reflected by the laser on the target cigarette pack, and in the acquisition area, perform gradient calculation on the pixel points of the cigarette pack texture image in the four directions of horizontal H, vertical V, diagonal D1, and diagonal D2 to identify the change of cigarette pack texture, select a pixel point (p, q) in the cigarette pack texture image as the starting point of the analysis, and define the diffusion radius r, calculate the Euclidean distance between each pixel point and the starting point, and apply the diffusion coefficient for weight adjustment to further determine the precise area for analyzing the cigarette pack texture in the acquisition area, ensure that the precise area of ​​analysis covers the key texture features of the target cigarette pack, and perform denoising and contrast enhancement preprocessing on the captured cigarette pack texture image in the precise area to improve the image quality; Cigarette Packet Texture Image Processing and Feature Extraction The acquired cigarette pack texture image is converted into a grayscale image to simplify the data and reduce the computational complexity, and the grayscale image is divided into three partitions: highlight, brighter, and darker according to the grayscale value. The custom highlight partition corresponds to the high-reflection area of ​​the target cigarette pack, the brighter partition corresponds to the medium-reflection area of ​​the target cigarette pack, and the darker partition corresponds to the low-reflection and / or shadow area of ​​the target cigarette pack. The Sobel operator in four directions of horizontal H, vertical V, diagonal D1, and diagonal D2 is applied to each partition to extract the surface texture features of the cigarette pack image in each partition, and the texture sensitivity coefficient T that characterizes the texture change of the cigarette pack is obtained. sf , based on the texture sensitivity coefficient T sf Generate a binary cigarette pack texture image; Calculating focus error For each monitored characteristic pixel point, the deviation between its actual position and the ideal position in the current binary cigarette package texture image is calculated to obtain a focus error E used to evaluate the focus accuracy, so as to determine the accuracy of the generated binary cigarette package texture image; Adjust the focus of the binocular camera and verify The focal length is judged and adjusted according to the obtained focus error E. After adjustment, the binary cigarette package texture image is recaptured and the feature pixel monitoring and focus error calculation process is repeated until the current focus error is reduced to within the system preset range, and the focus effect is verified to ensure that the feature pixels are clear and / or the edges are sharp, and the focus error is minimized.

3. The cigarette package texture recognition method based on binocular camera and convolutional neural network according to claim 2 is characterized in that: In the process of obtaining and preprocessing the cigarette pack texture image, the gradient calculation is performed on the pixel points of the cigarette pack texture image in four directions. The specific operation of identifying the cigarette pack texture change is as follows: Horizontal gradient for the cigarette pack texture image The cigarette pack texture changes are identified by comparing the brightness difference between each pixel (p, q) in the cigarette pack texture image and the pixels in the adjacent columns in the horizontal direction: Vertical gradient for cigarette pack texture image Identifying cigarette pack texture changes is done by comparing the brightness differences of adjacent rows in the same row: Diagonal gradient for cigarette pack texture image and The texture changes of cigarette packs are identified by comparing the brightness differences of diagonally adjacent pixels (p, q) from the upper left to the lower right and from the upper right to the lower left: In the formula, h p,q is the pixel value at the pixel position (p, q) in the cigarette pack texture image.

4. The cigarette package texture recognition method based on binocular camera and convolutional neural network according to claim 2 is characterized in that: In the process of cigarette packet texture image processing and feature extraction, the specific operations of implementing grayscale post-partitioning corresponding to the reflection area are as follows: Firstly, a grayscale image G(p,q) with a single grayscale value is generated by weighted averaging the red, green and blue channels of the obtained cigarette pack texture image. Secondly, grayscale thresholds T1 and T2 are determined, wherein the custom grayscale threshold T1 is used to distinguish between darker and lighter partitions, and the grayscale threshold T2 is used to distinguish between lighter and highlighted partitions, and the grayscale image G(p,q) is divided into three partitions according to the grayscale thresholds; Again, the partitioning result is applied to the top, bottom, left, right, front and back of the target cigarette packet to obtain the surface partitioning result.

5. The cigarette package texture recognition method based on binocular camera and convolutional neural network according to claim 2 or 4, characterized in that: In the process of cigarette pack texture image processing and feature extraction, the Sobel operator is used to extract the surface texture features of the cigarette pack texture image in each partition. This is based on calculating the grayscale change rate of the cigarette pack texture image in each partition in the horizontal H, vertical V, diagonal D1, and diagonal D2 directions. The specific operations are as follows: A 3×3 Sobel operator matrix is ​​constructed in the four directions of horizontal H, vertical V, diagonal D1, and diagonal D2, respectively. The middle value in the 3×3 Sobel operator matrix is ​​set to 0 to indicate that the Sobel operator is insensitive to changes in the four directions of horizontal H, vertical V, diagonal D1, and diagonal D2. At the same time, the positive and negative values ​​in the Sobel operator are set to represent the brightness changes of the cigarette pack texture image in each partition in the four directions. When it is a positive value, it means that the brightness changes in the horizontal H direction increase from left to right, the brightness increases from top to bottom in the vertical V direction, the brightness changes from upper left to lower right in the diagonal D1 direction, and the brightness changes from upper right to lower left in the diagonal D2 direction. When it is a negative value, the brightness changes in the four directions decrease.

6. The cigarette package texture recognition method based on binocular camera and convolutional neural network according to claim 2 is characterized in that: The texture sensitivity coefficient T that characterizes the texture change of the cigarette package is obtained sf The specific operations are as follows: First, by calculating the pixel points (p j ,q j ) is weighted according to its gradient amplitude in four directions, and the average texture grayscale value of the cigarette pack texture image in each partition is calculated In the formula, h i (p j ,q j ) is the pixel point in the i-th partition (p j ,q j ) gray value, N i is the pixel point in the i-th partition (p j ,q j ) quantity, W ij is the pixel point (p j ,q j ) is calculated based on the gradient magnitude; Secondly, according to the average texture gray value The brightness B and contrast C of the cigarette pack texture image in the partition are used to calculate the texture sensitivity coefficient T sf : Where a and b are the texture sensitivity coefficients T sf The minimum and maximum values ​​of G(p j ,q j ) is the pixel point (p j ,q j ) is the comprehensive gradient amplitude in four directions, β is the adjustment coefficient, which is used to balance the brightness B and contrast C to the texture sensitivity coefficient T sf The influence of the texture sensitivity coefficient T sf The value range is 7. The cigarette package texture recognition method based on binocular camera and convolutional neural network according to claim 2 or 6, characterized in that: Based on the texture sensitivity coefficient T sf When generating a binary cigarette pack texture image, it is necessary to dynamically adjust the threshold P by setting t Then, for each partition i, according to the pixel points (p j ,q j ) grayscale value and dynamically adjusted threshold value P t The comparison results are classified to generate a binary cigarette pack texture image representing the surface texture features of the target cigarette pack in different areas, where:

8. The cigarette package texture recognition method based on binocular camera and convolutional neural network according to claim 1, characterized in that: All the details at least include the clarity and integrity of the pattern, text, barcode, blur, missing or ghosting of the printing on the target cigarette package, as well as the integrity of the sealing seam or folding. The detail size D is the minimum size when the seal is not tight and / or the seam is uneven; wherein the constructed geometric constraint relationship includes: Ensure that the viewing angle of the binocular camera satisfies the viewing angle constraints required to cover the width and height of the target cigarette package; ensure that the viewing angle of the binocular camera can cover the focal length f constraint required for the maximum size of the target cigarette package; ensure that the resolution R of the binocular camera satisfies the resolution constraint required to capture the detail size D on the target cigarette package; and the distance constraint SD between the binocular camera and the target cigarette package.

9. The cigarette package texture recognition method based on binocular camera and convolutional neural network according to claim 1, characterized in that: The minimum focal length f min The calculation process is based on the focus strategy constructed, with the resolution R as the end condition, and is obtained by iterative solution. The specific process is: First, according to the detail size D to be monitored on the target cigarette package and the corresponding pixel size d on the binocular camera sensor, the required resolution R of the binocular camera is determined as the end condition of the focusing strategy; Secondly, select an initial estimated focal length f0 based on experience or the recommended settings of the binocular camera, and calculate the angle θ required for the binocular camera to cover the width and height of the target cigarette pack. w and θ h ; Again, according to the calculated viewing angle θ w and θ h , determine the maximum viewing angle θ max , the distance between the binocular camera and the target cigarette pack is calculated through the focal length f constraint relationship, and the actual resolution R required when the distance between the binocular camera and the target cigarette pack is appropriate is determined accordingly act ; Finally, by comparing the actual resolution R act The resolution target value R is set according to the detail recognition requirements of the cigarette package monitoring system. tar , perform the step of fine-tuning the focus, where If R act <R tar , you need to fine-tune the focus: Repeat the above steps until R act >R tar , end, otherwise, directly record the final focal length end.

10. A cigarette pack texture recognition system based on binocular camera and convolutional neural network, characterized in that: include: A memory and a processor, wherein the memory includes a cigarette pack texture recognition program based on a binocular camera and a convolutional neural network, and the cigarette pack texture recognition program based on a binocular camera and a convolutional neural network, when executed by the processor, implements the cigarette pack texture recognition method according to claims 1 to 9.

Images