Method and system for monitoring working condition of cut tobacco production line based on digital twinning and machine vision

Abstract

The invention relates to the technical field of tobacco shred processing automatic detection, in particular to a tobacco shred processing production line working condition monitoring method and system based on digital twinning and machine vision, and the method comprises the steps: determining key features of tobacco shred processing and a production workshop, and constructing a tobacco shred processing digital twinning model based on the key features; video images of the tobacco shred processing production line for working condition detection are collected in real time through visual equipment and recognized; and driving virtual and real mapping of the acquired video image data and the tobacco shred processing digital twinning model according to a preset tobacco shred processing key event, so as to perform real-time control analysis on the working condition of the tobacco shred processing production line based on the tobacco shred processing digital twinning model. The physical world and the digital world of the tobacco shred processing production line are mapped by using the digital twinborn visual identification technology, so that the working condition of the tobacco shred processing production line is presented in real time in the digital world, the abnormal state of the working condition is conveniently speculated and rehearsed, the online real-time monitoring of the tobacco shred processing working condition is realized, and the working efficiency is improved. And the production efficiency and the automation degree of tobacco shred making are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic detection of cut tobacco production, and in particular to a method and system for monitoring the working conditions of a cut tobacco production line based on digital twin and machine vision. Background Art

[0002] Tobacco is formed after complex processes such as initial tobacco baking, moisturizing, leaf stem separation, re-baking, slicing, flavoring, shredding, drying and mixing. In order to reduce labor costs and realize the application of "Internet +" in the tobacco industry, the tobacco industry uses artificial intelligence means such as machine vision to build cigarette factories with intelligent manufacturing capabilities, so that the cigarette industry can develop towards digitalization, networking and intelligence. In the process of intelligent manufacturing of tobacco, it is necessary to monitor the working conditions in real time to deal with some abnormal situations. At present, the detection of tobacco production conditions usually relies on manual work, which has high detection costs and low efficiency, and cannot detect abnormalities in a timely and effective manner. Summary of the invention

[0003] To this end, the present invention provides a method and system for monitoring the operating conditions of a tobacco production line based on digital twin and machine vision, which applies digital twin technology and machine vision technology to the operating condition detection in the tobacco production process, solving the problems of low efficiency and other problems existing in the existing manual detection.

[0004] According to the design scheme provided by the present invention, on the one hand, a method for monitoring the working condition of a tobacco production line based on digital twin and machine vision is provided, comprising:

[0005] Determine the key features of tobacco production and production workshops, and build a digital twin model of tobacco production based on the key features;

[0006] Use visual equipment to collect and identify video images of the tobacco production line in real time for working condition detection; and drive the virtual-real mapping of the collected video image data and the tobacco production digital twin model based on the preset tobacco production key events, so as to conduct real-time control and analysis of the tobacco production line working conditions based on the tobacco production digital twin model;

[0007] Among them, in the working condition detection, if the video image is identified as the first inspection image before the silk production line is loaded, the pre-trained foreign body detection model is used to perform foreign body detection on the first inspection image before the silk production line is loaded, so as to determine whether there are foreign bodies on the silk production line; if the video image is identified as the material image of the feeding bin conveyor belt, the image segmentation algorithm is used to detect the material on the feeding bin conveyor belt to determine whether the feeding bin conveyor belt is empty; if the video image is identified as the material image at the storage cabinet outlet, the visual recognition algorithm is used to detect the material flow and blockage status of the storage cabinet outlet to output the material flow at the storage cabinet outlet and determine whether there is a blockage at the storage cabinet outlet; if the video image is identified as the material image of the cloth car belt line in front of the mixing cabinet, the visual recognition algorithm is used to detect the material on the cloth car belt to determine whether the cloth car is empty.

[0008] As a method for monitoring the working condition of a tobacco production line based on digital twin and machine vision of the present invention, a tobacco production digital twin model is further constructed based on key features, including:

[0009] A three-dimensional model of the tobacco production line is established using digital modeling software, and a digital twin model of the tobacco production line is constructed by rendering and building scenes for the three-dimensional model.

[0010] As a working condition monitoring method of a tobacco production line based on digital twin and machine vision of the present invention, further, a video image of the tobacco production line for working condition detection is collected in real time by using a visual device, including:

[0011] Setting measurement and control points for the silk production line, and deploying visual equipment at each measurement and control point of the silk production line, the measurement and control points of the silk production line include a first measurement and control point for collecting the first inspection image before loading, a second measurement and control point for collecting the image of the material on the conveyor belt of the feeding bin, a third measurement and control point for collecting the image of the material at the exit of the storage cabinet, and a fourth measurement and control point for collecting the image of the material on the belt line of the mixing cabinet cloth car;

[0012] The visual equipment at each measuring and controlling point of the tobacco production line is used to collect video images of each measuring and controlling point of the tobacco production line in real time.

[0013] As a working condition monitoring method of a tobacco production line based on digital twin and machine vision of the present invention, a pre-trained foreign body detection model is further used to perform foreign body detection on the first inspection image before loading of the tobacco production line, including:

[0014] Preprocessing the first inspection image before feeding the silk production line, wherein the preprocessing includes image denoising and image enhancement;

[0015] The pre-processed first inspection image is input into the pre-trained foreign body detection model, and the foreign body detection model is used to determine whether there are foreign bodies in the first inspection image, wherein the foreign bodies are materials not on the silk production line;

[0016] Among them, the foreign object detection model is built based on the ResNet-50 network and is trained using image sample data with foreign object labels.

[0017] As a working condition monitoring method of a tobacco production line based on digital twin and machine vision of the present invention, further, an image segmentation algorithm is used to detect materials on a conveyor belt of a feeding bin, comprising:

[0018] Preprocessing the material image of the feeding bin conveyor belt, wherein the preprocessing includes image denoising;

[0019] Perform image segmentation on the pre-processed material image of the feeding bin conveyor belt to obtain the conveyor belt area image;

[0020] Extract the color features of the conveyor belt area image, compare it with the color feature template of the unloaded conveyor belt image obtained in advance, and obtain the color feature comparison result; reconstruct the conveyor belt area image to obtain the three-dimensional reconstructed surface of the conveyor belt, so as to obtain the flatness and surface height of the conveyor belt using the three-dimensional reconstructed surface;

[0021] The color feature comparison results, conveyor belt flatness and surface height are combined to determine whether the feeding bin conveyor belt is empty.

[0022] As a working condition monitoring method of a tobacco production line based on digital twin and machine vision of the present invention, further, image segmentation is performed on the material image of the feeding bin transmission belt, comprising:

[0023] The pixels of the material image are clustered using the fuzzy C-means clustering algorithm to obtain the membership of each pixel in the image to all class centers, and the class of the pixel is determined based on the membership;

[0024] In the clustering process, the local edge information is integrated into the image pixel clustering, and the fuzzy factor is set according to the local variation coefficient and the neighborhood grayscale similarity, and the fuzzy factor is used to measure the influence of the neighborhood pixel on the central pixel.

[0025] As a working condition monitoring method of a tobacco production line based on digital twin and machine vision of the present invention, further, a visual recognition algorithm is used to detect the material flow rate and the blocking state of the storage cabinet outlet, comprising:

[0026] The volume of the material in the material image at the storage cabinet outlet is obtained by using a pre-trained volume measurement model, so as to obtain the weight of the material at the storage cabinet outlet based on the material volume and in combination with the material density. The volume measurement model obtains the material size by using the image depth and calculates the material volume according to the material size.

[0027] A three-dimensional reconstruction algorithm is used to reconstruct the upper surface of the material at the cabinet outlet to obtain a three-dimensional curved surface of the material at the cabinet outlet; the height of the upper surface of the material is obtained based on the three-dimensional curved surface of the material, and when the height of the upper surface of the material is greater than a preset threshold, it is determined that the material at the cabinet outlet is blocked; and / or, the length of the cabinet outlet belt line is obtained through image segmentation, and when the belt line length is less than a preset length, it is determined that the cabinet outlet is blocked.

[0028] As a working condition monitoring method of a tobacco production line based on digital twin and machine vision of the present invention, further, a three-dimensional reconstruction algorithm is used to reconstruct the upper surface of the material at the outlet of the storage cabinet, and it also includes:

[0029] The image is smoothed using an adaptive bilateral depth filtering algorithm, and the missing values ​​of the image are interpolated and sampled using bilinear interpolation to obtain an adaptive bilateral filtering output;

[0030] Based on the deep convolutional network, the super-resolution reconstructed image data of the material at the cabinet outlet is obtained, and the super-resolution reconstructed image data and the adaptive bilateral filtering output are weighted averaged to obtain the three-dimensional surface of the material at the cabinet outlet.

[0031] As a working condition monitoring method of a tobacco production line based on digital twin and machine vision of the present invention, further, a visual recognition algorithm is used to detect materials on a cloth conveyor belt, comprising:

[0032] The upper surface of the material on the cloth conveyor belt line is obtained based on the material image of the cloth conveyor belt line in front of the mixing cabinet and using a three-dimensional reconstruction algorithm; the height of the upper surface of the material on the cloth conveyor belt line is compared with a preset threshold value, and if the height is greater than the preset threshold value, it is determined that there is a material blockage abnormality on the cloth conveyor belt line;

[0033] And / or, the length of the fabric car belt line is obtained through image segmentation, and if the length of the fabric car belt line is less than a preset length, it is determined that there is a material blockage abnormality in the fabric car belt line.

[0034] On the other hand, the present invention also provides a tobacco production line operating condition monitoring system based on digital twin and machine vision, comprising: a model building module and an operating condition detection module, wherein:

[0035] The model building module is used to determine the key characteristics of tobacco cutting and production workshops, and build a digital twin model of tobacco cutting based on the key characteristics;

[0036] The working condition detection module is used to use visual equipment to collect and identify video images of the tobacco production line for working condition detection in real time; and drive the virtual-real mapping of the collected video image data and the tobacco production digital twin model based on the preset tobacco production key events, so as to conduct real-time control and analysis of the working condition of the tobacco production line based on the tobacco production digital twin model;

[0037] Among them, in the working condition detection, if the video image is identified as the first inspection image before the silk production line is loaded, the pre-trained foreign body detection model is used to perform foreign body detection on the first inspection image before the silk production line is loaded, so as to determine whether there are foreign bodies on the silk production line; if the video image is identified as the material image of the feeding bin conveyor belt, the image segmentation algorithm is used to detect the material on the feeding bin conveyor belt to determine whether the feeding bin conveyor belt is empty; if the video image is identified as the material image at the storage cabinet outlet, the visual recognition algorithm is used to detect the material flow and blockage status of the storage cabinet outlet to output the material flow at the storage cabinet outlet and determine whether there is a blockage at the storage cabinet outlet; if the video image is identified as the material image of the cloth car belt line in front of the mixing cabinet, the visual recognition algorithm is used to detect the material on the cloth car belt to determine whether the cloth car is empty.

[0038] Beneficial effects of the present invention:

[0039] 1. The present invention uses digital twin visual recognition technology to map the physical world of the tobacco production line with the digital world, so as to present the working conditions of the tobacco production line in real time in the digital world, facilitate the speculation and preview of abnormal working conditions, realize online real-time monitoring of the tobacco production conditions, and improve the production efficiency and automation level of tobacco production.

[0040] 2. In the working condition detection of the present invention, the first inspection before loading is carried out before the production line starts the daily production task to prevent the tools and materials used for maintenance from being left in the production line, that is, gloves, rags, tools such as screwdrivers, wrenches, pliers, pens and other non-tobacco production line equipment, and abnormalities are detected in time before entering the shredder to avoid serious production accidents and losses caused by damage to the shredder; visual recognition is used above the feeding bin to detect in real time whether the material in the feeding bin is empty, and an alarm is given if it is empty to ensure the normal operation of the automatic batch change system; visual recognition is used at the exit of the storage cabinet to detect whether the material in the storage cabinet is empty. At the same time, visual recognition is used at the outlet to detect material blockage and output a material blockage warning to effectively avoid the occurrence of material blockage, making the production line feeding more balanced and controllable, and effectively improving the quality level. Above the fabric car in front of the mixing cabinet, visual recognition is used to detect in real time whether the material on the fabric car belt line is empty. If it is empty, an alarm will be given to ensure the normal operation of the automatic batch changing system, stabilize the quality of tobacco making, and reduce production costs. It has good application prospects in the field of tobacco making condition detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the working condition monitoring process of the tobacco production line based on digital twin and machine vision in the embodiment;

[0042] Figure 2This is a schematic diagram of the foreign body detection algorithm process in the embodiment;

[0043] Figure 3 This is a schematic diagram of the network structure of the foreign body detection model in the embodiment;

[0044] Figure 4 It is a schematic diagram of the three-dimensional reconstructed surface of the conveyor belt with a small amount of material in the feeding bin and the material surface height curve in the embodiment;

[0045] Figure 5 This is a schematic diagram of the feeding bin conveyor belt emptying identification algorithm process in the embodiment;

[0046] Figure 6 It is a schematic diagram of the flow measurement algorithm principle at the outlet of the storage tank in the embodiment;

[0047] Figure 7 It is a schematic diagram of the three-dimensional reconstructed surface of the conveyor belt material at the storage cabinet outlet in the embodiment;

[0048] Figure 8 It is a schematic diagram of the principle of the conveyor belt material blockage detection algorithm at the storage cabinet outlet in the embodiment;

[0049] Fig. 9 This is a schematic diagram of the network structure for super-resolution reconstruction of a depth map in an embodiment;

[0050] Fig.10 Schematic diagram of the principle of the fabric delivery vehicle empty detection algorithm in the embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0052] For intelligent monitoring of tobacco production conditions, see Figure 1 As shown, a method for monitoring the working condition of a tobacco production line based on digital twin and machine vision is provided, comprising:

[0053] S101. Determine the key characteristics of tobacco processing and production workshops, and build a digital twin model of tobacco processing based on the key characteristics.

[0054] Specifically, digital modeling software can be used to build a three-dimensional model of the tobacco production line. By rendering the three-dimensional model and building scenes, the physical space of the tobacco production line and the digital space can be mapped to construct a digital twin model of the tobacco production line.

[0055] S102, using visual equipment to collect and identify video images of the tobacco production line for working condition detection in real time; and based on preset tobacco production key events, driving the virtual-real mapping of the collected video image data and the tobacco production digital twin model, so as to conduct real-time control and analysis of the working conditions of the tobacco production line based on the tobacco production digital twin model;

[0056] Among them, in the working condition detection, if the video image is identified as the first inspection image before the silk production line is loaded, the pre-trained foreign body detection model is used to perform foreign body detection on the first inspection image before the silk production line is loaded, so as to determine whether there are foreign bodies on the silk production line; if the video image is identified as the material image of the feeding bin conveyor belt, the image segmentation algorithm is used to detect the material on the feeding bin conveyor belt to determine whether the feeding bin conveyor belt is empty; if the video image is identified as the material image at the storage cabinet outlet, the visual recognition algorithm is used to detect the material flow and blockage status of the storage cabinet outlet to output the material flow at the storage cabinet outlet and determine whether there is a blockage at the storage cabinet outlet; if the video image is identified as the material image of the cloth car belt line in front of the mixing cabinet, the visual recognition algorithm is used to detect the material on the cloth car belt to determine whether the cloth car is empty.

[0057] Specifically, if the identified video image is the first inspection image before loading the tobacco production line, the video image of the tobacco production line for working condition detection is collected in real time by visual equipment, which can be designed to include:

[0058] Setting measurement and control points for the silk production line, and deploying visual equipment at each measurement and control point of the silk production line, the measurement and control points of the silk production line include a first measurement and control point for collecting the first inspection image before loading, a second measurement and control point for collecting the image of the material on the conveyor belt of the feeding bin, a third measurement and control point for collecting the image of the material at the exit of the storage cabinet, and a fourth measurement and control point for collecting the image of the material on the belt line of the mixing cabinet cloth car;

[0059] The visual equipment at each measuring and controlling point of the tobacco production line is used to collect video images of each measuring and controlling point of the tobacco production line in real time.

[0060] When workers are repairing and maintaining the silk thread production line, they often leave foreign objects due to negligence, such as: corrugated paper pieces, transparent plastic bag fragments, gloves, rags and other debris, screwdrivers, wrenches, pliers, pens and other maintenance tools. In order to ensure the normal operation and generation of the production line, it is necessary to identify the above foreign objects and give an alarm signal. Visual equipment such as industrial cameras can be installed above or on the side of the production line as needed. Since the production line will move forward after it is turned on, there is no need to include the entire production line in the picture. Only a section of the production line needs to be intercepted for identification, because the foreign objects will be pushed into the visual identification area by the advancing production line.

[0061] The feeding bin is the material buffer of the production line. Therefore, materials are constantly input and output during operation, resulting in the situation where the materials in the bin are sometimes more and sometimes less. Visual equipment such as cameras can be installed directly above the feeding bin to take a panoramic view of the bin and identify empty materials through algorithms and issue alarms and pictures.

[0062] The size of the storage cabinet is generally 15 meters long, 3 meters wide, 1.8-2 meters deep, and the material height does not exceed 1.5 meters. At the exit of the storage cabinet, the top cover of the cabinet is a movable material distribution car. Visual equipment such as TOF cameras with video functions can be installed on the upper side of the cabinet to provide real-time feedback on the material height at the exit. There is a conveyor belt at the bottom of the cabinet, and the speed is generally in the order of millimeters per second. The data sampling density of the material height curve at the exit of the storage cabinet can be once every 5 seconds. The flow rate of the discharge is predicted according to the material height, and the speed of the bottom belt of the storage cabinet is adjusted according to the flow rate to maintain a balanced discharge flow. After the cabinet material is discharged from the cabinet, it enters the belt line to detect the blockage of the belt line. The material level height can be used for detection, and a blockage warning will be issued when it exceeds a certain height.

[0063] In the fabric car empty detection, cameras and other visual equipment can be installed at the mixing bin. Samples are taken every 5 seconds. The status of the material on the fabric car belt is used to determine whether there is an empty situation and output an alarm signal in the empty state.

[0064] Among them, the pre-trained foreign body detection model is used to detect foreign bodies in the first inspection image before loading the silk production line, which can be designed to include:

[0065] Preprocessing the first inspection image before feeding the silk production line, wherein the preprocessing includes image denoising and image enhancement;

[0066] The pre-processed first inspection image is input into the pre-trained foreign body detection model, and the foreign body detection model is used to determine whether there are foreign bodies in the first inspection image, wherein the foreign bodies are materials not on the silk production line;

[0067] Among them, the foreign object detection model is built based on the ResNet-50 network and is trained using image sample data with foreign object labels.

[0068] In the field of anomaly detection, thanks to the rapid development of digital image processing and computer vision research, the use of basic image data for anomaly detection has become the first choice. Commonly used detection algorithms include image detection technology based on statistical information, morphology-based detection algorithms, and filtering-based detection algorithms. These detection algorithms can effectively detect certain specific anomalies. However, they cannot meet the requirements of abnormal characteristics or state diversity. In the embodiment of this case, computer vision technology based on deep learning is used to solve the problem of foreign body detection, and the feature extraction ability of deep convolutional neural networks and the implicit modeling ability of nonlinear mapping relationships are fully utilized to improve the accuracy of foreign body detection. At the same time, since the data operation operators contained in the convolutional neural network have high parallelism, the GPU is used to accelerate the learning, training and reasoning detection of the model, thereby realizing high-precision real-time detection of foreign matter contained in the material. Figure 2 As shown in the figure, the foreign body detection model includes four stages: foreign body image acquisition, detection model training, model migration, and real-time operation. The goal of the foreign body image acquisition stage is to use the industrial camera installed above the production line to obtain the raw material image on the production line in real time. At the same time, the collected original image is preprocessed to suppress the noise in the image and enhance the image visual effect, thereby improving the accuracy of subsequent foreign body detection. The foreign body detection model training stage uses the image dataset acquired by the industrial camera and the artificial foreign body marking information of the dataset to train a foreign body detection network based on a deep convolutional network using a GPU workstation, such as Figure 3 As shown, Backbone adopts the ResNet-50 structure.

[0069] It should be noted that ResNe-50 is a residual network, which is a deep network formed by stacking a series of sub-networks. A direct channel is added to the sub-network to allow the original input information to be directly transmitted to the subsequent layers. If the input is set to X and a certain parameterized network layer is set to H, then the output of this layer with X as input will be H(X). General CNN networks such as Alexnet / VGG will directly learn the expression of the parameter function H through training, and thus directly learn X->H(X). ResNe-50 is committed to using multiple parameterized network layers to learn the difference between input and output, that is, H(X)-X, that is, learning X->(H(X)-X)+X. In addition, the core structure of the Dilated Encoder module is to connect 4 layers of dilated convolutions with different expansion coefficients in series, and the expansion coefficients of each layer are 2, 4, 6, and 8 respectively. There is also a residual connection in it, the purpose of which is to combine two different receptive fields (the input receptive field and the processed receptive field).

[0070] The model migration phase aims to convert the foreign body detection model obtained by GPU workstation training into a Caffe model that can run on industrial control equipment. In the real-time operation phase, when the production line is running, the industrial camera transmits the raw material images taken to the industrial control equipment, which is used as the input of the foreign body detection Caffe model. After the model is processed, the relevant detection results are output. If the results contain foreign bodies, the industrial control equipment outputs an alarm signal and the type of foreign body.

[0071] Among them, if the video image is identified as an image of materials on the conveyor belt of the feeding bin, the image segmentation algorithm is used to detect the materials on the conveyor belt of the feeding bin, which can be designed to include:

[0072] Preprocessing the material image of the feeding bin conveyor belt, wherein the preprocessing includes image denoising;

[0073] Perform image segmentation on the pre-processed material image of the feeding bin conveyor belt to obtain the conveyor belt area image;

[0074] Extract the color features of the conveyor belt area image, compare it with the color feature template of the unloaded conveyor belt image obtained in advance, and obtain the color feature comparison result; reconstruct the conveyor belt area image to obtain the three-dimensional reconstructed surface of the conveyor belt, so as to obtain the flatness and surface height of the conveyor belt using the three-dimensional reconstructed surface;

[0075] The color feature comparison results, conveyor belt flatness and surface height are combined to determine whether the feeding bin conveyor belt is empty.

[0076] In the material buffer, materials are carried and transported by conveyor belts. The conveyor belt and the materials are different in color, and there is a significant difference. Therefore, this color difference can be used to determine whether the conveyor belt is empty. After the color features are obtained through a deep learning algorithm, it can be used as one of the information for evaluating the empty state. The surface of an empty conveyor belt is very flat, while the surface of a conveyor belt loaded with materials is uneven. In this embodiment of the case, a TOF camera can be used to obtain the depth information of the conveyor belt surface, and then its surface is reconstructed to obtain its three-dimensional surface. On this basis, the two metrics of flatness and height are used to determine whether the surface is empty. Figure 4 The figure shows the 3D reconstructed surface of a conveyor belt carrying a small amount of material, and the figure on the right shows the height curve of a certain section of the surface. It is easy to see that by scanning the height curves of each section and supplementing it with height information, it is possible to determine whether the conveyor belt is in an empty state. In order to improve the accuracy of conveyor belt empty recognition, the results of the above two recognition methods are further fused using the information fusion method. The industrial control equipment generates the final empty recognition result and outputs the empty confirmation information and empty image. The overall recognition algorithm is as follows: Figure 5As shown, the image captured by the camera is preprocessed to suppress noise in the image and improve the visual quality of the image. The conveyor belt area is segmented from the image using an image segmentation algorithm to determine the area of ​​interest for color feature comparison analysis as well as flatness and height analysis.

[0077] Specifically, in the image denoising method, the key to denoising is how to make full use of the prior laws of the image itself and what data estimation method to use to estimate the image. In order to use the low-rank property of the image for denoising, the observed image can be first divided into many small blocks of the same size. i , according to the similarity judgment criteria, search for similar image blocks, and vectorize these similar image blocks to form a similar block matrix P i On this basis, the low rank and minimum variance estimation theory of similar block matrices are used to estimate P i The noise of the image block is suppressed by estimating the image block, and the denoised image is reconstructed by the estimated value of the image block. In order to solve the problem of a small amount of residual noise, the denoised image is processed by the back projection method to further improve the denoising effect of the image. Specifically, considering the calculation speed, the Euclidean distance based on the grayscale feature is used to measure the similarity between image blocks. For any two image blocks p i and pj, their similarity can be expressed as

[0078]

[0079] Sim(p i ,p j ) is smaller, it means that the image block p i and p j For any given image block p r , we can use the block matching method to use the above formula as the metric to find the K most similar image blocks, denoted as For the convenience of discussion, the image block is represented in the form of a column vector, then p r Its similar blocks can form a similar block matrix

[0080]

[0081] The denoising goal is to use the noisy similar block matrix P r Estimate Q as accurately as possible r .

[0082] The minimum variance estimation theory is a very effective low-rank matrix estimation method, so we use the minimum variance estimation to estimate Q. Its goal is to solve the matrix M that satisfies the minimization problem.

[0083]

[0084] The estimated value of Q is obtained by multiplying the two matrices P and M. Since the objective function in the above formula is strictly convex and differentiable, it can be obtained by making its first-order derivative equal to zero

[0085] M=(P T P) -1 P T Q.

[0086] Therefore, the minimum variance estimate of Q can be expressed as

[0087]

[0088] The goal of image segmentation is to divide an image into a set of non-intersecting sub-regions, and the same region has the same or similar characteristics, where the characteristics can be grayscale, shape, texture, etc. In the implementation of this case, the goal of image segmentation is to use the image segmentation algorithm to segment the conveyor belt from the denoised image for subsequent empty detection. Accurate image segmentation can provide a reliable basis for empty detection. Due to the diversity and complexity of images, image segmentation is susceptible to external interference such as noise and uneven intensity. How to accurately segment the required target object has always been a bottleneck and one of the hot issues in image segmentation. Fuzzy C-means clustering algorithm (FCM) is a widely used and effective image segmentation method. It has the advantages of being in line with human cognitive characteristics, simple description, easy to implement, and good segmentation effect. Compared with other segmentation algorithms, FCM is essentially a soft clustering algorithm, that is, there is no need to simply indicate which pixels belong to or do not belong to which class, but to indicate to what extent the pixel belongs to this class. The quantity representing the degree is the degree of membership. Due to the uncertainty of the initial judgment of the belonging class, FCM can retain as much information as possible from the initial image. Therefore, in the embodiment of this case, the FCM algorithm is used to segment the conveyor belt.

[0089] Specifically, an improved FCM is adopted, and the pixels of the material image are clustered using the improved fuzzy C-means clustering algorithm to obtain the membership of each pixel in the image to all class centers, and the class of the pixel is determined based on the membership; wherein, in the clustering process, the local edge information is integrated into the image pixel clustering, and the fuzzy factor is set based on the local variation coefficient and the neighborhood grayscale similarity, and the fuzzy factor is used to measure the influence of the neighborhood pixel on the central pixel. It integrates the local edge information into the FCM clustering process, which can make full use of the image neighborhood information and local information, and redesign the fuzzy factor by introducing the local variation coefficient and defining the neighborhood grayscale similarity. The improved FCM algorithm can more accurately measure the influence of the neighborhood pixel on the central pixel, and can obtain more accurate edges while achieving noise resistance. In addition, the improved FCM algorithm adds a local segmentation post-processing process. Since the image grayscale level is much larger than the number of clusters, the pixel grayscale values ​​of each class in the segmentation result are very different, resulting in weak edges or pixels severely affected by noise not being correctly classified. To address this problem, the post-processing process makes full use of the local information of the image and selects a local window at the edge for re-segmentation. Since there are few gray levels of pixels in the local window and the difference in pixel gray values ​​in each category is small, local segmentation can classify pixels more accurately, thereby improving segmentation accuracy.

[0090] If the identified video image is an image of materials at the storage cabinet outlet, the visual recognition algorithm is used to detect the material flow and blocking state at the storage cabinet outlet, which can be designed to include:

[0091] Using a pre-trained volume measurement model to obtain the volume of the material in the material image at the cabinet outlet, so as to obtain the weight of the material at the cabinet outlet based on the material volume and combined with the material density. The volume measurement model uses the image depth to obtain the material size and calculates the material volume based on the material size.

[0092] A three-dimensional reconstruction algorithm is used to reconstruct the upper surface of the material at the cabinet outlet to obtain a three-dimensional curved surface of the material at the cabinet outlet; the height of the upper surface of the material is obtained based on the three-dimensional curved surface of the material, and when the height of the upper surface of the material is greater than a preset threshold, it is determined that the material at the cabinet outlet is blocked; and / or, the length of the cabinet outlet belt line is obtained through image segmentation, and when the belt line length is less than a preset length, it is determined that the cabinet outlet is blocked.

[0093] In order to achieve high-precision measurement of raw material usage, a raw material usage measurement model based on deep learning algorithm can be used, that is, a volume measurement model, such as Figure 6 As shown, the specific construction process can be described as follows:

[0094] During the training phase, a measurement model is generated based on a small sample learning algorithm and a high-precision parameter algorithm with the help of existing raw material depth image data to obtain the mapping relationship between the depth image data and the raw material volume. During the inference measurement phase, the depth image acquired by the TOF camera is used as the input of the measurement model, and the model will output the raw material volume at the current moment, and then calculate the weight of the raw material in combination with the raw material density.

[0095] After the material enters the belt line from the storage cabinet, if the material flow is too large and unevenly distributed, material blockage may occur at the front end of the belt line. In order to avoid affecting production efficiency due to material blockage on the belt line, it is necessary to detect in real time whether the belt line is in a material blockage state, and output abnormal signal of material blockage and related images. In the embodiment of this case, a technical solution of visual recognition with deep learning is adopted. The basic idea is to use a TOF camera with a camera function to obtain the height information of the material on the belt line in real time, and use a three-dimensional reconstruction algorithm to obtain the upper surface of the material. Figure 7 The figure shows a schematic diagram of the three-dimensional surface of the material calculated by the reconstruction algorithm. On this basis, the material surface height is calculated and compared with the set threshold to determine whether it exceeds the warning height. If it is greater than the threshold, it means that a blockage has occurred at the front end of the belt line, and then an abnormal blockage alarm is output. In addition, due to the large color difference between the belt line and the material, the length of the belt line in the image can be used to indirectly determine whether a blockage has occurred. When the length of the belt line is shorter than a certain threshold, it means that a blockage has occurred, and the industrial control needs to output a relevant alarm. Specifically, the principle of the blockage judgment algorithm at the cabinet outlet is as follows: Figure 8 As shown, the belt length and the surface height of the material reconstructed image are combined to comprehensively judge the material blocking state.

[0096] For the segmentation of belt lines, the improved FCM algorithm used in empty detection can be used. The improved FCM segmentation algorithm integrates edge information into the FCM clustering process, so the algorithm is also effective for segmenting belt lines. In the TOF depth data reconstruction method, accurate depth information of the scene provides key information for determining whether a blockage occurs. However, limited by the low resolution of the TOF camera, the depth values ​​obtained in practical applications usually contain strong noise. At the same time, since the resolution of the depth map is much lower than that of the color image and the viewpoint is different, only a very sparse depth map can be obtained after the depth map is projected onto the color image, and even a situation where a large area lacks depth values, that is, a hole problem occurs. In order to fill the missing depth values ​​in the color image, many algorithms assume that adjacent pixels with similar colors have similar depth values, and pixels with large color differences have different depth values. However, this type of method cannot effectively handle low-resolution depth maps containing a large number of holes. In response to the above problems, the embodiment of this case uses a joint adaptive bilateral depth filtering algorithm to smooth the depth map data and complete the missing values. The idea of ​​the joint adaptive bilateral depth filtering algorithm is concise and has very high computational efficiency. With the help of ordinary GPUs, real-time data processing can be achieved. After the data is filtered, bilinear interpolation is used to interpolate the missing values ​​of the depth map, and the depth map is refined by combining the color image texture information, thus solving the problem of image contour blur caused by bilinear interpolation.

[0097] like Fig. 9As shown in the figure, in order to further improve the accuracy of material surface reconstruction and reduce the misjudgment rate of material blocking, the material surface is reconstructed by a depth map super-resolution reconstruction algorithm based on a deep convolutional network. The depth map super-resolution reconstruction network has a 10-layer structure. The first layer is the input layer, which inputs a low-resolution depth map; the 2nd to 8th layers are convolutional layers, and the low-resolution depth map is passed through the convolutional layer to obtain a series of feature maps. These feature maps are nonlinearly mapped into high-resolution image blocks; the 9th layer is the deconvolution layer, which is upsampled through the deconvolution operation to further improve the resolution of the image block; and finally the reconstructed high-resolution depth map is output by the 10th layer. The convolution layer is the core component of the convolutional neural network, which has the characteristics of local connection and weight sharing. The operation performed by the convolution layer is: one or more feature maps of the previous layer are used as input, and convolution operations are performed with one or more convolution kernels to generate one or more outputs. The convolution operation can extract features well. It obtains the best parameters for different tasks by back-propagating errors, and learns the best convolution kernel for this task. The final reconstruction process is achieved through a deconvolution layer, which is equivalent to an upsampling operation. When the step size is 1, the convolution and deconvolution operations are essentially the same. The sampling factor is adjusted by adjusting the step size, and a relatively large convolution kernel is used to improve the reconstruction quality. For network training, the network continuously adjusts the network parameters by minimizing the cost between the reconstructed result and the true disparity map, using the mean square error as the cost function.

[0098] After the processed depth data is obtained by using the above-mentioned depth map reconstruction network, it is weighted averaged with the output result of the joint adaptive bilateral depth filter to obtain the final material surface data. Since the noise and missing values ​​in the original data have been well processed, the material surface data at this time has very high accuracy. Based on these surface data, the threshold processing algorithm is used to perform threshold analysis to determine whether a material blockage occurs. If a material blockage occurs, an alarm signal is issued to remind workers to deal with it in time.

[0099] Among them, if the recognized video image is the material image of the cloth car belt line in front of the mixing cabinet, the visual recognition algorithm is used to detect the material on the cloth car belt, which can be designed to include:

[0100] The upper surface of the material on the cloth conveyor belt line is obtained based on the material image of the cloth conveyor belt line in front of the mixing cabinet and using a three-dimensional reconstruction algorithm; the height of the upper surface of the material on the cloth conveyor belt line is compared with a preset threshold value, and if the height is greater than the preset threshold value, it is determined that there is a material blockage abnormality on the cloth conveyor belt line;

[0101] And / or, the length of the fabric car belt line is obtained through image segmentation, and if the length of the fabric car belt line is less than a preset length, it is determined that there is a material blockage abnormality in the fabric car belt line.

[0102] A similar technical solution to the feeding bin emptying identification can be used. The specific algorithm is as follows: Fig.10 As shown in the figure, a TOF camera with a video function is used to obtain the height information of the material on the belt line in real time, and the upper surface of the material is obtained using a three-dimensional reconstruction algorithm. On this basis, the material surface height is calculated and compared with the set threshold to determine whether it exceeds the warning height. If it is greater than the threshold, it means that a blockage has occurred at the front end of the belt line, and then an abnormal blockage alarm is output. In addition, due to the large color difference between the belt line and the material, the length of the belt line in the image can be used to indirectly determine whether a blockage has occurred. When the length of the belt line is shorter than a certain threshold, it means that a blockage has occurred, and the industrial control needs to output a relevant alarm.

[0103] Furthermore, based on the above method, an embodiment of the present invention also provides a tobacco production line operating condition monitoring system based on digital twin and machine vision, comprising: a model building module and an operating condition detection module, wherein:

[0104] The model building module is used to determine the key characteristics of tobacco cutting and production workshops, and build a digital twin model of tobacco cutting based on the key characteristics;

[0105] The working condition detection module is used to use visual equipment to collect and identify video images of the tobacco production line for working condition detection in real time; and drive the virtual-real mapping of the collected video image data and the tobacco production digital twin model based on the preset tobacco production key events, so as to conduct real-time control and analysis of the working condition of the tobacco production line based on the tobacco production digital twin model;

[0106] Among them, in the working condition detection, if the video image is identified as the first inspection image before the silk production line is loaded, the pre-trained foreign body detection model is used to perform foreign body detection on the first inspection image before the silk production line is loaded, so as to determine whether there are foreign bodies on the silk production line; if the video image is identified as the material image of the feeding bin conveyor belt, the image segmentation algorithm is used to detect the material on the feeding bin conveyor belt to determine whether the feeding bin conveyor belt is empty; if the video image is identified as the material image at the storage cabinet outlet, the visual recognition algorithm is used to detect the material flow and blockage status of the storage cabinet outlet to output the material flow at the storage cabinet outlet and determine whether there is a blockage at the storage cabinet outlet; if the video image is identified as the material image of the cloth car belt line in front of the mixing cabinet, the visual recognition algorithm is used to detect the material on the cloth car belt to determine whether the cloth car is empty.

[0107] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0108] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0109] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0110] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0111] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for monitoring the working condition of a tobacco production line based on digital twin and machine vision, characterized in that: Include: Determine the key features of tobacco production and production workshops, and build a digital twin model of tobacco production based on the key features; Use visual equipment to collect and identify video images of tobacco production lines in real time for working condition detection; And based on the preset key events of tobacco production, the collected video image data and the virtual-real mapping of the tobacco production digital twin model are driven to conduct real-time control and analysis of the tobacco production line working conditions based on the tobacco production digital twin model; Among them, in the working condition detection, if the video image is identified as the first inspection image before the silk production line is loaded, the pre-trained foreign body detection model is used to perform foreign body detection on the first inspection image before the silk production line is loaded, so as to determine whether there are foreign bodies on the silk production line; if the video image is identified as the material image of the feeding bin conveyor belt, the image segmentation algorithm is used to detect the material on the feeding bin conveyor belt to determine whether the feeding bin conveyor belt is empty; if the video image is identified as the material image at the storage cabinet outlet, the visual recognition algorithm is used to detect the material flow and blockage status of the storage cabinet outlet to output the material flow at the storage cabinet outlet and determine whether there is a blockage at the storage cabinet outlet; if the video image is identified as the material image of the cloth car belt line in front of the mixing cabinet, the visual recognition algorithm is used to detect the material on the cloth car belt to determine whether the cloth car is empty.

2. The method for monitoring the working condition of a tobacco production line based on digital twin and machine vision according to claim 1 is characterized in that: The digital twin model of tobacco production is constructed based on key features, including: A three-dimensional model of the tobacco production line is established using digital modeling software, and a digital twin model of the tobacco production line is constructed by rendering and building scenes for the three-dimensional model.

3. The method for monitoring the working condition of a tobacco production line based on digital twin and machine vision according to claim 1 is characterized in that: Use visual equipment to collect video images of tobacco production lines in real time for working condition detection, including: Setting measurement and control points for the silk production line, and deploying visual equipment at each measurement and control point of the silk production line, the measurement and control points of the silk production line include a first measurement and control point for collecting the first inspection image before loading, a second measurement and control point for collecting the image of the material on the conveyor belt of the feeding bin, a third measurement and control point for collecting the image of the material at the exit of the storage cabinet, and a fourth measurement and control point for collecting the image of the material on the belt line of the mixing cabinet cloth car; The visual equipment at each measuring and controlling point of the tobacco production line is used to collect video images of each measuring and controlling point of the tobacco production line in real time.

4. The method for monitoring the working condition of a tobacco production line based on digital twin and machine vision according to claim 1 is characterized in that: Use the pre-trained foreign body detection model to detect foreign bodies in the first inspection image before loading the silk production line, including: Preprocessing the first inspection image before feeding the silk production line, wherein the preprocessing includes image denoising and image enhancement; The pre-processed first inspection image is input into the pre-trained foreign body detection model, and the foreign body detection model is used to determine whether there are foreign bodies in the first inspection image, wherein the foreign bodies are materials not on the silk production line; Among them, the foreign object detection model is built based on the ResNet-50 network and is trained using image sample data with foreign object labels.

5. The method for monitoring the working condition of a tobacco production line based on digital twin and machine vision according to claim 1 is characterized in that: Use image segmentation algorithms to detect materials on the conveyor belt of the feeding bin, including: Preprocessing the material image of the feeding bin conveyor belt, wherein the preprocessing includes image denoising; Perform image segmentation on the pre-processed material image of the feeding bin conveyor belt to obtain the conveyor belt area image; Extract the color features of the conveyor belt area image, compare it with the color feature template of the unloaded conveyor belt image obtained in advance, and obtain the color feature comparison result; reconstruct the conveyor belt area image to obtain the three-dimensional reconstructed surface of the conveyor belt, so as to obtain the flatness and surface height of the conveyor belt using the three-dimensional reconstructed surface; The color feature comparison results, conveyor belt flatness and surface height are combined to determine whether the feeding bin conveyor belt is empty.

6. The method for monitoring the working condition of a tobacco production line based on digital twin and machine vision according to claim 5 is characterized in that: Image segmentation of the material image of the feeding bin conveyor belt, including: The pixels of the material image are clustered using the fuzzy C-means clustering algorithm to obtain the membership of each pixel in the image to all class centers, and the class of the pixel is determined based on the membership; In the clustering process, the local edge information is integrated into the image pixel clustering, and the fuzzy factor is set according to the local variation coefficient and the neighborhood grayscale similarity, and the fuzzy factor is used to measure the influence of the neighborhood pixel on the central pixel.

7. The method for monitoring the working condition of a tobacco production line based on digital twin and machine vision according to claim 1 is characterized in that: Use visual recognition algorithms to detect the material flow and blockage status of the storage tank outlet, including: Using a pre-trained volume measurement model to obtain the volume of the material in the material image at the cabinet outlet, so as to obtain the weight of the material at the cabinet outlet based on the material volume and combined with the material density. The volume measurement model uses the image depth to obtain the material size and calculates the material volume based on the material size. A three-dimensional reconstruction algorithm is used to reconstruct the upper surface of the material at the cabinet outlet to obtain a three-dimensional curved surface of the material at the cabinet outlet; the height of the upper surface of the material is obtained based on the three-dimensional curved surface of the material, and when the height of the upper surface of the material is greater than a preset threshold, it is determined that the material at the cabinet outlet is blocked; and / or, the length of the cabinet outlet belt line is obtained through image segmentation, and when the belt line length is less than a preset length, it is determined that the cabinet outlet is blocked.

8. The method for monitoring the working condition of a tobacco production line based on digital twin and machine vision according to claim 7 is characterized in that: The upper surface of the material at the storage tank outlet is reconstructed using a three-dimensional reconstruction algorithm, which also includes: The image is smoothed using an adaptive bilateral depth filtering algorithm, and the missing values ​​of the image are interpolated and sampled using bilinear interpolation to obtain an adaptive bilateral filtering output; Based on the deep convolutional network, the super-resolution reconstructed image data of the material at the cabinet outlet is obtained, and the super-resolution reconstructed image data and the adaptive bilateral filtering output are weighted averaged to obtain the three-dimensional surface of the material at the cabinet outlet.

9. The method for monitoring the working condition of a tobacco production line based on digital twin and machine vision according to claim 1, characterized in that: Use visual recognition algorithms to detect materials on the fabric conveyor belt, including: The upper surface of the material on the cloth conveyor belt line is obtained based on the material image of the cloth conveyor belt line in front of the mixing cabinet and using a three-dimensional reconstruction algorithm; the height of the upper surface of the material on the cloth conveyor belt line is compared with a preset threshold value, and if the height is greater than the preset threshold value, it is determined that there is a material blockage abnormality on the cloth conveyor belt line; And / or, the length of the fabric car belt line is obtained through image segmentation, and if the length of the fabric car belt line is less than a preset length, it is determined that there is a material blockage abnormality in the fabric car belt line.

10. A tobacco production line operating condition monitoring system based on digital twin and machine vision, characterized in that: Contains: model building module and working condition detection module, among which, The model building module is used to determine the key characteristics of tobacco cutting and production workshops, and build a digital twin model of tobacco cutting based on the key characteristics; The working condition detection module is used to use visual equipment to collect and identify video images of the tobacco production line for working condition detection in real time; and drive the virtual-real mapping of the collected video image data and the tobacco production digital twin model based on the preset tobacco production key events, so as to conduct real-time control and analysis of the working condition of the tobacco production line based on the tobacco production digital twin model; Among them, in the working condition detection, if the video image is identified as the first inspection image before the silk production line is loaded, the pre-trained foreign body detection model is used to perform foreign body detection on the first inspection image before the silk production line is loaded, so as to determine whether there are foreign bodies on the silk production line; if the video image is identified as the material image of the feeding bin conveyor belt, the image segmentation algorithm is used to detect the material on the feeding bin conveyor belt to determine whether the feeding bin conveyor belt is empty; if the video image is identified as the material image at the storage cabinet outlet, the visual recognition algorithm is used to detect the material flow and blockage status of the storage cabinet outlet to output the material flow at the storage cabinet outlet and determine whether there is a blockage at the storage cabinet outlet; if the video image is identified as the material image of the cloth car belt line in front of the mixing cabinet, the visual recognition algorithm is used to detect the material on the cloth car belt to determine whether the cloth car is empty.

Images