Belt foreign matter visual identification system based on image processing

Abstract

The invention discloses a belt foreign matter visual identification system based on image processing, and relates to the field of belt foreign matter visual identification. The system comprises an image acquisition module, a multi-stage image preprocessing module, a fusion type feature extraction module, a self-adaptive foreign matter recognition module and a configurable control output module. The image acquisition module performs continuous image acquisition on the surface of the running belt within a set time interval through an industrial camera to generate an original image sequence; the multi-stage image preprocessing module is connected with the image acquisition module, and the fusion type feature extraction module is connected with the image preprocessing module; the self-adaptive foreign matter recognition module is connected with the fusion type feature extraction module, and the configurable control output module is connected with the self-adaptive foreign matter recognition module and used for marking a connected domain meeting a dynamic threshold condition as a foreign matter area and outputting corresponding foreign matter position information to an alarm device or a control system according to a foreign matter category.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of belt foreign matter visual identification, and particularly relates to a belt foreign matter visual identification system based on image processing. BACKGROUND

[0002] In the current industrial production process, the conveyor belt as one of the core equipment of material transportation, its running stability is directly related to the safety and efficiency of the whole production line. The system is widely used in the automatic monitoring of industrial conveyor belt system, especially in the industries such as coal mine, steel, cement, port and other continuous material transportation scenes, for real-time detection of whether there are foreign matters on the running belt that may affect the safety of the equipment or the quality of the product.

[0003] However, in the actual operation process, due to human operation error, environmental factors or equipment failure, etc., non-expected foreign matters (such as stones, metal fragments, tools, packaging bags, etc.) often fall into the belt surface. If these foreign matters are not discovered and removed in time, the following serious consequences may occur: the foreign matters are stuck in the roller, the supporting roller or the crusher inlet, which is easy to cause equipment wear and even shutdown maintenance; small foreign matters may also cause mechanical structure damage due to impact under high-speed running state. SUMMARY

[0004] The purpose of the present application is to solve the above problems, and a belt foreign matter visual identification system based on image processing is provided, which is realized by multi-module fusion of multi-stage image preprocessing module, setting fusion feature extraction module, adaptive foreign matter identification module and the like, so as to improve the accuracy of foreign matter identification and realize accurate segmentation and verification of foreign matter area.

[0005] The present application specifically adopts the following technical solutions:

[0006] A belt foreign matter visual identification system based on image processing, comprising an image acquisition module, a multi-stage image preprocessing module, a fusion feature extraction module, an adaptive foreign matter identification module and a configurable control output module.

[0007] The image acquisition module continuously acquires images of the running belt surface within a set time interval through an industrial camera to generate a sequence of original images.

[0008] The multi-stage image preprocessing module is connected to the image acquisition module and is used to sequentially perform dynamic background modeling, adaptive illumination compensation and local contrast enhancement operations to obtain an enhanced target image. The dynamic background modeling of the multi-stage image preprocessing module adopts a sliding window update mechanism, which specifically includes: setting a local area with a window size of N x N; performing pixel-level weighted averaging on the current frame image and the previous M frame images to update the background image B(x, y): B(x, y) = a I t (x, y) + (1-a) Bprev (x, y) where I t (x,y) represents the pixel value of the current frame image at coordinate (x, y), B prev (x,y) represents the corresponding pixel value of the background image at the previous moment, a is the update weight coefficient, the value range is 0

[0009] The image after removing the background is subjected to histogram equalization processing;

[0010] The image contrast is adjusted by using the following formula:

[0011]

[0012] where I min and I max are the minimum and maximum pixel values in the local region of the image, and I min I max is adjusted adaptively according to the image noise level;

[0013] The multi-stage image preprocessing module includes an image fusion unit for pixel-level weighted fusion of visible light images and infrared images to generate a fused image as input for subsequent processing. The image fusion unit uses the following weighted fusion formula:

[0014] I f (x,y) = w1·I v (x,y) + w2·I i (x,y)

[0015] where I v (x,y) is the pixel value of the visible light image, I i (x,y) is the pixel value of the infrared image, w1+w2=1, and w1,w2∈[0,1], the weight value is dynamically adjusted according to the ambient light intensity.

[0016] The fused feature extraction module is connected to the image preprocessing module.

[0017] The fused feature extraction module extracts various features that are helpful for identifying foreign objects from the image after multi-stage image preprocessing. This process includes but is not limited to edge features, local texture features, and gradient histogram information:

[0018] Edge features: Canny edge detection algorithm is used to capture edge information in the image. The Canny algorithm includes: (1) using a Gaussian filter to smooth the image to remove noise; (2) calculating the gradient strength and direction of each pixel point; (3) applying Non-Maximum Suppression to refine the edges; (4) using a double threshold method to determine the final edges;

[0019] Local texture feature: the texture feature of the region around each pixel point is calculated by local binary pattern (LBP), and the basic formula of LBP is as follows:

[0020]

[0021] wherein g c is the gray value of the center pixel, g p is the gray value of the pth sampling point on the circumference, and s(x) is a sign function.

[0022] Gradient histogram information: the direction gradient information of the object in the image is captured by using the histogram of oriented gradients (HOG), and the calculation of the HOG feature description involves the following steps:

[0023] (1) the image is divided into small spatial regions, which are called cell units;

[0024] (2) for each cell unit, the gradient direction histogram is calculated;

[0025] (3) the histogram is normalized to reduce the influence of light changes;

[0026] The calculation method of the shape factor F s :

[0027] wherein A is the area of the connected domain, and P is the perimeter thereof; if F s is less than a set threshold T s , the region is determined as a foreign object candidate region, and the spatial consistency of the adjacent region is verified by the DBSCAN clustering algorithm;

[0028] If the shape factor is less than the set threshold, the region is marked as a foreign object candidate region, and the spatial consistency verification: the candidate region is verified again by using the DBSCAN clustering algorithm, the isolated noise points are removed, and the real foreign object region is retained;

[0029] The connection steps of the adaptive foreign object recognition module connected with the fusion feature extraction module: (1) the fusion feature extraction module transmits the composite feature map formed to the adaptive foreign object recognition module, (2) the adaptive foreign object recognition module performs connected domain marking, shape factor calculation and the like based on the received feature map, and uses the DBSCAN clustering algorithm to accurately segment and verify the foreign object region;

[0030] The configurable control output module is connected with the adaptive foreign object recognition module, which is used for marking the connected domain meeting the dynamic threshold condition as a foreign object region, and outputting the corresponding foreign object position information to the alarm device or control system according to the foreign object category.

[0031] Preferably, an edge computing module is further included, which is deployed in the field control cabinet, and is used for receiving data of the image acquisition module and performing image preprocessing, feature extraction and foreign matter recognition operations, and supporting GPU accelerated inference; the edge computing module is built-in with a lightweight CNN model, which is used for assisting feature extraction and foreign matter classification.

[0032] Preferably, the image acquisition module adopts a double-camera structure, and visible light images and infrared images are acquired respectively.

[0033] Preferably, the image acquisition module is installed on a fixed support above the belt, and a lens axis of the industrial camera is perpendicular to a running direction of the belt, and a field of view angle covers the entire belt width.

[0034] The present application has the following intended effects:

[0035] The present application has the following intended effects:

[0036] The present application has the following intended effects:

[0037] By the adaptive foreign matter recognition module, the accurate segmentation and verification of the foreign matter area are realized. The module performs region segmentation on the feature map based on a connected domain marking algorithm, and calculates parameters such as the area, shape factor and gray mean value of each connected domain, for preliminary determination of the foreign matter candidate region. Subsequently, the system introduces a DBSCAN clustering algorithm to perform secondary spatial consistency verification on the candidate region, to eliminate isolated noise points and retain the real foreign matter area. This layered recognition and clustering verification mechanism not only improves the reliability of recognition, but also reduces the false positive rate and the missed detection rate, so that the system still maintains stable performance in the complex environment of an industrial site. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A belt foreign matter vision recognition system based on image processing DETAILED DESCRIPTION

[0039] The specific implementation of the present application will be further described below in combination with the drawings and specific embodiments:

[0040] A belt foreign matter vision recognition system based on image processing, comprising an image acquisition module, a multi-stage image preprocessing module, a fusion type feature extraction module, an adaptive foreign matter recognition module and a configurable control output module;

[0041] The image acquisition module performs continuous image acquisition on the surface of the running belt within a set time interval through an industrial camera, to generate a raw image sequence;

[0042] The multi-stage image preprocessing module is connected to the image acquisition module, and is used to sequentially perform dynamic background modeling, adaptive illumination compensation and local contrast enhancement operations, to obtain an enhanced target image. The dynamic background modeling of the multi-stage image preprocessing module adopts a sliding window updating mechanism, and specifically includes: setting a local area with a window size of N×N; performing pixel-level weighted averaging on the current frame image and the previous M frame images to update the background image B(x,y): B(x,y)=α·I t (x,y)+(1-α)·B prev (x,y)wherein, I t (x,y) represents the pixel value of the current frame image at coordinates (x,y), B prev (x,y) represents the corresponding pixel value of the previous moment background image, and α is an updating weight coefficient, with a value range of 0<α<1, and α is dynamically adjusted according to the image motion amplitude;

[0043] The background-removed image is subjected to histogram equalization processing;

[0044] The image contrast is adjusted by using the following formula:

[0045]

[0046] where I min and I max are the minimum and maximum pixel values in the local region of the image, respectively, and I min I max is adaptively adjusted according to the image noise level;

[0047] The multi-stage image preprocessing module includes an image fusion unit for pixel-level weighted fusion of the visible light image and the infrared image to generate a fused image as input for subsequent processing. The image fusion unit uses the following weighted fusion formula:

[0048] I f (x,y) = w1·I v (x,y) + w2·I i (x,y)

[0049] where I v (x,y) is the pixel value of the visible light image, I i (x,y) is the pixel value of the infrared image, w1+w2 = 1, and w1, w2 ∈ [0, 1], and the weight values are dynamically adjusted according to the ambient light intensity;

[0050] The fused feature extraction module is connected to the image preprocessing module.

[0051] The fused feature extraction module aims to extract various features from the image after multi-stage image preprocessing that are helpful for identifying foreign objects. This process includes but is not limited to edge features, local texture features, and gradient histogram information.

[0052] Edge features: The Canny edge detection algorithm is used to capture edge information in the image. The Canny algorithm includes: (1) smoothing the image using a Gaussian filter to remove noise. (2) calculating the gradient strength and direction of each pixel point. (3) applying Non-Maximum Suppression to refine the edges. (4) using a double-threshold method to determine the final edges.

[0053] Local texture features: The Local Binary Pattern (LBP) is used to calculate the texture features in the region around each pixel point. The basic formula of LBP is:

[0054]

[0055] where g c is the gray value of the center pixel, g p is the gray value of the p-th sampling point on the circumference, and s(x) is the sign function.

[0056] Histogram of Gradient information: Histogram of Oriented Gradients (HOG) is used to capture the orientation gradient information of objects in the image. The calculation of HOG feature description involves the following steps:

[0057] (1) The image is divided into small spatial regions (called cell units).

[0058] (2) For each cell unit, the gradient direction histogram is calculated.

[0059] (3) Normalize these histograms to reduce the effects of illumination changes.

[0060] Formula example:

[0061] For shape factor calculation, the following formula is used:

[0062]

[0063] The connection relationship step of the fusion feature extraction module connected to the image preprocessing module is as follows: (1) The image acquisition module generates a raw image sequence and passes it to the multi-stage image preprocessing module. (2) The multi-stage image preprocessing module performs dynamic background modeling, adaptive illumination compensation, and local contrast enhancement operations on the image to generate an enhanced target image. (3) The enhanced image is sent to the fusion feature extraction module, where edge feature extraction, local texture feature analysis, and gradient histogram information processing are performed, and finally a composite feature map is formed for subsequent modules.

[0064] The adaptive foreign object recognition module is connected to the fusion feature extraction module,

[0065] The shape factor F s is calculated as follows:

[0066] Where A is the area of the connected domain, and P is its perimeter; if F s is less than a set threshold T s , the region is determined as a foreign object candidate region, and the DBSCAN clustering algorithm is used to verify the spatial consistency of its adjacent regions.

[0067] The adaptive foreign object recognition module performs region segmentation on the feature map based on the connected domain labeling algorithm, and calculates the area, shape factor, and gray mean value of each connected domain to preliminarily determine the foreign object candidate region. The core calculation process includes:

[0068]

[0069] If the shape factor is less than a set threshold, the region is marked as a foreign object candidate region, and the spatial consistency verification: the DBSCAN clustering algorithm is used to perform secondary verification on the candidate region, to remove isolated noise points and retain the real foreign object region.

[0070] The adaptive foreign matter recognition module is connected to the fusion feature extraction module, and the connection step is as follows: (1) the fusion feature extraction module transmits the formed composite feature map to the adaptive foreign matter recognition module; and (2) the adaptive foreign matter recognition module performs connected component labeling and calculates shape factors and other parameters based on the received feature map, and uses a DBSCAN clustering algorithm to accurately segment and verify the foreign matter region.

[0071] The configurable control output module is connected to the adaptive foreign matter recognition module, and is configured to mark a connected component as a foreign matter region if the connected component meets a dynamic threshold condition, and output corresponding foreign matter position information to an alarm device or a control system according to a foreign matter category; if a shape factor of a connected component is less than a set threshold value and the connected component passes a spatial consistency verification of the DBSCAN, the connected component is marked as a foreign matter region, and a corresponding alarm or control system action is triggered. The “dynamic threshold condition” specifically refers to a series of standards that are set in advance, such as a shape factor threshold value, an area size limit, and the like, and the standards can be adjusted according to an actual application scenario.

[0072] In addition, the system further includes an edge computing module, which is deployed in a field control cabinet and is configured to receive data of the image acquisition module and perform image preprocessing, feature extraction, and foreign matter recognition operations, and support GPU-accelerated inference; the edge computing module is internally provided with a lightweight CNN model, which is configured to assist feature extraction and foreign matter classification.

[0073] The image acquisition module adopts a double-camera structure and acquires visible light images and infrared images, respectively.

[0074] The image acquisition module is installed on a fixed support above a belt, a lens axis of an industrial camera is perpendicular to a running direction of the belt, and a field of view angle covers the entire belt width.

[0075] Of course, the above description is not a limitation on the present application, and the present application is not limited to the above examples, and changes, modifications, additions, or replacements made by those skilled in the art within the essential scope of the present application should also be within the protection scope of the present application.

Claims

1. An image processing-based belt foreign object visual recognition system, characterized by, The image acquisition module, the multi-stage image preprocessing module, the fusion feature extraction module, the adaptive foreign object recognition module and the configurable control output module are included. The image acquisition module acquires continuous images of the surface of the running belt within a set time interval through an industrial camera to generate a raw image sequence. The multi-stage image preprocessing module is connected with the image acquisition module, and is used for sequentially performing dynamic background modeling, adaptive illumination compensation and local contrast enhancement operations to obtain an enhanced target image. The dynamic background modeling of the multi-stage image preprocessing module adopts a sliding window updating mechanism, and specifically includes: setting a local region with a window size of N×N; performing pixel-level weighted average on a current frame image and previous M frame images to update a background image B(x, y): B(x, y) = α·I t (x,y)+(1-α)·B prev (x,y) wherein, I t (x,y) represents a pixel value of the current frame image at the coordinate (x, y), B prev (x,y) represents a corresponding pixel value of the previous moment background image, and α is an updating weight coefficient, and the value range is 0<α<1, and α is dynamically adjusted according to the image motion amplitude. The histogram equalization processing is performed on the image after the background is removed. The image contrast is adjusted by using the following formula: where I min and I max are the minimum and maximum pixel values within the local region of the image, respectively, and I min I max is adaptively adjusted according to the image noise level; The multi-stage image preprocessing module includes an image fusion unit for performing pixel-level weighted fusion on the visible light image and the infrared image to generate a fused image as an input for subsequent processing. I f (x,y) = w1 · I v (x,y) + w2 · I i (x,y) wherein I v (x,y) is a visible light image pixel value, I i (x,y) is an infrared image pixel value, w1+w2=1, and w1,w2∈[0,1], the weight value is dynamically adjusted according to the ambient light intensity; The fusion feature extraction module is connected to the image preprocessing module. The fusion feature extraction module extracts multiple features from the image after the multi-stage image preprocessing, which are helpful for identifying foreign objects, including but not limited to edge features, local texture features and histogram of gradient information. Edge features: Canny edge detection algorithm is used to capture edge information in the image, which includes: (1) using a Gaussian filter to smooth the image to remove noise; (2) calculating the gradient strength and direction of each pixel point; (3) applying non-maximum suppression to refine the edges; (4) using a double threshold method to determine the final edges. Local texture features: local binary pattern (LBP) is used to calculate the texture features in the surrounding area of each pixel point, and the basic formula of LBP is: where g c is the gray value of the center pixel, g p is the gray value of the pth sampling point on the circumference, and s(x) is a sign function. Gradient histogram information: direction gradient histogram (HOG) is used to capture the direction gradient information of objects in the image, and the calculation of HOG feature description involves the following steps: (1) divide the image into small spatial regions, called cell units; (2) for each cell unit, calculate the gradient direction histogram; (3) normalize the histogram to reduce the influence of light changes. Shape factor F s is calculated as follows: Wherein, A is the area of the connected domain, P is its perimeter; if F s is less than a set threshold T s , the region is determined as a foreign object candidate region, and its adjacent region is verified for spatial consistency by DBSCAN clustering algorithm. The adaptive foreign object recognition module is connected to the fusion feature extraction module. If the shape factor is less than a set threshold, the region is marked as a foreign object candidate area, and spatial consistency verification is performed using the DBSCAN clustering algorithm to remove isolated noise points and retain the real foreign object area. The adaptive foreign object recognition module is connected to the fusion feature extraction module in the following steps: (1) the fusion feature extraction module transmits the composite feature map to the adaptive foreign object recognition module, (2) the adaptive foreign object recognition module calculates parameters such as connected domain labeling and shape factor based on the received feature map, and uses the DBSCAN clustering algorithm to accurately segment and verify the foreign object area.

2. The image processing-based belt foreign object visual recognition system of claim 1, wherein, The configurable control output module is connected to the adaptive foreign object recognition module, which is used to mark the connected domain that meets the dynamic threshold condition as a foreign object area, and outputs the corresponding foreign object position information to the alarm device or control system according to the foreign object category. An edge computing module is also included, which is deployed in the on-site control cabinet and is used to receive data from the image acquisition module and perform image preprocessing, feature extraction and foreign object recognition operations, and supports GPU accelerated inference. The edge computing module has a built-in lightweight CNN model to assist feature extraction and foreign object classification.

3. The image processing based belt foreign object visual recognition system of claim 1, wherein, The image acquisition module adopts a double camera structure and respectively acquires visible light images and infrared images.

4. The image processing based belt foreign object visual recognition system of claim 1, wherein, The image acquisition module is installed on a fixed support above the belt, the lens axis of the industrial camera is perpendicular to the running direction of the belt, and the field of view angle covers the entire belt width.

Images