A multi-sensor fusion-based intelligent management method for a conveying belt

Abstract

The present application relates to a kind of based on multi-sensor fusion conveyor belt intelligent management method, belong to industrial equipment intelligent monitoring and diagnosis technical field.Therein, the method includes: the internal structure data of conveyor belt, surface three-dimensional profile data, surface 2D image data, carrier roller operating sound data and transmission chain vibration temperature data are collected;Data is preprocessed based on histogram equalization, time-frequency analysis and other algorithms to eliminate interference and enhance defect features;Based on the fusion model constructed by Bayes theory, the multi-source data is fused with confidence distance measure as an index;Intelligent identification of conveyor belt tear, foreign matter and carrier roller failure is realized by improved YOLOv7 network, sliding window fracture search algorithm and sound harmonic analysis;Alarm information is generated based on the diagnosis result, and hierarchical state display and fault history are recorded.Intelligent state monitoring, fault accurate identification and predictive maintenance of conveyor belt are realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent monitoring and diagnosis technology for industrial equipment, and specifically relates to an intelligent management method for conveyor belts based on multi-sensor fusion. Background Technology

[0002] In heavy industries such as coal, mining, ports, and steel, conveyor belts, as core equipment for the continuous transport of bulk materials, are the "vascular system" of material transportation. Conveyor belts undertake the long-distance transport of bulk materials, but their operating environment often involves extreme conditions such as high temperature, high humidity, dense dust, and severe vibration. Equipment aging, foreign object intrusion, and component wear can easily lead to malfunctions such as breakage and misalignment. According to industry statistics, annual production losses due to conveyor belt failures exceed tens of billions of yuan, and traditional management models are no longer sufficient to meet the stringent reliability requirements of modern industry.

[0003] Existing conveyor belt management technology has evolved from manual inspection to single-sensor monitoring. Early manual inspections relied on maintenance personnel periodically visually inspecting and tapping to determine equipment status. This was not only inefficient but also difficult to detect subtle defects in high-dust, high-noise environments, resulting in a high rate of missed detections. Delayed detection of major hazards often led to "small faults escalating into major accidents." To overcome the limitations of manual methods, the industry gradually introduced single-sensor monitoring technologies, such as X-ray sensors to detect internal steel core damage, visual sensors to identify surface foreign objects, and vibration sensors to monitor the status of transmission components. While X-ray online inspection instruments provide real-time visualization of internal structures, they cannot detect external defects such as surface tears; visual inspection systems are susceptible to changes in lighting and dust obstruction, with accuracy plummeting in dimly lit environments such as underground coal mines; and vibration sensors struggle to distinguish between normal operating vibrations and early fault signals, resulting in a persistently high false alarm rate.

[0004] The current technology system suffers from three major pain points: First, insufficient adaptability to extreme environments. In scenarios such as high altitudes with low oxygen and high temperature and humidity, the drift rate of a single sensor remains high, and equipment in some high-altitude mining areas often fails to monitor due to sensor performance degradation. Second, severe data silos exist, with different types of sensors operating independently. It is difficult to coordinate and analyze internal defect data from X-rays, surface feature data from vision, and acoustic signal data from sonar, creating "detection blind spots." For example, the acoustic characteristics of early wear in idler roller bearings cannot be correlated with abnormal temperature data, missing the fault warning window. Third, delayed decision-making and response. Traditional systems rely heavily on centralized data processing in the cloud. In high-speed conveyor belt operation scenarios, data transmission and analysis delays often lead to alarms being triggered only after a fault has occurred, thus losing the value of preventative maintenance.

[0005] With the deepening of Industry 4.0 and intelligent manufacturing, sensor technology is ushering in a technological revolution of multimodal fusion, edge intelligence, and wireless interconnection. Industrial sensors have evolved from single-parameter sensing to "multi-physical quantity integration." The successful application of multi-sensor fusion technology in fields such as intelligent driving has verified the feasibility of improving detection robustness through the complementarity of heterogeneous data. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides an intelligent management method for conveyor belts based on multi-sensor fusion. The objective of this invention can be achieved through the following technical solution:

[0007] A method for intelligent management of conveyor belts based on multi-sensor fusion includes:

[0008] S1: Collect data on the internal structure of the conveyor belt, the three-dimensional contour data of the surface, the 2D image data of the surface, the sound data of the idler rollers, and the vibration and temperature data of the transmission chain.

[0009] S2: The contrast of the surface 3D contour data is enhanced based on the histogram equalization algorithm, and dust interference is removed by combining the image dust removal algorithm; acoustic features are extracted by performing time-domain and frequency-domain analysis on the idler roller running sound data; interference elimination processing is performed on the acquired surface 2D image data by using corrosion morphology algorithm and uniform processing algorithm; noise reduction processing is performed on the transmission chain vibration temperature data; and internal defect features are highlighted by performing digital signal conversion on the internal structure data of the conveyor belt.

[0010] S3: Based on Bayesian theory, a mathematical model for multi-sensor information fusion is constructed. The confidence distance measure is used as the data fusion degree index. The optimal fusion number of multiple sensors is calculated through the confidence matrix and the fusion matrix to fuse X-ray, 3D contour, 2D image, sound and vibration temperature data.

[0011] S4: A YOLOv7 network based on an integrated hybrid void pyramid module and a spatial pyramid pooling parallel sampling structure, and a sliding window fracture defect search algorithm, combined with a conditional generative adversarial network to expand training samples and a focal loss function; the sliding window fracture defect search algorithm obtains the center position of the laser line based on the Canny contour extraction algorithm, and identifies tear defects by calculating contour features and the proportion of high-brightness pixels; and roller fault identification is achieved based on the harmonic feature analysis of sound signals.

[0012] S5: Based on the comparison between the detection results and the threshold, when it is determined that there is a foreign object or abnormality in the transmission chain, an alarm message is generated through the back-end server, the hierarchical status is displayed, and the fault history and handling suggestions are recorded.

[0013] As a preferred embodiment of the present invention, the processing of the surface three-dimensional contour data includes:

[0014] Based on the histogram equalization algorithm, the surface 3D contour image is divided into independent sub-blocks through local adaptive processing. A gray-level distribution histogram is calculated for each sub-block, and equalization adjustments are made based on the gray-level features of the sub-blocks. Based on the image dust removal algorithm, the surface 3D contour image is selected for pixel comparison to locate suspected dust areas with abnormal gray-level features. Feature matching verification of the suspected dust areas is performed by calling a preset dust feature library. When a dust area is confirmed, neighbor pixel restoration technology is used to reconstruct the pixels in the dust-covered area based on the gray-level values ​​of normal pixels surrounding the dust area, thereby removing dust interference.

[0015] Specifically, the time-domain and frequency-domain analysis of the idler roller operating sound data is performed using the following method:

[0016] Based on the time-domain analysis, the collected idler roller operating sound data is segmented and processed to extract the peak value, root mean square (RMS) value, and pulse count features of the sound signal. The peak value reflects the maximum intensity of the sound signal, the RMS value reflects the overall energy level of the sound signal, and the pulse count feature reflects the frequency of abnormal pulses in the sound signal. Based on the frequency-domain analysis, the time-domain sound signal is converted into a frequency-domain signal using signal transformation technology. The dominant frequency, frequency band energy ratio, and harmonic component features of the frequency-domain signal are extracted to generate an acoustic feature set characterizing the idler roller's operating state. The dominant frequency reflects the main vibration frequency of the idler roller, the frequency band energy ratio reflects the energy distribution in different frequency ranges, and the harmonic component features reflect the additional frequency components in the sound signal.

[0017] Specifically, the processing of the surface 2D image data includes:

[0018] Based on the corrosion morphology algorithm, fine dot-like interference in the surface 2D image is eroded and eliminated by selecting appropriate structural elements; the surface 2D image is smoothed by the uniform processing algorithm, and the mean value of the surface 2D image pixels is calculated by sliding window to eliminate irregular noise in the image background and output interference-free surface 2D image data.

[0019] Specifically, the method for filtering and noise reduction of the transmission chain vibration temperature data is as follows:

[0020] Based on wavelet threshold filtering technology, vibration data is decomposed into wavelet coefficients of different frequency scales by selecting appropriate wavelet basis functions and decomposition levels. A threshold function is set based on the difference between the wavelet coefficients of the noise signal and the effective vibration signal to suppress the wavelet coefficients containing noise. Wavelet reconstruction technology is used to restore the processed wavelet coefficients to time-domain vibration data, thus completing the noise reduction of the vibration data. Based on moving average filtering technology, the mean of the collected temperature data is calculated through a preset time window to eliminate instantaneous fluctuation interference in the temperature data and output a temperature change curve.

[0021] Specifically, the method for highlighting the internal defect features of the conveyor belt is as follows:

[0022] An initial digital image of the internal structure is generated by converting the analog signal acquired by the X-ray sensor into a digital signal. Based on an adaptive threshold segmentation algorithm, a segmentation threshold is set according to the grayscale difference between the conveyor belt substrate and the internal steel core in the initial digital image of the internal structure. The contour features of the steel core region are extracted by an edge detection algorithm to determine the shape and distribution of the steel core. Abnormal features of the steel core are identified based on the feature analysis of the steel core contour. Internal defect features are highlighted by the grayscale contrast of abnormal areas.

[0023] Specifically, the method for constructing a multi-sensor information fusion mathematical model is as follows:

[0024] Based on the sensor's measurement principle, adaptability to on-site conditions, and historical measurement accuracy, initial confidence levels are assigned to X-ray sensors, 3D laser sensors, 2D area array cameras, industrial stethoscopes, and vibration-temperature composite sensors. Confidence distance is used as the data fusion index. The fusion degree of the collected data is determined by calculating the feature similarity and consistency between sensor-collected data. A confidence matrix is ​​constructed to record the confidence relationships of the collected data. Based on these confidence relationships, a fusion matrix is ​​constructed to record the complementarity and redundancy of the collected data. Based on the confidence matrix and the fusion matrix, sensor combinations with strong information complementarity and low redundancy are selected through calculation to determine the optimal number of multi-sensor fusions, thus fusing multi-dimensional data.

[0025] Specifically, the YOLOv7 network integrating the hybrid hollow pyramid module and the spatial pyramid pooling parallel sampling structure includes:

[0026] Based on the hybrid void pyramid module, parallel convolution operations are performed on the input feature map using convolution kernels with varying void ratios. Multi-scale features of the target are extracted based on these kernels, and feature concatenation technology is used to fuse these multi-scale features, outputting a feature map containing scale information. Based on the spatial pyramid pooling parallel sampling structure, the feature map is synchronously sampled by setting a sampling window. Feature information is extracted from the region corresponding to the sampling window, and the feature information is integrated using a feature fusion model to complete the YOLOv7 network's identification of foreign objects on the conveyor belt.

[0027] Specifically, the process of expanding the training samples of the conditional generative adversarial network includes:

[0028] Existing conveyor belt defect samples are used as input data to the generative adversarial network (GAN), along with a random noise vector. By adjusting the parameters of the GAN, the defect morphology, background environment, and lighting conditions of the generated samples are controlled to generate multi-dimensional candidate samples. Based on the input of the discriminant network to the multi-dimensional candidate samples, the authenticity of the multi-dimensional candidate samples is determined according to the characteristics of real samples. Based on the determination results, the multi-dimensional candidate samples determined to be real are added to the training sample set to expand the training samples.

[0029] Specifically, the working process of the sliding window fracture defect search algorithm includes:

[0030] Based on the conveyor belt width characteristics acquired by the 3D laser sensor, the size of the sliding window is adaptively adjusted to cover the entire width range of the conveyor belt. The 3D contour image is scanned traversally based on the movement of the sliding window along the belt. The laser line in the area covered by the sliding window is extracted using the Canny contour extraction algorithm to determine the center position of the laser line. Based on the contour extraction, the continuity characteristics of the laser line contour and the distribution characteristics of high-brightness pixels are calculated. The continuity characteristics reflect the integrity of the laser line contour, and the distribution characteristics of high-brightness pixels reflect the reflective intensity distribution of the conveyor belt surface. When the continuity of the laser line contour is lower than a preset standard and the distribution density of high-brightness pixels is higher than a preset standard, a tear defect is determined in the area.

[0031] Specifically, the method for identifying idler roller faults based on harmonic feature analysis of sound signals is as follows:

[0032] Harmonic extraction is performed on the idler roller operation sound data collected by industrial sonar to separate the fundamental frequency and harmonic frequencies of the sound signal; the ratio of harmonic amplitude to fundamental amplitude is calculated to output a relative intensity chart of harmonics; the harmonic frequency and relative intensity chart are compared with the fault harmonic characteristics, and if the comparison result is in the preset fault area and the relative intensity of the harmonics exceeds the normal range, it is determined that the idler roller has a bearing wear or jamming fault.

[0033] Specifically, the hierarchical status display includes:

[0034] The status display hierarchy is divided into several levels, including the overall system level, regional system level, individual conveyor belt level, and key component level. The overall system level displays the overall operational health status of all conveyor belts, the regional system level displays the statistics of conveyor belt failures and operational trends within a region, the individual conveyor belt level displays the real-time operating parameters and defect distribution of that conveyor belt, and the key component level displays the raw data collected by sensors and the processed feature data. Switching between levels is possible based on user permissions and needs.

[0035] The beneficial effects of this invention are as follows:

[0036] By deploying X-ray sensors, 3D laser sensors, 2D area array cameras, industrial sonar, and vibration-temperature composite sensors, multi-dimensional data coverage of the conveyor belt's internal structure, surface contour, surface image, operating sound, and transmission status is achieved. X-ray sensors can capture deep defects such as internal steel core fractures and corrosion; 3D lasers and 2D area array cameras work together to acquire the surface's three-dimensional contour and two-dimensional details; industrial sonar focuses on the acoustic signals of idler roller operation; and vibration-temperature composite sensors monitor the dynamic status of the transmission chain. Compared to the limitations of traditional single sensors that can only cover single-level defects, this method forms a comprehensive perception system that "covers both internal and external aspects, synchronizes sound and image, and combines dynamic and static elements." It can comprehensively identify various faults in the conveyor belt, such as internal steel core damage, surface tears, foreign object intrusion, idler roller wear, and transmission chain abnormalities, completely eliminating the detection blind spots of traditional management, where "internal defects are difficult to detect, minor surface defects are easily missed, and abnormal operating sounds are difficult to capture."

[0037] To address the challenges of dust concentration, fluctuating lighting, and vibration interference in conveyor belt operating environments, this method employs targeted preprocessing schemes for different data types. For 3D contour data, local adaptive histogram equalization and neighborhood pixel repair for dust removal are used, avoiding the blurring of local details caused by overall equalization while accurately removing dust occlusion. For 2D image data, step-by-step processing using erosion morphology and uniformity algorithms effectively enhances details of dark defects, eliminates point interference, strengthens defect boundaries, and smooths background noise, making surface cracks and foreign object traces clearly visible. For vibration data, wavelet threshold filtering accurately separates effective vibration signals from noise, while moving average filtering eliminates instantaneous fluctuation interference for temperature data. For X-ray internal data, adaptive threshold segmentation and edge detection clearly distinguish the matrix from the steel core, highlighting internal defect characteristics. This significantly reduces the impact of extreme environments on data quality, significantly improving the signal-to-noise ratio and feature recognition of various data types, providing a highly reliable data foundation for subsequent fusion analysis and fault detection.

[0038] A multi-sensor information fusion mathematical model is constructed based on Bayesian theory. Through the logic of "initial confidence level allocation - confidence distance measurement calculation - confidence matrix and fusion matrix construction - optimal fusion number selection," optimal fusion of multi-dimensional data is achieved. On the one hand, by assigning initial confidence levels to different sensors that match their measurement accuracy and operational adaptability, the model ensures that high-reliability sensor data plays a core role in the fusion process. On the other hand, by recording data confidence relationships through the confidence matrix and analyzing data complementarity and redundancy through the fusion matrix, sensor combinations with "strong information complementarity and low redundancy" can be selected, avoiding misjudgments caused by single data biases. Compared to the limitations of traditional independent analysis of sensor data and the difficulty in coordination, multi-modal fusion significantly improves the "accuracy and robustness" of fault diagnosis, effectively reducing the problems of missed and misjudgments caused by the partiality of single data.

[0039] High-precision defect identification was achieved through multi-technology collaboration. For foreign object detection, a YOLOv7 network integrating a hybrid void pyramid module and a spatial pyramid pooling parallel sampling structure was employed. The hybrid void pyramid module can extract features of foreign objects at different scales through multi-void rate convolution, avoiding missed detection of small-sized foreign objects, while the spatial pyramid pooling parallel sampling improves feature extraction efficiency. Simultaneously, the use of a conditional generative adversarial network to expand training samples and a focus loss function significantly improves the generalization ability and accuracy of foreign object detection, especially for small-sized, low-contrast foreign objects. For surface tear defects, the sliding window fracture defect search algorithm, through adaptive window size adjustment, Canny contour extraction, and analysis of continuity and high-brightness pixel distribution, can accurately locate the tear area, avoiding detection deviations caused by changes in conveyor belt width and laser line offset. For idler roller faults, acoustic harmonic feature analysis can capture abnormal signals in the early wear stage of idler roller bearings, enabling early fault detection. This significantly improves the detection accuracy of various conveyor belt faults, significantly reduces the false alarm and missed detection rates, and provides accurate fault information for preventative maintenance.

[0040] In summary, this method, through sensor deployment optimization and data processing technology innovation, can be widely applied to various complex industrial scenarios, significantly expanding the application scope and practical value of the technology. Attached Figure Description

[0041] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0042] Figure 1 This is a flowchart illustrating an intelligent conveyor belt management method based on multi-sensor fusion according to the present invention.

[0043] Figure 2 This is a structural block diagram of the workflow of the sliding window fracture defect search algorithm of the present invention;

[0044] Figure 3 This is an overview of the architecture of the present invention. Detailed Implementation

[0045] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0046] Please see Figure 1 A method for intelligent management of conveyor belts based on multi-sensor fusion includes:

[0047] S1: Collect data on the internal structure of the conveyor belt, the three-dimensional contour data of the surface, the 2D image data of the surface, the sound data of the idler rollers, and the vibration and temperature data of the transmission chain.

[0048] S2: The contrast of the surface 3D contour data is enhanced based on the histogram equalization algorithm, and dust interference is removed by combining the image dust removal algorithm; acoustic features are extracted by performing time-domain and frequency-domain analysis on the idler roller running sound data; interference elimination processing is performed on the acquired surface 2D image data by using corrosion morphology algorithm and uniform processing algorithm; noise reduction processing is performed on the transmission chain vibration temperature data; and internal defect features are highlighted by performing digital signal conversion on the internal structure data of the conveyor belt.

[0049] S3: Based on Bayesian theory, a mathematical model for multi-sensor information fusion is constructed. The confidence distance measure is used as the data fusion degree index. The optimal fusion number of multiple sensors is calculated through the confidence matrix and the fusion matrix to fuse X-ray, 3D contour, 2D image, sound and vibration temperature data.

[0050] S4: A YOLOv7 network based on an integrated hybrid void pyramid module and a spatial pyramid pooling parallel sampling structure, and a sliding window fracture defect search algorithm, combined with a conditional generative adversarial network to expand training samples and a focal loss function; the sliding window fracture defect search algorithm obtains the center position of the laser line based on the Canny contour extraction algorithm, and identifies tear defects by calculating contour features and the proportion of high-brightness pixels; and roller fault identification is achieved based on the harmonic feature analysis of sound signals.

[0051] S5: Based on the comparison between the detection results and the threshold, when it is determined that there is a foreign object or abnormality in the transmission chain, an alarm message is generated through the back-end server, the hierarchical status is displayed, and the fault history and handling suggestions are recorded.

[0052] Specifically, the processing of the surface three-dimensional contour data includes:

[0053] Based on the histogram equalization algorithm, the surface 3D contour image is divided into independent sub-blocks through local adaptive processing. A gray-level distribution histogram is calculated for each sub-block, and equalization adjustments are made based on the gray-level features of the sub-blocks. Based on the image dust removal algorithm, the surface 3D contour image is selected for pixel comparison to locate suspected dust areas with abnormal gray-level features. Feature matching verification of the suspected dust areas is performed by calling a preset dust feature library. When a dust area is confirmed, neighbor pixel restoration technology is used to reconstruct the pixels in the dust-covered area based on the gray-level values ​​of normal pixels surrounding the dust area, thereby removing dust interference.

[0054] In this embodiment, in the scenario of monitoring conveyor belts in underground coal mines, the three-dimensional contour images acquired by the 3D laser sensor often exhibit blurred features due to high concentrations of dust coverage, requiring the following processing:

[0055] Local adaptive histogram equalization processing. The 3D contour image is divided into independent sub-blocks of 8×8 pixels, and a gray-level distribution histogram is calculated for each sub-block separately. For sub-blocks where the gray-level difference between the conveyor belt surface and the background is small (such as the conveyor belt edge area), equalization adjustment is performed based on its gray-level concentration range (100-150), expanding the gray-level range to 50-200 to enhance the edge contour clarity. For sub-blocks containing surface defect areas (such as protrusions and depressions), based on their gray-level discrete characteristics (including a wide range of distribution from 60-200), the gray-level contrast between the defects and the substrate is strengthened by histogram stretching, so that the gray-level difference of the originally blurry small protrusions (gray-level difference ≤10) is increased to ≥30, which is convenient for subsequent feature extraction.

[0056] Image dust removal processing. Three consecutive frames of 3D contour images are selected for pixel-by-pixel comparison to locate areas with abrupt grayscale value changes between 180-255 and irregular dot-like or flocculent morphology—these areas show no continuous positional change across the three frames (asynchronous with the conveyor belt movement) and are identified as suspected dust-covered areas. A pre-set dust feature library (containing over 200 morphological features of common coal dust and rock dust found in underground coal mines, including average particle size, grayscale distribution, and edge gradient) is used to perform feature matching on the suspected areas. When the morphological features of the suspected area (e.g., maximum diameter ≤ 5 pixels, edge gradient ≤ 15) match the coal dust features in the feature library by ≥ 85%, it is confirmed as a dust area. For confirmed dust areas, a 3×3 pixel neighborhood repair technique is used—based on the average grayscale value of the 8 normal pixels surrounding the dust area, the grayscale of the dust-covered pixels is reconstructed.

[0057] After the above processing, the grayscale mean square error of the dust interference area in the 3D contour image decreased from 45 to 12, and the contour clarity of defects such as protrusions and depressions larger than 0.5mm on the conveyor belt surface was improved by more than 60%, providing high-quality feature data for subsequent tear defect identification.

[0058] Specifically, the time-domain and frequency-domain analysis of the idler roller operating sound data is performed using the following method:

[0059] Based on the time-domain analysis, the collected idler roller operating sound data is segmented and processed to extract the peak value, root mean square (RMS) value, and pulse count features of the sound signal. The peak value reflects the maximum intensity of the sound signal, the RMS value reflects the overall energy level of the sound signal, and the pulse count feature reflects the frequency of abnormal pulses in the sound signal. Based on the frequency-domain analysis, the time-domain sound signal is converted into a frequency-domain signal using signal transformation technology. The dominant frequency, frequency band energy ratio, and harmonic component features of the frequency-domain signal are extracted to generate an acoustic feature set characterizing the idler roller's operating state. The dominant frequency reflects the main vibration frequency of the idler roller, the frequency band energy ratio reflects the energy distribution in different frequency ranges, and the harmonic component features reflect the additional frequency components in the sound signal.

[0060] Specifically, the processing of the surface 2D image data includes:

[0061] Based on the corrosion morphology algorithm, fine dot-like interference in the surface 2D image is eroded and eliminated by selecting appropriate structural elements; the surface 2D image is smoothed by the uniform processing algorithm, and the mean value of the surface 2D image pixels is calculated by sliding window to eliminate irregular noise in the image background and output interference-free surface 2D image data.

[0062] In this embodiment, in the monitoring scenario of bulk cargo conveyor belts in ports, images captured by 2D area array cameras often suffer from uneven lighting (such as local overexposure due to direct sunlight during the day, and blurring of dark areas due to insufficient supplemental lighting at night), dust, and water droplets, making it difficult to identify defects such as minor scratches and edge cracks on the conveyor belt surface. The following processing is required:

[0063] Erosion morphology algorithm processing. The image contains numerous tiny dot-like interferences (approximately 1-3 pixels in diameter) formed by dust particles and water droplets. These interferences resemble the local features of minor defects, easily leading to misidentification. A circular structuring element, appropriately sized to match the dot-like interferences, is selected to perform an erosion operation on the image. The structuring element slides along the image pixels one by one; when all pixels within the structuring element's coverage area are interference points, the central pixel is marked as the point to be eliminated. After processing, the dust, water droplets, and other dot-like interferences in the image are completely eroded away, while linear defects such as scratches and cracks on the conveyor belt surface are preserved intact due to the mismatch between their shape and the structuring element.

[0064] The image is processed using a uniform processing algorithm. Even after edge enhancement, the background area (the defect-free area of ​​the conveyor belt) still exhibits irregular grayscale fluctuations (local grayscale differences of 10-15) due to lighting variations and camera noise, easily masking small defects. A sliding window of appropriate size is used to traverse the image, with the average grayscale value of all pixels within the window serving as the new grayscale value for the center pixel. After processing, the grayscale fluctuations in the background area are suppressed (local grayscale difference ≤ 5), resulting in a uniform grayscale distribution. Meanwhile, the defective area, due to its significant grayscale difference from the background (grayscale difference ≥ 30), is clearly distinguished from the smooth background.

[0065] After the above processing, the lighting interference and point noise in the 2D image are effectively eliminated, and the outline and boundary features of defects such as fine scratches and edge cracks on the conveyor belt surface are clearly distinguishable, providing high-quality image data for the subsequent identification of foreign objects and defects by the YOLOv7 network.

[0066] Specifically, the method for filtering and noise reduction of the transmission chain vibration temperature data is as follows:

[0067] Based on wavelet threshold filtering technology, vibration data is decomposed into wavelet coefficients of different frequency scales by selecting appropriate wavelet basis functions and decomposition levels. A threshold function is set based on the difference between the wavelet coefficients of the noise signal and the effective vibration signal to suppress the wavelet coefficients containing noise. Wavelet reconstruction technology is used to restore the processed wavelet coefficients to time-domain vibration data, thus completing the noise reduction of the vibration data. Based on moving average filtering technology, the mean of the collected temperature data is calculated through a preset time window to eliminate instantaneous fluctuation interference in the temperature data and output a temperature change curve.

[0068] Specifically, the method for highlighting the internal defect features of the conveyor belt is as follows:

[0069] An initial digital image of the internal structure is generated by converting the analog signal acquired by the X-ray sensor into a digital signal. Based on an adaptive threshold segmentation algorithm, a segmentation threshold is set according to the grayscale difference between the conveyor belt substrate and the internal steel core in the initial digital image of the internal structure. The contour features of the steel core region are extracted by an edge detection algorithm to determine the shape and distribution of the steel core. Abnormal features of the steel core are identified based on the feature analysis of the steel core contour. Internal defect features are highlighted by the grayscale contrast of abnormal areas.

[0070] In this embodiment, the internal defect detection of a steel-core conveyor belt in an underground coal mine is used as the application scenario. The Otsu adaptive threshold segmentation algorithm is used to process the initial digital image. The specific process is as follows:

[0071] Statistical analysis of the initial image's grayscale histogram revealed a "double-peak" characteristic in the grayscale distribution—the first peak corresponds to the rubber substrate (grayscale peak 25000), and the second peak corresponds to the steel core (grayscale peak 45000), with a significant valley between the two peaks. The algorithm automatically set the segmentation threshold to the valley between the two peaks (calculated threshold 35000), dividing the image pixels into two categories: areas with grayscale values ​​> 35000 were identified as the steel core foreground area, and areas with grayscale values ​​≤ 35000 were identified as the rubber substrate background area. After segmentation, the steel core area and the substrate area were completely separated, with no adhesion or missegmentation. The edge contour of the steel core, which was originally obscured by the substrate grayscale, was initially revealed. However, minor defects inside the steel core (such as cracks with a diameter < 2mm) were still difficult to identify due to poor grayscale uniformity.

[0072] For the segmented steel core foreground region, the Canny edge detection algorithm is used to extract contour features. First, a 5×5 Gaussian filter (standard deviation σ=1.2) is applied to the steel core region to eliminate local gray-level fluctuations caused by X-ray noise and avoid false edge interference. The gradient magnitude and direction of the steel core region are calculated using the Sobel operator to locate edge pixels with abrupt gray-level changes. The "non-maximum suppression" algorithm is used to remove non-edge pixels and retain local maxima pixels only along the gradient direction to form fine edges. A dual threshold is set (high threshold 80, low threshold 40) to filter out strong edges (gradient value > 80) and connect weak edges (40 < gradient value ≤ 80), finally obtaining a continuous and complete steel core contour.

[0073] The extracted steel core contour was parameterized using a contour analysis tool: the steel core was determined to be a circular cross-section (normal diameter 8mm, corresponding to 12 image pixels), and was evenly distributed along the width of the conveyor belt (normal center-to-center spacing 15mm, corresponding to 22 image pixels), forming a normal distribution feature of "uniformly arranged circular contours", providing a benchmark for subsequent anomaly identification.

[0074] Specifically, the method for constructing a multi-sensor information fusion mathematical model is as follows:

[0075] Based on the sensor's measurement principle, adaptability to on-site conditions, and historical measurement accuracy, initial confidence levels are assigned to X-ray sensors, 3D laser sensors, 2D area array cameras, industrial stethoscopes, and vibration-temperature composite sensors. Confidence distance is used as the data fusion index. The fusion degree of the collected data is determined by calculating the feature similarity and consistency between sensor-collected data. A confidence matrix is ​​constructed to record the confidence relationships of the collected data. Based on these confidence relationships, a fusion matrix is ​​constructed to record the complementarity and redundancy of the collected data. Based on the confidence matrix and the fusion matrix, sensor combinations with strong information complementarity and low redundancy are selected through calculation to determine the optimal number of multi-sensor fusions, thus fusing multi-dimensional data.

[0076] Specifically, the YOLOv7 network integrating the hybrid hollow pyramid module and the spatial pyramid pooling parallel sampling structure includes:

[0077] Based on the hybrid void pyramid module, parallel convolution operations are performed on the input feature map using convolution kernels with varying void ratios. Multi-scale features of the target are extracted based on these kernels, and feature concatenation technology is used to fuse these multi-scale features, outputting a feature map containing scale information. Based on the spatial pyramid pooling parallel sampling structure, the feature map is synchronously sampled by setting a sampling window. Feature information is extracted from the region corresponding to the sampling window, and the feature information is integrated using a feature fusion model to complete the YOLOv7 network's identification of foreign objects on the conveyor belt.

[0078] In this embodiment, foreign object detection on a port bulk cargo conveyor belt is used as the application scenario. Targeting common foreign objects on the conveyor belt surface, such as coal piles, metal anchor residues, and woven bag fragments, the YOLOv7-tiny architecture is used as the basic framework. The original network's backbone (including CBS convolution and ELAN modules) and head (including the detection head and loss calculation layer) are retained. After the backbone outputs the feature map and before the head input, a Hybrid Hollow Pyramid Module (HAPM) and a Spatial Pyramid Pooling Parallel Sampling Structure (SPP-PPS) are inserted, forming a detection flow of "backbone→HAPM→SPP-PPS→head". The image of the conveyor belt surface (resolution 1280×720) acquired by a 2D area array camera is used as the network input. First, the size is normalized (scaled to 640×640 while maintaining the aspect ratio). Then, the influence of illumination fluctuations is eliminated by RGB channel mean subtraction (mean value is [123.68, 116.779, 103.939]). Finally, the pixel value is normalized (scaled to the range of 0-1) to obtain the standardized input feature map.

[0079] To address the issue of large size variations in foreign objects on conveyor belts (e.g., metal anchor rods 50-100mm long, woven bag fragments as small as 10-20mm), HAPM extracts features at different scales through parallel convolution with multiple dilation rates. HAPM receives the C3 feature map (256 channels, 80×80 size) output from the backbone and sets up three parallel convolution branches, using 3×3 convolution kernels with dilation rates of 1, 3, and 5 respectively, and each branch has 128 kernels. The three branches simultaneously perform convolution operations on the C3 feature map (all using the ReLU activation function), resulting in three sets of feature maps, each 80×80 in size and 128 channels. Then, feature concatenation technology is used to merge the three sets of feature maps along the channel dimension, outputting a 384-channel (128×3) 80×80 feature map. The multi-scale fusion feature map contains feature information of small, medium and large-sized foreign objects, avoiding the problems of missed detection of small foreign objects or incomplete features of large foreign objects caused by traditional single-scale convolution.

[0080] Specifically, the process of expanding the training samples of the conditional generative adversarial network includes:

[0081] Existing conveyor belt defect samples are used as input data to the generative adversarial network (GAN), along with a random noise vector. By adjusting the parameters of the GAN, the defect morphology, background environment, and lighting conditions of the generated samples are controlled to generate multi-dimensional candidate samples. Based on the input of the discriminant network to the multi-dimensional candidate samples, the authenticity of the multi-dimensional candidate samples is determined according to the characteristics of real samples. Based on the determination results, the multi-dimensional candidate samples determined to be real are added to the training sample set to expand the training samples.

[0082] Specifically, the working process of the sliding window fracture defect search algorithm includes:

[0083] Based on the conveyor belt width characteristics acquired by the 3D laser sensor, the size of the sliding window is adaptively adjusted to cover the entire width range of the conveyor belt. The 3D contour image is scanned traversally based on the movement of the sliding window along the belt. The laser line in the area covered by the sliding window is extracted using the Canny contour extraction algorithm to determine the center position of the laser line. Based on the contour extraction, the continuity characteristics of the laser line contour and the distribution characteristics of high-brightness pixels are calculated. The continuity characteristics reflect the integrity of the laser line contour, and the distribution characteristics of high-brightness pixels reflect the reflective intensity distribution of the conveyor belt surface. When the continuity of the laser line contour is lower than a preset standard and the distribution density of high-brightness pixels is higher than a preset standard, a tear defect is determined in the area.

[0084] In this embodiment, a steel-core conveyor belt in the main transport roadway of a coal mine is used as the application object. Addressing the surface tearing caused by coal gangue impact and steel core fatigue (common tear width 2-10mm, length 50-300mm), a 3D laser sensor (adapted to the explosion-proof environment of coal mines, with a laser line projection width of 2m and Z-axis accuracy of 0.1mm) is used to scan the operating conveyor belt, collecting three-dimensional contour data of the conveyor belt surface. An edge detection algorithm is used to identify the edge pixels on both sides of the conveyor belt (where grayscale values ​​change abruptly from 120-180 in the conveyor belt area to 20-50 in the background area). The spacing between the edge pixels on both sides in the image width direction is calculated. The actual width of the target conveyor belt is 1.2m, corresponding to 600 pixel units in the width dimension of the three-dimensional contour image (i.e., 1 pixel unit corresponds to 2mm).

[0085] To ensure the sliding window fully covers the width of the conveyor belt without including redundant background, the width dimension of the sliding window is set to 600 pixels (consistent with the width of the conveyor belt image, corresponding to an actual length of 1.2m), which precisely covers the entire lateral range of the conveyor belt. Combining the conveyor belt running speed (3m / s) and the sampling frequency of the 3D laser sensor (20 frames / second, each frame corresponding to a longitudinal length of 150mm for the conveyor belt), the length dimension of the sliding window is set to 25 pixels (corresponding to an actual length of 50mm). This size avoids both the dilution of local defects caused by an excessively long window and the reduction in detection efficiency caused by an excessively short window, thus achieving the scanning basis of "full lateral coverage and precise longitudinal segmentation".

[0086] Using the 3D contour image (single frame size 600×400 pixels, corresponding to a 1.2m×0.8m area of ​​the conveyor belt) output by the 3D laser sensor as the processing object, the sliding window moves sequentially along the running direction of the conveyor belt (image length dimension), with a movement step size set to 12.5 pixel units (corresponding to 25mm in reality). That is, each time the sliding window moves, there is a 50% vertical overlap area with the previous window. This overlap design can avoid missed detections caused by surface defects of the conveyor belt crossing the window boundary.

[0087] For a single-frame 3D contour image, the sliding window starts from the beginning of the image length dimension (upstream of the conveyor belt running direction) and moves sequentially to the end (downstream), requiring a total of 32 moves ((400-25)÷12.5 +1=32) to complete the full traversal of the single-frame image. At the same time, the algorithm is synchronized with the sampling frequency of the 3D laser sensor. After each frame of image traversal is completed, the next frame of data is automatically received, realizing real-time scanning of the continuous operation of the conveyor belt.

[0088] Specifically, the method for identifying idler roller faults based on harmonic feature analysis of sound signals is as follows:

[0089] Harmonic extraction is performed on the idler roller operation sound data collected by industrial sonar to separate the fundamental frequency and harmonic frequencies of the sound signal; the ratio of harmonic amplitude to fundamental amplitude is calculated to output a relative intensity chart of harmonics; the harmonic frequency and relative intensity chart are compared with the fault harmonic characteristics, and if the comparison result is in the preset fault area and the relative intensity of the harmonics exceeds the normal range, it is determined that the idler roller has a bearing wear or jamming fault.

[0090] Specifically, the hierarchical status display includes:

[0091] The status display hierarchy is divided into several levels, including the overall system level, regional system level, individual conveyor belt level, and key component level. The overall system level displays the overall operational health status of all conveyor belts, the regional system level displays the statistics of conveyor belt failures and operational trends within a region, the individual conveyor belt level displays the real-time operating parameters and defect distribution of that conveyor belt, and the key component level displays the raw data collected by sensors and the processed feature data. Switching between levels is possible based on user permissions and needs.

[0092] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multi-sensor fusion-based intelligent management method for a conveyor belt, characterized by, The method comprises the following steps: S1: collecting the internal structure data of the conveyor belt, the surface three-dimensional profile data, the surface 2D image data, the running sound data of the carrier roller and the vibration temperature data of the transmission chain; S2: enhancing the contrast of the surface three-dimensional profile data based on a histogram equalization algorithm, and removing dust interference by combining an image dust removal algorithm; extracting acoustic features by analyzing the running sound data of the carrier roller in the time domain and the frequency domain; processing the collected surface 2D image data by interference elimination through an erosion morphology algorithm and a uniform processing algorithm; filtering and denoising the vibration temperature data of the transmission chain; highlighting the internal defect features by converting the internal structure data of the conveyor belt into a digital signal; S3: constructing a multi-sensor information fusion mathematical model based on Bayesian theory, taking confidence distance measurement as a data fusion degree index, calculating the best fusion number of multiple sensors through a confidence matrix and a fusion matrix, and fusing the X-ray, 3D profile, 2D image, sound and vibration temperature data; S4: based on the YOLOv7 network with integrated mixed hollow pyramid module and spatial pyramid pooling parallel sampling structure and the sliding window crack defect search algorithm, combining the conditional generative adversarial network to expand the training samples and the focal loss function; the sliding window crack defect search algorithm obtains the laser line center position based on the Canny contour extraction algorithm, and identifies the tearing defect by calculating the contour feature and the proportion of high brightness pixels; based on the harmonic feature analysis of the sound signal, the carrier roller fault is identified; S5: based on the comparison of the detection result and the threshold value, when it is determined that there is foreign matter or transmission chain abnormality, alarm information is generated through the background server, hierarchical state display is performed, and fault history and processing suggestions are recorded.

2. The method of claim 1, wherein, The processing of the surface three-dimensional profile data in S2 comprises: based on the histogram equalization algorithm, the surface three-dimensional profile image is divided into independent sub-blocks by a local adaptive processing method, the gray distribution histogram of the independent sub-blocks is calculated, and the equalization adjustment is performed based on the gray features of the independent sub-blocks; based on the image dust removal algorithm, the surface three-dimensional profile image is selected for pixel comparison, the suspected dust area with abnormal gray features is located, the suspected dust area is verified by calling a preset dust feature library for feature matching; when it is confirmed as a dust area, the pixels of the dust covered area are reconstructed based on the gray values of the normal pixels around the dust area through the neighborhood pixel repair technology, and the dust interference is removed.

3. The method of claim 1, wherein, The time domain and frequency domain analysis of the carrier roller running sound data in S2 is as follows: based on the time domain analysis, the collected carrier roller running sound data is processed in segments, and the peak value, root mean square and pulse number features of the sound signal are extracted; The peak feature reflects the maximum intensity of the sound signal, the root mean square feature reflects the overall energy level of the sound signal, and the pulse number feature reflects the frequency of occurrence of abnormal pulses in the sound signal; based on the frequency domain analysis, the time domain sound signal is converted into a frequency domain signal through a signal transformation technique, the main frequency, frequency band energy proportion and harmonic component features of the frequency domain signal are extracted, and an acoustic feature set representing the running state of the roller is generated; the main frequency feature reflects the main vibration frequency of the roller running, the frequency band energy proportion feature reflects the energy distribution of different frequency intervals, and the harmonic component feature reflects the additional frequency components in the sound signal.

4. The method of claim 1, wherein, The processing of the surface 2D image data in S2 includes: Based on the erosion morphological algorithm, small point-shaped interference in the surface 2D image is eroded and eliminated by selecting an appropriate structural element; based on the uniform processing algorithm, the surface 2D image is background smoothed, the image pixels are mean calculated by a sliding window, the irregular noise in the image background is eliminated, and the surface 2D image data without interference is output.

5. The method of claim 1, wherein, The method for filtering and denoising the transmission chain vibration temperature data in S2 is: Based on wavelet threshold filtering technology, the vibration data is decomposed into wavelet coefficients of different frequency scales by selecting an appropriate wavelet basis function and decomposition level; based on the difference between the wavelet coefficients of noise signals and effective vibration signals, a threshold function is set to suppress the wavelet coefficients containing noise; the processed wavelet coefficients are restored to time domain vibration data by wavelet reconstruction technology, and the denoising of the vibration data is completed; Based on the sliding average filtering technology, the temperature data collected is mean calculated by a preset time window, the instantaneous fluctuation interference in the temperature data is eliminated, and the temperature change curve is output.

6. The method of claim 1, wherein, The highlighting of the internal defect features of the conveyor belt in S2 is achieved by: By converting the analog signal collected by the X-ray sensor into a digital signal, an initial internal structure digital image is generated; based on the adaptive threshold segmentation algorithm, a segmentation threshold is set according to the gray difference between the conveyor belt matrix and the internal steel core in the initial internal structure digital image; the profile features of the steel core region are extracted by the edge detection algorithm to determine the shape and distribution of the steel core; the abnormal features of the steel core are identified based on the feature analysis of the steel core profile; the internal defect features are highlighted by the gray contrast of the abnormal area.

7. The method of claim 1, wherein, The method for constructing a multi-sensor information fusion mathematical model in S3 is: Based on the measurement principle of the sensor, the adaptability of the field conditions and the historical measurement accuracy, the initial confidence of the X-ray sensor, the 3D laser sensor, the 2D area camera, the industrial auscultation sonar and the vibration temperature composite sensor is allocated, the confidence distance measure is taken as the data fusion degree index, the feature similarity and consistency between the collected data are calculated to determine the fusion degree of the collected data; The confidence matrix is constructed to record the confidence relationship of the collected data, and the fusion matrix is constructed based on the confidence relationship to record the complementarity and redundancy of the collected data. Based on the confidence matrix and fusion matrix, sensor combinations with strong information complementarity and low redundancy are selected through calculation and screening, the optimal number of multi-sensor fusions is determined, and multi-dimensional data is fused.

8. The method of claim 1, wherein, The YOLOv7 network integrating the hybrid hollow pyramid module and the spatial pyramid pooling parallel sampling structure described in S4 includes: Based on the hybrid void pyramid module, parallel convolution operations are performed on the input feature map using convolution kernels with varying void ratios. Multi-scale features of the target are extracted based on these kernels, and feature concatenation technology is used to fuse these multi-scale features, outputting a feature map containing scale information. Based on the spatial pyramid pooling parallel sampling structure, the feature map is synchronously sampled by setting a sampling window. Feature information is extracted from the region corresponding to the sampling window, and the feature information is integrated using a feature fusion model to complete the YOLOv7 network's identification of foreign objects on the conveyor belt.

9. The method of claim 1, wherein, The process of expanding the training samples of the conditional generative adversarial network described in S4 includes: Existing conveyor belt defect samples are used as input data to the generative adversarial network (GAN), along with a random noise vector. By adjusting the parameters of the GAN, the defect morphology, background environment, and lighting conditions of the generated samples are controlled to generate multi-dimensional candidate samples. Based on the input of the discriminant network to the multi-dimensional candidate samples, the authenticity of the multi-dimensional candidate samples is determined according to the characteristics of real samples. Based on the determination results, the multi-dimensional candidate samples determined to be real are added to the training sample set to expand the training samples.

10. The method of claim 1, wherein, The working process of the sliding window fracture defect search algorithm described in S4 includes: Based on the conveyor belt width characteristics acquired by the 3D laser sensor, the size of the sliding window is adaptively adjusted to cover the entire width range of the conveyor belt. The 3D contour image is scanned traversally based on the movement of the sliding window along the belt. The laser line in the area covered by the sliding window is extracted using the Canny contour extraction algorithm to determine the center position of the laser line. Based on the contour extraction, the continuity characteristics of the laser line contour and the distribution characteristics of high-brightness pixels are calculated. The continuity characteristics reflect the integrity of the laser line contour, and the distribution characteristics of high-brightness pixels reflect the reflective intensity distribution of the conveyor belt surface. When the continuity of the laser line contour is lower than a preset standard and the distribution density of high-brightness pixels is higher than a preset standard, a tear defect is determined in the area.

11. The method of claim 1, wherein, The method for identifying idler roller faults based on harmonic feature analysis of sound signals, as described in S4, is as follows: Harmonic extraction is performed on the idler roller operation sound data collected by industrial sonar to separate the fundamental frequency and harmonic frequencies of the sound signal; the ratio of harmonic amplitude to fundamental amplitude is calculated to output a relative intensity chart of harmonics; the harmonic frequency and relative intensity chart are compared with the fault harmonic characteristics, and if the comparison result is in the preset fault area and the relative intensity of the harmonics exceeds the normal range, it is determined that the idler roller has a bearing wear or jamming fault.

12. The method of claim 1, wherein, The hierarchical status display described in S5 includes: The status display hierarchy is divided into several levels, including the overall system level, regional system level, individual conveyor belt level, and key component level. The overall system level displays the overall operational health status of all conveyor belts, the regional system level displays the statistics of conveyor belt failures and operational trends within a region, the individual conveyor belt level displays the real-time operating parameters and defect distribution of that conveyor belt, and the key component level displays the raw data collected by sensors and the processed feature data. Switching between levels is possible based on user permissions and needs.

Images