Visual intelligent monitoring method and system for operation of belt conveyor

Abstract

The invention relates to a visual intelligent monitoring method and system for operation of a belt conveyor. The method comprises the following steps: collecting a video stream through visual sensing equipment deployed along a belt conveyor; performing dynamic image quality evaluation and enhancement at an edge computing node, and operating a lightweight deep learning model to detect typical anomalies in real time; meanwhile, uploading the data to a cloud end, extracting high-dimensional features by using a visual basic model, and comparing the high-dimensional features with a dynamic feature distribution model to identify unknown anomalies; when an exception is found, data from the non-visual sensing module in the same time period are automatically called and analyzed to perform collaborative verification; and through a verification result, carrying out fusion decision making and generating final diagnosis and early warning information. The system comprises a visual perception module, an edge computing gateway, a non-visual sensing module and a cloud intelligent analysis platform. According to the invention, omnibearing and high-reliability intelligent monitoring and early warning of the running state of the belt conveyor are realized.

Description

Technical Field

[0001] This invention relates to the field of belt conveyor technology, and in particular to a visual intelligent monitoring method and system for belt conveyor operation. Background Technology

[0002] Belt conveyors, as important continuous transport equipment, are widely used in industries such as mining, metallurgy, and coal to transport bulk materials or packaged goods. During long-term operation, they are prone to malfunctions such as belt misalignment, longitudinal tearing, bearing damage, and material blockage due to factors such as equipment aging, material impact, and environmental corrosion. These malfunctions not only lead to unplanned downtime and affect production efficiency, but may also cause equipment damage or even safety accidents.

[0003] Several belt conveyor monitoring solutions have been proposed in the existing technology. For example, Chinese patent application CN115908273A, entitled "A Method and System for Automatic Monitoring of Belt Misalignment Based on Deep Learning," proposes to monitor the belt status in real time through a GPU analysis server, construct a belt status dataset, and identify and alarm on misalignment based on deep learning technology. This method achieves automatic detection of belt misalignment through image processing technology, which reduces the burden of manual inspection to some extent. However, this existing technology still has obvious limitations in practical applications: First, its visual monitoring method is easily affected by environmental factors common in industrial sites such as dust, fog, and changes in lighting, and lacks an effective image quality assurance mechanism; second, the method mainly identifies preset misalignment faults and lacks the ability to detect other types of faults or unknown anomalies; third, model updates and optimization rely on historical data and lack online learning and adaptive optimization mechanisms; finally, this solution mainly relies on visual single-modal data and does not make full use of multi-source information such as vibration and sound for collaborative verification, affecting the reliability of diagnostic results.

[0004] Therefore, there is an urgent need in this field for an intelligent monitoring method and system for belt conveyors that can adapt to complex industrial environments, has strong generalization capabilities, supports the identification of unknown anomalies, and can optimize performance through continuous learning. Summary of the Invention

[0005] The present invention aims to address the shortcomings of the prior art by providing a visual intelligent monitoring method and system for belt conveyor operation.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a visual intelligent monitoring method for belt conveyor operation, comprising the following steps:

[0007] S1. Real-time video streams of the belt running area are collected by visual sensing devices deployed along the belt conveyor.

[0008] S2. On the edge computing node, perform dynamic image quality assessment and enhancement processing on the video stream, and run a lightweight deep learning model on the processed video stream to detect preset typical abnormal states in real time.

[0009] S3. Upload the video stream and edge processing results to the cloud analysis platform;

[0010] S4. In the cloud, high-dimensional features are extracted from video frames using a pre-trained visual basic model, and the high-dimensional features are compared with a dynamic feature distribution model built on historical normal operation data to identify abnormal states that deviate from the normal pattern and are not preset in the lightweight deep learning model.

[0011] S5. When visual analysis detects an anomaly, it automatically retrieves and analyzes data from non-visual sensing modules within the same time period for collaborative verification.

[0012] S6. Based on the verification results of visual analysis and non-visual sensor data, make fusion decisions and generate the final diagnostic and early warning information.

[0013] In particular, this method also includes an active learning loop for unknown cases:

[0014] Once the system identifies an unknown anomaly and generates an alert, it automatically packages the multimodal data within that time period into a case package to be confirmed.

[0015] After the operations and maintenance personnel confirm and label the case package, the system automatically includes it in an incremental learning sample library;

[0016] The cloud-based analytics platform regularly fine-tunes the visual base model and / or dynamic feature distribution model using an incremental learning sample library.

[0017] Specifically, in S2, dynamic image quality assessment and enhancement processing includes:

[0018] Calculate the signal-to-noise ratio and contrast of an image in real time;

[0019] When any indicator falls below its preset threshold, at least one of the following actions will be automatically triggered:

[0020] Activate the cleaning device integrated into the visual sensing device;

[0021] Switch to a fog-penetrating image restoration algorithm based on dark channel priors;

[0022] Automatically adjusts the intensity and angle of the fill light.

[0023] Specifically, in S4, the process of establishing and updating the dynamic feature distribution model includes:

[0024] During the system initialization phase, visual data of the belt conveyor under normal operating conditions is collected, and its feature vectors are extracted using the basic visual model to construct the initial normal feature distribution.

[0025] During system operation, the feature distribution is periodically updated incrementally using verified normal data to adapt to equipment aging and slow changes in operating conditions.

[0026] In particular, the deep learning model deployed on edge computing nodes serves as the student model, while the visual foundation model deployed in the cloud serves as the teacher model. There is a knowledge distillation relationship between the two: the student model improves its generalization ability and recognition accuracy by periodically synchronizing parameters or learning output features from the teacher model.

[0027] Specifically, in S4, identifying abnormal states that deviate from the normal pattern includes predictive monitoring of the health status of key components of the conveyor belt: by comparing the gradual trend of visual features of the same component in historical data, when a subtle change that conforms to a preset degradation pattern is identified, a predictive maintenance alarm for the degradation of the component's health is generated, rather than an emergency fault alarm.

[0028] Specifically, in S5, the non-visual sensing module includes distributed fiber optic vibration sensors deployed along the conveyor belt frame and an array of audio sensors deployed near key components; collaborative verification specifically includes:

[0029] When visual analysis identifies an abnormal area, it calls the data of the distributed fiber optic vibration sensor segment corresponding to that area for vibration spectrum analysis, and / or calls the data of the audio sensor array covering that area for sound source localization and abnormal sound pattern identification.

[0030] By aligning visual positioning information with position segment information from vibration sensors and sound source positioning information from acoustic sensors in a spatiotemporal manner, the overall confidence level of the anomaly diagnosis is significantly improved when the multi-source information is consistent in both space and time.

[0031] Specifically, in S6, the fusion decision-making adopts a confidence-weighted decision-making mechanism:

[0032] Assign a confidence weight Wv to the visual analysis results of S4 based on its model confidence and real-time image quality;

[0033] Assign a confidence weight Ws to the data analysis results of the non-visual sensing module of S5 based on its data quality and feature significance;

[0034] The final decision output is a weighted sum of visual and non-visual analysis results. When the confidence level of the weighted sum exceeds a preset threshold, an alert is confirmed.

[0035] A visual intelligent monitoring system for belt conveyor operation, used to implement a method for visual intelligent monitoring of belt conveyor operation, comprising:

[0036] The visual perception module includes one or more visual sensing devices deployed along the belt conveyor. The visual sensing devices are integrated with a self-cleaning device and a controllable supplementary lighting system for real-time acquisition of video streams of the belt running area.

[0037] The edge computing gateway, communicating with the vision perception module, internally stores and runs a lightweight deep learning model as a student model, and is configured as follows:

[0038] The received video stream is dynamically image quality evaluated, and the self-cleaning device, controllable supplementary lighting system, or fog-penetrating image restoration algorithm is triggered based on the evaluation results to enhance the video stream.

[0039] Run a lightweight deep learning model on the enhanced video stream to detect pre-defined typical anomalous states in real time;

[0040] The non-visual sensing module includes:

[0041] Distributed fiber optic vibration sensors are installed along the conveyor belt frame to provide vibration data with position information;

[0042] An array of audio sensors deployed near key components is used to provide audio data with sound source localization information;

[0043] The cloud-based intelligent analysis platform communicates with the edge computing gateway and non-visual sensing modules, and is equipped with a visual basic model unit as the teacher model. This unit is used to extract high-dimensional features from uploaded video frames. The student model and the teacher model form a knowledge distillation relationship.

[0044] In particular, the cloud-based intelligent analytics platform also includes:

[0045] The dynamic feature library stores dynamic feature distribution models built based on historical normal data; the visual basic model unit compares the extracted high-dimensional features with the dynamic feature distribution model to identify unknown anomalies that are not pre-defined.

[0046] The multimodal fusion diagnostic unit is configured as follows:

[0047] When the visual basic model unit identifies an anomaly, it automatically retrieves the segment data of the distributed fiber optic vibration sensor corresponding to the anomaly area within the same time period and / or the data of the audio sensor array covering the area, performs vibration spectrum analysis and sound source localization, and performs spatiotemporal alignment and collaborative verification of the visual, vibration, and acoustic localization information.

[0048] The early warning and decision-making unit adopts a confidence-weighted decision-making mechanism to perform fusion decision-making on the output of the multimodal fusion diagnostic unit and generate diagnostic and early warning information.

[0049] The active learning engine is configured to package unknown anomaly cases into a case package to be confirmed. After receiving annotations from operations and maintenance personnel, the package is incorporated into the incremental learning sample library and used to fine-tune the visual basic model unit and / or dynamic feature library.

[0050] The beneficial effects of this invention are:

[0051] It achieves reliable monitoring around the clock: Through dynamic image quality assessment and enhancement processing, it can automatically cope with complex environmental interferences such as dust, fog, and changes in lighting in industrial sites, ensuring that the visual monitoring system can obtain clear image data under different working conditions, significantly improving the system's environmental adaptability and monitoring reliability.

[0052] It has the ability to identify unknown anomalies: It adopts a two-layer architecture of "edge lightweight deep learning model + cloud visual basic model". The edge side is responsible for real-time detection of preset typical anomalies, while the cloud side effectively identifies unknown abnormal states that are not preset and deviate from the normal pattern by comparing high-dimensional features with dynamic feature distribution models, which greatly expands the fault detection range of the system.

[0053] A continuously optimized intelligent closed loop has been constructed: by introducing an active learning mechanism for unknown cases, the system can automatically include abnormal cases confirmed by operation and maintenance personnel into the incremental learning sample library, and regularly fine-tune the model, so that the system has the ability to continuously evolve and gradually improve the recognition accuracy and generalization ability.

[0054] Improved accuracy and reliability of diagnostic results: Through a multimodal data collaborative verification mechanism, when visual analysis identifies anomalies, it automatically retrieves and analyzes non-visual sensor data such as vibration and audio within the same time period, and performs spatiotemporal alignment and fusion decision-making, effectively reducing false alarm and false negative rates and significantly improving the overall confidence of anomaly diagnosis.

[0055] Predictive maintenance has been achieved: the system can identify subtle changes that conform to preset degradation patterns by comparing the gradual trend of visual features of components in historical data, thereby generating predictive maintenance alarms before failures occur, realizing the transformation from reactive maintenance to predictive maintenance and effectively reducing unplanned downtime.

[0056] The system resource and performance balance has been optimized: by the division of labor and cooperation between edge computing nodes and cloud platforms, and by establishing a knowledge distillation relationship between the two, the edge model can periodically learn from the cloud teacher model, thereby continuously improving the recognition accuracy and generalization ability of the lightweight edge model while ensuring real-time performance.

[0057] It provides precise decision support: by adopting a confidence-weighted fusion decision mechanism, it comprehensively considers the quality and significance of visual analysis results and non-visual sensor data to generate a weighted comprehensive confidence score, providing operation and maintenance personnel with more scientific and accurate early warning information to assist in rapid decision-making. Attached Figure Description

[0058] Figure 1 This is a flowchart of the method of the present invention;

[0059] Figure 2 This is a structural block diagram of the cloud-based intelligent analysis platform of the system of the present invention;

[0060] Figure 3 This is a schematic diagram of the sensor layout of the system of the present invention;

[0061] The following will describe in detail, with reference to the accompanying drawings, embodiments of the present invention. Detailed Implementation

[0062] The present invention will be further described below with reference to embodiments:

[0063] like Figure 1 As shown, a visual intelligent monitoring method for belt conveyor operation includes the following steps:

[0064] S1. Real-time video streams of the belt's operating area are collected using visual sensing devices deployed along the belt conveyor. Specifically, this step is the system's basic data input stage. High-performance industrial cameras are deployed along the belt conveyor at key locations (such as drive rollers, redirecting rollers, material drop points, and tensioning devices) to form a visual perception network. These devices continuously collect video footage of the belt and the materials above it at a certain frame rate, generating raw, unprocessed video stream data to provide raw materials for subsequent intelligent analysis.

[0065] Visual information can cover a wide area, providing the most intuitive images of the belt surface condition, material distribution, and belt misalignment. It is the most direct means of identifying anomalies such as tearing, misalignment, and large foreign objects. It requires no direct contact with the equipment, does not affect the normal operation of the belt conveyor, and avoids the mechanical complexity associated with sensor installation.

[0066] S2. On the edge computing node, dynamic image quality assessment and enhancement processing are performed on the video stream, and a lightweight deep learning model is run on the processed video stream to detect preset typical abnormal states in real time. Dynamic image quality assessment and enhancement processing is the core link to cope with complex industrial field environments and ensure the reliability of visual intelligent monitoring. It provides high-quality, clear and reliable visual input for subsequent deep learning model analysis, fundamentally reducing false detections and missed detections caused by image quality problems.

[0067] Typical abnormal states include, but are not limited to, the following categories:

[0068] a. Belt misalignment:

[0069] Visual characteristics: The belt centerline deviates from the belt conveyor frame centerline, and one edge of the belt is in contact with or close to contact with the idler or frame. For example, in a normal operation image, the distance between the belt edge and the side of the frame remains within a stable range. When the lightweight model detects that this distance is consistently less than a preset threshold (e.g., 50 mm), a "belt misalignment" alarm is triggered.

[0070] b. Large foreign object intrusion:

[0071] Visual characteristics: The presence of a significantly sized object on the belt, rollers, or frame that is not conveying material. For example, a broken piece of wood, a fallen tool, or a large rock stuck between the belt and rollers. The lightweight model can identify the outline and texture of these objects, which differ significantly from normal materials and equipment, and immediately issue an alarm to prevent mechanical damage to the equipment.

[0072] c. The conveyor belt is severely torn longitudinally.

[0073] Visual characteristics: Continuous, narrow, high-contrast dark lines appear on the belt surface, or irregular breaks appear at the belt edges. For example, when the belt is torn by a sharp foreign object, a visible tear is formed. The lightweight model can identify this serious and destructive failure by analyzing the continuity of the belt surface texture and trigger the highest level of emergency shutdown alarm.

[0074] d. Large-area material application:

[0075] Visual characteristics: A large amount of material accumulates in the non-carrying area of ​​the conveyor belt, such as under the idler rollers, at the bottom of the frame, or on the return belt. For example, due to blockage, misalignment, or impact, materials such as coal powder and ore may spill and accumulate from the belt. The lightweight model identifies spillage by recognizing a large number of pixels and their texture features that should not be present in specific areas (such as below the drop point).

[0076] The idler roller is completely jammed or detached:

[0077] Visual characteristics: The idler roller stops rotating, and its surface rubs violently against the running belt, generating smoke or dust; or the idler roller is missing from its installation position. For example, a stuck idler roller appears as a stationary cylinder in a video, while the idlers around it are rotating. The lightweight model can quickly identify such faults by analyzing the motion state of the idler rollers in consecutive frames, preventing fires caused by frictional heat.

[0078] f. Open flame and smoke:

[0079] Visual characteristics: Irregular, flickering bright spots (open flames) or grayish-white, semi-transparent diffused areas (smoke) appear in the image. For example, flames or smoke may be generated locally in equipment due to bearing overheating or electrical short circuits. The lightweight model, trained on a large number of flame and smoke images, can sensitively capture these special color, shape, and motion characteristics, enabling early warning of fires.

[0080] By deploying lightweight models at the edge specifically for real-time detection of typical anomalies, rapid response is achieved, gaining valuable time for emergency shutdowns and preventing the escalation of incidents. Simultaneously, this design offloads computational load, allowing the cloud platform to focus on more complex and time-consuming tasks of discovering and deeply analyzing unknown anomalies, thus constructing an efficient, reliable, and fully functional intelligent monitoring system.

[0081] Dynamic image quality assessment and enhancement processing specifically includes:

[0082] The system calculates the signal-to-noise ratio (SNR) and contrast ratio of images in real time. Specifically, dynamic image quality assessment metrics include: SNR: quantifying the ratio of effective information to noise (such as dust particles and electronic noise) in an image. The SNR decreases significantly in environments with high dust concentrations. Contrast ratio: measuring the difference between the brightest and darkest areas of an image. Contrast ratio decreases when there is insufficient lighting or lens smudges causing image blurring. Evaluation process: The system calculates the actual values ​​of the above metrics for each frame of the video stream or by sampling frames at a set frequency, and compares them with preset empirical thresholds. For example, when the SNR is below 25dB and / or the contrast ratio is below 0.3 (after normalization), the system determines that the current image quality does not meet the analysis requirements.

[0083] When any metric falls below its preset threshold, at least one of the following actions is automatically triggered, which can enhance performance in a synergistic way rather than through a single, fixed action:

[0084] The cleaning device integrated into the vision sensor is activated; for example, the vision sensor integrates compressed air nozzles or a micro-scraping mechanism, which the operator can select according to actual needs. When the evaluation module detects persistently poor image quality (e.g., contrast remains below a threshold for 5 seconds), it sends a pulse signal to the cleaning device. Directly removing water droplets, oil, and sticky dust adhering to the lens cover is a fundamental solution to image blur.

[0085] The system switches to a fog-penetrating image restoration algorithm based on dark channel priors. It first calculates the dark channel of the image to estimate the transmittance map caused by dust, and then reverse-engineers a clear image free of dust. This effectively eliminates the hazy effect caused by uniformly diffused dust and moisture, restoring obscured image details and colors.

[0086] The system automatically adjusts the intensity and angle of the fill light; for example, the fill light system consists of multiple independently controllable LED arrays with finely adjustable illumination angles. When the signal-to-noise ratio is low, the system can increase the fill light intensity; when the image is locally overexposed or has excessive shadows, the illumination angle of a specific array can be adjusted to achieve uniform lighting. It overcomes adverse conditions such as insufficient or uneven ambient lighting and backlighting. By actively improving the imaging environment, it enhances image quality from the source, reducing the burden on subsequent algorithm processing and the noise that may be introduced.

[0087] Meanwhile, the system does not perform these operations in isolation, but intelligently selects the optimal combination based on the image quality assessment results. For example, when the system detects a sudden drop in the image signal-to-noise ratio and a slow decrease in contrast, it may determine that there is a sudden dust storm. In this case, the system may first enhance the supplementary lighting to try to penetrate the dust. If the effect is not good, it will immediately activate the defogging algorithm for real-time correction, while the self-cleaning device serves as a follow-up guarantee against persistent pollution. This ensures that the lightweight deep learning model can obtain "clear" images under any operating conditions, greatly reducing false alarms and false negatives caused by environmental factors. Upgrading the traditional "periodic manual cleaning" or "fixed parameter shooting" to "on-demand self-cleaning and adaptive imaging" significantly reduces maintenance workload and ensures the continuous and effective operation of the system. Integrating image quality assessment, physical cleaning, optical adjustment, and algorithm enhancement into a complete, self-optimizing closed-loop system for image quality assurance.

[0088] S3. Upload the video stream and edge processing results to the cloud analytics platform. Specifically, after completing local processing and detection, the edge nodes upload the enhanced video stream, along with any detected anomalies (including time, location, type, confidence level, etc.), to the cloud analytics platform. The cloud provides virtually unlimited storage space for long-term preservation of high-quality live video and data, facilitating historical review, trend analysis, and model optimization. Aggregating data in the cloud enables the subsequent analysis of more complex and powerful models.

[0089] S4. In the cloud, high-dimensional features are extracted from video frames using a pre-trained visual basic model, and the high-dimensional features are compared with a dynamic feature distribution model built on historical normal operation data to identify abnormal states that deviate from the normal pattern and are not preset in the lightweight deep learning model.

[0090] The process of establishing and updating the dynamic feature distribution model includes:

[0091] During the system initialization phase, visual data of the belt conveyor under normal operating conditions (such as no load, half load, full load, etc.) are collected, and feature vectors are extracted using the basic visual model to construct the initial normal feature distribution.

[0092] During system operation, the feature distribution is periodically updated incrementally using verified normal data to adapt to equipment aging and slow changes in operating conditions. This enables the model to adaptively track the slow drift of the system's normal state, maintaining long-term monitoring accuracy.

[0093] Abnormal states that deviate from the normal pattern include, but are not limited to, the following categories:

[0094] These types of anomalies are "unrecognized" by the lightweight front-end model, but can be keenly detected by the cloud-based model:

[0095] (1) Composite anomalies: These are anomalies that are novel in characterization and are caused by the superposition of multiple minor normal events under specific spatiotemporal conditions. For example, the composite fault of "belt misalignment-material spillage-idler adhesion": the belt misaligns slightly for a moment, causing a small amount of material to spill and adhere to the idler below. Individually, each event may not trigger the alarm threshold of the front-end model, but their combination visually forms a unique texture and morphological pattern that has never been individually labeled.

[0096] (2) Progressive degradation anomaly: refers to the slow deterioration of equipment performance, with minimal changes in visual characteristics in the early stages, failing to reach the preset fault threshold. For example: early corrosion and fatigue of steel wire rope at conveyor belt joints: manifested as fine brownish-red staining or extremely small cracks on the rubber surface of the joint. This gradual change pattern of texture and color is something the front-end model has not learned. Early lubrication failure of idler roller bearings: before the formation of obvious hot spots, its operating posture may exhibit imperceptible shaking, causing a specific, non-periodic fluctuation pattern in the pixel values ​​of the background area in the video.

[0097] (3) Observed or unforeseen anomalies: These refer to faults that were not included in the model training phase due to the scarcity of samples. For example: Atypical longitudinal scratches on the belt: Tear marks caused by foreign objects of special shapes (such as a broken specific tool) that are different in shape from all scratch samples in the training set. Specific types of oil stains: Leakage patterns formed on the belt frame by a certain type of lubricating oil, whose reflective properties and shapes are completely different from known water stains and coal sludge stains.

[0098] (4) Abnormal interaction between environment and equipment: refers to a unique abnormal pattern caused by the interaction between environmental factors and equipment status. For example: fault development under specific light and dust concentration: under low-angle sunlight in winter, combined with dust adhesion at specific humidity, the slight out-of-roundness fault of the idler roller presents a unique alternating light and dark stripe, a visual pattern that has never appeared under other working conditions.

[0099] Anomaly Detection Mechanism: During the online monitoring phase, the cloud-based visual foundation model extracts high-dimensional features from uploaded video frames and calculates the probability value or reconstruction error of the feature vector belonging to the aforementioned "normal state space." When the probability value is lower than a preset threshold or the reconstruction error is higher than a preset threshold, the system determines that the current state "deviations from the normal mode." This method does not require pre-defining and labeling all fault types. It can detect any anomalies that do not conform to historical normal patterns, including novel faults never seen before, complex combinations of multiple minor anomalies, etc., realizing a paradigm shift from "supervised learning" to "semi-supervised / unsupervised learning." Through incremental updates, the system becomes a "living" system, capable of "evolving" along with the device, avoiding the performance degradation problem of traditional fixed models due to device aging.

[0100] Identifying abnormal states that deviate from the normal pattern includes predictive monitoring of the health status of key components of the belt conveyor: by comparing the gradual trend of visual features of the same component in historical data, when subtle changes that conform to a preset degradation pattern are identified, a predictive maintenance alarm for the health degradation of that component is generated, rather than an emergency fault alarm.

[0101] "Key components" include, but are not limited to: Conveyor belt body: such as joint cracking, wire rope corrosion, cover rubber wear, localized fatigue damage, etc. Idler rollers: such as early atypical vibration caused by lubrication failure, slight radial runout before bearing jamming, drum wear, aging of seals, etc. Rollers: such as localized wear and peeling of the rubber coating, minor deformation of the bearing housing, dynamic and static balance deviations, etc. Drive unit: such as changes in the paint layer on the surface of the motor and reducer housing, oil leakage patterns, and other indirect characteristics indicating the internal condition.

[0102] "Preset degradation pattern" refers to the pattern of visual characteristic changes that characterize the slow deterioration of component performance, derived from historical data analysis. Examples include: a trend of increasing texture roughness (such as the gradual change in the rubber surface of a conveyor belt joint); an expansion or deepening of the range of a specific color area (such as rusted areas or oily areas); slow deformation of component geometry (such as slight bending of idler rollers or small wavy lines on belt edges); and gradual changes in thermal imaging characteristics (such as a slow upward trend in temperature at a bearing location under the same operating conditions).

[0103] For example, a conveyor belt joint: the system establishes an independent feature time series for it. By continuously monitoring the evolution of the component's feature vector (e.g., the feature value characterizing the joint's texture complexity is slowly but steadily decreasing), and applying trend analysis algorithms (such as linear regression and time series prediction models), the system can determine whether it is developing along a known degradation trajectory. When a trend conforming to a preset degradation pattern is identified, the system does not trigger an emergency shutdown alarm, but instead generates a predictive maintenance alarm, such as: "Joint J001's health is degrading; based on the current trend, it is expected to require maintenance in 30 days." This operation elevates the equipment's maintenance strategy from "reactive maintenance" and "scheduled maintenance" to the advanced stage of "condition-based maintenance," enabling early detection of potential faults, avoiding unplanned downtime, and maximizing equipment lifespan.

[0104] The deep learning model deployed on edge computing nodes serves as the student model, while the visual foundation model deployed in the cloud serves as the teacher model. There is a knowledge distillation relationship between the two: the student model improves its generalization ability and recognition accuracy by periodically synchronizing parameters or learning output features from the teacher model.

[0105] Specifically, the student model not only learns how to classify correctly, but more importantly, it learns to mimic the teacher model's high-dimensional feature representation of the data and its "soft label" output (i.e., the probability distribution for different categories, not just the final classification result). The student model's parameters are then updated to minimize the difference between its output and the teacher model's output. This allows lightweight models deployed on resource-constrained edge devices to achieve performance close to that of powerful cloud models, significantly reducing false positives and false negatives at the edge. Cloud-edge collaboration constructs an efficient "teaching and learning" closed loop, allowing the intelligence level of the entire system to flow and balance between the cloud and the edge, which is key to achieving efficient and accurate distributed intelligent monitoring.

[0106] S5. When visual analysis identifies an anomaly, it automatically retrieves and analyzes data from non-visual sensing modules within the same time period for collaborative verification. The non-visual sensing modules include distributed fiber optic vibration sensors deployed along the conveyor belt frame and an array of audio sensors deployed near key components.

[0107] Collaborative verification specifically includes: when visual analysis identifies an abnormal area, calling the data of the distributed fiber optic vibration sensor segment corresponding to that area for vibration spectrum analysis, and / or calling the data of the audio sensor array covering that area for sound source localization and abnormal sound pattern identification;

[0108] By aligning visual positioning information with position segment information from vibration sensors and sound source positioning information from acoustic sensors in a spatiotemporal manner, the overall confidence level of the anomaly diagnosis is significantly improved when the multi-source information is consistent in both space and time.

[0109] For example, when cloud-based visual analysis identifies "abnormal dust rising in area X of the drive roller at a distance of 15 meters from the drive unit," the system will not indiscriminately call all sensor data, but will perform precise spatial correlation: It will call a specific section of the distributed fiber optic vibration sensor: based on the location information of "15 meters" provided by the visual system, the system will automatically retrieve the vibration timing data of the 14-16 meter section. It will also call the coverage data of the audio sensor array A: the system will locate the audio sensor array AX deployed near the drive unit that can cover the "15-meter" area and retrieve its synchronized audio data.

[0110] Multimodal feature analysis: The system performs parallel analysis on the retrieved data to find physical evidence that corroborates visual anomalies. Vibration spectrum analysis: Fast Fourier transform is performed on the retrieved data segments to analyze their spectral components. For example, it checks for peak values ​​of the "fault characteristic frequency of the outer ring of the idler roller bearing." Sound source localization and abnormal sound identification: For the data from the audio array AX, on the one hand, the location and distance of the abnormal sound source are calculated by calculating the time difference of sound arrival at different microphones; on the other hand, it analyzes whether the audio signal contains patterns that match "dry friction noise of the idler roller" or "metallic impact sound."

[0111] Spatiotemporal alignment and decision fusion: The system establishes a unified timestamp and spatial coordinate system: Time alignment: Checks whether the moment T when visually capturing dust rising falls within the same time window (e.g., ±1 second) as the moment when the vibration characteristic frequency appears and the abnormal sound erupts. Spatial alignment: Checks whether the abnormal location segment (14-16 meters) reported by the vibration sensor and the sound source location (AX) calculated by the sound source localization system spatially overlap with the abnormal area (drive roller X) reported by the vision. Consistency judgment: The system considers the anomaly to have been "co-verified" only when the abnormal information from the visual, vibration, and acoustic sources all point to the same target in both space and time, significantly increasing its overall confidence and triggering a high-priority alarm.

[0112] This step effectively distinguishes between real faults and visual artifacts, resulting in an extremely low false alarm rate. Multimodal information not only confirms anomalies but also helps to accurately locate the fault type, providing maintenance personnel with precise repair guidance. It enables early detection of latent faults, as the visual characteristics of some faults appear later, but vibration or acoustic characteristics may appear earlier, allowing for high-confidence warnings to be issued in the early stages of a fault. When a certain type of sensor temporarily fails or is severely interfered with, the system can still rely on other sensors to make judgments, ensuring the continuity of monitoring.

[0113] S6. Based on the verification results of visual analysis and non-visual sensor data, a fusion decision is made to generate the final diagnostic and early warning information; the fusion decision adopts a confidence-weighted decision mechanism.

[0114] Assign a confidence weight Wv to the visual analysis results of S4, based on its model confidence and real-time image quality. Specifically, model confidence Cv refers to the probability value that the cloud-based visual model outputs this abnormal result. For example, if the model judges "roller damage" with a 98% probability, the Cv value is very high. Real-time image quality Qv originates from the dynamic evaluation results of step S2 (such as signal-to-noise ratio and contrast); the clearer the image, the higher the Qv value. The confidence weight Wv is not a fixed value but is determined by two dynamic factors, i.e., Wv = f(Cv, Qv). For example, when the model confidence is high but the image quality is poor (such as dirt on the lens), the system will lower Wv because a high confidence under a low-quality image may be unreliable.

[0115] Assign a confidence weight Ws to the data analysis results of the S5's non-visual sensing module based on its data quality and feature saliency. Specifically, data quality Qs refers to the signal-to-noise ratio of the vibration or audio signal. For example, excessive ambient noise will reduce the Qs of the audio data. Feature saliency Fs refers to the degree of matching between the extracted anomalous features (such as specific vibration frequencies or anomalous sound patterns) and the standard fault template. The higher the matching degree, the higher the Fs value. The confidence weight Ws is also determined by two factors, namely, Ws = f(Qs, Fs). For example, detecting a crisp metallic impact sound against a quiet background will result in high Fs and Qs, thus obtaining a high Ws.

[0116] The final decision output is a weighted sum of visual and non-visual analysis results. An alert is confirmed when the weighted overall confidence score exceeds a preset threshold. Specifically, the final decision formula is: Overall Confidence Score = (V * Wv + S * Ws) / (Wv + Ws), where V is the normalized confidence score of the visual analysis results (0 ≤ V ≤ 1), and S is the normalized confidence score of the non-visual analysis results (0 ≤ S ≤ 1). Dynamic Threshold Judgment: The calculated Score is compared with a preset threshold (e.g., 0.8). The system only confirms an alert when the Score exceeds this threshold.

[0117] The credibility-weighted decision-making mechanism offers the following advantages: It enables situational adaptation, leading to more accurate decisions. The system can perceive its own "health status" and environmental conditions, dynamically adjusting its level of trust in different information. It quantifies uncertainty, making the decision-making process interpretable. It maximizes information utilization, improving system sensitivity and robustness. Even if the confidence level of a single information source is low, as long as multiple low-confidence information points to the same conclusion, the weighted aggregation may still exceed the threshold. It establishes a fault-tolerant and robust decision-making foundation. When a sensor completely fails, its corresponding data quality (Qs or Qv) drops to extremely low levels, and its weight approaches zero. The system automatically excludes it from the decision-making process, relying instead on other functioning sensors.

[0118] This method also includes an active learning loop for unknown cases:

[0119] Once the system identifies an unknown anomaly and generates an alert, it automatically packages the multimodal data within that time period into a case-to-confirmation package. Specifically, this data package includes not only the video clip that triggered the alarm but also distributed fiber optic vibration data and audio data from the same time period. This ensures that the samples used for subsequent learning are multimodal and context-complete, providing the richest information foundation for accurate diagnosis and model learning.

[0120] After the operations and maintenance personnel confirm and annotate the case package, the system automatically includes it in an incremental learning sample library; specifically, the packaged case is pushed to the operations and maintenance personnel's interface. The operations and maintenance personnel (domain experts) then make a judgment and annotation on the case based on the video footage, vibration spectrum, sound characteristics, and possible on-site verification results. Annotation results:

[0121] Confirmed as a new fault: Mark its specific fault type (e.g., "early pitting of idler roller bearings").

[0122] If a false alarm is identified, it will be marked as a "false alarm" and the reason may be noted (such as "light and shadow interference").

[0123] Confirmed as a new normal mode: If it is a normal operating condition that has not been recorded before (such as a special material gloss), it is marked as "normal".

[0124] The cloud-based analytics platform regularly fine-tunes the visual baseline model and / or dynamic feature distribution model using an incremental learning sample library. Specifically, the system periodically (e.g., weekly or monthly) uses newly verified cases (including new positive, negative, and normal samples) accumulated in the "incremental learning sample library" as training data to incrementally learn or fine-tune the cloud-based visual baseline model and dynamic feature distribution model. An incremental learning algorithm that does not significantly forget old knowledge is employed to ensure that the model absorbs new knowledge without compromising its existing strong recognition capabilities.

[0125] This process offers the following advantages: It enables the system to "self-evolve," with a continuously expanding knowledge base that grows to cover an increasing number of fault types; it significantly reduces long-term false alarm rates, becoming more accurate with use, as each correction by maintenance personnel serves as a learning opportunity. By absorbing these negative samples, the model gradually learns to distinguish between genuine faults and easily confused interference patterns (such as specific lighting, birds, or temporary personnel passing by); it adapts to equipment aging and operational condition drift, becoming more stable with use, as the dynamic feature distribution model continuously injects new normal samples to track the slow changes in the conveyor belt caused by long-term operation (such as slight belt elongation or darkening of the overall color due to aging), updating the definition of "normal" in a timely manner and avoiding misjudging these slow changes as abnormal; it greatly reduces model update and maintenance costs, automatically collecting and preprocessing valuable data during daily operation, requiring only simple confirmation by maintenance personnel to drive automatic model optimization; and it forms a human-machine collaborative expert system, perfectly combining human domain knowledge and judgment with the tireless and powerful computing capabilities of machines.

[0126] like Figures 2-3 As shown, a visual intelligent monitoring system for belt conveyor operation is used to implement a visual intelligent monitoring method for belt conveyor operation, comprising:

[0127] The visual perception module includes one or more visual sensing devices deployed along the belt conveyor. The visual sensing devices are integrated with self-cleaning devices and controllable supplementary lighting systems to collect video streams of the belt running area in real time. Through self-cleaning and supplementary lighting, the visual perception module can actively ensure that clear video streams can be collected under any harsh working conditions (dust, water vapor, low light), laying a solid foundation for subsequent analysis.

[0128] The edge computing gateway, communicating with the vision perception module, internally stores and runs a lightweight deep learning model as a student model, and is configured as follows:

[0129] The received video stream is dynamically image quality evaluated, and the self-cleaning device, controllable supplementary lighting system, or fog-penetrating image restoration algorithm is triggered based on the evaluation results to enhance the video stream.

[0130] Run a lightweight deep learning model on the enhanced video stream to detect pre-defined typical anomalous states in real time;

[0131] The non-visual sensing module includes:

[0132] Distributed fiber optic vibration sensors, deployed along the conveyor frame, provide vibration data with location information. For example, Φ3mm tight-buffered optical cables can be used, fixed to the inside of the conveyor's longitudinal beams with clips, providing passive and continuous sensing throughout the entire process. Any vibration event can be located to a specific meter-level length range, thus directly providing the location segment information of the vibration source.

[0133] An array of audio sensors deployed near key components provides audio data with sound source localization information; for example, it consists of multiple noise-canceling microphones, mounted linearly or in an array on a rack near key components such as drive rollers, redirecting rollers, and tensioning devices. By calculating the time difference of sound reaching different microphones, spatial localization of abnormal sound sources can be achieved.

[0134] The cloud-based intelligent analysis platform communicates with the edge computing gateway and non-visual sensing modules, and is equipped with a visual basic model unit as the teacher model. This unit is used to extract high-dimensional features from uploaded video frames. The student model and the teacher model form a knowledge distillation relationship.

[0135] The cloud-based intelligent analytics platform also includes:

[0136] The dynamic feature library stores dynamic feature distribution models built based on historical normal data; the visual basic model unit compares the extracted high-dimensional features with the dynamic feature distribution model to identify unknown anomalies that are not pre-defined.

[0137] The multimodal fusion diagnostic unit is configured as follows:

[0138] When the visual basic model unit identifies an anomaly, it automatically retrieves the segment data of the distributed fiber optic vibration sensor corresponding to the anomaly area within the same time period and / or the data of the audio sensor array covering the area, performs vibration spectrum analysis and sound source localization, and performs spatiotemporal alignment and collaborative verification of the visual, vibration, and acoustic localization information.

[0139] The early warning and decision-making unit adopts a confidence-weighted decision-making mechanism to perform fusion decision-making on the output of the multimodal fusion diagnostic unit and generate diagnostic and early warning information.

[0140] The active learning engine is configured to package unknown anomaly cases into a case package to be confirmed. After receiving annotations from operations and maintenance personnel, the package is incorporated into the incremental learning sample library and used to fine-tune the visual basic model unit and / or dynamic feature library.

[0141] This system has the following advantages:

[0142] High reliability: Through multimodal spatiotemporal alignment verification and credibility-weighted decision-making, the problem of high false alarm rate in industrial field monitoring is fundamentally solved.

[0143] Full monitoring coverage: It combines real-time typical fault detection at the edge with the discovery of unknown anomalies in the cloud, ensuring both speed and breadth.

[0144] Self-evolution capability: Through knowledge distillation and active learning loop, the system has the ability to continuously optimize and adapt, with a long life cycle and low maintenance cost.

[0145] Intelligent decision-making: The decision-making process comprehensively considers data quality, model confidence, and multi-source consistency, far exceeding simple threshold judgment and more closely resembling human expert reasoning.

[0146] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0147] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0148] The present invention has been described above by way of example. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any improvements made by adopting the inventive concept and technical solution of the present invention, or direct application to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A visual intelligent monitoring method for belt conveyor operation, characterized in that, Includes the following steps: S1. Real-time video streams of the belt running area are collected by visual sensing devices deployed along the belt conveyor. S2. On the edge computing node, perform dynamic image quality assessment and enhancement processing on the video stream, and run a lightweight deep learning model on the processed video stream to detect preset typical abnormal states in real time. S3. Upload the video stream and edge processing results to the cloud analysis platform; S4. In the cloud, high-dimensional features are extracted from video frames using a pre-trained visual basic model, and the high-dimensional features are compared with a dynamic feature distribution model built on historical normal operation data to identify abnormal states that deviate from the normal pattern and are not preset in the lightweight deep learning model. S5. When visual analysis identifies an anomaly, it automatically retrieves and analyzes data from non-visual sensing modules within the same time period for collaborative verification. S6. Based on the verification results of visual analysis and non-visual sensor data, make fusion decisions and generate the final diagnostic and early warning information.

2. The visual intelligent monitoring method for belt conveyor operation according to claim 1, characterized in that, This method also includes an active learning loop for unknown cases: Once the system identifies an unknown anomaly and generates an alert, it automatically packages the multimodal data within that time period into a case package to be confirmed. After the operations and maintenance personnel confirm and label the case package, the system automatically includes it in an incremental learning sample library; The cloud-based analytics platform regularly fine-tunes the visual base model and / or dynamic feature distribution model using an incremental learning sample library.

3. The visual intelligent monitoring method for belt conveyor operation according to claim 1, characterized in that, In S2, dynamic image quality assessment and enhancement processing specifically includes: Calculate the signal-to-noise ratio and contrast of the image in real time; When any indicator falls below its preset threshold, at least one of the following actions will be automatically triggered: Activate the cleaning device integrated into the visual sensing device; Switch to a fog-penetrating image restoration algorithm based on dark channel priors; Automatically adjusts the intensity and angle of the fill light.

4. The visual intelligent monitoring method for belt conveyor operation according to claim 1, characterized in that, In S4, the process of establishing and updating the dynamic feature distribution model includes: During the system initialization phase, visual data of the belt conveyor under normal operating conditions is collected, and its feature vectors are extracted using the basic visual model to construct the initial normal feature distribution. During system operation, the feature distribution is periodically updated incrementally using verified normal data to adapt to equipment aging and slow changes in operating conditions.

5. The visual intelligent monitoring method for belt conveyor operation according to claim 1, characterized in that, Deep learning models deployed on edge computing nodes act as student models, while the visual foundational model deployed in the cloud acts as the teacher model. A knowledge distillation relationship exists between the two: the student model periodically synchronizes parameters from the teacher model or learns its output features. To improve its generalization ability and recognition accuracy.

6. The visual intelligent monitoring method for belt conveyor operation according to claim 1, characterized in that, In S4, identifying abnormal states that deviate from the normal pattern includes predictive monitoring of the health status of key components of the belt conveyor: by comparing the gradual trend of visual features of the same component in historical data, when a subtle change that conforms to a preset degradation pattern is identified, a predictive maintenance alarm for the degradation of the component's health is generated, rather than an emergency fault alarm.

7. The visual intelligent monitoring method for belt conveyor operation according to claim 1, characterized in that, In S5, the non-visual sensing module includes distributed fiber optic vibration sensors deployed along the conveyor belt frame and an array of audio sensors deployed near key components; collaborative verification specifically includes: When visual analysis identifies an abnormal area, it calls the data of the distributed fiber optic vibration sensor segment corresponding to that area for vibration spectrum analysis, and / or calls the data of the audio sensor array covering that area for sound source localization and abnormal sound pattern identification. By aligning visual positioning information with position segment information from vibration sensors and sound source positioning information from acoustic sensors in a spatiotemporal manner, the overall confidence level of the anomaly diagnosis is significantly improved when the multi-source information is consistent in both space and time.

8. The visual intelligent monitoring method for belt conveyor operation according to claim 1, characterized in that, In S6, the fusion decision-making adopts a credibility-weighted decision-making mechanism: Assign a confidence weight Wv to the visual analysis results of S4 based on its model confidence and real-time image quality; Assign a confidence weight Ws to the data analysis results of the non-visual sensing module of S5 based on its data quality and feature significance; The final decision output is a weighted sum of visual and non-visual analysis results. When the confidence level of the weighted sum exceeds a preset threshold, an alert is confirmed.

9. A visual intelligent monitoring system for belt conveyor operation, used to implement the visual intelligent monitoring method for belt conveyor operation as described in any one of claims 1-8, characterized in that, include: The visual perception module includes one or more visual sensing devices deployed along the belt conveyor. The visual sensing devices are integrated with a self-cleaning device and a controllable supplementary lighting system for real-time acquisition of video streams of the belt running area. The edge computing gateway, communicating with the vision perception module, internally stores and runs a lightweight deep learning model as a student model, and is configured as follows: The received video stream is dynamically image quality evaluated, and the self-cleaning device, controllable supplementary lighting system, or fog-penetrating image restoration algorithm is triggered based on the evaluation results to enhance the video stream. Run a lightweight deep learning model on the enhanced video stream to detect pre-defined typical anomalous states in real time; The non-visual sensing module includes: Distributed fiber optic vibration sensors are installed along the conveyor belt frame to provide vibration data with position information; An array of audio sensors deployed near key components is used to provide audio data with sound source localization information; The cloud-based intelligent analysis platform communicates with the edge computing gateway and non-visual sensing modules, and is equipped with a visual basic model unit as the teacher model. This unit is used to extract high-dimensional features from uploaded video frames. The student model and the teacher model form a knowledge distillation relationship.

10. The belt conveyor operation visual intelligent monitoring system according to claim 9, characterized in that, The cloud-based intelligent analytics platform also includes: The dynamic feature library stores dynamic feature distribution models built based on historical normal data; the visual basic model unit compares the extracted high-dimensional features with the dynamic feature distribution model to identify unknown anomalies that are not pre-defined. The multimodal fusion diagnostic unit is configured as follows: When the visual basic model unit identifies an anomaly, it automatically retrieves the segment data of the distributed fiber optic vibration sensor corresponding to the anomaly area within the same time period and / or the data of the audio sensor array covering the area, performs vibration spectrum analysis and sound source localization, and performs spatiotemporal alignment and collaborative verification of the visual, vibration, and acoustic localization information. The early warning and decision-making unit adopts a confidence-weighted decision-making mechanism to perform fusion decision-making on the output of the multimodal fusion diagnostic unit and generate diagnostic and early warning information. The active learning engine is configured to package unknown anomaly cases into a case package to be confirmed. After receiving annotations from operations and maintenance personnel, the package is incorporated into the incremental learning sample library and used to fine-tune the visual basic model unit and / or dynamic feature library.

Images