An intelligent detection system for residual coal unloading in skips based on image recognition

Abstract

This invention discloses an intelligent detection system for residual coal unloading in skips based on image recognition, comprising the following modules: a data acquisition module, using an intrinsically safe mining camera positioned at the skip unloading location to acquire unloading video sequences; a calibration and segmentation module, used to perform region calibration and frame-by-frame image segmentation to form a residual feature sequence; a temporal modeling module, including a neural oscillator network, used for rhythm modeling and phase discrimination, outputting temporal discrimination results; a fusion and judgment module, used to combine spatial parameters and temporal results to generate a fused residual score; an alarm recording module, used to trigger an alarm and generate snapshot and video recordings when residual anomalies occur; and a linkage control module, used to transmit the judgment status and output linkage signals to the hoisting system safety loop. This invention achieves accurate detection and safe control of residual coal unloading in skips through camera recognition and neural oscillator network modeling.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring in mines, and in particular to an intelligent detection system for residual coal unloading in skips based on image recognition. Background Technology

[0002] In the hoisting and transportation process of coal mine main shafts, the skip serves as the primary coal-bearing and unloading device. The efficiency and safety of coal unloading operations directly affect production efficiency and the underground working environment. However, in current production, coal residue often remains in the skip after unloading. If this residue is not removed in time, it can easily lead to a decrease in the skip's loading capacity, coal spillage, and uneven loading in the next operation. In severe cases, it can even cause the skip to jam, increase the impact load on the wire rope, and bring significant safety risks.

[0003] Currently, most coal mining enterprises still rely on manual monitoring and video inspections to monitor the unloading of coal in the skips. Inspectors visually observe the skips for residue through video feeds. However, this method relies on human experience, is highly subjective, and makes it difficult to quantify the degree of residue in a timely and accurate manner. Furthermore, the mine entrance environment is complex, with interference factors such as coal dust, fog, and strobe lights. Manual observation is prone to fatigue and missed detections, leading to the inability to promptly identify and address abnormalities.

[0004] Some mines have attempted to use image recognition or sensor technology to detect coal unloading in skips, but conventional image segmentation methods are limited by computational complexity and mine environmental noise, making it difficult to balance detection accuracy and real-time performance. At the same time, existing systems often ignore the temporal characteristics of coal unloading residues, making it impossible to effectively distinguish between short-term coal dust obscuration and continuous residue states, and they also lack a safety linkage mechanism with the hoisting system.

[0005] Therefore, how to provide an intelligent detection system for residual coal unloading in skips based on image recognition is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose an intelligent detection system for coal residue in skip unloading based on image recognition. This invention achieves precise quantification of coal residue by deploying intrinsically safe mining cameras at the skip unloading location and using lightweight image segmentation methods to obtain a mask of the residue area and extract area and height parameters. The residue feature sequence is input into a modified neural oscillator network, which uses suppression and memory units to model and correct high-frequency disturbances and low-frequency trends, effectively distinguishing between short-term occlusion and continuous residue, thus improving detection robustness and accuracy. Upon anomaly detection, the system can trigger audible and visual alarms and generate alarm records containing images and parameters. Simultaneously, it outputs linkage control signals to the hoisting system via hard contacts and industrial Ethernet dual channels to achieve hook stop or speed limit control, thereby constructing a safety closed loop integrating detection, judgment, and control, significantly reducing the safety risks posed by coal residue in skip unloading.

[0007] An intelligent detection system for residual coal unloading in a skip based on image recognition, according to an embodiment of the present invention, includes the following modules:

[0008] The acquisition module is a mine-use intrinsically safe camera installed at the coal unloading position of the skip, used to acquire video sequences of the coal unloading process;

[0009] The calibration and segmentation module is used to perform a coal unloading area calibration method on the video sequence to determine the restricted detection area, and to perform image segmentation processing frame by frame within the restricted detection area to obtain a residual area mask within the coal unloading area. Based on the residual area mask, residual area parameters and residual height parameters are calculated to form a residual feature sequence with timestamps.

[0010] The temporal modeling module includes a neural oscillator network for performing rhythmic modeling and phase discrimination on the residual feature sequence, and outputting temporal discrimination results.

[0011] The fusion and judgment module is used to fuse the residual area parameter and residual height parameter with the time-series discrimination result to obtain a fused residual score;

[0012] The alarm recording module is used to trigger an audible and visual alarm when an abnormality in coal unloading residue is detected, and to capture and record video slices of the corresponding frame to generate an alarm record;

[0013] The linkage control module is used to transmit the fusion residual score and judgment status to the ground display terminal for presentation and storage via Ethernet, and output linkage control signals to the lifting system safety control loop based on the judgment status.

[0014] Furthermore, the modules are interconnected using the following method:

[0015] A mine-use intrinsically safe camera is deployed at the coal unloading location of the skip to collect video sequences of the coal unloading process. The coal unloading area is then calibrated using a coal unloading area calibration method to determine the restricted detection area.

[0016] Lightweight image segmentation is performed frame by frame in the restricted detection area to obtain the residual area mask in the coal unloading area, and the residual area parameter and residual height parameter are calculated to form a residual feature sequence with timestamps;

[0017] The residual feature sequence is input into a neural oscillator network to perform rhythmic modeling and phase discrimination on the residual feature sequence, and outputs a time-series discrimination result containing residual stability indicators and rhythm consistency indicators;

[0018] The residual area parameter and residual height parameter are fused with the time-series discrimination result to obtain a fused residual score. When the fused residual score exceeds the alarm threshold and continues for a set time, it is determined to be an abnormal coal unloading residue. When the fused residual score is lower than the recovery threshold and continues for a set time, it is determined to be a return to normal.

[0019] When an abnormality is detected in the coal unloading residue, an audible and visual alarm is triggered, and the corresponding frame is captured and video sliced ​​to generate an alarm record.

[0020] The fusion residual score and judgment status are transmitted to the ground display terminal via Ethernet for presentation and storage. Based on the judgment status, a linkage control signal is output to the lifting system safety control loop to realize hook stop or speed limit control. When the judgment status is restored to normal, the alarm is deactivated and normal monitoring is restored.

[0021] Furthermore, the method for calibrating the coal unloading area includes:

[0022] Before performing segmentation, distortion correction, illumination normalization, and noise filtering are performed on the acquired video sequences;

[0023] The image projection boundary is calculated based on the geometric dimensions of the coal unloading port and the camera installation angle, and a rectangular or polygonal region is generated within the projection boundary as a restricted detection area.

[0024] The system identifies the positions of preset marker points within a restricted detection area and corrects the restricted detection area based on the marker point positions, brightness distribution, and device vibration during operation.

[0025] Furthermore, obtaining the residual region mask includes:

[0026] A lightweight convolutional neural network is used to classify image blocks in restricted detection areas pixel by pixel to obtain the residual probability of each pixel.

[0027] The residual probability map is thresholded to generate a binary mask;

[0028] Morphological filtering is performed on the binary mask to remove isolated noise, thereby obtaining the residual region mask.

[0029] Furthermore, the construction of the residual feature sequence includes:

[0030] The number of pixels in the residual area mask is counted, and the residual area parameters are calculated based on the pixel-physical size mapping relationship calibrated by the camera.

[0031] The pixel position difference between the highest and lowest points in the residual area mask is detected and converted into physical height through perspective transformation to obtain the residual height parameter;

[0032] The residual area parameter and the residual height parameter are bound to the data collection timestamp to form a triplet sequence arranged in chronological order, which serves as the residual feature sequence.

[0033] Furthermore, the modeling of residual feature sequences by the neural oscillator network includes:

[0034] The residual area parameters and residual height parameters arranged in chronological order are normalized and mapped to oscillator input signals. The residual area parameters are mapped to oscillator amplitude signals, the residual height parameters are mapped to oscillator frequency signals, and the centroid trajectory of the residual region mask is mapped to phase drive signals.

[0035] Suppression units and memory units are set between the coupled oscillation units to weaken high-frequency interference and enhance the preservation of low-frequency continuous characteristics, respectively.

[0036] The output combines a residual persistence score with phase synchronization, frequency difference, and stability indicators.

[0037] Furthermore, the suppression unit comprises two parts: a high-frequency disturbance detector and a variable suppression gate.

[0038] The high-frequency disturbance detector takes the brightness change rate of the restricted detection area and the motion energy of the mask boundary of the residual area as input, performs high-pass filtering on the input signal and calculates the energy within the time window to obtain the disturbance intensity index;

[0039] The variable inhibition gate generates an inhibition factor with a value of 0 to 1 based on the perturbation intensity index, and applies the inhibition factor to the coupling weights of the neural oscillator network.

[0040] The memory unit comprises two parts: a leakage integrator and a window steady-state evaluator.

[0041] The leakage integrator performs exponential sliding integral on the time series of residual area and residual height parameters, with the time constant adaptively set according to the shift work conditions.

[0042] The window steady-state evaluator calculates the phase drift variance and frequency stability within a preset time window and outputs the persistence weight.

[0043] Furthermore, the generation of the fusion residual score includes:

[0044] The residual area and residual height parameters are normalized and synthesized into a spatial score. The spatial reliability is obtained by calculating the overlap rate of the mask boundaries of the residual regions in adjacent frames and the contrast of the video frames.

[0045] The timing discrimination results are smoothed and normalized into timing scores within a time window, and the timing reliability is obtained by statistically analyzing the phase concentration and frequency change amplitude of the neural oscillator network units.

[0046] Weights are assigned based on the relative magnitudes of spatial reliability and temporal reliability, and the spatial score and temporal score are weighted and summed according to the weights to generate a fusion residual score.

[0047] Different alarm thresholds and recovery thresholds are set in the determination of the fusion residue score, and the judgment is performed in combination with the continuous frame count. When the fusion residue score continuously exceeds the alarm threshold, it is determined to be an abnormal coal unloading residue. When the fusion residue score continuously falls below the recovery threshold, it is determined to be normal.

[0048] Furthermore, the generation of the alarm record includes:

[0049] When an abnormality is detected in coal unloading residue, the video frame corresponding to the trigger time is locked and the residual area is masked and rendered to form a captured image.

[0050] Extract a video clip several seconds before and after the trigger time as the abnormal recording;

[0051] The index information of the captured images and abnormal videos, along with the residual area parameters, residual height parameters, time-series discrimination results, and fused residual scores of the corresponding frames, are stored together to form an alarm record;

[0052] The alarm log records the trigger timestamp, duration of the abnormality, and output status of the linkage control signal.

[0053] Furthermore, the output of the linkage control signal includes two methods: hard contact output and network output, wherein:

[0054] The hard-contact output drives normally open or normally closed contacts through an intrinsically safe isolation circuit, and directly connects the abnormal judgment result to the hoist safety circuit to realize hook stop or speed limit control.

[0055] The network output transmits the judgment status and fusion residual score to the upper monitoring system via industrial Ethernet using standard communication protocols, so that the dispatch center can display and archive them in real time, and use them as a redundancy check for the hard contact output.

[0056] If any output channel fails or communication is interrupted, the other channel can still independently maintain stop or speed limit control.

[0057] The beneficial effects of this invention are:

[0058] This invention utilizes an intrinsically safe mining camera deployed at the coal unloading location of the skip, combined with a lightweight image segmentation method to obtain a mask of the residual area, and calculates the residual area and height parameters. This enables precise quantification of coal unloading residues, overcoming the shortcomings of manual inspections that rely on experience and have strong subjectivity, and achieving automation and objectivity in residue detection.

[0059] This invention inputs residual feature sequences into an adapted neural oscillator network, utilizing its rhythmic modeling and phase discrimination capabilities, along with suppression and memory units, to achieve dynamic evolution modeling of residual states under complex operating conditions. This effectively distinguishes between short-term disturbances and persistent residuals, significantly improving the robustness and accuracy of the detection system.

[0060] This invention automatically triggers audible and visual alarms and generates alarm records containing images and parameters when an abnormality in coal unloading is detected. Simultaneously, it outputs linkage control signals to the hoisting system via hard contacts and industrial Ethernet dual channels to achieve hook stop or speed limit control, ensuring safe handling of abnormal coal unloading situations. This system can significantly reduce accident risks and improve the safety and reliability of mine hoisting and transportation. Attached Figure Description

[0061] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0062] Figure 1 This is a structural diagram of an intelligent detection system for residual coal unloading in a skip based on image recognition, as proposed in this invention.

[0063] Figure 2 This is a data flow diagram of alarm recording and linkage control for an intelligent detection system for residual coal unloading in a skip based on image recognition proposed in this invention. Detailed Implementation

[0064] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0065] refer to Figures 1-2 An intelligent detection system for residual coal unloading in skips based on image recognition includes the following modules:

[0066] The acquisition module is a mine-use intrinsically safe camera installed at the coal unloading position of the skip, used to acquire video sequences of the coal unloading process;

[0067] The calibration and segmentation module is used to perform a coal unloading area calibration method on the video sequence to determine the restricted detection area, and to perform image segmentation processing frame by frame within the restricted detection area to obtain a residual area mask within the coal unloading area. Based on the residual area mask, residual area parameters and residual height parameters are calculated to form a residual feature sequence with timestamps.

[0068] The temporal modeling module includes a neural oscillator network for performing rhythmic modeling and phase discrimination on the residual feature sequence, and outputting temporal discrimination results.

[0069] The fusion and judgment module is used to fuse the residual area parameter and residual height parameter with the time-series discrimination result to obtain a fused residual score;

[0070] The alarm recording module is used to trigger an audible and visual alarm when an abnormality in coal unloading residue is detected, and to capture and record video slices of the corresponding frame to generate an alarm record;

[0071] The linkage control module is used to transmit the fusion residual score and judgment status to the ground display terminal for presentation and storage via Ethernet, and output linkage control signals to the lifting system safety control loop based on the judgment status.

[0072] In this embodiment, the modules are interconnected using the following method:

[0073] A mine-use intrinsically safe camera is deployed at the coal unloading location of the skip to collect video sequences of the coal unloading process. The coal unloading area is then calibrated using a coal unloading area calibration method to determine the restricted detection area.

[0074] Lightweight image segmentation is performed frame by frame in the restricted detection area to obtain the residual area mask in the coal unloading area, and the residual area parameter and residual height parameter are calculated to form a residual feature sequence with timestamps;

[0075] The residual feature sequence is input into a neural oscillator network to perform rhythmic modeling and phase discrimination on the residual feature sequence, and outputs a time-series discrimination result containing residual stability indicators and rhythm consistency indicators;

[0076] The residual area parameter and residual height parameter are fused with the time-series discrimination result to obtain a fused residual score. When the fused residual score exceeds the alarm threshold and continues for a set time, it is determined to be an abnormal coal unloading residue. When the fused residual score is lower than the recovery threshold and continues for a set time, it is determined to be a return to normal.

[0077] When an abnormality is detected in the coal unloading residue, an audible and visual alarm is triggered, and the corresponding frame is captured and video sliced ​​to generate an alarm record.

[0078] The fusion residual score and judgment status are transmitted to the ground display terminal via Ethernet for presentation and storage. Based on the judgment status, a linkage control signal is output to the lifting system safety control loop to realize hook stop or speed limit control. When the judgment status is restored to normal, the alarm is deactivated and normal monitoring is restored.

[0079] In this embodiment, the method for calibrating the coal unloading area includes:

[0080] The acquired coal unloading video sequence was preprocessed frame by frame. Geometric distortion correction was performed on the video frames using pinhole camera model parameters to eliminate barrel distortion caused by the camera lens. Histogram equalization was used to normalize the image grayscale distribution to reduce brightness differences caused by flickering of the wellhead lights and uneven local lighting. Spatiotemporal denoising was performed on the video frames using a 3×3 median filter to filter out high-frequency noise points caused by coal dust and electromagnetic interference.

[0081] Based on the physical geometric dimensions of the coal unloading port and the fixed installation height and pitch angle of the camera, the perspective projection boundary of the coal unloading port in the image is calculated. This boundary is described using a quadrilateral, and a corresponding polygonal mask region is generated in the video frame as the only restricted detection region, ensuring that subsequent segmentation calculations are performed only within this region.

[0082] The system identifies the positions of preset marker points within a restricted detection area. In this embodiment, high-reflectivity patches are fixedly installed on both sides of the coal unloading port frame as marker points. The system detects the coordinates of the marker points in each frame using a template matching algorithm and compares their real-time positions with the initial reference positions. When a slight shift in the camera due to vibration or impact is detected, the system automatically corrects the vertex coordinates of the restricted detection area based on the marker point shift; simultaneously, it compensates for the image brightness distribution to ensure that the restricted detection area remains consistent with the actual position of the coal unloading port throughout the entire operating cycle.

[0083] In this embodiment, obtaining the residual region mask includes:

[0084] The calibrated and determined restricted detection regions are cropped from the preprocessed video frames and uniformly scaled into 256×256 pixel image blocks, which are then used as input for image segmentation.

[0085] Image patches are input into a lightweight convolutional neural network based on an improved MobileNetV3 architecture. This network is adapted and improved from the original MobileNetV3 as follows:

[0086] Input layer: Standard convolution is used instead of depthwise separable convolution to enhance the perception of low-contrast residual coal edges;

[0087] Backbone network: The inverse residual structure of MobileNetV3 is retained, but the SE attention mechanism is introduced in the intermediate layer to suppress coal dust background noise and enhance the feature response of the residual region;

[0088] Output layer: The original fully connected classification head is replaced with a pixel-wise Softmax classification head, and the output is a residual probability map with the same size as the input image.

[0089] The network outputs the residual confidence score for each pixel, ranging from 0 to 1, forming a probability map. The network outputs the confidence score value for belonging to the "residual coal" category pixel by pixel, forming a probability map of the same size as the input image.

[0090] Threshold segmentation is performed on the probabilistic map. The threshold is fixed at 0.5. Pixels larger than the threshold are identified as residual regions, and pixels smaller than the threshold are identified as background, thereby generating a binary map.

[0091] The binary image is subjected to opening and closing operations using a 3×3 convolution kernel. Isolated noise points are removed first, and then the edge shape is smoothed to obtain a continuous and smooth residual region mask. This mask is saved synchronously with the original video frame and serves as the sole basis for subsequent calculations of residual area and residual height parameters.

[0092] In this embodiment, the construction of the residual feature sequence includes:

[0093] The residual area parameters are calculated based on the residual region mask. The number of pixels in the mask is counted, and combined with the pixel-to-physical-size mapping ratio obtained during camera calibration, the pixel count is converted into the actual physical area. This conversion ratio is obtained through calibration board measurement during system installation and debugging to ensure the physical quantification accuracy of the area calculation.

[0094] In the residual area mask, the pixel row coordinates of the highest and lowest points of the residual area are detected to obtain the pixel height difference. To convert the pixel height difference into the actual height, this implementation uses a checkerboard calibration board to complete camera calibration during the system installation and debugging phase, obtaining a proportionality coefficient between the pixel coordinates and the actual physical dimensions. Specifically, the calibration board is fixed to the inner wall of the coal unloading port. Under the condition of known grid spacing of the calibration board, video frames are acquired and the pixel positions of the calibration points are extracted. The ratio of the pixel height difference to the actual height difference is calculated to determine the conversion ratio from pixel to physical height. During operation, the system directly multiplies the pixel height difference of the mask by this proportionality coefficient to obtain the residual height parameter. To improve stability, a 5-frame moving average is used to smooth the height parameter and suppress fluctuations caused by single-frame jitter.

[0095] To ensure time consistency, during the processing of each frame, the residual area parameter and the residual height parameter are simultaneously bound to the acquisition timestamp to form a triplet (area, height, time).

[0096] The triples of all consecutive frames are arranged in chronological order to form a residual feature sequence, which is then cached in a circular queue for subsequent use by the neural oscillator network. This residual feature sequence is the sole input data source for temporal modeling and can fully reflect the dynamic evolution trend of the residual state during coal unloading.

[0097] In this embodiment, the modeling of residual feature sequences by the neural oscillator network includes:

[0098] The area and height parameters in the residual feature sequence are linearly normalized. The area parameter is normalized to the 0-1 range based on the maximum capacity area of ​​the unloading port; the height parameter is normalized to the 0-1 range based on the effective height of the unloading port. The resulting area value is used as the amplitude input of the oscillator, and the height value is used as the frequency input of the oscillator.

[0099] The centroid trajectory of the residual mask is extracted, and its angular change over time is used as the phase-driving signal input to the oscillator. The oscillator network consists of multiple coupled units, with adjacent units connected by coupling weights. During operation, the phase of each unit updates over time and is affected by the amplitude, frequency, and phase drive. The network incorporates suppression units and memory units; the former is used to weaken high-frequency disturbances such as coal dust, while the latter is used to preserve the slow-changing trend of the residual material.

[0100] During the operation of the oscillator network, the phase synchronization degree is obtained by statistically analyzing the consistency of the phases of each oscillator unit. The calculation formula is as follows:

[0101]

[0102] Where N is the number of oscillating units, θ i (t) represents the phase of the i-th unit, R φ (t) represents the phase synchronization degree.

[0103] The instantaneous frequency distribution of each unit is statistically analyzed, its standard deviation is calculated and normalized to obtain the frequency stability, which reflects the overall stability of the network at the frequency level.

[0104] The memory unit performs a sliding integral on the time-series data of residual area and height to obtain a persistent weight between 0 and 1, which is used to enhance the contribution of long-term trends.

[0105] m(t)=λm(t-1)+(1-λ)max(A n (t), H n (t)),λ∈(0,1);

[0106] Where A n (t),H n(t) represents the normalized area and height, λ is the fixed leakage factor (set during the debugging phase), and m(t) is the persistence weight.

[0107] The residual persistence score is obtained by combining phase synchronization, frequency stability, and persistence weights using a geometric mean.

[0108] R c (t)=(R φ (t)·S f (t)·m(t)) 1 / 3 ;

[0109] Where R φ (t) represents the phase synchronization degree, S f (t) represents frequency stability, m(t) represents persistence weight, and R c (t) represents the residual persistence score. This score, as a temporal discrimination result, is used to determine residual anomalies after being fused with spatial features.

[0110] In this embodiment, the suppression unit includes a high-frequency disturbance detector and a variable suppression gate, which is used to reduce the impact of high-frequency interference such as dust bursts, light flickering and mechanical vibration on the modeling results of the neural oscillator network.

[0111] The high-frequency disturbance detector works by analyzing in real time the average brightness change of the restricted detection area and the displacement of the mask boundary in the residual area during coal unloading video. The system first calculates the brightness change rate and boundary motion between adjacent frames, then uses high-pass filtering to retain only rapidly changing components. Within a fixed time window, the system statistically analyzes the energy of these high-frequency components to obtain an index reflecting the disturbance intensity.

[0112] The variable suppression gate dynamically adjusts the coupling level of the neural oscillator network based on the perturbation intensity. When the perturbation intensity is high, the suppression gate reduces the interaction between oscillating units, preventing the network from exhibiting false synchronization due to short-term anomalies such as dust or flicker. When the perturbation intensity weakens, the suppression gate gradually restores the normal coupling level to ensure that the network can sensitively reflect the true residual trend.

[0113] The memory unit consists of a leakage integrator and a window steady-state evaluator, which are used to maintain the low-frequency trend of the residual state and enhance the ability to distinguish long-term stability.

[0114] The leakage integrator first compares the normalized residual area parameter and height parameter in each frame, selecting the larger value as the representative value for the current frame. Then, this representative value is weighted with the integration result from the previous time step according to a fixed ratio to obtain a new integration value. Over time, the integrator gradually forms a trend quantity between zero and one, reflecting the long-term accumulation or slow decay of the residual state. This mechanism can suppress the influence of short-term jitter or illumination interference on the results, enabling the system to maintain a stable overall trend judgment.

[0115] The window steady-state evaluator statistically analyzes the operational status of a neural oscillator network over a fixed time period, primarily checking two aspects: first, whether the phases of different oscillating units are concentrated within a small range; and second, whether the frequencies of each unit remain relatively stable during this time period. If the evaluation results show concentrated phases and small frequency fluctuations, the network is considered to be in a steady state; if the phases are dispersed and the frequency fluctuates, the network is considered unstable.

[0116] The memory unit ultimately weights and combines the trend quantity obtained from the leakage integrator with the steady-state result of the window steady-state estimator to form a persistence weight between zero and one. The larger the persistence weight, the more likely the residual state is to be a real and long-term accumulation; the smaller the weight, the more likely the residual state is caused by transient disturbances or noise.

[0117] The persistence weights, along with the inhibition factor output by the inhibition unit, are then applied to the neural oscillator network: the inhibition factor directly affects the coupling strength between oscillator units, weakening high-frequency disturbances; the persistence weights, on the other hand, affect the final residual persistence score, enhancing the contribution of long-term stable signals. By combining these two approaches, the system can stably distinguish between short-term noise and true residual noise under complex operating conditions.

[0118] In this embodiment, the generation of the fusion residue score includes:

[0119] The residual area and residual height of each frame are linearly normalized to the range of 0 to 1; the two are combined into a spatial score of a single frame at a fixed ratio (the ratio is fixed during the debugging phase and does not change with operation).

[0120] The mask overlap (i.e., the cross-union ratio of the masks in two adjacent frames) is calculated frame by frame. The average of these overlaps is taken within the decision window to obtain a boundary consistency score (0-1, with a higher value indicating a more stable boundary). Tenengrad sharpness is calculated for each frame and linearly mapped to the 0-1 range using the minimum / maximum reference values ​​determined during installation and debugging. The average of the mapping results is taken within the decision window to obtain a visibility score (0-1, with a higher value indicating greater sharpness). The spatial reliability (0-1) is obtained by taking the geometric mean of the boundary consistency score and the visibility score. The geometric mean is achieved by multiplying and then taking the square root to simultaneously satisfy the requirement that "a poor result in any component leads to an overall decrease in quality."

[0121] The residual persistence score output by the neural oscillator network is taken and a sliding average is performed within the decision window to obtain the window time series score (0~1).

[0122] The phase of each oscillating unit is converted into a unit vector (with cosine and sine components). The length of the average vector of all unit vectors is calculated. The length range is naturally between 0 and 1, which is used as the phase concentration (the closer to 1, the more consistent the phase of the group).

[0123] Calculate the instantaneous frequency of each oscillation unit within the decision window, and perform standard deviation statistics on the frequency dispersion of all units; then use the upper limit value fixed during the debugging stage for linear mapping and truncation to obtain a stability score of 0 to 1 (the smaller the dispersion, the larger the score). Take the geometric mean of "phase concentration" and "frequency stability" to obtain the timing reliability (0 to 1).

[0124] The weights are proportionally allocated to the relative magnitudes of "spatial reliability" and "temporal reliability," and normalized so that the sum of the two weights is 1 (the side with higher reliability automatically receives a larger weight). The "window spatial score" and "window temporal score" are then weighted and summed using the above weights to obtain the fusion residual score (0-1) for this window.

[0125] To avoid misjudgments caused by jitter, a dual threshold + window consistency mechanism is adopted:

[0126] Alarm triggering: If the fusion residual score of all frames is not lower than the alarm threshold within the judgment window, it is judged as "abnormal coal unloading residue".

[0127] Recovery trigger: If the fusion residual score of all frames is not higher than the recovery threshold within the judgment window, it is judged as "recovered to normal".

[0128] The alarm threshold, recovery threshold, and window length are fixed once during the debugging phase and do not change adaptively during operation.

[0129] In this embodiment, the generation of the alarm record includes:

[0130] When the fusion residual score determines that there is an anomaly in the coal unloading residue, the system immediately locks the video frame corresponding to the trigger moment and overlays a mask of the residual area onto that frame image to form a captured image with a clear indication of the residual area. At the same time, the system captures video segments a few seconds before and after the trigger moment, splices them together, saves them as an anomaly recording, and generates corresponding recording index information.

[0131] The system stores the captured image, the abnormal video recording index, the residual area parameters and residual height parameters corresponding to the trigger frame, the timing discrimination results output by the neural oscillator network, and the fused residual score together to form a complete alarm record. This alarm record also further records the trigger timestamp, the duration of the abnormality, and the output status of the linkage control signal.

[0132] All alarm records are written to the database or log file in chronological order as structured entries, ensuring subsequent retrieval and traceability. When the ground display terminal retrieves alarm records, it can simultaneously present captured images, abnormal video clips, parameter values, and linkage status, assisting staff in quickly determining the nature and severity of the anomaly, and archiving them as historical data for safety analysis and operation and maintenance optimization.

[0133] In this embodiment, the output of the linkage control signal includes two modes: hard contact output and network output. Both can be enabled simultaneously, and the other channel can independently maintain control when either channel fails.

[0134] The hard-contact output is connected to the hoist's safety circuit via an intrinsically safe isolation circuit. When the system determines an abnormality in coal unloading residue, the intrinsically safe isolation circuit drives the normally open or normally closed contact to actuate, directly writing the abnormality determination result into the hoist's safety circuit. Upon receiving a stop or speed limit signal, the safety circuit immediately executes the corresponding protective measures to ensure that the hoisting operation stops or slows down in the event of a residual abnormality. This output method features high real-time performance and low dependence on external networks, and can take effect directly in case of emergencies.

[0135] The network output is transmitted via industrial Ethernet to achieve data interaction with the upper-level monitoring system. While outputting the anomaly determination result, the system also sends the residual score, alarm level, and linkage signal status to the dispatch center in the form of data packets using standard communication protocols. The dispatch center can display the current status in real time on the interface and automatically archive and generate an operation log. The network output serves both as a redundancy check for hard-connector outputs and as a means to provide the dispatch center with richer data information.

[0136] In actual operation, when any output channel experiences a circuit failure or communication interruption, the other channel can still independently maintain stop or speed limit control, ensuring that safety control does not fail due to dependence on a single channel, thereby improving the overall reliability and fault tolerance of the system.

[0137] Example 1:

[0138] To verify the feasibility of this invention in practice, it was applied to the online monitoring and renovation project of the north and south skip coal unloading ports of a mine's main shaft, and a 7-day field test was conducted.

[0139] One intrinsically safe mining camera with a resolution of 1920×1080 and a frame rate of 25 frames per second is installed at each coal unloading point. The camera is positioned approximately 0.9 meters from the center of the unloading point, with a pitch angle set to 18°. High-reflectivity markers are also fixed on both sides of the unloading point for area correction. The camera is connected to an intrinsically safe edge computer equipped with an ARMA76×2+A55×4 processor and 4GB of memory. It is connected to the ground monitoring center via a gigabit industrial Ethernet network and simultaneously outputs hard-contact signals to the hoist safety circuit.

[0140] The system performs frame-by-frame preprocessing on the acquired video sequences, including distortion correction using pinhole model parameters, uniform illumination distribution through histogram equalization and gamma adjustment, and noise from coal dust and electromagnetic interference through 3×3 median filtering. Based on the geometric parameters of the coal unloading port and the camera installation pose, the perspective projection boundary is calculated and a quadrilateral restricted detection area is generated. Then, real-time detection of marker point coordinates is used to correct the area offset caused by vibration, thereby ensuring the stability and reliability of the detection area.

[0141] In the image segmentation stage, the restricted region is cropped into 256×256 pixel image blocks, which are then input into a lightweight convolutional neural network based on an improved MobileNetV3 architecture. This network employs standard convolutions in the input layer to enhance the perception of low-contrast edges, introduces an SE attention mechanism in the backbone network to suppress coal dust noise, and uses pixel-wise softmax classification in the output layer to generate a residual probability map of the same size as the input. The system performs thresholding and morphological filtering on the probability map to obtain a continuous and smooth residual region mask.

[0142] The system calculates residual area and height parameters based on a mask. The area parameter is obtained by converting the number of mask pixels to the camera calibration ratio. During the calibration process, a checkerboard pattern is used to measure the correspondence between pixels and physical area, with an error of approximately ±3.7%. The height parameter is calculated by detecting the pixel difference between the highest and lowest points of the mask and then combining it with the calibration ratio to convert it into physical height, with an error of approximately ±4.5%. The obtained area and height parameters are bound to a timestamp to form a continuous sequence of triples, constituting a residual feature sequence and stored in a cache.

[0143] In the temporal modeling phase, the residual area and height parameters are normalized into amplitude and frequency signals, respectively, and the mask centroid trajectory is extracted as the phase-driven signal input to the neural oscillator network. This network consists of multiple coupled units, including suppression and memory units. The suppression unit dynamically adjusts the coupling weights of the oscillator units by analyzing the high-frequency components of the brightness change rate and boundary motion, thereby reducing interference from dust bursts and light flicker. The memory unit consists of a leakage integrator and a window steady-state evaluator. The former performs sliding integrals on the changes in area and height to maintain long-term trends, while the latter evaluates phase concentration and frequency stability to generate persistence weights. After combined effects, the network outputs a residual persistence score, reflecting whether the residual exhibits long-term stable characteristics.

[0144] During the fusion and judgment phase, the system synthesizes area and height into a spatial score, and calculates spatial reliability by combining boundary overlap and image sharpness. Simultaneously, the continuous score output by the oscillator network is smoothed and used as a temporal score, and temporal reliability is calculated by combining phase concentration and frequency stability. The relative magnitudes of the two types of reliability determine the final weighting ratio, resulting in the fused residual score. When this score consistently exceeds the alarm threshold within a window, the system determines it as "abnormal coal unloading residue"; when the score consistently falls below the recovery threshold, it is determined as "returned to normal." The threshold is fixed during the debugging phase and is not adaptively adjusted during operation.

[0145] When a residual anomaly is detected, the system immediately locks the frame at the trigger moment and overlays a mask to generate a captured image. Simultaneously, it extracts 5-second video clips before and after the trigger and saves them as anomaly recordings. Alarm log entries include the captured image, recording index, residual area parameters, residual height parameters, timing discrimination results, fused residual score, trigger timestamp, anomaly duration, and linkage signal status, all uniformly stored in the database for subsequent retrieval and review.

[0146] In the linkage control stage, the system, on the one hand, drives normally open contacts through intrinsically safe isolation circuits to directly write abnormal signals into the hoist safety circuit, thereby stopping the hook or limiting speed; on the other hand, it transmits the judgment status and fusion score to the upper-level monitoring system via industrial Ethernet for real-time display and archiving by the dispatch center. When any channel fails, the other channel can still maintain control independently, ensuring safety redundancy.

[0147] Table 1 Results of the Invention and Comparative Methods

[0148]

[0149]

[0150] During a 7-day continuous on-site test, the system collected a total of 89.6 hours of video, automatically extracting frames and saving over 800,000 frames through event triggering. All 326 manually verified genuine residual events were detected. Compared with traditional single-threshold spatial judgment methods, the present invention improves the event recall rate to 97.5%, the event precision rate to 95.1%, and reduces the false alarm rate to 0.42 times per hour. The proportion of false alarms caused by dust and flicker is less than 15%, which is significantly better than the comparative methods, fully demonstrating the engineering feasibility and robustness of the present invention.

[0151] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An intelligent detection system for residual coal unloading in skips based on image recognition, characterized in that, Includes the following modules: The acquisition module is a mine-use intrinsically safe camera installed at the coal unloading position of the skip, used to acquire video sequences of the coal unloading process; The calibration and segmentation module is used to perform a coal unloading area calibration method on the video sequence to determine the restricted detection area, and to perform image segmentation processing frame by frame within the restricted detection area to obtain a residual area mask within the coal unloading area. Based on the residual area mask, residual area parameters and residual height parameters are calculated to form a residual feature sequence with timestamps. The temporal modeling module includes a neural oscillator network for performing rhythmic modeling and phase discrimination on the residual feature sequence, and outputting temporal discrimination results. The modeling of residual feature sequences by the neural oscillator network includes: The residual area parameters and residual height parameters arranged in chronological order are normalized and mapped to oscillator input signals. The residual area parameters are mapped to oscillator amplitude signals, the residual height parameters are mapped to oscillator frequency signals, and the centroid trajectory of the residual region mask is mapped to phase drive signals. Suppression units and memory units are set between the coupled oscillation units to weaken high-frequency interference and enhance the preservation of low-frequency continuous characteristics, respectively. The output combines a residual persistence score with phase synchronization, frequency difference, and stability indicators; The suppression unit comprises two parts: a high-frequency disturbance detector and a variable suppression gate. The high-frequency disturbance detector takes the brightness change rate of the restricted detection area and the motion energy of the mask boundary of the residual area as input, performs high-pass filtering on the input signal and calculates the energy within the time window to obtain the disturbance intensity index; The variable inhibition gate generates an inhibition factor with a value of 0 to 1 based on the perturbation intensity index, and applies the inhibition factor to the coupling weights of the neural oscillator network. The memory unit comprises two parts: a leakage integrator and a window steady-state evaluator. The leakage integrator performs exponential sliding integral on the time series of residual area and residual height parameters, with the time constant adaptively set according to the shift work conditions. The window steady-state evaluator calculates the phase drift variance and frequency stability within a preset time window and outputs the persistence weight; The fusion and judgment module is used to fuse the residual area parameter and residual height parameter with the time-series discrimination result to obtain a fused residual score; The alarm recording module is used to trigger an audible and visual alarm when an abnormality in coal unloading residue is detected, and to capture and record video slices of the corresponding frame to generate an alarm record; The linkage control module is used to transmit the fusion residual score and judgment status to the ground display terminal for presentation and storage via Ethernet, and output linkage control signals to the lifting system safety control loop based on the judgment status.

2. The intelligent detection system for residual coal unloading in skip based on image recognition according to claim 1, characterized in that, The modules are connected in the following way: A mine-use intrinsically safe camera is deployed at the coal unloading location of the skip to collect video sequences of the coal unloading process. The coal unloading area is then calibrated using a coal unloading area calibration method to determine the restricted detection area. Lightweight image segmentation is performed frame by frame in the restricted detection area to obtain the residual area mask in the coal unloading area, and the residual area parameter and residual height parameter are calculated to form a residual feature sequence with timestamps; The residual feature sequence is input into a neural oscillator network to perform rhythmic modeling and phase discrimination on the residual feature sequence, and outputs a time-series discrimination result containing residual stability indicators and rhythm consistency indicators; The residual area parameter and residual height parameter are fused with the time-series discrimination result to obtain a fused residual score. When the fused residual score exceeds the alarm threshold and continues for a set time, it is determined to be an abnormal coal unloading residue. When the fused residual score is lower than the recovery threshold and continues for a set time, it is determined to be a return to normal. When an abnormality is detected in the coal unloading residue, an audible and visual alarm is triggered, and the corresponding frame is captured and video sliced ​​to generate an alarm record. The fusion residual score and judgment status are transmitted to the ground display terminal via Ethernet for presentation and storage. Based on the judgment status, a linkage control signal is output to the lifting system safety control loop to realize hook stop or speed limit control. When the judgment status is restored to normal, the alarm is deactivated and normal monitoring is restored.

3. The intelligent detection system for residual coal unloading in skips based on image recognition according to claim 2, characterized in that, The method for calibrating the coal unloading area includes: Before performing segmentation, distortion correction, illumination normalization, and noise filtering are performed on the acquired video sequences; The image projection boundary is calculated based on the geometric dimensions of the coal unloading port and the camera installation angle, and a rectangular or polygonal region is generated within the projection boundary as a restricted detection area. The system identifies the positions of preset marker points within a restricted detection area and corrects the restricted detection area based on the marker point positions, brightness distribution, and device vibration during operation.

4. The intelligent detection system for residual coal unloading in skip based on image recognition according to claim 2, characterized in that, The acquisition of the residual region mask includes: A lightweight convolutional neural network is used to classify image blocks in restricted detection areas pixel by pixel to obtain the residual probability of each pixel. The residual probability is thresholded to generate a binary mask; Morphological filtering is performed on the binary mask to remove isolated noise, thereby obtaining the residual region mask.

5. The intelligent detection system for residual coal unloading in skip based on image recognition according to claim 2, characterized in that, The construction of the residual feature sequence includes: The number of pixels in the residual area mask is counted, and the residual area parameters are calculated based on the pixel-physical size mapping relationship calibrated by the camera. The pixel position difference between the highest and lowest points in the residual area mask is detected and converted into physical height through perspective transformation to obtain the residual height parameter; The residual area parameter and the residual height parameter are bound to the data collection timestamp to form a triplet sequence arranged in chronological order, which serves as the residual feature sequence.

6. The intelligent detection system for residual coal unloading in skip based on image recognition according to claim 2, characterized in that, The generation of the fusion residual score includes: The residual area and residual height parameters are normalized and synthesized into a spatial score. The spatial reliability is obtained by calculating the overlap rate of the mask boundaries of the residual regions in adjacent frames and the contrast of the video frames. The timing discrimination results are smoothed and normalized into timing scores within a time window, and the timing reliability is obtained by statistically analyzing the phase concentration and frequency change amplitude of the neural oscillator network units. Weights are assigned based on the relative magnitudes of spatial reliability and temporal reliability, and the spatial score and temporal score are weighted and summed according to the weights to generate a fusion residual score. Different alarm thresholds and recovery thresholds are set in the determination of the fusion residue score, and the judgment is performed in combination with the continuous frame count. When the fusion residue score continuously exceeds the alarm threshold, it is determined to be an abnormal coal unloading residue. When the fusion residue score continuously falls below the recovery threshold, it is determined to be normal.

7. The intelligent detection system for residual coal unloading in skip based on image recognition according to claim 2, characterized in that, The generation of the alarm record includes: When an abnormality is detected in coal unloading residue, the video frame corresponding to the trigger time is locked and the residual area is masked and rendered to form a captured image. Extract a video clip several seconds before and after the trigger time as the abnormal recording; The index information of the captured images and abnormal videos, along with the residual area parameters, residual height parameters, time-series discrimination results, and fused residual scores of the corresponding frames, are stored together to form an alarm record; The alarm log records the trigger timestamp, duration of the abnormality, and output status of the linkage control signal.

8. The intelligent detection system for residual coal unloading in skip based on image recognition according to claim 2, characterized in that, The output of the linkage control signal includes two methods: hard contact output and network output, wherein: The hard-contact output drives normally open or normally closed contacts through an intrinsically safe isolation circuit, and directly connects the abnormal judgment result to the hoist safety circuit to realize hook stop or speed limit control. The network output transmits the judgment status and fusion residual score to the upper monitoring system via industrial Ethernet using standard communication protocols, so that the dispatch center can display and archive them in real time, and use them as a redundancy check for the hard contact output. If any output channel fails or communication is interrupted, the other channel can still independently maintain stop or speed limit control.

Images