A coal-rock cutting state monitoring method and device based on energy consumption and visual collaborative perception and electronic equipment

Abstract

The application discloses a coal cutter coal and rock cutting state monitoring method and device based on CR-EVFNet and electronic equipment. The method synchronously collects energy consumption time sequence signals and cutting area image signals; through an ETR-NPLGB module, NeuralProphet is used to separate low-frequency trends, periodic disturbances and transient residuals, and energy consumption side confidence is output by LightGBM; through an ATISP-MY7 module, ISP enhancement parameters are generated, and visual side confidence and spatial position information are output by Mobile-YOLOv7; through an RGFusion module, according to the dynamic weighting of the dual-mode reliability, the coal seam, rock, gangue or coal and rock transition cutting state is output. The application can improve the monitoring accuracy and robustness under low illumination, dust and load fluctuation conditions.

Description

Technical Field

[0001] This invention relates to the fields of intelligent monitoring of coal mining machines, coal and rock identification, multimodal information fusion, time series analysis, image enhancement and target detection, and in particular to a method, device and electronic equipment for monitoring the coal and rock cutting status of coal mining machines based on energy consumption and visual collaborative perception. Background Technology

[0002] The coal mining machine is a key piece of equipment in fully mechanized mining faces, and its operating status directly affects coal mining efficiency, equipment safety, and the service life of key transmission components. In actual coal mining, the coal mining machine drum needs to continuously cut through coal seams, rocks, and interbedded rock areas. Because coal, rock, and interbedded rock media differ significantly in hardness, structure, crushing characteristics, and spatial distribution, the cutting load changes when the drum cutting teeth enter different media, leading to changes in the coal mining machine's motor energy consumption, transmission chain vibration, and overall machine operating status.

[0003] In existing coal mining machine condition monitoring methods, some schemes mainly rely on vibration signals to diagnose faults in key mechanical components such as bearings and gears. While these methods can reflect the operating status of mechanical components to a certain extent, under the complex cutting conditions of coal mining machines, sudden changes in the external coal and rock medium can also be transmitted along the transmission chain and cause abnormal vibrations. When the cutting teeth cut into rock or interbedded rock areas, the resulting impact response can easily overlap with early bearing fault characteristics, making it difficult for diagnostic methods relying solely on vibration signals to distinguish whether the abnormal impact originates from the degradation of the mechanical components themselves or from changes in the external cutting load.

[0004] To achieve the sensing of coal and rock cutting status by coal mining machines, existing technologies include monitoring schemes based on energy consumption analysis. These schemes typically collect motor current, voltage, power, or energy consumption signals, and use changes in motor load to infer the current cutting status of the coal mining machine. However, during coal mining machine operation, factors such as traction speed adjustments, periodic drum cutting, fluctuations in base power, equipment inertia, and changes in geological conditions occur. The raw energy consumption signal often contains low-frequency trend fluctuations, periodic disturbances, and transient impact components simultaneously. When the cutting teeth momentarily cut into hard rock or interbedded rock, the resulting transient energy consumption changes are easily masked by the background of base operating power and gradually changing load, leading to insufficient stability of single energy consumption monitoring methods in identifying sudden changes in the coal and rock medium.

[0005] On the other hand, machine vision-based coal and rock recognition methods can directly acquire spatial image information of the cut area, offering the advantage of intuitively reflecting the distribution and boundary positions of coal and rock. However, in underground coal mining environments, the cut area typically suffers from low illumination, dust obstruction, water mist interference, uneven lighting, and image noise, resulting in poor image quality of the acquired coal and rock. Traditional image enhancement methods usually rely on fixed parameters or manual experience for adjustment, making it difficult to adaptively adjust enhancement strategies for images with different degrees of degradation. This can easily lead to problems such as insufficient enhancement, over-enhancement, loss of detail, or noise amplification, thus affecting the accuracy of subsequent coal and rock target detection.

[0006] Furthermore, underground equipment is typically limited by intrinsic safety, explosion-proof requirements, and edge computing power and power consumption, making it difficult to directly deploy complex deep learning models on the edge computing devices of coal mining machines or fully mechanized mining faces. Therefore, how to integrate the internal energy consumption response of the coal mining machine and external visual spatial information to achieve stable monitoring of the cutting status of coal, rock, and interbedded rock, while ensuring real-time performance and deployment feasibility, is an urgent problem to be solved in the intelligent operation and fault diagnosis of coal mining machines.

[0007] In summary, the existing technology has at least the following shortcomings:

[0008] First, single vibration monitoring methods are insufficient to distinguish between sudden changes in external cutting load and internal mechanical fault characteristics;

[0009] Second, single energy consumption monitoring methods are easily affected by low-frequency load fluctuations, traction speed regulation and periodic disturbances, making it difficult to stably extract transient characteristics caused by sudden changes in coal and rock media.

[0010] Third, single visual monitoring methods are easily affected by low lighting and dusty environments underground, and the accuracy of coal and rock target recognition decreases after image quality degrades;

[0011] Fourth, existing high-complexity deep learning models are insufficient to meet the requirements of real-time performance, low power consumption, and lightweight design for downhole edge deployment;

[0012] Fifth, existing methods lack a collaborative discrimination mechanism for coal and rock cutting status that can adaptively adjust the weights of energy consumption information and visual information according to different working conditions.

[0013] Therefore, it is necessary to propose a method, device, and electronic equipment for monitoring the coal and rock cutting status of coal mining machines based on energy consumption and visual collaborative perception, so as to improve the accuracy, robustness, and engineering deployment adaptability of coal and rock cutting status identification in complex underground environments. Summary of the Invention

[0014] The purpose of this invention is to provide a method, device, and electronic equipment for monitoring the coal and rock cutting status of a coal mining machine based on energy consumption and visual collaborative perception. Specifically, it constructs a CR-EVFNet coal and rock cutting status energy consumption-visual fusion monitoring network to solve the problems of insufficient anti-interference ability of single perception mode in existing coal mining machine coal and rock cutting status monitoring methods, difficulty in stably extracting abrupt changes in coal and rock medium characteristics, decreased visual recognition accuracy in low light and dust environments, insufficient multi-source information fusion capability, and poor adaptability to underground edge deployment.

[0015] To achieve the above objectives, this invention provides a method for monitoring the coal and rock cutting status of a coal mining machine based on CR-EVFNet. The CR-EVFNet includes an ETR-NPLGB energy consumption transient residual decoupling module, an ATISP-MY7 visual adaptive enhancement recognition module, and an RGFusion reliability gating fusion module. The method includes the following steps:

[0016] The current, voltage, power or energy consumption timing signals of the coal cutting motor and / or traction motor of the coal mining machine are collected simultaneously, and the coal and rock image signals of the cutting area are collected simultaneously.

[0017] The energy consumption time series signal is divided into a sliding time window, and a time matching relationship is established between the sliding time window and the coal and rock image frame at the corresponding time.

[0018] The sliding time window is input into the ETR-NPLGB energy consumption transient residual decoupling module, and the NeuralProphet is used to perform time series decomposition on the energy consumption sequence to obtain the low-frequency trend term, periodic disturbance term, autoregressive dynamic term and transient residual term.

[0019] Extract residual energy, root mean square of residual, transient peak value, residual rate of change, impact duration and kurtosis features from the transient residual term to construct an energy consumption residual feature vector;

[0020] The energy consumption residual feature vector is input into the LightGBM state mapping model, which outputs the confidence vectors of coal seam, rock, interbedded gangue and coal-rock transition categories on the energy consumption side, and calculates the energy consumption side reliability score based on the residual impact intensity, energy consumption side category confidence and energy consumption signal noise level.

[0021] The coal and rock image corresponding to the sliding time window is input into the ATISP-MY7 visual adaptive enhancement recognition module. The ISP enhancement parameters are generated using depth-separable convolution, Swing Transformer and Cross-Transformer and adaptive enhancement is performed. Then it is input into the Mobile-YOLOv7 target detection network to obtain the visual side category confidence vector and spatial location information.

[0022] A visual reliability score is calculated based on image sharpness, low-light conditions, dust interference, image contrast, and visual detection confidence.

[0023] The energy consumption side reliability score and the vision side reliability score are input into the RGFusion reliability gating fusion module to dynamically determine the energy consumption side weight and the vision side weight. Specifically, when the vision side reliability score is lower than a first threshold and the energy consumption side reliability score is higher than a second threshold, the energy consumption side weight is increased and the vision side weight is decreased. When the vision side reliability score is not lower than the first threshold, the vision side weight is increased, and the spatial location information is used as the spatial constraint or auxiliary output information for fusion discrimination.

[0024] The monitoring results of coal and rock cutting status of the coal mining machine are obtained by fusing and judging the visual side weight, the energy consumption side weight, the visual side category confidence vector and the energy consumption side category confidence vector.

[0025] Output the coal and rock cutting status monitoring results of the coal mining machine. The coal and rock cutting status monitoring results include at least one of the following: coal seam cutting status, rock cutting status, interbedded rock cutting status, coal-rock transition cutting status, spatial location information of the coal-rock boundary area, and abnormal cutting prompt information.

[0026] This invention constructs a CR-EVFNet coal and rock cutting status energy consumption-visual fusion monitoring network by simultaneously acquiring energy consumption time-series signals and cutting area image signals during the operation of the coal mining machine. On the energy consumption side, the ETR-NPLGB module uses NeuralProphet to remove low-frequency trends and periodic disturbances, retains the transient residuals corresponding to abrupt changes in the coal and rock medium, and generates energy consumption-side category confidence vectors using LightGBM. On the visual side, the ATISP-MY7 module uses depthwise separable convolution, Swing Transformer, and Cross-Transformer to dynamically generate ISP enhancement parameters, and Mobile-YOLOv7 identifies the coal and rock regions. On the fusion side, the RGFusion module performs gated weighting based on bimodal reliability scores to obtain the coal and rock cutting status monitoring results of the coal mining machine.

[0027] Through the above methods, the present invention enables the energy consumption model, visual model and reliability fusion model to form a clear collaborative link, thereby improving the accuracy, robustness and engineering deployment adaptability of cutting status identification in coal, rock, interbedded gangue and coal-rock transition areas under complex conditions such as low illumination, dust obstruction, basic load fluctuation, traction speed regulation and periodic cutting disturbance.

[0028] The present invention also provides a coal and rock cutting status monitoring device for a coal mining machine based on energy consumption and visual collaborative perception, including a multi-source data acquisition module, an ETR-NPLGB energy consumption transient residual decoupling module, a LightGBM state mapping unit, an ATISP-MY7 visual adaptive enhancement recognition module, a Mobile-YOLOv7 target detection unit, an RGFusion reliability gating fusion module, and a result output module.

[0029] The multi-source data acquisition module is used to synchronously acquire energy consumption time-series signals and cutting area image signals during the operation of the coal mining machine, and to establish the time correspondence between the energy consumption time-series signals and the cutting area image signals.

[0030] The ETR-NPLGB energy consumption transient residual decoupling module is used to preprocess the energy consumption time series signal and construct the energy consumption time series window. NeuralProphet is used to separate the background load for each energy consumption time series window to obtain the low-frequency trend term, periodic disturbance term, autoregressive dynamic term and transient residual term, and the energy consumption residual feature vector is extracted from the transient residual term.

[0031] The LightGBM state mapping unit is used to obtain the energy consumption side coal and rock cutting state characterization result based on the energy consumption residual feature vector. The energy consumption side coal and rock cutting state characterization result includes the energy consumption side category confidence vector, and calculates the energy consumption side reliability score based on the residual impact intensity, energy consumption side category confidence, and energy consumption signal stability.

[0032] The ATISP-MY7 visual adaptive enhancement recognition module is used to evaluate the image quality of the image signal of the truncated area corresponding to the energy consumption time series window, generate ISP enhancement parameters, and perform adaptive image enhancement processing.

[0033] The Mobile-YOLOv7 target detection unit is used to identify coal and rock targets in the enhanced cut area image, and obtain the visual side category confidence vector and the spatial location information of the coal-rock boundary area, interbedded rock area or coal-rock transition area.

[0034] The RGFusion reliability gating fusion module is used to perform normalized weight calculation on the visual side reliability score and the energy consumption side reliability score, dynamically determine the visual side weight and the energy consumption side weight, and perform fusion discrimination based on the visual side category confidence vector and the energy consumption side category confidence vector.

[0035] In one embodiment, the RGFusion reliability-gated fusion module includes a reliability evaluation unit, a weight calculation unit, and a fusion decision unit. The reliability evaluation unit is used to calculate the visual-side reliability score and the energy consumption-side reliability score based on the image degradation degree and the energy consumption transient residual response, respectively. The weight calculation unit is used to normalize the reliability scores on both sides and dynamically determine the visual-side weight and the energy consumption-side weight. The fusion decision unit is used to perform fusion discrimination based on the visual-side category confidence vector, the energy consumption-side category confidence vector, the visual-side weight, and the energy consumption-side weight.

[0036] The result output module is used to output the monitoring results of the coal and rock cutting status of the coal mining machine.

[0037] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the above-mentioned method for monitoring the coal and rock cutting status of a coal mining machine based on energy consumption and visual collaborative perception.

[0038] Compared with the prior art, the present invention has at least the following beneficial effects:

[0039] First, this invention constructs a CR-EVFNet coal and rock cutting state energy consumption-visual fusion monitoring network, combining three functionally defined model modules—ETR-NPLGB, ATISP-MY7, and RGFusion—into a complete monitoring link, avoiding merely a functional description of "simple fusion of energy consumption and vision."

[0040] Second, this invention uses the ETR-NPLGB energy consumption transient residual decoupling module to separate the background load of the energy consumption time series signal, obtain the low-frequency trend term, periodic disturbance term, autoregressive dynamic term and transient residual term, and extract residual energy, residual root mean square, transient peak value, residual change rate, impact duration and kurtosis characteristics from the transient residual term, which can reduce the impact of traction speed adjustment, basic operating power change, equipment inertia and drum periodic cutting disturbance on coal and rock condition identification.

[0041] Third, this invention uses the LightGBM state mapping model to perform nonlinear mapping on the energy consumption residual feature vector, which can use gradient boosting decision trees to characterize the nonlinear relationship between the energy consumption residual features and the coal and rock cutting state under complex cutting conditions, thereby improving the energy consumption side state characterization capability.

[0042] Fourth, this invention uses the ATISP-MY7 visual adaptive enhancement recognition module to perform task-driven adaptive enhancement of the cut area image, and uses the Mobile-YOLOv7 target detection network to obtain the visual side category confidence vector and the spatial location information of the coal-rock boundary area, interbedded rock area and coal-rock transition area, thereby improving the intuitiveness and spatial positioning capability of coal-rock cutting status perception.

[0043] Fifth, this invention uses depth-separable convolution, Swing Transformer, and Cross-Transformer to dynamically generate gamma correction, white balance, color correction, multi-scale sharpening, and denoising parameters, which can mitigate the adverse effects of low light, dust, water mist, uneven illumination, and noise interference on coal and rock target identification.

[0044] Sixth, this invention uses the Mobile-YOLOv7 lightweight target detection network for coal and rock target identification, which reduces the number of model parameters and computational complexity while ensuring detection performance, and is conducive to real-time deployment in downhole edge computing devices.

[0045] Seventh, this invention dynamically determines the visual-side weights and energy-side weights through the RGFusion reliability-gated fusion module. Visual-side reliability is jointly determined by image sharpness, low-light intensity, dust interference level, image contrast, and visual detection confidence. Energy-side reliability is jointly determined by residual energy, transient peak value, impact duration, energy-side category confidence, and energy-side signal stability. When the visual modality is affected by low light or dust interference, the system increases the energy-side weights; when the visual information is clear and reliable, the system increases the visual-side weights.

[0046] Eighth, the RGFusion dynamic weight fusion of the present invention is not a fixed ratio fusion, but a normalization adjustment based on the reliability of transient residuals on the energy consumption side and the reliability of image degradation on the visual side; compared with single visual recognition, it can maintain the continuity of state discrimination in scenarios with dust obstruction, low illumination or water mist interference; compared with single energy consumption recognition, it can use visual spatial position to correct the state output in scenarios with traction speed regulation, basic power fluctuation and periodic truncation disturbance.

[0047] Ninth, the coal and rock cutting status monitoring results output by this invention can not only be used for adjusting the traction speed of the coal mining machine, adjusting the drum height, adjusting the cutting power and controlling the cutting path, but also serve as prior information on external working conditions for bearing fault diagnosis. This helps to distinguish between sudden changes in external cutting load and internal mechanical fault characteristics, thereby improving the reliability of fault diagnosis for key components of the coal mining machine. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments are briefly described below. It should be understood that the following drawings are only used to illustrate the technical solutions of the present invention and do not constitute a limitation on the scope of protection of the present invention. Where there is no conflict, the technical features of the various drawings and their corresponding embodiments can be combined with each other.

[0049] Figure 1 A flowchart of a coal mining machine coal and rock cutting status monitoring method based on CR-EVFNet provided for an embodiment of the present invention.

[0050] Figure 2 This is a structural block diagram of a coal mining machine coal and rock cutting status monitoring device based on CR-EVFNet, provided for an embodiment of the present invention.

[0051] Figure 3 A schematic diagram of the ETR-NPLGB energy consumption transient residual decoupling module and the LightGBM state mapping unit provided in the embodiments of the present invention.

[0052] Figure 4 This is a schematic diagram of the ATISP-MY7 visual adaptive enhancement recognition module and the Mobile-YOLOv7 target detection unit provided in the embodiments of the present invention.

[0053] Figure 5 This is a schematic diagram of the RGFusion reliability gating fusion module provided in an embodiment of the present invention.

[0054] Figure 6 This is a schematic diagram illustrating the output and application of coal and rock cutting status monitoring results provided in an embodiment of the present invention.

[0055] The reference numerals in the attached figures are explained as follows:

[0056] 100. Multi-source data acquisition module;

[0057] 110. Energy consumption signal acquisition unit;

[0058] 120. Image signal acquisition unit;

[0059] 130. Time synchronization unit;

[0060] 200. ETR-NPLGB energy consumption transient residual decoupling module;

[0061] 210. Energy consumption signal preprocessing unit;

[0062] 220. NeuralProphet timing decoupling unit;

[0063] 230. Transient residual feature extraction unit;

[0064] 300, LightGBM state mapping unit;

[0065] 310. Energy consumption category confidence generation unit;

[0066] 320. Energy consumption side status output unit;

[0067] 400. ATISP-MY7 Visual Adaptive Enhancement Recognition Module;

[0068] 410. Image quality evaluation unit;

[0069] 420. Image Adaptive Enhancement Unit;

[0070] 430. Parameter Interaction Unit;

[0071] 440. Image signal processing unit;

[0072] 500, Mobile-YOLOv7 target detection unit;

[0073] 510. Lightweight feature extraction unit;

[0074] 520. Target detection unit;

[0075] 530. Visual category confidence generation unit;

[0076] 600, RGFusion Reliability Gated Fusion Module;

[0077] 610. Reliability Evaluation Unit;

[0078] 620. Weight Calculation Unit;

[0079] 630. Integrated decision-making unit;

[0080] 700. Result Output Module. Detailed Implementation

[0081] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments described below are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should all fall within the scope of protection of the present invention.

[0082] It should be noted that, where there is no conflict, the various technical features in the embodiments of this invention can be combined with each other. In this invention, "CR-EVFNet" refers to an energy consumption-visual fusion monitoring network for coal and rock cutting; "ETR-NPLGB" refers to an energy consumption transient residual decoupling model composed of NeuralProphet temporal decoupling, transient residual feature extraction, and LightGBM state mapping; "ATISP-MY7" refers to a visual adaptive enhancement and recognition model composed of adaptive Transformer-ISP image enhancement and Mobile-YOLOv7 target detection; and "RGFusion" refers to a reliability fusion model that uses gating weighting based on energy consumption-side reliability scores and visual-side reliability scores.

[0083] Example 1

[0084] like Figure 1 As shown in the figure, this embodiment provides a method for monitoring the coal and rock cutting status of a coal mining machine based on CR-EVFNet. It shows the overall process of synchronous acquisition of coal mining machine energy consumption time-series signal and cutting area image signal, time correspondence processing, ETR-NPLGB energy consumption side status characterization, ATISP-MY7 visual side status characterization, RGFusion reliability gating fusion discrimination and monitoring result output.

[0085] The method includes the following steps.

[0086] S1. Synchronously acquire energy consumption timing signals and cutting area image signals during the operation of the coal mining machine.

[0087] The energy consumption timing signal includes at least one of the following: current signal, voltage signal, power signal, energy consumption signal, and load torque estimation signal of the coal mining machine traction motor and / or cutting motor. The cutting area image signal includes at least one of the following: video stream, continuous image frame, or single frame image acquired by a mining industrial camera device located near the coal mining machine body, rocker arm, drum, hydraulic support, working face support, or cutting area.

[0088] like Figure 2 As shown, it illustrates the connection relationships between the multi-source data acquisition module 100 and its energy consumption signal acquisition unit 110, image signal acquisition unit 120, time synchronization unit 130, energy consumption feature extraction module 200, energy consumption state mapping module 300, visual image enhancement module 400, coal and rock target recognition module 500, fusion discrimination module 600 and result output module 700.

[0089] During the operation of the coal mining machine, the energy consumption time sequence signal and the image signal of the cutting area are collected synchronously by the multi-source data acquisition module 100.

[0090] Specifically, the energy consumption signal acquisition unit 110 acquires the current, voltage, power, energy consumption, or estimated load torque of the coal mining machine's traction motor and / or cutting motor. The energy consumption timing signal can come from the motor driver, electrical sensors, power acquisition device, internal data interface of the coal mining machine control system, or the fully mechanized mining face monitoring system.

[0091] Image signals are acquired by the image signal acquisition unit 120 in front of the coal mining machine drum, near the drum, or in the cutting area. The image signal acquisition unit 120 can be a mining industrial camera, and its installation position can be the coal mining machine body, rocker arm, near the drum, hydraulic support, working face support, or other positions where images of the cutting area can be acquired.

[0092] The time synchronization unit 130 timestamps the energy consumption timing signal and the cropped area image signal, so that the energy consumption change within the same time window corresponds to the image state. For energy consumption signals and image signals with different sampling frequencies, synchronization can be achieved by timestamp alignment, interpolation alignment, sliding window matching, or nearest neighbor frame matching.

[0093] For example, the continuous energy consumption time series signal is divided into several energy consumption subsequences according to a preset time window, and each energy consumption subsequence is associated with one or more frames of cropped area images within the corresponding time range, thereby forming a joint sample of energy consumption and vision.

[0094] S2. Process the energy consumption time sequence signal and the cut-off area image signal to establish a time correspondence between them.

[0095] Specifically, the collected energy consumption time-series signal is preprocessed by the energy consumption signal preprocessing unit 210. The preprocessing includes at least one of outlier removal, missing value compensation, noise suppression, resampling, normalization, and sliding window segmentation.

[0096] In one implementation, the continuous energy consumption time-series signal is divided into multiple energy consumption time-series windows, and each energy consumption time-series window corresponds to one or more frames of cropped region images.

[0097] For anomalies caused by communication jitter, sensor interference, or instantaneous data acquisition abnormalities during coal mining machine operation, correction methods such as threshold judgment, median filtering, moving average, or statistical anomaly detection can be used. For missing data, linear interpolation, forward padding, backward padding, or data compensation based on adjacent periods can be used to fill in the missing data.

[0098] To mitigate the impact of different motor power ratings, operating speeds, and sampling units on the model input, the energy consumption sequence can be normalized. The normalized energy consumption sequence can be represented as:

[0099] E' = normalize(E)

[0100] Where E represents the original energy consumption time series signal, E' represents the preprocessed energy consumption time series signal, and normalize represents the normalization function.

[0101] S3. Input the sliding time window into the ETR-NPLGB energy consumption transient residual decoupling module, use NeuralProphet to perform time-series decoupling and background load separation, and extract the energy consumption residual feature vector from the transient residual term.

[0102] Specifically, the energy consumption time series signal in the energy consumption time series window is decomposed into a time series to obtain a low-frequency trend term, a periodic disturbance term, an autoregressive dynamic term, and a transient residual term.

[0103] The low-frequency trend term is used to characterize the background of slow load changes caused by the adjustment of the coal mining machine traction speed, the change of basic operating power, the slow change of geological conditions, or the inertia of the equipment; the periodic disturbance term is used to characterize the rotation of the drum, the periodic coal feeding of the cutting teeth, or the periodic load fluctuation; the autoregressive dynamic term is used to characterize the impact of short-term historical energy consumption on the current window; and the transient residual term is used to characterize the short-term impact response generated when the cutting teeth cut into the rock, interbedded rock, or the region of sudden change in coal and rock hardness.

[0104] like Figure 3 As shown, it illustrates the processing steps of obtaining the energy consumption side coal and rock cutting state characterization results after the energy consumption time series signal is decoupled by NeuralProphet, extracted by transient residual features, and mapped by LightGBM.

[0105] In this embodiment, the background load is separated from the processed energy consumption time series signal by the ETR-NPLGB energy consumption transient residual decoupling module, and the energy consumption residual feature vector is extracted from the transient residual term to obtain the feature input for energy consumption side state characterization.

[0106] During the coal cutting process, the raw energy consumption signal typically contains low-frequency trend components, periodic disturbance components, transient response components, and random disturbance components. The low-frequency trend component characterizes the background of slowly varying loads caused by adjustments in the coal mining machine's traction speed, changes in basic operating power, gradual changes in geological conditions, and equipment inertia. The periodic disturbance component characterizes the rotation of the drum, the periodic entry of the cutting teeth into the coal, and periodic load fluctuations. The transient response component characterizes the short-term impact response generated when the cutting teeth cut into rock, interbedded rock, or areas of abrupt changes in coal-rock hardness. The random disturbance component characterizes sensor noise, acquisition interference, or other random errors.

[0107] In one implementation, background load separation of energy consumption time series signals can be performed using filtering separation, wavelet decomposition, empirical mode decomposition, variational mode decomposition, time-frequency analysis, residual analysis, deep time series feature extraction, or statistical feature extraction. In a preferred embodiment, the NeuralProphet time series decoupling unit in ETR-NPLGB is used to decouple trends, periods, and residuals.

[0108] In a preferred embodiment, the energy consumption time-series signal is decomposed using the NeuralProphet time-series decoupling model, and the decomposition process is expressed as follows:

[0109] y(t)=g(t)+s(t)+h(t)+AR(t)+e(t)

[0110] Where y(t) represents the original energy consumption observation sequence of the coal mining machine, g(t) represents the low-frequency trend term, s(t) represents the periodic term, h(t) represents the sudden event term, AR(t) represents the autoregressive dynamic term, and e(t) represents the random error term.

[0111] Furthermore, the low-frequency trend term and periodic disturbance term are separated from the original energy consumption sequence, and the transient residual term is obtained based on the local mutation part in the sudden event term, autoregressive dynamic term and random error term; the transient residual feature extraction unit 230 extracts residual energy, residual root mean square, transient peak value, residual change rate, impact duration and kurtosis feature from the transient residual term.

[0112] Through the above background load separation and timing decoupling processes, the background effects caused by traction speed regulation, basic power fluctuations and drum periodic loads can be weakened from the original energy consumption signal, so that the transient energy consumption characteristics related to changes in coal and rock media can be preserved and enhanced.

[0113] S4. Input the energy consumption residual feature vector into the LightGBM state mapping model to obtain the energy consumption side coal and rock cutting state characterization results.

[0114] Specifically, the residual energy, root mean square residual, transient peak value, residual change rate, impact duration, and kurtosis characteristics are used to construct the energy consumption residual feature vector X_energy. The energy consumption residual feature vector is then input into the LightGBM state mapping model to output the energy consumption side coal and rock cutting state characterization results.

[0115] The LightGBM state mapping unit 300 takes the energy consumption residual feature vector as input and learns the nonlinear relationship between the energy consumption residual feature and the coal and rock cutting state through a gradient boosting decision tree.

[0116] During model training, the LightGBM state mapping model can employ histogram splitting, leaf-first growth, unilateral gradient sampling, and mutually exclusive feature binding strategies to balance state recognition accuracy, training efficiency, and edge deployment feasibility.

[0117] In a preferred embodiment, the LightGBM state mapping model outputs energy consumption-side category confidence vectors corresponding to coal seam cutting state, rock cutting state, interbedded rock cutting state, and coal-rock transition cutting state.

[0118] The energy consumption-side coal and rock cutting status characterization results include an energy consumption-side category confidence vector, and at least one of an energy consumption-side continuous status score or an energy consumption-side status change trend. For example, the energy consumption-side category confidence vector can be expressed as:

[0119] P_energy = F_energy(X_energy)

[0120] Where X_energy represents the energy consumption side feature vector, F_energy represents the energy consumption side state discrimination model or state mapping model, and P_energy represents the energy consumption side category confidence vector.

[0121] like Figure 4 As shown, it illustrates the process by which coal and rock images are enhanced by the ATISP-MY7 visual adaptive enhancement and recognition module, and then processed by Mobile-YOLOv7 target detection to obtain visual side category confidence vectors and spatial location information.

[0122] S5. Input the image signal of the cut-off region corresponding to the energy consumption time series window into the ATISP-MY7 visual adaptive enhancement recognition module to perform image quality evaluation and adaptive enhancement processing.

[0123] Specifically, the image signal of the cut area is evaluated based on at least one of image sharpness, low light level, dust interference level, image contrast and edge integrity, and image quality improvement processing is performed based on the image quality evaluation results.

[0124] The visual image enhancement module 400 performs image quality evaluation and improvement processing on the image signal of the cut area. In order to improve the stability of visual recognition in low-light, dusty and water mist environments underground, adaptive enhancement processing is performed on the image signal of the cut area before coal and rock target recognition.

[0125] In one embodiment, prior to coal and rock target identification, the image signal of the cut area is further subjected to image quality improvement processing. The image quality improvement processing includes at least one of brightness enhancement, white balance, color correction, detail sharpening, noise reduction, contrast enhancement, and image dehazing.

[0126] In a preferred embodiment, the image signal of the cut-off region is adaptively enhanced by the adaptive Transformer-ISP unit in the ATISP-MY7 visual adaptive enhancement recognition module to obtain the enhanced image of the cut-off region.

[0127] The adaptive Transformer-ISP unit includes a feature extraction network, a parameter interaction network, and an image signal processing module.

[0128] The feature extraction network includes a depthwise separable convolutional network and a Swing Transformer network. The depthwise separable convolutional network is used to extract shallow local texture features of the cropped region image, and the Swing Transformer network is used to extract global context features of the cropped region image. The parameter interaction network uses a Cross-Transformer Block to perform cross-attention calculation on image features and initial ISP parameters to generate or update ISP enhancement parameters suitable for the current image degradation state. The image signal processing module performs quality improvement processing on the cropped region image based on the updated ISP enhancement parameters.

[0129] The ISP enhancement parameters include at least one of gamma correction parameters, white balance parameters, color correction matrix parameters, multi-scale sharpening parameters, and noise reduction parameters.

[0130] Specifically, the image quality evaluation unit 410 evaluates the image sharpness, low-light level, dust interference level, image contrast, and edge integrity of the original truncated area image to obtain the image quality evaluation result. The image quality improvement unit 420 invokes or generates an image quality improvement strategy based on the image quality evaluation result, and may include a feature extraction network to extract local texture features and global contextual features. Local texture features include coal and rock texture, edges, fissures, blocky structures, and coal-rock boundary details; global contextual features include the overall image brightness distribution, color distribution, contrast distribution, and spatial structural relationships.

[0131] In a preferred embodiment, the feature extraction network in the image quality improvement unit 420 includes a depthwise separable convolutional network and a Swing Transformer network. The depthwise separable convolutional network is used to extract shallow local texture features of the image, while the Swing Transformer network is used to extract global contextual features of the image with lower computational overhead.

[0132] The parameter interaction unit 430 generates ISP enhancement parameters based on the local and global features. The parameter interaction unit 430 employs a Cross-Transformer Block to perform cross-attention calculations on the image features and initial ISP parameters to obtain enhancement parameters suitable for the current image degradation state. This method enables the image enhancement parameters to adaptively adjust according to the brightness, dust interference level, and texture degradation level of the current image.

[0133] The image signal processing unit 440 performs enhancement processing on the original truncated area image based on the image enhancement parameters output by the parameter interaction unit 430. Specifically, brightness adjustment can employ gamma correction to improve the brightness and contrast of low-light areas; white balance and color correction are used to improve color deviations under complex downhole lighting conditions; detail sharpening is used to enhance coal and rock boundaries, fissures, and texture details; and noise reduction processing is used to reduce the impact of dust, low light, and sensor noise on image recognition.

[0134] Through the above adaptive enhancement processing, an enhanced cropped region image that is more suitable for object detection model recognition can be obtained.

[0135] S6. Input the enhanced cut area image into the Mobile-YOLOv7 target detection network to obtain the visual representation results of the coal and rock cut status.

[0136] Specifically, the improved cut area image is input into the Mobile-YOLOv7 target detection network to identify coal seam areas, rock areas, interbedded rock areas, coal-rock transition areas, or coal-rock boundaries, and outputs a visual representation of the coal-rock cut-off state. Specifically, the Mobile-YOLOv7 target detection unit 500 performs target recognition on the enhanced cut area image to obtain a visual representation of the coal-rock cut-off state.

[0137] The visual-side coal and rock cutting state characterization result includes a visual-side category confidence vector and spatial location information of the coal-rock boundary region, interbedded rock region, or coal-rock transition region. The spatial location information includes at least one of the following: target detection box, pixel region, segmentation mask, center point position, boundary line position, or positional relationship relative to the drum cutting region.

[0138] The lightweight feature extraction unit 510 performs multi-scale feature extraction on the image of the cut area, and the target detection unit 520 detects coal, rock, interbedded rock, coal-rock transition area or coal-rock boundary area based on the extracted image features.

[0139] The Mobile-YOLOv7 target detection network includes a lightweight backbone network, a feature fusion network, and a detection head. The lightweight backbone network is used to reduce the number of model parameters and computational complexity, the feature fusion network is used to fuse coal and rock textures and boundary features at different scales, and the detection head is used to output class confidence and spatial location information.

[0140] In a preferred embodiment, the Mobile-YOLOv7 target detection network uses YOLOv7 as the basic framework for target detection and replaces the original high-complexity backbone network of YOLOv7 with MobileNetV3 basic blocks to reduce the number of model parameters and computational load, thereby improving the real-time inference capability on downhole edge devices.

[0141] The visual-side category confidence generation unit 530 outputs the visual-side coal-rock cutting state characterization result. The visual-side coal-rock cutting state characterization result may include a visual-side category confidence vector, and spatial location information of the coal-rock boundary region, interbedded rock region, or coal-rock transition region. For example, the visual-side category confidence vector can be represented as:

[0142] P_visual = F_visual(I)

[0143] Where I represents the cropped region image after image quality improvement, F_visual represents the visual target recognition model, and P_visual represents the visual category confidence vector.

[0144] S7. Input the visual side reliability score and the energy consumption side reliability score into the RGFusion reliability gating fusion module to dynamically determine the visual side weight and the energy consumption side weight.

[0145] Specifically, such as Figure 5 As shown, the RGFusion reliability gating fusion module 600 includes a reliability evaluation unit 610, a weight calculation unit 620, and a fusion decision unit 630. The reliability evaluation unit 610 is used to calculate the visual side reliability score and the energy consumption side reliability score respectively. The weight calculation unit 620 is used to normalize the reliability scores on both sides and dynamically determine the visual side weight and the energy consumption side weight. The fusion decision unit 630 is used to obtain the fusion category confidence vector based on the energy consumption side category confidence vector, the visual side category confidence vector, the energy consumption side weight, and the visual side weight, and to determine the coal cutting status monitoring result of the coal mining machine.

[0146] In this embodiment, the RGFusion reliability-gated fusion module 600 adaptively fuses and discriminates the coal and rock cutting status characterization results on the energy consumption side and the visual side to obtain the final coal and rock cutting status monitoring results of the coal mining machine. The modal reliability evaluation information includes visual image clarity, low illumination level, dust interference level, image contrast, visual detection confidence, residual energy, transient peak value, impact duration, energy consumption status discrimination confidence, energy consumption signal stability, and noise level within the energy consumption time series window.

[0147] The reliability evaluation unit 610 normalizes image clarity, low illumination level, dust interference level, image contrast, and visual detection confidence, and weights them according to preset visual reliability weighting coefficients to determine the visual reliability score R_visual; and normalizes residual energy, transient peak value, impact duration, energy consumption category confidence, energy consumption signal stability, and noise level within the energy consumption time window, and weights them according to preset energy consumption reliability weighting coefficients to determine the energy consumption reliability score R_energy.

[0148] w_energy + w_visual = 1

[0149] Where w_energy represents the energy consumption weight and w_visual represents the visual weight.

[0150] In one implementation, the weight calculation unit 620 calculates the weights on both sides according to w_energy = (R_energy+ε) / (R_energy+R_visual+2ε) and w_visual = (R_visual+ε) / (R_energy+R_visual+2ε), where ε is a preset small constant to prevent the denominator from being zero. This normalized weight calculation method increases the energy consumption side contribution when the visual side degrades while the energy consumption side residual response is reliable, and increases the visual side contribution when the visual side is clear and the detection confidence is high.

[0151] In one implementation, R_visual increases with image sharpness, image contrast, and visual detection confidence, and decreases with increasing low-light intensity and dust interference. R_energy increases with increasing residual energy, transient peak value, transient duration, and energy consumption category confidence, and decreases with increasing noise level within the energy consumption time window. When R_visual is below a first threshold and R_energy is above a second threshold, it indicates that the visual modality is significantly affected by low light intensity, dust, or occlusion, and the energy consumption transient residual response is reliable. In this case, the visual weight w_visual is reduced, and the energy consumption weight w_energy is increased, making the system rely more on the energy consumption transient response characteristics for state discrimination.

[0152] When R_visual is not lower than the first threshold, it indicates that the visual modality can reflect the spatial distribution of coal and rock relatively stably. At this time, the visual side weight w_visual is increased, and the spatial location information of the visual side is used as the spatial constraint or auxiliary output information for fusion discrimination.

[0153] S8. Perform RGFusion fusion discrimination based on the visual side weight, the energy consumption side weight, the visual side category confidence vector and the energy consumption side category confidence vector to obtain the coal and rock cutting status monitoring results of the coal mining machine.

[0154] In one implementation, the energy consumption weight, the visual weight, and the fusion discrimination result are represented as follows:

[0155] w_energy = (R_energy+ε) / (R_energy+R_visual+2ε), w_visual = (R_visual+ε) / (R_energy+R_visual+2ε), P_fusion = w_energy × P_energy + w_visual × P_visual

[0156] Where ε is a preset small constant to prevent the denominator from being zero, P_fusion represents the fusion category confidence vector, P_energy represents the energy consumption-side category confidence vector, P_visual represents the visual-side category confidence vector, w_energy represents the energy consumption-side weight, and w_visual represents the visual-side weight.

[0157] Among them, w_energy and w_visual are dynamically adjusted according to the reliability evaluation results of the visual side and the reliability evaluation results of the energy consumption side, and w_energy + w_visual = 1.

[0158] S9. Output the monitoring results of the coal and rock cutting status of the coal mining machine.

[0159] The results output module 700 outputs the coal and rock cutting status monitoring results of the coal mining machine.

[0160] The coal and rock cutting status monitoring results of the coal mining machine include at least one of the following: coal seam cutting status, rock cutting status, interbedded rock cutting status, coal-rock transition cutting status, spatial location information of the coal-rock boundary area, status confidence information, status change trend information, and abnormal cutting prompt information.

[0161] The coal and rock cutting status monitoring results of the coal mining machine are used as prior information of external working conditions in coal mining machine traction speed adjustment, drum height adjustment, cutting power adjustment, cutting path control, fully mechanized mining face monitoring, or bearing fault diagnosis.

[0162] The current cutting status can be categorized as coal seam cutting, rock cutting, interbedded rock cutting, or coal-rock transition cutting. The spatial location of the coal-rock boundary can help determine whether the current cutting area of ​​the drum is close to rock or interbedded rock. The trend of status change can reflect whether there is a sudden change in the cutting medium of the coal mining machine.

[0163] Furthermore, the coal and rock cutting status monitoring results can be transmitted to the coal mining machine control system, the fully mechanized mining face monitoring system, or the fault diagnosis system. The coal mining machine control system can adjust the traction speed, drum height, cutting power, or cutting path based on the coal and rock cutting status monitoring results; the fault diagnosis system can use the coal and rock cutting status monitoring results as prior information on external operating conditions to distinguish between sudden changes in external cutting load and internal mechanical fault characteristics.

[0164] like Figure 6 As shown, it illustrates how the monitoring results of coal and rock cutting status are output to the coal mining machine control system, the fully mechanized mining face monitoring system, or the fault diagnosis system, and used as prior information of external working conditions for traction speed adjustment, drum height adjustment, cutting path control, or bearing fault diagnosis.

[0165] Example 2

[0166] like Figure 2 As shown, this embodiment also provides a coal and rock cutting status monitoring device for a coal mining machine based on CR-EVFNet. The device includes a multi-source data acquisition module 100, an ETR-NPLGB energy consumption transient residual decoupling module 200, a LightGBM state mapping unit 300, an ATISP-MY7 visual adaptive enhancement recognition module 400, a Mobile-YOLOv7 target detection unit 500, an RGFusion reliability gating fusion module 600, and a result output module 700.

[0167] The multi-source data acquisition module 100 is used to synchronously acquire energy consumption timing signals and cutting area image signals during the operation of the coal mining machine. The multi-source data acquisition module 100 may include an energy consumption signal acquisition unit 110, an image signal acquisition unit 120, and a time synchronization unit 130.

[0168] The energy consumption signal acquisition unit 110 is used to acquire the current, voltage, power, energy consumption, or load torque estimate of the traction motor and / or cutting motor of the coal mining machine. The image signal acquisition unit 120 is used to acquire image signals in front of the drum, near the drum, or in the cutting area. The time synchronization unit 130 is used to time-stamp the energy consumption timing signal and the cutting area image signal, and to establish a time correspondence between the energy consumption timing window and the corresponding cutting area image frame or image sequence.

[0169] The ETR-NPLGB energy consumption transient residual decoupling module 200 is used to separate the background load from the processed energy consumption time series signal and extract the energy consumption residual feature vector to obtain the feature input for energy consumption side state characterization, and send the energy consumption residual feature vector to the LightGBM state mapping unit 300.

[0170] In one embodiment, the ETR-NPLGB energy consumption transient residual decoupling module 200 includes an energy consumption signal preprocessing unit 210, a NeuralProphet time-series decoupling unit 220, and a transient residual feature extraction unit 230. The energy consumption signal preprocessing unit 210 performs outlier removal, missing value compensation, noise suppression, resampling, normalization, and sliding window segmentation on the energy consumption time-series signal. The NeuralProphet time-series decoupling unit 220 obtains a low-frequency trend term, a periodic perturbation term, an autoregressive dynamic term, and a transient residual term. The transient residual feature extraction unit 230 extracts the energy consumption residual feature vector from the transient residual term.

[0171] The LightGBM state mapping unit 300 includes an energy consumption-side category confidence generation unit 310 and an energy consumption-side state output unit 320. The energy consumption-side category confidence generation unit 310 is used to generate an energy consumption-side category confidence vector based on the energy consumption residual feature vector; the energy consumption-side state output unit 320 is used to output the energy consumption-side coal and rock cutting state characterization results, which include the energy consumption-side category confidence vector, the energy consumption-side continuous state score, or the energy consumption-side state change trend.

[0172] The ATISP-MY7 Visual Adaptive Enhancement Recognition Module 400 is used to evaluate the image quality of the image signal in the truncated area and generate ISP enhancement parameters to improve the stability of visual recognition in low-light, dusty and water mist environments.

[0173] In one embodiment, the ATISP-MY7 visual adaptive enhancement recognition module 400 includes an image quality evaluation unit 410, an image adaptive enhancement unit 420, a parameter interaction unit 430, and an image signal processing unit 440. Specifically, the image quality evaluation unit 410 evaluates image sharpness, low-light performance, dust interference, image contrast, and edge integrity; the image adaptive enhancement unit 420 extracts local texture features and global spatial features; the parameter interaction unit 430 generates ISP enhancement parameters suitable for the current image degradation state using the Cross-Transformer Block; and the image signal processing unit 440 performs quality improvement processing on the cropped area image based on the ISP enhancement parameters.

[0174] The Mobile-YOLOv7 object detection unit 500 includes a lightweight feature extraction unit 510, an object detection unit 520, and a visual-side category confidence generation unit 530. The lightweight feature extraction unit 510 is used to extract multi-scale features from the enhanced truncated region image; the object detection unit 520 is used to identify coal, rock, interbedded rock, coal-rock transition regions, or coal-rock boundary regions; the visual-side category confidence generation unit 530 is used to output the visual-side category confidence vector and the spatial location information of the coal-rock boundary region, interbedded rock region, or coal-rock transition region.

[0175] The RGFusion reliability-gated fusion module 600 includes a reliability evaluation unit 610, a weight calculation unit 620, and a fusion decision unit 630. The reliability evaluation unit 610 is used to determine the visual-side reliability score and the energy consumption-side reliability score based on at least one modal reliability evaluation information.

[0176] The weight calculation unit 620 is used to perform normalized weight calculation on the visual side reliability score and the energy consumption side reliability score to dynamically determine the energy consumption side weight and the visual side weight. Specifically, when the visual side reliability score is lower than the first threshold and the energy consumption side reliability score is higher than the second threshold, the visual side weight is reduced and the energy consumption side weight is increased; when the visual side reliability score is not lower than the first threshold, the visual side weight is increased.

[0177] The fusion decision unit 630 is used to obtain the fusion category confidence vector based on the energy consumption side category confidence vector, the visual side category confidence vector, the energy consumption side weight, and the visual side weight, and to determine the coal and rock cutting status category of the coal mining machine based on the fusion category confidence vector. At the same time, it sends the spatial location information of the coal and rock boundary area, the interbedded rock area, or the coal and rock transition area as auxiliary output information to the result output module 700.

[0178] The result output module 700 is used to output the monitoring results of coal and rock cutting status of the coal mining machine. The result output module 700 can be connected to the coal mining machine control system, the fully mechanized mining face monitoring system, edge computing equipment, fault diagnosis system or human-machine interaction terminal, and is used to output the coal and rock cutting status category, coal and rock boundary location, status confidence level, status change trend or abnormal cutting prompt information.

[0179] The modules in this embodiment can be implemented through software programs, or through hardware circuits, embedded processors, industrial computers, edge computing boxes, or combinations thereof. The modules can exchange data via bus, industrial Ethernet, serial port, CAN communication, wireless communication, or other industrial communication interfaces.

[0180] Example 3

[0181] This embodiment provides a specific application method.

[0182] During normal operation of the coal mining machine, the system collects energy consumption data of the traction motor and the cutting motor at a preset sampling frequency, and simultaneously collects images of the drum cutting area at a preset frame rate. The system first segments the energy consumption data into sliding windows and performs time matching on the image frames, so that each energy consumption time window corresponds to at least one frame of the cutting area image.

[0183] Subsequently, the system inputs the current energy consumption window into the ETR-NPLGB energy consumption transient residual decoupling module, strips away the low-frequency trend and periodic load background, extracts transient residual features, and obtains the energy consumption side coal and rock state confidence through the LightGBM state mapping model.

[0184] Simultaneously, the system inputs the current cut area image into the ATISP-MY7 visual adaptive enhancement and recognition module for image quality evaluation and adaptive enhancement processing. Then, the enhanced cut area image is input into the Mobile-YOLOv7 target detection network to obtain the visual-side category confidence vector, the spatial location of the coal-rock boundary region, and the detection confidence.

[0185] If the current image has obvious dust obstruction, low brightness, or low detection confidence, the RGFusion reliability-gated fusion module reduces the visual side weight and increases the energy consumption side weight; if the current image is clear and the target detection confidence is high, the RGFusion reliability-gated fusion module increases the visual side weight and combines the position of the visual detection box to determine the coal-rock boundary area.

[0186] Finally, the system outputs the current coal and rock cutting status of the coal mining machine. For example, when the system determines that the current cutting status is gradually changing from coal seam to rock or interbedded rock, it can send a prompt message to the coal mining machine control system to assist in adjusting the traction speed, drum height, or cutting power. At the same time, this status result can be transmitted to the bearing fault diagnosis module to interpret whether the abnormal impact in the vibration signal comes from a sudden change in the external cutting medium.

[0187] Example 4

[0188] This embodiment also provides an electronic device. The electronic device includes a processor and a memory, the memory storing a computer program. When the computer program is executed by the processor, it implements the CR-EVFNet-based coal mining machine coal and rock cutting status monitoring method described in any of the above embodiments.

[0189] The electronic device may be a coal mining machine controller, a mining edge computing device, an industrial computer, an embedded processor, a fully mechanized mining face monitoring device, or a server. The processor may be at least one of a central processing unit, a graphics processing unit, a digital signal processor, a microcontroller, a field-programmable gate array, or a neural network accelerator processor.

[0190] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in conjunction with the above embodiments, those skilled in the art should understand that modifications can be made to the technical solutions described in the above embodiments, or equivalent substitutions can be made to some of the technical features; these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the present invention, and should all be included within the scope of protection of the present invention.

Claims

1. A method for monitoring the coal and rock cutting status of a coal mining machine based on energy consumption and visual collaborative perception, characterized in that: A CR-EVFNet coal and rock cutting state energy consumption-visual fusion monitoring network is constructed. The CR-EVFNet includes an ETR-NPLGB energy consumption transient residual decoupling module, an ATISP-MY7 visual adaptive enhancement recognition module, and an RGFusion reliability gating fusion module. The method includes the following steps: The current, voltage, power or energy consumption timing signals of the coal mining machine's cutting motor and / or traction motor are collected simultaneously, as well as the coal and rock image signals of the cutting area are collected simultaneously. The energy consumption time series signal is divided into a sliding time window, and a time matching relationship is established between the sliding time window and the coal and rock image frame at the corresponding time. The sliding time window is input into the ETR-NPLGB energy consumption transient residual decoupling module, and the NeuralProphet is used to perform time series decomposition on the energy consumption sequence to obtain the low-frequency trend term, periodic disturbance term, autoregressive dynamic term and transient residual term. Extract residual energy, root mean square of residual, transient peak value, residual rate of change, impact duration and kurtosis features from the transient residual term to construct an energy consumption residual feature vector; The energy consumption residual feature vector is input into the LightGBM state mapping model, which outputs the confidence vectors of coal seam, rock, interbedded gangue and coal-rock transition categories on the energy consumption side, and calculates the energy consumption side reliability score based on the residual impact intensity, energy consumption side category confidence and energy consumption signal noise level. The coal and rock image corresponding to the sliding time window is input into the ATISP-MY7 visual adaptive enhancement recognition module, and the local texture features and global spatial features of the coal and rock image are extracted using depth-separable convolution and Swing Transformer. Cross-Transformer is used to perform cross-attention calculation on image features and initial ISP parameters to generate gamma correction parameters, white balance parameters, color correction matrix, multi-scale sharpening parameters and denoising parameters, and adaptive enhancement of coal and rock images is performed based on the ISP parameters. The enhanced coal and rock image is input into the Mobile-YOLOv7 target detection network, which outputs the confidence vectors of the coal seam, rock, interbedded rock, and coal-rock transition categories on the visual side, as well as the spatial location information of the coal-rock boundary region, interbedded rock region, or coal-rock transition region. The energy consumption side reliability score and the visual side reliability score are input into the RGFusion reliability gating fusion module to dynamically calculate the energy consumption side fusion weight and the visual side fusion weight, and the energy consumption side category confidence vector and the visual side category confidence vector are weighted and fused to obtain the coal and rock cutting status monitoring results of the coal mining machine. Output the coal and rock cutting status monitoring results of the coal mining machine. The monitoring results include at least one of the following: coal seam cutting status, rock cutting status, interbedded rock cutting status, coal-rock transition cutting status, spatial location information of the coal-rock boundary area, and abnormal cutting prompt information.

2. The method for monitoring the coal and rock cutting status of a coal mining machine according to claim 1, characterized in that: The energy consumption timing signal includes at least one of the following: current signal, voltage signal, power signal, energy consumption signal, and load torque estimation signal of the coal mining machine traction motor and / or cutting motor; The image signal of the cutting area includes at least one of the following: video stream, continuous image frame or single frame image acquired by a mining industrial camera device located near the coal mining machine body, rocker arm, drum, hydraulic support, working face support or cutting area. The preprocessing and construction of the energy consumption time series window includes performing at least one of the following on the energy consumption time series signal: outlier removal, missing value compensation, noise suppression, resampling, normalization, and sliding window segmentation. Each energy consumption time series window is made to correspond to one or more frames of cropped region images by means of timestamp alignment, interpolation alignment, sliding window matching, or nearest neighbor frame matching.

3. The method for monitoring the coal and rock cutting status of a coal mining machine according to claim 1, characterized in that: The ETR-NPLGB energy consumption transient residual decoupling module includes a NeuralProphet time-series decoupling unit, a residual feature extraction unit, and a LightGBM state mapping unit. The time-series decomposition of the energy consumption sequence using NeuralProphet includes: The energy consumption time series signal in the energy consumption time series window is decomposed into low-frequency trend term, periodic disturbance term, autoregressive dynamic term and transient residual term; The low-frequency trend term is used to characterize the background of slow load caused by the adjustment of the coal mining machine traction speed, the change of basic operating power, the slow change of geological conditions, or the inertia of the equipment. The periodic disturbance term is used to characterize the rotation of the drum, the periodic coal feeding of the cutting teeth, or the periodic load fluctuation. The transient residual term is used to characterize the short-time impact response generated when the cutting teeth cut into the rock, interbedded rock, or the area of ​​sudden change in coal and rock hardness. The energy consumption time series signal is decomposed using the NeuralProphet time series decoupling model. The decomposition process is as follows: y(t)=g(t)+s(t)+h(t)+AR(t)+e(t) Where y(t) represents the original energy consumption observation sequence of the coal mining machine, g(t) represents the low-frequency trend term, s(t) represents the periodic term, h(t) represents the sudden event term, AR(t) represents the autoregressive dynamic term, and e(t) represents the random error term; The low-frequency trend term and the periodic disturbance term are separated from the original energy consumption sequence, and the transient residual term is obtained based on the local mutation part in the sudden event term, the autoregressive dynamic term, and the random error term.

4. The method for monitoring the coal and rock cutting status of a coal mining machine according to claim 1, characterized in that: The step of inputting the energy consumption residual feature vector into the LightGBM state mapping model includes: The residual energy, root mean square residual, transient peak value, residual rate of change, impact duration, and kurtosis feature are combined to form the energy consumption residual feature vector X_energy, which is then input into the LightGBM state mapping model. The LightGBM state mapping model performs nonlinear mapping on the energy consumption residual feature vector based on gradient boosting decision tree, histogram splitting, leaf-first growth and unilateral gradient sampling. The LightGBM state mapping model outputs an energy consumption-side category confidence vector P_energy, which corresponds to the coal seam cutting state, rock cutting state, interbedded rock cutting state, and coal-rock transition cutting state.

5. The method for monitoring the coal and rock cutting status of a coal mining machine according to claim 1, characterized in that: The ATISP-MY7 visual adaptive enhancement and recognition module includes an adaptive Transformer-ISP unit and a Mobile-YOLOv7 target detection unit; the adaptive Transformer-ISP unit performs image enhancement including: The image signal of the cut area is evaluated based on at least one of the following: image sharpness, low light level, dust interference level, image contrast, and edge integrity. Depth-separable convolution is used to extract coal-rock boundary edges, local textures and fracture details, and SwinTransformer is used to extract global context features corresponding to low illumination, dust occlusion and spatial distribution of coal and rock. Cross-Transformer Block is used to perform cross-attention calculation on image features and initial ISP parameters to dynamically generate image signal processing parameters suitable for the current image degradation state. The image signal processing parameters include at least one of gamma correction parameters, white balance parameters, color correction matrix parameters, multi-scale sharpening parameters, and denoising parameters.

6. The method for monitoring the coal and rock cutting status of a coal mining machine according to claim 1, characterized in that: The Mobile-YOLOv7 target detection network includes a lightweight backbone network, a feature fusion network, and a detection head. The coal and rock target identification includes: The adaptively enhanced coal and rock images are input into the Mobile-YOLOv7 target detection network to identify coal seam regions, rock regions, interbedded rock regions, coal-rock transition regions, or coal-rock boundary regions. Output the visual characterization results of coal and rock cutting status; The visual coal and rock cutting state characterization results include the visual category confidence vector P_visual, and the spatial location information of the coal and rock boundary area, the interbedded rock area, or the coal and rock transition area. The spatial location information includes at least one of the following: target detection box, pixel region, segmentation mask, center point position, boundary line position, or positional relationship relative to the drum cutting area; The lightweight backbone network replaces the original, highly complex YOLOv7 backbone network with MobileNetV3 basic blocks to reduce the number of parameters and computational complexity while meeting the real-time deployment requirements of downhole edge devices.

7. The method for monitoring the coal and rock cutting status of a coal mining machine according to claim 1, characterized in that: The reliability score calculation method of the RGFusion reliability-gated fusion module includes: The image clarity, low light level, dust interference level, image contrast and visual detection confidence of the cropped area image are normalized and weighted according to the preset visual reliability weight coefficient to determine the visual reliability score R_visual. The residual energy, transient peak value, impact duration, energy consumption category confidence, energy consumption signal stability, and noise level within the energy consumption time window of the transient residual term are normalized and weighted according to the preset energy consumption reliability weight coefficient to determine the energy consumption reliability score R_energy. Among them, R_visual increases with the increase of image sharpness, image contrast and visual detection confidence, and decreases with the increase of low light intensity and dust interference; R_energy increases with the increase of residual energy, transient peak, impact duration and energy consumption category confidence, and decreases with the increase of noise level within the energy consumption time window. When R_visual is lower than the first threshold and R_energy is higher than the second threshold, the weight of the visual side is reduced and the weight of the energy side is increased; when R_visual is not lower than the first threshold, the weight of the visual side is increased, and the spatial location information of the coal-rock boundary area, interbedded rock area or coal-rock transition area obtained by the visual side is used to assist in the judgment of the coal-rock cutting state.

8. The method for monitoring the coal and rock cutting status of a coal mining machine according to claim 1, characterized in that: The fusion discrimination of the RGFusion reliability gating fusion module includes: The energy consumption side coal and rock cutting state characterization result is represented as the energy consumption side category confidence vector P_energy, and the category discrimination result in the visual side coal and rock cutting state characterization result is represented as the visual side category confidence vector P_visual. Based on the energy consumption side reliability evaluation result R_energy and the vision side reliability evaluation result R_visual, normalized weights are calculated to determine the energy consumption side weight w_energy and the vision side weight w_visual. The energy consumption-side weights and the visual-side weights are determined as follows, and the fusion category confidence vector P_fusion is obtained: w_energy = (R_energy+ε) / (R_energy+R_visual+2ε), w_visual = (R_visual+ε) / (R_energy+R_visual+2ε), P_fusion = w_energy × P_energy + w_visual × P_visual Where ε is a preset small constant to prevent the denominator from being zero, P_energy, P_visual and P_fusion represent the category confidence vectors corresponding to the coal seam cutting state, rock cutting state, interbedded rock cutting state and coal-rock transition cutting state, respectively, and w_energy and w_visual satisfy w_energy + w_visual = 1; The coal mining machine's coal and rock cutting status category is determined based on the fusion category confidence vector P_fusion, and the spatial location information of the coal and rock boundary area, interbedded rock area, or coal and rock transition area in the visual side coal and rock cutting status characterization result is used as auxiliary output information. The coal cutting status monitoring results of the coal mining machine are used for at least one of the following: adjustment of the coal mining machine traction speed, adjustment of the drum height, adjustment of the cutting power, control of the cutting path, or monitoring of the fully mechanized mining face.

9. A coal mining machine coal and rock cutting status monitoring device based on energy consumption and visual collaborative perception, characterized in that: This includes the CR-EVFNet coal and rock cutting energy consumption-visual fusion monitoring network, wherein the CR-EVFNet includes: The multi-source data acquisition module is used to synchronously acquire energy consumption time-series signals and cutting area image signals during the operation of the coal mining machine, and to establish a time correspondence between the energy consumption time-series signals and the image frames or image sequences in the cutting area image signals; The ETR-NPLGB energy consumption transient residual decoupling module is used to preprocess the energy consumption time series signal and construct an energy consumption time series window. NeuralProphet is used to perform time series decoupling and background load separation on each energy consumption time series window, and to extract the energy consumption residual feature vector from the transient residual term. The LightGBM state mapping unit is used to generate an energy consumption-side category confidence vector based on the energy consumption residual feature vector. The ATISP-MY7 visual adaptive enhancement recognition module is used to generate ISP enhancement parameters through the adaptive Transformer-ISP unit and adaptively enhance the image of the truncated region. The Mobile-YOLOv7 target detection unit is used to identify coal and rock targets in the enhanced coal and rock images and obtain visual side category confidence vectors and spatial location information. The RGFusion reliability gating fusion module is used to calculate the reliability score on the energy consumption side and the reliability score on the visual side. It dynamically determines the weights on the energy consumption side and the visual side based on the reliability scores on both sides, and performs fusion discrimination based on the weights on the energy consumption side, the weights on the visual side, the confidence vector of the energy consumption side category, and the confidence vector of the visual side category to obtain the monitoring results of the coal cutting status of the coal mining machine. The result output module is used to output the monitoring results of the coal and rock cutting status of the coal mining machine; The RGFusion reliability gating fusion module includes a reliability evaluation unit, a weight calculation unit, and a fusion decision unit. The reliability evaluation unit is used to calculate the visual reliability score and the energy consumption reliability score respectively. The weight calculation unit is used to dynamically determine the visual weight and the energy consumption weight according to the normalized weight calculation method. The fusion decision unit is used to obtain the fusion category confidence vector and determine the coal and rock cutting status category of the coal mining machine.

10. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the coal and rock cutting status monitoring method of any one of claims 1 to 8.

Images