Multi-mode working condition visual image recognition system for coal mining machine

Abstract

The present application relates to the field of intelligent mine and computer vision automatic identification technology, specifically to a multi-modal working condition visual image recognition system of a coal mining machine, comprising: an image sample generation module, configured to generate simulated interference images based on received reference images of ground coal mining machines in an offline configuration phase, and train a working condition discrimination program; an image filtering and restoration module, configured to receive real-time video streams output by underground cameras, and dynamically adjust filtering parameters; an image feature extraction module, configured to perform channel weighting and region weighting on restored images, and extract working condition feature data of the coal mining machine; a working condition state classification module, comprising a working condition discrimination program, configured to receive the working condition feature data of the coal mining machine and output a recognition score, and feed back the recognition score of the current frame and the recognition score fluctuation characteristics of continuous multiple frames to the image filtering and restoration module; the present application improves the feature extraction quality in harsh environments.

Description

Technical Field

[0001] This invention relates to the field of intelligent mining and computer vision automated recognition technology, specifically a multimodal working condition visual image recognition system for coal mining machines. Background Technology

[0002] With the continuous development of intelligent coal mining technology, monitoring the operating conditions of coal mining machines in underground fully mechanized mining faces has become increasingly important. This complex underground working environment is often filled with dust and water mist with concentrations exceeding the preset upper limit and electromagnetic interference with intensity exceeding the preset threshold, which brings many challenges to the extraction and identification of working condition characteristics.

[0003] Currently, monitoring the operating conditions of coal mining machines generally relies on conventional underground video monitoring equipment. Technicians mainly determine the current operating status of the coal mining machine by observing video footage or using conventional image processing techniques. However, traditional monitoring and identification methods often fail to effectively extract clear outlines and key features of the coal mining machine when faced with diffuse dust obstruction, local coal dust clumps, lens contamination, and random electromagnetic pulse noise. Although these methods can complete basic image acquisition, their anti-interference capabilities are usually lower than the preset standards. When the operating conditions of the coal mining machine are dynamically switched, the probability of misjudgment due to interference from single frames is relatively high, making it difficult to output identification results in real time and stably.

[0004] Therefore, how to accurately and stably identify the operating status of coal mining machines in complex underground environments with strong interference has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-modal visual image recognition system for coal mining machines, avoiding misjudgments caused by single-frame interference from dust, water mist, and electromagnetic fields. Furthermore, it is more effective in accurately and stably identifying the operating status of coal mining machines in complex underground environments with strong interference. To achieve the above objectives, this invention specifically adopts the following technical solutions:

[0006] The image sample generation module is used to generate simulated interference images based on the received ground coal mining machine reference images during the offline configuration phase, and to mix the simulated interference images with the acquired real underground images to form a sample set for training the working condition discrimination program.

[0007] The image filtering and restoration module is used to receive the real-time video stream from the downhole camera, perform hybrid filtering on the current frame image, output the restored image, and dynamically adjust the filtering parameters according to the calculated image quality assessment value and the received recognition score fluctuation characteristics.

[0008] The image feature extraction module is connected to the image filtering and restoration module. It is used to perform channel weighting and region weighting on the restored image to extract coal mining machine operating feature data.

[0009] The working condition classification module includes a working condition discrimination program, which receives the working condition feature data of the coal mining machine and outputs the recognition score. It dynamically determines the current judgment threshold by combining the current traction speed and the statistical features of historical recognition scores to output the working condition category result, and feeds back the recognition score of the current frame and the recognition score fluctuation features of multiple consecutive frames to the image filtering and restoration module.

[0010] Optionally, the specific process of the image filtering and restoration module performing hybrid filtering is as follows:

[0011] Obtain multiple pixel sub-blocks from the input image;

[0012] Calculate the local gray-level variance within each pixel sub-block as an evaluation metric for local noise intensity.

[0013] The local noise intensity evaluation index is compared with the low noise threshold and the high noise threshold respectively. The low noise threshold is less than the high noise threshold. The low noise threshold is the product of the mean gray-level variance of the historical M frames of background images without device motion and the first coefficient. The high noise threshold is the product of the mean gray-level variance and the second coefficient. The first coefficient is less than the second coefficient. M is a preset positive integer.

[0014] When the local noise intensity evaluation index is less than the low noise threshold, the mean filtered pixel value is used as the filtered output pixel value.

[0015] When the local noise intensity evaluation index is greater than the high noise threshold, the median filtered pixel value is used as the filtered output pixel value.

[0016] When the local noise intensity assessment index meets When, the weights of the mean filtering result Weights of the median filtering result The calculation formula is as follows:

[0017] ;

[0018] ;

[0019] in, To achieve a low noise threshold, For high noise threshold, It serves as an evaluation index for local noise intensity; and the sum of the weights of the mean filtering result and the median filtering result is 1.

[0020] Optionally, the specific process of adjusting the filtering parameters in the image filtering and restoration module is as follows:

[0021] The image quality assessment value of the restored image is obtained, and the standard deviation of the recognition scores of N consecutive frames is used as the recognition score fluctuation value, where N is a preset positive integer;

[0022] When the average image quality assessment value of the most recent N consecutive frames is lower than the preset quality threshold and the recognition score fluctuation value is higher than the stability threshold, the window size of the filter kernel in the hybrid filtering is increased, and the weight of the median filtering result in the weighted combination is increased relative to the previous frame.

[0023] Optionally, the specific process of the image feature extraction module extracting feature data is as follows:

[0024] The basic feature matrix is ​​obtained by extracting features from the restored image. The channel weight values ​​are adjusted according to the frequency domain interference intensity of each channel and the degree of matching with the standard reference of the coal mining machine to obtain the weighted feature matrix.

[0025] Obtain the key region map of the weighted feature matrix in the channel dimension, and divide the key region map into multiple grid regions;

[0026] Calculate the regional coefficient for each grid region. When the regional coefficient is greater than the preset occlusion threshold, assign the first weight to the corresponding region. When the regional coefficient is less than or equal to the preset occlusion threshold, assign the second weight to the corresponding region. This is done to perform spatial regional weighting on the weighted feature matrix and output the coal mining machine operating condition feature data.

[0027] Optionally, the specific process by which the operating condition classification module determines the current judgment threshold and output category is as follows:

[0028] Calculate the moving average and standard deviation of the historical recognition score sequence for the current candidate category;

[0029] The operating condition fluctuation adjustment coefficient is determined based on the current traction speed of the coal mining machine;

[0030] Based on the combined calculation of the moving average, standard deviation, and operating condition fluctuation adjustment coefficient, the current judgment threshold for the current frame is determined, wherein the current judgment threshold satisfies:

[0031] ;

[0032] in, This is the current judgment threshold. It is a moving average. Standard deviation, This is the operating condition fluctuation adjustment coefficient;

[0033] When the recognition score of the current frame is greater than or equal to the current judgment threshold, the corresponding working condition category result is output; otherwise, the historical stable working condition is output, which is the mode of the output categories in the most recent N frames, where N is a preset positive integer.

[0034] Optionally, the operating condition classification module also includes a multi-frame status verification process, specifically:

[0035] Compare the first working condition category with the highest recognition score in the current frame with the second working condition category, which is a historically stable working condition;

[0036] When the two are inconsistent, determine whether the maximum recognition score of the first working condition category is greater than the sum of the current judgment threshold and the preset fixed judgment margin;

[0037] If yes, output the first working condition category; otherwise, output the second working condition category and generate a category mutation marker to be fed back to the image filtering and restoration module to trigger the adjustment of the filter kernel size.

[0038] Optionally, the configuration process for the image sample generation module includes:

[0039] The differences in interference distribution between the simulated interference image and the actual noisy image in the mine, the differences in image content between the simulated interference image and the reference image of the ground coal mining machine, and the working condition discrimination deviation of the initial working condition discrimination program were obtained respectively.

[0040] Based on the differences in interference distribution, image content, and working condition discrimination bias, the image generation parameters of dust occlusion density, noise amplitude, and blur intensity are jointly updated, and the working condition discrimination program is retrained based on the updated sample set.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. This invention effectively suppresses local coal dust clumps and random electromagnetic pulse noise by implementing dynamic hybrid filtering and adaptive weighting of channels and regions; the system automatically reduces the weight of severely interfered channels and high dust-covered areas, and highlights the associated features of the coal mining machine; this solves the problem that traditional methods cannot extract clear contours under diffuse dust and lens contamination, and improves the feature extraction quality in harsh environments;

[0043] 2. This invention combines the current traction speed of the coal mining machine with the statistical characteristics of historical identification scores to dynamically adjust the judgment threshold for the classification of working conditions; in conjunction with the multi-frame status verification process, the system can verify the status by combining historical stable working conditions when a sudden change in a single frame is detected; this effectively solves the problem that traditional monitoring is easily affected by single-frame interference when the working conditions of the coal mining machine are dynamically switched, thus ensuring the stability and accuracy of the real-time output of working conditions.

[0044] 3. In the offline stage, this invention generates simulated interference images containing dust and noise based on reference images, and mixes them with real downhole images to form a sample set for training; by jointly reducing the differences in interference distribution, image content, and working condition discrimination bias, the image generation parameters are continuously updated; this fundamentally overcomes the weakness of conventional methods in anti-interference ability, and makes the trained working condition discrimination program highly match the real downhole degradation environment. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof; in the drawings:

[0046] Figure 1 This is a schematic diagram of the modules of the multimodal working condition visual image recognition system for coal mining machines provided in the embodiments of this application. Detailed Implementation

[0047] To make the above features and advantages of the present invention more apparent and understandable, exemplary embodiments are described below in detail with reference to the accompanying drawings.

[0048] A multi-modal working condition visual image recognition system for coal mining machines includes:

[0049] The image sample generation module is used to generate simulated interference images based on the received ground coal mining machine reference images during the offline configuration phase, and to mix the simulated interference images with the acquired real underground images to form a sample set for training the working condition discrimination program.

[0050] The image filtering and restoration module is used to receive the real-time video stream from the downhole camera, perform hybrid filtering on the current frame image, output the restored image, and dynamically adjust the filtering parameters according to the calculated image quality assessment value and the received recognition score fluctuation characteristics.

[0051] The image feature extraction module is connected to the image filtering and restoration module. It is used to perform channel weighting and region weighting on the restored image to extract coal mining machine operating feature data.

[0052] The working condition classification module includes a working condition discrimination program, which receives coal mining machine working condition feature data and outputs recognition scores. It dynamically determines the current judgment threshold by combining the current traction speed and historical recognition score statistical features to output the working condition category result, and feeds back the current frame recognition score and the recognition score fluctuation features of multiple consecutive frames to the image filtering and restoration module.

[0053] The specific process of the image filtering and restoration module performing hybrid filtering is as follows:

[0054] Obtain multiple pixel sub-blocks from the input image;

[0055] Calculate the local gray-level variance within each pixel sub-block as an evaluation metric for local noise intensity.

[0056] The local noise intensity evaluation index is compared with the low noise threshold and the high noise threshold respectively. The low noise threshold is less than the high noise threshold. The low noise threshold is the product of the mean gray-level variance of the historical M frames of background images without device motion and the first coefficient. The high noise threshold is the product of the mean gray-level variance and the second coefficient. The first coefficient is less than the second coefficient. M is a preset positive integer.

[0057] When the local noise intensity evaluation index is less than the low noise threshold, the mean filtered pixel value is used as the filtered output pixel value.

[0058] When the local noise intensity evaluation index is greater than the high noise threshold, the median filtered pixel value is used as the filtered output pixel value.

[0059] When the local noise intensity assessment index meets When, the weights of the mean filtering result Weights of the median filtering result The calculation formula is as follows:

[0060] ;

[0061] ;

[0062] in, To achieve a low noise threshold, For high noise threshold, It serves as an evaluation index for local noise intensity; and the sum of the weights of the mean filtering result and the median filtering result is 1.

[0063] The specific process of adjusting the filtering parameters in the image filtering and restoration module is as follows:

[0064] The image quality assessment value of the restored image is obtained, and the standard deviation of the recognition scores of N consecutive frames is used as the recognition score fluctuation value, where N is a preset positive integer;

[0065] When the average image quality assessment value of the most recent N consecutive frames is lower than the preset quality threshold and the recognition score fluctuation value is higher than the stability threshold, the window size of the filter kernel in the hybrid filtering is increased, and the weight of the median filtering result in the weighted combination is increased relative to the previous frame.

[0066] The identification system in this embodiment is set up in the underground operation scenario of the coal mining machine, such as... Figure 1 As shown, the underground camera in the system acts as the image acquisition entity, continuously outputting a real-time video stream containing the coal mining machine body, drum, coal wall, and surrounding dust; the image filtering and restoration module is deployed in the image processing unit connected to the underground camera;

[0067] The image feature extraction module and the working condition classification module are deployed on the same computing platform or in an industrial computing device connected to the aforementioned image processing unit; the coal mining machine control system provides the current traction speed; the image sample generation module is used in the offline training phase and does not participate in the direct processing of real-time underground video;

[0068] In this embodiment, the reference image of the surface coal mining machine refers to the image of the coal mining machine collected under conditions of weak interference, in which the outline of the coal mining machine, working parts and working posture are relatively clear, and it is used as the source image; the real underground image refers to the image actually collected underground, which contains dust, water mist, electromagnetic interference and local blur.

[0069] Simulated interference images refer to training images obtained by adding underground degradation features to the reference images of ground coal mining machines through the image sample generation module; this ensures that subsequent training inputs simultaneously include structurally clear but interference-simulated images and real underground images, thereby matching the trained anti-interference image feature extraction program and working condition discrimination program with the actual underground distribution.

[0070] During the offline configuration phase, the image sample generation module first receives a reference image of the surface coal mining machine and a real image of the underground mine. Based on the requirement to preserve the structural features of the coal mining machine components, the image sample generation module uses the reference image as its content base and employs a specific pixel degradation mapping function to superimpose dust occlusion, noise amplitude, and blur intensity onto the image. This degradation mapping function is as follows:

[0071] in, For the generated simulated interference image pixels, The reference image pixels for the ground coal mining machine are both measured in grayscale. This is a binary occlusion mask matrix in the [0,1] interval generated by the dust occlusion density. The kernel standard deviation is equal to the fuzzy intensity. Gaussian filter operator, The preset dust grayscale constant, This indicates that the amplitude is equal to the preset noise amplitude. The random impulse noise distribution;

[0072] Dust masking is used to simulate the regional masking caused by diffuse dust and local coal dust clouds underground; electromagnetic noise is used to simulate pulsed bright and dark spots, stripes, and random grayscale disturbances in images under equipment operating conditions; blur intensity is used to simulate the decrease in sharpness caused by lens contamination and scattering.

[0073] After the simulated interference image is generated, the image sample generation module combines the simulated interference image with the real downhole image according to the preset mixing ratio to form a sample set. The sample set is then input into the training computing device to train the anti-interference image feature extraction program and the working condition discrimination program.

[0074] The anti-interference image feature extraction program in this embodiment adopts a residual backbone network structure with the fully connected layers removed, such as the ResNet-50 deep residual network. Its input is a red-green-blue image matrix with the size adjusted to a fixed resolution. This matrix is ​​specifically represented as an input feature tensor with dimensions [3,224,224], which contains physical pixel values ​​that reflect the working form of the coal mining machine, the outline of the coal wall, and the ambient light and shadow.

[0075] After channel weighting and spatial aggregation in the network layers, the output is a multi-channel basic feature tensor. The dimension of this output feature matrix is ​​set to [2048, 7, 7]. The working condition discrimination program includes a global average pooling layer and a single-class fully connected layer. The input is the above-mentioned basic feature matrix of [2048, 7, 7]. After global pooling, it is mapped to a one-dimensional feature vector of length 2048. The output dimension is equal to the number of preset working condition categories. For example, if there are 4 preset working conditions, the output is a 1×4 prediction score vector.

[0076] During the training phase, the cross-entropy loss function is used to calculate the error loss between the recognition score distribution output by the working condition discrimination program and the actual working condition label, and the network parameters of the program are updated through the backpropagation algorithm; after the training is completed, the program parameters are fixed into the downhole operating equipment for use in the working phase.

[0077] During the working phase, the downhole camera outputs a real-time video stream from the downhole, and the image filtering and restoration module reads the current frame image frame by frame; the current frame image enters the hybrid filtering process, and then enters the image restoration process;

[0078] Image restoration processing employs unsharpened masking edge enhancement logic: the restored image is obtained using the unsharpened masking edge enhancement formula.

[0079] Blurry image extraction:

[0080] ;

[0081] High-frequency detail extraction:

[0082] ;

[0083] Enhanced detail restoration:

[0084] ;

[0085] in, The image after blending filtering is located at pixel coordinates. The gray value at the location is expressed in gray levels.

[0086] It is a Gaussian smoothing filter operator;

[0087] is the pixel grayscale value of the blurred image after Gaussian smoothing, with the dimension of grayscale level;

[0088] The extracted high-frequency detail information pixel values ​​are expressed in grayscale.

[0089] The preset enhancement coefficient is a fixed constant of 1.5 in this embodiment, which is a dimensionless pure number.

[0090] The output pixel grayscale value of the enhanced restored image is consistent with the input in terms of dimension, which is grayscale level; thus completing the sharpening enhancement and detail restoration, making the coal wall edge, drum outline and machine body component connection more easy to extract features in the subsequent process; after the image filtering and restoration module outputs the restored image, it sends the restored image to the image feature extraction module.

[0091] After receiving the restored image, the image feature extraction module first extracts basic features by the image feature extraction unit, and then performs channel weighting and region weighting on the basic features. Channel weighting refers to assigning different weights to different feature channels to reduce the channel response that is more affected by electromagnetic interference and retain the channel response that is related to the coal mining machine's operating conditions.

[0092] Region weighting refers to assigning different weights to different regions in an image, giving higher weight to the working area of ​​the coal mining machine, the coal wall contact area, and the area that is not severely obscured, while reducing the weight of areas that are severely obscured by dust and have low information contribution. After weighting, the image feature extraction module outputs the working condition feature data of the coal mining machine and sends the working condition feature data to the working condition classification module.

[0093] After receiving the coal mining machine's working condition feature data, the working condition discrimination program in the working condition classification module outputs the recognition score for each preset working condition category. The preset working condition categories include at least cutting, traveling, coal loading, and standby. When calculating the current working condition category, the classification module does not directly use a fixed threshold, but instead reads the current traction speed in the coal mining machine control system and reads the historical recognition score statistics in the system cache.

[0094] Based on the system characteristic that the data refresh cycle of the coal mining machine control system is greater than the frame rate cycle of the video stream from the underground camera, the classification module sets up a time-aligned cache queue and uses a nearest neighbor timestamp matching strategy to extract the traction speed data that is closest to the acquisition time of the current frame image from the cache queue as the current traction speed, so as to eliminate the risk of data asynchrony caused by physical communication delay.

[0095] The time-aligned cache queue is configured as a fixed-depth first-in-first-out circular queue. Each data node in the queue contains at least a timestamp field and a traction speed field. When the nearest neighbor timestamp matching strategy is executed, the classification module traverses all valid nodes in the cache queue, calculates the absolute difference between the acquisition timestamp of the current frame image and the timestamp field value in each data node, and selects the data node with the smallest absolute difference, and uses the traction speed recorded in it as the current traction speed.

[0096] Historical recognition score statistics include the average score and dispersion of candidate categories across several consecutive frames; the classification module adjusts the threshold based on the rate of change in working conditions reflected by the traction speed.

[0097] When the fluctuation of the traction speed exceeds the preset fluctuation threshold, the current working condition fluctuation adjustment coefficient is switched to a lower value; when the traction speed is less than the preset speed threshold or the fluctuation of the working condition is lower than the preset fluctuation threshold, the current working condition fluctuation adjustment coefficient is switched to a higher value to update the current judgment threshold in a coordinated manner. This can avoid misjudgment caused by dust obstruction in a single frame, and also avoid the threshold being too high when the working condition is actually switched, which would cause a lag in the switching.

[0098] After obtaining the maximum category recognition score of the current frame, the classification module further calculates the standard deviation of the recognition scores of N consecutive frames, and feeds back the maximum category recognition score of the current frame and the standard deviation of the recognition scores of N consecutive frames to the image filtering and restoration module; N is a preset positive integer, preferably a positive integer less than the preset value, in order to balance the stability and real-time performance of recognition.

[0099] For the hybrid filtering process, in this embodiment, it is executed by the filtering calculation unit in the image filtering and restoration module; the input image is divided into multiple pixel sub-blocks according to a preset grid, and each pixel sub-block contains a group of adjacent pixels; since underground interference has regional differences when it is not uniformly distributed across the entire image, for example, the dust is heavier near the drum coal throwing area, and local electromagnetic noise may only appear in local grid areas, so it is necessary to judge the local noise intensity by block;

[0100] After the division is completed, the filtering calculation unit calculates the local gray-level variance for each pixel sub-block as an evaluation index of local noise intensity. The local gray-level variance reflects the degree of dispersion of pixel gray-level changes within the sub-block: when the variance is less than the first variance threshold, it indicates that the gray-level change is gradual and usually dominated by diffuse noise; when the variance is greater than the second variance threshold, it indicates that there are abrupt points or abnormal pulses with gray-level gradients greater than the preset threshold, and it is more likely to contain electromagnetic pulse interference.

[0101] The filtering calculation unit compares the local noise intensity evaluation index with the low noise threshold and the high noise threshold respectively; the low noise threshold and the high noise threshold are not fixed constants in advance, but are determined based on the mean gray-level variance of the historical M-frame background images without device motion.

[0102] Background images without equipment motion refer to background frames captured when the main components of the coal mining machine have no obvious displacement and there is no large mechanical movement in the image. They are used to characterize the basic noise level of the underground environment. In the calculation, the mean of the actual grayscale variance of the historical M frames of background images without equipment motion is first obtained.

[0103] When the lens is completely obscured by dust, forming a single grayscale background that results in a mean actual grayscale variance of 0, to avoid subsequent calculation errors caused by the high noise threshold being equal to the low noise threshold, the system uses the following formula to calculate the low noise threshold and the high noise threshold:

[0104] Bottom line cutoff correction:

[0105] ;

[0106] Low noise threshold calculation:

[0107] ;

[0108] High noise threshold calculation:

[0109] ;

[0110] in, For history The mean of the actual local grayscale variance of a frame of background image without device motion, with the dimension being the square of the grayscale level;

[0111] For a preset, very small positive number, such as This is used to prevent extremely solid color backgrounds from causing the threshold difference to be 0, thus triggering a division-by-zero error; its dimension must be consistent with the variance, which is the square of the gray level;

[0112] This is the corrected final mean of the grayscale variance, with the dimension being the square of the grayscale level;

[0113] As the first coefficient, It is the second coefficient, and All of them are dimensionless pure numbers;

[0114] and The dimensions of the low noise threshold and high noise threshold obtained are the square of the gray level; since the first coefficient is explicitly limited to be less than the second coefficient, this bottom line constraint mathematically guarantees that the low noise threshold is always less than the high noise threshold.

[0115] The values ​​of the first and second coefficients can be set according to the strength of the downhole background disturbance. Generally, the first coefficient corresponds to the boundary of mild interference, and the second coefficient corresponds to the boundary of strong interference. Those skilled in the art can use historical downhole sample statistics to make the first coefficient fall within the range that makes most sub-blocks of the background image judged as low noise, and make the second coefficient fall within the range that makes most abnormal sub-blocks judged as high noise under the conditions of cutting dust and strong electromagnetic interference.

[0116] When the local noise intensity assessment index is less than the low noise threshold, the filtering calculation unit uses the mean-filtered pixel value as the output pixel value. In low noise conditions, the image is mainly affected by random disturbances, and mean filtering is more conducive to smoothing small fluctuations without excessively destroying the regional structure.

[0117] When the local noise intensity assessment index is greater than the high noise threshold, the filtering calculation unit uses the median filtered pixel value as the output pixel value. High variance usually corresponds to sudden bright spots, dark spots, or impulse interference. Median filtering is more effective in suppressing isolated outliers. The specific weighting formula and weighted output logic are as follows:

[0118] Mean filter weights:

[0119] ;

[0120] Median filter weights:

[0121] ;

[0122] Weighted filter output:

[0123] ;

[0124] in, This is the local noise intensity evaluation index for the current pixel sub-block, representing the local gray-level variance, with the dimension being the square of the gray level.

[0125] and These are the low noise threshold and the high noise threshold, with the dimension being the square of the gray level;

[0126] and Since the dimensions of the numerator and denominator are both squares of gray levels, their dimensions cancel each other out after division. Therefore, both are dimensionless pure numbers and satisfy the following condition. ;

[0127] coordinates within the sub-block The mean filtering result at the location, This is the result of median filtering, with the dimension being gray levels;

[0128] The weighted and fused filtered output pixel value is expressed in grayscale. The closer the local noise intensity is to the high noise threshold, the greater the contribution of median filtering.

[0129] The closer to the low noise threshold, the greater the contribution of mean filtering. Therefore, the output pixel value can change continuously with the local noise level and will not form abrupt changes between adjacent sub-blocks. Finally, the filtered outputs of each pixel sub-block are reconstructed into a whole frame of filtered image and then sent to the subsequent image restoration processing unit.

[0130] In this embodiment, the process of adjusting the filtering parameters based on feedback from the subsequent stage is completed by the quality evaluation unit and the parameter adjustment unit in the image filtering and restoration module. After image restoration is completed, the quality evaluation unit calculates the image quality assessment value using the following image sharpness evaluation function. :

[0131] ;

[0132] in, To restore the height of the image, The width of the restored image is measured in pixels.

[0133] To restore the image in coordinates The horizontal grayscale gradient at that location; To restore the image in coordinates The vertical grayscale gradient at a given location, with dimensions of gray levels per pixel;

[0134] The grayscale gradient magnitude of a single pixel is calculated, with the dimension being grayscale level / pixel;

[0135] This is an image quality assessment value, representing the spatial average level of grayscale gradient magnitude of all pixels in the entire image, with the dimension of grayscale level / pixel;

[0136] Image quality assessment values ​​reflect the overall clarity of an image: if the average grayscale gradient is greater than the first gradient threshold, it usually indicates that the edges are clear and the outlines of the parts are well defined; if the average grayscale gradient is less than the second gradient threshold, it usually indicates that the image is affected by blurring, dust occlusion, or excessive smoothing and exceeds the preset assessment lower limit.

[0137] Within the same system cycle, the image filtering and restoration module receives the maximum category recognition score of the current frame from the working condition classification module, and also receives the standard deviation of the recognition scores of N consecutive frames. The larger the standard deviation of the recognition scores of N consecutive frames, the more obvious the fluctuation of the classification result and the more unstable the current image processing result.

[0138] The parameter adjustment unit averages the image quality evaluation values ​​of the most recent N consecutive frames to obtain the average level of image sharpness in recent times; it reads the recognition score fluctuation value of the current frame and compares it with the preset quality threshold and stability threshold respectively; the preset quality threshold is set by historical sample statistics and represents the minimum level of sharpness that the system can accept; the stability threshold is set by historical running data and represents the boundary where the recognition result changes from a stable state to obvious fluctuation.

[0139] When the average image quality assessment value of the most recent N consecutive frames is lower than the preset quality threshold and the current frame recognition score fluctuation value is higher than the stability threshold, the parameter adjustment unit determines that the current filtering intensity has not met the preset image restoration requirements. It has neither restored the image quality to a usable level nor affected the stability of the working condition recognition.

[0140] Therefore, the parameter adjustment unit adjusts the window size of the mean filter kernel and the median filter kernel from the first kernel size to the second kernel size, where the second kernel size is larger than the first kernel size, in order to expand the local statistical range; at the same time, the weight of the median filter result is increased by a preset increment relative to the previous frame, which is specifically set to a fixed value of 0.1 in this embodiment, with a weight upper limit of 1.0;

[0141] Since sudden strong interference usually manifests as significant pulse-type outliers and local violent fluctuations, increasing the filter kernel and increasing the median filter ratio helps to reduce the impact of strong interference on the image. After the adjustment, the new filter parameters will be applied to the input frame after the pipeline calculation delay is completed, or applied after alignment through the closed-loop buffer queue.

[0142] In a typical work scenario, the underground camera captures images of a significant increase in dust near the coal mining machine drum. After the image filtering and restoration module calculates the local grayscale variance of the current frame block by block, it will find that some sub-blocks have moved from the low noise region to the medium-high noise region, which causes the weight of the median filtering result corresponding to the sub-block to increase accordingly.

[0143] If the clarity of the restored image remains low for several consecutive frames, and the maximum recognition score output by the working condition classification module fluctuates significantly, the parameter adjustment unit further increases the filter kernel size and continues to increase the proportion of median filtering.

[0144] After this processing, abnormal noise and local noise in the restored image will be suppressed, and the stability of the coal mining machine outline and coal wall edge will be improved, thus making the subsequent working condition classification output more stable.

[0145] The specific process of the image feature extraction module extracting feature data is as follows:

[0146] The basic feature matrix is ​​obtained by extracting features from the restored image. The channel weight values ​​are adjusted according to the frequency domain interference intensity of each channel and the degree of matching with the standard reference of the coal mining machine to obtain the weighted feature matrix.

[0147] Obtain the key region map of the weighted feature matrix in the channel dimension, and divide the key region map into multiple grid regions;

[0148] Calculate the regional coefficient for each grid region. When the regional coefficient is greater than the preset occlusion threshold, assign the first weight to the corresponding region. When the regional coefficient is less than or equal to the preset occlusion threshold, assign the second weight to the corresponding region. This is to perform spatial dimension regional weighting on the weighted feature matrix and output the coal mining machine operating condition feature data.

[0149] The specific process by which the operating condition classification module determines the current judgment threshold and output category is as follows:

[0150] Calculate the moving average and standard deviation of the historical recognition score sequence for the current candidate category;

[0151] The operating condition fluctuation adjustment coefficient is determined based on the current traction speed of the coal mining machine;

[0152] Based on the combined calculation of the moving average, standard deviation, and operating condition fluctuation adjustment coefficient, the current judgment threshold for the current frame is determined, wherein the current judgment threshold satisfies:

[0153] ;

[0154] in, This is the current judgment threshold. It is a moving average. Standard deviation, This is the operating condition fluctuation adjustment coefficient;

[0155] When the recognition score of the current frame is greater than or equal to the current judgment threshold, the corresponding working condition category result is output; otherwise, the historical stable working condition is output. The historical stable working condition is the mode of the output categories in the most recent N frames, where N is a preset positive integer.

[0156] The operating condition classification module also includes a multi-frame status verification process, specifically:

[0157] Compare the first working condition category with the highest recognition score in the current frame with the second working condition category, which is a historically stable working condition;

[0158] When the two are inconsistent, determine whether the maximum recognition score of the first working condition category is greater than the sum of the current judgment threshold and the preset fixed judgment margin;

[0159] If yes, output the first working condition category; otherwise, output the second working condition category and generate a category mutation marker to be fed back to the image filtering and restoration module to trigger the adjustment of the filter kernel size.

[0160] This implementation mainly describes the data processing process between the image feature extraction module and the working condition classification module; the restored image output by the image filtering and restoration module enters the image feature extraction module, the working condition classification module judges the extraction result, and feeds back the judgment result and the verification result to the front-end processing unit;

[0161] In this embodiment, the basic feature matrix refers to the multi-channel feature data formed by the image feature extraction unit after convolution extraction of the restored image; the standard reference pattern of the coal mining machine refers to the reference feature pattern obtained by training with a sample set, which is used to characterize the key components of the coal mining machine and the relatively stable appearance structure under typical working conditions.

[0162] The key area map refers to the two-dimensional region response map obtained after filtering in the channel dimension, which is used to represent the distribution of more noteworthy regions in the current image; the region coefficient is the relative ratio between the average response of a certain region and the maximum response of the entire image, which is used to determine whether the region may be occluded or whether it contains strong and effective information.

[0163] After receiving the restored image, the image feature extraction module extracts the basic feature matrix. The basic feature matrix usually contains multiple channels, each corresponding to a type of local texture, edge, brightness change or structural combination response. Since residual interference in the downhole environment may be more obvious in some channels, all channels cannot be used equally directly, but channel weighting is required first.

[0164] During channel weighting, the image feature extraction module calculates the frequency domain interference intensity for each channel of the basic feature matrix; the frequency domain interference intensity is determined by the ratio of the spectral energy of that channel in a preset high-frequency band to the total spectral energy of that channel.

[0165] In the specific calculation, the image feature extraction module performs a two-dimensional discrete Fourier transform on the two-dimensional feature map of this channel to convert it to the frequency domain, setting the high-frequency cutoff radius with the center of the spectrum as the origin as... The spectral energy and frequency domain interference intensity are calculated using the following formulas:

[0166] Preset spectral energy within the high-frequency band:

[0167] ;

[0168] Total spectral energy across the entire frequency domain:

[0169] ;

[0170] Frequency domain interference intensity:

[0171] ;

[0172] in, Represents the channel feature map in the frequency domain coordinates Frequency components at that location; The magnitude of the corresponding frequency component depends on the response value of the input feature tensor. It represents the high-frequency spectrum energy, with the dimension being the square of the characteristic amplitude; This represents the total spectral energy, with the same dimension of the square of the characteristic amplitude. Frequency domain interference intensity;

[0173] Electromagnetic interference and sharp noise often form abnormal energy concentrations in the high-frequency band. Although the normal structural characteristics of a coal mining machine also contain edge high-frequency components, they do not present a large number of irregular high-frequency spikes in multiple areas. Therefore, when the proportion of high-frequency energy in a certain channel is too high, it indicates that the channel may be dominated by noise, and its channel weight should be reduced.

[0174] Meanwhile, the image feature extraction module also calculates the normalized correlation coefficient between the feature map of each channel and the standard reference pattern of the coal mining machine as the degree of matching. The higher the normalized correlation coefficient, the more the channel matches the standard structure or typical working condition appearance of the coal mining machine, and the more likely it reflects effective working condition information rather than random interference. Therefore, when the degree of matching of a certain channel is higher than the correlation threshold, the weight of the corresponding channel is increased.

[0175] In practice, the image feature extraction module sets the initial weight values ​​for each channel to be... For a specific channel, the module adjusts the weights through two consecutive logical steps, with the specific adjustment formula as follows:

[0176] The transition weights are obtained by adjusting downwards based on the frequency domain interference intensity. :

[0177] ;

[0178] The new weight value for this channel is obtained by adjusting the weight upwards based on the degree of matching. :

[0179] ;

[0180] Channel scaling is applied to the current channel's basic feature matrix:

[0181] ;

[0182] in, The initial weights are dimensionless; The frequency domain interference intensity of this channel is dimensionless. The frequency domain energy threshold is dimensionless. The normalized correlation coefficient is used to measure the degree of matching between the channel feature map and the standard reference image of the coal mining machine. It is dimensionless. The relevance threshold is dimensionless. The preset excitation coefficient is a dimensionless constant. and These are the pixel response values ​​of the channel feature matrix before and after scaling; after this weighted filtering, the weighted feature matrix can be obtained.

[0183] In addition, the energy threshold and correlation threshold are determined based on the statistical results of the sample set. The energy threshold corresponds to the boundary where the channel has been significantly affected by high-frequency interference, and the correlation threshold corresponds to the boundary where the channel has formed a stable match with the standard reference pattern. Both are set according to the distribution of each channel in the training sample. Specifically, the energy threshold is set to the 85th to 95th percentile value of the frequency domain interference intensity distribution of each channel, and the correlation threshold is set to the mean value of the matching degree distribution of each channel. This makes the noise channel suppressed by mathematical proportional attenuation, and the effective channel is clearly preserved.

[0184] After obtaining the weighted feature matrix, the image feature extraction module generates a maximum value map and an average value map in the channel dimension. The maximum value map reflects the strongest response of each position in all channels, which helps to highlight local key structures. The average value map reflects the overall average response of each position, which helps to maintain the integrity of the region. The module performs a weighted combination of the maximum value map and the average value map according to a preset ratio to obtain the key area map.

[0185] In this embodiment, based on the spatial mapping requirements of feature fusion, the preset ratio is clearly defined as 0.5 for each, that is, the pixel value of each position in the maximum value map multiplied by 0.5 is added to the corresponding pixel value in the average value map multiplied by 0.5.

[0186] Since using only the maximum value chart can easily amplify local abnormal bright spots, while using only the average value chart may weaken the outline of key components, the above-mentioned weighted combination of equal proportions can achieve a robust balance between local prominence and overall stability.

[0187] The image feature extraction module divides the key region image into multiple regions according to a preset grid size. In this embodiment, based on the spatial mapping requirements of feature fusion, the pixel-level fusion calculation formula for the key region image is defined as follows:

[0188] ;

[0189] in, To generate a key area map in spatial coordinates Feature pixel value at; The maximum value in spatial coordinates Feature pixel value at; The average value plot in spatial coordinates Feature pixel value at;

[0190] If the calculated regional coefficient is greater than the first coefficient threshold, it means that the overall response of the region is greater than the first response threshold. This usually indicates that the outline of the coal mining machine components is clear, the proportion of the obscured area is lower than the preset area proportion, or the relevant features of the working condition are concentrated. If the regional coefficient is less than the second coefficient threshold, it means that the gray scale feature of the region is lower than the preset lower limit, the clarity is lower than the preset lower limit, or the distribution dispersion is higher than the preset dispersion. It may be affected by dust obscuration, local pollution, or background interference.

[0191] Among them, the preset occlusion threshold Determined by offline system statistics, the calculation logic is as follows: extract the coefficients of reference image regions without dust occlusion from the sample set, calculate their global average value, and multiply by a preset reduction factor. The value ranges from 0.6 to 0.75, and the specific formula is as follows:

[0192] ;

[0193] in, Set the preset occlusion threshold; The preset reduction factor is S; S is the total number of grid regions drawn from the reference image. For the reference image, the first Unobstructed area coefficient for each region;

[0194] The image feature extraction module compares the region coefficient with the occlusion threshold. When the region coefficient is higher than the occlusion threshold, the region is assigned a first weight; when the region coefficient is less than or equal to the occlusion threshold, it is assigned a second weight, where the second weight is less than the first weight.

[0195] In this embodiment, the first weight is set to a value range of 0.8 to 1.0 to maintain or moderately enhance the features of the effective region; the second weight is set to a value range of 0.1 to 0.3 to significantly suppress the interference features introduced by the occluded region.

[0196] Therefore, the proportion of areas affected by occlusion exceeding the preset occlusion assessment threshold in subsequent features is significantly reduced, while the proportion of key working areas is increased to the preset weight value; the image feature extraction module performs regional weighting processing on the weighted feature matrix according to the corresponding weight of each region, outputs the final coal mining machine working condition feature data, and sends it to the working condition classification module.

[0197] Specifically, since the weighted feature matrix is ​​a three-dimensional basic tensor containing the channel dimension, and the weights corresponding to each region form a two-dimensional spatial weight map, the image feature extraction module expands and broadcasts the two-dimensional spatial weight map along the channel dimension to make its dimensional structure strictly aligned with the weighted feature matrix, and performs the Hadamard product operation on the corresponding elements with the weighted feature matrix, thereby completing the uniform application of weights to each region without destroying the internal correlation structure between feature channels;

[0198] In this embodiment, the calculation process of the current judgment threshold is performed by the threshold calculation unit in the working condition classification module. The threshold calculation unit obtains the recognition score sequence corresponding to the current candidate category from the historical frames cached by the system. For example, if the candidate category with the highest score in the current frame is truncation, then the score records of this category in several historical frames are read.

[0199] After the reading is completed, the threshold calculation unit calculates the moving average and standard deviation of the recognition score sequence; the moving average reflects the recent typical score level of the category, and the standard deviation reflects the recent fluctuation of the category;

[0200] The threshold calculation unit obtains the current traction speed from the coal mining machine control system. The traction speed reflects how fast the coal mining machine's operating status changes. When the speed is greater than the first traction speed threshold, the frequency of coal mining machine action changes and image posture changes is higher than the preset change frequency. When the speed is less than the second traction speed threshold, the working condition is relatively stable.

[0201] The segmented mapping rule refers to dividing the traction speed into several intervals according to the ratio of the rated traction speed of the coal mining machine, with each interval corresponding to a preset adjustment coefficient;

[0202] The segmented mapping rule in this embodiment introduces a dual-threshold hysteresis loop mechanism to suppress the oscillation of the adjustment coefficient caused by the fluctuation of the traction speed near the critical point of the interval. For the switching boundary between the low-speed stable state and the medium-speed state, a first lower threshold, such as 28%, and a first upper threshold, such as 32%, are set.

[0203] When the traction speed increases from low to high and exceeds the first upper limit threshold, the operating condition fluctuation adjustment coefficient switches from a higher value in the low-speed state, preferably a fixed value of 1.3, to a fixed value of 1.0 in the medium-speed state; when the traction speed decreases from high to low and falls below the first lower limit threshold, the adjustment coefficient switches back from 1.0 to 1.3.

[0204] Similarly, for the switching boundary between medium speed and high speed fluctuation, a second lower threshold, such as 68%, and a second upper threshold, such as 72%, are set; when the traction speed exceeds the second upper threshold, the adjustment coefficient is switched from a fixed value of 1.0 to a lower value of the high speed state, preferably a fixed value of 0.6, and then switched back to 1.0 when it falls below the second lower threshold.

[0205] This processing mechanism not only clearly quantifies the adjustment coefficient values ​​for each state, but also effectively isolates the physical jitter of hardware sensors through time-sequence hysteresis logic. It quantitatively and smoothly introduces the strength of the actual operating state changes of the coal mining machine into the classification and judgment system, thereby reducing the system's dependence on a single visual feature and avoiding the lack of interpretability in the parameter configuration process.

[0206] The threshold calculation unit calculates the current judgment threshold based on the moving average + standard deviation × operating condition fluctuation adjustment coefficient. If the current traction speed corresponds to a large fluctuation, the operating condition fluctuation adjustment coefficient is set to a lower value, so as to weaken the effect of the standard deviation on raising the threshold and thus avoid the threshold being too high when the actual operating condition is switched.

[0207] If the current traction speed corresponds to a relatively stable state, the operating condition fluctuation adjustment coefficient is set to a higher value, which enhances the effect of the standard deviation on the threshold, thereby avoiding false switching caused by single-frame interference during stable operation; after the current judgment threshold is calculated, the operating condition discrimination program receives the coal mining machine operating condition feature data of the current frame and outputs the corresponding recognition score.

[0208] The threshold calculation unit compares the recognition score with the current judgment threshold: when the recognition score is greater than or equal to the current judgment threshold, the corresponding working condition category result is output; otherwise, the historical stable working condition is output; the historical stable working condition is defined as the mode of the output category in the most recent N frames, which represents the most common and relatively stable working condition state in the recent period.

[0209] In the specific statistical logic, if multiple categories have the same frequency of occurrence and are tied for the highest frequency in the output categories of the most recent N frames, then according to the principle of temporal priority, the category that appears most recently in the current frame is taken as the mode to ensure the uniqueness of the classification system.

[0210] For the multi-frame state verification process, in this embodiment, it is executed by the state verification unit in the working condition state classification module; the state verification unit reads the first working condition category with the highest recognition score in the current frame output and its maximum recognition score, calculates the mode from the output categories of the most recent N frames, uses the mode as the historical statistical recognition result of the historical frame, and uses it as the second working condition category; here, the first working condition category represents the most likely instantaneous judgment in the current frame, and the second working condition category represents the stable judgment in the recent continuous results;

[0211] The state verification unit compares the first working condition category with the second working condition category. If they are consistent, it means that the current frame judgment is consistent with the recent stable state, and the current category output can be directly maintained, and the category mutation flag is generated as 0. If they are inconsistent, it means that there is a possibility of category mutation in the current frame. This category mutation may be due to the actual working condition switch of the coal mining machine, or it may be due to the instantaneous recognition error caused by dust obstruction, electromagnetic interference or lens contamination. Therefore, further state verification needs to be performed.

[0212] During further verification, the status verification unit compares the maximum recognition score with the current judgment threshold plus the fixed judgment margin; the fixed judgment margin is a preset positive number, such as 0.1 to 0.15, which is set by historical sample statistics and its function is to add auxiliary judgment constraints when the category changes.

[0213] If the maximum recognition score is greater than the sum of the current judgment threshold and the preset fixed judgment margin, it means that the current frame’s support for the first working condition category is greater than the sum of the current judgment threshold and the preset fixed judgment margin. The category change can be considered reliable. Therefore, the first working condition category is output and the category mutation flag is set to 1.

[0214] If the maximum recognition score does not exceed the higher threshold, it means that although a high-scoring candidate category appears in the current frame, the feature data does not meet the preset judgment condition for negating the historical stable working condition. Therefore, the second working condition category is output, and this inconsistency event is also fed back as a category mutation situation for the previous filtering adjustment. When the image filtering and restoration module receives the category mutation flag as 1, it triggers the adjustment of the filtering kernel size to cope with the situation of large changes in image conditions recently.

[0215] In a typical recognition process, the drum area in the restored image is obscured by local dust. After the working condition discrimination program calculates the current frame, the coal loading category recognition score is slightly higher than the cutting category, but the output category of the most recent N frames is cutting. At this time, the state verification unit finds that the first working condition category is inconsistent with the second working condition category, and continues to judge whether the maximum recognition score exceeds the current judgment threshold + fixed judgment margin.

[0216] If the threshold is not exceeded, the output of the historically stable cut category will be maintained to avoid incorrect switching caused by single-frame occlusion. If coal loading-related features are continuously displayed in subsequent frames and the maximum recognition score significantly exceeds the above-mentioned higher threshold, the state verification unit will accept the new category. Through this processing, the key area information output by the image feature extraction module and the historical statistical rules in the working condition classification module work together to make the category judgment more suitable for continuous underground operation scenarios.

[0217] The configuration process for the image sample generation module includes:

[0218] The differences in interference distribution between the simulated interference image and the actual noisy image in the mine, the differences in image content between the simulated interference image and the reference image of the ground coal mining machine, and the working condition discrimination deviation of the initial working condition discrimination program were obtained respectively.

[0219] Based on the differences in interference distribution, image content, and working condition discrimination bias, the image generation parameters of dust occlusion density, noise amplitude, and blur intensity are jointly updated, and the working condition discrimination program is retrained based on the updated sample set.

[0220] This embodiment illustrates the specific configuration process of the image sample generation module during the offline configuration phase; the module is deployed on a training computing device, and the input data includes reference images of ground coal mining machines and actual noisy images of the mine.

[0221] The reference image of the surface coal mining machine provides clear information about the working conditions, while the actual noisy image of the mine provides information about the distribution of underground interference. By using both types of images together in the configuration, the generated simulated interference image can retain the semantics of the coal mining machine's working conditions and closely approximate the appearance of underground interference.

[0222] At the start of the configuration process, the image sample generation module first acquires a reference image of the ground coal mining machine and an actual noisy image of the mine. The reference image contains known actual working condition markings, while the actual noisy image of the mine reflects the true combination of underground dust, electromagnetic noise, and fuzziness.

[0223] The image sample generation module generates simulated interference images based on the reference image. The initial image generation parameters include at least dust occlusion density, noise amplitude, and blur intensity. Dust occlusion density is used to control the area ratio of the occluded region in the image and the occlusion attenuation coefficient. Noise amplitude is used to control the strength of random noise and impulse noise. Blur intensity is used to control the degree to which the image edges and textures are weakened.

[0224] After generating simulated interference images, the image sample generation module does not directly use them as the final training samples, but first performs three types of difference and bias calculations;

[0225] The first category is the difference in interference distribution; the image sample generation module statistically analyzes the gray-level histograms of the simulated interference image and the actual noisy image of the mine, and compares the gray-level distribution patterns of the two; at the same time, frequency domain analysis is performed on the two types of images to obtain the frequency domain intensity distribution, and then the differences in energy distribution between the two in different frequency bands are compared.

[0226] In the specific comparison, the Bach distance is used to measure the difference in gray-level histogram distribution. For the difference in frequency domain energy distribution, in order to avoid the scale imbalance problem caused by the unbounded domain of the conventional relative entropy, this implementation uses the Jensen-Shannon divergence, which has symmetry and whose domain strictly converges to the [0,1] interval, for measurement. After unifying the domain scale, the two are added together according to the preset weights to obtain the quantified interference distribution difference index.

[0227] Since spatial domain occlusion and frequency domain pulses have an equal impact on overall image degradation, this implementation explicitly sets the preset weights for both to 0.5. The interference distribution difference index is calculated using the following joint error equation. This avoids the logical fallacy, from a mathematical perspective, where a sudden increase in the value of a single unbounded indicator can overshadow the contribution of another indicator.

[0228] ;

[0229] in, Represents the Bhattacharyya distance metric function. This is the histogram vector of the grayscale distribution of the simulated interference image; This is the histogram vector of grayscale distribution in the actual noisy image of the mine. This represents the JS divergence calculation operator. To simulate the energy distribution probability matrix of the interference image in the frequency domain; This represents the probability matrix of the energy distribution in the frequency domain of an actual noisy image of the mine.

[0230] If the simulated interference image is too uniform in grayscale distribution, or if the energy proportions in the high-frequency and low-frequency bands deviate significantly from the actual noisy image, it indicates that the current image generation parameters fail to accurately reflect the downhole interference and should be adjusted further.

[0231] The reason for setting the above interference distribution difference index is that the underground dust will usually change the basic gray level distribution of the image, and electromagnetic noise and stripe interference will change the frequency domain intensity distribution. Comparing one of them alone is not enough to reflect whether the degradation is close to the real situation.

[0232] The second category is image content difference; the image sample generation module extracts contours from the simulated interference image and the reference image to form the corresponding basic contour matrix, and calculates the mean square error between the two.

[0233] In this process, the image sample generation module uses the Canni edge detection algorithm to perform Gaussian smoothing, gradient calculation, non-maximum suppression and double threshold edge connection processing on the simulated interference image and the reference image respectively to extract the image contour. The extracted binarized edge distribution matrix is ​​the basic contour matrix.

[0234] The mean squared error is calculated by calculating the sum of squared differences between corresponding elements of the two basic contour matrices pixel by pixel, and then taking the average value of the spatial dimension. The smaller the mean squared error, the closer the outer contour of the coal mining machine, the boundaries of key components, and the overall working posture in the simulated interference image are to the reference image. The larger the mean squared error, the more the original content has been excessively destroyed during the generation process, which may lead to distortion of key working conditions such as drums, machine body connections, or coal flow patterns.

[0235] The module determines whether the contour difference meets the preset conditions based on the mean square error value; the preset conditions represent the boundaries where degradation has been added but the content is still recognizable, in order to avoid the generated image looking like a downhole image but no longer being able to maintain the correct working condition information;

[0236] The third category is the working condition discrimination bias; the image sample generation module inputs the simulated interference image into the initial working condition discrimination program, obtains the predicted probability score vector of each working condition category, and obtains the predicted probability score of the category corresponding to the actual working condition label of the reference image.

[0237] In specific quantification, the absolute value of the difference between the ideal confidence level (a constant value of 1.0) and the predicted probability score corresponding to the actual working condition label is calculated and output as the working condition discrimination bias of the simulated interference image. The bias value ranges from 0 to 1. The larger the bias value, the more the currently generated degraded interference has excessively damaged the core visual features, resulting in a significant decrease in the model's recognition confidence. The smaller the bias value, the more the working condition discrimination program can still extract and determine the correct working condition information from the interference image.

[0238] After obtaining the above three types of results, the image sample generation module updates the image generation parameters in the direction of reducing the difference in interference distribution, reducing the difference in image content, and reducing the deviation in working condition discrimination.

[0239] In specific adjustments, the adjustment step size is determined based on the difference between the calculated difference indicators and their target tolerance thresholds. For each image generation parameter, the adjustment step size is based on the parameter value before adjustment. The preset learning compensation coefficient for this parameter The difference index value calculated so far and the target tolerance threshold corresponding to this difference. Calculate the adjusted value of this parameter. Its update logic satisfies the formula:

[0240] ;

[0241] in, This is the adjusted value for the parameter; The value of this parameter before adjustment; This is the preset learning compensation coefficient for this parameter; The value of the difference index obtained from the current calculation; This is the target tolerance threshold corresponding to the difference;

[0242] To ensure dimensional balance and physical consistency on both sides of the formula, different physical properties are considered for the generation parameters, such as noise amplitude with gray dimensions or dimensionless occlusion ratio, and various difference indicators, such as Bach distance and root mean square error, and the learning compensation coefficient. The dimension is always equal to the dimension of the target parameter divided by the dimension of the corresponding difference index. In essence, it has the dual functions of scaling and dimension conversion. The system pre-constructs a set of compensation mappings that are independent between each generated parameter and the corresponding difference index to avoid cross-dimensional interference.

[0243] If the difference in interference distribution is greater than the first distribution difference threshold while the difference in image content is less than the first content difference threshold, then according to the above formula, the values ​​of dust occlusion density, noise amplitude and blur intensity are increased with a positive learning compensation coefficient first, so that the simulated image is closer to the actual downhole noise distribution.

[0244] If the mean square error of the contour of the image content difference is greater than the first contour error threshold, it indicates that the degradation is too severe or the method of action is inappropriate. The parameters that will significantly damage the contour should be reduced by using a negative learning compensation coefficient according to the above formula. If the working condition discrimination deviation is greater than the first deviation threshold, it indicates that although the current image has a degraded appearance, it is not conducive to retaining the working condition discrimination information. Similarly, the relevant parameters should be adjusted by multiplying the deviation rate by the learning compensation factor.

[0245] In the above closed-loop update process of parameters, in order to prevent the parameter adjustment direction from conflicting when multiple difference indicators are too large at the same time, that is, the distribution difference requires the increase of parameters while the content difference requires the decrease of parameters, resulting in the oscillation that cannot converge, the system decouples the parameters through a state machine timing constraint strategy: the parameters are allowed to be increased in the positive direction based on the interference distribution difference only when the image content difference and the working condition discrimination deviation simultaneously meet the corresponding target tolerance threshold.

[0246] Once either of the latter two exceeds the threshold, the positive increment permission is forcibly revoked and the system enters the negative feedback callback phase. Through this gradual update closed-loop control based on priority damping and deviation quantization feedback, the image generation parameters will continuously stabilize and approach the convergence equilibrium state where the degradation distribution is realistic and the working condition content is recognizable.

[0247] While updating the image generation parameters, the image sample generation module also updates the working condition discrimination program parameters. Specifically, the updated image generation parameters are used to regenerate the simulated interference image, and the new simulated interference image is mixed with the real downhole image to form an updated sample set. The updated sample set is then input into the training computing device to retrain the working condition discrimination program.

[0248] After retraining, the working condition discrimination program improves its ability to recognize degraded images and can continue to be used as a new initial working condition discrimination program in the next round of deviation calculation. Through this iterative mechanism, the image generation parameters and the working condition discrimination program parameters are optimized synchronously until the generated simulated interference image reaches a convergent equilibrium state in terms of interference distribution fidelity, image content retention rate, and working condition discriminability.

[0249] In a typical training scenario, although the initially generated simulated interference image has obvious noise, the proportion of high-frequency energy is lower than that of the actual noisy image in the mine, indicating that the electromagnetic interference simulation is insufficient; at the same time, the contour mean square error value is small, indicating that the content is well preserved.

[0250] However, the initial working condition discrimination program exhibits class bias in some simulated images. At this point, the image sample generation module will appropriately increase the noise amplitude and keep the blur intensity within a range that does not significantly damage the contour, then regenerate the simulated interference image and retrain the working condition discrimination program. If the contour mean square error value is subsequently found to increase significantly, it indicates that the adjustment has affected the preservation of content, and the blur intensity or dust occlusion density should be adjusted back.

[0251] After multiple rounds of adjustments, the generated images can closely approximate the actual noisy images in terms of grayscale histogram and frequency domain intensity distribution, while maintaining consistency between the coal mining machine's operating condition markings and classification results, thus providing a more suitable training basis for the real-time underground recognition stage.

[0252] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-modal working condition visual image recognition system for coal mining machines, characterized in that, include: The image sample generation module is used to generate simulated interference images based on the received ground coal mining machine reference images during the offline configuration phase, and to mix the simulated interference images with the acquired real underground images to form a sample set for training the working condition discrimination program. The image filtering and restoration module is used to receive the real-time video stream from the downhole camera, perform hybrid filtering on the current frame image, output the restored image, and dynamically adjust the filtering parameters according to the calculated image quality assessment value and the received recognition score fluctuation characteristics. An image feature extraction module, connected to the image filtering and restoration module, is used to perform channel weighting and region weighting on the restored image to extract coal mining machine operating condition feature data. The working condition classification module includes the working condition discrimination program, which is used to receive the working condition feature data of the coal mining machine and output the recognition score, dynamically determine the current judgment threshold by combining the current traction speed and the statistical features of historical recognition scores to output the working condition category result, and feed back the recognition score of the current frame and the recognition score fluctuation features of multiple consecutive frames to the image filtering and restoration module. The specific process by which the operating condition classification module determines the current judgment threshold and output category is as follows: Calculate the moving average and standard deviation of the historical recognition score sequence for the current candidate category; The operating condition fluctuation adjustment coefficient is determined based on the current traction speed of the coal mining machine; Based on the combined calculation of the sliding average, the standard deviation, and the operating condition fluctuation adjustment coefficient, the current judgment threshold for the current frame is determined, wherein the current judgment threshold satisfies: in, The current determination threshold is... The moving average value is... The standard deviation is... The operating condition fluctuation adjustment coefficient is the coefficient described above. When the recognition score of the current frame is greater than or equal to the current judgment threshold, the corresponding working condition category result is output; otherwise, the historical stable working condition is output, which is the mode of the output categories in the most recent N frames, where N is a preset positive integer. The operating condition classification module also includes a multi-frame status verification process, specifically: Compare the first working condition category with the highest identification score in the current frame with the second working condition category, which is the historical stable working condition; When the two are inconsistent, determine whether the maximum recognition score of the first working condition category is greater than the sum of the current judgment threshold and the preset fixed judgment margin; If yes, the first working condition category is output; if no, the second working condition category is output, and a category mutation marker is generated and fed back to the image filtering and restoration module to trigger the adjustment of the filter kernel size.

2. The multi-modal working condition visual image recognition system for coal mining machines according to claim 1, characterized in that, The specific process of the image filtering and restoration module performing hybrid filtering is as follows: Obtain multiple pixel sub-blocks from the input image; Calculate the local gray-level variance within each pixel sub-block as an evaluation index for local noise intensity. The local noise intensity evaluation index is compared with a low noise threshold and a high noise threshold, respectively. The low noise threshold is less than the high noise threshold. The low noise threshold is the product of the mean gray-level variance of the historical M frames of background images without device motion and a first coefficient. The high noise threshold is the product of the mean gray-level variance and a second coefficient. The first coefficient is less than the second coefficient. M is a preset positive integer. When the local noise intensity evaluation index is less than the low noise threshold, the mean filtered pixel value is used as the filtered output pixel value. When the local noise intensity evaluation index is greater than the high noise threshold, the median filtered pixel value is used as the filtered output pixel value. When the local noise intensity assessment index meets When, the weights of the mean filtering result Weights of the median filtering result The calculation formula is as follows: in, The low noise threshold, The high noise threshold, The local noise intensity evaluation index is defined as follows; and the sum of the weights of the mean filtering result and the median filtering result is 1.

3. The multi-modal working condition visual image recognition system for coal mining machines according to claim 1, characterized in that, The specific process of adjusting the filtering parameters in the image filtering and restoration module is as follows: The image quality assessment value of the restored image is obtained, and the standard deviation of the recognition scores of N consecutive frames is used as the recognition score fluctuation value, where N is a preset positive integer; When the average image quality assessment value of the most recent N consecutive frames is lower than the preset quality threshold, and the recognition score fluctuation value is higher than the stability threshold, the window size of the filter kernel in the hybrid filtering is increased, and the weight of the median filtering result in the weighted combination is increased relative to the previous frame.

4. The multi-modal working condition visual image recognition system for coal mining machines according to claim 1, characterized in that, The specific process of the image feature extraction module extracting feature data is as follows: The restored image is subjected to feature extraction to obtain a basic feature matrix. The channel weight values ​​are adjusted according to the frequency domain interference intensity of each channel and the degree of matching with the standard reference pattern of the coal mining machine to obtain a weighted feature matrix. Obtain the key region map of the weighted feature matrix in the channel dimension, and divide the key region map into multiple grid regions; Calculate the regional coefficient for each grid region. When the regional coefficient is greater than a preset occlusion threshold, assign a first weight to the corresponding region. When the regional coefficient is less than or equal to the preset occlusion threshold, assign a second weight to the corresponding region. This is done to perform spatial dimension regional weighting on the weighted feature matrix and output the coal mining machine operating condition feature data.

5. The multi-modal working condition visual image recognition system for coal mining machines according to claim 1, characterized in that, The configuration process of the image sample generation module includes: The differences in interference distribution between the simulated interference image and the actual noisy image in the mine, the differences in image content between the simulated interference image and the reference image of the ground coal mining machine, and the working condition discrimination deviation of the initial working condition discrimination program are obtained respectively. Based on the differences in interference distribution, the differences in image content, and the working condition discrimination bias, the image generation parameters of dust occlusion density, noise amplitude, and blur intensity are jointly updated, and the working condition discrimination program is retrained based on the updated sample set.

Images