A video image processing method and device for a coal mining face

Abstract

This invention relates to the field of video image processing technology, and discloses a method and apparatus for processing video images of coal mining faces. The method includes acquiring the original video stream and performing distortion correction using camera intrinsic parameters; preprocessing the image through block histogram enhancement and Gaussian filtering; extracting straight line features using the maximum inter-class variance method and morphological operations, and generating a set of valid straight lines based on the least squares method; determining reference straight lines according to the number of straight lines using a hierarchical strategy, and calculating the rotation angle by using anonymization point clustering to identify parallel structures; finally, constructing an affine transformation matrix to perform rotation correction on the image, and cropping invalid edge regions to output the final image. This invention solves the problem of tilt distortion in monitoring images caused by limited underground installation space and non-horizontal installation, effectively eliminating perspective interference and outputting monitoring images that maintain horizontal geometric properties, facilitating the observation of the real-time attitude of equipment such as hydraulic supports.

Description

Technical Field

[0001] This invention relates to the field of video image processing technology, specifically to a method and apparatus for video image processing in coal mining faces. Background Technology

[0002] As the core area of ​​coal mine production, real-time monitoring of equipment status at the coal face is crucial for ensuring safe production. Due to the complex underground geological conditions and the frequent movement of hydraulic supports, the installation space for cameras at the working face is extremely limited, making it difficult to ensure that the camera equipment is installed in a strictly horizontal position. This limitation in physical installation conditions directly results in a prevalent tilting phenomenon in the acquired monitoring videos, making the images displayed on the terminal inconsistent with the horizontal visual habits of the human eye. This makes it difficult for monitoring personnel to intuitively and accurately determine the actual horizontal posture of key equipment such as hydraulic supports and scraper conveyors.

[0003] In attempting to correct tilted images, the unique working environment underground presents significant challenges to image processing. Coal mining faces typically have low illumination and high concentrations of coal dust, resulting in monitoring images generally exhibiting low contrast, high noise, and blurred details. This leads to minimal grayscale differences between cable troughs, support edges, and the coal wall background. Furthermore, to cover a wider field of view, cameras are usually equipped with wide-angle lenses, inevitably introducing barrel distortion, causing originally straight linear features to appear curved in the image. Existing processing methods often struggle to accurately extract the straight-line features needed for correction when faced with such low-quality images exhibiting geometric distortion, resulting in a lack of reliable data foundation for subsequent processing steps.

[0004] Furthermore, existing technologies typically employ simple edge detection or Hough transform to find reference lines when calculating image rotation angles. However, coal mining face roadways exhibit deep extension characteristics, and parallel lines in three-dimensional space appear to converge or tilt in two-dimensional images due to perspective effects. Existing angle calculation methods often ignore this perspective rule, easily misjudging non-horizontal interference lines affected by perspective as horizontal reference lines, or lacking effective verification mechanisms when single line features exhibit abnormalities. This results in significant deviations in the calculated rotation angles, making accurate image correction impossible under complex working conditions. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a video image processing method and apparatus for coal mining faces, which solves the problem that coal mining face cameras suffer from geometric distortion and tilt in the monitoring images due to limited installation space and lack of strict horizontal calibration, making it impossible to accurately reflect the horizontal posture of equipment such as hydraulic supports and scraper conveyors.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] The first aspect of this invention provides a video image processing method for a coal mining face, the method comprising:

[0008] S1. Use the image acquisition unit to acquire the original video stream of the coal mining face scene and generate the original image data;

[0009] S2. Use camera intrinsic parameter data to perform distortion correction on the original image data to generate a distortion-free image;

[0010] S3. Enhance and smooth the distortion-reduced image to generate a preprocessed image;

[0011] S4. Perform binarization and morphological operations on the preprocessed image, extract line features, and filter to obtain a set of valid lines;

[0012] S5. Count the number of lines in the valid line set, determine the reference line based on the number of lines, and calculate the rotation angle based on the parameters of the reference line.

[0013] S6. Rotate the preprocessed image using a rotation angle, and then crop the rotated image to obtain the final image;

[0014] S7. Output the final image through the display unit.

[0015] Preferably, in step S3, the enhancement process employs a block-based strategy, dividing the distortion-reduced image into multiple rectangular sub-blocks, statistically analyzing the gray-level histograms of the rectangular sub-blocks, setting cropping limits, and using an interpolation algorithm to smoothly fuse the boundary pixels between the rectangular sub-blocks, resulting in the enhanced image. The smoothing process then uses a Gaussian convolution kernel to traverse the pixels in the enhanced image, calculating the new gray-level value of the center pixel through convolution operations and generating a pre-processed image. This step increases the gray-level difference between the cable trough, support edges, and the coal wall background, and filters out high-frequency noise points.

[0016] Preferably, in step S4, the binarization process uses the maximum inter-class variance method to statistically analyze the gray-level histogram of the preprocessed image, calculating the gray-level value that maximizes the variance between the foreground pixel class and the background pixel class as the optimal binarization threshold. The preprocessed image is then segmented using this optimal threshold to obtain a binary image. Morphological operations utilize preset structuring elements to first dilate the binary image, then erode it to form continuous boundary candidate regions. This step connects edge breaks caused by illumination occlusion, forming continuous linear features.

[0017] Preferably, in step S4, the process of extracting straight line features and selecting a set of valid straight lines is as follows: A region growing algorithm is used to traverse the candidate boundary line regions, and connected regions whose projected span lengths meet preset conditions are selected as valid linear features; the coordinate set of all pixels within the connected region is obtained, and a nonlinear least squares method is used to fit the coordinate set to a straight line. To ensure the linearity of the extracted features, the root mean square error of all pixels within the connected region relative to the fitted straight line is calculated, and fitted straight lines with root mean square errors less than a preset error threshold are retained to form a set of valid straight lines.

[0018] Preferably, the specific method for calculating the root mean square error of all pixels in the connected domain relative to the fitted line is as follows: under the constraint of normalizing the coefficients of the linear equation, calculate the sum of squares of the vertical Euclidean distances from all pixels to the fitted line, calculate the average value of the sum of squares, and take the square root of the average value to obtain the root mean square error.

[0019] Preferably, in step S5, the strategy for determining the reference line based on the number of lines includes: when the set of valid lines contains only one line, that line is determined as the reference line; when the set of valid lines contains two lines, the absolute values ​​of the slopes of the two lines are calculated respectively, and the line with the smallest absolute value of the slope is selected as the reference line.

[0020] Preferably, in step S5, when the effective set of lines contains more than two lines, a screening strategy based on disappearance points is adopted: a set of line pairs is constructed, and the intersection points of any two non-coincident lines are calculated to obtain a set of disappearance points; a clustering algorithm is used to classify the set of disappearance points, and the cluster containing the most disappearance points and the smallest intra-cluster variance is selected as the target cluster; a subset of lines corresponding to the disappearance points within the target cluster is determined, and the line with the smallest absolute slope value is selected from the subset and determined as the reference line. This strategy utilizes the geometric constraints of disappearance points to identify groups of parallel lines in three-dimensional space, eliminating non-horizontal interference lines caused by perspective effects and determining the correction benchmark.

[0021] Preferably, the method for calculating the intersection point of any two non-coincident lines includes: solving the equations of the two lines simultaneously; when the calculated value of the denominator of the equations of the two lines is greater than a preset small threshold, solving for the x-coordinate and y-coordinate of the intersection point and using it as the disappearance point in the disappearance point set.

[0022] Preferably, step S6 specifically involves: calculating the coordinates of the center point of the image width on the reference line as the rotation center, constructing an affine transformation matrix, and performing a rotation operation on the preprocessed image using the rotation angle and rotation center to obtain the rotated image; then calculating the minimum bounding rectangle of the effective pixels in the rotated image, and cropping the rotated image according to the minimum bounding rectangle to obtain the final image, thereby removing the invalid region caused by the rotation.

[0023] A second aspect of the present invention provides a video image processing device for a coal mining face, comprising:

[0024] The image acquisition unit, data processing unit, and display unit are connected sequentially via data transmission lines.

[0025] The image acquisition unit is configured to capture real-time video streams of the coal mining face and convert optical signals into digital image signals, and transmit the acquired raw image data to the data processing unit.

[0026] The display unit is connected to the data processing unit and is used to output the corrected horizontal viewing angle monitoring image. The data processing unit includes a distortion correction module, an image preprocessing module, a feature extraction module, an angle calculation module, and a geometric correction module, which are connected in sequence.

[0027] The distortion correction module is used to correct distortion in the original image data using camera intrinsic data to generate a distortion-free image.

[0028] The image preprocessing module is used to enhance and smooth the distorted image to generate a preprocessed image.

[0029] The feature extraction module is used to perform binarization and morphological operations on the preprocessed image, extract line features, and filter to obtain a set of valid lines.

[0030] The angle calculation module is used to count the number of lines in the valid line set, determine the reference line based on the number of lines, and calculate the rotation angle based on the parameters of the reference line.

[0031] The geometric correction module is used to rotate the preprocessed image using a rotation angle, and then crop the rotated image to obtain the final image, which is then sent to the display unit.

[0032] This invention, through the aforementioned technical solution, utilizes camera intrinsic parameters to correct image distortion introduced by wide-angle lenses, restoring the geometric properties of straight-line features in the image. Adaptive contrast enhancement and Gaussian filtering enhance the grayscale difference between equipment edges and the background while suppressing noise interference. In particular, the adoption of a case-by-case angle calculation strategy utilizes the slope of straight lines when features are simple, and uses the geometric constraints of anonymization points to identify parallel structures in three-dimensional space when features are rich, reducing calculation errors caused by abnormal single-line features. Finally, by rotating and cropping, a monitoring screen with a horizontal visual effect is output, facilitating observation of the equipment status on the work surface.

[0033] This invention provides a method and apparatus for processing video images of coal mining faces. It has the following beneficial effects:

[0034] 1. This invention restores the linearity of image geometric features through distortion correction, and combines adaptive histogram enhancement and Gaussian filtering to solve the problems of low image contrast and noise interference caused by low illumination and high dust environment in underground mines. It enhances the grayscale difference between the edges of cable troughs and hydraulic supports and the background of coal walls, and improves the accuracy of subsequent straight line feature extraction.

[0035] 2. This invention determines the reference line by adopting a hierarchical strategy based on the number of effective lines. In particular, in multi-line scenarios, it uses the elimination point clustering to identify parallel structures in three-dimensional space, eliminating non-horizontal interference lines caused by perspective effects, avoiding misjudgments caused by abnormal features of a single line, and ensuring the reliability of the rotation angle calculation benchmark under complex working conditions.

[0036] 3. This invention corrects the image by using the calculated rotation angle and cropping the invalid area of ​​the edge according to the minimum bounding rectangle. This eliminates the screen tilt caused by the inability to install the camera horizontally due to the limited installation space in the coal mining face, and outputs a horizontal monitoring screen that conforms to the visual habits of the human eye, making it easier for monitoring personnel to intuitively judge the real-time posture of the equipment. Attached Figure Description

[0037] Figure 1 This is a hardware connection diagram of the video image processing device for a coal mining face provided in an embodiment of the present invention;

[0038] Figure 2 This is a flowchart illustrating the video image processing method for a coal mining face provided in an embodiment of the present invention.

[0039] Figure 3 This is a block diagram of the internal functional modules of the data processing unit of the video image processing device for a coal mining face provided in an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] See attached document Figure 1 and attached Figure 2This invention provides a method and apparatus for processing video images of a coal mining face. The apparatus includes an image acquisition unit, a data processing unit, and a display unit. The image acquisition unit is configured to capture a video stream or image sequence containing a scene of the coal mining face. The data processing unit is communicatively connected to the image acquisition unit and is used to receive raw image data and execute subsequent image processing algorithms. The display unit is connected to the data processing unit and is used to output a corrected horizontal viewing angle monitoring screen. The data processing unit includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the following image processing steps.

[0042] See attached document Figure 2 This invention provides a video image processing method for coal mining faces, which is executed by electronic equipment or an image processing device, and specifically includes the following steps:

[0043] Step S1: Image Acquisition. The image acquisition unit (such as an intrinsically safe camera) is installed at the coal face. Strict level calibration is not required for its installation; only that the camera is approximately level with the object being filmed, allowing for some installation error. The image acquisition unit acquires the raw video stream containing the working face scene (such as hydraulic supports and scraper conveyors) and transmits it to the data processing unit. An image coordinate system is established, with the upper left corner of the image set as the origin. The axis extends to the right along the width of the image. The axis extends downwards along the height direction of the image.

[0044] Step S2: Image Distortion Removal. The data processing unit acquires the camera intrinsic parameters, including radial and tangential distortion coefficients, obtained through pre-calibration of the image acquisition unit offline. Based on the camera intrinsic parameters, the processor remaps the coordinates of each pixel in the original image using a distortion correction model. This process corrects barrel or pincushion distortion introduced by the wide-angle lens, generating a distortion-removed image, thus ensuring that the extracted straight-line features of the support edge conform to the true physical geometric properties.

[0045] Step S3: Image Preprocessing. This step includes two stages: enhancement and smoothing, aiming to generate a high-quality preprocessed image. First, the distortion-reduced image is enhanced. Specifically, the processor performs adaptive contrast enhancement on the distortion-reduced image, using an adaptive histogram equalization algorithm that limits contrast to enhance the grayscale differences between the cable trough, support edges, and coal wall background. Subsequently, the processor uses a Gaussian filtering algorithm to smooth the enhanced image, employing a Gaussian kernel function to perform a weighted average of the pixel neighborhood to filter out subtle high-frequency noise points caused by coal dust particles, generating the preprocessed image.

[0046] Step S4: Extracting Line Features. This step aims to extract a set of effective lines that can represent the pose of the working surface from the preprocessed image. First, the preprocessed image is binarized. The processor analyzes the grayscale histogram of the preprocessed image and dynamically calculates the optimal binarization threshold using Otsu's method or the maximum entropy method, converting the preprocessed image into a binary image. For edge breaks caused by illumination occlusion in the binary image, the processor performs morphological closing operations using structuring elements. That is, it first performs dilation to fill the edge gaps and then performs erosion to restore the original edge shape, thereby connecting the broken edge segments and forming continuous boundary line candidate regions. Finally, line features are extracted and the effective line set is obtained by filtering. The processor uses the 8-neighborhood region growing method to traverse the binary image, counting the total number of pixels contained in each connected region and their corresponding values ​​in the region. Span in the axial direction. Processor reserved. Connected regions whose axial span length is greater than or equal to 80% of the total image width are considered as effective linear features, and short, unstructured texture interference is eliminated.

[0047] For each retained continuous boundary line, the processor obtains the coordinates of all pixels it contains and fits the straight line equation using a nonlinear least squares method. Simultaneously, the processor calculates the root mean square error (RMSE) of the fitted line. If the calculated RMS error exceeds a preset error threshold, the line fitting quality is deemed unqualified and discarded. The error threshold is related to the image resolution. For example, for an image with a resolution of 640×480 pixels, the error threshold is set to 10 pixels; for an image with a resolution of 1280×960 pixels, the error threshold is set to 20 pixels. For other resolutions, the threshold is linearly derived proportionally to form a set of valid lines.

[0048] Step S5: Calculate the rotation angle of the image based on different cases. The processor calculates the rotation angle based on the valid line set selected in step S4, according to the following three cases. :

[0049] Scenario 1: When the set of valid lines contains only one line, the processor directly uses the slope parameter (i.e., the coefficients of the line equation) of that line. and Calculate the arctangent value using the negative of the ratio to obtain the rotation angle. .

[0050] Scenario 2: When the set of valid lines contains two lines, the processor calculates the absolute value of the slope of each line, selects the line with the smallest absolute slope (i.e., the one closest to the horizontal state) as a reference, and calculates the rotation angle based on the parameters of this reference line. .

[0051] Scenario 3: When the valid set of lines contains more than two lines, the processor calculates the coordinates of the intersection of any two lines to obtain a set of disappearance points. The processor determines whether there are multiple sets of disappearance points with unique disappearance points (i.e., multiple disappearance points are distributed within a small area). The processor uses the K-means clustering algorithm to classify the set of disappearance points. When a class contains multiple disappearance points and these disappearance points are distributed within a preset Euclidean distance range, it is determined that the corresponding set of lines that generated the disappearance points of this class are parallel to each other in three-dimensional space. The processor selects the line with the smallest absolute slope from this set of parallel lines as the final reference line and calculates the rotation angle based on the parameters of this reference line. .

[0052] Step S6: Image Correction. This step uses the calculated rotation angle to perform geometric transformation and post-processing on the image. The processor calculates the reference line determined in step S5 that lies within the image. The coordinates of the center point of the axis pixel count. For example, for a 640×480 image, calculate... When the coordinate is 320, it is on the reference line. Axis coordinates This coordinate point The rotation center is determined. The processor constructs an affine transformation matrix to rotate the image by an angle around this rotation center. Specifically, for each pixel in the image, its new coordinates after rotation are calculated using a rotation matrix, and the pixel values ​​are filled using bilinear interpolation. This yields the rotated image. Next, the black borders after rotation are processed. The processor calculates the extreme coordinates of all valid pixels in the rotated image, thereby determining the minimum bounding rectangle that can contain all valid pixels. The processor then crops the image based on the range of the minimum bounding rectangle, retaining the region where the minimum bounding rectangle contains all valid pixels.

[0053] Step S7: Output Display. The data processing unit sends the cropped image signal to the display unit. The display unit presents the automatically leveled coal face monitoring screen, ensuring that the roof, floor, or support beams in the screen always maintain a horizontal visual effect.

[0054] See attached document Figure 1 and attached Figure 2 First, step S1 is executed. The image acquisition unit captures the real-time video stream of the coal mining face and converts it into a digital image sequence, which is then transmitted to the data processing unit. In the processing logic of the data processing unit, a two-dimensional image coordinate system is established, with the upper left corner of the image plane set as the origin. Define the horizontal direction to the right as The positive axis is used to represent the pixel position along the width of the image; the vertical downward direction is defined as... The positive axis is used to represent the pixel position in the image's height direction. Due to the confined space in coal mining faces, image acquisition units are typically equipped with wide-angle lenses, and their installation location is limited by hydraulic support conditions, resulting in the acquisition of raw images... Geometric distortion exists.

[0055] Next, step S2 is executed. The data processing unit calls the pre-stored camera intrinsic data to process the raw image. Distortion correction is performed. Camera intrinsic data is established based on a polynomial distortion model, including radial and tangential distortion coefficients. The data processing unit iterates through each pixel in the original image and calculates its corresponding distortion-free ideal coordinates. Let the original image... The pixel coordinates in the image are The corresponding corrected ideal image coordinates are The specific mapping calculation formula is as follows:

[0056] ;

[0057] ;

[0058] in, and These represent the corrected horizontal and vertical coordinates, respectively. and These represent the horizontal and vertical coordinates in the original distorted image, respectively. This represents the Euclidean distance from a pixel to the optical center, which satisfies... ; , , These represent the first, second, and third order radial distortion coefficients, respectively; , These represent the tangential distortion coefficients in the horizontal and vertical directions, respectively.

[0059] The data processing unit reconstructs and generates a distortion-free image based on the calculated coordinate mapping relationship using an interpolation algorithm. .

[0060] Then step S3 is executed. The data processing unit processes the distortion-corrected image. Image preprocessing is performed; step S3 includes adaptive contrast enhancement and Gaussian filtering.

[0061] In adaptive contrast enhancement processing, the data processing unit will remove the distortion from the image. Divided into non-overlapping segments The algorithm iteratively processes rectangular sub-blocks. For each sub-block, its grayscale histogram is calculated, and a cropping limit is set. Pixels with frequencies exceeding the cropping limit in the histogram are evenly distributed across other grayscale levels to limit excessive local contrast amplification. After processing, bilinear interpolation is used to smoothly blend the boundary pixels between the sub-blocks, generating a contrast-enhanced image. This process significantly improves the visual boundary clarity between the cable trough support, scraper conveyor support, and the coal wall background.

[0062] After adaptive contrast enhancement, the data processing unit applies a Gaussian filtering algorithm to smooth and denoise the image. The data processing unit uses a Gaussian convolution kernel to iterate through the pixels in the contrast-enhanced image. The weight distribution function of the Gaussian convolution kernel is calculated as follows:

[0063] ;

[0064] in, Indicates the position of the Gaussian convolution kernel. The weight value at the location; This represents the horizontal distance offset of the sampling points within the convolution kernel relative to the center of the convolution kernel; This represents the vertical distance offset of the sampling point within the convolution kernel relative to the center of the convolution kernel; This represents the standard deviation of the Gaussian distribution, which determines the smoothness of the filter. The base of the natural logarithm; This represents the decay factor of the distance weight.

[0065] The data processing unit calculates the new grayscale value of the center pixel through convolution operations to generate a preprocessed image. Step S3 utilizes the smoothing properties of the Gaussian function to filter out subtle high-frequency noise caused by suspended coal dust particles in the image, while preserving the edge features of the large-scale support structure in the image, providing high-quality input data for subsequent line detection.

[0066] See attached document Figure 1 and attached Figure 2 After completing the image preprocessing operation, the data processing unit then executes step S4, which is to extract the straight line features.

[0067] The data processing unit first processes the preprocessed image. Binarization processing is performed. Given that in coal mining face scenarios, metal components such as cable troughs of hydraulic supports, side guards, and scraper conveyor supports exhibit high brightness under supplemental lighting, while the coal wall, roof, and floor background exhibit relatively low brightness, the image grayscale histogram shows a bimodal distribution. The data processing unit uses the Otsu's algorithm (maximum inter-class variance method) to statistically analyze the grayscale histogram of the preprocessed image, calculating the grayscale value that maximizes the variance between the foreground and background pixel classes, and determining this grayscale value as the optimal binarization threshold. The data processing unit utilizes the optimal binarization threshold. The preprocessed image is segmented to generate a binary image. In a binary image, a gray value greater than or equal to... Pixels are marked as logical "1", representing potential foreground edges of the support structure; grayscale values ​​less than The pixels are marked as logical "0" to represent the background environment.

[0068] To address the potential breakage of straight lines due to the complex physical environment downhole (such as localized dust obstruction or support gaps), the data processing unit processes binary images... Perform morphological closing operations. Morphological closing operations consist of two stages: dilation followed by erosion. First, a predefined structuring element (e.g., a 5×5 pixel rectangle) is used to dilate the binary image, expanding the foreground region boundary to fill tiny holes inside the support edge and connect adjacent broken line segments. Then, the same structuring element is used to erode the dilated image, shrinking the foreground region boundary back to its original size, thus eliminating unnecessary expansion introduced by dilation and ultimately forming a continuous and complete boundary line candidate region.

[0069] The line parameters for each boundary line are then extracted. The data processing unit uses an 8-neighbor connected region growing algorithm to traverse the binary image after closing operations. Starting from any unlabeled foreground pixel, the 8-neighbor connected region growing algorithm recursively checks its eight neighboring pixels, merging all interconnected foreground pixels into a single connected region. During the generation of connected regions, the data processing unit counts the total number of pixels contained within each connected region. And calculate the horizontal direction of the connected component. Projected span length in the axial direction .

[0070] To eliminate sporadic noise or interference from non-target objects, the data processing unit filters connected components based on the physical characteristics of the horizontal field of view of the coal mining face equipment. The data processing unit obtains the physical resolution width of the current image. Only retain those that meet the requirements. Connected components that meet the length condition are considered valid candidate line features, while other connected components that do not meet the length condition are considered invalid noise and discarded.

[0071] For each candidate connected component retained after filtering, the data processing unit obtains the set of coordinates of all pixels within it. The data processing unit uses the nonlinear least squares method to fit a straight line to the coordinate set. A general straight line equation model is established:

[0072] ;

[0073] in, Represents the horizontal coordinate variable in the image coordinate system; Represents the ordinate variable in the image coordinate system; In the equation of a straight line The coefficient reflects the horizontal component of the normal vector of the line; In the equation of a straight line The coefficient reflects the perpendicular component of the normal vector of the line; The constant term in the equation of a straight line reflects the distance between the line and the origin.

[0074] During the fitting process, normalization constraints are introduced. The optimal parameters are determined through iterative calculation. , and .

[0075] To quantify the accuracy of the fitted line, the data processing unit calculates the root mean square error (RMSE) of all pixels within the connected region relative to the fitted line. Based on the above normalization constraints, the RMS error... The calculation formula is as follows:

[0076] ;

[0077] in, The calculated root mean square error is used to measure the degree of dispersion of the pixel set from the fitted line. This represents the square root operation; This represents the weighting factor used to calculate the average of the sum of squared errors; This indicates the range from the first pixel to the second pixel. The values ​​of each pixel are summed up. Indicates the first The x-coordinate value of each pixel; Indicates the first The ordinate value of each pixel; In the equation of a straight line The coefficient reflects the horizontal component of the normal vector of the line; In the equation of a straight line The coefficient reflects the perpendicular component of the normal vector of the line; The constant term in the equation of the straight line reflects the distance between the line and the origin. In the formula... Item in Under the condition of, geometrically representing the first The vertical Euclidean distance from each pixel to the fitted line.

[0078] The data processing unit will calculate the root mean square error. With respect to the preset error threshold Perform a comparison. If... Greater than the error threshold If the length of the connected component meets the requirement, but its linearity is insufficient (possibly due to a curved pipe or an irregular reflective strip), it is discarded; if Less than or equal to the error threshold If the connected component is true, then it is considered a valid linear boundary. Error threshold. The error threshold is set based on the resolution of the current image. For example, for an image with a resolution of 640×480 pixels, the error threshold is... Set to 10 pixels; for images with a resolution of 1280×960 pixels, the error threshold is... The initial resolution is set to 20 pixels, and other resolutions are adjusted accordingly. This adaptive mechanism ensures that the algorithm maintains a consistent line selection standard across camera devices with varying resolutions.

[0079] See attached document Figure 1 and attached Figure 2 Following the line parameter extraction steps described above, the data processing unit executes step S5, which calculates the image rotation angle based on different cases. The data processing unit then compiles the set of valid lines retained in step S4. The number of lines included determines the angle calculation logic to be executed for each line.

[0080] When the effective set of lines When the data contains only a single straight line, the data processing unit directly obtains the fitting parameters of that straight line. and To address situations where poor lighting conditions at the coal mining face limit the extraction of only single, long-distance features (such as scraper conveyor chutes), the data processing unit uses this straight line as a horizontal reference. The unit then calculates the inclination angle of this line using the arctangent function, which serves as the rotation angle to be corrected. The calculation formula is as follows:

[0081] ;

[0082] in, Indicates the rotation angle of the image; Represents the arctangent function; In the equation of a straight line The coefficient reflects the horizontal component of the normal vector of the line; In the equation of a straight line The coefficient reflects the perpendicular component of the normal vector of the line; This represents the slope of the line on the image plane.

[0083] When the effective set of lines When a given set contains two straight lines, the data processing unit calculates the absolute values ​​of the slopes of both lines and determines the reference line based on the principle of minimum slope. Let the parameters of the two lines be... and The data processing unit calculates the absolute value of the slope of each straight line. :

[0084] ;

[0085] in, Indicates the first The absolute value of the slope of a straight line; Indicates the first The x-coordinate of the equation of a straight line The coefficient; Indicates the first The ordinate of the equation of a straight line The coefficient; The value can be 1 or 2, representing the first and second lines respectively.

[0086] Data processing unit comparison and The size. If The system determines that the first line is closer to horizontal and selects it as the reference line; otherwise, it selects the second line as the reference line. After selecting the reference line, the data processing unit obtains the parameters of the reference line. and And according to the formula Calculating the rotation angle can effectively eliminate interfering lines with large tilt angles in the image (such as the diagonal rods or columns of hydraulic supports), ensuring the accuracy of the calibration benchmark.

[0087] When the effective set of lines When a data set contains more than two straight lines, the data processing unit introduces a disappearance point geometric constraint mechanism to identify true parallel structures in three-dimensional space. The data processing unit constructs a set of line pairs and iterates through any two non-coincident lines in the valid line set. and For each pair of lines, the data processing unit solves the equations of the lines simultaneously and calculates the intersection point of the two lines on the image plane, i.e., the disappearance point. The x-coordinate of the disappearance point and ordinate The calculation is as follows:

[0088] ;

[0089] ;

[0090] in, and These represent the x and y coordinates of the disappearance point, respectively; , , For the first Fitting parameters for the straight lines; , , For the first Fitting parameters for a straight line.

[0091] During the calculation process, the data processing unit detects the denominator. If the denominator is less than a preset small threshold (for example, the small threshold can be set to 10), the value will be calculated. 6 This indicates that the two lines are approximately parallel on the image plane and do not have a valid intersection point within a finite distance. The data processing unit skips the calculation of this line pair.

[0092] The data processing unit collects all calculated disappearance points to form a set. The K-means clustering algorithm was used to construct a set of the disappearance points. Cluster analysis is performed. In practical applications, the uniqueness of disappearance points is determined by the fact that multiple disappearance points fall within a relatively small region. After clustering, the data processing unit counts the number of disappearance points in each cluster and selects the cluster with the largest number of disappearance points and the smallest intra-cluster variance as the target cluster. The target cluster represents the existence of a set of lines in the image that are parallel to each other in three-dimensional space (corresponding to rows of hydraulic supports or conveyor components), that is, determining that the set of lines are parallel in the physical world.

[0093] The data processing unit identifies the subset of lines corresponding to the disappearance points within the target cluster. Within this subset, the data processing unit recalculates the absolute slope of each line and selects the line with the smallest absolute slope as the final reference line. The data processing unit then obtains the parameters of the final reference line. and According to the formula Determine the final rotation angle of the image.

[0094] See attached document Figure 1 and attached Figure 2 Following the image rotation angle calculation steps described above, the data processing unit executes step S6, i.e., image correction. The data processing unit first determines the coordinates of the geometric center point of the rotation operation. The data processing unit calculates the reference line located on the image. The coordinates of the center point of the axis pixel count. Specifically, obtain the physical resolution width of the image. and the x-coordinate of the geometric center point of the rotation operation Set to half the image width, that is Subsequently, the data processing unit will Substitute the equation of the reference line determined in step S5 Calculate the ordinate of the geometric center point of the rotation operation on the reference line. The calculation formula is as follows:

[0095] ;

[0096] in, The ordinate of the geometric center point of the rotation operation; , , The equation coefficients of the reference line determined in step S5; The x-coordinate of the geometric center point of the rotation operation.

[0097] Determine the coordinates of the geometric center point of the rotation operation Then, the data processing unit calculates the rotation angle based on step S5. Construct an affine transformation matrix and perform a rotation operation on the preprocessed image. For any pixel in the image... Its rotated coordinates The calculation is as follows:

[0098] ;

[0099] ;

[0100] in, and These represent the x and y coordinates of the pixel after the rotation operation, respectively; and These represent the x and y coordinates of the pixel before rotation, respectively; Indicates the rotation angle of the image; and These represent the cosine function and the sine function, respectively.

[0101] The data processing unit uses bilinear interpolation to calculate coordinates. The pixel grayscale value is used to complete the geometric correction of the image, restoring the reference line in the image to a horizontal state. In addition to the bilinear interpolation algorithm, this embodiment can also use the nearest neighbor interpolation algorithm or the bicubic interpolation algorithm to calculate the pixel grayscale value, so as to achieve different balances between calculation speed and image quality.

[0102] The edge regions of the rotated image are then processed. Because the rotation operation causes displacement of the four corner points of the original image matrix, the background of the output image contains invalid black areas. The data processing unit iterates through all valid pixels (i.e., non-background fill pixels) in the rotated image and counts their position within the background. shaft and The coordinate extreme values ​​along the axis are used to calculate the minimum bounding rectangle that can contain all valid pixels. The data processing unit crops the rotated image according to the coordinate range of the minimum bounding rectangle, cropping to the smallest region that contains all valid pixels and removing invalid background areas.

[0103] Finally, step S7 is executed. The data processing unit sends the image data, after rotation correction and cropping, to the display unit. The display unit outputs and displays the processed coal face image. At this time, the scraper conveyor or support features in the image are in a horizontal position, making it easier for monitoring personnel to observe the actual situation of the working face.

[0104] See attached document Figure 3 The video image processing device for the coal mining face includes an image acquisition unit, a data processing unit, and a display unit connected in sequence via data transmission lines or buses.

[0105] The image acquisition unit is configured to be installed below the hydraulic support or roof beam of the coal mining face to capture real-time video streams of the coal mining face and convert optical signals into digital image signals. The output of the image acquisition unit is communicatively connected to the input of the data processing unit, transmitting the acquired raw image data to the data processing unit.

[0106] The data processing unit is equipped with a memory and a processor. The processor executes a computer program stored in the memory to implement image processing logic. Specifically, the processor can be one or more combinations of a central processing unit, a graphics processing unit, a digital signal processor, a field-programmable gate array, or an application-specific integrated circuit. From the perspective of functional module division, the data processing unit includes a distortion correction module, an image preprocessing module, a feature extraction module, an angle calculation module, and a geometric correction module, which are connected in sequence by signals.

[0107] The distortion correction module receives the raw image sent by the image acquisition unit at its input. Internally, the distortion correction module stores pre-calibrated camera intrinsic parameter data, which is used to perform geometric correction on the raw image according to a polynomial distortion model, eliminating distortion caused by the wide-angle lens, and then transmitting the generated distortion-free image to the image preprocessing module.

[0108] The image preprocessing module is connected to the distortion correction module and is used to receive the distorted image. The image preprocessing module is configured to first divide the image into multiple sub-blocks and perform contrast-limited adaptive histogram equalization to enhance the edge contrast of the cable trough and support structure; then, it uses a Gaussian filtering algorithm to smooth the image and filter out coal dust noise. The image preprocessing module simultaneously sends the preprocessed image to both the feature extraction module and the geometric correction module.

[0109] The feature extraction module is connected to the image preprocessing module. The feature extraction module is configured to perform line feature detection and parameter extraction on the preprocessed image. Specifically, the feature extraction module uses the maximum inter-class variance method to determine the threshold and performs binarization, connects broken edges through morphological closing operations, and extracts connected components using region growing. For connected components that meet the length condition, the feature extraction module uses nonlinear least squares to fit the line parameters and selects the effective line set based on the root mean square error. The feature extraction module then transmits the extracted effective line set and its parameters to the angle calculation module.

[0110] The angle calculation module, connected to the feature extraction module, receives a valid set of lines. It is configured to execute an adaptive angle calculation strategy based on the number of lines in the set: when the set contains only one line, it directly calculates the tilt angle of that line; when the set contains two lines, it selects the line with the smaller absolute slope as the reference line to calculate the tilt angle; when the set contains multiple lines, it constructs line pairs, solves for disappearance points, uses a clustering algorithm to identify spatial parallel structures, and selects the line with the smallest absolute slope from the target parallel structures as the reference line to calculate the tilt angle. The angle calculation module then sends the calculated rotation angle and reference line parameters to the geometric correction module.

[0111] The geometric correction module is connected to both the image preprocessing module and the angle calculation module. The geometric correction module receives the preprocessed image as the object of operation and the rotation angle and reference line parameters as the basis for transformation. It is configured to calculate the coordinates of the rotation center, construct an affine transformation matrix to perform reverse rotation correction on the preprocessed image, calculate the minimum bounding rectangle of the effective pixels in the corrected image, and crop invalid regions at the image edges.

[0112] The output of the geometric correction module is connected to the display unit, which sends the final image after rotation correction and cropping to the display unit for real-time display, so that the key equipment features of the coal mining face in the display screen remain horizontal.

Claims

1. A method for processing video images of a coal mining face, characterized in that, Includes the following steps: S1. Use the image acquisition unit to acquire the original video stream of the coal mining face scene and generate the original image data; S2. Use camera intrinsic parameter data to perform distortion correction on the original image data to generate a distortion-free image; S3. Enhance and smooth the distorted image to generate a preprocessed image. The enhancement process specifically includes: dividing the distorted image into multiple rectangular sub-blocks, calculating the gray-level histogram of the rectangular sub-blocks and setting a cropping limit value, and using an interpolation algorithm to smoothly fuse the boundary pixels between the rectangular sub-blocks to obtain the enhanced image; the smoothing process specifically involves using a Gaussian convolution kernel to traverse the pixels in the enhanced image, calculating the new gray-level value of the center pixel through convolution operations, and generating the preprocessed image. S4. Perform binarization and morphological operations on the preprocessed image, extract line features, and filter to obtain a set of valid lines. The binarization specifically includes: performing statistical analysis on the grayscale histogram of the preprocessed image using the maximum inter-class variance method, calculating the grayscale value that maximizes the variance between the foreground pixel class and the background pixel class, and using it as the optimal binarization threshold; using the optimal binarization threshold to segment the preprocessed image to obtain a binary image; the morphological operations specifically involve using a preset structuring element to first perform a dilation operation on the binary image, then perform an erosion operation to form a continuous boundary line candidate region. S5. Count the number of lines contained in the set of valid lines, determine the reference line based on the number of lines, and calculate the rotation angle based on the parameters of the reference line. S6. Rotate the preprocessed image using the rotation angle, and crop the rotated image to obtain the final image; S7. Output the final image through the display unit.

2. The method for processing video images of a coal mining face according to claim 1, characterized in that, In step S4, the extraction of line features and the filtering to obtain a set of valid lines specifically includes: The region growing algorithm is used to traverse the candidate regions of the boundary line and select connected components whose projected span length meets the preset conditions as effective linear features. Obtain the coordinate set of all pixels in the connected domain, perform line fitting on the coordinate set using the nonlinear least squares method, calculate the root mean square error of all pixels in the connected domain relative to the fitted line, and retain the fitted lines whose root mean square error is less than a preset error threshold to form the effective line set.

3. The method for processing video images of a coal mining face according to claim 2, characterized in that, The calculation of the root mean square error of all pixels in the connected region relative to the fitted straight line specifically includes: Under the constraint of normalizing the coefficients of the linear equation, the sum of squares of the vertical Euclidean distances from all pixels to the fitted line is calculated, the average value of the sum of squares is calculated, and the square root of the average value is taken to obtain the root mean square error.

4. The method for processing video images of a coal mining face according to claim 1, characterized in that, In step S5, determining the reference line based on the number of lines specifically includes: When the set of valid lines contains only one line, that line is determined as the reference line; When the set of valid lines contains two lines, the absolute values ​​of the slopes of the two lines are calculated respectively, and the line with the smallest absolute value of the slope is selected and determined as the reference line.

5. The method for processing video images of a coal mining face according to claim 1, characterized in that, In step S5, determining the reference line based on the number of lines specifically includes: When the set of valid lines contains more than two lines, construct a set of line pairs and calculate the intersection points of any two non-coincident lines to obtain a set of disappearance points. The set of disappearance points is classified using a clustering algorithm, and the cluster containing the most disappearance points and having the smallest intra-cluster variance is selected as the target cluster. A subset of straight lines corresponding to the disappearance points within the target cluster is determined, and the straight line with the smallest absolute slope value is selected from the subset of straight lines and determined as the reference straight line.

6. The method for processing video images of a coal mining face according to claim 5, characterized in that, The calculation of the intersection points of any two non-coincident lines to obtain the set of disappearance points specifically includes: By solving the equations of the two lines simultaneously, when the calculated value of the denominator of the two line equations is greater than a preset small threshold, the x-coordinate and y-coordinate of the intersection point are obtained and used as the disappearance point in the set of disappearance points.

7. The method for processing video images of a coal mining face according to claim 1, characterized in that, Step S6 specifically includes: Calculate the coordinates of the center point of the image width on the reference line as the rotation center, construct an affine transformation matrix, and use the rotation angle and the rotation center to perform a rotation operation on the preprocessed image to obtain the rotated image; Calculate the minimum bounding rectangle of the effective pixels in the rotated image, and crop the rotated image according to the minimum bounding rectangle to obtain the final image.

8. A video image processing device for a coal mining face, characterized in that, The method for processing video images of a coal mining face according to any one of claims 1-7 includes: The image acquisition unit is configured to capture real-time video streams of the coal mining face, convert optical signals into digital image signals, and then transmit the acquired raw image data to the data processing unit. The display unit is connected to the data processing unit and is used to output a corrected horizontal viewing angle monitoring image; The data processing unit includes a distortion correction module, an image preprocessing module, a feature extraction module, an angle calculation module, and a geometric correction module, which are connected in sequence by signals. The distortion correction module is used to perform distortion correction on the original image data using camera intrinsic parameter data to generate a distortion-free image. The image preprocessing module is used to enhance and smooth the distortion-reduced image to generate a preprocessed image. The feature extraction module is used to perform binarization and morphological operations on the preprocessed image, extract line features, and filter to obtain a set of valid lines; The angle calculation module is used to count the number of lines contained in the set of valid lines, determine the reference line based on the number of lines, and calculate the rotation angle based on the parameters of the reference line. The geometric correction module is used to rotate the preprocessed image using the rotation angle, crop the rotated image to obtain the final image, and send the final image to the display unit.

Images