Fully mechanized coal mining face dynamic video image fusion method

Abstract

The invention relates to a fully mechanized coal mining face dynamic video image fusion method. The method comprises the following steps: extracting representative feature points in an image; performing pairwise matching on the feature points by using normalized mutual closing and mutual information; a homography matrix between the two images is calculated through the coordinates of the corresponding points, and then a projective transformation function is called; one image is converted into a view angle of another image, in order to ensure robustness, a ransac algorithm is used to screen feature points, and obvious outboard points are filtered out; and performing grid discretization on the image to be spliced, calculating a homography matrix of each grid, and splicing the grids. According to the invention, high-precision image processing and real-time data transmission technologies are combined, the smoothness and accuracy of the video splicing effect are ensured, the video fusion rate is improved, a comprehensive and high-efficiency solution is provided to meet the application requirements of different scenes, and the method plays an important role in improving the mining efficiency of a fully mechanized coal mining face and ensuring safe production.

Description

Technical Field

[0001] The present invention belongs to video fusion technology, and in particular to a dynamic video image fusion method for a fully mechanized mining working face. Background Art

[0002] The full name of the fully mechanized coal mining face is a fully mechanized coal mining face, which is equipped with a series of closely coordinated mechanical equipment such as high-power coal mining machines, scraper conveyors, hydraulic supports, etc. In the fully mechanized mining face of a modern large coal mine, while the coal mining machine cuts coal on the coal seam, the scraper conveyor runs synchronously to transport the coal to the transfer machine, and the hydraulic support automatically moves forward according to a certain step distance, realizing a high degree of mechanization and automation of the coal mining process.

[0003] The comprehensive mining working face is equipped with a monitoring system. Some comprehensive mining working faces use a single monitor, which cannot comprehensively monitor the comprehensive mining working face in real time; some comprehensive mining working faces are installed with multiple monitors and multiple monitoring angles, which require frequent changes of angles, causing visual fatigue and prone to video loopholes. Real-time monitoring is not in place, and comprehensive monitoring of the comprehensive mining working face cannot be achieved. Summary of the invention

[0004] In view of this, the object of the present invention is to provide a dynamic video image fusion method for a fully mechanized mining working face to solve the problems raised in the above-mentioned background technology.

[0005] In order to achieve the above purpose, the technical solutions adopted are as follows:

[0006] The dynamic video image fusion method of the fully mechanized mining working face comprises the following steps:

[0007] S1: Monitor Installation

[0008] Multiple cameras are installed on the shearer support to monitor the fully mechanized mining face in real time;

[0009] S2: Camera calibration and projection transformation

[0010] Use the video reading module to read the Rtsp signal source of the camera and convert it into a video frame image. Manually select four points on a captured frame image to define the target rectangular area of ​​the projection transformation. Then, based on the locally preset orthodontic parameters, correct the image and save the configuration file locally in Npy format.

[0011] S3: Video reading, image registration and fusion stitching

[0012] Read multiple camera videos that need to be stitched, and apply the orthogonal file saved in step 2 to continuously perform image registration, brightness and contrast adjustment, orthogonalization and transformation on multiple captured video frames to ensure that the video frames meet the stitching standards. Then, use the algorithm to stitch multiple images side by side into a long video frame. Apply the video frame cutting technology at the junction of two images to cut the video frames of a certain width into a proportional number of vertical strips, calculate their fusion weights, and smoothly fuse the previous and next two frames of images together;

[0013] S4: Video and camera switching

[0014] Through the customized reading module, the bracket number where the current coal mining machine is located is continuously obtained, and compared and calculated with the configuration file to determine the center number of the current video stitching camera group. Based on the real-time comparison between the current number and the number read by the reading module, the video stitching screen will automatically refresh the content, reuse the corresponding image processing configuration file and parameters, and update the video in real time to ensure that the picture matches the position of the coal mining machine;

[0015] S5: Output stitched video

[0016] According to the image quality, it is divided into two versions: half and three quarters of the original quality. According to the video output type, it is divided into two versions: Rtsp and Rtmp. The output is a window version that is spliced ​​together from multiple videos, has a quality of three quarters of the original and is displayed on top.

[0017] As a further improvement of the present invention, SURF, SIFT, FAST, and ORB operators are used to extract representative feature points in the image.

[0018] As a further improvement of the present invention, normalized cross-correlation and mutual information are used to perform pairwise matching of feature points.

[0019] As a further improvement of the present invention, the homography matrix H between the two images is calculated by the coordinates of the corresponding points, and then the projection transformation function is called;

[0020]

[0021] Transform one image to the perspective of another image. To ensure robustness, the RANSAC algorithm is used to screen the feature points and filter out obvious outliers.

[0022] As a further improvement of the present invention, the image with splicing is discretized into grids, and the homography matrix of each grid is calculated and spliced ​​respectively.

[0023] The beneficial effects of the present invention are:

[0024] The present invention combines high-precision image processing and real-time data transmission technology to ensure the smoothness and accuracy of video stitching effects, improve the video fusion rate, and provide a comprehensive and efficient solution to meet the application needs of different scenarios. It plays an important role in improving the mining efficiency of comprehensive mining faces and ensuring safe production. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. In order to enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application are clearly and completely described. Obviously, the described embodiments are only embodiments of a part of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of this application.

[0026] The dynamic video image fusion method of the fully mechanized mining working face comprises the following steps:

[0027] S1: Monitor Installation

[0028] Multiple cameras are installed on the shearer support to monitor the fully mechanized mining face in real time;

[0029] S2: Camera calibration and projection transformation

[0030] Use the video reading module to read the Rtsp signal source of the camera and convert it into a video frame image. Manually select four points on a captured frame image to define the target rectangular area of ​​the projection transformation. Then, based on the locally preset orthodontic parameters, correct the image and save the configuration file locally in Npy format.

[0031] S3: Video reading, image registration and fusion stitching

[0032] Read multiple camera videos that need to be stitched, and apply the orthogonal file saved in step 2 to continuously perform image registration, brightness and contrast adjustment, orthogonalization and transformation on multiple captured video frames to ensure that the video frames meet the stitching standards. Then, use the algorithm to stitch multiple images side by side into a long video frame. Apply the video frame cutting technology at the junction of two images to cut the video frames of a certain width into a proportional number of vertical strips, calculate their fusion weights, and smoothly fuse the previous and next two frames of images together;

[0033] S4: Video and camera switching

[0034] Through the customized reading module, the bracket number where the current coal mining machine is located is continuously obtained, and compared and calculated with the configuration file to determine the center number of the current video stitching camera group. Based on the real-time comparison between the current number and the number read by the reading module, the video stitching screen will automatically refresh the content, reuse the corresponding image processing configuration file and parameters, and update the video in real time to ensure that the picture matches the position of the coal mining machine;

[0035] S5: Output stitched video

[0036] According to the image quality, it is divided into two versions: half and three quarters of the original quality. According to the video output type, it is divided into two versions: Rtsp and Rtmp. The output is a window version that is spliced ​​together from multiple videos, has a quality of three quarters of the original and is displayed on top.

[0037] In the present invention, feature extraction is performed, and operators such as SURF, SIFT, FAST, and ORB are used to extract representative feature points in the image.

[0038] Then, image registration is performed, and NCC (normalized cross correlation), MI (mutual information), etc. are used to match feature points in pairs.

[0039] Calculate the homography matrix: Calculate the homography matrix H between the two images through the coordinates of the corresponding points, and then call the projection transformation function;

[0040]

[0041] Transform one image to the perspective of another image. To ensure robustness, the RANSAC algorithm is used to screen the feature points and filter out obvious outliers.

[0042] Deformation and fusion: Discretize the image with stitching into as many grids as possible, calculate the homography matrix for each grid separately, and stitch them separately.

[0043] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A dynamic video image fusion method for a fully mechanized mining face, characterized in that: The following steps are involved: S1: Monitor Installation Multiple cameras are installed on the shearer support to monitor the fully mechanized mining face in real time; S2: Camera calibration and projection transformation Use the video reading module to read the Rtsp signal source of the camera and convert it into a video frame image. Manually select four points on a captured frame image to define the target rectangular area of ​​the projection transformation. Then, based on the locally preset orthodontic parameters, correct the image and save the configuration file locally in Npy format. S3: Video reading, image registration and fusion stitching Read multiple camera videos that need to be stitched, and apply the orthogonal file saved in step 2 to continuously perform image registration, brightness and contrast adjustment, orthogonalization and transformation on multiple captured video frames to ensure that the video frames meet the stitching standards. Then, use the algorithm to stitch multiple images side by side into a long video frame. Apply the video frame cutting technology at the junction of two images to cut the video frames of a certain width into a proportional number of vertical strips, calculate their fusion weights, and smoothly fuse the previous and next two frames of images together; S4: Video and camera switching Through the customized reading module, the bracket number where the current coal mining machine is located is continuously obtained, and compared and calculated with the configuration file to determine the center number of the current video stitching camera group. Based on the real-time comparison between the current number and the number read by the reading module, the video stitching screen will automatically refresh the content, reuse the corresponding image processing configuration file and parameters, and update the video in real time to ensure that the picture matches the position of the coal mining machine; S5: Output stitched video According to the image quality, it is divided into two versions: half and three quarters of the original quality. According to the video output type, it is divided into two versions: Rtsp and Rtmp. The output is a window version that is spliced ​​together from multiple videos, has a quality of three quarters of the original and is displayed on top.

2. The method for dynamic video image fusion of fully mechanized mining working face according to claim 1 is characterized in that: Use SURF, SIFT, FAST, and ORB operators to extract representative feature points in the image.

3. The method for dynamic video image fusion of fully mechanized mining working face according to claim 2 is characterized in that: Normalized cross-correlation and mutual information are used to perform pairwise matching of feature points.

4. The method for dynamic video image fusion of fully mechanized mining working face according to claim 3 is characterized in that: Calculate the homography matrix H between the two images through the coordinates of the corresponding points, and then call the projection transformation function; Transform one image to the perspective of another image. To ensure robustness, the RANSAC algorithm is used to screen the feature points and filter out obvious outliers.

5. The method for fusion of dynamic video images of fully mechanized mining working face according to claim 4 is characterized in that: The spliced ​​image is discretized into grids, and the homography matrix is ​​calculated for each grid and then spliced ​​separately.

Images