Adaptive coal mining machine tracking multi-camera video stitching method

Abstract

The present invention discloses an adaptive coal mining machine follow-up tracking multi-camera video stitching method, which belongs to the field of intelligent monitoring technology. The method combines the camera group adaptive follow-up tracking method with the coal mining machine video stitching algorithm to improve the stitching effect of the coal mining machine monitoring video under the condition of large parallax of the coal mining working face, and reduce the influence of the camera monitoring blind spot on the coal mining machine status monitoring; firstly, the position of the machine body is marked with a color fluorescent belt, and its real space coordinates are obtained through the camera calibration principle; based on the camera group adaptive follow-up tracking method, the cameras deployed on the coal mining working face are grouped and activated, and the video stitching task of the coal mining machine screen is completed by the currently activated camera group. The coal mining machine video stitching algorithm uses an image stitching algorithm to stitch the corresponding frame images of two monitoring videos, and selects SIFT feature points based on the RANSAC algorithm for the registration of the two images; finally, the two images are stitched by searching for the optimal stitching line to realize large-scale monitoring of coal mining machine operations.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent monitoring, and relates to a method for splicing videos of multiple cameras while adaptively tracking a coal mining machine. Background Art

[0002] Coal mining machines are important equipment for the mechanization and modernization of coal mine production. If a failure occurs, the entire coal mining face will be interrupted. Its operating status directly affects the safety level and economic production benefits of coal mines. Since the coal mining machine is a large and complex system integrating mechanical, electrical and hydraulic systems, its body is very long, with high operational complexity and the inability to detect working failures in time. Traditional video monitoring systems cannot cover the entire body, which will inevitably lead to monitoring blind spots and increase the rate of missed fault detection.

[0003] Therefore, the fault monitoring system for large coal mining machines requires the cooperation of multiple cameras to realize the coordinated acquisition of coal mining machine images under a large field of view span in order to complete the overall monitoring of the coal mining machine. In summary, in order to timely discover the operation failure of the coal mining machine, extend the service life of the coal mining machine, and avoid various economic losses caused by equipment failure, it is of great significance to design an adaptive coal mining machine tracking multi-camera video splicing method.

[0004] At present, the research on the stitching technology of underground video surveillance in coal mines has made certain progress. For example, Zhao Jianhong et al. proposed to use image processing units to realize the fusion of video images in order to solve the problem of poor real-time performance of video image stitching; Guan Zenglun et al. proposed to reduce the dimension of feature points and extract feature points only in the area of ​​interest to improve the accelerated robust algorithm to improve the real-time performance and stitching effect of the underground video stitching algorithm in order to solve the problem of low real-time performance of the accelerated robust feature algorithm; Du Chunhui et al. used the nonlinear anti-distortion model (NAM) correction algorithm to eliminate image distortion, used the accelerated robust feature (SURF) detection algorithm to detect feature points, and used the bilinear interpolation method to perform image fusion to achieve video stitching.

[0005] The above methods have proposed certain technical improvements for underground video surveillance splicing, but there is no adaptive dynamic scheduling for the camera group used for underground monitoring, and they are mainly oriented to the monitoring scenes of static targets or environments. The follow-up monitoring of dynamic large targets (such as coal mining machines) has not been fully studied. At the same time, the existing splicing algorithm cannot handle the blurring and distortion of video splicing results under large parallax conditions in mines, which limits the vertical expansion of the monitoring range under wide field of view and large parallax scenes. Summary of the invention

[0006] Purpose of the invention: In order to solve the problems existing in the prior art, the present invention provides an adaptive coal mining machine tracking multi-camera video stitching method.

[0007] Technical solution: In order to achieve the above-mentioned invention object, the present invention adopts the following technical solution:

[0008] The method for splicing videos of multiple cameras for adaptive coal mining machine tracking includes the following steps:

[0009] 1) Mark the ROI in the center of the coal mining machine as the positioning anchor point for the camera calibration algorithm;

[0010] 2) Use the color component feature extraction algorithm to retrieve the ROI anchor points in the images collected by each camera, and use a certain algorithm time interval t q Perform ROI anchor point polling retrieval on the video data of each camera in order from left to right. If a camera retrieves the anchor point, the subsequent cameras stop searching. With this camera as the center, the number of cameras on the left and right sides is determined according to the length of the body;

[0011] 3) Then, according to the mapping relationship between the image space coordinates and the real space coordinates in the camera calibration, the camera group to be called is adaptively determined and the video in the group is stitched, and finally the result is displayed on the monitor.

[0012] Furthermore, in step 1), the ROI marking is specifically as follows: first, the acquired image is converted from RGB to HSV color gamut, and the color factor threshold in the HSV space is determined according to the color of the color fluorescent band, and the rectangular ROI is identified based on this threshold, and the pixel position of the centroid of the rectangular ROI is obtained, and marked as the identification anchor point of the camera follow-up; then, the image space coordinates of the anchor point are projected into the real space coordinate system using the camera calibration method.

[0013] Furthermore, the camera calibration method comprises the following steps:

[0014] 1.1) Establish the image space coordinate system: The image space coordinate system is divided into the image physical coordinate system and the image pixel coordinate system: the image physical coordinate system xo0y uses the actual physical size as the unit of measurement, and the origin is located at o0; the pixel coordinate system uov uses the pixel point as the unit of measurement, and the origin is located at o; u and v represent the number of columns and rows of pixels in the digital image respectively; the x-axis and y-axis of the image coordinate system are parallel to the u-axis and v-axis of the pixel coordinate system and have the same direction; (u0, v0) is the pixel coordinate of the physical coordinate system origin o0, and the size of each pixel of the photosensitive device in the camera is dx*dy, then the transformation relationship from the physical coordinate system coordinate (x, y) to the pixel coordinate system coordinate (u, v) is expressed as:

[0015]

[0016] 1.2) Establish the camera coordinate system: camera coordinate system x c yc z c The optical axis of the camera is used as the Z axis, c is the optical center, x c Axis and y c The axes are parallel to the x-axis and y-axis of the image coordinate system respectively; o c o0 is the focal length of the camera f; world space coordinate system: a three-dimensional coordinate system defined by the user, used to describe the position of objects and cameras in three-dimensional space; X w , Y w , Z w Indicates; point p represents the imaging coordinates in the physical coordinate system of the image, and point b represents the coordinates of the object point in the camera coordinate system;

[0017] According to the principle of similar triangles, the relationship between the camera coordinate system and the image physical coordinate system is calculated as:

[0018]

[0019]

[0020]

[0021] 1.3) Establish the real space coordinate system: Translate and rotate the camera coordinates to obtain the coordinates of the ROI anchor point in the real space coordinate system:

[0022]

[0023] Where R represents a rotation matrix, t s Represents the translation vector, which is called the extrinsic parameter matrix. It is determined by factors such as the camera deployment position and its angle with the horizontal plane. It is used to represent the rotation and translation transformation relationship of the camera coordinates relative to the world coordinates. Finally, the world coordinates O of the ROI anchor point are calculated. a (X wa ,Y wa ,Z wa ).

[0024] Further in step 2), after determining the position of the ROI anchor point in space, the cameras deployed on the coal mining tunnel are grouped and activated using the camera group adaptive tracking method to ensure that the video monitoring range can be adaptively adjusted following the movement of the coal mining machine, so that the monitoring angle can cover the entire coal mining machine; the camera group adaptive tracking algorithm process is as follows:

[0025] 2.1) Initialization parameters: length of the shearer L, vertical distance between the shearer and the camera H ce , the distance between adjacent cameras d, the horizontal field of view angle of the camera θ h ;

[0026] 2.2) A single camera is labeled C i , {i|i=1,2,3…N}, where N represents the number of all cameras deployed in the lane; camera C i The world coordinate of i (X wc ,Y wc ,Z wc );

[0027] 2.3) Initialize the polling time interval t of the ROI anchor point coordinate recognition algorithm q , the anchor recognition algorithm running speed k (recognizes k pictures per second), the camera frame rate Fr (outputs Fr pictures per second), and the initial polling camera C1;

[0028] 2.4) Unit polling time t q Internal, camera output t q *Fr images, ROI recognition algorithm can process t q *k pictures, so we need to q * Randomly sample t from Fr images q *k sheets are processed by anchor point recognition algorithm;

[0029] 2.5) If an ROI anchor point is identified in the image, its real space coordinates O are calculated a (X wa ,Y wa ,Z wa ), let r = i and record the current camera as C r , execute step 6; if no anchor point is detected, set i=i+1 and execute step 2.4) for the next camera data;

[0030] 2.6) It is known that the horizontal length of the camera is 2H ce tan(θ h / 2), x h For camera C r Horizontal distance from the leftmost side of the field of view to the ROI, y h C r Horizontal distance from the rightmost side of the field of view to ROI:

[0031] x h =2H tan(θ h / 2)+|X wc -X wa | (1)

[0032] y h =2H tan(θ h / 2)-x h (2)

[0033] 2.7) Let C r The number of cameras required on the left is n L , the number of cameras required on the right is n R , then the following conditions are met:

[0034] x h +n L *d>L / 2 (3)

[0035] y h +n R *d>L / 2 (4)

[0036] 2.8) Find n L , n R , the camera group number range to be activated is

[0037] Further in step 3), the image stitching uses the RANSAC algorithm to filter the foreground of the coal mining machine body containing the most feature matching points, and removes irrelevant background feature matching points; finally, an optimal stitching line is found on the local plane to achieve the stitching of the input image; the activated camera number that monitors the current state of the coal mining machine At the same time, the images collected by each of them are I j , j is the index number of the camera in the activated camera group, j∈[rn L ,r+n R ]; The purpose of the stitching algorithm is to combine all the images I j Perform two-by-two splicing in sequence, from and Start until and Perform large parallax stitching on the surveillance images of each camera in the group.

[0038] The large parallax image stitching algorithm based on the optimal stitching line further comprises the following steps:

[0039] Input: image I to be stitched r and I t ; Output: splicing result I res ;

[0040] 3.1) Use SIFT algorithm to process the input image I r ,I t Perform feature point detection and matching respectively to obtain the initial feature point matching pair M0;

[0041] 3.2) Iterate the RANSAC algorithm on M0, let ω represent the number of iterations, and obtain a set of feature matching points M ω , and calculate its corresponding homography matrix Hω ;

[0042] 3.3) Update the initial feature matching point M0 = M0-M ω , ω=ω+1, repeat step 3.2) to get H ω ;

[0043] 3.4) Repeat step 3.3) until M0 < 10 and stop the loop to obtain the homography matrix set {H ω}, and the number of matrices in the set is n H ;

[0044] 3.5) Use the set {H ω}, let v be the set {H ω}, initialize v = 1, and let

[0045] 3.6) Calculate the local alignment image The corresponding stitching line loss function E(l), l represents the image attribute label of each pixel in the overlapping area of ​​the image to be stitched (which of the two images it belongs to);

[0046] 3.7) Use the graph cut algorithm to minimize the loss function and obtain the attribute label result l of the pixels in the overlapping area of ​​the image v , and record the current suture line energy E(l v );

[0047] 3.8) Let v = v + 1, update Repeat steps 3.6)-3.7) until v>n H , and obtain the loss function results of each seam line {E v}; Compare sets {E v}, select the local alignment image corresponding to the minimum value And synthesize the final splicing result along the seam line I res .

[0048] The SIFT algorithm is further implemented in two stages: the first stage is the establishment of the scale space; the second stage is the location of the extreme points;

[0049] The establishment of the scale space includes the following steps: first, the input image is taken as the bottom layer, and the results after each downsampling are stacked up in sequence to obtain an image pyramid; then, in order to keep the image scale in the pyramid continuous, a Gaussian function is used to blur each layer of the image pyramid to generate multiple images of different scales, thereby constructing a Gaussian pyramid; the calculation formula of Gaussian blur is:

[0050]

[0051] Where I(x,y) is the input image, is a convolution operation, L(x,y,σ) is an image with a scale of σ, G(x,y,σ) is a Gaussian function with a variable scale, and the Gaussian function expression is:

[0052]

[0053] Among them, (x, y) represents the position of the pixel in the image, σ is the standard deviation of the Gaussian function, which is also called the scale factor in the scale space; the Gaussian difference function is used instead of the differential operation, and the Gaussian difference function expression is:

[0054]

[0055] By comparing the size of each pixel with the pixels in the same layer and the two adjacent layers, the local extreme points in the Gaussian difference pyramid are found as candidate feature points;

[0056] The extreme point positioning is the search result of the extreme point in the scale space in the discrete condition. The method adopted is to perform curve fitting on the Gaussian difference pyramid space by fitting the three-dimensional quadratic function. The Maclaurin expansion of the three-dimensional quadratic function fitting the Gaussian difference function in the scale space is:

[0057]

[0058] in is the first-order partial derivative of the difference function D(x,y,σ) with respect to the position X(x,y), is the second-order partial derivative; by taking the derivative of D(X) and making the derivative result zero, we can find the offset of the extreme point:

[0059]

[0060] When the offset of the extreme point is greater than 0.5, it needs to be changed to the neighborhood position, and the extreme point position is determined by repeated iteration until the offset is less than 0.5; if the offset still does not meet the conditions after 5 iterations, the extreme point needs to be deleted;

[0061] Determine the direction for each feature point; when assigning a direction to a feature point, it is achieved by counting the gradient direction and gradient magnitude of the pixels in the neighborhood of the feature point. The calculation formula for the gradient direction T(x,y) and gradient magnitude a(x,y) of the pixel point p(x,y) is:

[0062]

[0063]

[0064] Calculate the gradient direction and magnitude of all pixels in the neighborhood of the feature point and express them in the form of a histogram; determine the direction corresponding to the histogram peak as the main direction of the feature point, and determine the direction higher than 80% of the main direction as the auxiliary direction of the SIFT feature point of the input image;

[0065] After obtaining the SIFT feature points of the input image, the K nearest neighbor algorithm is used to match the feature points of the two images to be stitched: first, the distances between all feature point descriptors in the feature point sets of the two images are calculated; then, for each feature point, the K closest points are selected as candidate matching points, and for each feature point, the two closest points are selected as matching candidates, and a ratio is obtained by taking the quotient between the closest distance and the second closest distance.

[0066] Further, the RANSAC algorithm uses an iterative method to find a set of feature matching point subsets that fit the global data among all SIFT feature matching points; the RANSAC algorithm finds a foreground feature matching point subset M and calculates the optimal homography matrix H; the steps of the RANSAC algorithm process are:

[0067] Input: initial feature matching point set M0; Output: optimal feature matching point (interior point) subset M, optimal homography matrix H;

[0068] 5.1) Assume that the number of iterations s = 1, and randomly select a subset M containing 4 pairs of feature matching points from the initial feature matching points M0 s As an interior point, calculate its homography matrix H s ;

[0069] 5.2) Use H to convert the remaining feature points s Transform and calculate the distance between the transformed feature point and its corresponding feature matching point. If the distance does not exceed the threshold ε, add this pair of feature points to the inner point set M s , record the number of all internal points n at this time s , and let s = s + 1, let M0 = M s ;

[0070] 5.3) Repeat steps 5.1)-5.2) until the number of iterations s is greater than the maximum value set, or the number of internal points n s Stop iteration when it is greater than the set maximum value;

[0071] 5.4) Select the set of interior points with the largest number as the optimal set of interior points M, and calculate the corresponding homography matrix H.

[0072] Furthermore, based on the optimal stitching line, I0 and I1 are two input images, Ω is the target overlapping area of ​​the two images, all the pixels on the left side of the stitching line come from I0, and all the pixels on the right side come from I1; Ω is the pixel space occupied by the fusion result (overlapping area), La = {0, 1} represents the set of optional labels, and each pixel point p∈Ω is labeled l p ∈La, the suture line can be obtained; when l p = 0, indicating that the RGB value of point p comes from I0. p =1, indicating that the RGB value of point p comes from I1; the purpose of the stitching algorithm is to find a set of labels l that can minimize the energy equation, and the energy equation expression is:

[0073]

[0074] E(l) is composed of the data item E data and the smoothing term E smooth N is the set of all adjacent pixel pairs, and the data item is the label l for pixel p. p The cost of the smoothing term is to label a pair of adjacent pixels (p, q) (l p ,l q ) cost; in the fusion result pixel space, the pixel point p position is in image I lp When there are valid RGB values ​​in E data (p,l p )=0, when there is no valid RGB value, E data (p,l p )=∞, where l p ∈{0, 1}.

[0075] Beneficial effect: Compared with the prior art, the adaptive coal mining machine tracking multi-camera video stitching method of the present invention combines the camera group adaptive tracking method with the coal mining machine video stitching algorithm, improves the stitching effect of the coal mining machine monitoring video under the condition of large parallax of the coal mining face, and reduces the influence of the camera monitoring blind spot on the coal mining machine status monitoring. First, a color fluorescent tape is used to artificially mark the machine body to mark the real position of the coal mining machine in space. The centroid of the rectangular mark is called the ROI (Regions of Interest) anchor point, which will be used as the positioning reference of the subsequent pixel space. The spatial position of the coal mining machine image is determined by extracting the color factor of the ROI anchor point, and then the image coordinates are mapped to the spatial coordinates by the camera calibration method to determine the real position of the coal mining machine in space. Based on the camera group adaptive tracking method, the cameras deployed on the coal mining face are grouped and activated, and the currently activated camera group completes the video stitching task of the coal mining machine screen, realizing adaptive tracking monitoring of the coal mining machine. The coal mining machine video stitching algorithm mainly uses the image stitching algorithm to stitch the corresponding frame images of two monitoring videos: first, SIFT (Scale-invariant feature transform) feature point detection is performed on the image, and the feature points are paired using the clustering algorithm. The scale and rotation invariance of the SIFT operator are used to improve the robustness of the matching point detection in the coal mining face scene to monitor the change of viewing angle, affine transformation, etc. Then, the RANSAC (RANdomSAmple Consensus) algorithm is used to screen the foreground feature matching points of the coal mining machine, remove irrelevant feature matching points, and perform image registration based on the obtained feature matching point group. Finally, the input images are stitched by searching for the optimal stitching line to achieve large-scale monitoring of the coal mining machine working process, timely discover possible operating faults in the coal mining machine working process, reduce production safety risks, and reduce direct economic losses caused by untimely fault handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a schematic diagram of a method for splicing videos of coal mining machines by adaptively following multiple cameras;

[0077] Figure 2 It is a schematic diagram of the ROI anchor point calibration method for coal mining machines;

[0078] Figure 3 It is a schematic diagram of camera calibration;

[0079] Figure 4 The camera adaptively follows the 2D image;

[0080] Figure 5 It is a schematic diagram of the process of large parallax coal mining machine image stitching algorithm based on the optimal stitching line;

[0081] Figure 6 It is a schematic diagram of the stitching order in the activation camera group;

[0082] Figure 7 This is the SIFT algorithm flow chart;

[0083] Figure 8 It is a schematic diagram of the scale space pyramid;

[0084] Fig. 9 It is a schematic diagram of the main direction distribution of feature points;

[0085] Fig.10 is a schematic diagram of stitch line image fusion;

[0086] Fig.11 are input video frames A, B, C and output video frame D;

[0087] Fig.12 are the input video frames A, B and the output video frame. DETAILED DESCRIPTION

[0088] The structure and performance of the present invention will be further described below in conjunction with the accompanying drawings.

[0089] like Figure 1-12 As shown in the figure, the adaptive coal mining machine tracking multi-camera video stitching method includes the following steps: manually mark the ROI in the central part of the coal mining machine (using a colored fluorescent tape) as the positioning anchor point of the camera calibration algorithm. Use the color component feature extraction algorithm to retrieve the ROI anchor point in the image collected by each camera, and use the color component feature extraction algorithm to extract the ROI anchor point at a certain algorithm time interval t q The ROI anchor point polling and retrieval of the video data of each camera is performed in order from left to right. If one or several cameras retrieve the anchor point, the subsequent cameras stop searching. Then, according to the mapping relationship between the pixel coordinates and the real coordinates in the camera calibration, the camera group to be called is adaptively determined and the video in the group is stitched, and the result is finally displayed on the monitor. Figure 1 As shown:

[0090] Coal mining machine ROI anchor point calibration

[0091] Due to the large size of the coal mining machine and the limited width of the coal mine tunnel, a single camera cannot monitor the complete coal mining machine screen. It is necessary to use a camera group to splice the monitoring screen to achieve complete monitoring of the working status of the coal mining machine. At the same time, due to the long depth of the coal mining machine working tunnel, the fixed camera group cannot perform uninterrupted follow-up monitoring of the coal mining machine in the working state. As the mining progresses, there will inevitably be monitoring blind spots. Therefore, it is necessary to track the position of the coal mining machine in real time to provide a reference for the adaptive follow-up algorithm of the camera group.

[0092] This method uses the ROI anchor point calibration method to determine the position of the coal mining machine: first, a colored fluorescent band is used to mark the center position of the coal mining machine with a rectangular frame ROI. Then, the ROI calibration area in the video data is retrieved based on the color component feature extraction algorithm, and t q The algorithm polling interval is , and the ROI anchor point is retrieved for the video captured by each camera according to the camera arrangement order, and the ROI is determined in the order from left to right. If a camera is retrieved first, the following cameras stop searching, and the current camera is used as the benchmark for adaptive video stitching.

[0093] The process of ROI anchor point calibration for coal mining machine is as follows: Figure 2 As shown. First, the acquired image is converted from RGB to HSV color gamut, and the color factor threshold in the HSV space is determined according to the color of the color fluorescent band. Based on this threshold, the rectangular ROI is identified, and the pixel position of the centroid of the rectangular ROI is obtained and marked as the recognition anchor point for camera tracking. Then, the pixel position coordinates of the anchor point are projected into the world space position coordinate system using the camera calibration method.

[0094] Camera Calibration Method

[0095] Image space coordinate system: The image space coordinate system can be divided into ① image physical coordinate system and ② image pixel coordinate system: The image physical coordinate system xo0y uses the actual physical size as the unit of measurement, with the origin at o0; the pixel coordinate system uov uses the pixel point as the unit of measurement, with the origin at o, u and v represent the number of columns and rows of pixels in the digital image respectively; the x-axis and y-axis of the image coordinate system are parallel to the u-axis and v-axis of the pixel coordinate system and have the same direction. (u0,v0) is the pixel coordinate of the physical coordinate system origin o0. The size of each pixel of the photosensitive device in the camera is dx*dy, so the transformation relationship from the physical coordinate system coordinate (x,y) to the pixel coordinate system coordinate (u,v) can be expressed as:

[0096]

[0097] Camera coordinate system: Camera coordinate system x c y c z c The optical axis of the camera is used as the Z axis, c is the optical center, x c Axis and y c The axes are parallel to the x-axis and y-axis of the image coordinate system respectively. Figure 3 Middle c o0 is the focal length f of the camera.

[0098] World space coordinate system: A three-dimensional coordinate system defined by the user, used to describe the position of objects and cameras in three-dimensional space.w , Y w , Z w express.

[0099] Figure 3 Point p represents the imaging coordinates in the image physical coordinate system, and point b represents the coordinates of the object point in the camera coordinate system. According to the principle of similar triangles, the relationship between the camera coordinate system and the image physical coordinate system can be deduced as follows:

[0100]

[0101]

[0102]

[0103] By translating and rotating the camera coordinates according to certain parameters, the coordinates of the ROI anchor point in the world coordinate system can be obtained:

[0104]

[0105] Where R represents a rotation matrix, t s Represents the translation vector, which is called the extrinsic parameter matrix. It is determined by factors such as the camera deployment position and its angle with the horizontal plane. It is used to represent the rotation and translation transformation relationship of the camera coordinates relative to the world coordinates. Finally, the world coordinates O of the ROI anchor point are calculated. a (X wa ,Y wa ,Z wa ).

[0106] Camera group adaptive tracking method

[0107] After determining the position of the ROI anchor point in space, the cameras deployed in the coal mining tunnel are grouped and activated using the camera group adaptive tracking method to ensure that the activated camera group can adaptively adjust following the movement of the coal mining machine, so that the monitoring angle can follow and cover the entire view of the coal mining machine.

[0108] like Figure 4 , given the length L of the coal mining machine body and the vertical distance H between the coal mining machine and the camera in the coal mining operation scenario ce , the distance between adjacent cameras d, the horizontal field of view angle of the camera θ h , Coal mining machine ROI anchor point position world coordinate O a (X wa ,Y wa ,Z wa ) and other conditions. The camera group required to be activated is obtained according to the following algorithm, so as to complete the follow-up acquisition of the complete image of the coal mining machine.

[0109] The process of the camera group adaptive tracking method is as follows:

[0110] 1) Initialization parameters: length of the shearer L, vertical distance between the shearer and the camera H ce , the distance between adjacent cameras d, the horizontal field of view angle of the camera θ h ;

[0111] 2) A single camera is labeled C i , {i|i=1,2,3…N}, where N represents the number of all cameras deployed in the lane. Camera C i The world coordinate of i (X wc ,Y wc ,Z wc );

[0112] 3) Initialize the polling time interval t of the ROI anchor point coordinate recognition algorithm q , the anchor recognition algorithm running speed k (can recognize k pictures per second), the camera frame rate Fr (output Fr pictures per second), and the initial polling camera C1;

[0113] 4) Unit polling time t q Inside, the camera output t q *Fr images, ROI recognition algorithm can process t q *k pictures, so we need to q * Randomly sample t from Fr images q *k sheets are processed by anchor point recognition algorithm;

[0114] 5) If a ROI anchor point is identified in the image, its world coordinates O are calculated a (X wa ,Y wa ,Z wa ), let r = i and record the current camera as C r , execute step 6; if no anchor point is detected, set i=i+1 and execute step 4 for the next camera data;

[0115] 6) It is known that the horizontal length of the camera is 2H ce tan(θ h / 2), x h For camera C r Horizontal distance from the leftmost side of the field of view to the ROI, y h C r Horizontal distance from the rightmost side of the field of view to ROI:

[0116] x h =2H tan(θ h / 2)+|X wc -Xwa | (1)

[0117] y h =2H tan(θ h / 2)-x h (2)

[0118] 7) Let C r The number of cameras required on the left is n L , the number of cameras required on the right is n R , the following conditions should be met:

[0119] x h +n L *d>L / 2 (3)

[0120] y h +n R *d>L / 2 (4)

[0121] 8) Find n L , n R , the camera group number range to be activated is

[0122] The conditions of the coal mining face are complex, and the monitoring video effect is easily affected by factors such as lighting, coal dust, and occlusion. Video stitching technology needs to first use the SIFT algorithm to describe the image information, and obtain feature description points that are highly robust to noise, light, and perspective changes. Then, based on the matching of feature points, the input image is projected on the plane to suppress blur, ghosting, and other phenomena in the stitching results. However, in the large parallax stitching scene of the coal mining machine, there are often multiple corresponding planes in the image, which can be roughly divided into the coal mining machine foreground plane and the irrelevant information background plane. Each plane requires a homography matrix H to realize the perspective transformation of the projection plane. The existing methods cannot achieve alignment on all spatial planes, and the homography matrix that can fit the most feature matching points must be selected as the image alignment model. This method uses the RANSAC algorithm to screen the coal mining machine body foreground containing the most feature matching points and eliminate irrelevant background feature matching points. Finally, an optimal stitching line is found on the local plane to achieve the stitching of the input image. As Figure 5 shown.

[0123] The camera group adaptive tracking method has obtained the number of the activated camera that monitors the current state of the coal mining machine. At the same time, the images collected by each of them are I j , j is the index number of the camera in the activated camera group, j∈[rn L ,r+n R ]. The purpose of the stitching algorithm is to combine all the images I j Splice them two by two in sequence. and Start until and The splicing order is Figure 6 As shown:

[0124] This method performs large parallax stitching on the surveillance images of each camera in the group. The large parallax image stitching algorithm process based on the optimal stitching line is as follows:

[0125] Input: image I to be stitched r and I t

[0126] Output: splicing result I res .

[0127] 1) Use SIFT algorithm to process the input image I r ,I t Perform feature point detection and matching respectively to obtain the initial feature point matching pair M0;

[0128] 2) Iterate the RANSAC algorithm on M0, let ω represent the number of iterations, and obtain a set of feature matching points M ω , and calculate its corresponding homography matrix H ω ;

[0129] 3) Update the initial feature matching point M0 = M0-M ω , ω=ω+1, repeat step 2) to get H ω ;

[0130] 4) Repeat step 3) until M0 < 10 and stop the loop to obtain the homography matrix set {H ω}, and the number of matrices in the set is n H ;

[0131] 5) Use the set {H ω}, let v be the set {H ω}, initialize v = 1, and let

[0132] 6) Calculate the local alignment image The corresponding stitching line loss function E(l), l represents the image attribute label of each pixel in the overlapping area of ​​the image to be stitched (which of the two images it belongs to);

[0133] 7) Use the graph cut algorithm to minimize the loss function and obtain the attribute label result l of the pixels in the overlapping area of ​​the image v , and record the current suture line energy E(l v );

[0134] 8) Let v = v + 1, update Repeat steps 6)-7) until v>n H , and obtain the loss function results of each seam line {E v}; Compare sets {E v}, select the local alignment image corresponding to the minimum value And synthesize the final splicing result along the seam line I res .

[0135] SIFT Algorithm

[0136] Scale-invariant feature transform (SIFT) is an algorithm for detecting local features. It is mainly used to detect and describe local features in images, find extreme points in the spatial scale, and extract their position, scale, and rotation invariants. The essence of the algorithm is to find key points (feature points) with prominent features in different scale spaces and locate their gradient directions, so that they can still express the characteristics of the image under changes in lighting, angle, position, and noise. The SIFT feature detection algorithm is implemented in two stages: the first stage is the establishment of the scale space, which represents the image in a multi-scale space by searching for stable points in all possible scales; the second stage is the precise positioning of the extreme points, accurately determining the position and scale of the extreme points, and removing unstable edge response points and low-contrast points. Figure 7 As shown, the main steps of the SIFT algorithm for image feature point extraction include:

[0137] Scale space construction

[0138] The construction of the scale space is to find candidate feature points that may be rotationally and scale-invariant within all scales of the image. The image pyramid can be represented by a Gaussian pyramid. First, the input image is taken as the bottom layer, and the results of each downsampling are stacked up in sequence to obtain the image pyramid. Then, in order to keep the image scale within the pyramid continuous, the Gaussian function and each layer of the image pyramid are used to blur, generate multiple images of different scales, and thus construct a Gaussian pyramid. The calculation formula for Gaussian blur is:

[0139]

[0140] Where I(x,y) is the input image, is a convolution operation, L(x,y,σ) is an image with a scale of σ, G(x,y,σ) is a Gaussian function with a variable scale, and the Gaussian function expression is:

[0141]

[0142] Among them, (x, y) represents the position of the pixel in the image, σ is the standard deviation of the Gaussian function, which is also called the scale factor in the scale space. The larger σ is, the larger the image scale is, and the smaller σ is, the smaller the image scale is. The constructed Gaussian pyramid structure is as follows Figure 8 (a) shown.

[0143] In order to obtain candidate feature points, the Gaussian difference function can be used instead of differential operations, which is faster and more efficient when detecting image extreme points. The Gaussian difference function expression is:

[0144]

[0145] It can be seen from the above formula that the Gaussian difference function can be realized by taking the difference between two adjacent layers of the Gaussian pyramid. The construction process of the Gaussian difference pyramid is as follows: Figure 8 (b) shows that by comparing the size of each pixel with the pixels in the same layer and the two adjacent layers, the local extreme points in the Gaussian difference pyramid are found as candidate feature points.

[0146] Precise positioning of extreme points:

[0147] The extreme points in the scale space are the retrieval results in discrete conditions, which are sensitive to noise and edges. Therefore, the detected local extreme points need to be further accurately positioned to become feature points. The method adopted is to curve fit the Gaussian difference pyramid space by fitting a three-dimensional quadratic function to accurately determine the position and scale of the key points. At the same time, in order to enhance the stability of matching, it is also necessary to remove low-contrast key points and unstable edge response points to improve the anti-noise ability.

[0148] The Maclaurin expansion of the three-dimensional quadratic function fitting the scale-space Gaussian difference function is:

[0149]

[0150] in is the first-order partial derivative of the difference function D(x,y,σ) with respect to the position X(x,y), is the second-order partial derivative; by taking the derivative of D(X) and making the derivative result zero, the offset of the extreme point can be obtained:

[0151]

[0152] When the extreme point offset is greater than 0.5, it needs to be changed to the neighborhood position, and the extreme point position is determined by repeated iteration until the offset is less than 0.5. If the offset still does not meet the conditions after 5 iterations, the extreme point needs to be deleted to ensure that all extreme points found are stable.

[0153] Feature point main direction assignment

[0154] In order to improve the robustness of the feature descriptor to image rotation, it is necessary to determine the direction for each feature point. When assigning a direction to a feature point, this can be achieved by counting the gradient direction and gradient magnitude of the pixels in the neighborhood of the feature point. The calculation formula for the gradient direction T(x,y) and gradient magnitude a(x,y) of the pixel point p(x,y) is:

[0155]

[0156]

[0157] Calculate the gradient direction and magnitude of all pixels in the neighborhood of the feature point and express them in the form of a histogram, such as Fig. 9 The direction corresponding to the histogram peak is determined as the main direction of the feature point, and the direction higher than 80% of the main direction is determined as the auxiliary direction of the feature point, so as to achieve the purpose of improving the stability of the key point.

[0158] Feature point pairing

[0159] After obtaining the SIFT feature points of the input image, the K nearest neighbor algorithm is used to match the feature points of the two images to be stitched: first, the distance between all feature point descriptors in the feature point set of the two images is calculated. Then, for each feature point, the K closest points are selected as candidate matching points. For each feature point, the two closest points are selected as matching candidates. The quotient between the closest distance and the second closest distance is used to obtain a ratio. If the ratio value is large, a pair of feature points corresponding to the closest distance is selected as the final feature matching point.

[0160] RANSAC algorithm

[0161] RANSAC can find a subset of feature matching points that can fit the global data among all SIFT feature matching points through an iterative method. Due to the significant feature of the coal mining machine as the main content of the image in the coal mining machine operation monitoring scenario, the feature matching point group obtained by the RANSAC algorithm can be used to describe the foreground coal mining machine plane.

[0162] The RANSAC algorithm searches for a subset of foreground feature matching points M and calculates the optimal homography matrix H.

[0163] Steps of the RANSAC algorithm flow:

[0164] Input: Initial feature matching point set M0

[0165] Output: optimal feature matching point (interior point) subset M, optimal homography matrix H

[0166] 1) Assume that the number of iterations s = 1, and randomly select a subset M containing 4 pairs of feature matching points from the initial feature matching points M0 s As an interior point, calculate its homography matrix H s ;

[0167] 2) Use H to convert the remaining feature points s Transform and calculate the distance between the transformed feature point and its corresponding feature matching point. If the distance does not exceed the threshold ε, add this pair of feature points to the inner point set M s , record the number of all internal points n at this time s , and let s = s + 1, let M0 = M s ;

[0168] 3) Repeat steps 1)-2) until the number of iterations s is greater than the maximum value set, or the number of internal points n s Stop iteration when it is greater than the set maximum value;

[0169] Select the set of interior points with the largest number as the optimal set of interior points M, and calculate the corresponding homography matrix H.

[0170] Optimal seam image fusion

[0171] Seam-based image fusion can be seen as a process of assigning labels to pixels on both sides of a seam. Different labels represent different sources of pixels. The seam is between two adjacent pixels from different input images.

[0172] like Fig.10 As shown, I0 and I1 are two input images, Ω is the target overlapping area of ​​the two images, and all the pixels on the left side of the stitching line come from I0, and all the pixels on the right side come from I1. Fig.11 (a)(b)(c) are input video frames A, B, C; Fig.11 (d) is the output video frame D; Fig.12 (a)(b) are input video frames A and B; Fig.12 (c) is the output video frame.

[0173] The optimal seam line requires that the pixels on both sides of it are as similar as possible in color and structure to suppress the traces of artificial splicing. To describe the similarity of the pixels on both sides of the seam line, it is modeled as a seam line loss function. The optimal seam line search is transformed into a problem of minimizing the loss function: Ω is the pixel space (overlapping area) occupied by the fusion result, La = {0, 1} represents the set of optional labels, and each pixel point p∈Ω is labeled l p ∈La, the suture line can be obtained. p = 0, indicating that the RGB value of point p comes from I0. p=1, indicating that the RGB value of point p comes from I1. The purpose of the stitching algorithm is to find a set of labels l that can minimize the energy equation. The energy equation is expressed as:

[0174]

[0175] E(l) is composed of the data item E data and the smoothing term E smooth N is the set of all adjacent pixel pairs, and the data item is the label l for pixel p. p The cost of the smoothing term is to label a pair of adjacent pixels (p, q) (l p ,l q ) cost. In the fusion result pixel space, the pixel point p position is in image I lp When there are valid RGB values ​​in E data (p,l p )=0, when there is no valid RGB value, E data (p,l p )=∞, where l p ∈{0, 1}.

Claims

1. An adaptive coal mining machine tracking multi-camera video stitching method, characterized by: The steps include: 1) Mark the ROI in the center of the coal mining machine as the positioning anchor point for the camera calibration algorithm; 2) Use the color component feature extraction algorithm to retrieve the ROI anchor points in the images collected by each camera, and use a certain algorithm time interval t q Perform ROI anchor point polling retrieval on the video data of each camera in order from left to right. If a camera retrieves the anchor point, the subsequent cameras stop searching. 3) Then, according to the mapping relationship between the image space coordinates and the real space coordinates in the camera calibration, the camera group to be called is adaptively determined and the video in the group is stitched, and finally the result is displayed on the monitor.

2. The adaptive coal mining machine tracking multi-camera video stitching method according to claim 1 is characterized by: In step 1), the ROI marking is specifically as follows: first, the acquired image is converted from RGB to HSV color gamut, the color factor threshold in the HSV space is determined according to the color of the color fluorescent band, the rectangular ROI is identified based on the threshold, and the image position coordinates of the centroid of the rectangular ROI are obtained, which are marked as the identification anchor point of the camera tracking; The image space coordinates of the anchor points are then projected into the real space coordinate system using the camera calibration method.

3. The adaptive coal mining machine tracking multi-camera video stitching method according to claim 2 is characterized by: The camera calibration method comprises the following steps: 1.1) Establish the image space coordinate system: The image space coordinate system is divided into the image physical coordinate system and the image pixel coordinate system: The image physical coordinate system xo0y uses the actual physical size as the unit of measurement, and the origin is located at o0; the image pixel coordinate system uov uses the pixel point as the unit of measurement, and the origin is located at o; u and v represent the number of columns and rows of pixels in the digital image respectively; the x-axis and y-axis of the image space coordinate system are parallel to the u-axis and v-axis of the pixel coordinate system and have the same direction; (u0, v0) is the pixel coordinate system coordinate of the physical coordinate system origin o0, and the size of each pixel of the photosensitive device in the camera is dx*dy, then the transformation relationship from the physical coordinate system coordinate (x, y) to the pixel coordinate system coordinate (u, v) is expressed as: 1.2) Establish the camera coordinate system: camera coordinate system x c y c z c The optical axis of the camera is used as the Z axis, c is the optical center, x c Axis and y c The axes are parallel to the x-axis and y-axis of the image space coordinate system respectively; o c o0 is the focal length of the camera f; point p represents the imaging coordinates in the image physical coordinate system, and point b represents the coordinates of the object point in the camera coordinate system; according to the principle of similar triangles, the relationship between the camera coordinate system and the image physical coordinate system is: 1.3) Establishing the real space coordinate system: The real space coordinate system is a user-defined three-dimensional coordinate system used to describe the position of objects and cameras in the real three-dimensional space. w , Y w , Z w express; By translating and rotating the camera coordinates, we can get the coordinates of the ROI anchor point in the real space coordinate system: Where R represents a rotation matrix, t s Represents the translation vector, which is called the extrinsic parameter matrix. It is determined by the camera deployment position and the angle between it and the horizontal plane. It is used to represent the rotation and translation transformation relationship between the camera coordinates and the world coordinates. Finally, the world coordinates O of the ROI anchor point are calculated. a (X wa ,Y wa ,Z wa ).

4. The adaptive coal mining machine tracking multi-camera video stitching method according to claim 1 is characterized by: In step 2), after determining the position of the ROI anchor point in space, the cameras deployed on the coal mining tunnel are grouped and activated using the camera group adaptive tracking method to ensure that the camera group can cover the entire coal mining machine after activation, so that the monitoring range can be adaptively adjusted following the movement of the coal mining machine; The process of the camera group adaptive tracking method is as follows: 2.1) Initialization parameters: length of the shearer L, vertical distance between the shearer and the camera H ce , the distance between adjacent cameras d, the horizontal field of view angle of the camera θ h ; 2.2) A single camera is labeled C i , {i|i=1,2,3…N}, where N represents the number of all cameras deployed in the lane; Camera C i The world coordinate of i (X wc ,Y wc ,Z wc ); 2.3) Initialize the polling time interval t of the ROI anchor point coordinate recognition algorithm q , anchor point recognition algorithm running speed k, camera frame rate Fr, initial polling camera C1; 2.4) Unit polling time t q Internal, camera output t q *Fr images, ROI recognition algorithm can process t q *k pictures, so we need to q * Randomly sample t from Fr images q *k sheets are processed by anchor point recognition algorithm; 2.5) If a ROI anchor point is identified in the image, its world coordinates O are calculated a (X wa ,Y wa ,Z wa ), let r = i and record the current camera as C r , execute step 6; if no anchor point is detected, set i=i+1 and execute step 2.4) for the next camera data; 2.6) It is known that the horizontal length of the camera is 2H ce tan(θ h / 2), x h For camera C r Horizontal distance from the leftmost side of the field of view to the ROI, y h C r Horizontal distance from the rightmost side of the field of view to ROI: x h =2Htan(θ h / 2)+|X wc -X wa |(1) y h =2Htan(θ h / 2)-x h (2) 2.7) Let C r The number of cameras required on the left is n L , the number of cameras required on the right is n R , then the following conditions are met: x h +n L *d>L / 2(3) y h +n R *d>L / 2(4) 2.8) Find n L , n R , the camera group number range to be activated is 5. The adaptive coal mining machine tracking multi-camera video stitching method according to claim 4 is characterized in that: In step 3), the image stitching is to use the RANSAC algorithm to select the foreground of the coal mining machine body containing the most feature matching points, and eliminate irrelevant background feature matching points; finally, find an optimal stitching line on the local plane to achieve the stitching of the input image; monitor the number of the activated camera of the current coal mining machine status At the same time, the images collected by each of them are I j , j is the index number of the camera in the activated camera group, j∈[rn L ,r+n R ]; The purpose of the stitching algorithm is to combine all the images I j Perform two-by-two splicing in sequence, from and Start until and Perform large parallax stitching on the surveillance images of each camera in the group.

6. The adaptive coal mining machine tracking multi-camera video stitching method according to claim 5 is characterized by: The large parallax image stitching algorithm based on the optimal stitching line includes the following steps: Input: image I to be stitched r and I t ; Output :Splicing result I res ; 3.1) Use SIFT algorithm to process the input image I r ,I t Perform feature point detection and matching respectively to obtain the initial feature point matching pair M0; 3.2) Iterate the RANSAC algorithm on M0, let ω represent the number of iterations, and obtain a set of feature matching points M ω , and calculate its corresponding homography matrix H ω ; 3.3) Update the initial feature matching point M0 = M0-M ω , ω=ω+1, repeat step 3.2) to get H ω ; 3.4) Repeat step 3.3) until M0 < 10 and stop the loop to obtain the homography matrix set {H ω }, and the number of matrices in the set is n H ; 3.5) Use the set {H ω }, let v be the matrix of the set {H ω }, initialize v = 1, and let 3.6) Calculate the local alignment image The corresponding stitching line loss function E(l), l represents the image attribute label of each pixel in the overlapping area of ​​the image to be stitched; 3.7) Use the graph cut algorithm to minimize the loss function, obtain the attribute label result lv of the pixels in the overlapping area of ​​the image, and record the current stitching line energy E(l v ); 3.8) Let v = v + 1, update Repeat steps 3.6)-3.7) until v>n H , and obtain the loss function results of each seam line {E v }; Compare sets {E v }, and select the local alignment image corresponding to the minimum seam line energy. And synthesize the final splicing result I along this optimal stitching line res .

7. The adaptive coal mining machine tracking multi-camera video stitching method according to claim 6 is characterized by: The SIFT algorithm is implemented in two stages: the first stage is the establishment of scale space; the second stage is the location of extreme points; The establishment of the scale space includes the following steps: first, the input image is taken as the bottom layer, and the results after each downsampling are stacked up in sequence to obtain an image pyramid; then, in order to keep the image scale in the pyramid continuous, each layer of the image pyramid is Gaussian blurred using a Gaussian function to generate multiple images of different scales, thereby constructing a Gaussian pyramid; the calculation formula of Gaussian blur is: Where I(x,y) is the input image, is a convolution operation, L(x,y,σ) is an image with a scale of σ, G(x,y,σ) is a Gaussian function with a variable scale, and the Gaussian function expression is: Among them, (x, y) represents the position of the pixel in the image, σ is the standard deviation of the Gaussian function, which is also called the scale factor in the scale space; the Gaussian difference function is used instead of the differential operation, and the Gaussian difference function expression is: By comparing the size of each pixel with the pixels in the same layer and the two adjacent layers, the local extreme points in the Gaussian difference pyramid are found as candidate feature points; The extreme point positioning is the search result of the extreme point in the scale space in the discrete condition. The method adopted is to perform curve fitting on the Gaussian difference pyramid space by fitting the three-dimensional quadratic function. The Maclaurin expansion of the three-dimensional quadratic function fitting the Gaussian difference function in the scale space is: in is the first-order partial derivative of the difference function D(x,y,σ) with respect to the position X(x,y), is the second-order partial derivative; by taking the derivative of D(X) and making the derivative result zero, we can find the offset of the extreme point: When the offset of the extreme point is greater than 0.5, it needs to be changed to the neighborhood position, and the extreme point position is determined by repeated iteration until the offset is less than 0.5; if the offset still does not meet the conditions after 5 iterations, the extreme point needs to be deleted; Determine the direction for each feature point; when assigning a direction to a feature point, this is achieved by counting the gradient direction and gradient magnitude of the pixels in the neighborhood of the feature point; Calculate the gradient direction and magnitude of all pixels in the neighborhood of the feature point and express them in the form of a histogram; determine the direction corresponding to the histogram peak as the main direction of the feature point, and determine the direction higher than 80% of the main direction as the auxiliary direction of the SIFT feature point of the input image; After obtaining the SIFT feature points of the input image, the K nearest neighbor algorithm is used to match the feature points of the two images to be stitched: first, the distances between all feature point descriptors in the feature point sets of the two images are calculated; then, for each feature point, the K closest points are selected as candidate matching points, and for each feature point, the two closest points are selected as matching candidates, and a ratio is obtained by taking the quotient between the closest distance and the second closest distance.

8. The adaptive coal mining machine tracking multi-camera video stitching method according to claim 7 is characterized by: The RANSAC algorithm uses an iterative method to find a set of feature matching point subsets that fit the global data among all SIFT feature matching points; the RANSAC algorithm finds the foreground feature matching point subset M and calculates the optimal homography matrix H; the steps of the RANSAC algorithm process are: Input: initial feature matching point set M0; Output: optimal feature matching point subset M, optimal homography matrix H; 5.1) Assume that the number of iterations s = 1, and randomly select a subset M containing 4 pairs of feature matching points from the initial feature matching points M0 s As an interior point, calculate its homography matrix H s ; 5.2) Use H to convert the remaining feature points s Transform and calculate the distance between the transformed feature point and its corresponding feature matching point. If the distance does not exceed the threshold ε, add this pair of feature points to the inner point set M s , record the number of all internal points n at this time s , and let s = s + 1, let M0 = M s ; 5.3) Repeat steps 5.1)-5.2) until the number of iterations s is greater than the maximum value set, or the number of internal points n s Stop iteration when it is greater than the set maximum value; 5.4) Select the set of interior points with the largest number as the optimal set of interior points M, and calculate the corresponding homography matrix H.

9. The adaptive coal mining machine tracking multi-camera video stitching method according to claim 8 is characterized in that: Based on the optimal stitching line, I0 and I1 are two input images, Ω is the target overlapping area of ​​the two input images, all the pixels on the left side of the stitching line come from I0, and all the pixels on the right side come from I1; Ω is the pixel space occupied by the fusion result, La = {0, 1} represents the set of optional labels, and each pixel point p∈Ω is labeled l p ∈La can get the suture line; When l p = 0, indicating that the RGB value of point p comes from I0. p =1, indicating that the RGB value of point p comes from I1; the purpose of the optimal seam search algorithm is to find a set of labels l that can minimize the energy equation. The energy equation expression is: E(l) is composed of the data item E data and the smoothing term E smooth N is the set of all adjacent pixel pairs, and the data item is the label l for pixel p. p The cost of the smoothing term is to label a pair of adjacent pixels (p, q) (l p ,l q ) cost; in the fusion result pixel space, the pixel point p position is in image I lp When there are valid RGB values ​​in E data (p,l p )=0, when there is no valid RGB value, E data (p,l p )=∞, where lp∈{0,1}.

Images