Multi-view monitoring video stitching method and system based on energy diagram estimation

Abstract

The invention discloses a multi-view monitoring video stitching method and system based on energy diagram estimation. The method comprises the following steps: acquiring video data and camera positions of a plurality of monitoring cameras of a road section to be stitched; extracting video frame features of each camera video, dividing pixels in video frames into background pixels or foreground pixels, and matching the background pixels; performing visual depth layering on the video frame by using a multi-layer RANSAC (Random Sample Consensus) method; local matching, layer-by-layer splicing and spliced layer fusion are carried out on video frames which are obtained by two cameras at different positions at the same moment and have overlapped areas layer by layer; and splicing the video frames shot by all the cameras to complete the splicing of the multi-view monitoring video to obtain a panoramic video. The method has the advantages of being small in calculation amount, high in calculation speed and good in splicing effect, the balance of algorithm efficiency and splicing precision is kept, and the method high in efficiency and robustness is provided for the field of surveillance video splicing.

Description

Technical Field

[0001] The present invention belongs to the technical field of video splicing, and in particular relates to a multi-view surveillance video splicing method and system based on energy graph estimation. Background Art

[0002] In recent years, with the continuous advancement of "smart transportation" construction projects in various places, the coverage of traffic monitoring has also been continuously expanded. However, a large number of complex monitoring screens make it difficult for managers to intuitively obtain the traffic conditions in the monitoring area. In the current traffic monitoring system, multiple monocular cameras in a small range are mostly used to monitor road traffic conditions. There are problems such as frequent switching of perspectives and fragmentation of spatial relationships between different screens, which is not conducive to rapid handling of traffic accidents, road congestion and other problems.

[0003] Video stitching is an extension of image stitching, which refers to the technology of seamlessly stitching several overlapping video sequences into wide-view or even panoramic videos. It creates a video with a wider field of view by merging a group of videos with overlapping fields of view. The typical video stitching process includes feature extraction, corresponding point matching, image registration, and image synthesis. Through feature extraction and corresponding point matching, corresponding points can be found between images of different perspectives. Then, based on these corresponding points, the image registration process converts the images into a coordinate system to align them. Since the aligned images have overlapping areas, the final image synthesis step determines the color information of the overlapping areas and their nearby positions.

[0004] Video stitching technology can solve the above problems in traffic monitoring management. By applying video stitching technology to surveillance videos, managers can intuitively understand the current road traffic situation. However, existing video stitching technology has multiple problems when facing the scene of traffic monitoring with multiple moving objects. In the stage of establishing the initial stitching model, although image registration has been widely studied to improve the quality of image alignment, it is still a challenging problem when there are large parallax, lens distortion and exposure differences between input images. Existing image registration algorithms assume that the input images should be captured by cameras that are far enough away from the scene to reduce parallax. However, this limits the application of video stitching to the field of surveillance videos with long placement intervals, large parallax, lens distortion and exposure differences. In the stage of continuous stitching and fusion of video frames, video stitching also faces visual artifacts caused by temporal inconsistency and time constraints. The transition of stitching position or changes in lighting conditions between consecutive frames will cause additional visual artifacts. In addition, traditional video fusion algorithms have the problems of large computational complexity and slow video generation speed, which makes it difficult to apply to the field of surveillance video stitching that requires fast response. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a multi-view surveillance video stitching method and system based on energy map estimation. In the stage of establishing the initial stitching model, the problem of difficulty in direct matching between surveillance videos due to large parallax is alleviated by layered local alignment, and an initial stitching template is obtained; in the stage of continuous splicing and fusion of video frames, the video stitching line is dynamically calculated by the energy map estimation algorithm, and the minimum change stitching line between video frames that bypasses the moving object is found, so as to efficiently deal with the obvious stitching line and object ghosting problems caused by the fast moving speed of objects and the presence of multiple moving objects in the surveillance video.

[0006] In order to achieve the above object, the present invention provides a multi-view surveillance video stitching method based on energy map estimation, comprising the following steps:

[0007] Step 1, obtaining video data and camera positions of multiple surveillance cameras of the road section to be spliced;

[0008] Step 2, using a feature extraction algorithm to extract video frame features of each camera video, using a background modeling algorithm to perform background modeling on the video frame, and dividing the pixels in the video frame into background pixels or foreground pixels;

[0009] Step 3, matching the background pixels in the video frames based on the nearest neighbor search, and completing the background matching of the video frames taken by different cameras;

[0010] Step 4: Based on the video frame features extracted in step 2, a multi-layer RANSAC method is used to perform visual depth stratification on the video frames with overlapping areas acquired by cameras at two different positions at the same time, and objects at different visual depths are separated into different layers;

[0011] Step 5, locally matching the video frames with overlapping areas acquired by cameras at two different positions at the same time layer by layer;

[0012] Step 6: Using a minimum energy-based stitching line search method, the video frames with overlapping areas acquired by cameras at two different positions at the same time are stitched layer by layer, and the stitched video frames of each layer are fused to complete the stitching of the two video frames;

[0013] Step 7, perform the stitching operation of steps 4 to 6 on the video frames captured by all cameras to complete the stitching of multi-view surveillance videos and obtain a panoramic video.

[0014] Furthermore, in step 2, a feature extraction algorithm is used to extract the video frame features of each camera video, including the color features and grayscale values ​​of the pixels in the video frame; a Gaussian mixture model GMM is used to model the background of the video frame, and the specific operations are as follows: ① Organize the extracted features of all camera video frames into a data set and divide them into a training set and a test set, the training set is used to train the GMM model, and the test set is used to test the training effect of the GMM model; ② Initialize the parameters of the GMM using a random initialization method, including the mean and variance of each Gaussian component, and the weight of each Gaussian component, and use the maximum expectation algorithm to iteratively update the parameters of the model to maximize the likelihood function. The trained GMM model contains the Gaussian component parameters of each feature dimension; ③ Input the extracted video frame features into the trained GMM model, calculate the probability that each pixel in the video frame belongs to the background, and classify the pixels according to the probability value, dividing them into background pixels or foreground pixels.

[0015] Furthermore, in step 3, the Euclidean distance between each background pixel in the video frame and the background pixels of other video frames is calculated. If the Euclidean distance between the background pixels is less than the set threshold, the two background pixels are considered to match. The above calculation and judgment operations are performed on all video frames in turn to complete the background matching of all video frames.

[0016] Furthermore, in step 4, the homography matrix of each layer is first calculated: Let F represent the matching feature point pairs, and Represent the i-th pair of feature points in the first video frame and the second video frame respectively, and their relationship is expressed as N is the total number of matched feature points; for each pair of feature points, an augmented matrix A and a target vector b are formed, which can be specifically expressed as:

[0017]

[0018]

[0019] In the formula, and Respectively represent the coordinates of the points in the matching feature point pair in the first video frame and the second video frame;

[0020] The homography matrix H maps points in one video frame to points in another video frame, thereby achieving matching of video images at different angles. The least squares method is used to estimate the parameters of the homography matrix. The calculation formula for estimating the homography matrix using the least squares method is:

[0021] h=argmin||Ah-b|| (3)

[0022] Where h is a 9-dimensional vector, representing the parameters of the homography matrix;

[0023] By rearranging the vector h obtained by the least squares method, we get the homography matrix H:

[0024]

[0025] Where h1, h2, h3, h4, h5, h6, h7, h8, h9 are vectors of each dimension in h;

[0026] The above matrix H has 9 parameters and 8 degrees of freedom, which requires 8 pairs of feature points to solve. For the first layer of RANSAC, 8 different pairs of feature points are extracted from all the feature point pairs multiple times. The homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature point pairs according to this homography matrix is ​​calculated. The homography matrix H1 with the smallest residual sum is taken for the first layer of visual depth stratification. The threshold T is set. After all feature point pairs are mapped using H1, those with residuals less than T are regarded as inliers, otherwise they are regarded as outliers. All the obtained inliers are taken as points in the first layer of visual depth; then different 8 pairs of feature points are extracted multiple times from all the external points, and the homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature point pairs according to this homography matrix is ​​calculated, and the homography matrix H2 with the smallest residual sum is taken for the second layer of visual depth stratification. After all feature point pairs are mapped using H2, those with residuals less than T are regarded as inliers and points in the second layer of visual depth; this is repeated until there are no external points and the visual depth stratification is completed.

[0027] Furthermore, in step 5, the video frames with overlapping areas acquired by the cameras at two different positions at the same time are matched layer by layer, aligned respectively, and the source video frames are divided into M×N grids. When matching each layer, different grids are locally matched respectively, and the feature points use the distance between the center point of the grid and the nearest neighbor point in different layers to select the deformation mode of the deformation mapping of the grid. The grid g j From its center point c j Indicates that the grid g j The homography matrix of By using weights Combined H of N0 layers k The calculation formula is:

[0028]

[0029]

[0030]

[0031] In the formula, H k Represents the homography matrix of the kth layer; Indicates the grid center point c in layer k j The closest feature point; || || 2 Represents the Euclidean norm, used to calculate feature points and the grid center point c j The distance between them; σ represents the standard deviation of the Gaussian function; is a Gaussian weight associated with the position, used to convert the distance into a weight value, so that closer positions have higher weights and farther positions have lower weights; It represents the overall proportion of Gaussian weight.

[0032] Exponential terms in formulas It represents the square of the Euclidean distance between two feature points divided by the variance of the Gaussian function. The smaller the value, the closer the two feature points are, and the larger the corresponding weight is.

[0033] Calculate the homography matrix for each grid, based on the homography matrix of the grid The grid g j The source video frame pixel at position p′ in The target position p in the target video frame is obtained to complete the matching.

[0034] Furthermore, in step 6, the energy map at the pixel (i, j) at time t is calculated The specific calculation formula is:

[0035]

[0036] In the formula, They represent the energy graphs at the positions of the video frame pixels (i, j) at time t and t-1 respectively. The attenuation factor α determines the contribution of the energy graph at the previous moment to the formation of the current energy graph. t (i, j) is the union of the pixels contained in any foreground object detected in the video frames of different perspectives at time t, that is:

[0037]

[0038] In the formula, ∨ represents the logical OR operation, and They represent the corresponding layer instance segmentation results in video frames of different perspectives, S t When (i, j) is in the overlapping area, the value is 1, otherwise, the value is 0, that is:

[0039]

[0040] Where Ω0 is the set of pixel positions of foreground objects in the overlapping area;

[0041] In order to solve the problem that long-term memory of object energy may produce invalid gap boundaries, the contribution of the energy map of the previous frames is adjusted by the following formula:

[0042]

[0043] Where τ is the threshold value of the time window that defines the cumulative energy. Moments with energy values ​​less than τ will not be considered, thereby preventing excessive redundant information from the past from interfering with the current energy map estimation. τ is set to Where N * is the number of frames to maintain the energy, and α is the attenuation factor. By adjusting the attenuation factor α, the contribution of each frame to the output energy map can be controlled.

[0044] The median point of the first row of the overlapping area is used as the starting point of the stitching line, and the minimum energy pixel point is found row by row as the stitching point, and the stitching line is obtained by connecting. If the median point of the first row is the foreground point, the non-foreground pixel point with the smallest offset from the column where the median point is located is selected as the starting point of the stitching line; when searching for the stitching line path, first compare the energy map on the left side of the current stitching point and the energy diagram on the right The size and energy diagram The calculation formula is as follows:

[0045]

[0046]

[0047]

[0048] Where (x, y) is the location of the current splicing point; (i, j) is the pixel position involved in the energy map judgment; is the energy map at the pixel (i, j) at time t; E p (i, j) is the distance contribution coefficient at the pixel (i, j); H is the height of the overlapping area; [] is the rounding operation; ψ() is an indicator function. hour, The value is 1, otherwise, the value is 0; β1 and β2 are weight parameters.

[0049] Select the side with smaller energy map to ensure that the stitching line extends in the direction with minimum estimated energy. To speed up the processing time, only check the 2p-1 pixels below the current stitching line point (i, j) on the side with smaller energy map each time. This calculation process can be achieved by using greedy search:

[0050]

[0051] In the formula, (x *,y * ) represents the pixel with the lowest energy, Represents the energy diagram at time t (i+1, k), when When k∈{yp,y-p+1,y}, When k∈{y-p+1,y,y+p-1}, , k∈{y,y+p-1,y+p}, p is the set offset, p≥2.

[0052] The minimum energy pixel searched is assigned as a stitching point only when it is unique and does not belong to a foreground object. If the minimum energy pixel does not belong to a foreground object but is not unique, the pixel with the smallest column offset from the current stitching point is selected as the stitching point. If the minimum energy pixel is unique but belongs to a foreground object, the pixel with the smallest energy among the non-foreground object pixels is selected as the stitching point.

[0053] Furthermore, in step 7, the video frames captured by all cameras are stitched together in steps 4 to 6. For the two stitched video frames, they are regarded as one frame and the stitching operations of steps 4 to 6 are performed with other video frames with overlapping areas, and finally the stitching of multi-view surveillance videos is completed to obtain a panoramic video.

[0054] The present invention also provides a multi-view surveillance video stitching system based on energy map estimation, which is used to implement the multi-view surveillance video stitching method based on energy map estimation as described above.

[0055] Furthermore, the invention comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute the multi-view surveillance video stitching method based on energy map estimation as described above.

[0056] Alternatively, it includes a readable storage medium having a computer program stored thereon, and when the computer program is executed, the multi-view surveillance video stitching method based on energy map estimation as described above is implemented.

[0057] Compared with the prior art, the present invention has the following advantages:

[0058] 1) The present invention fully considers the characteristics of large parallax and many moving objects in surveillance videos, and adopts a local matching alignment algorithm in the initial splicing template establishment stage, thereby avoiding the image deformation problem caused by large parallax;

[0059] 2) In the stage of finding the video splicing line, the present invention adopts a path finding method based on energy map estimation. According to the characteristic that the pixels of the foreground object have large energy on the energy map, the problem is converted into finding the path with the minimum energy. By comparing the size of the energy maps on the left and right sides of the current splicing point, it is ensured to extend in the direction with the minimum estimated energy, thereby solving the problem of obvious splicing lines and object ghosting on the objects caused by the fast moving speed of objects and the presence of multiple moving objects in the surveillance video;

[0060] 3) The present invention has the characteristics of small calculation amount, fast calculation speed and good stitching effect, maintains the balance between algorithm efficiency and stitching accuracy, and provides a method with high efficiency and parallax robustness for the field of surveillance video stitching. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The present invention is a flowchart of a multi-view surveillance video stitching method based on energy map estimation according to an embodiment of the present invention.

[0062] Figure 2 Schematic diagram of the splicing line searching process according to an embodiment of the present invention.

[0063] Figure 3 This is a graph showing the result of finding the stitching line according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention provides a multi-view surveillance video stitching method and system based on energy map estimation. The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0065] Example 1

[0066] like Figure 1 As shown, an embodiment of the present invention provides a multi-view surveillance video stitching method based on energy map estimation, comprising the following steps:

[0067] Step 1: Obtain video data and camera positions of multiple surveillance cameras on the road section to be spliced.

[0068] Step 2: Use a feature extraction algorithm to extract the video frame features of each camera video, use a background modeling algorithm to perform background modeling on the video frame, and divide the pixels in the video frame into background pixels or foreground pixels.

[0069] This embodiment uses the SIFT algorithm to extract the video frame features of each camera video, including the color features and grayscale values ​​of the pixels in the video frame. The Gaussian mixture model GMM is used to model the background of the video frame. The specific operations are as follows: ① Organize all the extracted camera video frame features into a data set and divide them into a training set and a test set. The training set is used to train the GMM model, and the test set is used to test the training effect of the GMM model. ② Initialize the parameters of the GMM by random initialization, including the mean and variance of each Gaussian component, and the weight of each Gaussian component. Use the Expectation Maximization Algorithm (EM) to iteratively update the parameters of the model to maximize the likelihood function. The GMM model after training contains the Gaussian component parameters (mean and variance) of each feature dimension; ③ Input the extracted video frame features into the trained GMM model, calculate the probability that each pixel in the video frame belongs to the background, and classify the pixels according to the probability value, dividing them into background (non-moving object) pixels or foreground (moving object) pixels.

[0070] Step 3, matching the background pixels in the video frames based on the nearest neighbor search to complete the background matching of the video frames shot by different cameras.

[0071] Calculate the Euclidean distance between each background pixel in a video frame and the background pixels of other video frames. If the Euclidean distance between the background pixels is less than the set threshold, the two background pixels are considered to match. Perform the above calculation and judgment operations on all video frames in turn to complete the background matching of all video frames. Through background matching, the correspondence between the pixels with the same name in the background part of the video frames of different cameras is established, the fusion and unification of the background part of the video frames is completed, the incoherence of the background color is reduced, and the foundation is laid for subsequent video splicing, and the quality of the subsequent spliced ​​videos is improved.

[0072] Step 4: Based on the video frame features extracted in step 2, a multi-layer RANSAC method is used to perform visual depth stratification on the video frames with overlapping areas acquired by cameras at two different positions at the same time, and objects at different visual depths are separated into different layers.

[0073] First, calculate the homography matrix of each layer: Let F represent the matching feature point pairs, and Represent the i-th pair of feature points in the first video frame and the second video frame respectively, and their relationship is expressed as N is the total number of matched feature points; for each pair of feature points, an augmented matrix A and a target vector b are formed, which can be specifically expressed as:

[0074]

[0075]

[0076] In the formula, and Respectively represent the coordinates of the points in the matching feature point pair in the first video frame and the second video frame.

[0077] The homography matrix H maps points in one video frame to points in another video frame, thereby achieving matching of video images at different angles. The least squares method is used to estimate the parameters of the homography matrix. The calculation formula for estimating the homography matrix using the least squares method is:

[0078] h=argmin||Ah-b|| (3)

[0079] Where h is a 9-dimensional vector representing the parameters of the homography matrix.

[0080] By rearranging the vector h obtained by the least squares method, we get the homography matrix H:

[0081]

[0082] Where h1, h2, h3, h4, h5, h6, h7, h8, and h9 are vectors of various dimensions in h.

[0083] The above matrix H has 9 parameters and 8 degrees of freedom, which requires 8 pairs of feature points to solve. For the first layer of RANSAC, 8 different pairs of feature points are extracted from all the feature point pairs multiple times. The homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature point pairs according to this homography matrix is ​​calculated. The homography matrix H1 with the smallest residual sum is taken for the first layer of visual depth stratification. The threshold T is set. After all feature point pairs are mapped using H1, those with residuals less than T are regarded as inliers, otherwise they are regarded as outliers. All the obtained inliers are taken as points in the first layer of visual depth; then different 8 pairs of feature points are extracted multiple times from all the external points, and the homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature point pairs according to this homography matrix is ​​calculated, and the homography matrix H2 with the smallest residual sum is taken for the second layer of visual depth stratification. After all feature point pairs are mapped using H2, those with residuals less than T are regarded as inliers and points in the second layer of visual depth; this is repeated until there are no external points and the visual depth stratification is completed.

[0084] Step 5: Locally match the video frames with overlapping areas acquired by cameras at two different positions at the same time layer by layer.

[0085] The video frames acquired by two cameras at different positions at the same time are matched layer by layer and aligned separately to avoid object deformation caused by overall image matching and obtain a smooth transition overall. In order to simplify the matching calculation and reduce the computational complexity of the splicing method, the source video frame is divided into M×N grids. When matching each layer, different grids are locally matched. The feature point needs to use the distance between the center point of the grid and the nearest neighbor point in different layers to select the deformation mode of the grid deformation mapping. The grid g j From its center point c j Represents. Grid g j The homography matrix of By using weights Combined H of N0 layers k The calculation formula is:

[0086]

[0087]

[0088]

[0089] In the formula, H k Represents the homography matrix of the kth layer; Indicates the grid center point c in layer k j The closest feature point; || || 2 Represents the Euclidean norm, used to calculate feature points and the grid center point c j The distance between them; σ represents the standard deviation of the Gaussian function; is a Gaussian weight associated with the position, used to convert the distance into a weight value, so that closer positions have higher weights and farther positions have lower weights; It represents the overall proportion of Gaussian weight.

[0090] Exponential terms in formulas It represents the square of the Euclidean distance between two feature points divided by the variance of the Gaussian function. The smaller the value, the closer the two feature points are, and the larger the corresponding weight is.

[0091] Calculate the homography matrix for each grid, based on the homography matrix of the grid The grid g j The source video frame pixel at position p′ in The target position p in the target video frame is obtained to complete the matching.

[0092] The correspondence between the pixel positions of the source video frame and the pixel positions of the target video frame is stored in a pixel mapping table. When performing video frame matching, the transformed video frame can be obtained by searching for the corresponding pixels in the source video frame instead of repeatedly transforming, thereby speeding up the calculation.

[0093] Step 6, using the minimum energy-based stitching line search method, the video frames with overlapping areas acquired by cameras at two different positions at the same time are stitched layer by layer, and the stitched video frames of each layer are fused to complete the stitching of the two video frames.

[0094] Calculate the energy representation of objects located in different layers in the overlapping area of ​​two video frames obtained by cameras at different positions (Formulas (8)-(10)). If it is a foreground object, it has a higher importance and higher energy, and the stitching line should bypass this part of the pixels. Otherwise, if it is a background part, the stitching line is located here and ghosting will not be generated due to the movement of the object.

[0095] For the energy of the foreground object, the instance segmentation result of the foreground object can be represented by the binary map S t (i,j) indicates that S t (i,j) represents the union of the pixels contained in any foreground object detected in video frames of different perspectives:

[0096]

[0097] In the formula, ∨ represents the logical OR operation. and They represent the corresponding layer instance segmentation results in video frames of different perspectives, S t When (i, j) is in the overlapping area, the value is 1, otherwise, the value is 0, that is:

[0098]

[0099] Where Ω0 is the set of pixel positions of foreground objects in the overlapping region.

[0100] For a certain pixel position at a certain moment, the energy map of the pixel at this moment is calculated by combining the union of the energy map of the previous moment and the pixels contained in the foreground object at the current moment, so that the energy map has temporal coherence.

[0101] Energy map at pixel (i, j) at time t The calculation formula is:

[0102]

[0103] In the formula, Represent the energy graphs of the video frames at time t and t-1 respectively, St (i, j) is the union of the pixels contained in any foreground object detected in the video frames of different perspectives at time t, and the attenuation factor α determines the contribution of the energy map at the previous moment to the formation of the current energy map.

[0104] Due to its recursive structure, the time-consistent energy graph It is actually the weighted sum of a set of object energies from 0 to time t, and its importance decays over time. That is, the current energy map will remember the objects that moved through the previous frame without storing the previous frames, reducing the need for storage space. Specifically, in order to solve the problem that long-term memory of object energy may produce invalid gap boundaries, the contribution of the energy maps of the previous frames is adjusted by the following formula:

[0105]

[0106] Where τ is the threshold of the time window that defines the cumulative energy. Moments with energy values ​​less than τ will not be considered, thereby preventing excessive redundant information from past moments from interfering with the current energy map estimation.

[0107] Specifically, τ can be set as Where N * is the number of frames to maintain the energy, and α is the attenuation factor. By adjusting the attenuation factor α, the contribution of each frame to the output energy map can be controlled.

[0108] The middle point of the first row of the overlapping area is used as the starting point of the stitching line, and the minimum energy pixel point is searched row by row as the stitching point, and the stitching line is obtained by connecting. If the middle point of the first row is a foreground point, the non-foreground pixel point with the smallest offset from the column where the middle point is located is selected as the starting point of the stitching line. When searching for the stitching line path, in order to avoid sudden stitching lines on each row, first compare the energy map on the left side of the current stitching point. and the energy diagram on the right Select the side with the smaller energy map to ensure that the stitching line area selected below has lower energy as a whole (i.e. contains fewer foreground objects), so as to make the stitching line better overall and consider more pixels (avoid being misled by local low-energy pixels).

[0109] Calculate the energy map on the left side of the current splicing point and the energy diagram on the right The calculation formula is as follows:

[0110]

[0111]

[0112]

[0113] Where (x, y) is the location of the current splicing point; (i, j) is the pixel position involved in the energy map judgment; is the energy map at the pixel (i, j) at time t; E p (i, j) is the distance contribution coefficient at the pixel (i, j); H is the height of the overlapping area; [] is the rounding operation; ψ() is an indicator function. hour, The value is 1, otherwise, the value is 0; β1 and β2 are weight parameters;

[0114] Select the side with smaller energy map to ensure that the stitching line extends in the direction with minimum estimated energy. To speed up the processing time, only check the 2p-1 pixels below the current stitching line point (i, j) on the side with smaller energy map each time. This calculation process can be achieved by using greedy search:

[0115]

[0116] In the formula, (x * ,y * ) represents the pixel with the lowest energy, Represents the energy diagram at time t (i+1, k), when When k∈{yp,y-p+1,y}, When k∈{y-p+1,y,y+p-1}, When k∈{y,y+p-1,y+p}, p is the set offset, p≥2, and the value of p in this embodiment is 2.

[0117] The minimum energy pixel searched is assigned as a stitching point only when it is unique and does not belong to a foreground object. If the minimum energy pixel does not belong to a foreground object but is not unique, the pixel with the smallest column offset from the current stitching point is selected as the stitching point. If the minimum energy pixel is unique but belongs to a foreground object, the pixel with the smallest energy among the non-foreground object pixels is selected as the stitching point.

[0118] like Figure 2 As shown, the grid represents pixels, the gray shadow part represents the pixels of the foreground object, and the position selected by the red dotted line is the energy map on the left side of the current splicing point (the black dot in the first row) The calculation range, the position selected by the green dotted line is the current splicing point (the black dot in the first row) Energy diagram on the right The energy diagram on the left side of this embodiment is smaller than the energy diagram on the right side, that is, Therefore, the three dots within the blue frame below the current stitching point are compared. Since the white dot directly below the current stitching point belongs to the pixel of the foreground object, the pixel with the smallest energy among the two dots on the left is selected as the stitching point. In this embodiment, the black dot has the smallest energy and is used as the stitching point.

[0119] Figure 3 The splicing path calculated by the splicing line finding algorithm proposed in the present invention is shown.

[0120] Step 7, perform the stitching operation of steps 4 to 6 on the video frames captured by all cameras to complete the stitching of multi-view surveillance videos and obtain a panoramic video.

[0121] Perform the stitching operation of steps 4 to 6 on the video frames taken by all cameras. For the two stitched video frames, treat them as one frame and perform the stitching operation of steps 4 to 6 with other video frames with overlapping areas. Finally, complete the stitching of multi-view surveillance videos to obtain a panoramic video.

[0122] Example 2

[0123] Based on the same inventive concept, the present invention also provides a multi-view surveillance video stitching system based on energy graph estimation, including a processor and a memory, the memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute a multi-view surveillance video stitching method based on energy graph estimation as described above.

[0124] Example 3

[0125] Based on the same inventive concept, the present invention also provides a multi-perspective surveillance video stitching system based on energy graph estimation, including a readable storage medium, on which a computer program is stored. When the computer program is executed, a multi-perspective surveillance video stitching method based on energy graph estimation as described above is implemented.

[0126] In specific implementation, the method proposed in the technical solution of the present invention can be implemented by technical personnel in this field using computer software technology to realize automatic operation process. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of the present invention and computer equipment running the corresponding computer program, should also be within the scope of protection of the present invention.

[0127] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific implementation cases described, or replace them with similar methods, without departing from the spirit of the present invention or exceeding the scope defined by the appended claims.

Claims

1. A multi-view surveillance video stitching method based on energy map estimation, characterized in that: The following steps are involved: Step 1, obtaining video data and camera positions of multiple surveillance cameras of the road section to be spliced; Step 2, using a feature extraction algorithm to extract video frame features of each camera video, using a background modeling algorithm to perform background modeling on the video frame, and dividing the pixels in the video frame into background pixels or foreground pixels; Step 3, matching the background pixels in the video frames based on the nearest neighbor search, and completing the background matching of the video frames taken by different cameras; Step 4: Based on the video frame features extracted in step 2, a multi-layer RANSAC method is used to perform visual depth stratification on the video frames with overlapping areas acquired by cameras at two different positions at the same time, and objects at different visual depths are separated into different layers; Step 5, locally matching the video frames with overlapping areas acquired by cameras at two different positions at the same time layer by layer; Step 6: Using a minimum energy-based stitching line search method, the video frames with overlapping areas acquired by cameras at two different positions at the same time are stitched layer by layer, and the stitched video frames of each layer are fused to complete the stitching of the two video frames; Step 7, perform the stitching operation of steps 4 to 6 on the video frames captured by all cameras to complete the stitching of multi-view surveillance videos and obtain a panoramic video.

2. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: In step 2, a feature extraction algorithm is used to extract the video frame features of each camera video, including the color features and grayscale values ​​of the pixels in the video frame; a Gaussian mixture model GMM is used to model the background of the video frame. The specific operations are as follows: ① Organize the extracted features of all camera video frames into a data set and divide them into a training set and a test set. The training set is used to train the GMM model, and the test set is used to test the training effect of the GMM model. ② Initialize the parameters of the GMM using a random initialization method, including the mean and variance of each Gaussian component and the weight of each Gaussian component. Use the maximum expectation algorithm to iteratively update the parameters of the model to maximize the likelihood function. The trained GMM model contains the Gaussian component parameters of each feature dimension. ③ Input the extracted video frame features into the trained GMM model, calculate the probability that each pixel in the video frame belongs to the background, and classify the pixels according to the probability value, dividing them into background pixels or foreground pixels.

3. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: In step 3, the Euclidean distance between each background pixel in the video frame and the background pixels of other video frames is calculated. If the Euclidean distance between the background pixels is less than the set threshold, the two background pixels are considered to match. The above calculation and judgment operations are performed on all video frames in turn to complete the background matching of all video frames.

4. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: In step 4, the homography matrix of each layer is first calculated: Let F represent the matching feature point pairs, and Represent the i-th pair of feature points in the first video frame and the second video frame respectively, and their relationship is expressed as N is the total number of matched feature points; for each pair of feature points, an augmented matrix A and a target vector b are formed, which can be specifically expressed as: In the formula, and Respectively represent the coordinates of the points in the matching feature point pair in the first video frame and the second video frame; The homography matrix H maps points in one video frame to points in another video frame, thereby achieving matching of video images at different angles. The least squares method is used to estimate the parameters of the homography matrix. The calculation formula for estimating the homography matrix using the least squares method is: h=argmin||Ah-b|| (3) Where h is a 9-dimensional vector, representing the parameters of the homography matrix; By rearranging the vector h obtained by the least squares method, we get the homography matrix H: Where h1, h2, h3, h4, h5, h6, h7, h8, h9 are vectors of each dimension in h; The above matrix H has 9 parameters and 8 degrees of freedom, which requires 8 pairs of feature points to solve. For the first layer of RANSAC, 8 different pairs of feature points are extracted from all the feature point pairs multiple times. The homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature point pairs according to this homography matrix is ​​calculated. The homography matrix H1 with the smallest residual sum is taken for the first layer of visual depth stratification. The threshold T is set. After all feature point pairs are mapped using H1, those with residuals less than T are regarded as inliers, otherwise they are regarded as outliers. All the obtained inliers are taken as points in the first layer of visual depth; then different 8 pairs of feature points are extracted multiple times from all the external points, and the homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature point pairs according to this homography matrix is ​​calculated, and the homography matrix H2 with the smallest residual sum is taken for the second layer of visual depth stratification. After all feature point pairs are mapped using H2, those with residuals less than T are regarded as inliers and points in the second layer of visual depth; this is repeated until there are no external points and the visual depth stratification is completed.

5. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: In step 5, the video frames with overlapping areas acquired by cameras at two different positions at the same time are matched layer by layer, aligned respectively, and the source video frames are divided into M×N grids. When matching each layer, different grids are locally matched respectively. The feature points use the distance between the center point of the grid and the nearest neighbor point in different layers to select the deformation mode of the deformation mapping of the grid. The grid g j From its center point c j Indicates that the grid g j The homography matrix of By using weights Combined H of N0 layers k The calculation formula is: In the formula, H k Represents the homography matrix of the kth layer; Indicates the grid center point c in layer k j The closest feature point; || || 2 Represents the Euclidean norm, used to calculate feature points and the grid center point c j The distance between them; σ represents the standard deviation of the Gaussian function; is a Gaussian weight associated with the position, used to convert the distance into a weight value, so that closer positions have higher weights and farther positions have lower weights; It represents the overall proportion of Gaussian weight; Exponential terms in formulas It represents the square of the Euclidean distance between two feature points divided by the variance of the Gaussian function. The smaller the value, the closer the two feature points are, and the larger the corresponding weight is. Calculate the homography matrix for each grid, based on the homography matrix of the grid The grid g j The source video frame pixel at position p′ in The target position p in the target video frame is obtained to complete the matching.

6. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: In step 6, the median point of the first row of the overlapping area is used as the starting point of the stitching line, and the minimum energy pixel point is found row by row as the stitching point, and the stitching line is obtained by connecting. If the median point of the first row is a foreground point, the non-foreground pixel point with the smallest column offset from the median point is selected as the starting point of the stitching line; when searching for the stitching line path, first compare the energy map on the left side of the current stitching point And the energy diagram on the right The size and energy diagram The calculation formula is as follows: In the formula, (x, y) is the location of the current splicing point; (i, j) is the pixel position involved in the energy map judgment; is the energy map at the pixel (i, j) at time t; E p (i, j) is the distance contribution coefficient at the pixel (i, j); H is the height of the overlapping area; [ ] is the rounding operation; ψ( ) is an indicator function. hour, The value is 1, otherwise, the value is 0; β1 and β2 are weight parameters; Select the side with smaller energy map to ensure that the stitching line extends in the direction with minimum estimated energy. To speed up the processing time, only check the 2p-1 pixels below the current stitching line point (i, j) on the side with smaller energy map each time. This calculation process can be achieved by using greedy search: In the formula, represents the pixel with the lowest energy, Represents the energy diagram at time t (i+1, k), when When k∈{yp,y-p+1,y}, When k∈{y-p+1,y,y+p-1}, When k∈{y,y+p-1,y+p}, p is the set offset, p≥2; The minimum energy pixel searched is assigned as a stitching point only when it is unique and does not belong to a foreground object. If the minimum energy pixel does not belong to a foreground object but is not unique, the pixel with the smallest column offset from the current stitching point is selected as the stitching point. If the minimum energy pixel is unique but belongs to a foreground object, the pixel with the smallest energy among the non-foreground object pixels is selected as the stitching point.

7. The multi-view surveillance video stitching method based on energy map estimation according to claim 6, characterized in that: The calculation formula for the energy diagram in step 6 is: In the formula, They represent the energy graphs at the positions of the video frame pixels (i, j) at time t and t-1 respectively. The attenuation factor α determines the contribution of the energy graph at the previous moment to the formation of the current energy graph. t (i, j) is the union of the pixels contained in any foreground object detected in the video frames of different perspectives at time t, that is: In the formula, ∨ represents the logical OR operation. and They represent the corresponding layer instance segmentation results in video frames of different perspectives, S t When (i, j) is in the overlapping area, the value is 1, otherwise, the value is 0, that is: Where Ω0 is the set of pixel positions of foreground objects in the overlapping area; In order to solve the problem that long-term memory of object energy may produce invalid gap boundaries, the contribution of the energy map of the previous frames is adjusted by the following formula: Where τ is the threshold value of the time window that defines the cumulative energy. Moments with energy values ​​less than τ will not be considered, thereby preventing excessive redundant information from the past from interfering with the current energy map estimation. τ is set to Where N * is the number of frames to maintain the energy, and α is the attenuation factor. By adjusting the attenuation factor α, the contribution of each frame to the output energy map can be controlled.

8. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: In step 7, the video frames captured by all cameras are stitched together in steps 4 to 6. For the two stitched video frames, they are regarded as one frame and the stitching operations of steps 4 to 6 are performed with other video frames with overlapping areas. Finally, the stitching of multi-view surveillance videos is completed to obtain a panoramic video.

9. A multi-view surveillance video splicing system based on energy map estimation, characterized in that: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute a multi-view surveillance video stitching method based on energy map estimation as described in any one of claims 1-8.

10. A multi-view surveillance video splicing system based on energy map estimation, characterized in that: It includes a readable storage medium, on which a computer program is stored. When the computer program is executed, a multi-view surveillance video stitching method based on energy map estimation as described in any one of claims 1 to 8 is implemented.

Images