A multi-view surveillance video splicing method and system based on energy map estimation

Abstract

The present invention discloses a multi-view surveillance video stitching method and system based on energy graph estimation, comprising obtaining video data and camera positions of multiple surveillance cameras on a road section to be stitched; extracting video frame features of each camera video, dividing pixels in the video frame into background pixels or foreground pixels, and matching background pixels; using a multi-layer RANSAC method to perform visual depth stratification on the video frames; performing layer-by-layer local matching, layer-by-layer stitching, and stitching layer fusion on video frames with overlapping areas acquired at the same time by cameras at two different locations; stitching the video frames captured by all cameras to complete the stitching of multi-view surveillance videos and obtain a panoramic video. The present invention has the characteristics of low computational complexity, high computational speed, and good stitching effect, maintaining a balance between algorithm efficiency and stitching accuracy, and providing a highly efficient and robust method for the field of surveillance video stitching.

Description

Technical Field

[0001] The present invention belongs to the technical field of video splicing, and in particular relates to a multi-view surveillance video splicing method and system based on energy map estimation. Background Art

[0002] In recent years, with the continuous advancement of "smart transportation" construction projects across various regions, the coverage of traffic monitoring has continued to expand. However, the large number of complex monitoring images makes it difficult for managers to intuitively understand the traffic conditions in the monitored area. In current traffic monitoring systems, road traffic conditions are often monitored using multiple monocular cameras within a small area. This leads to problems such as frequent switching of perspectives and fragmented spatial relationships between different images, which is not conducive to quickly handling traffic accidents and road congestion.

[0003] Video stitching, an extension of image stitching, refers to the technology of seamlessly stitching together several overlapping video sequences into wide-view or even panoramic videos. It creates a video with a wider field of view by merging a set of videos with overlapping fields of view. The typical video stitching process includes feature extraction, corresponding point matching, image registration, and image synthesis. Feature extraction and corresponding point matching can find corresponding points between images from different perspectives. Based on these corresponding points, the image registration process converts the images into a coordinate system to align them. Since the aligned images have overlapping areas, the final image synthesis step determines the color information of the overlapping areas and their surrounding areas.

[0004] Video stitching technology can address the aforementioned challenges in traffic monitoring and management. By applying video stitching technology to surveillance videos, managers can gain an intuitive understanding of current road traffic conditions. However, existing video stitching technologies face several challenges when applied to traffic monitoring scenarios involving multiple moving objects. While image registration has been widely studied to improve image alignment during the initial stitching model development phase, it remains a challenging problem when input images exhibit large parallax, lens distortion, and exposure variations. Existing image registration algorithms assume that input images should be captured by cameras sufficiently far from the scene to minimize parallax. However, this limits the application of video stitching to surveillance videos with large parallax, lens distortion, and exposure variations, which are often located far apart. During the continuous stitching and fusion of video frames, video stitching also faces visual artifacts caused by temporal inconsistencies and time constraints. Transitions in stitching positions or changes in lighting conditions between consecutive frames can lead to additional visual artifacts. Furthermore, traditional video fusion algorithms suffer from high computational complexity and slow video generation speed, making them difficult to apply to surveillance video stitching applications where rapid response is crucial. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention provides a multi-view surveillance video stitching method and system based on energy graph estimation. During the initial stitching model establishment phase, a layered local alignment approach is used to alleviate the difficulty in directly matching surveillance videos due to large parallax differences, resulting in an initial stitching template. During the continuous stitching and fusion phase of video frames, an energy graph estimation algorithm is used to dynamically calculate the video stitching line, finding the minimum change stitching line between video frames that bypasses moving objects. This effectively addresses the issues of visible stitching lines and ghosting caused by fast-moving objects and the presence of multiple moving objects in surveillance videos.

[0006] In order to achieve the above object, the present invention provides a multi-view surveillance video splicing method based on energy map estimation, comprising the following steps:

[0007] Step 1: Obtain video data and camera positions of multiple surveillance cameras on the road section to be spliced;

[0008] Step 2: Using a feature extraction algorithm to extract the video frame features of each camera video, using a background modeling algorithm to perform background modeling on the video frame, and dividing the pixels in the video frame into background pixels or foreground pixels;

[0009] Step 3: Match the background pixels in the video frames based on the nearest neighbor search to complete the background matching of the video frames taken by different cameras;

[0010] Step 4: Based on the video frame features extracted in step 2, a multi-layer RANSAC method is used to perform visual depth layering on the video frames with overlapping areas acquired by two cameras at different positions at the same time, separating objects at different visual depths into different layers;

[0011] Step 5: Perform local matching layer by layer on the video frames with overlapping areas acquired by two cameras at different positions at the same time;

[0012] Step 6: Using a minimum energy-based stitching line search method, the video frames with overlapping areas acquired by two cameras at different positions at the same time are stitched layer by layer, and the stitched video frames of each layer are fused to complete the stitching of the two video frames;

[0013] Step 7: Perform the stitching operations of steps 4 to 6 on the video frames captured by all cameras to complete the stitching of the multi-view surveillance video and obtain a panoramic video.

[0014] Furthermore, in step 2, a feature extraction algorithm is used to extract the video frame features of each camera video, including the color features and grayscale values ​​of the pixels in the video frame; a Gaussian mixture model GMM is used to model the background of the video frame, and the specific operations are as follows: ① All the extracted camera video frame features are organized into a data set and divided into a training set and a test set. The training set is used to train the GMM model, and the test set is used to test the training effect of the GMM model; ② The parameters of the GMM are initialized by random initialization, including the mean and variance of each Gaussian component and the weight of each Gaussian component. The maximum expectation algorithm is used to iteratively update the parameters of the model to maximize the likelihood function. The GMM model after training contains the Gaussian component parameters of each feature dimension; ③ The extracted video frame features are input into the trained GMM model, the probability of each pixel in the video frame belonging to the background is calculated, and the pixels are classified according to the probability value, dividing them into background pixels or foreground pixels.

[0015] Furthermore, in step 3, the Euclidean distance between each background pixel in the video frame and the background pixels of other video frames is calculated. If the Euclidean distance between the background pixels is less than the set threshold, the two background pixels are considered to match. The above calculation and judgment operations are performed on all video frames in turn to complete the background matching of all video frames.

[0016] Furthermore, in step 4, the homography matrix of each layer is first calculated: Let F represent the matching feature point pairs, and Represent the i-th pair of feature points in the first video frame and the second video frame respectively, and their relationship is expressed as N is the total number of matched feature points; for each pair of feature points, an augmented matrix A and a target vector b are formed, which can be expressed as follows:

[0017]

[0018]

[0019] Where, and Represent the coordinates of the points in the matching feature point pair in the first video frame and the second video frame respectively;

[0020] The homography matrix H maps points in one video frame to points in another video frame, thereby achieving matching of video images at different angles. The least squares method is used to estimate the parameters of the homography matrix. The calculation formula for estimating the homography matrix using the least squares method is:

[0021] h=argmin||Ah-b|| (3)

[0022] Where h is a 9-dimensional vector representing the parameters of the homography matrix;

[0023] By rearranging the vector h obtained by the least squares method, we get the homography matrix H:

[0024]

[0025] Where h1, h2, h3, h4, h5, h6, h7, h8, and h9 are vectors of various dimensions in h;

[0026] The above matrix H has 9 parameters and 8 degrees of freedom, which requires 8 pairs of feature points to solve. For the first layer of RANSAC, 8 different pairs of feature points are extracted from all the feature point pairs multiple times. The homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature point pairs according to this homography matrix is ​​calculated. The homography matrix H1 with the smallest residual sum is taken to perform the first layer of visual depth stratification. The threshold T is set. After all feature point pairs are mapped using H1, those with residuals less than T are regarded as inliers, otherwise they are regarded as outliers. All the obtained inliers are used as points in the first layer of visual depth; then different 8 pairs of feature points are extracted multiple times from all the outliers, and the homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature point pairs according to this homography matrix is ​​calculated. The homography matrix H2 with the smallest residual sum is taken for the second layer of visual depth stratification. After all feature point pairs are mapped using H2, those with residuals less than T are regarded as inliers and as points in the second layer of visual depth; this is repeated until there are no outliers and the visual depth stratification is completed.

[0027] Furthermore, in step 5, the video frames with overlapping areas acquired by two cameras at different positions at the same time are matched layer by layer, aligned respectively, and the source video frames are divided into M×N grids. When matching each layer, different grids are locally matched respectively. The feature points use the distance between the center point of the grid and the nearest neighbor point in different layers to select the deformation mode of the grid deformation mapping. The grid g j From its center point c j Indicates that the grid g j The homography matrix is ​​expressed as By using weights Comprehensive H of N0 layers k The calculation formula is:

[0028]

[0029]

[0030]

[0031] Where H k represents the homography matrix of the k-th layer; Indicates the grid center point c in layer k j The nearest feature point; || || 2 Represents the Euclidean norm, used to calculate feature points and the grid center point c j The distance between them; σ represents the standard deviation of the Gaussian function; is a Gaussian weight associated with the position, used to convert the distance into a weight value, so that closer positions have higher weights and farther positions have lower weights; It represents the overall proportion of Gaussian weight.

[0032] Exponential terms in formulas It represents the square of the Euclidean distance between two feature points divided by the variance of the Gaussian function. The smaller the value, the closer the two feature points are, and the greater the corresponding weight.

[0033] Calculate the homography matrix of each grid, based on the homography matrix of the grid The grid g j The source video frame pixel at position p′ in Get its target position p in the target video frame and complete the matching.

[0034] Furthermore, in step 6, the energy map at the pixel (i, j) at time t is calculated The specific calculation formula is:

[0035]

[0036] Where, They represent the energy maps at the position of pixel (i, j) in the video frame at time t and t-1 respectively. The attenuation factor α determines the contribution of the energy map at the previous moment to the formation of the current energy map. S t (i, j) is the union of the pixels contained in any foreground object detected in the video frames of different perspectives at time t, that is:

[0037]

[0038] In the formula, ∨ represents the logical OR operation, and They represent the corresponding layer instance segmentation results in video frames of different perspectives, S t When (i, j) is in the overlapping area, the value is 1, otherwise, the value is 0, that is:

[0039]

[0040] Where Ω0 is the set of pixel positions of foreground objects in the overlapping region;

[0041] In order to solve the problem that long-term memory of object energy may produce invalid gap boundaries, the contribution of the energy map of the previous frames is adjusted by the following formula:

[0042]

[0043] Where τ is the threshold value of the time window that defines the cumulative energy. Moments with energy values ​​less than τ will not be considered, thereby preventing excessive redundant information from the past from interfering with the current energy map estimation. τ is set to where N * is the number of frames to maintain the energy, and α is the attenuation factor. By adjusting the attenuation factor α, the contribution of each frame to the output energy map can be controlled.

[0044] The median point of the first row of the overlapping area is used as the starting point of the stitching line, and the minimum energy pixel point is found row by row as the stitching point. The stitching line is connected. If the median point of the first row is the foreground point, the non-foreground pixel point with the smallest offset from the column where the median point is located is selected as the starting point of the stitching line. When searching for the stitching line path, first compare the energy map on the left side of the current stitching point. and the energy diagram on the right The size and energy diagram of The calculation formula is as follows:

[0045]

[0046]

[0047]

[0048] Where (x, y) is the location of the current splicing point; (i, j) is the pixel position involved in energy map judgment; is the energy map at the pixel (i, j) at time t; E p (i, j) is the distance contribution coefficient at the pixel (i, j); H is the height of the overlapping area; [] is the rounding operation; ψ() is an indicator function. hour, The value is 1, otherwise, the value is 0; β1 and β2 are weight parameters.

[0049] Select the side with the smaller energy map to ensure that the stitching line extends in the direction with the minimum estimated energy. To speed up processing time, only check the 2p-1 pixels below the current stitching line point (i, j) on the side with the smaller energy map each time. This calculation process can be achieved by using greedy search:

[0050]

[0051] In the formula, (x *,y * ) represents the pixel with the lowest energy, Represents the energy diagram at time t (i+1, k), when When k∈{yp,y-p+1,y}, When k∈{y-p+1,y,y+p-1}, When k∈{y,y+p-1,y+p}, p is the set offset, and p≥2.

[0052] Only when the minimum energy pixel found is unique and does not belong to the foreground object will it be assigned as the stitching point. If the minimum energy pixel does not belong to the foreground object but is not unique, the pixel with the smallest column offset from the current stitching point is selected as the stitching point. If the minimum energy pixel is unique but belongs to the foreground object, the pixel with the smallest energy among the non-foreground object pixels is selected as the stitching point.

[0053] Furthermore, in step 7, the video frames captured by all cameras are stitched together in steps 4 to 6. For the two stitched video frames, they are regarded as one frame and stitched together in steps 4 to 6 with other video frames with overlapping areas, and finally the stitching of multi-view surveillance videos is completed to obtain a panoramic video.

[0054] The present invention also provides a multi-view surveillance video splicing system based on energy map estimation, which is used to implement the multi-view surveillance video splicing method based on energy map estimation as described above.

[0055] Furthermore, the system includes a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute the multi-view surveillance video splicing method based on energy map estimation as described above.

[0056] Alternatively, it includes a readable storage medium having a computer program stored thereon, and when the computer program is executed, the multi-view surveillance video splicing method based on energy map estimation as described above is implemented.

[0057] Compared with the prior art, the present invention has the following advantages:

[0058] 1) The present invention fully considers the characteristics of large parallax and many moving objects in surveillance videos, and adopts a local matching alignment algorithm in the initial splicing template establishment stage, thereby avoiding the image deformation problem caused by large parallax;

[0059] 2) When searching for the video stitching line, the present invention uses a path-finding method based on energy graph estimation. Based on the characteristic that pixels of foreground objects have high energy on the energy graph, the problem is transformed into finding the path with the minimum energy. By comparing the energy graphs on the left and right sides of the current stitching point, the method ensures that the path extends in the direction with the minimum estimated energy. This solves the problem of obvious stitching lines and ghosting caused by objects moving quickly or with multiple moving objects in surveillance videos.

[0060] 3) The present invention has the characteristics of small computational complexity, fast computational speed, and good splicing effect, maintaining a balance between algorithm efficiency and splicing accuracy, and providing a highly efficient and parallax-robust method for the field of surveillance video splicing. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flowchart of a multi-view surveillance video stitching method based on energy map estimation according to an embodiment of the present invention.

[0062] Figure 2 Schematic diagram of the splicing line search process according to an embodiment of the present invention.

[0063] Figure 3 This is the result diagram of finding the splicing line according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention provides a multi-view surveillance video splicing method and system based on energy map estimation. The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0065] Example 1

[0066] like Figure 1 As shown, an embodiment of the present invention provides a multi-view surveillance video stitching method based on energy map estimation, comprising the following steps:

[0067] Step 1: Obtain video data and camera positions of multiple surveillance cameras on the road section to be spliced.

[0068] Step 2: Use a feature extraction algorithm to extract the video frame features of each camera video, use a background modeling algorithm to perform background modeling on the video frame, and divide the pixels in the video frame into background pixels or foreground pixels.

[0069] This embodiment uses the SIFT algorithm to extract the video frame features of each camera video, including the color features and grayscale values ​​of the pixels in the video frame. A Gaussian mixture model (GMM) is used to model the background of the video frame. The specific operations are as follows: ① All the extracted camera video frame features are organized into a data set and divided into a training set and a test set. The training set is used to train the GMM model, and the test set is used to test the training effect of the GMM model. ② The parameters of the GMM are initialized using a random initialization method, including the mean and variance of each Gaussian component and the weight of each Gaussian component. The expectation maximization algorithm (EM) is used to iteratively update the model parameters to maximize the likelihood function. The trained GMM model contains the Gaussian component parameters (mean and variance) of each feature dimension. ③ The extracted video frame features are input into the trained GMM model, the probability of each pixel in the video frame belonging to the background is calculated, and the pixels are classified according to the probability value, dividing them into background (non-moving object) pixels or foreground (moving object) pixels.

[0070] Step 3: Match the background pixels in the video frames based on the nearest neighbor search to complete the background matching of the video frames shot by different cameras.

[0071] The Euclidean distance between each background pixel in a video frame and the background pixels of other video frames is calculated. If the Euclidean distance between the background pixels is less than a set threshold, the two background pixels are considered to match. This calculation and judgment operation is performed sequentially for all video frames to complete background matching for all video frames. Through background matching, the correspondence between the same-named pixels in the background of video frames from different cameras is established, completing the fusion and unification of the background parts of the video frames, reducing the inconsistency of the background color, laying the foundation for subsequent video stitching, and improving the video quality of the subsequent stitching.

[0072] In step 4, based on the video frame features extracted in step 2, a multi-layer RANSAC method is used to perform visual depth stratification on the video frames with overlapping areas acquired by cameras at two different positions at the same time, separating objects at different visual depths into different layers.

[0073] First, calculate the homography matrix of each layer: Let F represent the matching feature point pairs, and Represent the i-th pair of feature points in the first video frame and the second video frame respectively, and their relationship is expressed as N is the total number of matched feature points; for each pair of feature points, an augmented matrix A and a target vector b are formed, which can be expressed as follows:

[0074]

[0075]

[0076] Where, and Represent the coordinates of the points in the matching feature point pair in the first video frame and the second video frame respectively.

[0077] The homography matrix H maps points in one video frame to points in another video frame, thereby achieving matching of video images at different angles. The least squares method is used to estimate the parameters of the homography matrix. The calculation formula for estimating the homography matrix using the least squares method is:

[0078] h=argmin||Ah-b|| (3)

[0079] Where h is a 9-dimensional vector representing the parameters of the homography matrix.

[0080] By rearranging the vector h obtained by the least squares method, we get the homography matrix H:

[0081]

[0082] Where h1, h2, h3, h4, h5, h6, h7, h8, and h9 are vectors of various dimensions in h.

[0083] The above matrix H has 9 parameters and 8 degrees of freedom, which requires 8 pairs of feature points to solve. For the first layer of RANSAC, 8 different pairs of feature points are extracted from all the feature point pairs multiple times. The homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature point pairs according to this homography matrix is ​​calculated. The homography matrix H1 with the smallest residual sum is taken to perform the first layer of visual depth stratification. The threshold T is set. After all feature point pairs are mapped using H1, those with residuals less than T are regarded as inliers, otherwise they are regarded as outliers. All the obtained inliers are used as points in the first layer of visual depth; then different 8 pairs of feature points are extracted multiple times from all the outliers, and the homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature point pairs according to this homography matrix is ​​calculated. The homography matrix H2 with the smallest residual sum is taken for the second layer of visual depth stratification. After all feature point pairs are mapped using H2, those with residuals less than T are regarded as inliers and as points in the second layer of visual depth; this is repeated until there are no outliers and the visual depth stratification is completed.

[0084] Step 5: Perform local matching layer by layer on the video frames with overlapping areas acquired by cameras at two different locations at the same time.

[0085] The video frames captured by two cameras at different positions at the same time are matched layer by layer and aligned separately to avoid object deformation caused by overall image matching and obtain a smooth transition. In order to simplify the matching calculation and reduce the computational complexity of the splicing method, the source video frame is divided into M×N grids. When matching each layer, different grids are locally matched. The feature points need to use the distance between the center point of the grid and the nearest neighbor point in different layers to select the deformation method of the grid deformation mapping. The grid g j From its center point c j Represents. Grid g j The homography matrix is ​​expressed as By using weights Comprehensive H of N0 layers k The calculation formula is:

[0086]

[0087]

[0088]

[0089] Where H k represents the homography matrix of the k-th layer; Indicates the grid center point c in layer k j The nearest feature point; || || 2 Represents the Euclidean norm, used to calculate feature points and the grid center point c j The distance between them; σ represents the standard deviation of the Gaussian function; is a Gaussian weight associated with the position, used to convert the distance into a weight value, so that closer positions have higher weights and farther positions have lower weights; It represents the overall proportion of Gaussian weight.

[0090] Exponential terms in formulas It represents the square of the Euclidean distance between two feature points divided by the variance of the Gaussian function. The smaller the value, the closer the two feature points are, and the greater the corresponding weight.

[0091] Calculate the homography matrix of each grid, based on the homography matrix of the grid The grid g j The source video frame pixel at position p′ in Get its target position p in the target video frame and complete the matching.

[0092] The correspondence between the pixel positions of the source video frame and the pixel positions of the target video frame is stored in a pixel mapping table. When performing video frame matching, the transformed video frame can be obtained by searching for the corresponding pixels in the source video frame instead of repeatedly performing the transformation, thereby speeding up the calculation.

[0093] In step 6, a minimum energy-based stitching line search method is used to stitch the video frames with overlapping areas acquired by two cameras at different positions at the same time layer by layer, and the stitched video frames of each layer are fused to complete the stitching of the two video frames.

[0094] Calculate the energy representation of objects located in different layers in the overlapping area of ​​two video frames acquired by cameras at different positions (Formulas (8)-(10)). If it is a foreground object, it has higher importance and higher energy, and the stitching line should bypass this part of the pixels. Conversely, if it is a background part, the stitching line is located here to avoid ghosting due to object movement.

[0095] For the energy of foreground objects, the instance segmentation results of foreground objects can be represented by binary images S t (i,j) indicates that S t (i, j) represents the union of the pixels contained in any foreground object detected in video frames at different viewpoints:

[0096]

[0097] In the formula, ∨ represents the logical OR operation, and They represent the corresponding layer instance segmentation results in video frames of different perspectives, S t When (i, j) is in the overlapping area, the value is 1, otherwise, the value is 0, that is:

[0098]

[0099] Where Ω0 is the set of pixel positions of foreground objects in the overlapping region.

[0100] For a certain pixel position at a certain moment, the energy map of the pixel at this moment is calculated by combining the union of the energy map of the previous moment and the pixels contained in the foreground object at the current moment, so that the energy map has temporal coherence.

[0101] Energy map at pixel (i, j) at time t The calculation formula is:

[0102]

[0103] Where, Represent the energy graphs of the video frames at time t and t-1 respectively, St (i, j) is the union of the pixels contained in any foreground object detected in the video frames of different perspectives at time t, and the attenuation factor α determines the contribution of the energy map at the previous moment to the formation of the current energy map.

[0104] Due to its recursive structure, the time-consistent energy graph It is actually the weighted sum of a set of object energies from time 0 to time t, with its importance decaying over time. This means that the current energy map remembers objects that moved through the previous frame without having to store previous frames, reducing storage space requirements. Specifically, to address the issue of long-term memory of object energies potentially creating invalid gap boundaries, the contribution of the energy maps of previous frames is adjusted using the following formula:

[0105]

[0106] Where τ is the threshold that defines the time window of cumulative energy. Moments with energy values ​​less than τ will not be considered, thereby preventing excessive redundant information from past moments from interfering with the current energy map estimation.

[0107] Specifically, τ can be set as where N * is the number of frames to maintain the energy, and α is the attenuation factor. By adjusting the attenuation factor α, the contribution of each frame to the output energy map can be controlled.

[0108] The median point of the first row of the overlapping area is used as the starting point of the stitching line, and the minimum energy pixel point is found row by row as the stitching point, and the stitching line is obtained by connecting. If the median point of the first row is a foreground point, the non-foreground pixel point with the smallest offset from the column where the median point is located is selected as the starting point of the stitching line. When searching for the stitching line path, in order to avoid sudden stitching lines on each row, first compare the energy map on the left side of the current stitching point. and the energy diagram on the right Selecting the side with the smaller energy map ensures that the stitching line area selected below has lower energy as a whole (i.e. contains fewer foreground objects), thereby making the stitching line better overall and considering more pixels (avoiding being misled by local low-energy pixels).

[0109] Calculate the energy map on the left side of the current splicing point and the energy diagram on the right The calculation formula is as follows:

[0110]

[0111]

[0112]

[0113] Where (x, y) is the location of the current splicing point; (i, j) is the pixel position involved in energy map judgment; is the energy map at the pixel (i, j) at time t; E p (i, j) is the distance contribution coefficient at the pixel (i, j); H is the height of the overlapping area; [] is the rounding operation; ψ() is an indicator function. hour, The value is 1, otherwise, the value is 0; β1 and β2 are weight parameters;

[0114] Select the side with the smaller energy map to ensure that the stitching line extends in the direction with the minimum estimated energy. To speed up processing time, only check the 2p-1 pixels below the current stitching line point (i, j) on the side with the smaller energy map each time. This calculation process can be achieved by using greedy search:

[0115]

[0116] In the formula, (x * ,y * ) represents the pixel with the lowest energy, Represents the energy diagram at time t (i+1, k), when When k∈{yp,y-p+1,y}, When k∈{y-p+1,y,y+p-1}, When k∈{y,y+p-1,y+p}, p is the set offset, p≥2, and the value of p in this embodiment is 2.

[0117] Only when the minimum energy pixel found is unique and does not belong to the foreground object will it be assigned as the stitching point. If the minimum energy pixel does not belong to the foreground object but is not unique, the pixel with the smallest column offset from the current stitching point is selected as the stitching point. If the minimum energy pixel is unique but belongs to the foreground object, the pixel with the smallest energy among the non-foreground object pixels is selected as the stitching point.

[0118] like Figure 2 As shown, the grid represents pixels, the gray shadow part represents the pixels of the foreground object, and the position selected by the red dotted line is the energy map on the left side of the current splicing point (the black dot in the first row) The calculation range, the position selected by the green dotted line is the energy diagram on the right side of the current splicing point (the black dot in the first row) The energy graph on the left side of this embodiment is smaller than the energy graph on the right side, i.e. Therefore, the three dots within the blue frame below the current stitching point are compared. Since the white dot directly below the current stitching point belongs to the pixel of the foreground object, the pixel with the smallest energy among the two dots on the left is selected as the stitching point. In this embodiment, the black dot has the smallest energy and is used as the stitching point.

[0119] Figure 3 The splicing path calculated by the splicing line finding algorithm proposed in this invention is shown.

[0120] Step 7: Perform the stitching operations of steps 4 to 6 on the video frames captured by all cameras to complete the stitching of the multi-view surveillance video and obtain a panoramic video.

[0121] Perform steps 4 to 6 on the video frames captured by all cameras. For the two spliced ​​video frames, treat them as one frame and perform steps 4 to 6 on the video frames with overlapping areas. Finally, complete the splicing of multi-view surveillance videos to obtain a panoramic video.

[0122] Example 2

[0123] Based on the same inventive concept, the present invention also provides a multi-view surveillance video stitching system based on energy graph estimation, including a processor and a memory, the memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute the above-mentioned multi-view surveillance video stitching method based on energy graph estimation.

[0124] Example 3

[0125] Based on the same inventive concept, the present invention also provides a multi-perspective surveillance video stitching system based on energy graph estimation, including a readable storage medium, on which a computer program is stored. When the computer program is executed, a multi-perspective surveillance video stitching method based on energy graph estimation as described above is implemented.

[0126] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0127] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A multi-view surveillance video splicing method based on energy map estimation, characterized in that: The following steps are involved: Step 1: Obtain video data and camera positions of multiple surveillance cameras on the road section to be spliced; Step 2: Using a feature extraction algorithm to extract the video frame features of each camera video, using a background modeling algorithm to perform background modeling on the video frame, and dividing the pixels in the video frame into background pixels or foreground pixels; Step 3: Match the background pixels in the video frames based on the nearest neighbor search to complete the background matching of the video frames taken by different cameras; Step 4: Based on the video frame features extracted in step 2, a multi-layer RANSAC method is used to perform visual depth layering on the video frames with overlapping areas acquired by two cameras at different positions at the same time, separating objects at different visual depths into different layers; Step 5: Perform local matching layer by layer on the video frames with overlapping areas acquired by two cameras at different positions at the same time; Step 6: Using a minimum energy-based stitching line search method, the video frames with overlapping areas acquired by two cameras at different positions at the same time are stitched layer by layer, and the stitched video frames of each layer are fused to complete the stitching of the two video frames; The median point of the first row of the overlapping area is used as the starting point of the stitching line, and the minimum energy pixel point is found row by row as the stitching point. The stitching line is connected. If the median point of the first row is the foreground point, the non-foreground pixel point with the smallest offset from the column where the median point is located is selected as the starting point of the stitching line. When searching for the stitching line path, first compare the energy map on the left side of the current stitching point. and the energy diagram on the right The size and energy diagram of 、 The calculation formula is as follows: （12） （13） （14） Where, is the location of the current splicing point; is the pixel position involved in energy map judgment; for t Moment Pixel Energy diagram at the location; Pixels The distance contribution coefficient at the position; H is the height of the overlapping area; It is a round-up operation; is an indicator function, when hour, The value is 1, otherwise, the value is 0; and is the weight parameter; Select the side with smaller energy graph to ensure that the stitching line extends in the direction with minimum estimated energy. In order to speed up the processing time, only the current stitching line point on the side with smaller energy graph is checked each time. Below pixels, which is calculated using a greedy search: （15） Where, represents the pixel with the lowest energy, express t time The energy diagram at hour, ,when hour, ,when hour, , p is the set offset, ; Only when the minimum energy pixel found is unique and does not belong to the foreground object will it be assigned as the splicing point. If the minimum energy pixel does not belong to the foreground object but is not unique, the pixel with the smallest column offset from the current splicing point is selected as the splicing point. If the minimum energy pixel is unique but belongs to the foreground object, the pixel with the smallest energy among the non-foreground object pixels is selected as the splicing point. Step 7: Perform the stitching operations of steps 4 to 6 on the video frames captured by all cameras to complete the stitching of the multi-view surveillance video and obtain a panoramic video.

2. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: In step 2, a feature extraction algorithm is used to extract the video frame features of each camera video, including the color features and grayscale values ​​of the pixels in the video frame; a Gaussian mixture model (GMM) is used to model the background of the video frame. The specific operations are as follows: all the extracted camera video frame features are organized into a data set and divided into a training set and a test set. The training set is used to train the GMM model, and the test set is used to test the training effect of the GMM model. The parameters of the GMM are initialized by random initialization, including the mean and variance of each Gaussian component and the weight of each Gaussian component. The maximum expectation algorithm is used to iteratively update the parameters of the model to maximize the likelihood function. The GMM model after training contains the Gaussian component parameters of each feature dimension; the extracted video frame features are input into the trained GMM model, the probability of each pixel in the video frame belonging to the background is calculated, and the pixels are classified according to the probability value, dividing them into background pixels or foreground pixels.

3. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: In step 3, the Euclidean distance between each background pixel in the video frame and the background pixels of other video frames is calculated. If the Euclidean distance between the background pixels is less than the set threshold, the two background pixels are considered to match. The above calculation and judgment operations are performed on all video frames in sequence to complete the background matching of all video frames.

4. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: In step 4, we first calculate the homography matrix of each layer: let F represent the matching feature point pairs, and Represents the first video frame and the second video frame respectively. i For feature points, the relationship is expressed as , N is the total number of matched feature points; for each pair of feature points, an augmented matrix is ​​formed A and the target vector b , specifically expressed as: （1） （2） Where, and Represent the coordinates of the points in the matching feature point pair in the first video frame and the second video frame respectively; Homography matrix H Map the points in one video frame to the points in another video frame to achieve matching of video images at different angles. Use the least squares method to estimate the parameters of the homography matrix. The calculation formula for estimating the homography matrix using the least squares method is: （3） Where, h Is a 9-dimensional vector representing the parameters of the homography matrix; The vector obtained by the least squares method h Rearrange to get the homography matrix H : （4） Where, 、 、 、 、 、 、 、 、 for h The vectors of each dimension in ; The above matrix H has 9 parameters and 8 degrees of freedom, which requires 8 pairs of feature points to solve. For the first layer of RANSAC, 8 different pairs of feature points are extracted from all the feature point pairs multiple times. The homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature point pairs according to this homography matrix is ​​calculated. The homography matrix with the minimum residual sum is taken. Perform the first layer of visual depth stratification, set the threshold T, and use all feature point pairs After mapping, the points with residuals less than T are considered as inliers, otherwise they are considered as outliers. All the obtained inliers are used as points in the first layer of visual depth. Then, 8 different pairs of feature points are extracted from all the outliers multiple times. The homography matrix is ​​solved based on the 8 pairs of feature points extracted each time, and the residual sum caused by the mapping of the remaining feature points according to this homography matrix is ​​calculated. The homography matrix with the minimum residual sum is taken. Perform the second layer of visual depth stratification and use all feature points to After mapping, the residuals smaller than T are considered as inliers and are used as points in the second layer of visual depth. This process is repeated until there are no outliers, and the visual depth layering is completed.

5. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: In step 5, the video frames with overlapping areas acquired by two cameras at different positions at the same time are matched layer by layer, aligned respectively, and the source video frames are divided into When matching each layer, different grids are matched locally. The feature points use the distance between the center point of the grid and the nearest neighbor point in different layers to select the deformation mode of the grid deformation mapping. From its center point Represents the grid The homography matrix is ​​expressed as , by using weights comprehensive layers The calculation formula is: （5） （6） （7） Where, represents the homography matrix of the k-th layer; Indicates the grid center point in layer k The nearest feature point; Represents the Euclidean norm, used to calculate feature points and the grid center point The distance between represents the standard deviation of the Gaussian function; is a Gaussian weight associated with the position, used to convert the distance into a weight value, so that closer positions have higher weights and farther positions have lower weights; It represents the overall proportion of Gaussian weight; Exponential terms in formulas Represents the square of the Euclidean distance between two feature points divided by the variance of the Gaussian function. The smaller this value is, the closer the two feature points are, and the greater the corresponding weight is. Calculate the homography matrix of each grid, based on the homography matrix of the grid The grid Middle position The source video frame pixels are Get its target position in the target video frame p , the matching is completed.

6. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: The calculation formula for the energy map in step 6 is: （8） Where, 、 Respectively t 、 t -1 video frame pixel at a moment Energy map at position, attenuation factor Determines the contribution of the energy graph at the previous moment to the formation of the current energy graph, for t The union of the pixels contained in any foreground object detected in the video frames at different perspectives at all times, that is: （9） In the formula, ∨ represents the logical OR operation, 、 and They represent the corresponding layer instance segmentation results in video frames of different perspectives, When it is in the overlapping area, the value is 1, otherwise, the value is 0, that is: （10） Where, is the set of pixel positions of foreground objects in the overlapping region; In order to solve the problem that long-term memory of object energy may produce invalid gap boundaries, the contribution of the energy map of the previous frames is adjusted by the following formula: （11） Where, is the threshold value that defines the time window of accumulated energy, and the energy value is less than The moment will not be considered, so as to prevent excessive redundant information of the past moments from interfering with the current energy map estimation. Set to ,in is the number of frames to maintain energy, is the attenuation factor, by adjusting the attenuation factor , controls the contribution of each frame to the output energy map.

7. The multi-view surveillance video stitching method based on energy map estimation according to claim 1, characterized in that: In step 7, the video frames captured by all cameras are stitched together in steps 4 to 6. For the two stitched video frames, they are regarded as one frame and stitched together with other video frames with overlapping areas in steps 4 to 6. Finally, the stitching of the multi-view surveillance video is completed to obtain a panoramic video.

8. A multi-view surveillance video splicing system based on energy map estimation, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute a multi-view surveillance video splicing method based on energy map estimation as described in any one of claims 1 to 7.

9. A multi-view surveillance video splicing system based on energy map estimation, characterized in that: The method comprises a readable storage medium having a computer program stored thereon. When the computer program is executed, the method for splicing multi-view surveillance videos based on energy map estimation according to any one of claims 1 to 7 is implemented.

Images