A High-Precision Planar Homography Calculation Method for Urban Surveillance Videos

Abstract

The present invention discloses a high-precision planar homography calculation method for urban surveillance videos. First, a frame of the surveillance camera video is read, and corresponding point pairs are collected on the video frame image and the remote sensing image to obtain the image coordinates and geographic coordinates of the corresponding points. Then, geometric features in the image are extracted to calculate the vanishing line of the ground plane. Finally, based on the corresponding points and the vanishing line, a high-precision homography matrix between the surveillance video and the two-dimensional map is calculated. The present invention ensures and improves the accuracy of homography matrix calculation.

Description

Technical Field

[0001] The present invention relates to a high-precision planar homography calculation method for urban surveillance videos. The present invention belongs to the fields of public security and smart cities and involves computer vision recognition technology. Background Art

[0002] After spatializing surveillance cameras / videos and unifying them into a unified geographical space, realizing the unified expression, visualization, and analysis of real-time and massive surveillance video content has become a hot topic in current fields such as public security, transportation, and urban management. Establishing a mutual mapping between videos and geographical spaces is one of the core bases for unifying their spatial frameworks and has attracted the attention of many academic and industrial circles. For the mapping between surveillance videos and two-dimensional / three-dimensional geographical space data, it is mainly achieved by selecting corresponding points in videos and remote sensing images / three-dimensional scene data and calculating the homography matrix to realize the mapping from video images to two-dimensional / three-dimensional geographical spaces. The main references are as follows:

[0003] [1] Zhang Xingguo, Liu Xuejun, Wang Sining, etc. Mutual mapping between surveillance videos and 2D geographical space data [J]. Journal of Wuhan University (Information Science Edition), 2015, 40(8): 1130 - 1136.

[0004] [2] Xie Yujia, Mao Bo, Wang Feiyue. A method and system for mapping surveillance video targets to a three-dimensional geographical scene model [P]. Jiangsu Province: CN201910285390.4, 2023-04-18.

[0005] [3] Liu Xuejun, Wang Meizhen, Yu Jinhui, etc. A method for mapping urban road surveillance videos to a two-dimensional map [P]. Jiangsu Province: CN201910051462.9, 2023-05-26.

[0006] [4] Zhang Xingguo, Li Xiaodi, Zhang Li, Ren Shuai, Liu Mohan. A method, system, and storage medium for mapping iron tower surveillance videos considering semantic and depth information. Chinese invention patent. Application number: 2024106314985, filing date: 2024-05-21.

[0007] The present invention is mainly oriented to urban surveillance videos, and mainly serves the monitoring of moving targets in the fields of public security, transportation, etc. The high-precision homography calculation method is the core foundation for acquiring the trajectory of moving targets and analyzing their behavior. The shooting direction of such surveillance cameras is usually at a small angle to the horizontal direction, which leads to a large difference in the viewing angle between the surveillance camera screen and the orthophotored remote sensing image, and requires a high precision for the mapping method. Low-precision plane homography easily leads to inaccurate positioning of targets in the surveillance screen, especially objects far away from the surveillance camera, thereby affecting subsequent visualization, analysis and application. The traditional plane homography calculation method is mainly divided into two steps: 1) selecting at least 4 control point pairs; 2) forming the point pairs into constraint conditions and calculating the homography matrix using direct linear transformation. The accuracy of the homography matrix calculation is seriously affected by the quality of the control points. However, due to the difference in viewing angle and resolution between remote sensing images and surveillance videos, it is usually difficult to obtain the same-name points, the number of same-name point pairs obtained is small, and the distribution is usually uneven, so that the calculation accuracy of the homography matrix cannot be guaranteed. How to select the control points in the scene and how to make full use of the scene geometric information constraints to ensure and improve the accuracy of the homography matrix calculation are the key issues to be solved by the present invention. Summary of the invention

[0008] The key problem to be solved by the present invention is to establish a mapping relationship between the video image space and the two-dimensional geographic space by collecting the same-name points in the video image and the geographic space and the regular geometric features of the scene for the city surveillance camera. Therefore, the present invention proposes a method for high-precision mapping of surveillance video and two-dimensional map using the same-name points and image geometric features.

[0009] The specific plan is as follows:

[0010] A high-precision planar homography calculation method for urban surveillance videos comprises the following steps:

[0011] S1: intercept the video frame image of the connected surveillance camera, collect the same-name point pairs on the image and remote sensing image, and obtain the spatial position of the same-name points, including image coordinates and two-dimensional geographic coordinates;

[0012] S2: Extract geometric features from the image, including parallel line groups and circles, and comprehensively calculate the vanishing lines of the ground plane;

[0013] S3: Extract the spatial positions of the same-name points according to S1 and determine the subset of same-name points involved in the calculation of the homography matrix;

[0014] S4: According to the same-name points and vanishing lines, a two-dimensional homography matrix is calculated to establish a mutual mapping relationship between the video image and the geographic space.

[0015] Furthermore, the specific steps of obtaining the image coordinates and two-dimensional geographic coordinates of the points with the same name in step S1 are:

[0016] S1.1: Obtain the image coordinates (u, v) of no less than 3 points in sequence in the video frame image;

[0017] S1.2: Obtain the two-dimensional geographic coordinates (X, Y) of the homologous points of each image point on the remote sensing image in sequence.

[0018] Furthermore, the specific steps for extracting the parallel line groups in step S2 are as follows:

[0019] S2.1.1: Initialize the flag flag1 indicating whether there are multiple groups of parallel line groups in the image. If there are multiple groups of parallel line groups, set flag1 = 1; otherwise, set flag1 = 0;

[0020] S2.1.2: Automatically / manually extract multiple parallel line groups on the ground plane, which are combinations of line segments converging to one point in the image;

[0021] S2.1.3: Take the parallel line group with the most line segments as the first group. If there are non-parallel line segments misclassified into this group, remove them. The processed line segment group is denoted as SegC1;

[0022] S2.1.4: Take the parallel line group with the second most line segments as the second group. If there are non-parallel line segments misclassified into this group, remove them. The processed line segment group is denoted as SegC2.

[0023] Furthermore, the specific steps for extracting circles in step S2 are as follows:

[0024] S2.2.1: Initialize the flag flag2 indicating whether there are more than 2 circles in the image. If there are multiple circles, set flag2 = 1; otherwise, set flag2 = 0;

[0025] S2.2.2: Automatically / manually extract multiple circles on the ground plane, including concentric circles and multiple separated circles, which are imaged as ellipses in the image;

[0026] S2.2.3: Among all the extracted circles, select the 2 circles closest to the lower part of the video frame image, denoted as C1 and C2 respectively.

[0027] Furthermore, the specific steps for calculating the ground plane vanishing line in step S2 are as follows:

[0028] S2.3.1: If the value of flag1 is 1, then according to the characteristics of the extracted parallel line groups, calculate the intersection points of all line segments in SegC1 and SegC2 respectively, denoted as VP1 and VP2, which are the vanishing points in two directions of the ground plane;

[0029] S2.3.2: Calculate the straight line determined by the two points VP1 and VP2, denoted as VL1;

[0030] VL1 = VP1 × VP2 (1)

[0031] S2.3.3: If the value of flag2 is 1, then according to the extracted circular features, calculate the virtual intersection points of the two circles, denoted as M i and M j ;

[0032] S2.3.4: Calculate the line determined by the two virtual intersection points M i and M j , denoted as VL2;

[0033] VL2 = M i ×M j (2)

[0034] S2.3.5: According to the following formula, calculate the plane vanishing line VL and represent VL as ax + by + c = 0;

[0035]

[0036] Furthermore, the specific steps for determining the subset of corresponding points participating in the homography matrix calculation in step S3 are as follows:

[0037] S3.1: If the number of corresponding point pairs is greater than 3 and less than 6, then retain all point pairs;

[0038] S3.2: If the number of corresponding point pairs is greater than 6, then eliminate one group of point pairs each time according to S3.3 and S3.4 until only 6 pairs remain;

[0039] S3.3: If there are points located at the image edge, then eliminate them;

[0040] S3.4: If there are multiple points close to each other, then eliminate the points farther from the image center.

[0041] Furthermore, the specific steps for calculating the two-dimensional homography matrix in step S4 are:

[0042] S4.1: According to the correspondence between the image coordinates and two-dimensional coordinates of multiple obtained corresponding points, each group of corresponding points provides 2 equalities as shown in formula (4), where (h1, h2, h3, h4, h5, h6, h7, h8, 1) T is the vector composed of the elements of the sought homography matrix, (u, v) are the corresponding image coordinates, and (X, Y) are the two-dimensional geographical coordinates of the corresponding points;

[0043]

[0044] S4.2: According to the solved ground plane vanishing line, provide 2 equalities as shown in formula (5), where (a, b, c) is the homogeneous representation of the vanishing line in the video frame;

[0045]

[0046] S4.3: Form a system of equations according to Equations (4) and (5), calculate the overdetermined solution of the system of equations, and form a homography matrix according to Equation (6);

[0047]

[0048] The beneficial effects of the present invention are as follows: By using the method of the present invention, a high-precision homography matrix can be obtained. Based on this, a high-precision association between video image content and the geospatial can be established, realizing the spatialization of moving targets such as people and vehicles and other static entities in the video, and providing method support for the precise positioning and analysis of the spatio-temporal trajectories of moving targets. Description of the Drawings

[0049] Figure 1 is the implementation flowchart of the present invention.

[0050] Figure 2 is the acquisition of corresponding point pairs. Among them, the left figure is the points collected in the remote sensing image, and the right figure is the corresponding points collected in the surveillance video frame.

[0051] Figure 3 is the schematic diagram of the input image cropping rule. Detailed Embodiments

[0052] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0053] The technical solutions of the present invention will be further described below with reference to the drawings.

[0054] As Figure 1 shown, the present invention provides a high-precision planar homography calculation method for urban surveillance videos, and the specific steps are as follows:

[0055] First step, prepare relevant equipment. Prepare a laptop computer and connect to the network.

[0056] Second step, corresponding points between video images and the geospatial.

[0057] S2.1: Obtain the address of the surveillance camera video stream, open the link address, and intercept the video frame with fewer moving targets.

[0058] S2.2: Open the remote sensing image and observe the corresponding points that appear simultaneously in the video frame and the remote sensing image.

[0059] S2.3: As shown in Figure 2 , the current example monitoring scenario can collect the image coordinates and geographical coordinates of 8 pairs of homologous points.

[0060] Step 3: Extract geometric features and calculate the vanishing line of the ground plane.

[0061] S3.1: Extract the parallel line groups in the video frame. As shown in Figure 3 , the red lines are the parallel line groups in 2 directions; the blue and orange ones are the circular manhole covers.

[0062] S3.2: Calculate the vanishing lines determined by the parallel line groups and the vanishing lines determined by the 2 blue circles respectively.

[0063] S3.3: Calculate the vanishing line of the ground plane based on the vanishing lines calculated from the two types of geometric features, as shown by the yellow line in Figure 2 .

[0064] Step 4: Selection of homologous points.

[0065] S4.1: According to the basis for homologous point selection, Figure 2 when there are more than 6 pairs of candidate homologous points, only fewer points are needed during actual collection.

[0066] S4.2: Eliminate the 8th pair of homologous points.

[0067] S4.3: Eliminate the 5th pair of homologous points.

[0068] Step 5: Calculate the two-dimensional homography matrix.

[0069] S5.1: According to the image coordinates and two-dimensional plane coordinates of the N groups (3 <= N <= 6) of homologous point pairs collected, construct N groups of equations as shown in Equation (4).

[0070]

[0071] S5.2: According to the calculated vanishing line, construct a group of equations as shown in Equation (5).

[0072]

[0073] S5.3: Combine all the equations of Equation (4) and Equation (5) to form a comprehensive system of equations, and calculate the solutions of the equations. S5.4: Convert the solutions of the equations calculated according to Equation (6) into a two-dimensional homography matrix.

[0074]

[0075] The technical means disclosed by the present invention are not limited to those disclosed in the above embodiments, but also include technical solutions formed by any combination of the above technical features. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A high-precision planar homography calculation method for urban surveillance videos, characterized in that, The steps include: S1: intercept the video frame image of the connected surveillance camera, collect the same-name point pairs on the image and remote sensing image, and obtain the spatial position of the same-name points, including image coordinates and two-dimensional geographic coordinates; S2: Extract geometric features from the image, including parallel line groups and circles, and comprehensively calculate the vanishing lines of the ground plane; S3: Extract the spatial positions of the same-name points according to S1 and determine the subset of same-name points involved in the calculation of the homography matrix; S4: Calculate the two-dimensional homography matrix based on the same-name points and vanishing lines, and establish the mutual mapping relationship between the video image and the geographic space; The specific steps of extracting the parallel line group in step S2 are: S2.1.1: Initialize flag1, which indicates whether the image contains multiple sets of parallel line groups. If so, set flag1=1; otherwise, set flag1=0; S2.1.2: Automatically / manually extract multiple parallel line groups on the ground plane, which are line segments converging to a point in the image; S2.1.3: The parallel line group with the most line segments is taken as the first group. If there are non-parallel line segments that are mistakenly classified into this group, they will be removed. The processed line segment group is recorded as SegC1; S2.1.4: The parallel line group with the second largest number of line segments is taken as the second group. Any non-parallel line segments that are mistakenly classified into this group are removed. The processed line segment group is recorded as SegC2; The specific steps of circle extraction in step S2 are: S2.2.1: Initialize flag2 to indicate whether the image contains more than two circles. If so, set flag2=1; otherwise, set flag2=0. S2.2.2: Automatically / manually extract multiple circles on the ground plane, including concentric circles and separated circles, which are imaged as ellipses in the image; S2.2.3: Among all the extracted circles, select the two circles closest to the bottom of the video frame image and record them as C1 and C2 respectively; The specific steps of calculating the ground plane vanishing line in step S2 are: S2.3.1: If the flag1 value is 1, then the intersection points of all line segments in SegC1 and SegC2 are calculated based on the extracted parallel line group features, and are recorded as VP1 and VP2, respectively, which are the vanishing points in two directions of the ground plane; S2.3.2: Compute the straight line defined by the two points VP1 and VP2, denoted as VL1; VL1=VP1×VP2(1) S2.3.3: If the value of flag2 is 1, calculate the virtual intersection points of the two circles based on the extracted circular features, denoted as M i and M j ; S2.3.4: Calculate the straight line determined by the two virtual dots M i and M j , denoted as VL2; VL2 = M i × M j (2) S2.3.5: Calculate the plane vanishing line VL according to the following formula, and express VL as ax+by+c=0; 2. The high-precision planar homography calculation method for urban surveillance videos according to claim 1, characterized in that, The specific steps of obtaining the image coordinates and two-dimensional geographic coordinates of the points with the same name in step S1 are: S1.1: Obtain image coordinates (u, v) of no less than 3 points in the video frame image in sequence; S1.2: Obtain the two-dimensional geographic coordinates (X, Y) of the same-name points of each image point on the remote sensing image in turn.

3. A high-precision planar homography calculation method for urban surveillance videos according to claim 1, characterized in that The specific steps of determining the subset of points of the same name participating in the calculation of the homography matrix in step S3 are as follows: S3.1: If the number of pairs of points with the same name is greater than 3 and less than 6, all pairs are retained; S3.2: If the number of pairs of points with the same name is greater than 6, then remove one pair of points at a time according to S3.3 and S3.4 until only 6 pairs remain; S3.3: If there are points located at the edge of the image, remove them; S3.4: If there are multiple points close to each other, eliminate the points that are farther from the image center.

4. A high-precision planar homography calculation method for urban surveillance videos according to claim 1, characterized in that The specific steps for calculating the 2D homography matrix in step S4 are as follows: S4.1: According to the correspondence between the image coordinates and the two-dimensional coordinates of multiple homologous points, each group of homologous points provides two equations as shown in Equation (4), where (h1, h2, h3, h4, h5, h6, h7, h8, 1) T is the vector composed of the elements of the sought homography matrix, (u, v) are the homologous image coordinates, and (X, Y) are the two-dimensional geographical coordinates of the homologous points; S4.2: According to the solved ground plane vanishing line, provide two equations as shown in Equation (5), where (a, b, c) is the homogeneous representation of the vanishing line in the video frame; S4.3: Form a system of equations based on Equation (4) and Equation (5), calculate the overdetermined solution of the system of equations, and form the homography matrix according to Equation (6);

Images