Urban monitoring video-oriented high-precision plane homography calculation method

Abstract

The invention discloses a high-precision plane homography calculation method for an urban monitoring video, and the method comprises the steps: firstly reading a frame image of a monitoring camera video, collecting homonymy point pairs on the video frame image and a remote sensing image, and obtaining the image coordinates and geographic coordinates of homonymy points; extracting geometric features in the image, and calculating a disappearance line of the ground plane; and finally, according to the homonymy points and the vanishing lines, calculating a high-precision homography matrix between the monitoring video and the two-dimensional map. According to the invention, the precision of homography matrix calculation is ensured and improved.

Description

Technical Field

[0001] The present invention relates to a high-precision planar homography calculation method for urban surveillance videos. The present invention belongs to the fields of public security and smart cities and relates to computer vision recognition technology. Background Art

[0002] Unifying surveillance cameras / videos into a unified geographic space after spatialization to achieve unified expression, visualization, and analysis of real-time, massive surveillance video content has become a hot topic in the fields of public security, transportation, and urban management. Establishing a mutual mapping between video and geographic space is one of the core foundations of a unified spatial framework, which has attracted the attention of many academic and industry circles. For the mapping of surveillance video and two-dimensional plane / three-dimensional geographic space data, the mapping of video images to two-dimensional plane / three-dimensional geographic space is mainly achieved by selecting corresponding points in the video and remote sensing image / three-dimensional scene data and calculating the homography matrix. The main references are as follows:

[0003] [1] Zhang Xingguo, Liu Xuejun, Wang Sining, et al. Mutual mapping between surveillance video and 2D geospatial data[J]. Journal of Wuhan University (Information Science Edition), 2015, 40(8): 1130-1136.

[0004] [2] Xie Yujia, Mao Bo, Wang Feiyue. A method and system for mapping surveillance video targets to three-dimensional geographic scene models[P]. Jiangsu Province: CN201910285390.4, 2023-04-18.

[0005] [3] Liu Xuejun, Wang Meizhen, Yu Jinhui, et al. A method for mapping urban road surveillance video to two-dimensional map [P]. Jiangsu Province: CN201910051462.9, 2023-05-26.

[0006] [4] Zhang Xingguo, Li Xiaodi, Zhang Li, Ren Shuai, Liu Mohan. A tower surveillance video mapping method, system and storage medium considering semantic and depth information. Chinese invention patent. Application number: 2024106314985, acceptance date: 2024-05-21.

[0007] The present invention is mainly oriented to urban surveillance videos, and mainly serves the monitoring of moving targets in the fields of public security, transportation, etc. The high-precision homography calculation method is the core foundation for acquiring the trajectory of moving targets and analyzing their behavior. The shooting direction of such surveillance cameras is usually at a small angle to the horizontal direction, which leads to a large difference in the viewing angle between the surveillance camera screen and the orthophotored remote sensing image, and requires a high precision for the mapping method. Low-precision plane homography easily leads to inaccurate positioning of targets in the surveillance screen, especially objects far away from the surveillance camera, thereby affecting subsequent visualization, analysis and application. The traditional plane homography calculation method is mainly divided into two steps: 1) selecting at least 4 control point pairs; 2) forming the point pairs into constraint conditions and calculating the homography matrix using direct linear transformation. The accuracy of the homography matrix calculation is seriously affected by the quality of the control points. However, due to the difference in viewing angle and resolution between remote sensing images and surveillance videos, it is usually difficult to obtain the same-name points, the number of same-name point pairs obtained is small, and the distribution is usually uneven, so that the calculation accuracy of the homography matrix cannot be guaranteed. How to select the control points in the scene and how to make full use of the scene geometric information constraints to ensure and improve the accuracy of the homography matrix calculation are the key issues to be solved by the present invention. Summary of the invention

[0008] The key problem to be solved by the present invention is to establish a mapping relationship between the video image space and the two-dimensional geographic space by collecting the same-name points in the video image and the geographic space and the regular geometric features of the scene for the city surveillance camera. Therefore, the present invention proposes a method for high-precision mapping of surveillance video and two-dimensional map using the same-name points and image geometric features.

[0009] The specific plan is as follows:

[0010] A high-precision planar homography calculation method for urban surveillance videos comprises the following steps:

[0011] S1: intercept the video frame image of the connected surveillance camera, collect the same-name point pairs on the image and remote sensing image, and obtain the spatial position of the same-name points, including image coordinates and two-dimensional geographic coordinates;

[0012] S2: Extract geometric features from the image, including parallel line groups and circles, and comprehensively calculate the vanishing lines of the ground plane;

[0013] S3: Extract the spatial positions of the same-name points according to S1 and determine the subset of same-name points involved in the calculation of the homography matrix;

[0014] S4: According to the same-name points and vanishing lines, a two-dimensional homography matrix is ​​calculated to establish a mutual mapping relationship between the video image and the geographic space.

[0015] Furthermore, the specific steps of obtaining the image coordinates and two-dimensional geographic coordinates of the points with the same name in step S1 are:

[0016] S1.1: Obtain image coordinates (u, v) of no less than 3 points in the video frame image in sequence;

[0017] S1.2: Obtain the two-dimensional geographic coordinates (X, Y) of the same-name points of each image point on the remote sensing image in turn.

[0018] Furthermore, the specific steps of extracting the parallel line group in step S2 are:

[0019] S2.1.1: Initialize flag1, which indicates whether the image contains multiple sets of parallel line groups. If so, set flag1=1; otherwise, set flag1=0;

[0020] S2.1.2: Automatically / manually extract multiple parallel line groups on the ground plane, which are line segments converging to a point in the image;

[0021] S2.1.3: The parallel line group with the most line segments is taken as the first group. If there are non-parallel line segments that are mistakenly classified into this group, they will be removed. The processed line segment group is recorded as SegC1;

[0022] S2.1.4: The parallel line group with the second largest number of line segments is taken as the second group. Any non-parallel line segments that are mistakenly classified into this group are removed. The processed line segment group is recorded as SegC2.

[0023] Furthermore, the specific steps of circle extraction in step S2 are:

[0024] S2.2.1: Initialize flag2 to indicate whether the image contains more than two circles. If so, set flag2=1; otherwise, set flag2=0.

[0025] S2.2.2: Automatically / manually extract multiple circles on the ground plane, including concentric circles and separated circles, which are imaged as ellipses in the image;

[0026] S2.2.3: Among all the extracted circles, select the two circles closest to the bottom of the video frame image and record them as C1 and C2 respectively.

[0027] Furthermore, the specific steps of calculating the ground plane vanishing line in step S2 are:

[0028] S2.3.1: If the flag1 value is 1, then the intersection points of all line segments in SegC1 and SegC2 are calculated based on the extracted parallel line group features, and are recorded as VP1 and VP2, respectively, which are the vanishing points in two directions of the ground plane;

[0029] S2.3.2: Compute the straight line defined by the two points VP1 and VP2, denoted as VL1;

[0030] VL1=VP1×VP2 (1)

[0031] S2.3.3: If the flag2 value is 1, then the virtual point where the two circles intersect is calculated based on the extracted circular features, denoted as M i and M j ;

[0032] S2.3.4: Calculate the value of M i and M j The straight line determined by the two imaginary points is denoted as VL2;

[0033] VL2=M i ×M j (2)

[0034] S2.3.5: Calculate the plane vanishing line VL according to the following formula, and express VL as ax+by+c=0;

[0035]

[0036] Furthermore, the specific steps of determining the subset of points of the same name participating in the calculation of the homography matrix in step S3 are as follows:

[0037] S3.1: If the number of pairs of points with the same name is greater than 3 and less than 6, all pairs are retained;

[0038] S3.2: If the number of pairs of points with the same name is greater than 6, then remove one pair of points at a time according to S3.3 and S3.4 until only 6 pairs remain;

[0039] S3.3: If there are points located at the edge of the image, remove them;

[0040] S3.4: If there are multiple points that are close to each other, the points that are far from the center of the image are eliminated.

[0041] Furthermore, the specific steps of calculating the two-dimensional homography matrix in step S4 are:

[0042] S4.1: According to the corresponding relationship between the image coordinates and the two-dimensional coordinates of multiple points with the same name, each group of points with the same name provides two equations as shown in equation (4), where (h1,h2,h3,h4,h5,h6,h7,h8,1) T is the vector composed of the elements of the homography matrix, (u,v) are the image coordinates of the same name, and (X,Y) are the two-dimensional geographic coordinates of the same name point;

[0043]

[0044] S4.2: According to the solved ground plane vanishing line, provide two equations as shown in equation (5), where (a, b, c) is the homogeneous expression of the vanishing line in the video frame;

[0045]

[0046] S4.3: Form an equation system according to equation (4) and equation (5), calculate the overdetermined solution of the equation system, and form a homography matrix according to equation (6);

[0047]

[0048] The beneficial effects of the present invention are as follows: a high-precision homography matrix can be obtained by using the method of the present invention, based on which a high-precision association between video image content and geographic space can be established, and the spatialization of moving targets such as people and cars and other static entities in the video can be realized, which can provide method support for the precise positioning and analysis of the spatiotemporal trajectory of moving targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is an implementation flow chart of the present invention.

[0050] Figure 2 It is the collection of same-name point pairs, where the left picture shows the points collected in the remote sensing image, and the right picture shows the same-name points collected in the surveillance video frame.

[0051] Figure 3 It is a schematic diagram of the input image cropping rules. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0053] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0054] like Figure 1 As shown, the present invention provides a high-precision planar homography calculation method for urban surveillance video, and the specific steps are:

[0055] Step 1: Prepare relevant equipment: Prepare a laptop and connect it to the Internet.

[0056] Step 2: The same-name points between video images and geographic space.

[0057] S2.1: Get the address of the surveillance camera video stream, open the link address, and capture the video frames with fewer moving targets.

[0058] S2.2: Open the remote sensing image and observe the points with the same name that appear simultaneously in the video frame and the remote sensing image.

[0059] S2.3: If Figure 2 As shown, the current example monitoring scene can collect 8 pairs of image coordinates and geographic coordinates of the same-name points.

[0060] The third step is to extract geometric features and calculate the ground plane vanishing line.

[0061] S3.1: Extract parallel line groups in the video frame, such as Figure 3 As shown, red is a group of parallel lines in two directions; blue and orange are circular manhole covers.

[0062] S3.2: Calculate the vanishing line determined by the parallel line group and the vanishing line determined by the two blue circles respectively.

[0063] S3.3: Based on the vanishing lines calculated from the two types of geometric features, the vanishing lines of the ground plane are calculated, such as Figure 2 Indicated by the yellow line.

[0064] Step 4: Select points with the same name.

[0065] S4.1: Select the basis based on the same name points, Figure 2 When there are more than 6 candidate pairs of points with the same name, only fewer points are needed for actual collection.

[0066] S4.2: Eliminate the 8 pairs of points with the same name.

[0067] S4.3: Eliminate the number 5 pairs of points with the same name.

[0068] Step 5: Calculate the two-dimensional homography matrix.

[0069] S5.1: Based on the collected N groups (3<=N<=6) of image coordinates and two-dimensional plane coordinates of the same-name point pairs, construct N groups of equations as shown in formula (4).

[0070]

[0071] S5.2: Based on the calculated vanishing lines, construct a set of equations as shown in equation (5).

[0072]

[0073] S5.3: Combine all equations of equation (4) and equation (5) to form a comprehensive set of equations and calculate the solutions of the equations. S5.4: Convert the solutions of the equations calculated according to equation (6) into a two-dimensional homography matrix.

[0074]

[0075] The technical means disclosed in the present invention are not limited to the technical means disclosed in the above-mentioned embodiments, but also include technical solutions composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. A high-precision planar homography calculation method for urban surveillance videos, characterized in that: The steps include: S1: intercept the video frame image of the connected surveillance camera, collect the same-name point pairs on the image and remote sensing image, and obtain the spatial position of the same-name points, including image coordinates and two-dimensional geographic coordinates; S2: Extract geometric features from the image, including parallel line groups and circles, and comprehensively calculate the vanishing lines of the ground plane; S3: Extract the spatial positions of the same-name points according to S1 and determine the subset of same-name points involved in the calculation of the homography matrix; S4: According to the same-name points and vanishing lines, a two-dimensional homography matrix is ​​calculated to establish a mutual mapping relationship between the video image and the geographic space.

2. According to claim 1, a high-precision planar homography calculation method for urban surveillance video is characterized in that: The specific steps of obtaining the image coordinates and two-dimensional geographic coordinates of the points with the same name in step S1 are: S1.1: Obtain image coordinates (u, v) of no less than 3 points in the video frame image in sequence; S1.2: Obtain the two-dimensional geographic coordinates (X, Y) of the same-name points of each image point on the remote sensing image in turn.

3. The high-precision planar homography calculation method for urban surveillance video according to claim 1 is characterized in that: The specific steps of extracting the parallel line group in step S2 are: S2.1.1: Initialize flag1, which indicates whether the image contains multiple sets of parallel line groups. If so, set flag1=1; otherwise, set flag1=0; S2.1.2: Automatically / manually extract multiple parallel line groups on the ground plane, which are line segments converging to a point in the image; S2.1.3: The parallel line group with the most line segments is taken as the first group. If there are non-parallel line segments that are mistakenly classified into this group, they will be removed. The processed line segment group is recorded as SegC1; S2.1.4: The parallel line group with the second largest number of line segments is taken as the second group. Any non-parallel line segments that are mistakenly classified into this group are removed. The processed line segment group is recorded as SegC2.

4. The high-precision planar homography calculation method for urban surveillance video according to claim 3 is characterized in that: The specific steps of circle extraction in step S2 are: S2.2.1: Initialize flag2 to indicate whether the image contains more than two circles. If so, set flag2=1; otherwise, set flag2=0. S2.2.2: Automatically / manually extract multiple circles on the ground plane, including concentric circles and separated circles, which are imaged as ellipses in the image; S2.2.3: Among all the extracted circles, select the two circles closest to the bottom of the video frame image and record them as C1 and C2 respectively.

5. The high-precision planar homography calculation method for urban surveillance video according to claim 4 is characterized in that: The specific steps of calculating the ground plane vanishing line in step S2 are: S2.3.1: If the flag1 value is 1, then the intersection points of all line segments in SegC1 and SegC2 are calculated based on the extracted parallel line group features, and are recorded as VP1 and VP2, respectively, which are the vanishing points in two directions of the ground plane; S2.3.2: Compute the straight line defined by the two points VP1 and VP2, denoted as VL1; VL1=VP1×VP2(1) S2.3.3: If the flag2 value is 1, then the virtual point where the two circles intersect is calculated based on the extracted circular features, denoted as M i and M j ; S2.3.4: Calculate the value of M i and M j The straight line determined by the two imaginary points is denoted as VL2; VL2=M i ×M j (2) S2.3.5: Calculate the plane vanishing line VL according to the following formula, and express VL as ax+by+c=0; 6. The high-precision planar homography calculation method for urban surveillance video according to claim 1 is characterized in that: The specific steps of determining the subset of points of the same name participating in the calculation of the homography matrix in step S3 are as follows: S3.1: If the number of pairs of points with the same name is greater than 3 and less than 6, all pairs are retained; S3.2: If the number of pairs of points with the same name is greater than 6, then remove one pair of points at a time according to S3.3 and S3.4 until only 6 pairs remain; S3.3: If there are points located at the edge of the image, remove them; S3.4: If there are multiple points that are close to each other, the points that are far from the center of the image are eliminated.

7. The high-precision planar homography calculation method for urban surveillance video according to claim 1 is characterized in that: The specific steps of calculating the two-dimensional homography matrix in step S4 are: S4.1: According to the corresponding relationship between the image coordinates and the two-dimensional coordinates of multiple points with the same name, each group of points with the same name provides two equations as shown in equation (4), where (h1,h2,h3,h4,h5,h6,h7,h8,1) T is the vector composed of the elements of the homography matrix, (u,v) are the image coordinates of the same name, and (X,Y) are the two-dimensional geographic coordinates of the same name point; S4.2: According to the solved ground plane vanishing line, provide two equations as shown in equation (5), where (a, b, c) is the homogeneous expression of the vanishing line in the video frame; S4.3: Form an equation system according to equation (4) and equation (5), calculate the overdetermined solution of the equation system, and form a homography matrix according to equation (6);

Images