Method for mapping a scene captured in a video surveillance image to a real physical world

Abstract

The application provides a method for mapping a shooting scene in a video monitoring image to a real physical world, which only needs to place a standard two-dimensional reference object on a road surface of a monitoring scene where a video monitoring device is located, collect multiple pictures with the reference object for labeling by the video monitoring device, manually label labeled pixel points on the pictures for labeling, select a corresponding mapping relation f(x, y), utilize known sizes and position relations on the real two-dimensional reference object, form several equation groups about the mapping function f(x, y) based on distances between the labeled pixel points, solve the equation groups by using a gradient descent method, and obtain the closest values of each parameter in f(x, y), i.e. the mapping function of the shooting scene to the real physical world under the camera.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, specifically to a method for mapping the scene captured in a video surveillance image to the real physical world. Background Technology

[0002] Currently, numerous dedicated surveillance cameras and video monitoring devices are installed on roads for collecting basic road traffic data. However, when using video sensors / monitoring equipment for applications such as vehicle speed measurement, the video acquisition equipment itself cannot directly obtain the correspondence between the captured scene and the real physical world. The acquisition of some information still requires the assistance of additional equipment (such as radar). For example, patent CN109615870A discloses a traffic detection system based on millimeter-wave radar and video. It collects target information through millimeter-wave radar and video equipment, converts world coordinates into image plane coordinates, determines the position of the target detected by the millimeter-wave radar sensor projected onto the image plane, and finds the mapping relationship between the scene captured by the video acquisition equipment and the real physical world.

[0003] In an effort to reduce costs, some regions wish to use existing fixed video surveillance equipment for traffic speed measurement. However, for the reasons mentioned above, additional auxiliary equipment such as radar is required to achieve the speed measurement function. But for some less economically developed areas, even simply adding radar equipment incurs significant costs in terms of initial investment and subsequent maintenance.

[0004] In existing technologies, such as those used in the field of intelligent vehicles, there are complex methods based on software, such as machine learning models, to calibrate and map monitoring equipment to the coordinate system of the real physical world. However, the monitoring equipment installed in vast rural areas is not installed all at once, but rather gradually laid out along with the national road infrastructure construction. Even within the same area, the types and models of monitoring equipment may differ. If such a complex mapping method is used, each type of monitoring equipment needs to be calibrated and modeled separately. Although no new hardware is needed, it still requires a significant amount of time for technical personnel to design and debug, resulting in high costs. Summary of the Invention

[0005] To address the problem that existing fixed video surveillance equipment cannot directly obtain the correspondence between the captured scene and the real physical world, and that the collection of some information still requires the assistance of additional sensor equipment or a large amount of manpower, which leads to an increase in overall costs, this invention provides a method for mapping the captured scene to the real physical world in video surveillance images. This method does not require the addition of additional sensor equipment and, based on a simple method, can estimate the relationship between the size of the captured scene and the distance to the real physical world in fixed video surveillance in rural areas.

[0006] The technical solution of this invention is as follows: a method for mapping a shooting scene in a video surveillance image to the real physical world, characterized by comprising the following steps:

[0007] S1: Prepare reference objects. According to the preset ground reference object positions, place the reference objects on the road surface of the monitoring scene where the video surveillance equipment is located.

[0008] The reference object uses a standard two-dimensional reference object; the position of the reference object includes at least 3 non-overlapping positions;

[0009] S2: Extract video frames from the historical video clips when the reference object was placed from the data collected by the video surveillance equipment to obtain a surveillance image;

[0010] The surveillance images include the reference objects distributed at different locations of the ground reference objects, which are respectively denoted as: annotation images;

[0011] S3: The images are manually annotated, and the annotation position is the fixed position of the known size of the reference object in each image, denoted as: annotation pixel;

[0012] S4: For video surveillance equipment, set the mapping function of image pixels with respect to the actual physical distance;

[0013] Using the labeled pixels, a system of equations about the mapping function is formed based on the positional relationships between the labeled pixels and their physical distances; the specific method is as follows:

[0014] a1: Establish a camera coordinate system for the image used for annotation;

[0015] a2: Determine the deformation relationship between the shooting scene and the real physical world as f-deformation;

[0016] The judgment method is as follows:

[0017] When the road within the monitoring range of the video surveillance equipment is a straight road, the f-shape becomes a linear deformation;

[0018] When the road within the monitoring range of dual video surveillance devices is a non-straight road, the deformation f is a proportional deformation;

[0019] Then: For the coordinates Point(x,y) of the corresponding labeled pixel point in the labeled image, the actual labeled pixel point Point' on the reference object is (x',y');

[0020] That is: Point'(x',y')=f(x,y)Point(x,y);

[0021] Therefore, solving for the mapping relationship between the monitoring scene and the real physical world becomes solving for the parameters in f(x,y);

[0022] a3: In the camera coordinate system, for each of the P labeled pixels Point(x,y) of a reference object, P sets of coordinates can be obtained;

[0023] Assumptions: Each reference object has P labeled pixels Point(x,y); there are Q reference object positions, then on Q labeled images, there are a total of N = Q * P labeled pixels Point(x,y);

[0024] a4: Based on the actual size of the reference object, the positional relationship between each labeled pixel is known, so the distance Dis between any two labeled pixels is known;

[0025] Dis = distance(Point' i -Point' j ),

[0026] Where distance is the Euclidean distance between two points, and i and j are the indexes of the labeled pixels, which are positive integers greater than or equal to 1 and less than or equal to N;

[0027] Then, based on the distance between any two labeled pixels, the following relationship is obtained:

[0028] Dis=distance(f(x,y)Point(x,y) i -f(x,y)Point(x,y) j );

[0029] a5: Construct a system of equations based on the known distances Dis between all labeled pixels;

[0030] Suppose we construct a system of M equations, resulting in the following relationship:

[0031] Dis1=distance(f(x,y)Point(x,y) i1-f(x,y)Point(x,y) j1 ); ...

[0033] Dis q = distance(f(x,y)Point(x,y) iq -f(x,y)Point(x,y) jq ); ...

[0035] Dis M distance(f(x,y)Point(x,y) iM -f(x,y)Point(x,y) jM );

[0036] In the formula, M is the known distance Dis, q is the index of the known distance, and 1≤q≤M;

[0037] S5: Solve the system of equations using the gradient descent method to obtain the mapping function between the shooting scene of the video surveillance equipment and the real physical world;

[0038] make

[0039]

[0040] To solve this problem, a relation is constructed based on the least squares method:

[0041] By applying the standard gradient descent method to find the minimum value of G, we can obtain the parameters in f(x,y) and thus obtain the mapping relationship between the scene captured in the video surveillance image and the real physical world f(x,y).

[0042] Its further features are:

[0043] When the deformation relationship f is a linear deformation: the origin pixel represents a horizontally and b vertically, with each horizontal pixel increasing by m and each vertical pixel increasing by n.

[0044] The physical coordinates of the image point (x,y) relative to the origin in the real world are f(x,y)=(a+(x-1)*m,b+(y-1)*n);

[0045] The parameters that need to be solved for f(x, y) are a, b, m and n;

[0046] When the deformation relationship f is a proportional deformation: the origin pixel represents a horizontally and b vertically, each horizontal pixel becomes m times the size of the previous pixel, and each vertical pixel becomes n times the size of the previous pixel.

[0047] The physical coordinates of the image point (x, y) relative to the origin in the real world are f(x, y) = (a*m) (x-1) ,b*n (y-1) );

[0048] The parameters that need to be solved for f(x, y) are a, b, m and n;

[0049] The reference object includes: a 1m*1m square cardboard; the corresponding marking positions are the pixels where the four right angles of the reference object are located.

[0050] The reference object positions are set to three locations on the road in the monitoring screen: the upper left, middle, and lower right.

[0051] In step a3, the camera coordinate system uses the upper left corner of the monitoring image as the origin to construct a mapping relationship between the monitoring scene and the real physical world.

[0052] This application provides a method for mapping the captured scene in video surveillance images to the real physical world. It only requires placing a standard-style two-dimensional reference object on the road surface of the monitored scene where the video surveillance equipment is located. Multiple images with the reference object are captured by the video surveillance equipment. The pixels on these images are manually labeled. Then, a corresponding mapping relationship f(x,y) is selected. Using the known dimensions and positional relationships of the real two-dimensional reference object, a system of equations about the mapping function f(x,y) is formed based on the distances between the labeled pixels. The system of equations is solved using the gradient descent method to obtain the closest value for each parameter in f(x,y), thus yielding the mapping function between the captured scene and the real physical world. This method does not require pre-calibration for each type of video surveillance equipment or additional hardware. It achieves the mapping relationship between the captured scene in video surveillance images and the real physical world with simple operation and low cost; it is particularly suitable for various types of video surveillance equipment on vast rural roads where precise solutions are not required. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the overall process of mapping the captured scene in video surveillance images to the real physical world in this application.

[0054] Figure 2 This is a schematic diagram showing the placement of reference objects;

[0055] Figure 3 A diagram showing the markings of four points on a reference object;

[0056] Figure 4 Example of steps for calculating mapping relationships. Detailed Implementation

[0057] like Figure 1As shown, the present invention includes a method for mapping a scene captured in a video surveillance image to the real physical world, which includes the following steps.

[0058] S1: Prepare reference objects. According to the preset ground reference object positions, place the reference objects on the road surface of the monitoring scene where the video surveillance equipment is located.

[0059] The reference object uses a standard style two-dimensional reference object; the reference object position includes at least 3 non-overlapping positions.

[0060] At the same time, it fully extracts the mapping relationship between video surveillance images and the real physical world.

[0061] S2: Extract video frames from historical video clips when placing reference objects in the data collected by the video surveillance equipment to obtain surveillance images; the surveillance images include the distribution of reference objects at different ground reference object locations, which are respectively denoted as: annotation images.

[0062] S3: Manually annotate the images used for annotation. The annotation position is the fixed pixel point where the known size of the reference object is located in each image, denoted as: annotation pixel point.

[0063] like Figure 2 As shown, the reference object in this embodiment includes: a 1m*1m square cardboard; as Figure 3 As shown, the corresponding annotation positions are the pixels where the four right angles of the reference object are located. That is, in this embodiment, the annotation pixels are... Figure 3 The four points marked in red;

[0064] The reference object's position color is set to the upper left, middle, and lower right parts of the road in the monitoring screen. In practice, the reference object needs to be placed on the road surface three times, sequentially in the upper left, middle, and lower right parts of the road in the monitoring screen. Historical video clips of the reference object placement are extracted and converted into several frames. From these frames, three images are selected showing the reference object placed in the upper left, middle, and lower right parts of the road surface.

[0065] To ensure a sufficient number of equations can be established, this application uses a square paper template with four points, positioned at the upper left, middle, and lower right corners of the road in the monitoring image. Simultaneously, a simple two-dimensional reference is used to reduce overall cost. Furthermore, using a 1m x 1m square paper template facilitates the establishment of mathematical relationships between labeled pixels, simplifying the labeling process.

[0066] S4: For video surveillance equipment, set the mapping function of image pixels with respect to the actual physical distance;

[0067] Using the labeled pixels, and based on the positional relationships between them and their physical distances, a system of equations concerning the mapping function is formed; the specific method is as follows:

[0068] a1: Establish a camera coordinate system for the images used for annotation;

[0069] a2: Determine the deformation relationship between the shooting scene and the real physical world as f-deformation;

[0070] The judgment method is as follows:

[0071] When the road within the monitoring range of the video surveillance equipment is a straight road, the f-shape becomes a linear deformation;

[0072] When the road within the monitoring range of dual video surveillance devices is a non-straight road, the deformation f is a proportional deformation.

[0073] In vast rural areas, video surveillance equipment is sparsely installed, and most roads requiring speed measurement are relatively straight. This is because actual speed measurement doesn't require precise real-time vehicle speeds, but only a speed range for subsequent judgment. To simplify the operation, this application pre-defines two deformation relationships based on actual road conditions.

[0074] Then: For the coordinates Point(x,y) of the corresponding labeled pixel point in the labeled image, the actual labeled pixel point Point' on the reference object is (x',y');

[0075] That is: Point'(x',y')=f(x,y)Point(x,y);

[0076] Therefore, solving the mapping relationship between the monitoring scene and the real physical world is transformed into solving the parameters in f(x,y).

[0077] In this method, uncertain parameters are treated as terms to be solved. Based on prior knowledge of the reference objects placed multiple times (e.g., the board is 1 meter wide), several systems of equations about the parameters are formed.

[0078] a3: In the camera coordinate system, for each of the P labeled pixels Point(x,y) of a reference object, P sets of coordinates can be obtained;

[0079] Assumptions: Each reference object has P labeled pixels Point(x,y); there are Q reference object positions, so on the Q labeled images, there are a total of N = Q * P labeled pixels Point(x,y).

[0080] a4: Based on the actual size of the reference object, the positional relationship between each labeled pixel is known, so the distance Dis between any two labeled pixels is known;

[0081] Dis = distance(Point' i -Point' j ),

[0082] Where distance is the Euclidean distance between two points, and i and j are the indexes of the labeled pixels, which are positive integers greater than or equal to 1 and less than or equal to N;

[0083] Specifically: distance(Point' i -Point' j ) = sqrt[(x' i -x' j ) 2 +(y' i -y' j ) 2 ].

[0084] Therefore, based on the distance between any two labeled pixels, the following relationship is obtained:

[0085] Dis=distance(f(x,y)Point(x,y) i -f(x,y)Point(x,y) j );

[0086] That is: distance(f(x,y)Point(x,y) i -f(x,y)Point(x,y) j )-Dis=0.

[0087] a5: Construct a system of equations based on the known distances Dis between all labeled pixels;

[0088] Suppose we construct a system of M equations, resulting in the following relationship:

[0089] Dis1=distance(f(x,y)Point(x,y) i1 -f(x,y)Point(x,y) j1 ); ...

[0091] Dis q = distance(f(x,y)Point(x,y) iq -f(x,y)Point(x,y) jq ); ...

[0093] Dis M distance(f(x,y)Point(x,y) iM -f(x,y)Point(x,y) jM );

[0094] In the formula, M is the known distance Dis, q is the index of the known distance, and 1≤q≤M.

[0095] S5: Solve the system of equations using the gradient descent method to obtain the mapping function between the shooting scene of the video surveillance equipment and the real physical world;

[0096] make

[0097]

[0098] To solve this problem, a relation is constructed based on the least squares method:

[0099] This will affect the distance(f(x,y)Point(x,y)) i -f(x,y)Point(x,y) j Solving for )-Dis=0 is transformed into solving for (distance(f(x,y)Point(x,y)) i -f(x,y)Point(x,y) j )-Dis) 2 Finding the minimum value;

[0100] By applying the standard gradient descent method to find the minimum value of G, we can obtain the parameters in f(x,y) and thus obtain the mapping relationship between the scene captured in the video surveillance image and the real physical world f(x,y).

[0101] like Figure 4 As shown, the deformation relationship f is a linear deformation. In this embodiment, the linear deformation is implemented based on an arithmetic sequence.

[0102] That is: When: the origin pixel represents 'a' horizontally and 'b' vertically, each horizontal pixel increases by 'm', and each vertical pixel increases by 'n'; in this embodiment, the length unit is cm.

[0103] In the image, the physical coordinates of a point (x, y) relative to the origin in the real world are f(x, y) = (a + (x-1) * m, b + (y-1) * n);

[0104] Therefore, in this embodiment, the mapping method f(x, y) between the physical world and the surveillance image in this scenario needs to be found, and the parameters to be solved are a, b, m and n.

[0105] For each surveillance video, a mapping function of image pixels with respect to the real physical distance is set. Using the pixels where the right angles are marked, a system of equations about the mapping function is formed based on the relationship between the pixels and their distance in the physical world (100cm). The system of equations is solved using the gradient descent method to obtain the mapping function between the scene captured by the camera and the real physical world.

[0106] based on Figure 2 A square two-dimensional reference object is used, with P value of 4 and Q value of 3. The placement diagram yields the pixel coordinates of 12 points, grouped into sets of 4 (the four right-angled vertices of a reference object), resulting in 12 specific (x, y) coordinates in 3 groups of 4.

[0107] Based on the formula f(x,y)=(a+(x-1)*m,b+(y-1)*n), we obtain 12 f(x,y) values; according to the characteristics of a square, in this example, the physical distance between adjacent vertices is 100cm, and the physical distance between relative vertices is...

[0108] Within each group, based on the pairwise distances of the four points, we can obtain 6 equations (the lengths of the four sides plus the two diagonals); that is, when the reference is a square cardboard, the value of M is set to 18; a total of 18 equations are obtained, which can be regarded as a system of equations about a, b, m, n, in the form of:

[0109]

[0110] At this point, the problem of mapping the captured scene to the physical world is transformed into a problem of solving a system of equations.

[0111]

[0112] Based on the least squares method, the following relationships are constructed:

[0113] The problem of mapping the captured scene to the physical world is transformed into the problem of finding the minimum value of the function G. By directly applying the standard gradient descent method to find the minimum value of G, we can obtain a, b, m, n.

[0114] When the deformation relationship f is a proportional deformation: the origin pixel represents a horizontally and b vertically. Each horizontal pixel becomes m times the size of the previous pixel, and each vertical pixel becomes n times the size of the previous pixel.

[0115] The physical coordinates of the image point (x, y) relative to the origin in the real world are f(x, y) = (a*m) (x-1) ,b*n (y-1) );

[0116] The parameters that need to be solved for f(x, y) are a, b, m and n.

[0117] The solution process for proportional deformation is the same as that for arithmetic deformation.

[0118] Using the technical solution of this invention, the deformation relationship f between the shooting scene in a vast rural area and the real physical world is first preset to be linear deformation and proportional deformation. Then, the parameters of f(x,y) are obtained based on the minimum value. The parameters obtained in this way are the closest values, not the exact values. However, in practical applications, when measuring vehicle speed, it is not necessary to obtain the exact value of the vehicle speed; only the speed range is needed for the next step of judgment. Therefore, the value obtained by the method in this application is practical, and because the steps are simple and the calculation process can be made into a fixed program for automatic execution, the cost is extremely low, the implementation efficiency is fast, and it is very suitable for widespread use.

Claims

1. A method for mapping a scene captured in a video surveillance image to the real physical world, characterized in that, It includes the following steps: S1: Prepare reference objects. According to the preset ground reference object positions, place the reference objects on the road surface of the monitoring scene where the video surveillance equipment is located. The reference object uses a standard two-dimensional reference object; the position of the reference object includes at least 3 non-overlapping positions; S2: Extract video frames from the historical video clips when the reference object was placed from the data collected by the video surveillance equipment to obtain a surveillance image; The surveillance images include the reference objects distributed at different locations of the ground reference objects, which are respectively denoted as: annotation images; S3: The images are manually annotated, and the annotation position is the fixed position of the known size of the reference object in each image, denoted as: annotation pixel; S4: For video surveillance equipment, set the mapping function of image pixels with respect to the actual physical distance; Using the labeled pixels, a system of equations about the mapping function is formed based on the positional relationships between the labeled pixels and their physical distances; the specific method is as follows: a1: Establish a camera coordinate system for the image used for annotation; a2: Determine the deformation relationship between the shooting scene and the real physical world as f-deformation; The judgment method is as follows: When the road within the monitoring range of the video surveillance equipment is a straight road, the f-shape becomes a linear deformation; When the road within the monitoring range of dual video surveillance devices is a non-straight road, the deformation f is a proportional deformation; Then: for the coordinates Point(x,y) of the corresponding labeled pixel point in the labeled image, the actual labeled pixel point Point' on the reference object is (x',y'); That is: Point'(x',y')=f(x,y)Point(x,y); Therefore, solving for the mapping relationship between the monitoring scene and the real physical world becomes solving for the parameters in f(x,y); a3: In the camera coordinate system, for each of the P labeled pixels Point'(x',y') of the reference object, P sets of coordinates can be obtained; Assumptions: Each reference object has P labeled pixels Point'(x',y'); there are Q reference object positions, then on Q labeled images, there are a total of N = Q * P labeled pixels Point'(x,y); a4: Based on the actual size of the reference object, the positional relationship between each of the labeled pixels is known, so the distance Dis between any two labeled pixels is known; Dis=distance(Point' i -Point' j ), Where distance is the Euclidean distance between two points, and i and j are the pixel indexes of the labeled points, which are positive integers greater than or equal to 1 and less than or equal to N; Then, based on the distance between any two labeled pixels, the following relationship is obtained: Dis=distance(f(x,y)Point(x,y) i -f(x,y)Point(x,y) j ); a5: Construct a system of equations based on the known distances Dis between all labeled pixels; Suppose we construct a system of M equations, resulting in the following relationship: Dis1=distance(f(x,y)Point(x,y) i1 -f(x,y)Point(x,y) j1 ); .... Dis q =distance(f(x,y)Point(x,y) iq -f(x,y)Point(x,y) jq ); .... Dis M =distance(f(x,y)Point(x,y) iM -f(x,y)Point(x,y) jM ); In the formula, q is the index of the distance equation, and 1≤q≤M; S5: Solve the system of equations using the gradient descent method to obtain the mapping function between the shooting scene of the video surveillance equipment and the real physical world; make To solve this problem, a relation is constructed based on the least squares method: By applying the standard gradient descent method to find the minimum value of G, we can obtain the parameters in f(x,y) and thus obtain the mapping relationship between the scene captured in the video surveillance image and the real physical world f(x,y).

2. The method for mapping a shooting scene to the real physical world in a video surveillance image according to claim 1, characterized in that: When the deformation relationship f is a linear deformation: the origin pixel represents a horizontally and b vertically, with each horizontal pixel increasing by m and each vertical pixel increasing by n. The physical coordinates of the image point (x,y) relative to the origin in the real world are f(x,y)=(a+(x-1)*m,b+(y-1)*n); The parameters that need to be solved for f(x, y) are a, b, m and n.

3. The method for mapping a shooting scene to the real physical world in a video surveillance image according to claim 1, characterized in that: When the deformation relationship f is a proportional deformation: the origin pixel represents a horizontally and b vertically, each horizontal pixel becomes m times the size of the previous pixel, and each vertical pixel becomes n times the size of the previous pixel. The physical coordinates of the image point (x, y) relative to the origin in the real world are f(x, y) = (a*m) (x-1) ,b*n (y -1) ); The parameters that need to be solved for f(x, y) are a, b, m and n.

4. The method for mapping a shooting scene to the real physical world in a video surveillance image according to claim 1, characterized in that: The reference object includes a 100cm*100cm square cardboard; the corresponding marking positions are the pixels where the four right angles of the reference object are located.

5. The method for mapping a shooting scene to the real physical world in a video surveillance image according to claim 1, characterized in that: The reference points are set to be located at the upper left, middle, and lower right of the road in the monitoring screen.

6. The method for mapping a shooting scene to the real physical world in a video surveillance image according to claim 1, characterized in that: In step a3, the camera coordinate system uses the upper left corner of the monitoring image as the origin to construct a mapping relationship between the monitoring scene and the real physical world.

Images