Method for mapping shooting scene and real physical world in video monitoring image

Abstract

According to the method for mapping the shooting scene and the real physical world in the video monitoring image provided by the invention, only the two-dimensional reference object of a standard style needs to be placed on the road surface of the monitoring scene where the video monitoring equipment is located, and a plurality of marking pictures with the reference object are acquired through the video monitoring equipment; marking pixel points on the marking picture manually, then selecting a corresponding mapping relation f (x, y), forming a plurality of equation sets related to the mapping function f (x, y) based on the distance between the marking pixel points by utilizing the known size and position relation on a real two-dimensional reference object, and solving the equation sets by utilizing a gradient descent method to obtain f (x, y); y) to obtain the mapping function of the shooting scene of the camera and the real physical world.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a method for mapping a shooting scene in a video surveillance image to the real physical world. Background Art

[0002] At present, a large number of front-end devices such as dedicated monitoring cameras and video monitoring have been installed on the road for the collection of basic road traffic data. However, when the video sensor / monitoring equipment is used for applications such as speed measurement of vehicles, the video acquisition equipment itself cannot directly obtain the correspondence between the captured scene and the real physical world, and the collection of some information still needs to be realized with the help of additional equipment (such as radar). For example, the patent with publication number CN109615870A discloses a traffic detection system based on millimeter-wave radar and video. It collects target information through millimeter-wave radar and video equipment, converts world coordinates into image plane coordinates, determines the position of the target projected on the image plane detected by the millimeter-wave radar sensor, and finds the mapping relationship between the scene captured by the video acquisition equipment and the real physical world.

[0003] In order to reduce costs, some regions hope to use the existing fixed video surveillance equipment for traffic speed measurement. However, based on the above reasons, in order to achieve the speed measurement function, it is necessary to install auxiliary equipment such as radar. However, for some regions with poor economic development, even if only some radar equipment is added, the cost of these equipment and the subsequent maintenance costs of the equipment are also a relatively large expense.

[0004] In the existing technology, such as in the field of smart cars, there are also complex methods based on software such as machine learning models to achieve calibration and mapping of monitoring equipment and real physical world coordinate systems. However, the monitoring equipment installed in the vast rural areas is not installed uniformly at one time, but is gradually laid out with the national road infrastructure construction. Even the types and models of monitoring equipment in the same area may be inconsistent. If this complex method is used for mapping, each type of monitoring equipment needs to be calibrated and the model built separately. Although no new hardware equipment is required, it also requires a lot of technical personnel's design and debugging work hours, which also requires a high cost. Summary of the invention

[0005] In order to solve the problem that the existing fixed video surveillance equipment cannot directly obtain the corresponding relationship between the captured scene and the real physical world, and the collection of some information also needs the assistance of additional sensor equipment, or requires a lot of manpower, which leads to the problem of increased overall cost. The present invention provides a method for mapping the shooting scene in the video surveillance image and the real physical world. It does not need to add additional sensor equipment. Based on a simple method, it can realize the estimation of the relationship between the size of the captured scene and the distance in the real physical world in fixed video surveillance in rural areas.

[0006] The technical solution of the present invention is as follows: a method for mapping a shooting scene in a video surveillance image to a real physical world, characterized in that it comprises the following steps:

[0007] S1: Prepare reference objects, and place the reference objects on the road surface of the monitoring scene where the video monitoring device is located according to the preset ground reference object positions;

[0008] The reference object uses a standard two-dimensional reference object; the reference object position includes at least 3 non-overlapping positions;

[0009] S2: extracting video frames from the historical video clips when the reference object is placed from the data collected by the video surveillance device to obtain a surveillance picture;

[0010] The monitoring picture includes the reference objects distributed at different positions of the ground reference objects, which are respectively recorded as: a picture for annotation;

[0011] S3: manually annotating the annotation pictures, where the annotated positions are the pixel points of the fixed positions of the known sizes of the reference objects in each picture, recorded as annotated pixel points;

[0012] S4: For the video surveillance device, a mapping function of image pixels with respect to the real physical distance is set;

[0013] Using the labeled pixel points, according to the positional relationship between the labeled pixel points and the distance between the labeled pixel points in the physical world, a system of equations about the mapping function is formed; the specific method is as follows:

[0014] a1: Establishing a camera coordinate system for the annotation image;

[0015] a2: Determine the deformation relationship between the shooting scene and the real physical world as f deformation;

[0016] The judgment method is:

[0017] When the road within the monitoring range of the video surveillance equipment is a straight road, the f deformation becomes a linear deformation;

[0018] When the road within the monitoring range of the dual video surveillance devices is a non-linear road, the deformation f becomes a geometric deformation;

[0019] Then: for the coordinates Point(x,y) of the corresponding labeled pixel point in the labeled image, the actual labeled pixel point Point' on the reference object is (x',y');

[0020] That is: Point'(x',y')=f(x,y)Point(x,y);

[0021] Then, solving the mapping relationship between the monitoring scene and the real physical world becomes solving the parameters in f(x,y);

[0022] a3: In the camera coordinate system, for each reference object's P labeled pixel points Point(x,y), P sets of coordinates can be obtained;

[0023] Assume that: each reference object has a total of P labeled pixel points Point(x,y); there are Q reference object positions, then on the Q labeled images, there are a total of N=Q*P labeled pixel points Point(x,y);

[0024] a4: According to the actual size of the reference object, the positional relationship between each of the annotated pixels is known, and the distance Dis between any two annotated pixels is known;

[0025] Dis=distance(Point' i -Point' j ),

[0026] Wherein, distance is the Euclidean distance calculation method between two points, i and j are the serial numbers of the labeled pixels, and their values ​​are positive integers greater than or equal to 1 and less than or equal to N;

[0027] Then, according to the distance between any two of the labeled pixels, the following relationship is obtained:

[0028] Dis=distance(f(x,y)Point(x,y) i -f(x,y)Point(x,y) j );

[0029] a5: Construct a set of equations based on the distance Dis between all known labeled pixels;

[0030] Assuming that a total of M equations are constructed, the following relationship is obtained:

[0031] Dis 1 =distance(f(x,y)Point(x,y)i1 -f(x,y)Point(x,y) j1 ); ....

[0033] Dis q =distance(f(x,y)Point(x,y) iq -f(x,y)Point(x,y) jq ); ....

[0035] Dis M distance(f(x,y)Point(x,y) iM -f(x,y)Point(x,y) jM );

[0036] Where M is the known distance Dis, q is the serial number of the known distance, 1≤q≤M;

[0037] S5: Using the gradient descent method to solve the equation group, a mapping function between the shooting scene of the video surveillance device and the real physical world is obtained;

[0038] make

[0039]

[0040] To solve, a relationship is constructed based on the least squares method:

[0041] By applying the standard gradient descent method to find the minimum value of G, we can get the parameters in f(x, y) and obtain the mapping relationship f(x, y) between the shooting scene in the video surveillance image and the real physical world.

[0042] It is further characterized by:

[0043] When the deformation relationship f is a linear deformation: the origin pixel represents a horizontally and b vertically, each pixel increases by m horizontally and n vertically;

[0044] The physical coordinates of the image point (x, y) in the real world relative to the origin are f(x, y) = (a + (x-1) * m, b + (y-1) * n);

[0045] The parameters of f(x, y) that need to be solved are a, b, m and n;

[0046] When the deformation relationship f is geometric deformation: the origin pixel represents a horizontally and b vertically, each pixel in the horizontal direction becomes m times the previous pixel, and each pixel in the vertical direction becomes n times the previous pixel;

[0047] The physical coordinates of the image point (x, y) in the real world relative to the origin are f(x, y) = (a*m (x-1) ,b*n (y-1) );

[0048] The parameters of f(x, y) that need to be solved are a, b, m and n;

[0049] The reference object includes: a 1m*1m square cardboard; the corresponding marking positions are the pixel points where the four right angles of the reference object are located;

[0050] The reference position colors are set to: the upper left, middle, and lower right parts of the road in the monitoring image;

[0051] In step a3, the camera coordinate system takes the upper left corner of the monitoring image as the origin to construct a mapping relationship between the monitoring scene and the real physical world.

[0052] The present application provides a method for mapping a shooting scene in a video surveillance image to the real physical world. It only needs to use a two-dimensional reference object of a standard style to be placed on the road surface of the surveillance scene where the video surveillance device is located, collect multiple annotated pictures with reference objects through the video surveillance device, manually annotate the annotated pixels on the annotated pictures, and then select the corresponding mapping relationship f(x, y), use the known size and position relationship on the real two-dimensional reference object, form several equations about the mapping function f(x, y) based on the distance between the annotated pixels, solve the equations using the gradient descent method, and solve the closest value of each parameter in f(x, y), that is, obtain the mapping function between the shooting scene under the camera and the real physical world. This method does not require prior calibration of each type of video surveillance equipment, and does not require the addition of additional hardware equipment. It can achieve the solution of the mapping relationship between the shooting scene in the video surveillance image and the real physical world with simple operation and very low cost; it is particularly suitable for various types of video surveillance equipment on large rural roads that do not require precise solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a general flow chart of the method for mapping the shooting scene in the video surveillance image to the real physical world in this application;

[0054] Figure 2 Schematic diagram for placing reference objects;

[0055] Figure 3 Mark the four points of the reference object with a schematic diagram;

[0056] Figure 4 This is an example of the steps for calculating the mapping relationship. DETAILED DESCRIPTION

[0057] like Figure 1As shown, the present invention includes a method for mapping a shooting scene in a video surveillance image with the real physical world, which includes the following steps.

[0058] S1: Prepare reference objects, and place the reference objects on the road surface of the monitoring scene where the video monitoring equipment is located according to the preset ground reference object positions;

[0059] The reference object uses a standard style 2D reference object; the reference object positions include at least 3 non-overlapping positions.

[0060] At the same time, the mapping relationship between video surveillance images and the real physical world is fully extracted.

[0061] S2: In the data collected by the video surveillance equipment, video frames in the historical video clips when the reference objects are placed are extracted to obtain a surveillance picture; the surveillance picture includes reference objects distributed at different ground reference positions, which are respectively recorded as: annotation pictures.

[0062] S3: Manually annotate the annotation images, and the annotated positions are the pixels at the fixed positions of the known sizes of the reference objects in each image, recorded as annotated pixels.

[0063] like Figure 2 As shown, the reference objects in this embodiment include: 1m*1m square cardboard; Figure 3 As shown, the corresponding marked positions are the pixel points where the four right angles of the reference object are located. That is, in this embodiment, the marked pixel points are Figure 3 The four points in red;

[0064] The reference position is set to the upper left, middle, and lower right of the road in the monitoring screen. In specific implementation, the reference needs to be placed on the road surface three times, in the upper left, middle, and lower right of the road in the monitoring screen. The historical video clips of the placement of the reference are extracted and converted into several frames of pictures, from which the pictures of the reference placed in the upper left, middle, and lower right of the road surface are found, a total of three pictures.

[0065] In order to ensure that a sufficient number of equations can be established, this application uses four square paper plates, which are placed at the upper left, middle, and lower right parts of the road in the monitoring screen. At the same time, the use of simple two-dimensional reference objects is to reduce the overall cost. The use of 1m*1m square paper plates is to more easily establish the mathematical relationship between the annotated pixel points and simplify the workload in the annotation.

[0066] S4: For the video surveillance device, a mapping function of image pixels with respect to the real physical distance is set;

[0067] Using the labeled pixel points, according to the positional relationship between the labeled pixel points and the distance between the labeled pixel points in the physical world, a set of equations about the mapping function is formed; the specific method is as follows:

[0068] a1: Establish the camera coordinate system for the annotation image;

[0069] a2: Determine the deformation relationship between the shooting scene and the real physical world as f deformation;

[0070] The judgment method is:

[0071] When the road within the monitoring range of the video surveillance equipment is a straight road, the f deformation becomes a linear deformation;

[0072] When the road within the monitoring range of the dual video surveillance devices is a non-linear road, the deformation f becomes a geometric deformation.

[0073] In rural areas, video surveillance equipment is sparsely installed, and most of the roads that need speed measurement are relatively straight roads. In actual speed measurement, it is not necessary to accurately obtain the real-time speed of the vehicle, but only to know the speed range of the vehicle for subsequent judgment. In order to simplify the operation steps, two deformation relationships are preset in this application according to the actual road conditions.

[0074] Then: for the corresponding labeled pixel coordinate Point(x,y) in the labeled image, the actual labeled pixel Point' on the reference object is (x',y');

[0075] That is: Point'(x',y')=f(x,y)Point(x,y);

[0076] Then, the solution of the mapping relationship between the monitoring scene and the real physical world is transformed into the solution of the parameters in f(x, y).

[0077] In this method, the uncertain parameters are taken as the items to be solved. Based on the prior knowledge of the reference objects placed multiple times (for example, the board is 1 meter wide), several equations about the parameters are formed.

[0078] a3: In the camera coordinate system, for each reference object’s P labeled pixel points Point(x,y), P sets of coordinates can be obtained;

[0079] Assume that each reference object has P labeled pixel points Point(x,y); there are Q reference object positions, then there are N = Q*P labeled pixel points Point(x,y) on the Q labeled images.

[0080] a4: According to the actual size of the reference object, the positional relationship between each annotated pixel is known, so the distance Dis between any two annotated pixels is known;

[0081] Dis=distance(Point' i -Point' j ),

[0082] Wherein, distance is the Euclidean distance calculation method between two points, i and j are the serial numbers of the labeled pixels, and their values ​​are positive integers greater than or equal to 1 and less than or equal to N;

[0083] Specifically: distance(Point' i -Point' j )=sqrt[(x' i -x' j )2+(y' i -y' j )2].

[0084] Then, according to the distance between any two labeled pixels, the following relationship is obtained:

[0085] Dis=distance(f(x,y)Point(x,y) i -f(x,y)Point(x,y) j );

[0086] That is: distance(f(x,y)Point(x,y) i -f(x,y)Point(x,y) j )-Dis=0.

[0087] a5: Construct a set of equations based on the distance Dis between all known labeled pixels;

[0088] Assuming that a total of M equations are constructed, the following relationship is obtained:

[0089] Dis 1 =distance(f(x,y)Point(x,y) i1 -f(x,y)Point(x,y) j1 ); ....

[0091] Dis q =distance(f(x,y)Point(x,y) iq -f(x,y)Point(x,y) jq ); ....

[0093] Dis M distance(f(x,y)Point(x,y) iM -f(x,y)Point(x,y) jM );

[0094] Wherein, M is the known distance Dis, q is the serial number of the known distance, 1≤q≤M.

[0095] S5: Using the gradient descent method to solve the equation group, a mapping function between the shooting scene of the video surveillance device and the real physical world is obtained;

[0096] make

[0097]

[0098] To solve, a relationship is constructed based on the least squares method:

[0099] distance(f(x,y)Point(x,y) i -f(x,y)Point(x,y) j )-Dis=0, converted to (distance(f(x,y)Point(x,y) i -f(x,y)Point(x,y) j )-Dis) 2 Finding the minimum value;

[0100] By applying the standard gradient descent method to find the minimum value of G, we can get the parameters in f(x, y) and obtain the mapping relationship f(x, y) between the shooting scene in the video surveillance image and the real physical world.

[0101] like Figure 4 As shown, the deformation relationship f is a linear deformation. In this embodiment, the linear deformation is implemented based on an arithmetic progression.

[0102] That is, when: the origin pixel represents a horizontally and b vertically, each pixel increases by m horizontally and n vertically; in this embodiment, the unit of length is cm.

[0103] The physical coordinates of a point (x, y) in the image relative to the origin in the real world are f(x, y) = (a+(x-1)*m, b+(y-1)*n);

[0104] Then, in this embodiment, the mapping method f(x, y) between the physical world and the surveillance image in this scene is found, and the parameters that need to be solved are a, b, m and n.

[0105] For each surveillance video, a mapping function of image pixels to the real physical distance is set. Using the pixel points where the marked right angles are located, according to the relationship between the pixel points and their distance in the physical world (100 cm), several sets of equations about the mapping function are formed. The gradient descent method is used to solve the set of equations to obtain the mapping function between the scene captured by the camera and the real physical world.

[0106] based on Figure 2 The square two-dimensional reference object, P takes the value of 4, and Q takes the value of 3. The placement diagram obtains the pixel coordinates of 12 points, with each 4 points (the four right-angle vertices of a reference object) forming a group, that is, 12 specific (x, y) points are obtained in 3 groups of 4 points;

[0107] Based on the formula f(x,y) = (a + (x-1) * m, b + (y-1) * n), we get 12 f(x, y); according to the characteristics of the square, we know that the physical distance between adjacent vertices in this example is 100 cm, and the physical distance between relative vertices is

[0108] In each group, according to the distance between the four points, 6 equations can be obtained (four side lengths plus two diagonals); that is, when the reference is a square cardboard, the value of M is set to 18; a total of 18 equations are obtained, which can be regarded as a system of equations about a, b, m, and n, as follows:

[0109]

[0110] At this point, the mapping problem between the captured scene and the physical world is transformed into the problem of solving a system of equations.

[0111]

[0112] According to, the least squares method constructs the relationship:

[0113] Convert the mapping problem between the captured scene and the physical world into the problem of finding the minimum value of the function G. Directly apply the standard gradient descent method to find the minimum value of G to obtain a, b, m, and n.

[0114] When the deformation relationship f is geometric deformation: the origin pixel represents a horizontally and b vertically, each pixel in the horizontal direction becomes m times the previous pixel, and each pixel in the vertical direction becomes n times the previous pixel;

[0115] The physical coordinates of the image point (x, y) in the real world relative to the origin are f(x, y) = (a*m (x-1) ,b*n (y-1) );

[0116] The parameters of f(x, y) that need to be solved are a, b, m and n.

[0117] The solution process of geometric deformation is the same as that of arithmetic deformation.

[0118] After using the technical solution of the present invention, the deformation relationship f between the shooting scene in the vast rural area and the real physical world is first preset to be linear deformation and geometric deformation, and then the parameters of f(x, y) are obtained based on the minimum value. The closest value of each parameter obtained in this way is not the precise value. However, in actual applications, when measuring the speed of a vehicle, it is not necessary to obtain the precise value of the speed. It is only necessary to obtain the speed range to make the next judgment. Therefore, the value obtained by the method in this application is practical, and because the steps are simple and the calculation process can be made into a fixed program for automatic execution, the cost is extremely low and the implementation efficiency is fast, which is very suitable for promotion and use.

Claims

1. A method for mapping a shooting scene in a video surveillance image to the real physical world, characterized in that: It includes the following steps: S1: Prepare reference objects, and place the reference objects on the road surface of the monitoring scene where the video monitoring device is located according to the preset ground reference object positions; The reference object uses a standard two-dimensional reference object; the reference object position includes at least 3 non-overlapping positions; S2: extracting video frames from the historical video clips when the reference object is placed from the data collected by the video surveillance device to obtain a surveillance picture; The monitoring picture includes the reference objects distributed at different positions of the ground reference objects, which are respectively recorded as: a picture for annotation; S3: manually annotating the annotation pictures, where the annotated positions are the pixel points of the fixed positions of the known sizes of the reference objects in each picture, recorded as annotated pixel points; S4: For the video surveillance device, a mapping function of image pixels with respect to the real physical distance is set; Using the labeled pixel points, according to the positional relationship between the labeled pixel points and the distance between the labeled pixel points in the physical world, a system of equations about the mapping function is formed; the specific method is as follows: a1: Establishing a camera coordinate system for the annotation image; a2: Determine the deformation relationship between the shooting scene and the real physical world as f deformation; The judgment method is: When the road within the monitoring range of the video surveillance equipment is a straight road, the f deformation becomes a linear deformation; When the road within the monitoring range of the dual video surveillance devices is a non-linear road, the deformation f becomes a geometric deformation; Then: for the coordinates Point(x,y) of the corresponding labeled pixel point in the labeled image, the actual labeled pixel point Point' on the reference object is (x',y'); That is: Point'(x',y')=f(x,y)Point(x,y); Then, solving the mapping relationship between the monitoring scene and the real physical world becomes solving the parameters in f(x,y); a3: In the camera coordinate system, for each reference object's P labeled pixel points Point(x,y), P sets of coordinates can be obtained; Assume that: each reference object has a total of P labeled pixel points Point(x,y); there are Q reference object positions, then on the Q labeled images, there are a total of N=Q*P labeled pixel points Point(x,y); a4: According to the actual size of the reference object, the positional relationship between each of the annotated pixels is known, and the distance Dis between any two annotated pixels is known; Dis=distance(Point' i -Point' j ), Wherein, distance is the Euclidean distance calculation method between two points, i and j are the serial numbers of the labeled pixels, and their values ​​are positive integers greater than or equal to 1 and less than or equal to N; Then, according to the distance between any two of the labeled pixels, the following relationship is obtained: Dis=distance(f(x,y)Point(x,y) i -f(x,y)Point(x,y) j ); a5: Construct a set of equations based on the distance Dis between all known labeled pixels; Assuming that a total of M equations are constructed, the following relationship is obtained: Dis1=distance(f(x,y)Point(x,y) i1 -f(x,y)Point(x,y) j1 ); .... Dis q =distance(f(x,y)Point(x,y) iq -f(x,y)Point(x,y) jq ); .... Dis M distance(f(x,y)Point(x,y) iM -f(x,y)Point(x,y) jM ); Where M is the known distance Dis, q is the serial number of the known distance, 1≤q≤M; S5: Using the gradient descent method to solve the equation group, a mapping function between the shooting scene of the video surveillance device and the real physical world is obtained; make To solve, a relationship is constructed based on the least squares method: By applying the standard gradient descent method to find the minimum value of G, we can get the parameters in f(x, y) and obtain the mapping relationship f(x, y) between the shooting scene in the video surveillance image and the real physical world.

2. According to claim 1, a method for mapping a shooting scene in a video surveillance image to a real physical world is characterized by: When the deformation relationship f is a linear deformation: the origin pixel represents a horizontally and b vertically, each pixel increases by m horizontally and n vertically; The physical coordinates of the image point (x, y) in the real world relative to the origin are f(x, y) = (a + (x-1) * m, b + (y-1) * n); The parameters of f(x, y) that need to be solved are a, b, m and n.

3. According to claim 1, a method for mapping a shooting scene in a video surveillance image to a real physical world is characterized by: When the deformation relationship f is geometric deformation: the origin pixel represents a horizontally and b vertically, each pixel in the horizontal direction becomes m times the previous pixel, and each pixel in the vertical direction becomes n times the previous pixel; The physical coordinates of the image point (x, y) in the real world relative to the origin are f(x, y) = (a*m (x-1) ,b*n (y -1) ); The parameters of f(x, y) that need to be solved are a, b, m and n.

4. According to claim 1, a method for mapping a shooting scene in a video surveillance image to a real physical world is characterized by: The reference object includes: a 1m*1m square cardboard; the corresponding marking positions are the pixel points where the four right angles of the reference object are located.

5. According to claim 1, a method for mapping a shooting scene in a video surveillance image to a real physical world is characterized by: The reference position colors are set to be: the upper left, middle and lower right parts of the road in the monitoring image.

6. According to claim 1, a method for mapping a shooting scene in a video surveillance image to a real physical world is characterized by: In step a3, the camera coordinate system takes the upper left corner of the monitoring image as the origin to construct a mapping relationship between the monitoring scene and the real physical world.

Images