Camera-based vehicle monitoring video speed measurement method and device

Abstract

The application discloses a kind of camera-based vehicle monitoring video speed measurement method and device, the method comprises: TOF camera is erected above road;Selecting marker is placed on road, and adjust TOF camera so that it can completely photograph marker;Depth information image and color image obtained by TOF camera are registered;Based on the depth information image and color image of registration, determine the two-dimensional coordinates of each pixel point in color image corresponding to actual road plane, obtain coordinate calibration matrix, complete calibration;Yolov4 target identification network is used to identify and track automobile target in color image, and the space position and time interval of automobile in different video frames are calculated to obtain the driving speed of automobile.The application not only improves the monitoring video automobile speed measurement precision, and greatly simplifies the complexity of calibration step.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of vehicle video speed measurement, and particularly relates to a camera-based vehicle monitoring video speed measurement method and device. BACKGROUND

[0002] Among a series of current vehicle speed measurement methods, the video speed measurement method directly analyzes the video images captured by the road monitoring camera, identifies and tracks the vehicles appearing in the images, and then converts the positions of the vehicles in the images into their spatial coordinates on the road surface according to the calibration matrix of the monitoring camera obtained in advance. In this way, the displacement distance of the vehicles between different frames can be calculated, and finally the moving speed of the vehicles can be calculated in combination with the acquisition frame rate of the monitoring camera. Compared with other vehicle speed measurement methods, the video speed measurement method obtains more intuitive information. This method can track vehicles and measure their speed in real time. Once it is found that a vehicle is speeding, the driving condition of the vehicle is recorded and saved by using video shooting. In the later stage, the traffic control department can provide intuitive and effective evidence when punishing and handling complaints of speeding vehicles. At the same time, the vehicle video speed measurement method has low cost and can be realized only by using a monitoring camera without introducing other complex equipment, which has small installation and construction amount and is easy to maintain.

[0003] However, in the research and application of vehicle video speed measurement, this method is also affected by the calibration accuracy of the camera. In the research of machine vision, the most commonly used camera calibration method is the "Zhang calibration method" proposed by Zhang Zhengyou in 1998. The size of the calibration board used in this method needs to account for more than 1 / 4 of the shooting scene. For speed measurement cameras that shoot larger scenes, it is not only expensive to make a larger area calibration board, but also makes the calibration process complex and difficult. Therefore, a Chinese patent with the publication number CN114782549A and the name of camera calibration method and system based on fixed-point identification proposes a method of placing a number of equidistant roadblocks on the road as calibration points to calibrate the monitoring camera. Although this method avoids using a larger area calibration board, it is affected by the imaging law of the camera lens. The captured image presents a situation of near large and far small. The row spacing between adjacent roadblocks in the image is not the same, but a nonlinear change. This will seriously affect the calibration accuracy of the camera. In the image, the roadblock is not a calibration point in the strict sense, but an object with a certain area. Therefore, the selection of the calibration point also limits the calibration accuracy of the camera. At the same time, the equidistant placement accuracy of the roadblock is also difficult to guarantee, which limits the application of this method in actual scenes. SUMMARY

[0004] The application aims to provide a camera-based vehicle monitoring video speed measurement method and device, and solve the problems of traditional vehicle video speed measurement systems, such as high labor cost, low precision, and difficult calibration, so as to realize high-precision calibration of monitoring cameras and finally improve the accuracy of the video vehicle speed measurement system.

[0005] The technical solution for achieving the object of the application is a camera-based vehicle monitoring video speed measurement method, comprising:

[0006] The TOF camera is erected above the road;

[0007] An identification object is selected and placed on the road, and the TOF camera is adjusted to be able to completely shoot the identification object;

[0008] The depth information image and the color image obtained by the TOF camera are registered;

[0009] Based on the registered depth information image and color image, the two-dimensional coordinates of each pixel point in the color image corresponding to the actual road plane are determined, the coordinate calibration matrix is obtained, and the calibration is completed;

[0010] The Yolov4 target recognition network is used to identify and track the vehicle target in the color image, and the driving speed of the vehicle is calculated according to the spatial position and time interval of the vehicle in different video frames.

[0011] Further, the TOF camera is erected above the road, including erecting the TOF monitoring camera above the road, shooting the road surface at a certain angle from above, so that it can record the monitored road section completely, and the yaw angle and roll angle are usually zero.

[0012] Further, the identification object includes a regular object with a convex shape, which is placed on the road so that it can be clearly recorded by the color detector of the TOF camera, and the identification object and its contour can be clearly recorded by the depth detector.

[0013] Further, the registration of the depth information image and the color image obtained by the TOF camera specifically includes:

[0014] The iterative threshold segmentation algorithm is used to segment the identification object region in the depth information image and the color image, respectively;

[0015] The centroid coordinates of the identification object region in the depth information image and the color image are determined based on the segmented identification object region, respectively;

[0016] The difference between the centroid coordinates of the depth information image and the color image is determined, and the depth information image and the color image are registered through affine transformation.

[0017] Further, the iterative threshold segmentation algorithm is used to segment the marker region in the depth information image and the color image respectively, which can be divided into the following steps:

[0018] Step one: use Gaussian filter to filter and denoise the image;

[0019] Step two: determine the initial threshold T k according to the maximum gray value G max and the minimum gray value G min of the filtered and denoised image k T max = (G min + G k ) / 2;

[0020] Step three: segment the image into target and background parts according to the threshold T k , and then calculate the average gray values G O and G B of the two parts;

[0021] Step four: determine the new threshold T k+1 according to the average gray values of the target and background parts O T B = (G k+1 + G k ) / 2;

[0022] Step five: repeat step three and step four until the new threshold T k+1 is equal to the threshold T k of the last iteration or the maximum number of iterations is reached;

[0023] Step six: separate the marker region from the background region in the image according to the obtained threshold T k , set the gray value of the marker region to 1 and the gray value of the background region to 0.

[0024] Further, the method for calculating the centroid coordinates of the marker region is as follows: according to the marker region in the color image and the depth information image obtained after threshold segmentation, the average value of all pixel point coordinates of the marker region is calculated to obtain the centroid coordinates of the marker region in the depth information image and the color image.

[0025] Further, the registration of the depth information image and the color image through affine transformation specifically includes the following steps:

[0026] Step one: calculate the difference (X diff , Y diff ) of the centroid coordinates of the marker region in the two images;

[0027] Step two: establish a 2*3 size affine transformation matrix T according to the difference of the centroid coordinates, since only translation is considered and rotation and scaling are not considered, so the affine transformation matrix T is

[0028] Step three: affine transform the depth information image according to the affine transformation matrix T to make it registered with the color image, so the position of each pixel point in the new depth information image is

[0029] Further, the two-dimensional coordinates of each pixel point in the color image corresponding to the actual road plane are determined, which are used for subsequent positioning and speed measurement of the target vehicle, and the specific process includes:

[0030] Step one: first, determine the coordinate axis direction of the monitored road surface, select the width direction of the road surface as the coordinate axis x-axis direction, and select the length direction of the road surface as the coordinate axis y-axis direction;

[0031] Step two: since the TOF monitoring camera is used to shoot the road surface at a certain angle, the yaw angle and roll angle are usually zero, at this time the y-axis coordinate value of each row of pixels in the image shot by the camera is the same and the x-axis coordinate value of these pixels corresponding to the actual road surface is uniformly distributed;

[0032] Step three: select two longitudinal virtual or solid lines in the color road image recorded by the TOF camera as reference lines, select one of the reference lines as the road space coordinate axis y-axis, the x-axis coordinate value of all points on this line is x0=0. Measure the actual distance w between the two reference lines, so that the x-axis coordinate value of all pixel points on the other reference line corresponding to the actual road surface is x1=w. Then, calculate the x-axis coordinate value of other pixel points on each row according to the uniform distribution principle, so that the x-axis coordinate value of all pixel points in the entire image corresponding to the actual road surface is obtained;

[0033] Step four: according to the camera imaging law, the farther the target from the camera, the smaller the size in the image, so in the image, the road presents the phenomenon of narrow on the top and wide on the bottom, and in the monitoring picture, the pixel between the adjacent rows on the top of the picture corresponds to the larger y-axis interval of the actual road surface, while the pixel between the adjacent rows on the bottom of the picture corresponds to the smaller y-axis interval of the actual road surface, at this time, the row index value of the pixel point in the image and its coordinate value on the y-axis of the actual road surface are a non-linear mapping relationship. In order to accurately calibrate the y-axis coordinate value of each pixel point in the image, we use the depth detection function of the TOF camera, according to the depth information image taken by the TOF camera, to obtain the distance between each point on the road surface and the TOF camera, and then according to the distance H between the TOF camera and the ground, the horizontal distance d between the TOF camera and the y-axis of the space coordinate axis and the x-axis coordinate value of each pixel point corresponding to the actual road surface, the y-axis coordinate value of each pixel point corresponding to the actual road surface can be accurately calculated by using the Pythagorean theorem;

[0034] Step five: combine all the coordinates of the pixel points corresponding to the actual road surface obtained in steps three and four, and arrange them in the form of a coordinate calibration matrix, so as to complete the calibration of the camera.

[0035] A kind of vehicle monitoring video speed based on monocular TOF camera based on camera vehicle monitoring video speed device, including TOF camera, marker, image registration module, calibration module and Yolov4 target identification network module;Wherein the marker is placed on the road, the TOF camera is used to shoot depth information image and color image, the image registration module is used to register depth information image and color image, the calibration module is used to determine the two-dimensional coordinate of each pixel point in color image corresponding to actual road plane, obtains coordinate calibration matrix;The Yolov4 target identification network module is used to identify and track the vehicle target in color image, and the space position and time interval of vehicle in different video frames are used to calculate the driving speed of vehicle.

[0036] Compared with the prior art, the beneficial effects of the present application are as follows:

[0037] (1) the present application uses TOF camera instead of traditional color camera to monitor vehicle speed, by using TOF camera to shoot color video image and depth information image, the two are combined to calibrate the camera, which overcomes the difficulty of y-axis coordinate calibration in traditional calibration method;

[0038] (2) the present application uses iterative threshold segmentation algorithm to segment the color image and depth information image before calibration, and the depth information image and the color image are registered based on the centroid coordinates by affine transformation, and the calibration after registration can realize the high-precision calibration of the monitoring camera;

[0039] (3) The Yolov4 target recognition network is introduced to realize high-precision recognition and positioning of the automobile, and finally the camera calibration matrix is obtained to measure the moving speed of the automobile, thereby further improving the measurement precision of the automobile video speed measurement. BRIEF DESCRIPTION OF DRAWINGS

[0040] Fig. 1(a) is a color image of the marker taken by the TOF camera, and Fig. 1(b) is a depth information image taken by the TOF camera.

[0041] Fig. 2(a) is a color image of the marker extracted after threshold segmentation, and Fig. 2(b) is a depth information image of the marker extracted after threshold segmentation.

[0042] Figure 3 It is a matching result image of the depth information image and the color image.

[0043] Figure 4 It is a schematic diagram of the reference line and the coordinate axis direction.

[0044] Figure 5 It is a schematic diagram of the x-axis coordinate value of the two reference lines.

[0045] Figure 6 It is a schematic diagram of the TOF camera calibration principle.

[0046] Figure 7 It is an effect diagram of the TOF camera recognizing, tracking and measuring the moving ball. DETAILED DESCRIPTION

[0047] The preferred embodiments will be described in detail below with reference to the accompanying drawings. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present application and its applications.

[0048] The embodiment provides a method applicable to expressway vehicle video speed measurement, comprising: erecting a TOF camera above a road surface, adjusting a monitoring camera angle, so that the monitoring camera is directed towards the same direction as the road surface and monitors the road surface in a top-down manner; placing an identification object with a convex feature on the monitored road surface, so that the identification object can be simultaneously captured by a depth detector and a color detector of the TOF camera; recording a color image and a depth information image of the road section by using the color detector and the depth detector of the TOF camera; separating an identification object region in the color image and the depth information image by using a threshold segmentation algorithm respectively; finding a centroid position of the identification object region in the color image and the depth information image by using a centroid positioning algorithm; matching the depth information image and the color image by using affine transformation according to the centroid positions of the identification object in the two images, so that the problem of non-matching of the color image and the depth information image is solved; removing the identification object, selecting two straight lane lines in a color video picture as reference lines, and combining the depth information image to calculate the calibration of the TOF camera according to a geometric relationship, so that the spatial coordinates of each pixel in the color picture corresponding to an actual road surface are obtained; identifying a vehicle target from the color video by using a target recognition algorithm and tracking the vehicle target, calculating a moving distance of the vehicle according to the spatial coordinates of the vehicle in different video frames, and calculating a driving speed of the vehicle on the road according to a shooting frame rate of the camera. The method specifically comprises

[0049] The embodiment uses a TOF (Time of flight, TOF) camera to measure the speed of a moving ball in an indoor environment. The maximum detection depth of the TOF camera is 3 m, the height of the camera from the ground is 0.45 m, two white reference lines in a shooting picture are used to simulate parallel lane lines on an actual expressway, the distance between the two lines is 0.64 m, and the horizontal distance of the TOF camera from the left reference line is 0.3 m.

[0050] In the embodiment, the TOF camera is first erected above the ground, and the ground is shot in a top-down manner with a certain angle. The yaw angle and the roll angle are adjusted to be zero. Then a regular-shaped ball is selected as an identification object and placed on the ground. After that, the TOF camera is turned on, and the position, contour, whether there is an occlusion and the definition of the ball in the depth information image and the color image are observed. According to the actual situation, the position of the ball is continuously adjusted, and finally the depth information image and the color image at the best position are recorded, as shown in FIG. 1(a) and FIG. 1(b).

[0051] The threshold segmentation algorithm is used to separate the ball region in the depth information image and the color image. The specific steps are as follows:

[0052] Step one: a Gaussian convolution kernel with a size of 3*3 and a variance of 1 is used to perform Gaussian filtering and denoising on the image.

[0053] Step 2: Based on the maximum grayscale value G of the filtered image max and minimum gray value G min Determine the initial threshold T k For T k =(G max +G min ) / 2;

[0054] Step 3: Based on the threshold T k The image is segmented into two parts: the target and the background. Then, the average gray value G of these two regions is calculated. O and G B ;

[0055] Step 4: Determine a new threshold T based on the calculated average gray values ​​of the target and background regions. k+1 =(G O +G B ) / 2;

[0056] Step 5: Repeat steps 3 and 4 until a new threshold T is reached. k+1 Equal to the threshold T of the previous iteration k Or, it can reach the maximum number of iterations set, which we have set to 50 here;

[0057] Step 6: Based on the obtained threshold T k The spherical region in the image is separated from the background region. The gray value of the spherical region is set to 1, and the gray value of the background region is set to 0, as shown in Figure 2(a) and Figure 2(b).

[0058] The centroid extraction algorithm is used to calculate the coordinates of the centroid point of the sphere region in the depth information image and the color image. The centroid coordinates are obtained by calculating the average value of the coordinates of all pixels in the target region.

[0059] The steps for registering the TOF camera depth information image with the color image are as follows:

[0060] Step 1: Based on the coordinates of the spherical region in the depth information image and the color image, calculate the difference (X) between the centroid coordinates of the spherical region in the two images. diff ,Y diff );

[0061] Step 2: Construct a 2x3 affine transformation matrix t based on the difference in centroid coordinates. Since only translation is considered and rotation and scaling are ignored, the affine transformation matrix...

[0062] Step 3: Perform an affine transformation on the depth information image based on the affine transformation matrix t to register it with the color image. Therefore, the positions of each pixel in the resulting new depth information image are... After registration as Figure 3 ;

[0063] After completing the image registration, the color detector in the TOF camera can be calibrated according to the depth information image, combined with Figures 4-7 , the specific steps are as follows:

[0064] Step one: remove the identification object from the picture, use the color shooting function and depth shooting function of the TOF camera to record the color image and depth information image;

[0065] Step two: determine the direction of the ground space coordinate axis, select the direction perpendicular to the reference line as the space coordinate axis x axis direction, and select the direction parallel to the reference line as the space coordinate axis y axis direction;

[0066] Step three: select the left reference line as the space coordinate axis y axis, so that the x axis coordinate value of all points on this line is 0, and the x axis coordinate value of all points on the right reference line is 0.64;

[0067] Step four: we ignore the interference of lens aberration on the shooting image, and can consider that the actual space x axis coordinate value corresponding to each row of pixel points in the picture is uniformly distributed. According to the number of pixels between the two reference lines in the image and the distance between the two reference lines in the actual space, the x axis coordinate value of other pixel points in each row is calculated by using the equidistant filling method, so that the x axis coordinate value of each pixel point in the shooting image corresponding to the actual space is obtained, which is represented by calx;

[0068] Step five: then according to the depth image obtained by the TOF camera, the distance Depth between each point on the road surface and the TOF camera is obtained, and the x axis coordinate value calx of each pixel point corresponding to the actual space is calculated, and the y axis coordinate value caly of each pixel point corresponding to the actual space is calculated by using the space geometric relationship;

[0069] Step six: in the depth image obtained by shooting, we see that due to the fact that the shooting scene exceeds the depth detection range of the TOF camera or due to the fact that the shooting scene is not uniformly reflected, there is a large error in the y axis coordinate value calculated finally, so we assign the y axis coordinate value of this part of the region to zero, so as to avoid the trouble of subsequent target tracking and speed measurement;

[0070] Step seven: arrange the x, y axis coordinate values (calx, caly) of the pixel points corresponding to the actual space into a matrix form, and we have completed the calibration of the TOF camera.

[0071] After the calibration of the camera is completed, the moving ball in the color video picture is recognized in combination with a Yolov4 target recognition network, and the ball is tracked with the center point of the identification frame as the spatial position of the ball, the displacement distance of the ball is measured every 10 frames, and the average speed of the ball in the time period is calculated according to the shooting frame rate of the camera.

[0072] A camera-based vehicle monitoring video speed measuring device, comprising a TOF camera, a marker, an image registration module, a calibration module and a Yolov4 target recognition network module; wherein the marker is placed on the road, the TOF camera is used to shoot depth information images and color images, the image registration module is used to register the depth information images and the color images, the calibration module is used to determine the two-dimensional coordinates of each pixel point in the color image corresponding to the actual road plane, and obtain the coordinate calibration matrix; the Yolov4 target recognition network module is used to identify and track the vehicle target in the color image, and calculate the driving speed of the vehicle according to the spatial position and time interval of the vehicle in different video frames.

[0073] It should be noted that each module of the speed measuring device contains all the technical features of the method, and will not be repeated here.

[0074] The present application aims at the problems of difficult calibration of ordinary cameras, poor calibration effect and the like affecting the measurement accuracy of the vehicle video speed measurement system, and proposes a monocular TOF camera vehicle video speed measurement method, uses a TOF camera instead of a traditional color camera as a monitoring probe, uses the depth detection function of the TOF camera, combines the color video images shot by the TOF camera, realizes high-precision calibration of the monitoring camera, and finally improves the accuracy of the video vehicle speed measurement system.

Claims

1. A camera-based vehicle monitoring video speed measurement method, characterized in that, The application relates to a method for calibrating a camera and a speed measurement system. The method comprises the following steps: a TOF camera is arranged above a road; a marker is selected and placed on the road, and the TOF camera is adjusted to be able to completely shoot the marker; a depth information image obtained by the TOF camera is registered with a color image; based on the registered depth information image and color image, the two-dimensional coordinates of each pixel point in the color image corresponding to an actual road plane are determined, a coordinate calibration matrix is obtained, and calibration is completed; a Yolov4 target recognition network is used to recognize and track a vehicle target in the color image, and the speed of the vehicle is calculated according to the spatial position and time interval of the vehicle in different video frames; the method for determining the two-dimensional coordinates of each pixel point in the color image corresponding to the actual road plane and obtaining the coordinate calibration matrix comprises the following steps: step one: the coordinate axis direction of the monitored road surface is determined, the width direction of the road surface is selected as the x-axis direction of the coordinate axis, and the length direction of the road surface is selected as the y-axis direction of the coordinate axis; Step three: calculate the y-axis coordinate value of each pixel point corresponding to the actual road surface, wherein, Depth is the distance between the pixel point obtained from the depth information image and the TOF camera, H is the distance between the measured TOF camera and the ground, d is the horizontal distance between the TOF camera and the y-axis of the space coordinate axis, and calx is the x-axis coordinate value of the pixel point corresponding to the actual road surface calculated in step two. step two: two longitudinal virtual lines or real lines passing through the image are selected as reference lines from the color image recorded by the TOF camera, one of the reference lines is selected as the y-axis of the road space coordinate axis, the x-axis coordinate value of all points on the reference line is x0=0, the actual distance w between the reference line and another reference line is measured, and the x-axis coordinate value of all pixel points on the other reference line corresponding to the actual road surface is determined as x1=w; the x-axis coordinate values of other pixel points on each row are calculated according to the principle of uniform distribution, and the x-axis coordinate values of all pixel points in the entire image corresponding to the actual road surface are obtained; 2. The camera-based car monitoring video speed measurement method according to claim 1, wherein, step four: the x-axis and y-axis coordinates of all pixel points corresponding to the actual road surface in steps two and three are combined to determine the coordinate calibration matrix, and the calibration of the camera is completed. the method for registering the depth information image and the color image obtained by the TOF camera comprises the following steps: an iterative threshold segmentation algorithm is used to segment the marker area in the depth information image and the color image respectively; the centroid coordinates of the marker area in the depth information image and the color image are determined respectively based on the segmented marker area; 3. The camera-based car monitoring video speed measurement method according to claim 2, wherein, the difference between the centroid coordinates of the depth information image and the color image is determined, and the depth information image and the color image are registered through affine transformation. the method for segmenting the marker area in the depth information image and the color image respectively by using the iterative threshold segmentation algorithm comprises the following steps: Step two: determine the initial threshold T according to the maximum gray value G and the minimum gray value G of the filtered image max min k k = (G max + G min ) / 2;​​​ Step three: according to threshold T k The image is divided into two parts, target and background, and the average gray value G of the two parts is calculated respectively O And G B ; Step four: Determine a new threshold T according to the average gray value of the target and background regions k+1 , T k+1 = (G O + G B ) / 2; Step five: repeat step three and step four until a new threshold T k+1 is equal to the threshold T of the last iteration k or a set maximum number of iterations is reached; Step six: the threshold value T obtained according to step five k The marker region and the background region are determined, the gray value of the marker region is set to 1, and the gray value of the background region is set to 0.

4. The camera-based car monitoring video speed measurement method according to claim 2, wherein, step one: Gaussian filtering is used to filter and denoise the depth information image and the color image; 5. The camera-based vehicle monitoring video speed measurement method of claim 2, wherein, the centroid coordinates of the marker area are the average values of the coordinates of all pixel points in the marker area. An affine transformation matrix T is established according to the difference of the barycentric coordinates, where X diff , Y diff are the difference of the barycentric coordinates; According to the affine transformation matrix T, the depth information image is subjected to affine transformation to make it registered with the color image, so that the position of each pixel point in the new depth information image is 6. The camera-based vehicle monitoring video speed measurement method according to any one of claims 1-5, characterized in that, the method for registering the depth information image and the color image through affine transformation comprises the following steps:

7. The camera-based car monitoring video speed measurement method according to claim 6, wherein, the yaw angle and the roll angle of the TOF camera are zero.

8. The camera-based car monitoring video speed measurement method according to claim 7, wherein, the marker is a regular object protruding from the road surface. the marker is a sphere.

9. A camera-based vehicle monitoring video speed measurement device implementing the method of any one of claims 1-8, characterized in that, The application relates to a TOF camera, a marker, an image registration module, a calibration module and a Yolov4 target recognition network module; wherein the marker is placed on a road, the TOF camera is used for shooting a depth information image and a color image, the image registration module is used for registering the depth information image and the color image, the calibration module is used for determining the two-dimensional coordinates of each pixel point in the color image corresponding to an actual road plane, and a coordinate calibration matrix is obtained; the Yolov4 target recognition network module is used for identifying and tracking a car target in the color image, and the driving speed of the car is calculated according to the spatial position and time interval of the car in different video frames.

Images