A real-time vehicle speed extraction method based on MF fusion algorithm for roadside monocular camera

Abstract

The present invention discloses a method for extracting real-time vehicle speed from a roadside monocular camera based on an MF fusion algorithm, comprising the following steps: S1: installing a monocular camera on the roadside to collect road video images; S2: performing frame processing on the road video images to form frame-by-frame images; S3: performing a monocular depth estimation algorithm using a MiDaS model; S4: obtaining two adjacent frame-by-frame images I t and I t+1 , using the Farneback optical flow method to calculate the optical flow between adjacent images; S5: By combining the optical flow vector with the depth map, the pixel speed is calculated to obtain the vehicle running speed. The present invention adopts the above-mentioned method for extracting real-time vehicle speed from a roadside monocular camera based on the MF fusion algorithm, predicts the pixel depth of each frame of the image through a depth estimation algorithm, and combines the optical flow method to capture the motion information of pixels in adjacent frames to calculate the actual speed of the vehicle, significantly improving the real-time performance and reliability of roadside traffic flow monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of road monitoring, and in particular to a method for extracting real-time vehicle speed from a roadside monocular camera based on an MF fusion algorithm. Background Art

[0002] In the field of traffic monitoring, especially roadside surveillance systems, extracting vehicle speed faces multiple technical challenges that impact the accuracy and reliability of traffic data. First, roadside surveillance videos often experience significant image obstruction, especially during high-traffic periods. The overlapping presence of other vehicles makes it difficult to identify and track specific vehicles. This obstruction makes it difficult for the monitoring system to effectively assess the actual speed of each vehicle, hindering comprehensive analysis of traffic flow.

[0003] Secondly, image distortion in surveillance videos is often unavoidable due to the camera's installation position and viewing angle. This distortion manifests as nonlinear characteristics in roadside monitoring, affecting target depth estimation and velocity calculation. While camera intrinsic parameter correction can often be used to correct image distortion, in practice, due to the specific nature of roadside settings, traditional intrinsic parameter correction methods often cannot accurately adapt to the changing viewing angles and distortion characteristics. Especially when using wide-angle lenses, the degree of distortion can vary significantly depending on the viewing angle, resulting in poor correction results. Furthermore, the lack of fixed reference objects in the surveillance scene limits the effective calibration process and makes obtaining camera intrinsic parameters difficult.

[0004] Therefore, extracting accurate speed information from monocular surveillance video has become a pressing task. To overcome these challenges, developing new technologies that can handle nonlinear distortion in dynamic scenes and directly extract vehicle speed data from video has become a research focus. These technologies must fully consider the unique environment of roadside surveillance to ensure reliable traffic data analysis in complex situations. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time vehicle speed extraction method for a roadside monocular camera based on the MF fusion algorithm. The method can accurately calculate the vehicle speed without relying on the camera's intrinsic parameters. The pixel depth of each frame is predicted by the depth estimation algorithm, and the motion information of pixels in adjacent frames is captured in combination with the optical flow method to calculate the actual speed of the vehicle.

[0006] The present invention provides a method for extracting real-time vehicle speed from a roadside monocular camera based on an MF fusion algorithm, comprising the following steps:

[0007] S1: Install a monocular camera on the roadside to collect road video images;

[0008] S2: performing frame processing on the road video image to form frame-by-frame images;

[0009] S3: Use the MiDaS model to perform a monocular depth estimation algorithm to predict the depth information of each frame and obtain the depth map of each frame.

[0010] S4: Acquire two adjacent frame-by-frame images I t and I t+1 ,Use Farneback optical flow method to calculate the optical flow between adjacent images, analyze the pixel changes between adjacent frame-by-frame images, and generate the optical flow vector of each pixel;

[0011] S5: By combining the optical flow vector with the depth map, the vehicle speed is obtained by calculating the pixel speed;

[0012] In step S5, the parallax displacement is obtained by the magnitude of the optical flow vector; the parallax displacement of the pixel point is expressed as the modulus of the optical flow vector:

[0013]

[0014] u(x,y) represents the horizontal optical flow component at the position (x,y) in the image; v(x,y) represents the vertical optical flow component at the position (x,y) in the image;

[0015] In step S5, the frame rate of the camera is f, and the time interval between adjacent frames is

[0016] The actual displacement Δs(x,y) is calculated by multiplying the parallax displacement of the pixel by the depth value:

[0017] Δs(x,y)=d(x,y)·D t (x,y);

[0018] Among them, D t (x,y) represents the depth change of pixel (x,y);

[0019] The speed of each pixel is obtained by dividing the actual displacement by the time interval:

[0020]

[0021] Preferably, in step S3,

[0022] The frame-by-frame image is adjusted to the size H*W required by the MiDaS model; the adjusted frame-by-frame image I prep Expressed as:

[0023] I prep =Resize(I,H,W);

[0024] Among them, I prepRepresents the adjusted frame-by-frame image; Resize means adjusting the image to the specified height H and width W, keeping the number of channels I of the image unchanged; I: represents the number of channels of the input image, representing the three color channels of the RGB image; H: represents the height of the image; W: represents the width of the image;

[0025] For the adjusted frame-by-frame image I prep Perform normalization processing, and the normalized frame-by-frame image I norm Expressed as:

[0026]

[0027] Among them, I norm Represents the normalized frame-by-frame image; μ is the mean used during training;

[0028] The normalized frame-by-frame image I norm Input the pre-trained MiDaS model M and use the convolutional neural network CNN to extract the normalized frame-by-frame image I norm Extract features and predict the depth information of each pixel. The output of the MiDaS model M is the depth map D:

[0029] D=M(I norm );

[0030] Where D is the depth map; M is the pre-trained MiDaS model;

[0031] Scale the depth map D back to the original image size H orgin *W orgin , the final depth map D final :

[0032] Among them, H orgin is the original image height; W orgin is the original image width;

[0033] Use color mapping to adjust the final depth map D final For visualization, map the depth values into the color space.

[0034] Preferably, in step S3, the normalized frame-by-frame image I is matched norm Each pixel in the final depth map D final The depth value information is used to calculate the distance of the detected target relative to the camera:

[0035] Distance=D final (x,y);

[0036] Distance is the distance information of the detection target to the monocular camera; x represents the horizontal coordinate of the pixel in the image; y represents the vertical coordinate of the pixel in the image.

[0037] Preferably, in step S4,

[0038] Calculate frame-by-frame image I t The spatial gradient and temporal gradient of the frame-by-frame image I t Use the convolution operation to calculate the horizontal and vertical gradients to get I x and I y ;

[0039]

[0040] Among them, I x Indicates that the convolution operation calculates the gradient in the horizontal direction; I y Indicates that the convolution operation calculates the gradient in the vertical direction;

[0041] Calculate adjacent frame-by-frame images I t and frame-by-frame images I t+1 Time gradient between

[0042]

[0043] in, Represents adjacent frame-by-frame images I t and frame-by-frame images I t+1 Time gradient between

[0044] A fixed-size window is selected as the neighborhood window for each pixel;

[0045] The brightness of the pixels in the selected neighborhood window is fitted by the least square method;

[0046] P(x,y)=a0+a1x+a2y+a3x 2 +a4y 2 +a5xy;

[0047] Among them, P(x,y) represents the brightness value of the pixel at position (x,y) in the image, that is, the grayscale intensity of the pixel; a0 is a constant term, which represents the average brightness of the pixels in the neighborhood; a1 is the linear variation coefficient in the x direction, which represents the change in brightness in the horizontal direction; a2 is the linear variation coefficient in the y direction, which represents the change in brightness in the vertical direction; a3 is the quadratic term coefficient in the x direction, which represents the curved change in brightness in the horizontal direction; a4 is the quadratic term coefficient in the y direction, which represents the curved change in brightness in the vertical direction; a5 is the mixed term coefficient, which represents the interaction effect of brightness in the horizontal and vertical directions.

[0048] Preferably, in step S4, an optical flow equation is constructed for each pixel according to the optical flow constraint:

[0049]

[0050] Among them, u represents the horizontal component of the optical flow, which usually represents the horizontal motion speed of the pixel in the image; v represents the vertical component of the optical flow, which usually represents the vertical motion speed of the pixel in the image;

[0051] Iteratively solve the motion vector for each pixel and update the estimated value using the least squares method until convergence;

[0052] Summarize the motion vector of each pixel into a velocity field;

[0053] V(x,y)=(μ(x,y),v(x,y));

[0054] Among them, V(x,y) represents the optical flow vector at the position (x,y) in the image, that is, the motion vector of the pixel; u(x,y) represents the horizontal optical flow component at the position (x,y) in the image; v(x,y) represents the vertical optical flow component at the position (x,y) in the image.

[0055] Therefore, the present invention adopts the above-mentioned method for real-time vehicle speed extraction from a roadside monocular camera based on the MF fusion algorithm to achieve accurate calculation of vehicle speed without relying on camera intrinsic parameters. The pixel depth of each frame image is predicted by the depth estimation algorithm, and the motion information of pixels in adjacent frames is captured in combination with the optical flow method to calculate the actual speed of the vehicle. This realizes accurate extraction of vehicle speed under nonlinear distortion conditions, significantly improving the real-time performance and reliability of roadside traffic flow monitoring.

[0056] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of the overall structure of a method for extracting real-time vehicle speed from a roadside monocular camera based on the MF fusion algorithm of the present invention;

[0058] Figure 2 The image depth map generated by the MiDaS depth estimation algorithm for the real-time vehicle speed extraction method of the roadside monocular camera based on the MF fusion algorithm of the present invention for the image frame;

[0059] Figure 3 This is a flowchart of calculating the optical flow vector using the Farneback optical flow method in a real-time vehicle speed extraction method for a roadside monocular camera based on the MF fusion algorithm of the present invention. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0061] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0062] The words "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0063] Example 1

[0064] like Figure 1-Figure 3 As shown, the present invention provides a method for extracting real-time vehicle speed from a roadside monocular camera based on an MF fusion algorithm, comprising the following steps:

[0065] S1: A monocular camera is installed on the roadside to cover the lane area to be monitored and collect road video images. The road video images collected by the monocular camera will be used for subsequent image analysis.

[0066] S2: performing frame processing on the road video image to form frame-by-frame images;

[0067] According to the frame rate of the road video image of the monocular camera (for example, 30 frames per second), the road video image data is decomposed into individual frame-by-frame images. For a 1-second road video image, 30 frame-by-frame images (such as Frame 1, Frame 2, Frame 3...Frame 30) will be obtained;

[0068] S3: Use the MiDaS model to perform a monocular depth estimation algorithm to predict the depth information of each frame and obtain the depth map of each frame.

[0069] S31: resize the frame-by-frame images to the size H*W required by the MiDaS model;

[0070] Each frame-by-frame image (e.g., Frame1) is resized to 384×384 pixels, which is the standard input size required by the model. The resized image is denoted as I prep,The standardized size allows the model to maintain high depth estimation accuracy at different resolutions.

[0071] Adjusted frame-by-frame image I prep It can be expressed as:

[0072] I prep =Resize(I,H,W);

[0073] Among them, I prep Represents the adjusted frame-by-frame image; Resize means adjusting the image to the specified height H and width W, keeping the number of channels I of the image unchanged; I: represents the number of channels of the input image, representing the three color channels of the RGB image; H: represents the height of the image; W: represents the width of the image;

[0074] S32: Analyzing the adjusted frame-by-frame image I prep Normalization is performed to normalize the pixel values from [0, 255] to [0, 1] and subtract the mean (according to the training data); the normalized frame-by-frame image I norm Expressed as:

[0075]

[0076] Among them, I norm Represents the normalized frame-by-frame image; μ is the mean used during training;

[0077] The normalized frame-by-frame image I norm Input the pre-trained MiDaS model M and use the convolutional neural network CNN to extract the normalized frame-by-frame image I norm The output of the MiDaS model M is a depth map D, which contains the relative depth information of each pixel in the image:

[0078] D=M(I norm );

[0079] Where D is the depth map; M is the pre-trained MiDaS model;

[0080] S34: Scale the depth map D back to the size H of the original image orgin *W orgin , so that the depth data can correspond to the pixels of the original image to the final depth map D final :

[0081] D final =Resize(D,H orgin ,W orgin );

[0082] Among them, H orginis the original image height; W orgin is the original image width;

[0083] S35: Use color mapping to adjust the final depth map D final For visualization, map the depth values into the color space.

[0084] S36: Matching normalized frame-by-frame images I norm Each pixel in the final depth map D final Depth value information, calculate the distance information of the object:

[0085] Distance=D final (x,y);

[0086] Distance is the distance information of the detection target to the monocular camera; x represents the horizontal coordinate of the pixel in the image; y represents the vertical coordinate of the pixel in the image.

[0087] S4: Acquire two adjacent frame-by-frame images I t and I t+1 , the Farneback optical flow method is used to calculate the optical flow between adjacent images, analyze the pixel changes between adjacent frame-by-frame images, and generate the optical flow vector of each pixel, thereby capturing the motion information of the object in the image;

[0088] S41: Calculate frame-by-frame image I t The spatial gradient and temporal gradient of the frame-by-frame image I t Use the convolution operation to calculate the horizontal and vertical gradients to get I x and I y :

[0089]

[0090] Among them, I x Indicates that the convolution operation calculates the gradient in the horizontal direction; I y Indicates that the convolution operation calculates the gradient in the vertical direction;

[0091] S42: Calculate adjacent frame-by-frame images I t and frame-by-frame images I t+1 Time gradient between A fixed-size window is selected as the neighborhood window for each pixel;

[0092]

[0093] in, Represents adjacent frame-by-frame images I t and frame-by-frame images I t+1 Time gradient between

[0094] S43: Perform brightness fitting on the pixels in the selected neighborhood window by the least squares method to obtain a quadratic polynomial model with corresponding coefficients a = (a0, a1, a2, a3, a4, a5) T ;

[0095] P(x,y)=a0+a1x+a2y+a3x 2 +a4y 2 +a5xy;

[0096] Among them, P(x,y) represents the brightness value of the pixel at position (x,y) in the image, that is, the grayscale intensity of the pixel; a0 is a constant term, which represents the average brightness of the pixels in the neighborhood; a1 is the linear variation coefficient in the x direction, which represents the change in brightness in the horizontal direction; a2 is the linear variation coefficient in the y direction, which represents the change in brightness in the vertical direction; a3 is the quadratic term coefficient in the x direction, which represents the curved change in brightness in the horizontal direction; a4 is the quadratic term coefficient in the y direction, which represents the curved change in brightness in the vertical direction; a5 is the mixed term coefficient, which represents the interactive effect of brightness in the horizontal and vertical directions.

[0097] S44: According to the optical flow constraint, construct the optical flow equation for each pixel:

[0098]

[0099] Among them, [u,v] T Represents the optical flow (or motion) velocity vector of each pixel; u represents the horizontal component of the optical flow, which usually represents the horizontal motion velocity of the pixel in the image; v represents the vertical component of the optical flow, which usually represents the vertical motion velocity of the pixel in the image;

[0100] S45: Iteratively solve the motion vector (u, v) for each pixel point and update the estimated value using the least squares method until convergence;

[0101] S46: Summarize the motion vector (u, v) of each pixel into a velocity field V(x, y), which represents the motion information of each point in the image.

[0102] V(x,y)=(μ(x,y),v(x,y));

[0103] Among them, V(x,y) represents the optical flow vector at the position (x,y) in the image, that is, the motion vector of the pixel; u(x,y) represents the horizontal optical flow component at the position (x,y) in the image; v(x,y) represents the vertical optical flow component at the position (x,y) in the image.

[0104] S5: By combining the optical flow vector with the depth map, the pixel speed is calculated to obtain the vehicle running speed;

[0105] S51: The parallax displacement is obtained by the magnitude of the optical flow vector.

[0106] The parallax displacement d(x,y) of the pixel point (x,y) is expressed as the modulus of the optical flow vector:

[0107]

[0108] u(x,y) represents the horizontal optical flow component at the position (x,y) in the image; v(x,y) represents the vertical optical flow component at the position (x,y) in the image;

[0109] S52: Assuming the frame rate of the camera is f (i.e., frames per second), the time interval between adjacent frames is If the frame rate is 30FPS, the time interval between adjacent frames is 0.03s;

[0110] S53: Calculate the actual displacement Δs(x,y) by multiplying the parallax displacement of the pixel by the depth value:

[0111] Δs(x,y)=d(x,y)·D t (x,y);

[0112] Among them, D t (x,y) represents the depth change of the pixel point (x,y), that is, the change in depth or distance of the position over time;

[0113] S54: The velocity v(x,y) of each pixel is obtained by dividing the actual displacement Δs(x,y) by the time interval Δt:

[0114]

[0115] Therefore, the present invention adopts the above-mentioned method for real-time vehicle speed extraction from a roadside monocular camera based on the MF fusion algorithm to achieve accurate calculation of vehicle speed without relying on camera intrinsic parameters. The pixel depth of each frame image is predicted by the depth estimation algorithm, and the motion information of pixels in adjacent frames is captured in combination with the optical flow method to calculate the actual speed of the vehicle. This realizes accurate extraction of vehicle speed under nonlinear distortion conditions, significantly improving the real-time performance and reliability of roadside traffic flow monitoring.

[0116] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can still be modified or replaced by equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for extracting real-time vehicle speed from a roadside monocular camera based on MF fusion algorithm, characterized in that: The following steps are involved: S1: Install a monocular camera on the roadside to collect road video images; S2: performing frame processing on the road video image to form frame-by-frame images; S3: Use the MiDaS model to perform a monocular depth estimation algorithm to predict the depth information of each frame and obtain the depth map of each frame. S4: Acquire two adjacent frame-by-frame images I t and I t+1 ,Use Farneback optical flow method to calculate the optical flow between adjacent images, analyze the pixel changes between adjacent frame-by-frame images, and generate the optical flow vector of each pixel; S5: By combining the optical flow vector with the depth map, the vehicle speed is obtained by calculating the pixel speed; In step S5, the parallax displacement is obtained by the magnitude of the optical flow vector; the parallax displacement of the pixel point is expressed as the modulus of the optical flow vector: u(x,y) represents the horizontal optical flow component at the position (x,y) in the image; v(x,y) represents the vertical optical flow component at the position (x,y) in the image; In step S5, the frame rate of the camera is f, and the time interval between adjacent frames is The actual displacement Δs(x,y) is calculated by multiplying the parallax displacement of the pixel by the depth value: Δs(x,y)=d(x,y)·D t (x,y); Among them, D t (x,y) represents the depth change of pixel (x,y); The speed of each pixel is obtained by dividing the actual displacement by the time interval:

2. The method for extracting real-time vehicle speed from a roadside monocular camera based on the MF fusion algorithm according to claim 1, characterized in that: In step S3, the frame-by-frame image is adjusted to the size H*W required by the MiDaS model; the adjusted frame-by-frame image I prep Expressed as: I prep =Resize(I,H,W); Among them, I prep Represents the adjusted frame-by-frame image; Resize means adjusting the image to the specified height H and width W, keeping the number of channels I of the image unchanged; I: represents the number of channels of the input image, representing the three color channels of the RGB image; H: represents the height of the image; W: represents the width of the image; For the adjusted frame-by-frame image I prep Perform normalization processing, and the normalized frame-by-frame image I norm Expressed as: Among them, I norm represents the normalized frame-by-frame image; μ is the mean used during training; The normalized frame-by-frame image I norm Input the pre-trained MiDaS model M and use the convolutional neural network CNN to extract the normalized frame-by-frame image I norm Extract features and predict the depth information of each pixel. The output of the MiDaS model M is the depth map D: D=M(I norm ); Where D is the depth map; M is the pre-trained MiDaS model; Scale the depth map D back to the original image size H orgin *W orgin , get the final depth map D final ; Among them, H orgin is the original image height; W orgin is the original image width; Use color mapping to adjust the final depth map D final For visualization, map the depth values into the color space.

3. The method for extracting real-time vehicle speed from a roadside monocular camera based on the MF fusion algorithm according to claim 2 is characterized in that: In step S3, the normalized frame-by-frame image I is matched norm Each pixel in the final depth map D final The depth value information is used to calculate the distance of the detected target relative to the camera: Distance=D final (x,y); Distance is the distance information of the detection target to the monocular camera; x represents the horizontal coordinate of the pixel in the image; y represents the vertical coordinate of the pixel in the image.

4. The method for extracting real-time vehicle speed from a roadside monocular camera based on the MF fusion algorithm according to claim 1, characterized in that: In step S4, the frame-by-frame image I is calculated t The spatial gradient and temporal gradient of the frame-by-frame image I t Use the convolution operation to calculate the gradient in the horizontal and vertical directions to get I x and I y : Among them, I x Indicates that the convolution operation calculates the gradient in the horizontal direction; I y Indicates that the convolution operation calculates the gradient in the vertical direction; in, Represents adjacent frame-by-frame images I t and frame-by-frame images I t+1 Time gradient between A fixed-size window is selected as the neighborhood window for each pixel; The brightness of the pixels in the selected neighborhood window is fitted by the least square method; P(x,y)=a0+a1x+a2y+a3x 2 +a4y 2 +a5xy; Among them, P(x,y) represents the brightness value of the pixel at position (x,y) in the image, that is, the grayscale intensity of the pixel; a0 is a constant term, which represents the average brightness of the pixels in the neighborhood; a1 is the linear variation coefficient in the x direction, which represents the change in brightness in the horizontal direction; a2 is the linear variation coefficient in the y direction, which represents the change in brightness in the vertical direction; a3 is the quadratic term coefficient in the x direction, which represents the curved change in brightness in the horizontal direction; a4 is the quadratic term coefficient in the y direction, which represents the curved change in brightness in the vertical direction; a5 is the mixed term coefficient, which represents the interactive effect of brightness in the horizontal and vertical directions.

5. The method for extracting real-time vehicle speed from a roadside monocular camera based on the MF fusion algorithm according to claim 4 is characterized in that: In step S4, the optical flow equation for each pixel is constructed according to the optical flow constraint: Among them, u represents the horizontal component of the optical flow, which usually represents the horizontal motion speed of the pixel in the image; v represents the vertical component of the optical flow, which usually represents the vertical motion speed of the pixel in the image; Iteratively solve the motion vector for each pixel and update the estimated value using the least squares method until convergence; Summarize the motion vector of each pixel into a velocity field; V(x,y)=(μ(x,y),v(x,y)); Among them, V(x,y) represents the optical flow vector at the position (x,y) in the image, that is, the motion vector of the pixel; u(x,y) represents the horizontal optical flow component at the position (x,y) in the image; v(x,y) represents the vertical optical flow component at the position (x,y) in the image.

Images