Vehicle speed recognition system based on YOLO neural network

Abstract

The invention discloses a vehicle speed recognition system based on a YOLO neural network, and the system comprises a video processing and camera calibration module which is used for converting a video into an image set, and judging whether to calibrate a traffic camera or not according to the percentage and a calibration threshold value. The vehicle identification and tracking module is used for identifying the vehicle set in each image based on the improved yolov8 neural network model and obtaining an identification result, tracking the vehicle set in the video based on the identification result and obtaining a tracking result, and obtaining pixel displacement data of the vehicle according to the tracking result; the depth map generation module is used for generating a depth map for the image based on a Pixelform model, and the vehicle speed calculation module is used for fitting a scaling factor based on the depth map, calculating the actual speed of the vehicle between adjacent images based on pixel displacement data and the scaling factor, and calculating the traffic flow speeds in different directions based on the actual speed of the vehicle and a speed measurement period. According to the invention, the vehicle positioning precision and the vehicle speed estimation precision of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle speed recognition, and in particular to a vehicle speed recognition system based on a YOLO neural network. Background Art

[0002] The current monocular camera vehicle speed measurement method mainly relies on prior camera information and actual road data to complete the vehicle speed measurement task. In fact, due to the large number of traffic cameras, almost all of them have different angles, and the difficulty in obtaining actual road data, the vehicle speed measurement task is difficult to carry out. When using the YOLO model to estimate vehicle speed, there is a large delay, low computational efficiency, and low estimation accuracy. Summary of the invention

[0003] The present invention provides a vehicle speed recognition system based on YOLO neural network to overcome the above technical problems.

[0004] A vehicle speed recognition system based on YOLO neural network includes a video processing and camera calibration module, a vehicle recognition and tracking module, a depth map generation module and a vehicle speed calculation module.

[0005] The video processing and camera calibration module is used to convert the video recorded by the traffic camera into an image set in chronological order, calculate the percentage of pixel points in the image set whose pixel values ​​of adjacent images change to zero and the calibration threshold, and determine whether to calibrate the traffic camera according to the percentage and the calibration threshold. If the traffic camera is calibrated, re-record the video according to the calibrated traffic camera and convert it into a new image set. Otherwise, do not re-acquire a new image set.

[0006] The vehicle recognition and tracking module is used to build and improve the yolov8 neural network model, recognize the vehicle set in each image based on the improved yolov8 neural network model and obtain the recognition result, track the vehicle set in the video based on the recognition result and obtain the tracking result, the tracking result includes the bounding box and center coordinates of each vehicle, and obtain the pixel displacement data of the vehicle according to the tracking result.

[0007] The depth map generation module is used to filter a part of the image set from the image set based on a selection strategy, wherein the selection strategy is to obtain the first ten frames of images after the start of recording from the image set, and to generate a depth map estimating relative displacement for each image in the part of the image set based on a Pixelformer model.

[0008] The vehicle speed calculation module is used to fit the scaling factor based on the depth map, calculate the actual speed of the vehicle between adjacent images based on the vehicle's pixel displacement data and the scaling factor, and calculate the vehicle flow speed in different directions based on the vehicle's actual speed and speed measurement period.

[0009] Preferably, the calculating the percentage of pixels in the image set whose pixel values ​​of adjacent images change to zero and the calibration threshold comprises:

[0010] S11, calculate the percentage of the number of pixels with zero pixel change between the i-th frame image and the i-1-th frame image to the total number of pixels, expressed as q i ,

[0011] S12, calculate the q of each frame of the first n frames of the image set i , q i Add ascending order to an array L, n represents the length of the array, L = {q 0 ,q 1 ,…,q n},

[0012] S13, calculate the calibration threshold X according to the formula (1),

[0013]

[0014] where θ is a fixed offset.

[0015] Preferably, the building and improving the yolov8 neural network model includes building the yolov8 neural network model, setting the AIFI layer after the 9th layer of the backbone part, the AIFI layer processes the features output by the 9th layer through 8 attention heads, and setting a RepViT module after the terminal feature layer P5, and the RepViT module is used to extract global features from the features output by P5.

[0016] Preferably, tracking the vehicle set in the video based on the recognition result and obtaining the tracking result includes:

[0017] S21. Extract the bounding boxes of all detected vehicles from the recognition results, calculate the center coordinates of each bounding box, represent the bounding boxes of all vehicles and their center coordinates, the number of times each vehicle's bounding box appears in the time-continuous image, and the width and height information of the vehicle's bounding box as TrackingBox objects, and add the TrackingBox objects to the list L. tracking_boxes_cur_frame Inside,

[0018] S22. Create dictionary D tracking_objects , D tracking_objects It is used to store the target TrackingObject being tracked. The TrackingObject contains the position information of the vehicle in the previous frame. The position information in the previous frame includes the bounding boxes of several vehicles and the center coordinates of the bounding boxes of the vehicles.

[0019] S23, traverse D tracking_objects For all the tracking objects that already exist in L, for each tracking object, tracking_boxes_cur_frame Find the tracking frame closest to the tracked object from L tracking_boxes_cur_frame Finding the closest tracking box to the tracked object includes calculating the Euclidean distance δ between the tracked object and the vehicle's bounding box, setting the threshold D max , if δ <D max , indicating that it is the same vehicle, update the location information of the tracking object, and tracking_boxes_cur_frame Remove the bounding box of the corresponding vehicle from D. Otherwise, remove the tracked object from D. tracking_objects Delete

[0020] Get L tracking_boxes_cur_frame Assign new IDs to unmatched tracking frames in D, create new tracking objects, and add them to D tracking_objects dictionary,

[0021] S24: Repeat S21-S23 until all tracking objects in the video are added to Dtracking_objects.

[0022] Preferably, the scaling factor based on the depth map fitting comprises:

[0023] S31, based on D tracking_objects Obtain the pixel displacement data of each vehicle, and at the same time obtain the actual displacement of each vehicle between adjacent frames based on the manually annotated groundtruth data in the dataset.

[0024] S32, let D M (p,p'):=||p-p'|| 2 , D M (p, p') represents the estimated Euclidean distance between two image points p and p' based on the depth map M. The estimated Euclidean distance between the two image points is calculated by the pixel displacement data. The real distance between the two image points is obtained based on the actual displacement of each vehicle between adjacent frames. The real distance is represented by D(p, p'), D(p, p') = s m D M (p,p'), fitting s based on the real-world point distance M corresponding to the sequence of image point pairs m , according to formula (2), the scaling factor is obtained:

[0025]

[0026] in, represents the sequence of image point pairs, Indicates that there is no scale factor The estimated distance vector between point pair sequences, p i represents the i-th image point in the previous frame in the adjacent frame, p i ' represents the i-th image point in the next frame in the adjacent frame, s m represents the scale factor, Represents the scaling factor after fitting.

[0027] The present invention provides a vehicle speed recognition system based on the YOLO neural network, which improves the YOLO neural network model, reduces the number of network layers, reduces the entire network delay, avoids useless calculation of low-level features, improves the detection accuracy and calculation efficiency, and ultimately increases the overall accuracy of the improved YOLO neural network model, making the vehicle position information more accurate. By setting a vehicle recognition and tracking module, reliable support is provided for speed estimation, thereby improving the accuracy of the entire vehicle speed recognition system for speed estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 It is a system structure diagram of the present invention;

[0030] Figure 2 It is a diagram of the improved yolov8 network structure in an embodiment of the present invention;

[0031] Figure 3 is a structural diagram of the RepVit module in an embodiment of the present invention;

[0032] Figure 4 is a target detection and tracking diagram in an embodiment of the present invention;

[0033] Figure 5 is a depth map generated after the depth map model Pixelformer in the embodiment of the present invention processes the random frame;

[0034] Figure 6 4 is a processing diagram of the vehicle speed recognition system in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Figure 1 It is the system structure diagram of the present invention, such as Figure 1 As shown, the method of this embodiment includes a video processing and camera calibration module, a vehicle identification and tracking module, a depth map generation module and a vehicle speed calculation module.

[0037] The video processing and camera calibration module is used to convert the video recorded by the traffic camera into an image set in chronological order, calculate the percentage of pixel points in the image set whose pixel values ​​of adjacent images change to zero and the calibration threshold, and determine whether to calibrate the traffic camera according to the percentage and the calibration threshold. If the traffic camera is calibrated, re-record the video according to the calibrated traffic camera and convert it into a new image set. Otherwise, do not re-acquire a new image set.

[0038] The vehicle recognition and tracking module is used to build and improve the yolov8 neural network model, recognize the vehicle set in each image based on the improved yolov8 neural network model and obtain the recognition result, track the vehicle set in the video based on the recognition result and obtain the tracking result, the tracking result includes the bounding box and center coordinates of each vehicle, and obtain the pixel displacement data of the vehicle according to the tracking result.

[0039] The depth map generation module is used to filter a part of the image set from the image set based on a selection strategy, wherein the selection strategy is to obtain the first ten frames of images after the start of recording from the image set, and to generate a depth map estimating relative displacement for each image in the part of the image set based on a Pixelformer model.

[0040] The vehicle speed calculation module is used to fit the scaling factor based on the depth map, calculate the actual speed of the vehicle between adjacent images based on the vehicle's pixel displacement data and the scaling factor, and calculate the vehicle flow speed in different directions based on the vehicle's actual speed and speed measurement period.

[0041] Based on the above scheme, the present invention provides a vehicle speed recognition system based on the YOLO neural network, which improves the yolov8 neural network model. By setting the AIFI layer and the RepViT module, the number of network layers is reduced, the entire network delay is reduced, the useless calculation of low-level features is avoided, and the detection accuracy and calculation efficiency are improved. Finally, the overall accuracy of the improved yolov8 neural network model is increased, and the vehicle position information is more accurate. By setting the vehicle identification and tracking module, reliable support is provided for speed estimation, thereby improving the accuracy of the speed estimation of the entire vehicle speed recognition system.

[0042] Specifically, this embodiment provides the details of the vehicle speed recognition system, including a video processing and camera calibration module, a vehicle recognition and tracking module, a depth map generation module, and a vehicle speed calculation module.

[0043] The video processing and camera calibration module is used to convert the video recorded by the traffic camera into an image set in chronological order, calculate the percentage of pixel points in the image set whose pixel values ​​of adjacent images change to zero and the calibration threshold, and determine whether to calibrate the traffic camera based on the percentage and the calibration threshold. If the traffic camera is calibrated, the video is re-recorded based on the calibrated traffic camera and converted into a new image set. Otherwise, a new image set is not re-acquired.

[0044] Specifically, given a video O recorded by a traffic camera, the O stream is converted into an HLS or RTP stream, and then downsampled to a resolution as high as FHD, while limiting the frame rate to less than 30FPS, and the processed video is O'. Higher FPS is of limited help to the present invention, so a maximum limit is set. The present invention proposes a video speed estimation method, which requires the video to be decomposed into frames for processing. Therefore, the images not specifically mentioned in the concept of this embodiment are all taken as examples of the i-th frame of the video by default. Calculate the percentage of pixel value changes of adjacent images in the image set to zero and the calibration threshold, and determine whether to calibrate the traffic camera based on the percentage and the calibration threshold. The purpose of the calibrated traffic camera is to ensure that its viewing angle is always consistent, and calculate the corresponding unique scaling factor under the viewing angle.

[0045] The calculating percentage of pixel value changes of adjacent images in the image set to zero and the calibration threshold comprises:

[0046] S11, calculate the percentage of the number of pixels with zero pixel change between the i-th frame image and the i-1-th frame image to the total number of pixels, expressed as p i ,

[0047] S12, calculate the q of each frame of the first n frames of the image set i , q iAdd ascending order to an array L, n represents the length of the array, L = {q 0 ,q 1 ,…,q n}, specifically, in this embodiment, the value of n is 100. When the number of frames is less than 100, the threshold is initialized in the constructor, and an initial value is set (set to 0.4 in the subsequent implementation verification process), and it will not be dynamically calculated based on the frame data.

[0048] S13, calculate the calibration threshold X according to the formula (1),

[0049]

[0050] where θ is a fixed offset.

[0051] Specifically, since all subsequent quantities related to the calculation speed need to be performed at the same camera angle, in order to ensure that the camera angle remains consistent, it is necessary to continuously detect the degree of picture change. A scrolling list containing 100 frames of change values ​​is used to ensure robustness when there is sensor noise, air pollution, or insects flying over the picture. For each frame, the video processing and camera calibration module determines whether the percentage of pixel values ​​that change to zero is less than one-quarter of the threshold. First, the module calculates the percentage of pixels with a pixel change of 0 between the current frame and the previous frame to the total number of pixels. When the list length is greater than 100, the threshold X will be updated to the first quartile in the array L, added to the settable fixed offset θ. When When , it is considered that the camera viewing angle has changed significantly and the automatic calibration function is triggered. When the camera angle of view changes within a reasonable range, the subsequent modules are carried out normally.

[0052] The vehicle identification and tracking module is used to construct and improve the yolov8 neural network model, identify the vehicle set in each image based on the improved yolov8 neural network model and obtain the identification result.

[0053] The construction and improvement of the yolov8 neural network model includes constructing the yolov8 neural network model, setting the AIFI layer after the 9th layer of the backbone part, the AIFI layer processes the features output by the 9th layer through 8 attention heads, and setting a RepViT module after the terminal feature layer P5, the RepViT module is used to extract global features from the features output by P5. The RepVit module is a special visual Transformer that reduces the number of network layers and the overall network latency through reparameterization. The method of adding a layer of convolution after AIFI can also improve the accuracy of detection, ultimately increasing the overall accuracy of the network model and making the vehicle's location information more accurate.

[0054] Based on the recognition results, the vehicle set in the video is tracked and the tracking results are obtained, wherein the tracking results include the bounding box and center coordinates of each vehicle.

[0055] Tracking the vehicle set in the video based on the recognition result and obtaining the tracking result includes:

[0056] S21. Extract the bounding boxes of all detected vehicles from the recognition results, calculate the center coordinates of each vehicle's bounding box, represent all vehicle bounding boxes and their center coordinates, the number of frames in which they appear, and the width and height of the vehicle's bounding boxes as TrackingBox objects, and add the TrackingBox objects to the list L. tracking_boxes_cur_frame Inside,

[0057] S22. Create dictionary D tracking_objects , D tracking_objects It is used to store the target TrackingObject being tracked. The TrackingObject contains the position information of the vehicle in the previous frame. The position information in the previous frame includes the bounding boxes of several vehicles and the center coordinates of the bounding boxes of the vehicles.

[0058] S23, traverse D tracking_objects For all the tracking objects that already exist in L, for each tracking object, tracking_boxes_cur_frame Find the tracking frame closest to the tracked object from L tracking_boxes_cur_frame Finding the closest tracking box to the tracked object includes calculating the Euclidean distance δ between the tracked object and the vehicle's bounding box, setting the threshold D max , if δ <D max , indicating that it is the same vehicle, update the location information of the tracking object, and tracking_boxes_cur_frame Remove the bounding box of the corresponding vehicle from D. Otherwise, remove the tracked object from D. tracking_objects Delete

[0059] Get Ltracking_boxes_cur_frame Assign new IDs to unmatched tracking frames in D, create new tracking objects, and add them to D tracking_objects dictionary,

[0060] S24: Repeat S21-S23 until all the tracking objects in the video are added to D tracking_objects middle.

[0061] Obtain the pixel displacement data of the vehicle based on the tracking results.

[0062] Specifically, the vehicle identification and tracking module uses the improved yolov8 neural network to perform vehicle identification and tracking functions, in order to more accurately locate the vehicle position and make the detection box size more consistent with the real box. The AIFI (Attentional Feature Integration) structure is introduced on the backbone of the yolov8 network. AIFI is in the backbone part, the AIFI layer is located at the 10th layer, and the AIFI layer receives the output of the 9th layer as input. 8 attention heads are used. This layer is a computationally intensive operation, but it is only performed once on high-level features, balancing performance and computational efficiency. The RepViT module is introduced at the terminal feature layer P5 of the YOLOv8 model. Its role is to enhance the expressive power of these high-level features and provide more powerful feature support for the final target detection task. It helps to improve the detection performance of the YOLOv8 model in large objects and complex scenes.

[0063] The network structure of yolov8 is as follows Figure 2 The following is a detailed description of each layer structure and its functions:

[0064] 1.P1-P5 feature pyramid:

[0065] Used to extract features of different scales to help detect objects of different sizes. P1 to P5 represent different feature levels, usually from shallow to deep features.

[0066] 2. AIFI module: The main functions include feature integration, global integration of spatial dimension features, introduction of 2D position information, helping the model understand the spatial relationship of features, and using the self-attention mechanism to capture long-distance dependencies between features. AIFI is in the backbone part, the AIFI layer is located in the 10th layer, and the AIFI layer receives the output of the 9th layer as input. 8 attention heads are used. This layer is a computationally intensive operation, but it is only performed once on high-level features, balancing performance and computational efficiency. First, extract the dimension information of the input tensor (number of channels C, height H, width W). Generate 2D sine cosine position encoding. Flatten the 4D input tensor [B batch size, C, H, W] into a 3D tensor [B, H×W, C]. Call the forward method of the parent class to perform self-attention and feedforward network calculations. Reshape the result into the original 4D shape [B, C, H, W]. This module mainly completes multi-scale feature fusion. AIFI can be applied to feature maps of different scales to enhance the feature representation of each scale. At the same time, the features processed by AIFI can better transmit information between different scales and improve the semantic richness of the features.

[0067] 3.C (concat): Mainly used for feature fusion, such as concatenating the upsampled feature map with the Backbone P4 feature map to fuse feature information of different scales.

[0068] 4.U (unsampling) represents the upsampling layer, which is used to enlarge the feature map so that it can be merged with the features of higher layers.

[0069] 5.C2F module: Implement CSP Bottleneck faster through 2 convolutions, combining convolution operations and feature focusing to enhance the capture of important features.

[0070] 6.RepVitblock: The RepViT module is introduced at the terminal feature layer P5 of the YOLOv8 model. Its role is to enhance the expressiveness of these high-level features and provide more powerful feature support for the final target detection task. It helps to improve the detection performance of the YOLOv8 model in large objects and complex scenes. The first is the 3x3 DW (Depthwise Convolution) layer, a special convolution operation that applies the convolution kernel independently to each input channel instead of applying the same convolution kernel to all channels. This operation can reduce computational complexity. This is followed by the SE (Squeeze-and-Excitation) layer, which is an attention mechanism that can adaptively adjust the weights of different channels to enhance the expressiveness of the network. Next are two 1x1 Convolution layers, which perform deep convolution operations, which are used to change the number of feature channels.

[0071] 7.Detect layer: Outputs detection results, including bounding box (BBox) and category prediction (CLS).

[0072] 8.LOSS module: contains bounding box regression loss and classification loss, which are used for model training. The BBOX part handles the regression of the bounding box, and the CLS part handles the classification of the object.

[0073] Figure 3 This is the structure diagram of the RepVit module and the structure diagram after reparameterization.

[0074] 1.DW (Depthwise convolution) is a deep convolution that is used to reduce the number of parameters and calculations while retaining feature expression capabilities.

[0075] 2. SE (Squeeze-and-Excitation) is an attention mechanism used to adaptively adjust the weights of feature maps and enhance important features.

[0076] 3. PW (Pointwise) channel dimension feature fusion: 1x1 PW convolution can perform feature fusion on the channel dimension of the input feature map. It can also learn cross-channel correlations and extract richer and more advanced feature representations. Improve computational efficiency, 1x1 PW convolution has a small amount of computation. When the network depth is large, using 1x1 PW convolution can greatly reduce computational complexity and parameter quantity.

[0077] After the RepVit module is re-parameterized, the computational complexity of a 1x1 DW layer convolution can be saved, the parameters and computational complexity of the module can be reduced, and computational efficiency can be improved. The expressiveness of these high-level features can be enhanced, providing more powerful feature support for the final target detection task. This helps improve the detection performance of the YOLOv8 model in large objects and complex scenes.

[0078] The images in the training set are fed into the neural network, and the output is set to return only the vehicle category, and the detection results are returned, including the category ID, the predicted confidence score, and the bounding box coordinates. The bounding boxes of all detected vehicles are extracted from the object detection results, and their center coordinates are calculated. Figure 4 This is a picture of the target detection and tracking of the yolov8 customized network. The output result is set to detect only the type of vehicle in the picture and assign a unique ID to the vehicle for subsequent tracking. The bounding boxes of these vehicles and their center coordinates, the number of frames that appear continuously, and the width and height of the bounding boxes of the vehicles are represented as TrackingBox objects. These TrackingBox objects are added to a list L tracking_boxes_cur_frame Inside, L tracking_boxes_cur_frame Used to store the bounding boxes of all vehicles detected in the current frame.

[0079] By creating a dictionary D tracking_objects To complete the vehicle tracking function, the dictionary stores the target TrackingObject being tracked, and the TrackingObject contains the location information of the vehicle in the previous frame - including the bounding boxes of several vehicles and the center coordinates of the bounding boxes of the vehicles. This dictionary will be dynamically updated as the target detection and tracking proceed.

[0080] Traverse D tracking_objects For all tracked objects that already existed in the previous frame, find L tracking_boxes_cur_frame The tracking box that is closest to the tracked object in the previous frame. Calculate the Euclidean distance δ between them. Only when the distance is less than the preset threshold D max , it is considered a matching box.

[0081] If we find the delta of an object <D max , it is considered to be the same vehicle and the location information of the tracking object is updated. tracking_boxes_cur_frame Remove the box from the

[0082] If a previously tracked object cannot find a matching tracking box, remove it from D tracking_objects , and it is considered that the vehicle has left the screen.

[0083] If L tracking_boxes_cur_frame Assign new IDs to unmatched tracking frames in D, create new tracking objects, and add them to D tracking_objects Dictionary. Use the track_id variable to record the newly assigned ID. The final updated D tracking_objects It contains complete information about each tracked object in each frame of the entire video. The complete trajectory information can be obtained through the coordinate position when it first appears and the coordinate position when it finally appears.

[0084] If the video contains metadata, the FPS of the video can be obtained directly. If the video does not contain metadata, it is assumed that the time between frames is consistent, and the frame rate of each frame within a certain number of frames is calculated, and then the average is taken to obtain the frame number estimate.

[0085] The depth map generation module is used to filter a portion of the image set from the image set based on a selection strategy, and generate a depth map for estimating relative displacement for each image in the portion of the image set based on a Pixelformer model. Figure 5It is the depth map generated by the depth map model Pixelformer processing random frames in the video. Specifically, the Pixelformer depth map estimation model is used to generate the depth map M. The depth map model has limited inference speed. It is unnecessary and time-consuming to generate a depth map for every frame of the entire video, so only the depth maps of a certain number of frames are calculated. The selection method is to generate ten depth maps for the first ten frames under a certain viewing angle. When there are more than ten frames, the average value of the ten depth maps is used.

[0086] The vehicle speed calculation module is used to fit the scaling factor based on the depth map, calculate the actual speed of the vehicle between adjacent images based on the pixel displacement data and the scaling factor, and calculate the speed of traffic in different directions based on the actual speed of the vehicle and the speed measurement period. The scaling factor is used to convert the relative displacement in the picture into the real displacement of the vehicle, and each camera angle will produce a different scaling factor.

[0087] Specifically, in D tracking_objects The complete displacement of a vehicle is obtained from the image. Based on the detected vehicle bounding box, the displacement of each vehicle between adjacent frames (in pixels) is calculated. At the same time, the actual displacement of each vehicle between adjacent frames (in meters) is obtained based on the manually annotated groundtruth data in the dataset. Then, the scaling factor that converts pixel displacement into actual displacement is fitted through linear regression. This scaling factor is the so-called "scaling factor".

[0088] The depth map-based fitting scaling factor includes:

[0089] S31, based on D tracking_objects Obtain the pixel displacement data of each vehicle. At the same time, according to the manually annotated groundtruth data in the data set, set the average length of the vehicle to 4.8 meters, and obtain the actual displacement of each vehicle between adjacent frames.

[0090] S32. Use a simple pinhole camera model to convert points in image i and their associated depth estimates to real-world points. For a given camera parameter f (representing focal length), s x 、s y represents the aspect ratio scaling in the x and y directions, and u 0 , v 0 (denotes the base point), let p≡[u,v,1] denote the image point in Cartesian coordinates in the camera coordinate system, let p≡(r u,v ,θ u,v ,φ u,v ), represents the same point in spherical coordinates (still in the camera reference frame). Then, given the depth map M of the image I, so that M I (u,v) is the estimated depth of point p in meters, and the corresponding real-world point P in spherical coordinates (still in the camera coordinate system) can be obtained as follows:

[0091] P=(M I (u,p),θ u,v +Π,φ u,v +Π) (2)

[0092] Usually d(u,v)=s M M I (u,v), where d(u,v) is the true depth of point p, and s M is a constant independent of image i (set to 1 in the present invention).

[0093] Let D M (p,p'):=||p-p'|| 2 , D M (p, p') represents the estimated Euclidean distance between two image points p and p' based on the depth map M. The estimated Euclidean distance between the two image points is calculated by the pixel displacement data. The real distance between the two image points is obtained based on the actual displacement of each vehicle between adjacent frames. The real distance is represented by D(p, p'), D(p, p') = s M D M (p,p'), estimates s based on the real-world point distance M corresponding to the image point pair sequence m , according to formula (2), the scaling factor is obtained:

[0094]

[0095] in, represents the sequence of image point pairs, Indicates that there is no scale factor The estimated distance vector between point pair sequences, p i represents the i-th image point in the previous frame in the adjacent frame, p i ' represents the i-th image point in the next frame in the adjacent frame, Represents the scaling factor after fitting.

[0096] The center of mass of each vehicle's bounding box is tracked through the video and the center of mass line is derived. Then, the intersection points of each vehicle's ground-truth bounding box and this center of mass line are calculated. These intersection points serve as the input to the above least squares estimator. In addition, each tracked car is assigned a velocity direction based on the change in its y coordinate.

[0097] Generally, there are two directions of traffic flow in a traffic camera: the direction of movement toward the camera and the direction of movement away from the camera. In order to calculate the average speed in a certain direction, all vehicles in that direction are stored in the rolling average of the last γ = 60 seconds. Let V = (v 1 ,v 2 ,…,v n ) is the average speed sequence of all vehicles. 1 ,t 2 ,…,t n ) is the corresponding timestamp sequence of the last time the car was seen in the video, and it is calculated that:

[0098]

[0099]

[0100] This calculation can be done at any time t when the output is needed. The parameter γ can be configured according to the specific use case. Finally, V t , v*(t) represents the speed of traffic in a certain direction, v i represents the average speed of the i-th vehicle, N represents the number of vehicles, and t i represents the time when the i-th vehicle appears, and d is a constant. Specifically, for example, if the FPS of the video is 30, then the time difference between each frame is 1 / 30 second. If the vehicle moves 300 pixels between two frames, the scaling factor is 0.01 m / pixel, and the time difference between two frames is 1 / 30 second, then the actual distance is 3 meters and the speed is 90 m / s.

[0101] Based on the above content, this embodiment provides a processing process of a vehicle speed recognition system based on a YOLO neural network, which mainly includes seven parts, such as Figure 6 As shown, the first part is the data preprocessing part, the second part monitors whether the camera changes and automatically calibrates, the third part recognizes and tracks the vehicle in the picture, the fourth part estimates the FPS based on the input, the fifth part uses the depth map model to generate the depth map, the sixth part estimates the scaling factor, and the seventh part determines the vehicle direction and calculates the average vehicle speed.

[0102] Overall beneficial effects:

[0103] The present invention provides a vehicle speed recognition system based on the YOLO neural network, which improves the YOLO neural network model. By setting an AIFI layer and a RepViT module, the number of network layers is reduced, the entire network delay is reduced, the useless calculation of low-level features is avoided, the detection accuracy and calculation efficiency are improved, and finally the overall accuracy of the improved YOLO neural network model is increased, the vehicle position information is made more accurate, and reliable support is provided for speed estimation by setting a vehicle identification and tracking module, thereby improving the accuracy of the entire vehicle speed recognition system for speed estimation.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle speed recognition system based on YOLO neural network, characterized in that: It includes video processing and camera calibration module, vehicle recognition and tracking module, depth map generation module and vehicle speed calculation module. The video processing and camera calibration module is used to convert the video recorded by the traffic camera into an image set in chronological order, calculate the percentage of pixel points in the image set whose pixel values ​​of adjacent images change to zero and the calibration threshold, and determine whether to calibrate the traffic camera according to the percentage and the calibration threshold. If the traffic camera is calibrated, re-record the video according to the calibrated traffic camera and convert it into a new image set. Otherwise, do not re-acquire a new image set. The vehicle recognition and tracking module is used to build and improve the yolov8 neural network model, recognize the vehicle set in each image based on the improved yolov8 neural network model and obtain the recognition result, track the vehicle set in the video based on the recognition result and obtain the tracking result, the tracking result includes the bounding box and center coordinates of each vehicle, and obtain the pixel displacement data of the vehicle according to the tracking result. The depth map generation module is used to filter a part of the image set from the image set based on a selection strategy, wherein the selection strategy is to obtain the first ten frames of images after the start of recording from the image set, and to generate a depth map estimating relative displacement for each image in the part of the image set based on a Pixelformer model. The vehicle speed calculation module is used to fit the scaling factor based on the depth map, calculate the actual speed of the vehicle between adjacent images based on the vehicle's pixel displacement data and the scaling factor, and calculate the vehicle flow speed in different directions based on the vehicle's actual speed and speed measurement period.

2. A vehicle speed recognition system based on YOLO neural network according to claim 1, characterized in that: The calculation of the percentage of pixels in the adjacent images in the image set whose pixel value changes are zero and the calibration threshold comprises: S11, calculate the percentage of the number of pixels with zero pixel change between the i-th frame image and the i-1-th frame image to the total number of pixels, expressed as q i , S12, calculate the q of each frame of the first n frames of the image set i , q i Arrange in ascending order and add them to an array L, where n represents the length of the array, L = {q0, q1, ..., q n }, S13, calculate the calibration threshold X according to the formula (1), where θ is a fixed offset.

3. A vehicle speed recognition system based on YOLO neural network according to claim 1, characterized in that: The yolov8 neural network model is built and improved, including building the yolov8 neural network model, setting the 9th layer after the backbone part as the AIFI layer, the AIFI layer processes the features output by the 9th layer through 8 attention heads, and setting a RepViT module after the terminal feature layer P5, the RepViT module is used to extract global features from the features output by P5.

4. A vehicle speed recognition system based on YOLO neural network according to claim 1, characterized in that: Tracking the vehicle set in the video based on the recognition result and obtaining the tracking result includes: S21. Extract the bounding boxes of all detected vehicles from the recognition results, calculate the center coordinates of each bounding box, represent the bounding boxes of all vehicles and their center coordinates, the number of times each vehicle's bounding box appears in the time-continuous image, and the width and height information of the vehicle's bounding box as TrackingBox objects, and add the TrackingBox objects to the list L. tracking_boxes_cur_frame Inside, S22. Create dictionary D tracking_objects , D tracking_objects It is used to store the target TrackingObject being tracked. The TrackingObject contains the position information of the vehicle in the previous frame. The position information in the previous frame includes the bounding boxes of several vehicles and the center coordinates of the bounding boxes of the vehicles. S23, traverse D tracking_objects For all the tracking objects that already exist in L, for each tracking object, tracking_boxes_cur_frame Find the tracking frame closest to the tracked object from L tracking_boxes_cur_frame Finding the closest tracking box to the tracked object includes calculating the Euclidean distance δ between the tracked object and the vehicle's bounding box, setting the threshold D max , if δ <D max , indicating that it is the same vehicle, update the location information of the tracking object, and tracking_boxes_cur_frame Remove the bounding box of the corresponding vehicle from D. Otherwise, remove the tracked object from D. tracking_objects Delete Get L tracking_boxes_cur_frame Assign new IDs to unmatched tracking frames in D, create new tracking objects, and add them to D tracking_objects dictionary, S24: Repeat S21-S23 until all the tracking objects in the video are added to D tracking_objects middle.

5. A vehicle speed recognition system based on YOLO neural network according to claim 4, characterized in that: The depth map-based fitting scaling factor includes: S31, based on D tracking_objects Obtain the pixel displacement data of each vehicle, and at the same time obtain the actual displacement of each vehicle between adjacent frames based on the manually annotated groundtruth data in the dataset. S32, let D M (p,p'):=||p-p'||2, D M (p, p') represents the estimated Euclidean distance between two image points p and p' based on the depth map M. The estimated Euclidean distance between the two image points is calculated by the pixel displacement data. The real distance between the two image points is obtained based on the actual displacement of each vehicle between adjacent frames. The real distance is represented by D(p, p'), D(p, p') = s m D M (p,p'), fitting s based on the real-world point distance M corresponding to the sequence of image point pairs m , according to formula (2), the scaling factor is obtained: in, represents the sequence of image point pairs, Indicates that there is no scale factor The estimated distance vector between point pair sequences, p i represents the i-th image point in the previous frame in the adjacent frame, p i ′ represents the i-th image point in the next frame in the adjacent frame, s m represents the scale factor, Represents the scaling factor after fitting.

Images