A Real-time Traffic Load Intelligent Sensing Method Based on Deep Learning

Abstract

The present invention discloses a real-time traffic load intelligent perception method based on deep learning. First, by constructing an object detection network and a fine-grained classification network based on deep learning, the basic vehicle information in video surveillance is obtained. Then, load information matching is carried out to obtain the total vehicle weight and axle weight information from the dynamic weighing system, and match the basic vehicle information in video surveillance with the vehicle information in the dynamic weighing system. If a vehicle passes through the dynamic weighing device, the total vehicle weight, axle weight, wheelbase, and number of axles are obtained from the dynamic weighing system, the information is synchronized to the database, and associated with the unique vehicle identifier. If the vehicle does not pass through the dynamic weighing device, the vehicle weight information is obtained from the database according to the unique vehicle identifier. Finally, the whole process dynamic tracking of the vehicle trajectories in the video surveillance area is realized to obtain the spatio-temporal distribution of the vehicle loads on the bridge deck, monitor the load conditions of the entire bridge in real time, and conduct the whole-process tracking of the passing vehicles on the bridge.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep learning and computer vision, and particularly relates to a real-time traffic load intelligent perception method based on deep learning. Background Art

[0002] In existing traffic load perception applications, dynamic weighing devices and supporting bayonet cameras are generally used. The dynamic weighing system is a technology that uses sensors to measure the axle weight and total weight of a vehicle during driving. The bayonet camera can take pictures of vehicles and realize functions such as speed measurement and vehicle type identification according to the sensors configured thereon.

[0003] However, the dynamic weighing system can only identify vehicles at a single measurement point and cannot track and analyze the entire driving trajectory of vehicles. Moreover, the data transmission and processing of the dynamic weighing system require a certain time delay, cannot provide real-time feedback on the load information of vehicles, and cannot give early warnings of overloaded roads in a timely manner, thus reducing the efficiency and effectiveness of the dynamic weighing system in traffic management and road maintenance. Summary of the Invention

[0004] The problem to be solved by the present invention is: to provide a real-time traffic load intelligent perception method based on deep learning, which identifies, tracks, and matches the passing vehicle information by combining bridge deck video monitoring and dynamic weighing information, obtains the spatio-temporal distribution characteristics of vehicle loads, and objectively and accurately reflects traffic load characteristics.

[0005] The present invention adopts the following technical solutions: A real-time traffic load intelligent perception method based on deep learning, comprising the following steps:

[0006] S1. Obtain vehicle basic information: Through constructing a target detection network and a fine-grained classification network based on deep learning, perform intelligent identification on the bridge deck video monitoring to obtain the vehicle basic information in the video monitoring;

[0007] S2. Perform load information matching: Based on the vehicle basic information obtained in step S1, obtain the vehicle total weight and axle weight information from the dynamic weighing system, and match the vehicle basic information in the video monitoring with the vehicle information in the dynamic weighing system;

[0008] Judge whether the vehicle has passed through the dynamic weighing device: If the vehicle has passed through the dynamic weighing device, obtain the vehicle total weight, axle weight, axle distance, and number of axles from the dynamic weighing system, synchronize the information to the database, and associate it with the unique vehicle identifier; If the vehicle has not passed through the dynamic weighing device, obtain the vehicle weight information from the database according to the unique vehicle identifier;

[0009] S3. Perform vehicle trajectory tracking: Achieve full-process dynamic tracking of the trajectories of vehicles within the video surveillance area, including single-camera multi-target vehicle tracking and cross-camera vehicle re-identification, obtain the spatio-temporal distribution of vehicle loads on the bridge deck, and conduct full-process tracking of the vehicles passing on the bridge.

[0010] Specifically, the vehicle basic information includes: vehicle type, axles, loading condition, lane where the vehicle is located, and vehicle position coordinates. Obtaining the vehicle basic information includes the following sub-steps:

[0011] S1.1. Obtain a frame of image from the bridge traffic surveillance video.

[0012] S1.2. Use a vehicle target detection algorithm based on the YOLOv5 target detection network to identify vehicles in the image and obtain the vehicle type information and image coordinate information of the vehicles. The vehicle type information includes: small cars, trucks, medium and large buses, and non-motor vehicles.

[0013] S1.3. Construct a fine-grained classification network to perform fine-grained classification on each vehicle in the identified image and obtain the number of axles and loading condition information of each truck.

[0014] S1.4. Based on LaneNet, perform pixel-level segmentation on the lane lines, extract the lane line equations, and conduct automatic lane division and numbering.

[0015] S1.5. Use the lane line position information and the image coordinate information obtained from vehicle target detection to identify the lane where each vehicle is located.

[0016] S1.6. Map the position of the vehicle on the image to a known-length reference object, which includes road markings and guardrails, and estimate the relative position of the vehicle with respect to the surveillance camera.

[0017] S1.7. According to the vehicle relative position and lane information, combined with the longitude and latitude information, lane width information, and lane orientation information where the surveillance camera is located, calculate the offset distance and direction angle of the vehicle with respect to the camera, and obtain the absolute position of the vehicle in the world coordinate system.

[0018] Specifically, in step S3, performing vehicle trajectory tracking includes the following sub-steps:

[0019] S3.1. Implement multi-target dynamic tracking of vehicles based on the DeepSort algorithm.

[0020] S3.2. Generate hash code features of vehicles based on a deep hash network, and use the hash code to retrieve the vehicle in the historical vehicle hash code database to perform matching of vehicles between different surveillance cameras.

[0021] S3.3. If a vehicle is retrieved from the historical vehicle hash code database, it means the vehicle has appeared within the surveillance range of other monitoring cameras on the bridge. Obtain the unique identifier of the vehicle and update the hash code feature of the vehicle in the database.

[0022] S3.4. If a vehicle is not retrieved from the historical vehicle hash code database, it means the vehicle appears on the bridge for the first time. Generate a new unique identifier for the vehicle and store the unique identifier and the hash code feature of the vehicle obtained in step S3.2 in the historical vehicle hash code database.

[0023] Further, in step S3.1, multi-object dynamic tracking of vehicles is implemented based on the DeepSort algorithm, and the method is as follows:

[0024] S3.1.1. Obtain the position, size, and category of each vehicle in the image from step S1.2 to obtain the vehicle target detection results; use the Kalman filter to predict the current position according to the historical positions of each vehicle to obtain the tracker prediction results.

[0025] S3.1.2. Match the tracker prediction results with the detection results, including: feature matching, cascade matching, and IOU matching methods. The matching results are respectively successful matching, unmatched tracking results, and unmatched detection results.

[0026] In feature matching, introduce the motion information and appearance information of the target vehicle, extract the position offset of the target vehicle, calculate the Mahalanobis distance between the prediction result and the detection result, use the deep neural network to calculate the features of the tracking target and the detection target, calculate the cosine distance between the prediction result and the detection result. For the matching results with large Mahalanobis distance and cosine distance, it is determined as unmatched.

[0027] In cascade matching, introduce the priority information. The priority of the tracker with consecutive successful matches is higher, and the more times of matching failure, the lower the priority. Combine the priority information and use the Hungarian algorithm for matching.

[0028] In IOU matching, introduce the IOU information between the tracking target box and the detection target box. For the matching results with IOU less than the preset threshold, it is determined as unmatched.

[0029] S3.1.3. According to the matching results, update the tracker status of each target vehicle.

[0030] The tracker statuses are respectively: uncertain state, determined state, and deleted state. The initial state of the tracker is the uncertain state. Update the tracker status of each target vehicle, and the method is as follows:

[0031] If the number of successful matches is greater than the threshold, the tracker transitions from an uncertain state to a confirmed state; if a tracker that was previously in a confirmed state fails to match the tracking result with the detection result more than the threshold number of times, the tracker transitions from the confirmed state to a deleted state;

[0032] For successful matching cases, update the information of the tracker. If the number of successful matches is greater than the threshold, set the state of the tracker to the confirmed state;

[0033] For cases where the tracking result does not match, record the number of failed matches. If the number of failed matches is greater than the threshold, set the tracker to the deleted state;

[0034] For cases where the detection result does not match, create a new tracker and initialize it to the uncertain state.

[0035] S3.1.4. Update the vehicle feature set. For the target vehicle with a successful match, update the vehicle features calculated by the deep neural network and store the tracking result.

[0036] Furthermore, in step S3.1.2, match the predicted result of the tracker with the detection result,

[0037] Furthermore, in step S3.2, generate the hash code feature of the vehicle based on the deep hash network and retrieve the vehicle using the hash code in the historical vehicle hash code database. The method is as follows:

[0038] S3.2.1. Crop the vehicle image from a frame of the surveillance video and use the deep hash network to generate the hash code feature of the vehicle. The number of bits of the hash code is 256;

[0039] S3.2.2. Retrieve the vehicle using the hash code in the historical vehicle hash code database. The retrieval method is to calculate the Hamming distance between the hash code feature of the vehicle and the hash code features stored in the historical vehicle hash code database, and obtain the 10 vehicles corresponding to the hash codes with the smallest and less than the distance threshold;

[0040] S3.2.3. If 10 hash codes cannot be retrieved, the vehicle is not retrieved. If at least 10 hash codes can be retrieved, use the voting algorithm to obtain the vehicle retrieval result.

[0041] The technical solution of the present invention also provides: an electronic device, including:

[0042] One or more processors;

[0043] A storage device on which one or more programs are stored;

[0044] When the one or more programs are executed by the one or more processors, the one or more processors implement the real-time traffic load intelligent perception method based on deep learning described above.

[0045] The technical solution of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps in any one of the above real-time traffic load intelligent perception methods based on deep learning are implemented.

[0046] Compared with the prior art, the present invention adopts the above technical solutions and has the following technical effects:

[0047] 1. In the real-time traffic load intelligent perception method of the present invention, a target detection algorithm based on yolov5 is used to locate the position of the vehicle in the image, and combined with objects with known distances such as lanes and guardrails on the bridge, lane recognition and distance estimation are carried out, and the position of the vehicle in the image is converted into the position of the vehicle in the world coordinate system.

[0048] 2. In the real-time traffic load intelligent perception method of the present invention, a deep hash network is used to generate the hash code feature of the vehicle, and the hash code is used to retrieve the vehicle in the database to realize the matching of vehicles between different monitoring cameras. Combined with vehicle target tracking, the whole process tracking of the passing vehicles on the bridge is realized.

[0049] 3. In the real-time traffic load intelligent perception method of the present invention, the matching of vehicle information and load information is realized. Based on the vehicle information recognition and the whole process tracking of the vehicle, information such as the vehicle weight, axle weight, and number of axles of the vehicle can be further obtained, and the load condition of the whole bridge can be monitored in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flowchart of the real-time traffic load intelligent perception method of the present invention;

[0051] Figure 2 is a visualization effect diagram of the vehicle target detection result in the embodiment of the present invention;

[0052] Figure 3 is a visualization effect diagram of the lane line segmentation result in the embodiment of the present invention;

[0053] Figure 4 is a flowchart of the multi-target dynamic tracking of vehicles based on the DeepSort algorithm of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] To make the objectives, technical solutions, and advantages of the present invention clearer, the following further elaborates on the technical solutions of the application in conjunction with the accompanying drawings. The described embodiments are only a part of the embodiments involved in the present invention. All non-innovative embodiments made by other researchers in the field based on this embodiment fall within the protection scope of the present invention. At the same time, for the step numbers in the embodiments of the present invention, they are only set for the convenience of elaboration and explanation, and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0055] In an embodiment of the present invention, through a real-time traffic load intelligent perception method based on deep learning, the spatio-temporal distribution information of the real-time load of a bridge is obtained. By combining bridge deck video monitoring and dynamic weighing information, the passing vehicle information is identified, tracked, and matched to obtain the spatio-temporal distribution characteristics of the vehicle load.

[0056] In this embodiment, the laboratory conditions for testing are as follows: 1 NVIDIA Quadro RTX 6000 GPU with 24G video memory, and a total of 20,000 bridge road monitoring image data.

[0057] The real-time traffic load intelligent perception method of this embodiment is as Figure 1 shown, and includes steps: vehicle basic information acquisition, load information distribution, and vehicle trajectory tracking, specifically as follows:

[0058] S1. Vehicle basic information acquisition:

[0059] By constructing a target detection network and a fine-grained classification network based on deep learning, the monitoring video frames are intelligently identified to obtain the basic information of the vehicles in the picture.

[0060] The vehicle basic information includes but is not limited to: vehicle type, axles, loading situation, lane where the vehicle is located, and vehicle position coordinates.

[0061] The acquisition of vehicle basic information includes the following sub-steps:

[0062] Step S1.1. Obtain a frame of image from the bridge traffic monitoring video;

[0063] Step S1.2. Use a vehicle target detection algorithm based on the YOLOv5 target detection network to identify the vehicles in the image, and obtain the vehicle type and image coordinate information of the vehicles. The vehicle types are divided into 4 categories: small cars, trucks, medium and large buses, and non-motor vehicles;

[0064] Step S1.3. Construct a fine-grained classification network to perform fine-grained classification on each truck identified in step S1.2.

[0065] According to the vehicle target detection results, crop partial images of the regions where each truck is located, and use a fine-grained classification convolutional neural network to classify the number of axles and the loading situation of the trucks, so as to obtain the information on the number of axles and the loading situation of each truck; the number of axles is divided into two axles, three axles, four axles, five axles, six axles, and others, and the loading situation is divided into full load and no load.

[0066] The final output of the vehicle detection result is Info_detect, which is expressed as follows:

[0067] ;

[0068] Among them, box = [x1, y1, x2, y2], (x1, y1) and (x2, y2) respectively represent the upper left vertex coordinates and the lower right vertex coordinates of the vehicle detection frame, model represents the specific vehicle type, axes represents the number of axles of the vehicle, and cargo represents the loading situation.

[0069] In this embodiment, the visualization effect of the vehicle target detection result is as Figure 2 shown.

[0070] Step S1.4: Perform pixel-level segmentation on the lane lines based on LaneNet and extract the lane line equations.

[0071] The specific method is: use a lane line image segmentation convolutional neural network to segment the set of pixel points on the image of the lane line region, and then reduce the segmentation region of each lane to a curve with a width of 1, which is called a lane curve.

[0072] In this embodiment, the visualization effect of the lane line segmentation result is as Figure 3 shown.

[0073] Step S1.5: Use the lane line position information and the image coordinate information obtained by vehicle target detection to identify the lanes where each vehicle is located;

[0074] According to the coordinate information of each vehicle obtained by vehicle target detection, draw a ray along the x-axis of the image to the left from the position of the vehicle, and calculate the number of intersection points between this ray and each lane curve. The lane number where the vehicle is located is equal to the number of intersection points.

[0075] Step S1.6: Map the position of the vehicle on the image to these objects with known lengths such as road markings and guardrails, and estimate the relative position of the vehicle with respect to the monitoring camera.

[0076] Step S1.7: According to the relative position of the vehicle and the lane information, combined with the longitude and latitude information of the camera location, the lane width information and the lane orientation information, calculate the offset distance and the direction angle of the vehicle with respect to the camera, and obtain the position of the vehicle in the world coordinate system.

[0077] S2. Load information matching:

[0078] It aims to match the bridge deck video monitoring and dynamic weighing information. Based on the vehicle basic information obtained in step S1, and the information such as the total vehicle weight and axle weight obtained from the dynamic weighing system, it matches the video monitoring vehicle information with the dynamic weighing vehicle information. The specific steps are as follows:

[0079] Step S2.1. Determine whether the vehicle has passed through the dynamic weighing device according to the vehicle position information and the position information of the dynamic weighing device;

[0080] Step S2.2. If the vehicle has passed through the dynamic weighing device, obtain the total vehicle weight, axle weight, wheelbase, number of axles, etc., synchronize the information to the database, and associate it with the vehicle unique identifier;

[0081] Step S2.3. If the vehicle has not passed through the dynamic weighing device, obtain the vehicle weight information from the database according to the vehicle unique identifier;

[0082] S3. Vehicle trajectory tracking:

[0083] It aims to achieve the full-process dynamic tracking of the vehicle trajectories in the video monitoring area, including single-camera multi-target vehicle tracking and cross-camera vehicle re-identification, and then obtain the real-time and accurate spatio-temporal distribution of the bridge deck vehicle loads. The specific steps are as follows:

[0084] Step S3.1. Implement multi-target dynamic vehicle tracking based on the DeepSort algorithm. Obtain the vehicle target detection results from the position, size, and category of each vehicle in the image obtained in step S1; use the Kalman filter to predict the current position according to the historical positions of each bounding box, and obtain the tracker prediction results.

[0085] Implement multi-target dynamic vehicle tracking based on the DeepSort algorithm. The method is as follows:

[0086] S3.1.1. Obtain the vehicle target detection results from the position, size, and category of each vehicle in the image obtained in step S1.2:

[0087] ;

[0088] Among them, represents the i-th detected vehicle, represents the center coordinates of the vehicle in the image, represents the width and height of the vehicle, represents the category of the vehicle;

[0089] Use the Kalman filter to predict the current position according to the historical positions of each vehicle, and obtain the tracker prediction results:

[0090] ;

[0091] Among them, represents the predicted position of the i-th vehicle at time t, and the function is the Kalman filter prediction function, represents the position of the i-th vehicle at time ;

[0092] S3.1.2. Match the prediction result of the tracker with the detection result, which is expressed as:

[0093] ;

[0094] Among them, D represents the set of detection results, P represents the set of prediction results, is a matching pair, representing the matching result of the i-th vehicle, and the matching result includes: matching successfully, the tracking result not being matched, and the detection result not being matched;

[0095] Match the prediction result of the tracker with the detection result, and use feature matching, cascade matching, and IOU matching methods respectively.

[0096] In feature matching, the motion information and appearance information of the target are introduced, the position offset of the target is extracted, and the Mahalanobis distance between the prediction result and the detection result is calculated:

[0097] ;

[0098] Among them, x represents the detection result, represents the mean of the prediction result, represents the covariance matrix of the prediction result,

[0099] Use a deep neural network to calculate the features of the tracking target and the detection target, and calculate the cosine distance between the prediction result and the detection result:

[0100] ;

[0101] Among them, A and B represent the feature vectors of the tracking target and the detection target respectively;

[0102] Use a deep neural network to calculate the features of the tracking target and the detection target, and for the matching results with large Mahalanobis distance and cosine distance, they are determined as unmatched;

[0103] In cascade matching, priority information is introduced. The priority of the tracker with consecutive successful matches is higher, and the more times of matching failure, the lower the priority. The Hungarian algorithm is used for matching in combination with the priority information.

[0104] The IOU (Intersection over Union) information between the tracked target box and the detected target box is introduced in the IOU matching:

[0105] ;

[0106] Among them, A' and B' represent the tracked target box and the detected target box respectively.

[0107] There are three types of final matching results, namely successful matching, unmatched tracking result, and unmatched detection result. For the matching result with an IOU less than the preset threshold, it is determined as unmatched.

[0108] S3.1.3. Update the tracker status of each target vehicle according to the matching result:

[0109] ;

[0110] Among them, represents the tracker status of the i-th vehicle at time t, and the function represents the status update function.

[0111] There are three types of tracker statuses, namely undetermined state, determined state, and deleted state.

[0112] The initial status of the tracker is the undetermined state; if the number of successful matches is greater than the threshold, the tracker changes from the undetermined state to the determined state; if the number of times the tracking result fails to match the detection result for a tracker that was previously in the determined state is greater than the threshold, the tracker changes from the determined state to the deleted state.

[0113] For the case of successful matching, update the information of the tracker. If the number of successful matches is greater than the threshold, set the tracker status to the determined state.

[0114] For the case of unmatched tracking result, record the number of failed matches. If the number of failed matches is greater than the threshold, set the tracker to the deleted state.

[0115] For the case of unmatched detection result, create a new tracker and initialize it to the undetermined state.

[0116] S3.1.4. Update the vehicle feature set. For the target vehicle with successful matching, update the vehicle features calculated by the deep neural network and store the tracking result:

[0117] ;

[0118] Among them, represents the feature set of the i-th vehicle at time t, the function represents the deep neural network calculation function, represents the detection result of the i-th vehicle.

[0119] Finally, store the tracking results.

[0120] In this embodiment, the multi-object dynamic tracking process of vehicles based on the DeepSort algorithm is as Figure 4 shown.

[0121] Step S3.2: Generate the hash code features of the vehicle based on the deep hash network, and use the hash code to retrieve the vehicle in the historical vehicle hash code database. The specific method is as follows:

[0122] S3.2.1: Crop the vehicle image from a frame of the bridge traffic monitoring video, and use the deep hash network to generate the hash code features of the vehicle:

[0123] ;

[0124] Among them, the number of bits of the hash code is 256, I represents the input vehicle image, H represents the generated hash code features, and θ represents the weights of the deep hash network, .

[0125] S3.2.2: Use the hash code to retrieve the vehicle in the historical vehicle hash code database. The retrieval method is to calculate the Hamming distance between the hash code features of the vehicle and the hash code features stored in the historical vehicle hash code database:

[0126] ;

[0127] Among them, represent two hash code features respectively, D represents the Hamming distance, represents the exclusive OR operation, represent the hash code features of vehicle i and the hash code features of vehicle i stored in the historical vehicle hash code database respectively;

[0128] Obtain the vehicles corresponding to the 10 hash codes with the smallest and less than the distance threshold.

[0129] S3.2.3: If 10 hash codes cannot be retrieved, it is regarded as the vehicle not being retrieved. If at least 10 hash codes can be retrieved, use the voting algorithm to obtain the vehicle retrieval result.

[0130] Step S3.3: If the vehicle is retrieved from the historical vehicle hash code database, it means that the vehicle has appeared within the monitoring range of other monitoring cameras on the bridge surface. Then obtain the unique identifier of the vehicle and update the hash code features of the vehicle in the database.

[0131] Step S3.4: If the vehicle is not retrieved from the database, it indicates that the vehicle appears on the bridge for the first time. Then, a new unique identifier for the vehicle is generated, and the unique identifier and the hash code feature of the vehicle obtained in step S10 are stored in the historical vehicle hash code database.

[0132] In an embodiment of the present invention, an electronic device is further provided, including: one or more processors; a storage device storing one or more programs thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the real-time traffic load intelligent perception method described in any of the above embodiments.

[0133] In an embodiment of the present invention, a computer-readable storage medium is further provided, having a computer program stored thereon, and when the program is executed by a processor, the steps in any of the real-time traffic load intelligent perception methods in the above embodiments are implemented.

[0134] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A real-time traffic load intelligent perception method based on deep learning, characterized in that: The steps include: S1. Obtain basic vehicle information: By building a deep learning-based target detection network and a fine-grained classification network, intelligent recognition is performed on bridge deck video surveillance to obtain basic vehicle information in video surveillance; S2. Matching load information: Based on the basic vehicle information obtained in step S1, the gross vehicle weight and axle weight information are obtained from the dynamic weighing system, and the basic vehicle information in the video surveillance is matched with the vehicle information in the dynamic weighing system; Determine whether the vehicle has passed through a dynamic weighing device: If the vehicle has passed through a dynamic weighing device, the vehicle's gross weight, axle weight, wheelbase, and number of axles are obtained from the dynamic weighing system, the information is synchronized to the database, and associated with the vehicle's unique identifier; if the vehicle has not passed through a dynamic weighing device, the vehicle weight information is obtained from the database based on the vehicle's unique identifier; S3, vehicle trajectory tracking: the trajectory of the vehicle in the video surveillance area is dynamically tracked throughout the entire process, including single-camera multi-target vehicle tracking and cross-camera vehicle re-identification, the spatiotemporal distribution of bridge deck vehicle load is obtained, and the full-process tracking of vehicles passing on the bridge is performed, including the following sub-steps: S3.

1. Realize dynamic tracking of multiple targets based on DeepSort algorithm; S3.

2. Generate a hash code feature of the vehicle based on the deep hash network, and use the hash code to retrieve the vehicle in the historical vehicle hash code database to match the vehicle between different surveillance cameras. The method is as follows: S3.2.

1. Crop the vehicle image from a frame of the surveillance video and use the deep hash network to generate the vehicle’s hash code features: ； Among them, the number of bits of the hash code is 256. I represents the input vehicle image, H Represents the generated hash code feature, f stands for Deep hashing network function, θ represents the weight of the deep hashing network, ; S3.2.

2. Use the hash code to retrieve the vehicle in the historical vehicle hash code database. The retrieval method is: calculate the Hamming distance between the hash code feature of the vehicle and the hash code feature stored in the historical vehicle hash code database: ； in, Represent two hash code features respectively, D represents the Hamming distance, represents the exclusive OR operation, Respectively represent vehicles The hash code features and the vehicles stored in the historical vehicle hash code database Hash code feature, obtain the vehicles corresponding to the 10 smallest hash codes that are smaller than the distance threshold; S3.2.

3. If 10 hash codes cannot be retrieved, the vehicle is not retrieved. If at least 10 hash codes can be retrieved, a voting algorithm is used to obtain the vehicle retrieval result. S3.

3. If a vehicle is retrieved from the historical vehicle hash code database, then the vehicle has appeared within the surveillance range of other surveillance cameras on the bridge deck, obtain the unique identification of the vehicle, and update the hash code feature of the vehicle in the database; S3.

4. If the vehicle is not retrieved in the historical vehicle hash code database, the vehicle appears on the bridge deck for the first time, a new vehicle unique identifier is generated, and the unique identifier and the hash code feature of the vehicle obtained in step S3.2 are stored in the historical vehicle hash code database.

2. The real-time traffic load intelligent perception method based on deep learning according to claim 1 is characterized in that: The basic vehicle information includes: vehicle model, axle, cargo load, lane where the vehicle is located, and vehicle position coordinates; obtaining the basic vehicle information includes the following sub-steps: S1.1, obtaining a frame of image in the bridge traffic monitoring video; S1.

2. Use the vehicle target detection algorithm based on the YOLOv5 target detection network to identify the vehicle in the image and obtain the vehicle model information and image coordinate information; the vehicle model information includes: small cars, trucks, medium and large buses, and non-motor vehicles; S1.3, build a fine-grained classification network, perform fine-grained classification on each vehicle in the identified image, and obtain the number of axles and cargo information of each truck; S1.

4. Perform pixel-level segmentation of lane lines based on LaneNet, extract lane line equations, and automatically divide and number lanes; S1.5, using the lane line position information and the image coordinate information obtained by vehicle target detection, identifying the lane where each vehicle is located; S1.6, mapping the position of the vehicle on the image to a reference object of known length, the reference object including road markings and guardrails, estimating the relative position of the vehicle with respect to the surveillance camera; S1.

7. Based on the relative position of the vehicle and the lane information, combined with the latitude and longitude information of the surveillance camera, the lane width information and the lane orientation information, the offset distance and direction angle of the vehicle relative to the camera are calculated to obtain the absolute position of the vehicle in the world coordinate system.

3. The real-time traffic load intelligent perception method based on deep learning according to claim 2 is characterized in that: In step S1.3, each vehicle in the identified image is classified into fine-grained categories as follows: According to the YOLOv5 target detection results, part of the image of the area where each vehicle is located is cropped, and the number of axles and cargo load of the vehicle are classified using a fine-grained classification convolutional neural network to obtain the number of axles and cargo load information of each vehicle; Output the vehicle detection result Info_detect, which is as follows: ； Among them, box=[x1, y1, x2, y2], (x1,y1), (x2,y2) represent the coordinates of the upper left vertex and the lower right vertex of the vehicle detection box respectively, model represents the specific model, axes represents the number of axles of the vehicle, and cargo represents the cargo situation.

4. The real-time traffic load intelligent perception method based on deep learning according to claim 2 is characterized in that: In step S3.1, the vehicle multi-target dynamic tracking is realized based on the DeepSort algorithm, and the method is as follows: S3.1.

1. Obtain the position, size and category of each vehicle in the image from step S1.2, and obtain the vehicle target detection result: ； in, Indicates detected vehicles, represents the center coordinates of the vehicle in the image, Indicates the width and height of the vehicle, Indicates the type of vehicle; Based on the historical positions of each vehicle, the Kalman filter is used to predict the current position and obtain the tracker prediction result: ； in, Indicates Vehicles at time The predicted position of is the Kalman filter prediction function, Indicates Vehicles at time location; S3.1.2, Matching tracker prediction results and detection results, expressed as: ； in, Represents the set of detection results, Represents the set of prediction results, is a matching pair, indicating The matching results of vehicles, i =1,2,..., n ; The matching results include: successful matching, unmatched tracking results, and unmatched detection results; S3.1.

3. Update the tracker status of each target vehicle based on the matching results: ； in, Indicates Vehicles at time The tracker state of Represents the state update function; S3.1.

4. Update the vehicle feature set. For the successfully matched target vehicle, update the vehicle features calculated by the deep neural network and store the tracking results: ； in, Indicates Vehicles at time The feature set, function represents the deep neural network calculation function, Indicates The detection results of the vehicle.

5. The real-time traffic load intelligent perception method based on deep learning according to claim 4 is characterized in that: In step S3.1.2, matching the tracker prediction result with the detection result includes: feature matching, cascade matching and IOU matching methods; In feature matching, the target vehicle motion information and appearance information are introduced, the target vehicle position offset is extracted, and the Mahalanobis distance between the prediction result and the detection result is calculated: ； Among them, x represents the test result, represents the mean of the prediction results, Represents the covariance matrix of the prediction results, with superscript T Represents transposition and calculates the cosine distance between the prediction result and the detection result: ； Among them, A and B represent the feature vectors of the tracking target and the detection target respectively; Use a deep neural network to calculate the features of the tracked and detected targets, and determine that the matching results with large Mahalanobis distance and cosine distance are mismatched; In cascade matching, priority information is introduced. The priority of the tracker that has been matched successfully continuously is higher. The more matching failures, the lower the priority. Combined with the priority information, the Hungarian algorithm is used for matching. In IOU matching, the IOU information between the tracking target box and the detection target box is introduced: ； in, Represent the tracking target frame and the detection target frame respectively. and It is used to calculate the intersection area and union area of ​​the target box. For matching results whose IOU is less than the preset threshold, it is judged as a mismatch.

6. The real-time traffic load intelligent perception method based on deep learning according to claim 5 is characterized in that: In step S3.1.3, the tracker states are: uncertain state, determined state, and deleted state. The initial state of the tracker is uncertain state. The tracker state of each target vehicle is updated as follows: If the number of successful matches is greater than the threshold, the tracker changes from the uncertain state to the certain state; if the number of failed matches between the tracking result and the detection result of the tracker that was previously in the certain state is greater than the threshold, the tracker changes from the certain state to the deleted state; For successful matches, update the tracker information. If the number of successful matches is greater than the threshold, set the tracker state to a confirmed state. If the tracking result does not match, the number of matching failures is recorded. If the number of matching failures is greater than the threshold, the tracker is set to the deletion state; If the detection results do not match, a new tracker is created and initialized to an uncertain state.

7. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the real-time traffic load intelligent perception method based on deep learning as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps in the real-time traffic load intelligent perception method based on deep learning described in any one of claims 1 to 6 are implemented.

Images