Real-time traffic load intelligent sensing method based on deep learning
Through deep learning technology combining video surveillance and dynamic weighing information, real-time tracking and matching of vehicle loads is achieved, and the problem that existing traffic load perception systems cannot be tracked and warned in real time is solved, and the efficiency of traffic management is improved.
Patent Information
- Application Number
- CN202411556409.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-11-04
AI Technical Summary
The existing traffic load sensing system cannot track the vehicle's full driving trajectory and analyze the load characteristics in real time, resulting in low data transmission delay and overload warning efficiency.
Using a real-time traffic load intelligent perception method based on deep learning, the vehicle information is identified and load data is matched through joint bridge deck video surveillance and dynamic weighing information, and the vehicle trajectory tracking is tracked using YOLOv5 target detection and DeepSort algorithm to generate hash code features Perform cross-camera matching to achieve spatial and temporal distribution characteristics of vehicle loads.
Real-time monitoring and full-process tracking of vehicle load information is realized, the efficiency of traffic management and road maintenance is improved, and the road overloading can be timely warned of.
Smart Images

Figure CN119068398A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning and computer vision technology, and in particular to a real-time traffic load intelligent perception method based on deep learning. Background Art
[0002] In existing traffic load sensing applications, dynamic weighing equipment and supporting bayonet cameras are commonly used. The dynamic weighing system is a technology that uses sensors to measure the axle weight and total weight of a vehicle while it is traveling. The bayonet camera can take pictures of the vehicle and realize functions such as speed measurement and vehicle model recognition based on the sensors configured on it.
[0003] However, the dynamic weighing system can only identify vehicles at a single measurement point and cannot track and analyze the vehicle's entire driving trajectory. Moreover, the data transmission and processing of the dynamic weighing system requires a certain time delay, and cannot provide real-time feedback on vehicle load information or timely warn of road overload conditions, thus reducing the efficiency and effectiveness of the dynamic weighing system in traffic management and road maintenance. Summary of the invention
[0004] The problem to be solved by the present invention is to provide a real-time traffic load intelligent perception method based on deep learning, which identifies, tracks and matches the passing vehicle information by combining bridge deck video monitoring and dynamic weighing information, obtains the spatiotemporal distribution characteristics of vehicle loads, and objectively and accurately reflects the traffic load characteristics.
[0005] The present invention adopts the following technical solution: a real-time traffic load intelligent perception method based on deep learning, comprising the following steps: S1. Obtain basic vehicle information: By building a deep learning-based target detection network and a fine-grained classification network, intelligent recognition is performed on bridge deck video surveillance to obtain basic vehicle information in video surveillance; S2. Matching load information: Based on the basic vehicle information obtained in step S1, the gross vehicle weight and axle weight information obtained from the dynamic weighing system are matched with the basic vehicle information in the video surveillance and the vehicle information in the dynamic weighing system; Determine whether the vehicle has passed through a dynamic weighing device: If the vehicle has passed through a dynamic weighing device, the vehicle's gross weight, axle weight, wheelbase, and number of axles are obtained from the dynamic weighing system, the information is synchronized to the database, and associated with the vehicle's unique identifier; if the vehicle has not passed through a dynamic weighing device, the vehicle weight information is obtained from the database based on the vehicle's unique identifier; S3. Vehicle trajectory tracking: The vehicle trajectory in the video surveillance area is dynamically tracked throughout the entire process, including single-camera multi-target vehicle tracking and cross-camera vehicle re-identification, to obtain the spatiotemporal distribution of vehicle load on the bridge deck and to track the vehicles passing on the bridge throughout the entire process.
[0006] Specifically, the basic vehicle information includes: vehicle model, axle, cargo condition, lane where the vehicle is located, and vehicle position coordinates; obtaining the basic vehicle information includes the following sub-steps:
[0007] S1.1, obtaining a frame of image in the bridge traffic monitoring video;
[0008] S1.2. Use the vehicle target detection algorithm based on the YOLOv5 target detection network to identify the vehicle in the image and obtain the vehicle model information and image coordinate information; the vehicle model information includes: small cars, trucks, medium and large buses, and non-motor vehicles;
[0009] S1.3, build a fine-grained classification network, perform fine-grained classification on each vehicle in the identified image, and obtain the number of axles and cargo information of each truck;
[0010] S1.4. Perform pixel-level segmentation of lane lines based on LaneNet, extract lane line equations, and automatically divide and number lanes;
[0011] S1.5, using the lane line position information and the image coordinate information obtained by vehicle target detection, identifying the lane where each vehicle is located;
[0012] S1.6, mapping the position of the vehicle on the image to a reference object of known length, the reference object including road markings and guardrails, estimating the relative position of the vehicle with respect to the surveillance camera;
[0013] S1.7. Based on the relative position of the vehicle and the lane information, combined with the latitude and longitude information of the surveillance camera, the lane width information and the lane orientation information, the offset distance and direction angle of the vehicle relative to the camera are calculated to obtain the absolute position of the vehicle in the world coordinate system.
[0014] Specifically, in step S3, vehicle trajectory tracking is performed, including the following sub-steps:
[0015] S3.1. Realize dynamic tracking of multiple targets based on DeepSort algorithm;
[0016] S3.2. Generate a hash code feature of the vehicle based on a deep hash network, and use the hash code to retrieve the vehicle in a historical vehicle hash code database to match vehicles between different surveillance cameras;
[0017] S3.3. If a vehicle is retrieved from the historical vehicle hash code database, then the vehicle has appeared within the surveillance range of other surveillance cameras on the bridge deck, obtain the unique identification of the vehicle, and update the hash code feature of the vehicle in the database;
[0018] S3.4. If the vehicle is not retrieved in the historical vehicle hash code database, the vehicle appears on the bridge deck for the first time, a new vehicle unique identifier is generated, and the unique identifier and the hash code feature of the vehicle obtained in step S3.2 are stored in the historical vehicle hash code database.
[0019] Furthermore, in step S3.1, the vehicle multi-target dynamic tracking is implemented based on the DeepSort algorithm, and the method is as follows:
[0020] S3.1.1. Obtain the position, size and category of each vehicle in the image from step S1.2 to obtain the vehicle target detection result; use the Kalman filter to predict the current position based on the historical position of each vehicle to obtain the tracker prediction result.
[0021] S3.1.2, matching the tracker prediction results and the detection results, including: feature matching, cascade matching and IOU matching methods, the matching results are successful matching, tracking results not matching, and detection results not matching;
[0022] In feature matching, the target vehicle motion information and appearance information are introduced, the target vehicle position offset is extracted, the Mahalanobis distance between the prediction result and the detection result is calculated, the deep neural network is used to calculate the features of the tracking target and the detection target, and the cosine distance between the prediction result and the detection result is calculated. For the matching results with large Mahalanobis distance and cosine distance, it is judged as mismatch;
[0023] In cascade matching, priority information is introduced. The priority of the tracker that has been matched successfully continuously is higher. The more matching failures, the lower the priority. Combined with the priority information, the Hungarian algorithm is used for matching.
[0024] In IOU matching, the IOU information between the tracking target box and the detection target box is introduced, and the matching results whose IOU is less than the preset threshold are judged as mismatches.
[0025] S3.1.3, based on the matching results, update the tracker status of each target vehicle;
[0026] The tracker states are: uncertain state, determined state, and deleted state. The initial state of the tracker is uncertain. The method for updating the tracker state of each target vehicle is as follows:
[0027] If the number of successful matches is greater than the threshold, the tracker changes from the uncertain state to the certain state; if the number of failed matches between the tracking result and the detection result of the tracker that was previously in the certain state is greater than the threshold, the tracker changes from the certain state to the deleted state;
[0028] For successful matches, update the tracker information. If the number of successful matches is greater than the threshold, set the tracker state to a confirmed state.
[0029] If the tracking result does not match, the number of matching failures is recorded. If the number of matching failures is greater than the threshold, the tracker is set to the deletion state;
[0030] If the detection results do not match, a new tracker is created and initialized to an uncertain state.
[0031] S3.1.4. Update the vehicle feature set. For the successfully matched target vehicle, update the vehicle features calculated by the deep neural network and store the tracking results.
[0032] Further, in step S3.1.2, matching the tracker prediction result with the detection result,
[0033] Furthermore, in step S3.2, a hash code feature of the vehicle is generated based on the deep hash network, and the hash code is used to retrieve the vehicle in the historical vehicle hash code database, as follows:
[0034] S3.2.1. Crop the vehicle image from a frame of the surveillance video and use the deep hash network to generate the vehicle's hash code features. The number of bits of the hash code is 256.
[0035] S3.2.2, using the hash code to retrieve the vehicle in the historical vehicle hash code database, the retrieval method is to calculate the Hamming distance between the hash code features of the vehicle and the hash code features stored in the historical vehicle hash code database, and obtain the vehicles corresponding to the 10 smallest hash codes that are less than the distance threshold;
[0036] S3.2.3. If 10 hash codes cannot be retrieved, the vehicle is not retrieved. If at least 10 hash codes can be retrieved, a voting algorithm is used to obtain the vehicle retrieval result.
[0037] The technical solution of the present invention also provides: an electronic device, comprising:
[0038] one or more processors;
[0039] a storage device having one or more programs stored thereon;
[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned real-time traffic load intelligent perception methods based on deep learning.
[0041] The technical solution of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in any of the above-mentioned real-time traffic load intelligent perception methods based on deep learning are implemented.
[0042] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:
[0043] 1. The real-time traffic load intelligent perception method of the present invention uses a target detection algorithm based on YOLOv5 to locate the position of the vehicle on the image, and combines objects with known distances such as lanes and guardrails on the bridge to perform lane recognition and distance estimation, and convert the position of the vehicle on the image into the position of the vehicle in the world coordinate system.
[0044] 2. The real-time traffic load intelligent perception method of the present invention uses a deep hash network to generate a hash code feature of a vehicle, and uses the hash code to retrieve the vehicle in a database, thereby achieving vehicle matching between different surveillance cameras, and combining vehicle target tracking to achieve full-process tracking of vehicles passing on the bridge.
[0045] 3. The real-time traffic load intelligent perception method of the present invention realizes the matching of vehicle information and load information. On the basis of vehicle information identification and vehicle full-process tracking, it can further obtain vehicle weight, axle weight, number of axles and other information, and can monitor the load conditions of the entire bridge in real time. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 This is a flow chart of the real-time traffic load intelligent perception method of the present invention;
[0047] Figure 2 This is a visualization effect diagram of the vehicle target detection result according to an embodiment of the present invention;
[0048] Figure 3 This is a visualization effect diagram of the lane line segmentation result of an embodiment of the present invention;
[0049] Figure 4 This is a flow chart of vehicle multi-target dynamic tracking based on the DeepSort algorithm of the present invention. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the application is further elaborated in detail below in conjunction with the accompanying drawings. The described embodiments are only a part of the embodiments involved in the present invention. All non-innovative embodiments of other researchers in the field on this embodiment belong to the protection scope of the present invention. At the same time, for the step numbering in the embodiment of the present invention, it is only set for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0051] In one embodiment of the present invention, a real-time traffic load intelligent perception method based on deep learning is used to obtain the real-time spatiotemporal distribution information of the bridge load. By combining bridge deck video monitoring and dynamic weighing information, the passing vehicle information is identified, tracked and matched to obtain the spatiotemporal distribution characteristics of the vehicle load.
[0052] In this embodiment, the laboratory conditions of the test are as follows: 1 NVIDIA Quadro RTX 6000 GPU with 24G video memory and a total of 20,000 bridge and road monitoring image data.
[0053] The real-time traffic load intelligent perception method of this embodiment is as follows: Figure 1 As shown, the steps include: vehicle basic information acquisition, load information distribution and vehicle trajectory tracking, as follows:
[0054] S1. Obtaining basic vehicle information:
[0055] By building a deep learning-based target detection network and a fine-grained classification network, the surveillance video frames can be intelligently identified to obtain basic information about the vehicles in the picture.
[0056] The basic vehicle information includes, but is not limited to: vehicle model, axle, cargo condition, lane the vehicle is in, and vehicle location coordinates.
[0057] The basic vehicle information acquisition includes the following sub-steps:
[0058] Step S1.1, obtaining a frame of image in the bridge traffic monitoring video;
[0059] Step S1.2, using the vehicle target detection algorithm based on the YOLOv5 target detection network to identify the vehicle in the image, obtain the vehicle model and image coordinate information, and the vehicle model is divided into four categories: small car, truck, medium and large passenger car, and non-motor vehicle;
[0060] Step S1.3: construct a fine-grained classification network to perform fine-grained classification on each truck identified in step S1.2.
[0061] According to the vehicle target detection results, part of the image of the area where each truck is located is cropped, and the fine-grained classification convolutional neural network is used to classify the number of axles and cargo conditions of the trucks to obtain the information on the number of axles and cargo conditions of each truck; the number of axles is divided into two axles, three axles, four axles, five axles, six axles, and others, and the cargo conditions are divided into fully loaded and empty.
[0062] The final output vehicle detection result is Info_detect, which is as follows: ; Among them, box=[x1, y1, x2, y2], (x1, y1) and (x2, y2) represent the coordinates of the upper left vertex and lower right vertex of the vehicle detection box respectively, model represents the specific model, axes represents the number of axles of the vehicle, and cargo represents the cargo situation.
[0063] In this embodiment, the vehicle target detection result visualization effect is as follows: Figure 2 shown.
[0064] Step S1.4: Perform pixel-level segmentation on lane lines based on LaneNet and extract lane line equations.
[0065] The specific method is: use the lane line image segmentation convolutional neural network to segment the lane line area on the image. The pixel set, and then reduce the segmented area of each lane to a curve with a width of 1, called a lane curve.
[0066] In this embodiment, the lane line segmentation result visualization effect is as follows: Figure 3 shown.
[0067] Step S1.5, using the lane line position information and the image coordinate information obtained by vehicle target detection, identifying the lane where each vehicle is located;
[0068] According to the coordinate information of each vehicle obtained by vehicle target detection, a ray is drawn from the vehicle position to the left along the x-axis of the image, and the number of intersections between this ray and each lane curve is calculated. The number of the lane where the vehicle is located is equal to the number of intersections.
[0069] Step S1.6: Map the position of the vehicle on the image to reference objects of known length, such as road markings and guardrails, to estimate the relative position of the vehicle with respect to the surveillance camera.
[0070] Step S1.7, based on the relative position of the vehicle and the lane information, combined with the latitude and longitude information of the camera, the lane width information and the lane orientation information, calculate the offset distance and direction angle of the vehicle relative to the camera to obtain the position of the vehicle in the world coordinate system.
[0071] S2. Load information matching:
[0072] The purpose is to match the bridge deck video surveillance and dynamic weighing information. Based on the basic vehicle information obtained in step S1 and the vehicle gross weight, axle weight and other information obtained in the dynamic weighing system, the video surveillance vehicle information and the dynamic weighing vehicle information are matched. The specific steps are as follows:
[0073] Step S2.1, judging whether the vehicle has passed through the dynamic weighing device according to the vehicle position information and the dynamic weighing device position information;
[0074] Step S2.2: If the vehicle passes through the dynamic weighing equipment, the vehicle's gross weight, axle weight, wheelbase, number of axles, etc. are obtained, the information is synchronized to the database, and associated with the vehicle's unique identification;
[0075] Step S2.3: If the vehicle has not passed the dynamic weighing device, obtain the vehicle weight information from the database according to the vehicle unique identification;
[0076] S3. Vehicle trajectory tracking:
[0077] The aim is to achieve full-process dynamic tracking of vehicle trajectories within the video surveillance area, including single-camera multi-target vehicle tracking and cross-camera vehicle re-identification, and thus obtain real-time and accurate spatiotemporal distribution of bridge deck vehicle loads. The specific steps are as follows:
[0078] Step S3.1: Implement dynamic tracking of multiple vehicle targets based on the DeepSort algorithm. Obtain the vehicle target detection result from the position, size and category of each vehicle in the image obtained in step S1; use the Kalman filter to predict the current position based on the historical position of each frame to obtain the tracker prediction result.
[0079] The method of realizing dynamic tracking of multiple vehicle targets based on DeepSort algorithm is as follows:
[0080] S3.1.1. Obtain the position, size and category of each vehicle in the image from step S1.2, and obtain the vehicle target detection result: ; in, represents the i-th detected vehicle, represents the center coordinates of the vehicle in the image, Indicates the width and height of the vehicle, Indicates the type of vehicle;
[0081] Based on the historical positions of each vehicle, the Kalman filter is used to predict the current position and obtain the tracker prediction result: ; in, represents the predicted position of the i-th vehicle at time t, and the function is the Kalman filter prediction function, represents the i-th vehicle at time location;
[0082] S3.1.2, Matching tracker prediction results and detection results, expressed as: ; Where D represents the detection result set, P represents the prediction result set, is a matching pair, indicating the matching result of the i-th vehicle, wherein the matching result includes: successful matching, unmatched tracking result, and unmatched detection result;
[0083] Matching tracker predictions with detections using feature matching, cascade matching, and IOU matching methods, respectively.
[0084] The motion information and appearance information of the target are introduced into feature matching, the position offset of the target is extracted, and the Mahalanobis distance between the prediction result and the detection result is calculated: ; Among them, x represents the test result, represents the mean of the prediction results, Represents the covariance matrix of the prediction results,
[0085] Use a deep neural network to calculate the features of the tracked and detected targets, and calculate the cosine distance between the predicted and detected results: ; Among them, A and B represent the feature vectors of the tracking target and the detection target respectively;
[0086] Use a deep neural network to calculate the features of the tracked and detected targets, and determine that the matching results with large Mahalanobis distance and cosine distance are mismatched;
[0087] Priority information is introduced in cascade matching. The priority of the tracker with continuous successful matching is higher. The more matching failures, the lower the priority. The Hungarian algorithm is used for matching in combination with the priority information.
[0088] The IOU matching introduces the IOU (intersection-over-union) information between the tracking target box and the detection target box: ; Among them, A' and B' represent the tracking target box and the detection target box respectively.
[0089] There are three final matching results: successful matching, unmatched tracking results, and unmatched detection results. Matching results with IOU less than the preset threshold are judged as unmatched.
[0090] S3.1.3. Update the tracker status of each target vehicle based on the matching results: ; in, represents the tracker state of the i-th vehicle at time t, and the function Represents a state update function.
[0091] There are three states of the tracker: uncertain state, confirmed state, and deleted state.
[0092] The initial state of the tracker is uncertain; if the number of successful matches is greater than the threshold, the tracker changes from uncertain to certain; if the tracker that was previously in a certain state fails to match the tracking result with the detection result more than the threshold, the tracker changes from certain to deleted.
[0093] For successful matches, the tracker information is updated, and if the number of successful matches is greater than a threshold, the tracker state is set to a determined state.
[0094] In the case where the tracking results are not matched, the number of matching failures is recorded. If the number of matching failures is greater than the threshold, the tracker is set to the deletion state.
[0095] If the detection result does not match, a new tracker is created and initialized to an uncertain state.
[0096] S3.1.4. Update the vehicle feature set. For the successfully matched target vehicle, update the vehicle features calculated by the deep neural network and store the tracking results: ; in, represents the feature set of the i-th vehicle at time t, function represents the deep neural network calculation function, Represents the detection result of the i-th vehicle.
[0097] Finally, the tracking results are stored.
[0098] In this embodiment, the vehicle multi-target dynamic tracking process based on the DeepSort algorithm is as follows: Figure 4 shown.
[0099] Step S3.2: Generate a hash code feature of the vehicle based on the deep hash network, and use the hash code to retrieve the vehicle in the historical vehicle hash code database. The specific method is as follows:
[0100] S3.2.1. Cut out the vehicle image from a frame of the bridge traffic monitoring video and use the deep hashing network to generate the vehicle’s hash code features: ; Among them, the number of bits of the hash code is 256, I represents the input vehicle image, H represents the generated hash code feature, and θ represents the weight of the deep hash network. .
[0101] S3.2.2. Use the hash code to retrieve the vehicle in the historical vehicle hash code database. The retrieval method is to calculate the Hamming distance between the hash code feature of the vehicle and the hash code feature stored in the historical vehicle hash code database: ; in, Represent two hash code features respectively, D represents the Hamming distance, represents the exclusive OR operation, They respectively represent the hash code feature of vehicle i and the hash code feature of vehicle i stored in the historical vehicle hash code database;
[0102] Get the vehicles corresponding to the 10 smallest hash codes that are smaller than the distance threshold.
[0103] S3.2.3. If 10 hash codes cannot be retrieved, the vehicle is deemed not to have been retrieved. If at least 10 hash codes can be retrieved, a voting algorithm is used to obtain the vehicle retrieval result.
[0104] Step S3.3: If a vehicle is retrieved from the historical vehicle hash code database, it means that the vehicle has appeared within the monitoring range of other surveillance cameras on the bridge deck. Then, the unique identification of the vehicle is obtained, and the hash code feature of the vehicle in the database is updated.
[0105] Step S3.4: If the vehicle is not retrieved in the database, it means that the vehicle appears on the bridge deck for the first time, then a new vehicle unique identifier is generated, and the unique identifier and the hash code feature of the vehicle obtained in step S10 are stored in the historical vehicle hash code database.
[0106] In an embodiment of the present invention, an electronic device is also provided, including: one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by the one or more processors, the one or more processors implement the real-time traffic load intelligent perception method described in any of the above embodiments.
[0107] In an embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the program is executed by a processor, the steps in any one of the real-time traffic load intelligent perception methods in the above embodiments are implemented.
[0108] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A real-time traffic load intelligent perception method based on deep learning, It is characterized in that The steps include: S1. Obtain basic vehicle information: By building a deep learning-based target detection network and a fine-grained classification network, intelligent recognition is performed on bridge deck video surveillance to obtain basic vehicle information in video surveillance; S2. Matching load information: Based on the basic vehicle information obtained in step S1, the gross vehicle weight and axle weight information are obtained from the dynamic weighing system, and the basic vehicle information in the video surveillance is matched with the vehicle information in the dynamic weighing system; Determine whether the vehicle has passed through a dynamic weighing device: If the vehicle has passed through a dynamic weighing device, the vehicle's gross weight, axle weight, wheelbase, and number of axles are obtained from the dynamic weighing system, the information is synchronized to the database, and associated with the vehicle's unique identifier; if the vehicle has not passed through a dynamic weighing device, the vehicle weight information is obtained from the database based on the vehicle's unique identifier; S3. Vehicle trajectory tracking: The vehicle trajectory in the video surveillance area is dynamically tracked throughout the entire process, including single-camera multi-target vehicle tracking and cross-camera vehicle re-identification, to obtain the spatiotemporal distribution of vehicle load on the bridge deck and to track the vehicles passing on the bridge throughout the entire process.
2. According to the real-time traffic load intelligent perception method based on deep learning according to claim 1, It is characterized in that The basic vehicle information includes: vehicle model, axle, cargo load, lane where the vehicle is located, and vehicle position coordinates; obtaining the basic vehicle information includes the following sub-steps: S1.1, obtaining a frame of image in the bridge traffic monitoring video; S1.
2. Use the vehicle target detection algorithm based on the YOLOv5 target detection network to identify the vehicle in the image and obtain the vehicle model information and image coordinate information; the vehicle model information includes: small cars, trucks, medium and large buses, and non-motor vehicles; S1.3, build a fine-grained classification network, perform fine-grained classification on each vehicle in the identified image, and obtain the number of axles and cargo information of each truck; S1.
4. Perform pixel-level segmentation of lane lines based on LaneNet, extract lane line equations, and automatically divide and number lanes; S1.5, using the lane line position information and the image coordinate information obtained by vehicle target detection, identifying the lane where each vehicle is located; S1.6, mapping the position of the vehicle on the image to a reference object of known length, the reference object including road markings and guardrails, estimating the relative position of the vehicle with respect to the surveillance camera; S1.
7. Based on the relative position of the vehicle and the lane information, combined with the latitude and longitude information of the surveillance camera, the lane width information and the lane orientation information, the offset distance and direction angle of the vehicle relative to the camera are calculated to obtain the absolute position of the vehicle in the world coordinate system.
3. According to claim 2, the real-time traffic load intelligent perception method based on deep learning, It is characterized in that In step S1.3, each vehicle in the identified image is classified into fine-grained categories as follows: According to the YOLOv5 target detection results, part of the image of the area where each vehicle is located is cropped, and the number of axles and cargo load of the vehicle are classified using a fine-grained classification convolutional neural network to obtain the number of axles and cargo load information of each vehicle; Output the vehicle detection result Info_detect, which is as follows: ; Among them, box=[x1, y1, x2, y2], (x1,y1), (x2,y2) represent the coordinates of the upper left vertex and the lower right vertex of the vehicle detection box respectively, model represents the specific model, axes represents the number of axles of the vehicle, and cargo represents the cargo situation.
4. According to claim 1, the real-time traffic load intelligent perception method based on deep learning, It is characterized in that In step S3, vehicle trajectory tracking is performed, including the following sub-steps: S3.
1. Realize dynamic tracking of multiple targets based on DeepSort algorithm; S3.
2. Generate a hash code feature of the vehicle based on a deep hash network, and use the hash code to retrieve the vehicle in a historical vehicle hash code database to match vehicles between different surveillance cameras; S3.
3. If a vehicle is retrieved from the historical vehicle hash code database, then the vehicle has appeared within the surveillance range of other surveillance cameras on the bridge deck, obtain the unique identification of the vehicle, and update the hash code feature of the vehicle in the database; S3.
4. If the vehicle is not retrieved in the historical vehicle hash code database, the vehicle appears on the bridge deck for the first time, a new vehicle unique identifier is generated, and the unique identifier and the hash code feature of the vehicle obtained in step S3.2 are stored in the historical vehicle hash code database.
5. According to claim 4, the real-time traffic load intelligent perception method based on deep learning, It is characterized in that In step S3.1, the vehicle multi-target dynamic tracking is realized based on the DeepSort algorithm, and the method is as follows: S3.1.
1. Obtain the position, size and category of each vehicle in the image from step S1.2, and obtain the vehicle target detection result: ; in, Indicates detected vehicles, represents the center coordinates of the vehicle in the image, Indicates the width and height of the vehicle, Indicates the type of vehicle; Based on the historical positions of each vehicle, the Kalman filter is used to predict the current position and obtain the tracker prediction result: ; in, Indicates Vehicles at time The predicted position of is the Kalman filter prediction function, Indicates Vehicles at time location; S3.1.2, Matching tracker prediction results and detection results, expressed as: ; in, Represents the set of detection results, Represents the set of prediction results, is a matching pair, indicating The matching results of vehicles, i =1,2,..., n ; The matching results include: successful matching, unmatched tracking results, and unmatched detection results; S3.1.
3. Update the tracker status of each target vehicle based on the matching results: ; in, Indicates Vehicles at time The tracker state of Represents the state update function; S3.1.
4. Update the vehicle feature set. For the successfully matched target vehicle, update the vehicle features calculated by the deep neural network and store the tracking results: ; in, Indicates Vehicles at time The feature set, function represents the deep neural network calculation function, Indicates The detection results of the vehicle.
6. According to claim 5, the real-time traffic load intelligent perception method based on deep learning, It is characterized in that In step S3.1.2, matching the tracker prediction results with the detection results includes: feature matching, cascade matching and IOU matching methods, In feature matching, the target vehicle motion information and appearance information are introduced, the target vehicle position offset is extracted, and the Mahalanobis distance between the prediction result and the detection result is calculated: ; Among them, x represents the test result, represents the mean of the prediction results, Represents the covariance matrix of the prediction results, with superscript T Represents transposition and calculates the cosine distance between the prediction result and the detection result: ; Among them, A and B represent the feature vectors of the tracking target and the detection target respectively; Use a deep neural network to calculate the features of the tracked and detected targets, and determine that the matching results with large Mahalanobis distance and cosine distance are mismatched; In cascade matching, priority information is introduced. The priority of the tracker that has been matched successfully continuously is higher. The more matching failures, the lower the priority. Combined with the priority information, the Hungarian algorithm is used for matching. In IOU matching, the IOU information between the tracking target box and the detection target box is introduced: ; in, Represent the tracking target frame and the detection target frame respectively. and It is used to calculate the intersection area and union area of the target box. For matching results whose IOU is less than the preset threshold, it is judged as a mismatch.
7. According to claim 5, the real-time traffic load intelligent perception method based on deep learning, It is characterized in that In step S3.1.3, the tracker states are: uncertain state, determined state, and deleted state. The initial state of the tracker is uncertain state. The tracker state of each target vehicle is updated as follows: If the number of successful matches is greater than the threshold, the tracker changes from the uncertain state to the certain state; if the number of failed matches between the tracking result and the detection result of the tracker that was previously in the certain state is greater than the threshold, the tracker changes from the certain state to the deleted state; For successful matches, update the tracker information. If the number of successful matches is greater than the threshold, set the tracker state to a confirmed state. If the tracking result does not match, the number of matching failures is recorded. If the number of matching failures is greater than the threshold, the tracker is set to the deletion state; If the detection results do not match, a new tracker is created and initialized to an uncertain state.
8. According to claim 4, the real-time traffic load intelligent perception method based on deep learning, It is characterized in that In step S3.2, a hash code feature of the vehicle is generated based on the deep hash network, and the hash code is used to retrieve the vehicle in the historical vehicle hash code database, as follows: S3.2.
1. Crop the vehicle image from a frame of the surveillance video and use the deep hash network to generate the vehicle’s hash code features: ; The number of bits of the hash code is 256, I represents the input vehicle image, and H represents the generated hash code feature. f stands for Deep hash network function, θ represents the weight of the deep hash network, ; S3.2.
2. Use the hash code to retrieve the vehicle in the historical vehicle hash code database. The retrieval method is: calculate the Hamming distance between the hash code feature of the vehicle and the hash code feature stored in the historical vehicle hash code database: ; in, Represent two hash code features respectively, D represents the Hamming distance, represents the exclusive OR operation, Respectively represent vehicles The hash code features and the vehicles stored in the historical vehicle hash code database Hash code feature, obtain the vehicles corresponding to the 10 smallest hash codes that are smaller than the distance threshold; S3.2.
3. If 10 hash codes cannot be retrieved, the vehicle is not retrieved. If at least 10 hash codes can be retrieved, a voting algorithm is used to obtain the vehicle retrieval result.
9. An electronic device, It is characterized in that include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the real-time traffic load intelligent perception method based on deep learning as described in any one of claims 1 to 8.
10. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and when the program is executed by a processor, the steps in the real-time traffic load intelligent perception method based on deep learning described in any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Spatial distribution monitoring system of whole bridge deck moving load based on dynamic weighing and multi-video information fusion
CN109167956A
Bridge vehicle space-time load monitoring method
CN112444311A
Bridge vehicle load space-time distribution fine-grained identification method based on computer vision
CN113221682A
Real-time back calculation method for traffic flow load of medium-and-small-span bridge
CN115240111A
Large-span bridge vehicle tracking and vehicle load spectrum intelligent identification method based on computer vision
CN117409379A
