Bridge vehicle load judgment system and method based on deep learning and sensor dual backup
Through deep learning and sensor dual backup bridge vehicle load determination system, the YOLOv5 model and CBAM attention mechanism are used to identify vehicle loads, and combined with semantic segmentation and sensor data for verification, solving the problem of high labor costs, poor real-time and accurate in traditional methods. The problem of low degree achieves high accuracy and high reliability load monitoring.
Patent Information
- Application Number
- CN202411114592.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional bridge vehicle load determination methods rely on manual observation and sensor detection, and there are problems such as high labor cost, poor real-time performance and low accuracy.
Using a system based on deep learning and sensor dual backup, through image acquisition and data preprocessing, the YOLOv5 model is used to combine the Convolutional Block Attention Module (CBAM) attention mechanism for vehicle load recognition, and combined with vehicle semantic segmentation and sensor data for load calculation. Implement dual backup verification to improve system reliability.
It improves the accuracy and real-timeness of load determination, reduces labor costs, and enhances the reliability and fault tolerance of the system through the dual backup mechanism to ensure the safe operation of the bridge structure.
Smart Images

Figure CN118942014A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of bridge vehicle transportation, and in particular to a bridge vehicle load determination system and method based on deep learning and dual backup of sensors. Background Art
[0002] The traditional method of determining bridge vehicle load mainly relies on manual observation combined with sensor detection, which has problems such as high labor cost, poor real-time performance, and low accuracy.
[0003] With the development of deep learning technology, the bridge vehicle load determination system based on deep learning has gradually become a new solution. Deep learning technology has powerful feature extraction and classification recognition capabilities. It can be trained on a large amount of bridge vehicle driving data to realize the identification and load determination of vehicles under different load conditions, thereby improving the determination accuracy, reducing labor costs, and improving real-time performance. Summary of the invention
[0004] In order to solve the problems that the existing bridge vehicle load determination method mainly relies on manual observation, sensor monitoring and other means, which has high labor cost, poor real-time performance and low accuracy, the present invention provides a bridge vehicle load determination system and method based on deep learning and sensor dual backup.
[0005] The present invention is achieved by adopting the following technical solutions:
[0006] A bridge vehicle load determination method based on deep learning and sensor dual backup is shown in the attached Figure 7 As shown; including the following steps:
[0007] Step S1: collecting image sample data of various types of vehicles and cargoes, and performing data preprocessing and data enhancement on the image sample data to produce a training data set;
[0008] It includes two sets of solar panels and visual cameras. With the power supply of the solar panels, the cameras transmit the monitoring images to the monitoring service center in real time. The cameras collect the video stream of the scene above the bridge side, obtain the original image data set from the video stream, process and mark the original image data set to obtain the artificial marking data of the vehicle, and obtain the pre-processed image data set, wherein the artificial marking data includes the vehicle position and vehicle model;
[0009] The vehicle models include non-motor vehicles, passenger cars, light trucks, medium trucks, and heavy trucks;
[0010] The processing methods for the original image data set include any one of the following or any combination thereof: Gaussian blur, noise addition, HSV channel color transformation; it can effectively increase the number of data sets, have a certain effect on improving the accuracy of the model, help the learning rate converge and make the model fit.
[0011] Step S2: training the image sample data to obtain a target detection model capable of identifying vehicles and objects to be determined;
[0012] The deep learning algorithm YOLOv5 is used to train and establish a target detection model, which includes an input end (Input), a backbone network (Backbone), a neck module (Neck), a head module (Head) and an output end (Output);
[0013] The preprocessed image is input into the backbone network through the input end (Input), and the backbone network (Backbone) is used to perform feature extraction on the preprocessed image to extract feature maps with different feature scales; the neck module (Neck) upsamples and downsamples the feature maps of different feature scales extracted by the backbone network (Backbone) and re-integrates them into feature maps of different scales, and inputs them into the head module (Head); the head module (Head) is used to perform target box selection and target detection to obtain results, and the Convolutional Block Attention Module (CBAM) attention mechanism is introduced into the head module (Head); CBAM embeds the spatial attention module and the channel attention module between the convolutional layers to enhance the network's attention to important features; the output end (Output) is mainly used to regress the final result detected; as shown in the attached figure Figure 2 shown.
[0014] As attached Figure 3 As shown in the figure, the channel attention mechanism (Channel Attention Module) compresses the feature map by performing average pooling (Average Pooling) and maximum pooling (Max Pooling) on the feature map in the spatial dimension to generate a one-dimensional vector; then these vectors are input into the shared network, element by element, and the channel attention map is generated; the channel attention mechanism can be expressed as:
[0015]
[0016] Among them, M c(F) represents the output of the channel attention module; σ is the activation function; MLP is a multi-layer perceptron; AugPool(F) is a global average pooling operation; MaxPool(F) is a global maximum pooling operation; W 0 , W 1 is the weight parameter in MLP; is the global average pooling result; is the global maximum pooling result;
[0017] The Spatial Attention Module compresses the feature map by performing average pooling and maximum pooling on the channel dimension. Max Pool extracts the maximum value on the channel, and Avg Pool extracts the average value on the channel. The number of extractions for both is height multiplied by width. The extracted feature maps (with 1 channel) are then merged into a 2-channel feature map; the spatial attention mechanism can be expressed as:
[0018]
[0019] Among them, M s (F) represents the output of the spatial attention module; σ is the activation function; f 7×7 is a 7×7 convolution operation; AugPool(F) is a global average pooling operation; MaxPool(F) is a global maximum pooling operation; is the global average pooling result; is the global maximum pooling result.
[0020] By introducing the Convolutional Block Attention Module (CBAM) attention mechanism into the backbone network (Backbone) and the head module (Head), experimental optimization and comparative analysis were carried out. It was concluded that the target detection model with the Convolutional Block Attention Module (CBAM) attention mechanism introduced into the head module (Head) is the best, thereby effectively improving the accuracy of the target detection model in identifying the vehicle to be determined.
[0021] In order to verify the above beneficial effects, the following comparative experiments were conducted:
[0022] Comparative experiment 1: Use the original YOLOv5 model to train vehicle recognition on a lightweight dataset containing 1041 images; the steps are as follows:
[0023] Step 1: Input the preprocessed image into the backbone network through the input terminal, and the backbone network extracts feature maps with different feature scales;
[0024] Step 2: The neck module (Neck) upsamples and downsamples the feature maps of different feature scales extracted by the backbone network (Backbone) to re-fuse feature maps of different scales, and inputs them into the head module (Head);
[0025] Step 3: The head module (Head) performs target box selection and target detection to obtain the result, and the output end (Output) is used to regress the final detected result.
[0026] Comparative experiment 2: The original YOLOv5 model is used, and the CBAM attention mechanism is introduced in the backbone network layer to obtain a target detection model. The target detection model is trained for vehicle recognition on a lightweight dataset containing 1041 images. The steps are as follows:
[0027] Step 1: Input the preprocessed image into the backbone network through the input terminal, introduce the CBAM attention mechanism into the backbone network layer, and extract feature maps with different feature scales from the backbone network;
[0028] Step 2: The neck module (Neck) upsamples and downsamples the feature maps of different feature scales extracted by the backbone network (Backbone) to re-fuse feature maps of different scales, and inputs them into the head module (Head);
[0029] Step 3: The head module (Head) performs target box selection and target detection to obtain the result, and the output end (Output) is used to regress the final detected result.
[0030] Comparative experiment 3: The YOLOv5 model is used, and the CBAM attention mechanism is introduced in the head module (Head) layer to obtain the target detection model. The target detection model is trained for vehicle recognition on a lightweight dataset containing 1041 images. The steps are as follows:
[0031] Step 1: Input the preprocessed image into the backbone network through the input terminal, and the backbone network extracts feature maps with different feature scales;
[0032] Step 2: The neck module (Neck) upsamples and downsamples the feature maps of different feature scales extracted by the backbone network (Backbone) and re-integrates them into feature maps of different scales, and inputs them into the head module (Head). The CBAM attention mechanism is introduced in the head module (Head) layer;
[0033] Step 3: The head module (Head) performs target box selection and target detection to obtain the result, and the output end (Output) is used to regress the final detected result.
[0034] Comparative experiment 4: The YOLOv5 model is used, and the CBAM attention mechanism is introduced in both the backbone network layer and the head module layer to obtain a target detection model. The target detection model is trained on a lightweight dataset containing 1041 images.
[0035] Step 1: Input the preprocessed image into the backbone network through the input terminal, introduce the CBAM attention mechanism into the backbone network layer, and extract feature maps with different feature scales from the backbone network;
[0036] Step 2: The neck module (Neck) upsamples and downsamples the feature maps of different feature scales extracted by the backbone network (Backbone) and re-integrates them into feature maps of different scales, and inputs them into the head module (Head). The CBAM attention mechanism is introduced in the head module (Head) layer;
[0037] Step 3: The head module (Head) performs target box selection and target detection to obtain the result, and the output end (Output) is used to regress the final detected result.
[0038] The experimental results are shown in the following table;
[0039]
[0040] Among them, precision is accuracy, mAP@0.5 (average detection accuracy, when IoU intersection-over-union ratio ≥ 0.5).
[0041] Through the above four sets of comparative experiments, it is concluded that the target detection model that introduces the CBAM attention mechanism in the head module (Head) layer (Experiment 3) is the optimal model, and its mAP@0.5 is improved by 1.4% compared with the original YOLOv5 model.
[0042] Finally, the target detection model that introduces the CBAM attention mechanism in the head module (Head) layer is used to perform iterative vehicle recognition training on 5192 preprocessed images; the steps are consistent with those of Experiment 3, and the final target detection model is obtained. During the training process, the model loss gradually decreases with the increase in the number of training times. When the number of training times reaches about 100 times, the loss change of the model tends to be flat and basically reaches convergence; when the number of training times reaches 200 times, the loss of each vehicle detection model drops below 0.001, and the loss of the model basically no longer changes.
[0043] Select P precision and R recall to evaluate the vehicle detection model. Figure 4 , Attachment Figure 5 As shown; the PR curve mAP@0.5 reaches 99.3%, which has extremely high detection accuracy. The training results are shown in the following table;
[0044]
[0045] Among them, mAP@0.5 (average detection accuracy, when IoU intersection-over-union ratio ≥ 0.5).
[0046] Among them, Figure 4 The horizontal axis is the number of iterations, and the vertical axis from left to right is the positioning loss (Boxloss), confidence loss (Objectness loss), classification loss (Classfication loss), precision (Precision), and recall (Recall).
[0047] Based on the target detection model established in the above process, the video or picture to be detected is input into the input layer (Input), the trained target detection model is used for detection, and the detected video or picture is output in the output layer (Output), so as to identify the video image and screen out the vehicle with load to be determined.
[0048] The target detection vehicle models include: non-motor vehicles, passenger cars, light trucks, medium trucks, and heavy trucks; among which the load to be determined vehicle screening models include: light trucks, medium trucks, and heavy trucks.
[0049] Step S3: Segment the vehicle outline in the image using the vehicle semantic segmentation model; use camera calibration to obtain camera parameters; calculate the load to be determined and the size of the load based on the obtained vehicle outline and camera parameters; as shown in the attached Figure 6 shown.
[0050] The vehicle semantic segmentation model uses a deep learning network to extract image details, semantic information, and pixel features from the original labeled image, obtains feature maps to train a segmentation model, and then segments the vehicle contours in the image of the vehicle category to be determined based on the established segmentation model;
[0051] The camera calibration obtains the focal length, principal point position, position, and posture through the black and white chessboard calibration technology, thereby converting the pixel coordinates into world coordinates. The vehicle and cargo dimensions are calculated based on the obtained vehicle outline pixel coordinates and camera parameters;
[0052] Size calculation uses the camera calibration internal parameter matrix to convert the pixel coordinates of the vehicle outline after semantic segmentation into coordinates in the camera coordinate system; the camera coordinates are converted into world coordinates by using the camera's external parameter matrix. The size of the vehicle and the object can then be calculated using the spatial distance between world coordinate points.
[0053]
[0054] Among them, (X c , Y c , Z c ) is the coordinate in the camera coordinate system, (X w , Y w , Z w ) is the coordinate in the world coordinate system, (u, v) is the coordinate in the pixel coordinate system;
[0055] R is the rotation matrix, T is the translation vector, f x and f y is the focal length, c x and c y are the principal point coordinates, and D is the dimension distance.
[0056] Step S4: Determine the deadweight of the vehicle to be determined through the vehicle deadweight limit standard table; calculate the load weight through the load size and the density of the identified load type; compensate for the influence of speed on the bridge load force, and make a preliminary determination of the vehicle load;
[0057] The vehicle size and the load size are calculated based on the size, and the vehicle type and the load type obtained by the target detection are combined to obtain the load density and volume to obtain the load weight, and the vehicle weight is obtained according to the vehicle weight limit standard table. The load is determined by comprehensively comparing the vehicle weight limit standard table with the influence of the vehicle speed on the load through the total vehicle weight.
[0058] The vehicle speed is obtained by the vehicle speed detection algorithm. The vehicle detection and tracking of this algorithm is based on the established target detection model, replacing the vehicle cascade classifier. The main advantage of using a pre-established detection model for speed detection is its real-time and accuracy. It can quickly and accurately detect vehicles and calculate their speeds, avoid missed detections and false detections, and remain consistent with the vehicle category detection, and can quickly respond to real-time monitoring applications.
[0059] Vehicle speed detection calculates the vehicle speed by detecting and tracking the position difference of the same vehicle in different frame images:
[0060]
[0061] Among them, Δt is the time difference between different frame images, and Δs is the position difference of the same vehicle in different frame images; The influence of vehicle speed on bridge load:
[0062] P=kv 2 +m;
[0063] k is the coefficient related to the characteristics of the vehicle and the bridge; v is the speed of the vehicle; m is the gross weight of the vehicle, and P is the load on the bridge;
[0064] Compare with the vehicle weight limit standard table;
[0065] If P is greater than the weight limit of the detected vehicle type, the vehicle is initially determined to be overloaded. At the same time, the subsequent steps and the verification and determination of the backup module S5 determination results are analyzed. When the two backups produce different determination results, the system determines the correct result through consistency check. Reduce the error judgment caused by the error of a single backup module and improve the reliability of the system;
[0066] If P is less than the weight limit of the detected vehicle type, it is preliminarily determined that the vehicle load is not overloaded. At the same time, the subsequent step and the verification determination backup module S5 determine the result for analysis.
[0067] The vehicle weight limit standard table is as follows:
[0068]
[0069] Step S5: Check whether the load determination module determines correctly based on the sensor backup.
[0070] Based on the sensor backup, the load determination module is checked to see whether the determination is correct; the sensor uses a vibrating wire strain sensor as the main sensor element, which is installed on the lower surface of the bridge span. By monitoring the vibration of the bridge, the load size of the bridge can be effectively determined;
[0071] The introduction of backup based on sensors allows the system to detect and correct errors or inconsistencies between backup modules. When there is a serious deviation between the detection judgment and the predicted result, the backup judgment is switched. When a sudden failure occurs in the sensor judgment mode, it is switched to the deep learning backup judgment method. If a sudden problem occurs in the deep learning judgment, it is switched back to the sensor backup judgment method. When the two backups produce different judgment results, the system can determine the correct result through consistency checks. This fault-tolerant capability reduces the wrong judgment caused by errors in a single backup module and improves the reliability of the system.
[0072] Check the load calculation, obtain the deflection data through the vibrating wire strain sensor installed under the bridge span, and use the 4G wireless transmission module to transmit it to the cloud server through the mqtt protocol and store it in the database;
[0073] The surface strain gauge arranged on the surface of the concrete structure is subjected to the dual effects of deformation and temperature. Its temperature correction coefficient is the difference between the temperature correction coefficient of the surface strain gauge and the linear expansion coefficient of the structure. The calculation formula of the surface strain gauge is:
[0074] ε=k×ΔF+b×ΔT=k×(FF 0 )+(ba)×(TT 0 )
[0075] Where: ε is the strain of the measured structure, the unit is 10 -6 ;
[0076] k is the measurement sensitivity of the surface strain gauge, in units of 10 -6 / F;
[0077] △F is the change of the real-time measurement value of the surface strain gauge relative to the reference value, the unit is F;
[0078] F is the real-time measurement value of the surface strain gauge, in F;
[0079] F 0 is the reference value of the surface strain gauge, in F;
[0080] b is the temperature correction coefficient of the surface strain gauge, unit is 10 -6 / ℃;
[0081] △T is the change of the real-time temperature measurement value relative to the reference value, in °C;
[0082] T is the real-time measurement value of temperature, in °C;
[0083] T 0 is the reference value of temperature, in °C;
[0084] a is the linear expansion coefficient of the structure being measured, in units of 10-6 / ℃;
[0085] The linear expansion coefficient a of concrete is usually 8 to 12×10 -6 / ℃.
[0086] The relationship between its deflection and load is described by the beam deflection equation:
[0087]
[0088] The length of the beam is L, the cross section is rectangular, the unit is m; the elastic modulus of the material is E, the unit is pa; the moment of inertia is I, the unit is m 4 ; y is the disturbance, unit is m; concentrated load is P 1 , unit is N.
[0089] If P 1 If the weight is greater than the weight limit of the detected vehicle type, the vehicle is deemed to be overloaded;
[0090] If P 1 If the weight is less than the weight limit of the detected vehicle type, it is determined that the vehicle load is not overloaded;
[0091] If the judgment result is consistent with the deep learning judgment method, it is the final judgment.
[0092] If the sensor judgment result is inconsistent with the deep learning judgment method, the preliminary judgment will be invalid, the result will be recorded and re-judged and manually eliminated. If the sensor equipment fails, the final result of the deep learning method backup judgment will prevail; if the deep learning equipment fails, the final result of the sensor backup judgment will prevail.
[0093] Based on deep learning and sensor dual backup judgment functions, the deep learning judgment method can also be used as the sensor judgment method to realize the final backup verification judgment.
[0094] A bridge vehicle load determination system based on deep learning and sensor dual backup, as shown in the attached Figure 1 As shown, the system includes:
[0095] The image acquisition module is used to collect image sample data of various types of vehicles and cargoes, and perform data preprocessing and data enhancement on the image sample data to produce a training data set;
[0096] A detection and recognition module is used to train the image sample data to obtain a target detection model capable of identifying the vehicle and the object to be determined;
[0097] The size calculation module uses a deep learning semantic segmentation model to segment the vehicle outline in the image; uses camera calibration to obtain camera parameters; and calculates the size of the vehicle and the load to be determined based on the obtained vehicle outline and camera parameters;
[0098] The load determination module determines the deadweight of the vehicle to be determined through the vehicle deadweight limit standard table; calculates the load weight through the load size and the density of the identified load type; compensates for the influence of speed on the bridge load force, and makes a preliminary determination of the vehicle load;
[0099] Verify and determine the backup module, and determine whether the load determination module is correct based on the sensor backup.
[0100] The system is used to determine the vehicle load in the side-view scene of the bridge, and includes an image acquisition module, a detection and recognition module, a size calculation module, a load determination module, and a verification and determination backup module. Image data is collected, and based on a pre-established detection model, the detection and recognition module is used to detect and screen the abnormal vehicle model and speed of the bridge, including identifying whether the detected vehicle to be determined includes cargo; using the size estimation module to calculate the vehicle and cargo dimensions; using the load determination module to calculate and determine the vehicle load; and verifying the determination result through the verification and determination backup module based on the backup of the vibrating-string strain sensor. The embodiment of the present application can monitor the vehicles and cargo to be weighed in real time and perform dual backup load determination, realizing unattended operation, and fusing and comparing the load data measured by the sensor with the prediction results based on deep learning. Redundant backup is performed through deep learning and sensor technical means, which greatly improves the reliability and fault tolerance of the system and ensures the safety of bridge load monitoring.
[0101] An electronic device includes a processor and a memory; the memory stores computer program instructions that can be executed by the processor, and the processor calls the computer program instructions to execute a bridge vehicle load determination method based on deep learning and dual backup of sensors.
[0102] A computer-readable storage medium stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes a bridge vehicle load determination method based on deep learning and dual backup of sensors.
[0103] The structural design of the present invention is reasonable and reliable. The bridge vehicle load determination system based on deep learning and dual backup of sensors uses a deep learning model to process and analyze the image data of vehicles above the bridge, extracts the characteristic information of the vehicle, and matches it with the known load conditions, and finally realizes accurate determination of the vehicle load. In addition, the system can be combined with a variety of information sources such as sensor monitoring data and video surveillance. Introducing dual backup into the bridge vehicle load determination system based on deep learning can improve the reliability, fault tolerance and safety of the system. Through redundancy and backup, even if a single module fails, the other backup module can still ensure the normal operation of the system. Such a design strategy can ensure the safe operation of the bridge structure, realize all-round monitoring and load determination of vehicles above the bridge, and provide reliable protection for the safe operation of the bridge structure.
[0104] The bridge vehicle load determination method based on deep learning and dual backup of sensors has powerful feature extraction and classification recognition capabilities. It can be trained on a large amount of bridge vehicle driving data to realize the identification and load determination of vehicles under different load conditions, thereby improving the determination accuracy, reducing labor costs, and improving real-time performance. Introducing the dual backup verification and determination module into the load determination system can improve the reliability, fault tolerance and safety of the system, so as to ensure the safe operation of the bridge structure and vehicle status. BRIEF DESCRIPTION OF THE DRAWINGS
[0105] Figure 1 This is a structural diagram of the bridge vehicle load determination system based on deep learning and dual backup of sensors.
[0106] Figure 2 It is a schematic diagram of the structural block diagram of the YOLOv5 algorithm model in the present invention.
[0107] Figure 3 It is a schematic diagram of the CBAM attention mechanism module in the present invention.
[0108] Figure 4 This is a schematic diagram of the target detection model evaluation index in the present invention. Figure 1 .
[0109] Figure 5 This is a schematic diagram of the target detection model evaluation index in the present invention. Figure 2 .
[0110] Figure 6 It is a schematic diagram of the size calculation module in the present invention.
[0111] Figure 7 It is a flow chart of a bridge vehicle load determination method based on deep learning and dual backup of sensors.
[0112] Figure 8 It is a schematic diagram of the vehicle image to be determined after being collected and detected in an embodiment of the present invention.
[0113] Fig. 9 Schematic diagram of pixel coordinate points for calculating the size of a vehicle to be determined in an embodiment of the present invention. DETAILED DESCRIPTION
[0114] Example
[0115] A bridge vehicle load determination method based on deep learning and sensor dual backup, first: based on the deep learning algorithm YOLOv5 trained and established target detection model to identify and screen out the load to be determined vehicles, and through the vehicle semantic segmentation model and camera calibration to obtain the vehicle and load size, and finally combined with the vehicle weight limit standard table to determine whether the load to be determined vehicle is overloaded; second: based on the vibrating string strain sensor installed under the bridge span to obtain its deflection data, and through the relationship between deflection and load to obtain the load of the vehicle to be determined, determine whether the load to be determined vehicle is overloaded; the steps are as follows:
[0116] Step 1: The camera captures image A of the scene above the bridge side, and the pixel of image A is 1920×1080;
[0117] Step 2: Preprocess image A and input it into the target detection model for recognition;
[0118] Step 2.1: Input the preprocessed image A into the backbone network through the input terminal, and the backbone network extracts feature maps with different feature scales;
[0119] Step 2.2: The neck module (Neck) upsamples and downsamples the feature maps of different feature scales extracted by the backbone network (Backbone) and re-integrates them into feature maps of different scales, and inputs them into the head module (Head). The CBAM attention mechanism is introduced into the head module (Head) layer;
[0120] Step 2.3: The head module (Head) performs the target box selection and target detection to obtain the result, and the output end (Output) is used to regress the final result detected, and the vehicle type in image A is obtained as a heavy truck; the recognition confidence is 95%, which is a load determination vehicle, as shown in the attached figure. Figure 8 As shown; and judge its load, the load type in this embodiment is shielding load, so the load type is calculated according to the general density range of wood, and finally the sensor backup is comprehensively judged;
[0121] Step 3: Input image A into the vehicle semantic segmentation model to segment the vehicle contour; and use camera calibration to obtain camera parameters, and calculate the size of the vehicle to be determined and the load based on the obtained vehicle contour and camera parameters;
[0122] Step 3.1: Image A is input into the vehicle semantic segmentation model, and the vehicle contour of image A is segmented by the vehicle semantic segmentation model;
[0123] Step 3.2: Input the vehicle outline of image A into the camera calibration, and convert the pixel coordinates in the vehicle outline of image A into camera coordinates through the intrinsic parameter matrix of camera calibration; convert the camera coordinates of image A into world coordinates through the extrinsic parameter matrix of camera calibration; and then calculate the size of the vehicle using the spatial distance between the world coordinate points;
[0124] The specific process is:
[0125] The camera for image A is calibrated, and its intrinsic parameter matrix is [[107.66920862, 0.0, 929.23930868], [0, 315.17571656, 492.60404244], [0, 0, 1.0]], and the rotation matrix is [[0.99949837, -0.01750557, -0.02639233], [0.02374446, 0.96566183, 0.258 71496], [0.02095711, -0.25921185, 0.96559309]], the translation vector is [[133.47044955], [-4.44087654], [898.61288536]], the pixel coordinates required for acquisition are A(660, 590), B(880, 810), C(780, 700), D(710, 458), E(785, 448), as shown in the attached figure. Fig. 9 As shown, the depth value in the camera coordinate system of this embodiment is Z c = 6 meters, and the coordinate conversion formula of the size calculation module can be obtained, and the corresponding world coordinate point is
[0126] A 1 [-166.95677937, 240.05380631, -856.35364402],
[0127] B 1 [-154.60371164, 243.88351887, -855.59367353],
[0128] C 1 [-160.22326266, 241.95890739, -855.98836623],
[0129] D 1 [-164.23153221, 237.57843422, -857.07730195],
[0130] E 1 [-160.05868110, 237.32143721, -857.23685941],
[0131] Then, the vehicle and cargo dimensions can be obtained from the spatial distance calculation formula of the dimension calculation module, A 1 B 1 =12.955 m, A 1 C 1 =7.007 m, A 1 D 1 =3.752 m, D 1 E 1 =4.148 meters, so the vehicle length is detected to be 12.955 meters, the load length is 7 meters, the width is 4 meters, and the height is 3.75 meters;
[0132] Step 4: Determine the deadweight of the vehicle to be determined through the vehicle deadweight limit standard table; calculate the load weight through the load size and the density of the identified load type; compensate for the influence of speed on the bridge load force, and make a preliminary determination of the vehicle load;
[0133] Step 4.1: Obtain the load category through the target detection model, and obtain the load weight by combining the density and volume of the load category;
[0134] The load weight is calculated based on M = V × ρ = A 1 D 1 ×A 1 C 1 ×D 1 E 1 ×ρ=63t, M is the load weight, A 1 D 1 , A 1 C 1 , D 1 E 1 is the cargo size, ρ is the cargo density, and the density of wood is about 0.6t / m 3 .
[0135] Step 4.2: Obtain the deadweight and load limit of the heavy truck through the vehicle deadweight and weight limit standard table, and obtain the total weight of the heavy truck based on the deadweight of the heavy truck and the weight of the load; the deadweight of the heavy truck is 16.8t and the load limit is 48.8t.
[0136] Step 4.3: The speed of the heavy truck is 46.8 km / h through the speed detection algorithm;
[0137] Step 4.4: By combining the total weight of the heavy truck with the effect of vehicle speed on the load and comparing it with the weight limit of the heavy truck in the vehicle deadweight limit standard table, the load is obtained as P>M+m=79.8t>48.8t, where m is the deadweight of the vehicle; the initial judgment is that it is overloaded.
[0138] Step 5: Check whether the load determination module is correct based on the sensor backup.
[0139] Step 5.1: Obtain the deflection data of the bridge by installing a vibrating wire strain sensor under the bridge span, and obtain the vehicle load through the relationship between the deflection and the load;
[0140] The length of the beam is L = 20m, and the cross section is rectangular; the elastic modulus of the material is E = 2×10 11 pa; moment of inertia is I = 0.06m 4 ; The disturbance y is directly measured by the sensor, and the deflection data collected is y = 0.02m; the concentrated load is P 1 ;Depend on Available, P 1 =72t>48.8t;
[0141] Step 5.2: By comparing the vehicle load with the weight limit of heavy trucks in the vehicle weight limit standard table, it is found that the vehicle load is greater than the load limit of 48.8t and is determined to be overloaded.
[0142] Step 5.3: The judgment result is consistent with that of step 3, and the final judgment is that the heavy truck is overloaded.
[0143] The embodiment avoids the untimely and erroneous judgments caused by equipment failure in the traditional single sensor method. The present invention judges the bridge vehicle load based on deep learning and dual backup of sensors to achieve unmanned operation. The judgment results based on deep learning are comprehensively judged and analyzed with the load data measured by the sensor, which improves the reliability and fault tolerance of the system and ensures the safety of bridge load monitoring.
[0144] When identifying the vehicle and cargo to be determined, the embodiment of the present application can further use the trained deep learning model to detect and identify the vehicle's multi-dimensional feature information such as the vehicle's speed, license plate, traffic flow, etc. for the framework.
[0145] In the description of the present invention, it is necessary to understand that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0146] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A bridge vehicle load determination method based on deep learning and sensor dual backup, characterized in that: The steps include: Step S1: collecting image sample data of various types of vehicles and cargoes, and performing data preprocessing and data enhancement on the image sample data to produce a training data set; Step S2: training the image sample data to obtain a target detection model capable of identifying vehicles and objects to be determined; Step S3: Segment the vehicle outline in the image through a deep learning semantic segmentation model; obtain camera parameters using camera calibration; calculate the size of the vehicle to be determined and the load based on the obtained vehicle outline and camera parameters; Step S4: Determine the deadweight of the vehicle to be determined by using the vehicle deadweight limit standard table; Calculate the load weight by load size and the density of the identified load type; Compensate for the impact of speed on bridge load and make a preliminary assessment of vehicle load; Step S5: Check whether the load determination module determines correctly based on the sensor backup.
2. A bridge vehicle load determination method based on deep learning and sensor dual backup according to claim 1, characterized in that: The step S1: collecting a video stream of a scene above the side of the bridge through a camera, obtaining an original image data set from the video stream, processing and marking the original image data set to obtain manually marked data of the vehicle, and obtaining a preprocessed image data set, wherein the manually marked data includes a vehicle position and a vehicle model; The vehicle models include non-motor vehicles, passenger cars, light trucks, medium trucks, and heavy trucks; The processing method of the original image data set includes any one of the following or any combination thereof: Gaussian blur, noise addition, and HSV channel color transformation.
3. A bridge vehicle load determination method based on deep learning and sensor dual backup according to claim 2, characterized in that: Step S2: using the deep learning algorithm YOLOv5 to train and establish a target detection model, wherein the target detection model includes an input end (Input), a backbone network (Backbone), a neck module (Neck), a head module (Head) and an output end (Output); Inputting the preprocessed image into a backbone network through an input terminal (Input), wherein the backbone network (Backbone) is used to perform feature extraction on the preprocessed image to extract feature maps with different feature scales; The neck module (Neck) upsamples and downsamples the feature maps of different feature scales extracted by the backbone network (Backbone) and re-integrates them into feature maps of different scales, and inputs them into the head module (Head); the head module (Head) is used to select the target frame and detect the target to obtain the result, and the Convolutional Block Attention Module (CBAM) attention mechanism is introduced into the head module (Head); CBAM embeds the spatial attention module and the channel attention module between the convolutional layers to enhance the network's attention to important features; The output terminal (Output) is mainly used to regress the final result detected.
4. The method for determining bridge vehicle load based on deep learning and sensor dual backup according to claim 3 is characterized in that: The channel attention module compresses the feature map by performing average pooling and maximum pooling on the feature map in the spatial dimension to generate a one-dimensional vector; these vectors are then input into the shared network, element-by-element summed and merged to generate a channel attention map; the channel attention mechanism can be expressed as: Among them, M c (F) represents the output of the channel attention module; σ is the activation function; MLP is a multi-layer perceptron; AugPool(F) is a global average pooling operation; MaxPool(F) is a global maximum pooling operation; W0 and W1 are weight parameters in MLP; is the global average pooling result; is the global maximum pooling result; The spatial attention module compresses the feature map by performing average pooling and maximum pooling on the channel dimension respectively; Max Pool extracts the maximum value on the channel, and Avg Pool extracts the average value on the channel, and the number of extractions for both is height multiplied by width; then the extracted feature maps are merged into a 2-channel feature map; the spatial attention mechanism can be expressed as: Among them, M s (F) represents the output of the spatial attention module; σ is the activation function; f 7×7 is a 7×7 convolution operation; AugPool(F) is a global average pooling operation; MaxPool(F) is a global maximum pooling operation; is the global average pooling result; is the global maximum pooling result.
5. The method for determining bridge vehicle load based on deep learning and sensor dual backup according to claim 4 is characterized in that: Step S3: The vehicle semantic segmentation model uses a deep learning network to extract image details, semantic information and pixel features from the original labeled image, obtains a feature map to train a segmentation model, and segments the vehicle contour in the image of the vehicle category to be determined based on the established segmentation model; The camera calibration obtains the focal length, principal point position, position, and posture through the black and white chessboard calibration technology, thereby converting the pixel coordinates into world coordinates; the vehicle and cargo sizes are calculated based on the obtained vehicle outline pixel coordinates and camera parameters; Size calculation: The pixel coordinates of the vehicle outline after semantic segmentation are converted into coordinates in the camera coordinate system through the camera calibration intrinsic parameter matrix; the camera coordinates are converted into world coordinates by using the camera extrinsic matrix; and the size of the vehicle and the object can be calculated by the spatial distance between the world coordinate points; Among them, (X c , Y c , Z c ) is the coordinate in the camera coordinate system, (X w , Y w , Z w ) is the coordinate in the world coordinate system, (u, v) is the coordinate in the pixel coordinate system; R is the rotation matrix, T is the translation vector, f x and f y is the focal length, c x and c y are the principal point coordinates, and D is the dimension distance.
6. A bridge vehicle load determination method based on deep learning and sensor dual backup according to claim 5, characterized in that: The step S4: based on the size calculation, the vehicle size and the load size are obtained, and the vehicle type and the load type obtained by the target detection are combined to obtain the load density and volume to obtain the load weight, and the vehicle weight is obtained according to the vehicle weight limit standard table, and the load is determined by comprehensively comparing the vehicle weight limit standard table with the influence of the vehicle gross weight and the vehicle speed on the load; Vehicle speed detection calculates the vehicle speed by detecting and tracking the position difference of the same vehicle in different frame images: Among them, Δt is the time difference between different frame images, and Δs is the position difference of the same vehicle in different frame images; Effect of vehicle speed on bridge loads: P=kv 2 +m; k is the coefficient related to the characteristics of the vehicle and the bridge; v is the speed of the vehicle; m is the gross weight of the vehicle, and P is the load on the bridge; Compare with the vehicle weight limit standard table; If P is greater than the weight limit of the detected vehicle type, the vehicle load is preliminarily determined to be overloaded; at the same time, the judgment result of the backup module S5 is analyzed in the subsequent steps and checked. When the two backups produce different judgment results, the system determines the correct result through consistency check, reduces the wrong judgment caused by the error of a single backup module, and improves the reliability of the system; If P is less than the weight limit of the detected vehicle category, it is preliminarily determined that the vehicle load is not overloaded; at the same time, the judgment result of the backup module is analyzed in the subsequent steps and verification.
7. A bridge vehicle load determination method based on deep learning and sensor dual backup according to claim 6, characterized in that: The step S5: based on the sensor backup, checking whether the load determination module is correct; the sensor uses a vibrating wire strain sensor as the main sensor element, which is installed on the lower surface of the bridge span, and can effectively determine the load size of the bridge by monitoring the vibration of the bridge; Check the load calculation, obtain the deflection data through the vibrating wire strain sensor installed under the bridge span, and use the 4G wireless transmission module to transmit it to the cloud server through the mqtt protocol and store it in the database; The surface strain gauge arranged on the surface of the concrete structure is subjected to the dual effects of deformation and temperature. Its temperature correction coefficient is the difference between the temperature correction coefficient of the surface strain gauge and the linear expansion coefficient of the structure. The calculation formula of the surface strain gauge is: ε=k×ΔF+b×ΔT=k×(F-F0)+(ba)×(T-T0); Where: ε is the strain of the measured structure, the unit is 10 -6 ; k is the measurement sensitivity of the surface strain gauge, in units of 10 -6 / F; △F is the change of the real-time measurement value of the surface strain gauge relative to the reference value, the unit is F; F is the real-time measurement value of the surface strain gauge, in F; F0 is the reference value of the surface strain gauge, in F; b is the temperature correction coefficient of the surface strain gauge, unit is 10 -6 / ℃; △T is the change of the real-time temperature measurement value relative to the reference value, in °C; T is the real-time measurement value of temperature, in °C; T0 is the reference value of temperature, in °C; a is the linear expansion coefficient of the structure being measured, in units of 10 -6 / ℃; The linear expansion coefficient a of concrete is usually taken as: 8~12×10 -6 / ℃; The relationship between its deflection and load is described by the beam deflection equation: The length of the beam is L, the cross section is rectangular, the unit is m; the elastic modulus of the material is E, the unit is pa; the moment of inertia is I, the unit is m 4 ; y is the disturbance, the unit is m; The concentrated load is P1, in N; If P1 is greater than the weight limit of the detected vehicle type, the vehicle is determined to be overloaded; If P1 is less than the weight limit of the detected vehicle type, it is determined that the vehicle load is not overloaded; If the judgment result is consistent with the deep learning judgment method, it is the final judgment; If the sensor judgment result is inconsistent with the deep learning judgment method, the preliminary judgment will be invalid, the result will be recorded and re-judged and manually eliminated. If the sensor equipment fails, the final result of the deep learning method backup judgment will prevail; If the deep learning device fails, the sensor backup will determine the final result; Based on deep learning and sensor dual backup judgment functions, the deep learning judgment method can also be used as the sensor judgment method to realize the final backup verification judgment.
8. A bridge vehicle load determination system based on deep learning and sensor dual backup, characterized in that: The system includes: The image acquisition module is used to collect image sample data of various types of vehicles and cargoes, and perform data preprocessing and data enhancement on the image sample data to produce a training data set; A detection and recognition module is used to train the image sample data to obtain a target detection model capable of identifying the vehicle and the object to be determined; The size calculation module uses a deep learning semantic segmentation model to segment the vehicle outline in the image; uses camera calibration to obtain camera parameters; and calculates the size of the vehicle and the load to be determined based on the obtained vehicle outline and camera parameters; The load determination module determines the deadweight of the vehicle to be determined through the vehicle deadweight limit standard table; calculates the load weight through the load size and the density of the identified load type; compensates for the influence of speed on the bridge load force, and makes a preliminary determination of the vehicle load; Verify and determine the backup module, and determine whether the load determination module is correct based on the sensor backup.
9. An electronic device, characterized in that: It includes a processor and a memory; the memory stores computer program instructions that can be executed by the processor, and the processor calls the computer program instructions to execute a bridge vehicle load determination method based on deep learning and dual backup of sensors as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a computer, enable the computer to execute a bridge vehicle load determination method based on deep learning and dual backup of sensors as described in any one of claims 1 to 7.
Citation Information
Cited By
Bridge monitoring method, device and equipment based on multi-dimensional perception fusion
CN119714749A
Bridge monitoring method, device and equipment based on multi-dimensional perception fusion
CN119714749B
