A computer vision-based system and method for monitoring the spatiotemporal distribution of bridge traffic loads

Abstract

This invention provides a computer vision-based system and method for monitoring the spatiotemporal distribution of bridge traffic loads. The system includes a toll station vehicle dynamic weighing and capture unit, a bridge traffic video acquisition unit, a traffic flow group identification and tracking unit, and a traffic load spatiotemporal distribution information processing unit. The traffic flow group identification and tracking unit acquires bridge traffic features through a YOLO-based vehicle identification module, fuses toll station vehicle information streams and multi-level associated vehicle particle groups, and processes this information through a vehicle appearance feature matching and tracking module. This achieves multi-level matching and trajectory tracking of individual and group vehicles on the bridge. Finally, the traffic load spatiotemporal distribution information processing unit merges the traffic flow spatiotemporal distribution information (composed of trajectory and time information) with the traffic flow moving load information (composed of vehicle weight, speed, and road surface unevenness) and displays it in real time on a display terminal. This invention enables real-time monitoring of the spatiotemporal distribution of traffic loads, providing a guarantee for the safe operation of bridges.

Description

Technical Field

[0001] This invention relates to the field of bridge health monitoring technology, specifically to a computer vision-based system and method for monitoring the spatiotemporal distribution of bridge deck traffic loads. Background Technology

[0002] Bridges play a crucial role in my country's transportation infrastructure, and their structural safety and durability are paramount for safe operation. Besides being affected by service life, natural environment, and their own materials, bridges are also influenced by the distribution of moving loads from traffic flow on the bridge deck. The random distribution of moving loads from traffic flow on the bridge deck is a significant factor affecting bridge service performance and operational safety. These loads are not only random but also change with factors such as the development of the automotive industry and society, making them a type of multidimensional random variable moving load. Furthermore, they are influenced by factors such as vehicle speed, vehicle length, and axle load, further complicating the randomness of the traffic load. The statistical patterns of moving load distribution on the bridge deck are not only important for evaluating the long-term performance of bridge structures but also provide valuable reference for improving bridge design and construction quality, offering data support for bridge operators.

[0003] Computer vision technology is an important branch of artificial intelligence. It refers to the collection of images through cameras, and the subsequent recognition, trajectory tracking, and image-based measurements of target images or text by computers. Computer vision technology helps humans further process image information and achieve their goals more accurately and quickly. Currently, the applications of computer vision technology in civil engineering construction mainly include vehicle type classification, image and text processing, and crack detection.

[0004] Based on the above, this invention discloses a bridge deck traffic load spatiotemporal distribution monitoring system based on the computer vision target detection technology YOLO. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a computer vision-based system and method for monitoring the spatiotemporal distribution of bridge traffic loads, thereby addressing the shortcomings of the existing technology and enabling accurate and real-time monitoring of the moving load distribution of random traffic flow on the bridge.

[0006] This paper presents a computer vision-based method that combines a dynamic weighing system to monitor the spatiotemporal distribution of traffic loads on bridge decks. This method is applicable to different types of long-span bridges and can accurately and in real-time monitor the moving load distribution of random traffic flow on the bridge deck and record the key characteristics of the vehicles.

[0007] To solve the above-mentioned technical problems, the present invention includes:

[0008] A computer vision-based bridge traffic load spatiotemporal distribution monitoring system includes a toll station vehicle dynamic weighing and capture unit, a bridge traffic flow video acquisition unit, a traffic flow group identification and tracking unit, and a traffic load spatiotemporal distribution information processing unit. The vehicle dynamic weighing and capture unit is connected to both the traffic flow group identification and tracking unit and the traffic load spatiotemporal distribution information processing unit. The bridge traffic flow video acquisition unit, traffic flow group identification and tracking unit, and traffic load spatiotemporal distribution information processing unit are sequentially connected. The toll station vehicle dynamic weighing and capture unit acquires information streams of vehicles passing through the toll station. These information streams include vehicle weight, vehicle type, color, and time sequence information. The bridge traffic flow video acquisition unit captures random traffic flow images on the bridge surface. The traffic flow group identification and tracking unit includes a vehicle recognition module based on the YOLO framework and a vehicle appearance feature matching and tracking module. The vehicle recognition module is used to perform bridge surface calibration and detection on the real-time image transmission of random traffic flow on the bridge surface from the bridge surface video acquisition unit, obtaining the spatiotemporal coordinates, vehicle type, and color information of the random traffic flow on the bridge surface, and forming a multi-level associated vehicle particle group of the random traffic flow on the bridge surface; the vehicle appearance feature matching and tracking module is used to perform multi-level cyclic matching between the toll station vehicle information stream obtained by the toll station vehicle dynamic weighing and capture unit and the multi-level associated vehicle particle group of the random traffic flow on the bridge surface obtained by the vehicle recognition module, realizing the fusion of bridge surface vehicles and corresponding vehicle weights, and realizing trajectory tracking, speed estimation, and time information acquisition of the random traffic flow on the bridge surface; the traffic flow load spatiotemporal distribution information processing unit is used to process the driving trajectory, time, and speed information of the random traffic flow on the bridge surface obtained by the vehicle appearance feature matching and tracking module and the vehicle weight information obtained by the toll station vehicle dynamic weighing and capture unit to obtain the spatiotemporal distribution information of the traffic flow load on the bridge surface.

[0009] Furthermore, the toll station vehicle dynamic weighing and capture unit includes a toll lane PC and a vehicle separator, a capture camera, and a weighing instrument, all connected to the toll lane PC. The vehicle separator and the weighing instrument are bidirectionally connected. The vehicle separator is used to separate vehicles and provide start and end signals to the weighing instrument. The weighing instrument obtains vehicle load and axle count information by weighing axle weight. The capture camera is used to acquire and record the vehicle type and color information of the corresponding vehicle. The toll lane PC is used to summarize the weight, type, color, and time sequence information of all passing vehicles and form a toll station vehicle information stream.

[0010] Furthermore, the bridge traffic flow video acquisition unit includes an industrial camera array and a storage module and a communication module, both connected to the industrial camera array; the industrial camera array is used to acquire bridge traffic flow images, the storage module is used to store the bridge traffic flow images acquired by the industrial camera array, and the communication module is used to transmit the bridge traffic flow images acquired by the industrial camera array to the traffic flow identification and tracking subsystem in real time.

[0011] Furthermore, the vehicle recognition module includes a YOLO multimodal vehicle recognition model trained based on a VGG16 convolutional neural network and the PyTorch framework.

[0012] A computer vision-based method for monitoring the spatiotemporal distribution of bridge deck traffic loads includes the following steps:

[0013] Step S1: Obtain vehicle weight, vehicle type, color, and time sequence information of vehicles passing through the toll station, and form a toll station vehicle information stream; in this step, the toll station information stream X of the i-th vehicle... i Specifically:

[0014] X i =(x i1 ,x i2 ,x i3 ,x i4 )

[0015] Where, x i1 x i2 x i3 and x i4 These represent the vehicle model, color, weight, and time information of the i-th vehicle, respectively.

[0016] Step S21: Collect and transmit real-time images of random traffic flow across the entire bridge surface, and perform bridge surface calibration and detection on the real-time collected images to obtain the spatiotemporal coordinates, vehicle type, and color information of the random traffic flow on the bridge surface; in this step, the spatiotemporal distribution information stream Y of the j-th vehicle on the bridge surface... j Specifically:

[0017] Y j =(y j1 ,y j2 ,y j3 ,y j4 )

[0018] Among them, y j1 y j2 y j3 and y j4 These represent the vehicle type, color, location, and time information of the j-th vehicle on the bridge.

[0019] Step S22: Based on the vehicle type and color information of the random traffic flow on the bridge surface with spatiotemporal distribution characteristics obtained in Step S21, calculate the Euclidean distance between ordinary particles and the core particle vehicle in the middle lane within the video frame. Take Euclidean distance The smallest n cars are considered as a swarm of traffic particles, and the Euclidean distance is considered. The average of the sums is used to obtain the swarm weight W of the n vehicles closest to the core particle vehicle in the middle lane. best nSpecifically:

[0020]

[0021] and several different W best n A multi-level interconnected vehicle particle swarm is formed for traffic flow on the bridge surface.

[0022] Step S3: Perform multi-level cyclic matching on the toll station vehicle information flow obtained in step S1 and the multi-level associated vehicle particle swarm of the bridge surface random traffic flow obtained in step S22 to achieve the fusion of bridge surface vehicles and corresponding vehicle weights. Then, use the Deepsort multi-target tracking model to achieve trajectory tracking, speed estimation and time information acquisition of the bridge surface random traffic flow.

[0023] Step S4: The spatiotemporal distribution information of traffic flow, composed of trajectory and time information, is fused with the traffic flow moving load information, composed of vehicle weight, vehicle speed, and road surface unevenness, to obtain the spatiotemporal distribution of bridge deck traffic load.

[0024] Furthermore, in step S3, the toll station vehicle information stream is fused with the multi-level associated vehicle particle swarm and subjected to multi-level cyclic matching, specifically including the following steps:

[0025] Step S31: First, calculate y based on the Euclidean distance between the required monitoring area and the toll station. j4 The corresponding time confidence interval is used to perform ORB feature matching on vehicle images passing through the toll station and within the monitored area within the corresponding time period. This yields a matching similarity matrix A composed of a reference particle swarm of m vehicles at the toll station and a bridge surface vehicle particle swarm containing n vehicles. m*n , m>n, specifically:

[0026] a i1 =max{a 11 ,…,a m1},(i=1,2,…m)

[0027]

[0028] Among them, a mn This represents the ORB feature matching value between the m-th vehicle passing through the toll station and the n-th vehicle within the monitoring area of ​​the bridge surface, a. i1 Let Sum be the optimal matching value between the first vehicle in the bridge monitoring area and the m vehicle information streams at the toll station, and let A be the similarity matrix of this matching. m*n The sum of the optimal matching values ​​between n vehicles traveling on the bridge and m vehicle information streams at the toll station;

[0029] Step S32: Next, based on the continuous information flow Q of m vehicles passing through the toll station Xi mAnd the vehicle particle swarm W on the bridge surface within the monitored area best n x in both groups i1 With y j1 x i2 With y j2 Auxiliary judgment is performed, and when both are equal, it is considered that the individual matching of bridge traffic flow and toll station vehicles has been completed; when the overall matching accuracy reaches the set value, multi-level association matching of individual and group bridge traffic flow and toll station vehicles has been achieved.

[0030] Step S33: After obtaining a single Sum value, repeat steps S21, S22, S31 and S32. Using computer vision technology and ORB feature matching, perform multi-level cyclic matching between the bridge traffic flow and vehicles passing through the toll station to achieve multi-level association matching between the entire bridge traffic flow and vehicles at the toll station. Based on the vehicle weight obtained in step S1, complete the individual matching of vehicle-vehicle weight with the bridge traffic flow.

[0031] Furthermore, in step S4, the spatiotemporal distribution information of the traffic flow on the bridge is obtained by tracking the trajectories of individual vehicles and groups of random traffic flow on the bridge.

[0032] Furthermore, in step S4, the moving impact load caused by random traffic flow on the bridge is obtained based on the vehicle weight and speed of the corresponding vehicle and the road surface unevenness.

[0033] Furthermore, the moving impact load P caused by random traffic flow on the bridge surface is specifically as follows:

[0034] P = M * f

[0035] Where M represents the weight of vehicles on the bridge deck, and f represents the dynamic random impact coefficient caused by road surface unevenness, etc.

[0036] The beneficial effects of this invention are:

[0037] The YOLO multimodal vehicle model of this invention is applicable to real-time monitoring and timely early warning of the spatiotemporal distribution of random traffic flow moving loads on bridge decks in various locations and environments, and is low-cost and highly efficient. This invention utilizes deep learning algorithms and an improved YOLO framework to achieve spatiotemporal distribution monitoring of random traffic flow moving loads on bridge decks. In practical applications, it exhibits adaptive and self-learning characteristics, contributing to further improvements in the intelligent monitoring of bridge traffic load distribution. This invention enables long-term real-time monitoring of the moving load distribution of random traffic flow on bridge decks, providing bridge managers with real-time data and early warning information, and ensuring the safe operation of bridges. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the present invention;

[0039] Figure 2 This is a layout diagram of the industrial cameras in the bridge traffic flow video acquisition unit;

[0040] Figure 3 This is a diagram illustrating the spatiotemporal distribution of random traffic flow on the bridge surface as implemented in this embodiment.

[0041] Figure 4 It is a spatiotemporal distribution map of the random traffic flow load on the bridge deck. Detailed Implementation

[0042] The operation of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The description of these embodiments is used to help understand the specific content of the present invention, but does not constitute a limitation on the scope of the present invention, and only represents one embodiment of the present invention.

[0043] like Figure 1 As shown, this invention provides a computer vision-based bridge traffic load spatiotemporal distribution monitoring system, including a toll station vehicle dynamic weighing and capture unit, a bridge traffic flow video acquisition unit, a traffic flow group identification and tracking unit, and a traffic load spatiotemporal distribution information processing unit. The vehicle dynamic weighing and capture unit is connected to the traffic flow group identification and tracking unit and the traffic load spatiotemporal distribution information processing unit, respectively. The bridge traffic flow video acquisition unit, the traffic flow group identification and tracking unit, and the traffic load spatiotemporal distribution information processing unit are connected sequentially. The toll station vehicle dynamic weighing and capture unit is used to acquire the information flow of vehicles passing through the toll station. The toll station vehicle information flow includes vehicle weight, vehicle type, color, and time series information. The bridge traffic flow video acquisition unit is used to acquire random traffic flow images on the bridge. The traffic flow group identification and tracking unit includes a vehicle recognition module based on the YOLO framework and a vehicle appearance feature matching and tracking module. The vehicle recognition module is used to calibrate and detect the bridge surface in real-time from the bridge surface traffic flow video acquisition unit, obtaining the spatiotemporal coordinates, vehicle type, and color information of the bridge surface traffic flow, and forming a multi-level associated vehicle particle group of the bridge surface traffic flow. The vehicle appearance feature matching and tracking module is used to perform multi-level cyclic matching between the toll station vehicle information stream obtained from the toll station vehicle dynamic weighing and capture unit and the multi-level associated vehicle particle group of the bridge surface traffic flow obtained from the vehicle recognition module, realizing the fusion of bridge surface vehicles and their corresponding weights, and realizing trajectory tracking, speed estimation, and time information acquisition of the bridge surface traffic flow. The traffic flow load spatiotemporal distribution information processing unit is used to process the driving trajectory, time, and speed information of the bridge surface traffic flow obtained from the vehicle appearance feature matching and tracking module and the vehicle weight information obtained from the toll station vehicle dynamic weighing and capture unit to obtain the spatiotemporal distribution information of the bridge surface traffic flow load.

[0044] The toll station vehicle dynamic weighing and capture unit includes a toll lane PC and vehicle separators, capture cameras, and weighing instruments, all connected to the toll lane PC. The vehicle separators and weighing instruments are bidirectionally connected. The vehicle separators are used to separate vehicles and provide start and end signals to the weighing instruments. The weighing instruments obtain vehicle load and axle count information by weighing axle weight. The capture cameras are used to acquire and record the vehicle type and color information of the corresponding vehicles. The toll lane PCs are used to summarize the weight, type, color, and time sequence information of all passing vehicles and form a toll station vehicle information stream.

[0045] The bridge traffic flow video acquisition unit includes an industrial camera array and a storage module and a communication module, both connected to the industrial camera array. The industrial camera array is used to acquire images of the bridge traffic flow, the storage module is used to store the images of the bridge traffic flow acquired by the industrial camera array, and the communication module is used to transmit the images of the bridge traffic flow acquired by the industrial camera array to the traffic flow identification and tracking subsystem in real time.

[0046] The vehicle recognition module includes a YOLO multimodal vehicle recognition model trained based on a VGG16 convolutional neural network and the PyTorch framework.

[0047] This invention also provides a computer vision-based method for monitoring the spatiotemporal distribution of bridge deck traffic loads, comprising the following steps:

[0048] Step S1: Obtain vehicle weight, vehicle type, color, and time sequence information of vehicles passing through the toll station, and form a toll station vehicle information stream; in this step, the toll station information stream X of the i-th vehicle... i Specifically:

[0049] X i =(x i1 ,x i2 ,x i3 ,x i4 )

[0050] Where, x i1 x i2 x i3 and x i4 These represent the vehicle model, color, weight, and time information of the i-th vehicle, respectively.

[0051] Step S21: Collect and transmit real-time images of random traffic flow across the entire bridge surface, and perform bridge surface calibration and detection on the real-time collected images to obtain the spatiotemporal coordinates, vehicle type, and color information of the random traffic flow on the bridge surface; in this step, the spatiotemporal distribution information stream y of the j-th vehicle on the bridge surface... j Specifically:

[0052] Y j =(y j1 ,y j2 ,yj3 ,y j4 )

[0053] Among them, y j1 y j2 y j3 and y j4 These represent the vehicle type, color, location, and time information of the j-th vehicle on the bridge.

[0054] Step S22: Based on the vehicle type and color information of the random traffic flow on the bridge surface with spatiotemporal distribution characteristics obtained in Step S21, calculate the Euclidean distance between ordinary particles and the core particle vehicle in the middle lane within the video frame. Take Euclidean distance The smallest n cars are considered as a swarm of traffic particles, and the Euclidean distance is considered. The average of the sums is used to obtain the swarm weight W of the n vehicles closest to the core particle vehicle in the middle lane. best n Specifically:

[0055]

[0056] and several different W best n A multi-level interconnected vehicle particle swarm is formed for traffic flow on the bridge surface.

[0057] Step S3: Perform multi-level cyclic matching on the toll station vehicle information flow obtained in step S1 and the multi-level associated vehicle particle swarm of the bridge surface random traffic flow obtained in step S22 to achieve the fusion of bridge surface vehicles and corresponding vehicle weights. Then, use the Deepsort multi-target tracking model to achieve trajectory tracking, speed estimation and time information acquisition of the bridge surface random traffic flow.

[0058] Step S4: The spatiotemporal distribution information of traffic flow, composed of trajectory and time information, is fused with the traffic flow moving load information, composed of vehicle weight, vehicle speed, and road surface unevenness, to obtain the spatiotemporal distribution of bridge deck traffic load.

[0059] In step S3, the vehicle information stream at the toll station is fused with the multi-level associated vehicle particle swarm and subjected to multi-level cyclic matching. This specifically includes the following steps:

[0060] Step S31: First, calculate y based on the Euclidean distance between the required monitoring area and the toll station. j4 The corresponding time confidence interval is used to perform ORB feature matching on vehicle images passing through the toll station and within the monitored area within the corresponding time period. This yields a matching similarity matrix A composed of a reference particle swarm of m vehicles at the toll station and a bridge surface vehicle particle swarm containing n vehicles. m*n , m>n, specifically:

[0061] a i1 =max{a 11 ,…,a m1},(i=1,2,…m)

[0062]

[0063] Among them, a mn This represents the ORB feature matching value between the m-th vehicle passing through the toll station and the n-th vehicle within the monitoring area of ​​the bridge surface, a. i1 Let Sum be the optimal matching value between the first vehicle in the bridge monitoring area and the m vehicle information streams at the toll station, and let A be the similarity matrix of this matching. m*n The sum of the optimal matching values ​​between n vehicles traveling on the bridge and m vehicle information streams at the toll station;

[0064] Step S32: Next, based on the continuous information flow Q of m vehicles passing through the toll station Xi m And the vehicle particle swarm W on the bridge surface within the monitored area best n x in both groups i1 With y j1 x i2 With y j2 Auxiliary judgment is performed, and when both are equal, it is considered that the individual matching of bridge traffic flow and toll station vehicles has been completed; when the overall matching accuracy reaches the set value, multi-level association matching of individual and group bridge traffic flow and toll station vehicles has been achieved.

[0065] Step S33: After obtaining a single Sum value, repeat steps S21, S22, S31 and S32. Using computer vision technology and ORB feature matching, perform multi-level cyclic matching between the bridge traffic flow and vehicles passing through the toll station to achieve multi-level association matching between the entire bridge traffic flow and vehicles at the toll station. Based on the vehicle weight obtained in step S1, complete the individual matching of vehicle-vehicle weight with the bridge traffic flow.

[0066] In step S4, the spatiotemporal distribution information of the traffic flow on the bridge is obtained by tracking the trajectories of individual vehicles and groups of random traffic flow on the bridge.

[0067] In step S4, based on the vehicle weight and speed of the corresponding vehicle, and combined with the road surface unevenness, the moving impact load caused by random traffic flow on the bridge surface is obtained.

[0068] The moving impact load P caused by random traffic flow on the bridge deck is specifically as follows:

[0069] P = M * f

[0070] Where M represents the weight of vehicles on the bridge deck, and f represents the dynamic random impact coefficient caused by road surface unevenness, etc.

[0071] The YOLO multimodal vehicle model is trained on an image dataset from real-world scenarios. This dataset contains images of various vehicle types and images of traffic flow changing over time within the same scene. Data augmentation was performed on the dataset before training, including image flipping, cropping, HSV adjustment, and padding. The training process involved creating the vehicle model dataset using the labelimg software and continuously adjusting the training parameters based on the training results to obtain the optimal weights. The dataset creation process included: acquiring and preprocessing the required real-world scene images; manually labeling the target pixels in each image; performing data augmentation on the labeled dataset; and dividing the dataset proportionally into training, validation, and test sets.

[0072] Example:

[0073] This embodiment applies the invention to a large bridge, which is 1100 meters long and 33.5 meters wide. The bridge is a six-lane, two-way highway with a design speed of 120 km / h. Industrial cameras used to capture random traffic flow images on the bridge are deployed as follows: Figure 2 As shown, the specific implementation process is as follows:

[0074] First, the vehicle weight and time sequence information of vehicles passing through the toll station are obtained through the vehicle dynamic weighing and capture unit at the toll station. Then, the vehicle type and color of the passing vehicles are identified and obtained using the set YOLO model, forming a vehicle information stream at the toll station. The data format is shown in List 1.

[0075] Table 1. Vehicle information recorded by the YOLO model at the toll station.

[0076] Work Team Entry Time Toll collector number Model color Weight (kg) [1] Day shift 2020-05-1012:55:51

[00031]

[16] Type 6 truck [0] Blue 44150 [1] Day shift 2020-05-1012:55:07

[00031]

[16] Type 6 truck [0] Blue 48900 [1] Day shift 2020-05-1012:54:44

[00031]

[16] Type 6 truck [0] Red 45850 [1] Day shift 2020-05-1012:54:15

[00031]

[16] Type 6 truck [0] Blue 43650 [1] Day shift 2020-05-1012:53:51

[00031]

[16] Type 6 truck [0] Huang 15650 [1] Day shift 2020-05-1012:53:39

[00031] [1] Type I passenger car [0] White 1280 [1] Day shift 2020-05-1012:53:09

[00031]

[16] Type 6 truck [0] Red 43850 [1] Day shift 2020-05-1012:52:50

[00031]

[16] Type 6 truck [0] Huang 45250 [1] Day shift 2020-05-1012:52:32

[00031]

[16] Type 6 truck [0] Blue 47200 [1] Day shift 2020-05-1012:52:15

[00031] [1] Type I passenger car [0] White 1350 [1] Day shift 2020-05-1012:51:57

[00031]

[16] Type 6 truck [0] Red 15000 [1] Day shift 2020-05-1012:51:38

[00031]

[16] Type 6 truck [0] Red 45750 [1] Day shift 2020-05-1012:51:19

[00031]

[16] Type 6 truck [0] Red 15150 [1] Day shift 2020-05-1012:50:57

[00031]

[16] Type 6 truck [0] Red 15600 [1] Day shift 2020-05-1012:50:36

[00031] [1] Type I passenger car [0] Blue 1300 [1] Day shift 2020-05-1012:50:17

[00031]

[16] Type 6 truck [0] Blue 15750 [1] Day shift 2020-05-1012:49:47

[00031]

[16] Type 6 truck [0] Blue 15550 [1] Day shift 2020-05-1012:49:21

[00031]

[13] Type III truck [0] Red 15050 [1] Day shift 2020-05-1012:48:53

[00031]

[13] Type III truck [0] Huang 14900 [1] Day shift 2020-05-1012:48:19

[00031]

[16] Type 6 truck [0] Red 14850 [1] Day shift 2020-05-1012:48:02

[00031]

[16] Type 6 truck [0] Red 46600

[0077] Secondly, the industrial camera array in the bridge traffic flow video acquisition unit acquires and transmits real-time images of random traffic flow across the entire bridge surface. Based on the YOLO multimodal vehicle recognition model trained using the VGG16 convolutional neural network and the PyTorch framework, the bridge surface is calibrated and detected using the random traffic flow images acquired by the bridge traffic flow video acquisition unit. The spatiotemporal coordinates, vehicle type, and color information of the random traffic flow on the bridge surface are obtained, as shown in Table 2.

[0078] Table 2. Vehicle information identified by camera No. 1 on the bridge surface.

[0079]

[0080] Subsequently, data fusion and multi-level cyclic matching were performed on the vehicle information flow at the toll station and the multi-level associated vehicle particle swarm. Vehicles in the middle (second lane) of the two-way bridge lanes were designated as core particles, while vehicles in the first and third lanes were designated as ordinary particles (see Table 2). In this case, the number of bridge particle swarms was defined as n=7. Table 2 shows 7 core particles. The Euclidean distance between each core particle and its six nearest ordinary particles was calculated, and the average of the six minimum values ​​was used as the weight of the particle swarm. For example, the minimum weight of the particle swarm in Table 2 calculated in this case was 27.3, and this particle swarm was selected as the matching target. Simultaneously, the database particle swarm parameters for the toll station were set to m=13. Core vehicles were determined in Table 1 based on the characteristics of the bridge core particles. The first six vehicles and the last six vehicles, totaling 13 vehicles, were selected to form the reference particle swarm for the database. Further, matching was performed using the ORB algorithm, with vehicle color and model information used for auxiliary judgment. The top 7 particles with the highest matching degree were selected as the matched vehicle information. The weight of each vehicle was further determined based on Table 1. Based on the DeepSort multi-object tracking model, trajectory tracking and speed estimation of bridge traffic flow are achieved, and the spatiotemporal distribution information of random traffic flow on the bridge is obtained as follows: Figure 3 As shown.

[0081] Ultimately, by fusing the spatiotemporal distribution information of traffic flow, composed of trajectory and time information, with the mobile load information of traffic flow, composed of vehicle weight, vehicle speed, and road surface unevenness, the spatiotemporal distribution of traffic flow load on the bridge deck is monitored. This information is then displayed in real-time on a display terminal, providing a schematic diagram of the spatiotemporal distribution of random traffic flow load on the bridge deck, as shown in the image. Figure 4 As shown.

[0082] As described in this embodiment, the computer vision-based method for monitoring the spatiotemporal distribution of bridge traffic loads is applicable to real-time monitoring and timely early warning of the spatiotemporal distribution of random traffic loads on bridge surfaces in various locations and environments. This invention utilizes a deep learning algorithm based on an improved YOLO framework to monitor the spatiotemporal distribution of random traffic loads on bridge surfaces. In practical applications, it exhibits adaptive and self-learning characteristics, contributing to further improvements in the intelligent monitoring of bridge traffic load distribution. This invention enables long-term, real-time monitoring of the distribution of random traffic loads on bridge surfaces, providing bridge managers with real-time data and early warning information, thus ensuring the safe operation of bridges.

[0083] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A computer vision-based system for monitoring the spatiotemporal distribution of bridge traffic loads, characterized in that: It includes a toll station vehicle dynamic weighing and capture unit, a bridge surface traffic flow video acquisition unit, a traffic flow group identification and tracking unit, and a traffic flow load spatiotemporal distribution information processing unit; the vehicle dynamic weighing and capture unit is connected to the traffic flow group identification and tracking unit and the traffic flow load spatiotemporal distribution information processing unit respectively, and the bridge surface traffic flow video acquisition unit, traffic flow group identification and tracking unit, and traffic flow load spatiotemporal distribution information processing unit are connected in sequence; The toll station vehicle dynamic weighing and capture unit is used to acquire the toll station vehicle information stream; the toll station vehicle information stream includes vehicle weight, vehicle type, color and time sequence information; the bridge surface traffic flow video acquisition unit is used to capture random traffic flow images on the bridge surface. The traffic flow identification and tracking unit includes a vehicle identification module based on the YOLO framework and a vehicle appearance feature matching and tracking module. The vehicle identification module performs bridge surface calibration and detection on the real-time image transmission of random traffic flow from the bridge surface video acquisition unit, obtaining the spatiotemporal coordinates, vehicle type, and color information of the random traffic flow, and forming a multi-level associated vehicle particle group. This multi-level associated vehicle particle group uses vehicles in the middle lane of the bridge surface as core particles and vehicles in surrounding lanes as ordinary particles. By calculating the Euclidean distance between ordinary particles and core particles, the n closest vehicles are selected to form the traffic flow particle group, based on particle group weight parameters. A hierarchical group of related vehicles; The vehicle appearance feature matching and tracking module is used to combine the toll station vehicle information stream obtained by the toll station vehicle dynamic weighing and capture unit with the multi-level associated vehicle particle group of the bridge surface random traffic flow obtained by the vehicle recognition module. Through a three-level cyclic matching process of spatiotemporal distance filtering, ORB feature matching, and vehicle attribute verification, it realizes the individual and group association between bridge surface vehicles and toll station vehicles, thereby completing the fusion of bridge surface vehicles and corresponding vehicle weights, and realizing trajectory tracking, speed estimation, and time information acquisition of the bridge surface random traffic flow. The traffic flow load spatiotemporal distribution information processing unit is used to process the driving trajectory, time, and speed information of the bridge surface random traffic flow obtained by the vehicle appearance feature matching and tracking module with the vehicle weight information obtained by the toll station vehicle dynamic weighing and capture unit to obtain the spatiotemporal distribution information of the bridge surface traffic flow load.

2. The bridge deck traffic load spatiotemporal distribution monitoring system based on computer vision according to claim 1, characterized in that, The toll station vehicle dynamic weighing and capture unit includes a toll lane PC and a vehicle separator, a capture camera, and a weighing instrument, all connected to the toll lane PC. The vehicle separator and the weighing instrument are bidirectionally connected. The vehicle separator is used to separate vehicles and provide start and end signals to the weighing instrument. The weighing instrument obtains vehicle load and axle number information by weighing axle weight. The capture camera is used to acquire and record the vehicle type and color information of the corresponding vehicle. The toll lane PC is used to summarize the weight, type, color, and time sequence information of all passing vehicles and form a toll station vehicle information stream.

3. The bridge deck traffic load spatiotemporal distribution monitoring system based on computer vision according to claim 1, characterized in that: The bridge traffic video acquisition unit includes an industrial camera array and a storage module and a communication module, both connected to the industrial camera array; the industrial camera array is used to acquire bridge traffic images, and the storage module is used to store the bridge traffic images acquired by the industrial camera array. The communication module is used to transmit the images of traffic flow on the bridge surface acquired by the industrial camera array to the traffic flow identification and tracking subsystem in real time.

4. The bridge deck traffic load spatiotemporal distribution monitoring system based on computer vision according to claim 1, characterized in that: The vehicle recognition module includes a YOLO multimodal vehicle recognition model trained based on a VGG16 convolutional neural network and the PyTorch framework.

5. A method for monitoring the spatiotemporal distribution of bridge deck traffic load based on computer vision, characterized in that, Includes the following steps: Step S1: Obtain vehicle weight, vehicle type, color, and time sequence information of vehicles passing through the toll station, and form a toll station vehicle information stream; in this step, the toll station information stream of the i-th vehicle... Specifically: ； in, , , and These represent the vehicle model, color, weight, and time information of the i-th vehicle, respectively. Step S21: Acquire and transmit real-time images of random traffic flow across the entire bridge surface, and perform bridge surface calibration and detection on the real-time acquired images to obtain the spatiotemporal coordinates, vehicle type, and color information of the random traffic flow on the bridge surface; in this step, the spatiotemporal distribution information stream of the j-th vehicle on the bridge surface... Specifically: ； in , , and These represent the vehicle type, color, location, and time information of the j-th vehicle on the bridge. Step S22: Based on the vehicle type and color information of the random traffic flow on the bridge surface with spatiotemporal distribution characteristics obtained in Step S21, calculate the Euclidean distance between ordinary particles and the core particle vehicle in the middle lane within the video frame. Take Euclidean distance The smallest n cars are considered as a swarm of traffic particles, and the distance between them is Euclidean. The average of the sums is used to obtain the particle swarm weights of the n vehicles closest to the core particle vehicle in the middle lane. Specifically: ； and with weights Based on the classification criteria, multiple traffic flow particle groups are divided into different levels to form a multi-level associated vehicle particle group of bridge traffic flow; Step S3: Perform multi-level cyclic matching on the toll station information flow obtained in step S1 and the multi-level associated vehicle particle group of the random traffic flow on the bridge obtained in step S22. Lock the candidate set by spatiotemporal distance filtering, filter high similarity vehicle pairs by ORB feature matching, and verify the matching results by vehicle attribute verification. Then, iterate and cycle to cover all vehicles on the bridge to complete the individual and group association between bridge vehicles and toll station vehicles, thereby realizing the fusion of bridge vehicles and corresponding vehicle weights. Then, use the Deepsort multi-target tracking model to realize trajectory tracking, speed estimation and time information acquisition of the random traffic flow on the bridge. Step S4: The spatiotemporal distribution information of traffic flow, composed of trajectory and time information, is fused with the traffic flow moving load information, composed of vehicle weight, vehicle speed, and road surface unevenness, to obtain the spatiotemporal distribution of bridge deck traffic load.

6. The method for monitoring the spatiotemporal distribution of bridge deck traffic load based on computer vision according to claim 5, characterized in that, In step S3, the toll station vehicle information stream is fused with the multi-level associated vehicle particle group and subjected to multi-level cyclic matching. This specifically includes the following steps: Step S31: First, calculate the distance between the required monitoring area and the toll station according to the Euclidean distance. The corresponding time confidence interval is used to perform ORB feature matching on vehicle images passing through the toll station and within the monitored area within the corresponding time period. This yields a matching similarity matrix composed of a reference particle swarm of m vehicles at the toll station and a bridge surface vehicle particle swarm containing n vehicles. , m>n, specifically: ； ； in, This represents the ORB feature matching value between the m-th vehicle passing through the toll station and the n-th vehicle within the monitoring area to be matched. This represents the optimal matching value between the first vehicle within the bridge monitoring area and the m vehicle information streams at the toll station. For this matching similarity matrix The sum of the optimal matching values ​​between n vehicles traveling on the bridge and m vehicle information streams at the toll station; Step S32: Next, based on the m consecutive vehicle information streams passing through the toll station and the vehicle particle swarm on the bridge surface within the monitored area In both groups and , and Auxiliary judgment is performed, and when both are equal, it is considered that the individual matching of bridge traffic flow and toll station vehicles has been completed; when the overall matching accuracy reaches the set value, multi-level association matching of individual and group bridge traffic flow and toll station vehicles has been achieved. Step S33: After obtaining a single Sum value, repeat steps S21, S22, S31 and S32. Using computer vision technology and ORB feature matching, perform multi-level cyclic matching between the bridge traffic flow and vehicles passing through the toll station to achieve multi-level association matching between the entire bridge traffic flow and vehicles at the toll station. Based on the vehicle weight obtained in step S1, complete the individual matching of vehicle-vehicle weight with the bridge traffic flow.

7. The method for monitoring the spatiotemporal distribution of bridge deck traffic load based on computer vision according to claim 5, characterized in that, In step S4, the spatiotemporal distribution information of the traffic flow on the bridge is obtained by tracking the trajectories of individual vehicles and groups of random traffic flow on the bridge.

8. The method for monitoring the spatiotemporal distribution of bridge deck traffic load based on computer vision according to claim 5, characterized in that, In step S4, the moving impact load caused by random traffic flow on the bridge is obtained based on the vehicle weight and speed of the corresponding vehicle and the road surface unevenness.

9. A method for monitoring the spatiotemporal distribution of bridge deck traffic load based on computer vision, as described in claim 8, is characterized in that... Moving impact loads caused by random traffic flow on the bridge surface Specifically: ； Where M represents the weight of vehicles on the bridge deck, and f represents the dynamic random impact coefficient caused by road surface unevenness.

Images