Overloaded and oversized vehicle warning and tracking system based on AI and machine vision dual drive

Abstract

This invention discloses an overloaded and oversized vehicle early warning and tracking system based on AI and machine vision. The system comprises a non-contact vehicle weighing subsystem and a vehicle tracking subsystem. The non-contact vehicle weighing subsystem includes a thermal imaging acquisition component, a tire model recognition module, a deformation parameter calculation unit, a load prediction model, and a temperature correction module. The vehicle tracking subsystem includes a video acquisition module, a vehicle tracking module, a speed detection module, a speed correction module, and a total load calculation module. This invention uses machine vision to achieve non-contact dynamic, continuous, and rapid measurement of vehicle loads without disrupting road traffic facilities or the original vehicle structure.

Description

Technical Field

[0001] The present invention belongs to the technical field of transportation infrastructure operation and maintenance, and specifically relates to an overloaded and oversized vehicle early warning and tracking system and method based on dual drive of AI and machine vision. Background Art

[0002] In recent years, with economic development and technological advancements, the modern transportation industry has experienced rapid growth. Simultaneously, efforts to improve traffic inspections, regulate overloading and over-limit vehicles, and implement weight-based toll collection have also deepened, leading to the increasing use of vehicle load measurement systems. These systems aim to address the problem of overloaded vehicles and the resulting road damage, frequent traffic accidents, and deteriorating transportation markets.

[0003] With the rapid development of transportation toward information technology and intelligent systems, there are currently two main methods for measuring vehicle loads: detached measurement and on-board measurement. Detached methods include truck scales and dynamic bridge weighing. Truck scales are equipped with sensors to capture axle load signals and calculate vehicle weight. However, truck scale measurement equipment requires disruption to existing road structures, making installation difficult. Dynamic bridge weighing calculates vehicle weight based on changes in the bridge's influence line. Overweight vehicles may damage bridge structures, increasing safety risks. On-board methods include image recognition, laser ranging, and strain sensing. Image recognition-based methods require markers on the side of the vehicle's leaf springs and image acquisition sensors installed on the vehicle's underbody. Signal transmission is difficult and the camera's operating conditions are relatively demanding. Laser ranging measures the deformation of the leaf springs to calculate vehicle load, offering high accuracy but also high cost and certain installation requirements. Strain sensing equipment measures vehicle load by generating changes in strain through the axle or the load on the axle. Sensors need to be installed on the vehicle frame, axle and other parts, and corresponding changes need to be made to the vehicle connection parts, which has a certain impact on the vehicle's safety performance. In addition, the sensor device is prone to damage to the sensor unit under long-term vehicle vibration.

[0004] Therefore, how to avoid damage to road traffic facilities or vehicles during vehicle load measurement, measure vehicle load quickly and accurately, and reduce the technical difficulty and cost of measurement has become an urgent problem that technicians in this field need to solve. Summary of the Invention

[0005] Technical problem to be solved: The present invention provides an overloaded and oversized vehicle early warning and tracking system and method based on dual-drive of AI and machine vision. It can realize non-contact dynamic, continuous and rapid measurement of vehicle load through machine vision without damaging road traffic facilities or the original structure of the vehicle.

[0006] Technical solution:

[0007] An AI- and machine vision-based overloaded and oversized vehicle early warning and tracking system, comprising a non-contact vehicle weighing subsystem and a vehicle tracking subsystem;

[0008] The non-contact vehicle weighing subsystem includes a thermal imaging acquisition component, a tire model recognition module, a deformation parameter calculation unit, a load prediction model and a temperature correction module;

[0009] The tire model recognition module is used to identify the model of each tire of the vehicle, and obtain the tire's cross-sectional size information and tire pressure information based on the identified model; the thermal imaging acquisition component is located on both sides of the lane, and is used to take infrared thermal images of the side of the vehicle's single-axle tire to obtain corresponding temperature matrix data; the deformation parameter calculation unit calculates the mechanical deformation parameters of the tire based on the temperature matrix data of the vehicle's single-axle tire; the load prediction model is obtained based on the XGBoost model training, and the estimated tire load F is calculated based on the tire's mechanical deformation parameters and the tire's cross-sectional size and tire pressure information. 1i The temperature correction module is used to adjust the vehicle wheelbase L, the number of vehicle axles n, the temperature of each tire t i Perform the test, combined with the tire temperature t i Estimated tire load F 1i Perform temperature correction to obtain the temperature-corrected tire load F 2i =F 1i ×α t , the temperature-corrected tire load F 2i , the number of vehicle axles n is packaged into tire load information and sent to the vehicle tracking subsystem; i = 1, 2, ..., n; F 1i 、 F 2i and t i are the estimated tire load, corrected tire load and temperature of the i-th tire respectively; the temperature correction factor α t The value of is obtained from statistical regression;

[0010] The vehicle tracking subsystem includes a video acquisition module, a vehicle tracking module, a speed detection module, a speed correction module, and a total load calculation module; the video acquisition module is installed directly above the lane and is used to capture the lane area video in real time; the vehicle tracking module is trained based on the YoloX-DeepSort model and is used to process the video images captured by the video acquisition module, identify and track the target vehicle in the video image, and send the tracked position information to the speed detection module, which calculates the vehicle speed v and marks the calculated vehicle speed v on the target vehicle in the video image; the speed correction module combines the vehicle speed v to calculate the temperature-corrected tire load F 2iPerform speed correction to obtain the final single-axle tire load F i =F 2i ×β v The total load calculation module is based on the single-axle tire load F i Calculate the overall load Q of the target vehicle; speed correction factor β v The value of is obtained from statistical regression.

[0011] Furthermore, the vehicle tracking subsystem includes an early warning module; the early warning module obtains the load threshold of the target vehicle based on the vehicle wheelbase L and the number of vehicle axles n of the target vehicle, compares the overall load Q of the target vehicle with the load threshold, and determines whether the target vehicle is overloaded or exceeds the limit.

[0012] Furthermore, the tire model recognition module includes an optical image acquisition component, an OCR text recognition model, and a tire information query module;

[0013] The optical image acquisition components are located on both sides of the lane and are used to capture optical images of vehicle tires. The OCR text recognition model recognizes the character information corresponding to the tires; the tire information query module obtains the tire model, as well as the cross-sectional dimensions and tire pressure information corresponding to the tire model based on the character information corresponding to the tire.

[0014] Furthermore, the deformation parameter calculation unit processes the pixel temperature matrix data based on the OpenCV image processing algorithm, and obtains the mechanical deformation parameters of the target sample tire through image iterative geometric fitting and region growing algorithm detection; the mechanical deformation parameters include: the maximum pixel radius R of the tire, the maximum pixel area S1 of the tire, the pixel radius r of the wheel hub, the pixel length 1 of the tire contact with the ground, the pixel distance h from the tire center to the ground, the equivalent pixel area S2 of the tire after deformation, the image pixel area difference ΔS before and after the tire deformation, and the pixel length L of the tire contact dividing line;

[0015] The process of the deformation parameter calculation unit detecting and obtaining the mechanical deformation parameters of the target sample tire includes the following steps:

[0016] S11, generating a temperature image linearly related to temperature based on the pixel temperature matrix data; using the Sobel edge detection operator to calculate the pixel gradient amplitude of the temperature image, and performing image segmentation around the pixel points in the temperature image whose temperature difference is greater than a preset temperature difference threshold;

[0017] S12: retain the gradient amplitudes of the first 5% of the segmented image, extract the points with the largest gradient amplitudes, and color-mark them; sequentially select the color-marked pixels from the bottom of the image upwards, and use the selected pixels as seed points for the first tire contour fitting. The selected seed points are all pixels at the interface between the tire and the air;

[0018] S13, using the seed point selected in step S12, performing region growing on the seed point using a region growing algorithm to find a point adjacent to the seed point as a new seed point, and performing a second fitting of the tire outer contour;

[0019] S14, retaining the seed points selected in steps S12 and S13, and using the second fitted tire contour as a reference, finding the pixel point with the maximum gradient amplitude in the upper half of the tire from top to bottom, and performing a third tire contour fitting;

[0020] S15, using the tire outer contour fitted in step S14 as a reference, searching for a pixel point with the maximum gradient amplitude toward the tire center, using this pixel point as a seed point for fitting the wheel hub outer contour, and fitting the wheel hub outer contour;

[0021] S16, repeating the iteration to obtain the tire outer profile and the wheel hub outer profile that meet the preset error standard;

[0022] S17, finding a pixel gradient amplitude point at the intersection of the tire and the ground within an angle range of 45 degrees to the lower left and 45 degrees to the lower right of the tire center, and drawing a tire-ground dividing line based on the found pixel gradient amplitude point to obtain the interface between the tire and the ground after the tire is deformed;

[0023] S18, endpoint processing is performed on the image pixels at the interface between the tire and the ground from bottom to top, and the Y coordinate difference between two adjacent pixel points is calculated. When the Y coordinate difference between any pixel point and its adjacent pixel point is greater than a preset coordinate difference threshold, the pixel point is judged to be the endpoint of the tire-ground contact, and the actual contact pixel length between the tire and the ground is obtained.

[0024] Furthermore, the deformation parameter calculation unit uses a proportional factor α to correct the obtained mechanical deformation parameters of the tire:

[0025]

[0026] Where Rim is the wheel radius; r is the pixel radius of the wheel obtained by fitting.

[0027] Furthermore, the training process of the load prediction model includes:

[0028] S1, using a thermal imaging acquisition component to capture side thermal imaging images of a certain number of sample tires with different tire cross-sectional sizes, different tire pressures, different loads, and different temperatures under normal working conditions, to generate a sample tire set;

[0029] S2, extracting pixel temperature matrix data from each side thermal imaging image in the sample tire set to obtain surface temperature information of each sample tire; processing the pixel temperature matrix data of each sample tire using a deformation parameter calculation unit to obtain mechanical deformation parameters of the sample tire;

[0030] S3, taking the mechanical deformation parameters of each sample tire and the corresponding tire size and air pressure information as a group of training samples to generate a training sample set;

[0031] S4, a load prediction model is constructed based on the XGBoost model, and the load prediction model is trained using the training sample set in step S3; the trained load prediction model is used to process the imported mechanical deformation parameters, tire size, and air pressure information of the tire to be tested, and calculate the estimated tire load of the tire to be tested.

[0032] Furthermore, the non-contact vehicle weighing subsystem includes a model updating device; the model updating device is used to import new sample data into the load prediction model to update the load prediction model.

[0033] Furthermore, the vehicle tracking module is trained based on the YoloX-DeepSort model, extracts the pixel position of the lower midpoint of the rectangle of the wheel prediction box as the contact point between the target vehicle and the road surface, and calculates the actual position of the target vehicle through the shooting angle, focal length, position of the video acquisition module and the contact point position of the video acquisition module;

[0034] During the tracking process, the speed detection module records the timestamp of the target vehicle tire leaving the preset virtual detection area, and uses the vehicle wheelbase L and the time difference between the target vehicle leaving the virtual detection area to calculate the average speed of the target vehicle as the vehicle speed v of the target vehicle in the virtual detection area.

[0035] A method for early warning and tracking overloaded and oversized vehicles based on dual AI and machine vision is characterized in that the method is executed based on the aforementioned overloaded and oversized vehicle early warning and tracking system; the method comprises the following steps:

[0036] Identify the model of each tire of the vehicle entering the detection area, and obtain the tire cross-sectional size information and tire pressure information based on the identified model;

[0037] Collect infrared thermal images of the side of the vehicle's single-axle tire to obtain corresponding temperature matrix data; calculate the mechanical deformation parameters of the tire based on the temperature matrix data of the vehicle's single-axle tire;

[0038] According to the mechanical deformation parameters of the tire, the cross-sectional dimensions of the tire, and the tire pressure information, the estimated tire load F is calculated. 1i; Combined with tire temperature t i Estimated tire load F 1i Perform temperature correction to obtain the temperature-corrected tire load F 2i =F 1i ×α t ;

[0039] Real-time video capture of the lane area is performed, the video images captured by the video acquisition module are processed, the target vehicle in the video images is identified and tracked, and the tracked position information is sent to the speed detection module, which calculates the vehicle speed v and marks the calculated vehicle speed v on the target vehicle in the video image;

[0040] Tire load F after temperature correction combined with vehicle speed v 2i Perform speed correction to obtain the final single-axle tire load F i =F 2i ×β v ; According to the single axle tire load F i The overall load Q of the target vehicle is calculated.

[0041] Furthermore, the overloaded and oversized vehicle early warning and tracking method further includes:

[0042] According to the wheelbase L and the number of axles n of the target vehicle, the load threshold of the target vehicle is obtained by querying, and the overall load Q of the target vehicle is compared with the load threshold to determine whether the target vehicle is overloaded or exceeds the limit.

[0043] Beneficial effects:

[0044] (1) The image measurement method used in the overloaded and oversized vehicle early warning and tracking system based on dual drive of AI and machine vision of the present invention is extremely convenient to arrange compared with other methods. It adopts a non-contact method, does not cause any potential damage to the vehicle structure, and does not add any landmark identification objects to the vehicle body. Similarly, this method does not cause any destructive reconstruction to the main structure of the road. The detection equipment using this method has good mobility and does not need to fix the camera to a certain structure of the vehicle body, which reduces the difficulty of data transmission of the vehicle body sensor unit, weakens the impact of road conditions and environment, and is extremely convenient for the maintenance of the acquisition equipment.

[0045] (2) The present invention's overloaded and oversized vehicle early warning and tracking system, based on dual AI and machine vision drivers, captures the vehicle's outline. It only requires a high-speed thermal imaging camera installed outside the lane. The equipment is easy to set up, requiring a small number of cameras. The cameras can be replaced based on the accuracy and actual needs of the present invention, rationally maximizing the benefits of the needs. The newly installed camera only needs to perform simple distortion correction to quickly complete vehicle weight measurement, and it has strong mobility and reproducibility.

[0046] (3) The present invention's overloaded and oversized vehicle early warning and tracking system, based on dual AI and machine vision drivers, combines wheel load prediction with a deep learning visual tracking algorithm to achieve real-time tracking of vehicle loads. This system has important guiding significance for the early stages of bridge operation and maintenance, resolving the problem that traditional methods can only calculate bridge responses by assuming vehicle loads. The present invention's real-time acquisition and tracking method for vehicle loads provides an efficient and economical method for calculating vehicle load statistics during long-term bridge operation and maintenance.

[0047] (4) The present invention's overloaded and oversized vehicle early warning and tracking system, based on dual-drive AI and machine vision, uses a high-speed thermal imager to photograph vehicle tires. By switching the thermal imager's shooting mode (normal camera mode and thermal imaging mode), it can detect vehicle loads during the day or at night, and can also achieve dynamic and continuous detection of multiple vehicle loads simultaneously. Compared with traditional methods, the present invention has higher vehicle load detection efficiency and a wider detection period. At the same time, the present invention is less affected by external conditions such as vehicle speed and light, and is of great significance for promoting intelligent and unmanned operation and maintenance management of bridges. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a schematic diagram of the overloaded and oversized vehicle early warning and tracking system based on dual drive of AI and machine vision in a preferred embodiment of the present invention.

[0049] Figure 2 Schematic diagram of the coordinated arrangement of a high-speed thermal imager (thermal imaging acquisition component) and a camera (video acquisition module) according to a preferred embodiment of the present invention.

[0050] Figure 3 This is a schematic diagram of an implementation method for collaboratively arranging a high-speed thermal imager (thermal imaging acquisition component) and a camera (video acquisition module) according to a preferred embodiment of the present invention (arranged on a bridge).

[0051] Figure 4 This is a schematic diagram of the application of a high-speed thermal imager (thermal imaging acquisition component) according to a preferred embodiment of the present invention (in night mode).

[0052] Figure 5 This is a flow chart of the early warning principle of a preferred embodiment of the present invention.

[0053] The accompanying drawings are marked as follows: 1. Thermal imaging acquisition component; 2. Video acquisition module; 3. Data processing equipment (related program carrier of the overload and over-limit vehicle warning and tracking system); 4. Label applied to the target vehicle; 5. Tire deformation contour area under the high-speed thermal imager; 6. Tire image area taken by the high-speed thermal imager at night; 7. Tire temperature display area under the high-speed thermal imager. DETAILED DESCRIPTION

[0054] The following examples may enable those skilled in the art to more fully understand the present invention, but are not intended to limit the present invention in any way.

[0055] Figure 1 This is a schematic diagram of the overloaded and oversized vehicle early warning and tracking system based on AI and machine vision dual drive according to the preferred embodiment of the present invention. Figure 1 This embodiment discloses an overloaded and oversized vehicle warning and tracking system based on dual-drive of AI and machine vision. The overloaded and oversized vehicle warning and tracking system includes a non-contact vehicle weighing subsystem and a vehicle tracking subsystem.

[0056] The non-contact vehicle weighing subsystem includes a thermal imaging acquisition component 1, a tire model recognition module, a deformation parameter calculation unit, a load prediction model and a temperature correction module.

[0057] The tire model recognition module is used to identify the model of each tire of the vehicle, and obtain the tire's cross-sectional size information and tire pressure information based on the identified model; the thermal imaging acquisition component is located on both sides of the lane, and is used to take infrared thermal images of the side of the vehicle's single-axle tire to obtain corresponding temperature matrix data; the deformation parameter calculation unit calculates the mechanical deformation parameters of the tire based on the temperature matrix data of the vehicle's single-axle tire; the load prediction model is obtained based on the XGBoost model training, and the estimated tire load F is calculated based on the tire's mechanical deformation parameters and the tire's cross-sectional size and tire pressure information. 1i The temperature correction module is used to adjust the vehicle wheelbase L, the number of vehicle axles n, the temperature of each tire t i Perform the test, combined with the tire temperature t i Estimated tire load F 1i Perform temperature correction to obtain the temperature-corrected tire load F 2i =F 1i ×α t , the temperature-corrected tire load F 2i , the number of vehicle axles n is packaged into tire load information and sent to the vehicle tracking subsystem; i = 1, 2, ..., n; F 1i 、 F 2i and t i are the estimated tire load, corrected tire load and temperature of the i-th tire respectively; the temperature correction factor α t The value of is obtained from statistical regression.

[0058] The vehicle tracking subsystem includes a video acquisition module 2, a vehicle tracking module, a speed detection module, a speed correction module, and a total load calculation module; the video acquisition module is set up directly above the lane and is used to capture the lane area video in real time; the vehicle tracking module is trained based on the YoloX-DeepSort model and is used to process the video images captured by the video acquisition module, identify and track the target vehicle in the video image, and send the tracked position information to the speed detection module, which calculates the vehicle speed v and marks the calculated vehicle speed v on the target vehicle in the video image; the speed correction module combines the vehicle speed v to calculate the temperature-corrected tire load F 2i Perform speed correction to obtain the final single-axle tire load F i =F 2i ×β v The total load calculation module is based on the single-axle tire load F i Calculate the overall load Q of the target vehicle; speed correction factor β v The value of is obtained from statistical regression.

[0059] In this embodiment, except for the thermal imaging acquisition component 1 , the video acquisition module 2 and the optical image acquisition component, each module is a software program loaded into the data processing device 3 . Figure 2 and Figure 3 This is a schematic diagram of two system deployment methods, corresponding to toll stations and bridges respectively.

[0060] (1) Non-contact vehicle weighing subsystem (system A)

[0061] The tire model recognition module includes an optical image acquisition component, an OCR text recognition model, and a tire information query module; the optical image acquisition component is located on both sides of the lane and is used to capture optical images of the vehicle tires, and the OCR text recognition model recognizes the character information corresponding to the tires; the tire information query module obtains the tire model, as well as the cross-sectional dimensions and tire pressure information corresponding to the tire model based on the character information corresponding to the tire.

[0062] See also Figure 4 High-speed thermal imagers located on both sides of the lane are used to record the temperature information of the vehicle's single-axle tires. The mechanical deformation parameters of the single-axle tires are calculated based on the temperature matrix data of the vehicle's single-axle tires. The load prediction model obtained by training the XGBoost model is used to import the calculated mechanical deformation parameters of the tires, tire size, and tire pressure information into the load prediction model to calculate the load F of each tire. 1iThe number of high-speed thermal imagers deployed depends on the required accuracy of vehicle load calculations. They can be located on just one lane or on both sides. If high accuracy is required, they can be deployed on both sides of the lane to estimate the load for each tire. If low accuracy is required, they can be deployed on just one lane to infer the tire load on the other side based on the load calculation results on one side.

[0063] In this embodiment, the thermal imaging acquisition component 1 can be a high-resolution, high-frame-rate thermal imaging device with a resolution greater than or equal to 640*480. In this embodiment, a K26HE25 high-speed, high-frame-rate infrared thermal imaging acquisition component is used. The optical image acquisition component uses a Nikon D5600 SLR camera.

[0064] The deformation parameter calculation unit processes pixel temperature matrix data based on the OpenCV image processing algorithm and detects the mechanical deformation parameters of the target sample tire through image iterative geometric fitting and region growing algorithms. The mechanical deformation parameters include: tire maximum pixel radius R, tire maximum pixel area S1, wheel hub pixel radius r, tire-ground contact pixel length 1, tire center-to-ground pixel distance h, equivalent pixel area S2 after tire deformation, image pixel area difference ΔS before and after tire deformation, and tire-ground contact dividing line pixel length L. Preferably, the SG-Net algorithm can also be used to perform image segmentation on the tire image to separate the tire from the background, locate the tire-ground contact point, and then calculate and count the relevant deformation parameters.

[0065] In this implementation, a high-speed thermal imager captures thermal images of the tires on one side of the target vehicle as the vehicle slowly passes by, generating a raw CSV temperature data file. The temperature information corresponding to each pixel in the CSV temperature data file is extracted and redrawn into a temperature image based on a linear relationship. The Sobel operator is then used to perform edge detection on the temperature image, calculating the temperature gradient amplitude within the image. The point with the maximum gradient amplitude is found, and the top 5% of pixels are retained and color-coded.

[0066] Based on prior knowledge, the point with the maximum gradient amplitude at the contact surface between the lower half of the tire and the air most closely represents the tire's edge contour. Edge detection points at the tire-air interface are sequentially selected from the bottom of the image upward as seed points for fitting the tire's outer contour circle. The tire's outer contour is fitted using the first selected seed points, and a region growing operation is performed on these first selected seed points. A second seed point selection is performed to continue fitting the tire's outer contour. A third seed point selection is performed from the top down, further fitting the tire's outer contour. Using the tire's outer contour as the boundary, temperature gradient amplitude points are sequentially searched toward the center of the circle. These points are used as the edge pixels of the wheel hub for fitting the wheel hub's outer contour. Find the pixel gradient amplitude point at the junction of the tire and the ground within 45 degrees to the lower left and lower right of the tire center, use this point as a reference to make a dividing line between the tire and the ground, and obtain the interface between the tire and the ground after the tire is deformed. Perform endpoint processing on the image pixels of the interface between the tire and the ground from bottom to top, calculate the Y coordinate difference between two adjacent pixel points, and when the difference is greater than a certain threshold, judge that the point is the endpoint of the tire's contact with the ground, and obtain the actual contact pixel length between the tire and the ground. From then on, the outer contour edge segmentation of the tire and the wheel hub based on thermal imaging data is completed, and the following parameters are obtained: tire maximum pixel radius R, tire maximum pixel area S1, wheel hub pixel radius r, tire and ground contact pixel length l, pixel distance h from the tire center to the ground, equivalent pixel area S2 after tire deformation, image pixel area difference ΔS before and after tire deformation, and tire and ground contact dividing line pixel length L. Calculate the average temperature of the tire outer surface based on the thermal information captured by thermal imaging. like Figure 5 shown.

[0067] The optical image of the tire is collected by the optical acquisition component, and then the OCR recognition algorithm is called. The OCR recognition algorithm adopts character recognition technology based on deep learning. The PSENET character positioning network and the CRNN character recognition network are trained by the transfer learning method to form a two-stage character recognition method. The identifier information of the tire sidewall is used to obtain the tire size information (tire section height H, tire section width b, wheel hub radius Rim) and air pressure information atm.

[0068] According to the wheel size information, various deformation parameters measured in thermal imaging are corrected in size, and the obtained mechanical deformation parameters of the tire are corrected using the proportional factor α: The corrected mechanical deformation parameters are:

[0069] r true =α×r

[0070] R true =α×R

[0071] S 1true =α 2 ×S

[0072] l true =α×l

[0073] h true =α×h

[0074] S 2true =α 2 ×S2

[0075] ΔS true =α 2 ×ΔS

[0076] L true =α×L

[0077] Where r true 、R true 、S 1true 、l true 、h true 、S 2true , ΔS true and L true The following are the corrected wheel hub pixel radius, tire maximum pixel radius, tire maximum pixel area, tire-ground contact pixel length, tire center-to-ground pixel distance, tire equivalent pixel area after deformation, image pixel area difference before and after tire deformation, and tire-ground contact dividing line pixel length, respectively. R, S1, l, h, S2, ΔS, and L are the fitted tire maximum pixel radius, tire maximum pixel area, tire-ground contact pixel length, tire center-to-ground pixel distance, tire equivalent pixel area after deformation, image pixel area difference before and after tire deformation, and tire-ground contact dividing line pixel length, respectively. Specifically, as a vehicle slowly passes by, the thermal imaging acquisition assembly 1 captures and records the outer contour of the tire on one side of the vehicle body. The tire hub area and tire area are detected to obtain pixel points in the hub area and the tire area. An image scale factor is calculated based on the pixel points in the hub area and the wheel hub diameter (the hub area does not deform). The tire deformation is calculated based on the pixel points in the tire area and the image scale factor.

[0078] The XGboost model was trained based on the dataset collected from the indoor test to obtain a load prediction model. The 12 mechanical characteristics measured above were input into the load prediction model to obtain the predicted load of a single tire. Field verification showed that the error of this method was less than 5%, indicating good prediction results. Specifically, the training process of the load prediction model includes:

[0079] S1: Use thermal imaging acquisition component 1 to capture side thermal images of a certain number of sample tires with different tire cross-sectional dimensions, different tire pressures, different loads, and different temperatures under normal operating conditions to generate a sample tire set.

[0080] S2, extracting pixel temperature matrix data from each side thermal imaging image in the sample tire set to obtain surface temperature information of each sample tire; using a deformation parameter calculation unit to process the pixel temperature matrix data of each sample tire to obtain the mechanical deformation parameters of the sample tire.

[0081] S3: The mechanical deformation parameters (12 mechanical characteristics) of each sample tire and the corresponding tire size and air pressure information are used as a group of training samples to generate a training sample set.

[0082] S4, a load prediction model is constructed based on the XGBoost model, and the load prediction model is trained using the training sample set in step S3; the trained load prediction model is used to process the imported mechanical deformation parameters, tire size, and air pressure information of the tire to be tested, and calculate the estimated tire load of the tire to be tested.

[0083] During the training process, the mean square error (MSE) was used as the evaluation indicator, which showed good predictive performance in the finite element training results. The machine learning interpretable method shap was used to perform global and local interpretations between the various tire deformation parameters, find the most important parameters for predicting the tire model, and discard redundant parameters. During the model training process, the Bayesian parameter search method was used to optimize the XGBoost model to obtain the best predictive performance.

[0084] Furthermore, the non-contact vehicle weighing subsystem includes a model updating device; the model updating device is used to import new sample data into the load prediction model to update the load prediction model.

[0085] If this implementation scheme is capable of measuring the actual tire load, the tire load measured by third-party equipment can be combined with the 12 mechanical characteristics proposed in this embodiment to construct a data set for updating the model, realize the automatic update and self-learning mechanism of the model, and improve the universal capability of the load prediction model.

[0086] Since the tire generates heat through friction with the ground during driving, and the temperature of the tire changes under the influence of ambient temperature, this temperature change will cause the load F that the tire bears to be predicted based on tire deformation parameters alone. 1i There is an error between the actual load borne by the tire, so this embodiment introduces a temperature correction coefficient α to address this phenomenon. t The predicted value is corrected to make it more accurate. The tire load F after temperature correction2i =F 1i ×α t , α t is the temperature correction factor.

[0087] (2) Vehicle Tracking Subsystem (System B)

[0088] First, a camera 2 mounted high above captures images of the vehicle in motion. After camera calibration, the vehicle's speed v is detected using the YoloX-Deepsort network. Specifically, the vehicle tracking module, trained based on the YoloX-DeepSort model, extracts the pixel position of the lower midpoint of the rectangular wheel prediction frame as the contact point between the target vehicle and the road surface. The actual position of the target vehicle is calculated using the video acquisition module's shooting angle, focal length, position of the video acquisition module, and the contact point. During the tracking process, the speed detection module records the timestamp of the target vehicle's tire leaving the preset virtual detection zone. The average speed of the target vehicle is calculated using the vehicle wheelbase L and the time difference between the target vehicle's departure from the virtual detection zone, which serves as the target vehicle's speed v within the virtual detection zone.

[0089] The deformation of a moving car tire is not only related to the load and temperature it bears, but also to the speed of the car. The faster the car travels, the greater the tire deformation. Therefore, in order to fully and accurately predict the load borne by the tire, this embodiment also proposes a speed correction coefficient β v After the speed correction factor is corrected, a more accurate single-axle tire load F is obtained. i =F2×β v , β v The speed correction coefficient β is obtained by regression learning a large number of tire images at different driving speeds. v , and finally achieve speed correction.

[0090] Since the thermal imager is only deployed on one side of the vehicle, assuming that the number of axles on one side of the vehicle is n, the tire loads of the n adjacent axles constitute the overall vehicle load Q with a speed tag of v. The overall vehicle load Q is calculated as follows:

[0091] See also Figure 5 After predicting the loads of all vehicles within the system's detection range, the system can quickly determine and issue warnings in real time based on pre-set overload and overlimit thresholds. This embodiment can simultaneously monitor whether a specific vehicle's load exceeds a threshold, as well as whether the total load of vehicles within a certain range exceeds a threshold. Furthermore, this embodiment provides real-time and rapid warnings, which are of great significance to the safe operation of infrastructure such as bridges.

[0092] In this embodiment, this embodiment also discloses an overloaded and oversized vehicle early warning tracking based on dual-drive of AI and machine vision, including the following steps:

[0093] Step 1: Use a high-speed thermal imager to capture images of various tire models at different temperatures, pressures, and loads. Use the SG-Net algorithm to segment these images, separating the tire from the surrounding background to obtain images of the deformed tire. Use an optical character recognition (OCR) algorithm to identify text on the tire to determine tire pressure, dimensions, and other information. Normalize the images to obtain data about the deformed tire. This data is then loaded into the XGBoost machine learning prediction model to obtain predicted tire loads. Train the XGBoost machine learning model with a large amount of data to ensure that its predictions meet the expected accuracy standards.

[0094] Step 2: Integrate the SG-Net algorithm, OCR text recognition algorithm, XGBoost machine learning prediction model, and high-speed thermal imager in step 1 into a non-contact vehicle weighing subsystem (system A).

[0095] Step 3: Place a high-speed thermal imager next to a single lane such as a bridge or toll booth to capture images of the vehicle's tires while in motion. Simultaneously, record the number of axles and tire temperatures of the vehicle. Load the images and tire temperatures into the algorithms and machine learning prediction model to obtain the vehicle's single axle load.

[0096] Step 4: Build a vehicle tracking model based on the YoloX-Deepsort target tracking algorithm and integrate the tracking model with a camera to form a vehicle tracking subsystem (system B).

[0097] Step 5: Place a camera on an elevated road above the entrance of a bridge, toll booth, or other similar location, continuously capture images of vehicles below, and load the images into the vehicle tracking model based on the YoloX-Deepsort target tracking algorithm to obtain the vehicle's speed.

[0098] Step 6: The vehicle tracking subsystem (system B) receives the vehicle axle load and the number of axles from the non-contact vehicle weighing subsystem (system A). Combined with the vehicle speed measured in step 5, the vehicle axle load is speed-corrected and the number of axles is taken into account to arrive at the final vehicle axle load.

[0099] Step 7: Install the camera in the vehicle tracking subsystem (system B) on an elevated platform above the lane at the entrance of a toll booth or bridge. The camera captures images of vehicles traveling in the lane below and transmits them to the YoloX-Deepsort target tracking algorithm model in the vehicle tracking subsystem (system B). The model processes and calculates the image to determine the speed of the target vehicle and assigns a label to the target vehicle.

[0100] Step 8: The vehicle tracking subsystem (system B) receives the vehicle axle load and the number of axles from system A. The vehicle tracking subsystem (system B) calculates the overall load of the vehicle by integrating the vehicle axle load, the number of axles, and the target vehicle's speed.

[0101] Step 9: The vehicle tracking subsystem (system B) continuously detects the target vehicle's speed and receives the vehicle load information transmitted by the non-contact vehicle weighing subsystem (system A), thereby achieving dynamic real-time tracking of the target vehicle's load.

[0102] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that various improvements and modifications that do not depart from the principles of the present invention are considered to be within the scope of protection of this application.

Claims

1. An overloaded and oversized vehicle early warning and tracking system based on AI and machine vision dual drive, characterized by: The overloaded and oversized vehicle early warning and tracking system includes a non-contact vehicle weighing subsystem and a vehicle tracking subsystem; The non-contact vehicle weighing subsystem includes a thermal imaging acquisition component, a tire model recognition module, a deformation parameter calculation unit, a load prediction model and a temperature correction module; The tire model recognition module is used to identify the model of each tire of the vehicle and obtain the tire cross-sectional size information and tire pressure information based on the identified model. The thermal imaging acquisition component is located on both sides of the lane and is used to capture infrared thermal images of the side of the vehicle's single-axle tire to obtain corresponding temperature matrix data. The deformation parameter calculation unit calculates the mechanical deformation parameters of the tire based on the temperature matrix data of the vehicle single-axle tire; the load prediction model is obtained based on the XGBoost model training, and the estimated tire load F is calculated based on the mechanical deformation parameters of the tire and the tire cross-sectional dimensions and tire pressure information. 1i The temperature correction module is used to adjust the vehicle wheelbase L, the number of vehicle axles n, the temperature of each tire t i Perform the test, combined with the tire temperature t i Estimated tire load F 1i Perform temperature correction to obtain the temperature-corrected tire load F 2i =F 1i ×α t , the temperature-corrected tire load F 2i , the number of vehicle axles n is packaged into tire load information and sent to the vehicle tracking subsystem; i = 1, 2, ..., n; F 1i 、F 2i and t i are the estimated tire load, corrected tire load and temperature of the i-th tire respectively; Temperature correction factor α t The value of is obtained from statistical regression; The vehicle tracking subsystem includes a video acquisition module, a vehicle tracking module, a speed detection module, a speed correction module, and a total load calculation module; the video acquisition module is installed directly above the lane and is used to capture the lane area video in real time; the vehicle tracking module is trained based on the YoloX-DeepSort model and is used to process the video images captured by the video acquisition module, identify and track the target vehicle in the video image, and send the tracked position information to the speed detection module, which calculates the vehicle speed v and marks the calculated vehicle speed v on the target vehicle in the video image; the speed correction module combines the vehicle speed v to calculate the temperature-corrected tire load F 2i Perform speed correction to obtain the final single-axle tire load F i =F 2i ×β v The total load calculation module is based on the single-axle tire load F i Calculate the overall load Q of the target vehicle; speed correction factor β v The value of is obtained from statistical regression.

2. The overloaded and oversized vehicle early warning and tracking system based on AI and machine vision dual drive according to claim 1 is characterized in that: The vehicle tracking subsystem includes an early warning module; the early warning module obtains a load threshold of the target vehicle based on the target vehicle's wheelbase L and the number of vehicle axles n, compares the target vehicle's overall load Q with the load threshold, and determines whether the target vehicle is overloaded or out of limit.

3. The overloaded and oversized vehicle early warning and tracking system based on AI and machine vision dual drive according to claim 1 is characterized in that: The tire model recognition module includes an optical image acquisition component, an OCR text recognition model, and a tire information query module; The optical image acquisition components are located on both sides of the lane and are used to capture optical images of vehicle tires. The OCR text recognition model recognizes the character information corresponding to the tires; the tire information query module obtains the tire model, as well as the cross-sectional dimensions and tire pressure information corresponding to the tire model based on the character information corresponding to the tire.

4. The overloaded and oversized vehicle early warning and tracking system based on dual drive of AI and machine vision according to claim 1 is characterized in that: The deformation parameter calculation unit processes the pixel temperature matrix data based on the OpenCV image processing algorithm, and obtains the mechanical deformation parameters of the target sample tire through image iterative geometric fitting and region growing algorithm detection; The mechanical deformation parameters include: tire maximum pixel radius R, tire maximum pixel area S1, wheel hub pixel radius r, tire-ground contact pixel length l, tire center-to-ground pixel distance h, tire equivalent pixel area S2 after deformation, image pixel area difference ΔS before and after tire deformation, and tire-ground contact dividing line pixel length L; The process of the deformation parameter calculation unit detecting and obtaining the mechanical deformation parameters of the target sample tire includes the following steps: S11, generating a temperature image linearly related to temperature based on the pixel temperature matrix data; using the Sobel edge detection operator to calculate the pixel gradient amplitude of the temperature image, and performing image segmentation around the pixel points in the temperature image whose temperature difference is greater than a preset temperature difference threshold; S12: retain the gradient amplitudes of the first 5% of the segmented image, extract the points with the largest gradient amplitudes, and color-mark them; sequentially select the color-marked pixels from the bottom of the image upwards, and use the selected pixels as seed points for the first tire contour fitting. The selected seed points are all pixels at the interface between the tire and the air; S13, using the seed point selected in step S12, performing region growing on the seed point using a region growing algorithm to find a point adjacent to the seed point as a new seed point, and performing a second fitting of the tire outer contour; S14, retaining the seed points selected in steps S12 and S13, and using the second fitted tire contour as a reference, finding the pixel point with the maximum gradient amplitude in the upper half of the tire from top to bottom, and performing a third tire contour fitting; S15, using the tire outer contour fitted in step S14 as a reference, searching for a pixel point with the maximum gradient amplitude toward the tire center, using this pixel point as a seed point for fitting the wheel hub outer contour, and fitting the wheel hub outer contour; S16, repeating the iteration to obtain the tire outer profile and the wheel hub outer profile that meet the preset error standard; S17, finding a pixel gradient amplitude point at the intersection of the tire and the ground within an angle range of 45 degrees to the lower left and 45 degrees to the lower right of the tire center, and drawing a tire-ground dividing line based on the found pixel gradient amplitude point to obtain the interface between the tire and the ground after the tire is deformed; S18, endpoint processing is performed on the image pixels at the interface between the tire and the ground from bottom to top, and the Y coordinate difference between two adjacent pixel points is calculated. When the Y coordinate difference between any pixel point and its adjacent pixel point is greater than a preset coordinate difference threshold, the pixel point is judged to be the endpoint of the tire-ground contact, and the actual contact pixel length between the tire and the ground is obtained.

5. The overloaded and oversized vehicle early warning and tracking system based on AI and machine vision dual drive according to claim 4 is characterized in that: The deformation parameter calculation unit uses a proportional factor α to correct the obtained mechanical deformation parameters of the tire: Where Rim is the wheel radius; r is the pixel radius of the wheel obtained by fitting.

6. The overloaded and oversized vehicle early warning and tracking system based on AI and machine vision dual drive according to claim 1 is characterized in that: The training process of the load prediction model includes: S1, using a thermal imaging acquisition component to capture side thermal imaging images of a certain number of sample tires with different tire cross-sectional sizes, different tire pressures, different loads, and different temperatures under normal working conditions, to generate a sample tire set; S2, extracting pixel temperature matrix data from each side thermal imaging image in the sample tire set to obtain surface temperature information of each sample tire; processing the pixel temperature matrix data of each sample tire using a deformation parameter calculation unit to obtain mechanical deformation parameters of the sample tire; S3, taking the mechanical deformation parameters of each sample tire and the corresponding tire size and air pressure information as a group of training samples to generate a training sample set; S4, a load prediction model is constructed based on the XGBoost model, and the load prediction model is trained using the training sample set in step S3; the trained load prediction model is used to process the imported mechanical deformation parameters, tire size, and air pressure information of the tire to be tested, and calculate the estimated tire load of the tire to be tested.

7. The overloaded and oversized vehicle early warning and tracking system based on AI and machine vision dual drive according to claim 1 is characterized in that: The non-contact vehicle weighing subsystem includes a model updating device; the model updating device is used to import new sample data into the load prediction model to update the load prediction model.

8. The overloaded and oversized vehicle early warning and tracking system based on AI and machine vision dual drive according to claim 1 is characterized in that: The vehicle tracking module is trained based on the YoloX-DeepSort model, extracts the pixel position of the lower midpoint of the rectangle of the wheel prediction box as the contact point between the target vehicle and the road surface, and calculates the actual position of the target vehicle through the shooting angle, focal length, position of the video acquisition module and the contact point position of the video acquisition module; During the tracking process, the speed detection module records the timestamp of the target vehicle tire leaving the preset virtual detection area, and uses the vehicle wheelbase L and the time difference between the target vehicle leaving the virtual detection area to calculate the average speed of the target vehicle as the vehicle speed v of the target vehicle in the virtual detection area.

9. A method for early warning and tracking overloaded and oversized vehicles based on dual-drive of AI and machine vision, characterized in that: The overloaded and oversized vehicle early warning and tracking method is executed based on the overloaded and oversized vehicle early warning and tracking system described in any one of claims 1 to 8; the overloaded and oversized vehicle early warning and tracking method comprises the following steps: Identify the model of each tire of the vehicle entering the detection area, and obtain the tire cross-sectional size information and tire pressure information based on the identified model; Collect infrared thermal images of the side of the vehicle's single-axle tire to obtain corresponding temperature matrix data; calculate the mechanical deformation parameters of the tire based on the temperature matrix data of the vehicle's single-axle tire; According to the mechanical deformation parameters of the tire, the cross-sectional dimensions of the tire, and the tire pressure information, the estimated tire load F is calculated. 1i ; Combined with tire temperature t i Estimated tire load F 1i Perform temperature correction to obtain the temperature-corrected tire load F 2i =F 1i ×α t ; Real-time video capture of the lane area is performed, the video images captured by the video acquisition module are processed, the target vehicle in the video images is identified and tracked, and the tracked position information is sent to the speed detection module, which calculates the vehicle speed v and marks the calculated vehicle speed v on the target vehicle in the video image; Tire load F after temperature correction combined with vehicle speed v 2i Perform speed correction to obtain the final single-axle tire load F i =F 2i ×β v ; According to the single axle tire load F i The overall load Q of the target vehicle is calculated.

10. The overloaded and oversized vehicle early warning and tracking method based on dual-drive of AI and machine vision according to claim 9 is characterized in that: The overloaded and oversized vehicle early warning tracking method further includes: According to the wheelbase L and the number of axles n of the target vehicle, the load threshold of the target vehicle is obtained by querying, and the overall load Q of the target vehicle is compared with the load threshold to determine whether the target vehicle is overloaded or exceeds the limit.

Images