Road condition monitoring identification method, device and equipment based on machine vision and medium

Abstract

The invention relates to the technical field of data processing, and discloses a road condition monitoring recognition method, device and equipment based on machine vision and a medium. The method comprises the following steps: carrying out multi-source data acquisition on a road environment to obtain an original data set; performing dynamic time warping and multi-modal feature fusion processing on the original data set to obtain a spatial-temporal feature tensor; performing multi-scale pyramid decomposition and depth feature extraction on the spatial-temporal feature tensor to obtain a road condition feature vector; semantic analysis is carried out on the road condition feature vectors through feature mapping and knowledge graph enhancement, and a road condition attribute set is obtained; performing multi-level risk quantification and space-time propagation simulation on the road condition attribute set to obtain a graded early warning index and a risk distribution map; and performing multi-target tradeoff and causal chain reasoning on the graded early warning indexes and the risk distribution map to obtain a self-adaptive traffic regulation and control instruction set. According to the invention, the efficiency and accuracy of road condition monitoring and identification based on machine vision are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a road condition monitoring and identification method, device, equipment and medium based on machine vision. Background Art

[0002] Existing road condition monitoring and recognition technologies mainly rely on a combination of fixed cameras and manual inspections. These systems usually use video image analysis technology to identify abnormal conditions on the road, such as traffic congestion, accidents or road damage. Some advanced systems also integrate sensor networks to monitor environmental parameters such as road temperature and humidity in real time. At the same time, some regions have begun to try to use drones for road inspections to improve the flexibility and coverage of monitoring.

[0003] However, existing technologies still have some obvious shortcomings. First, a single data source (such as relying only on visible light images) is easily affected by weather and lighting conditions, resulting in unstable recognition accuracy. Second, most systems lack effective multi-source data fusion capabilities and cannot fully utilize the advantages of different types of sensors. Furthermore, traditional image processing methods perform poorly in complex scenarios, especially in identifying subtle road damage or predicting potential risks. Finally, existing systems generally lack intelligent decision-making support functions, making it difficult to automatically generate optimized traffic control strategies based on real-time road conditions. Summary of the invention

[0004] The present application provides a method, device, equipment and medium for machine vision-based road condition monitoring and identification, which are used to improve the efficiency and accuracy of machine vision-based road condition monitoring and identification.

[0005] In the first aspect, the present application provides a road condition monitoring and identification method based on machine vision, and the road condition monitoring and identification method based on machine vision includes: collecting multi-source data of the road environment to obtain an original data set, wherein the original data set includes: high-definition image stream, infrared thermal map, laser point cloud, millimeter wave radar data and environmental parameters; performing dynamic time warping and multimodal feature fusion processing on the original data set to obtain a spatiotemporal feature tensor; performing multi-scale pyramid decomposition and deep feature extraction on the spatiotemporal feature tensor to obtain a road condition feature vector; performing semantic analysis on the road condition feature vector through feature mapping and knowledge graph enhancement to obtain a road condition attribute set, wherein the road condition attribute set includes pavement type, damage status, attachment distribution and traffic flow parameters; performing multi-level risk quantification and spatiotemporal propagation simulation on the road condition attribute set to obtain graded warning indicators and risk distribution maps; performing multi-objective trade-offs and causal chain reasoning on the graded warning indicators and risk distribution maps to obtain an adaptive traffic control instruction set.

[0006] In a second aspect, the present application provides a road condition monitoring and identification device based on machine vision, and the road condition monitoring and identification device based on machine vision includes:

[0007] An acquisition module is used to collect multi-source data of the road environment to obtain an original data set, wherein the original data set includes: high-definition image stream, infrared thermal map, laser point cloud, millimeter wave radar data and environmental parameters;

[0008] A fusion module, used for performing dynamic time warping and multimodal feature fusion processing on the original data set to obtain a spatiotemporal feature tensor;

[0009] An extraction module, used for performing multi-scale pyramid decomposition and deep feature extraction on the spatiotemporal feature tensor to obtain a road condition feature vector;

[0010] A parsing module, used for performing semantic parsing on the road condition feature vector through feature mapping and knowledge graph enhancement to obtain a road condition attribute set, wherein the road condition attribute set includes road surface type, damage state, attachment distribution and traffic flow parameters;

[0011] A simulation module, used for performing multi-level risk quantification and spatiotemporal propagation simulation on the road condition attribute set to obtain a graded warning index and a risk distribution map;

[0012] The reasoning module is used to perform multi-objective weighing and causal chain reasoning on the graded warning indicators and risk distribution map to obtain an adaptive traffic control instruction set.

[0013] The third aspect of the present application provides a computer device, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via a bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned road condition monitoring and identification method based on machine vision are performed.

[0014] The fourth aspect of the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the above-mentioned road condition monitoring and identification method based on machine vision.

[0015] In the technical solution provided by this application, multi-source data acquisition significantly improves the comprehensiveness and reliability of data, including the comprehensive use of high-definition image streams, infrared thermal images, laser point clouds, millimeter-wave radar data and environmental parameters, so that road condition monitoring is no longer limited to a single data source, greatly enhancing the system's perception of complex road conditions. Dynamic time warping and multimodal feature fusion technology effectively solve the problem of spatiotemporal alignment of heterogeneous data, ensuring the consistency and comparability of data from different sources. Multiscale pyramid decomposition and deep feature extraction methods can capture the multi-scale characteristics of road conditions, from macroscopic road network structures to microscopic road surface details, which can be fully characterized, improving the accuracy and robustness of feature extraction. The introduction of feature mapping and knowledge graph enhancement combines machine learning with domain expert knowledge, greatly improving the accuracy and interpretability of semantic understanding of road conditions. Multi-level risk quantification and spatiotemporal propagation simulation technology can not only assess the current road condition risks, but also predict the evolution trend of risks, providing a scientific basis for active prevention and timely intervention. Finally, the application of multi-objective trade-offs and causal chain reasoning enables the system to find the optimal balance between multiple objectives such as safety, traffic efficiency and maintenance costs, and formulate more accurate and effective control strategies through causal analysis. Reduce road maintenance costs through accurate maintenance strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0017] Figure 1 A schematic diagram of an embodiment of a road condition monitoring and identification method based on machine vision in an embodiment of the present application;

[0018] Figure 2 A schematic diagram of an embodiment of a road condition monitoring and identification device based on machine vision in an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of the structure of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The embodiments of the present application provide a method, device, equipment and medium for road condition monitoring and identification based on machine vision. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiment of the present application, an embodiment of the road condition monitoring and identification method based on machine vision includes:

[0022] Step S101, collecting multi-source data of the road environment to obtain an original data set, wherein the original data set includes: high-definition image stream, infrared thermal map, laser point cloud, millimeter wave radar data and environmental parameters;

[0023] Step S102, performing dynamic time warping and multimodal feature fusion processing on the original data set to obtain a spatiotemporal feature tensor;

[0024] Step S103, performing multi-scale pyramid decomposition and deep feature extraction on the spatiotemporal feature tensor to obtain a road condition feature vector;

[0025] Step S104: semantically analyze the road condition feature vector through feature mapping and knowledge graph enhancement to obtain a road condition attribute set, wherein the road condition attribute set includes road surface type, damage state, attachment distribution and traffic flow parameters;

[0026] Step S105, performing multi-level risk quantification and spatiotemporal propagation simulation on the road condition attribute set to obtain a graded warning index and a risk distribution map;

[0027] Step S106: Perform multi-objective weighing and causal chain reasoning on the graded warning indicators and risk distribution map to obtain an adaptive traffic control instruction set.

[0028] It is understandable that the execution subject of the present application can be a road condition monitoring and recognition device based on machine vision, or a terminal or a server, which is not limited here. The present application embodiment is described by taking the server as the execution subject as an example.

[0029] Specifically, multi-source data collection is performed on the road environment to obtain the original data set. This involves using multiple sensor devices to simultaneously collect road environment information, including high-definition image streams captured by high-definition cameras, infrared heat maps obtained by infrared thermal imagers, laser point clouds obtained by laser radar scanning, millimeter-wave radar data detected by millimeter-wave radars, and environmental parameters measured by various environmental sensors. The high-definition image stream provides visible light information of the road scene, the infrared heat map can identify heat sources under low light conditions, the laser point cloud accurately depicts the three-dimensional structure of the road, the millimeter-wave radar data can detect moving objects, and the environmental parameters include temperature, humidity, light intensity, etc. The original data set is processed by dynamic time warping and multimodal feature fusion to obtain a spatiotemporal feature tensor. Dynamic time warping aims to solve the problem of inconsistent data collection frequencies of different sensors, and aligns each data stream to a unified time scale through methods such as interpolation or downsampling. Multimodal feature fusion uses deep learning techniques, such as multimodal autoencoders or attention mechanisms, to fuse different types of data features into a unified representation. The output of this step is a high-dimensional tensor that contains information on time, space, and multiple data modalities.

[0030] Subsequently, the spatiotemporal feature tensor is subjected to multi-scale pyramid decomposition and deep feature extraction to obtain a road condition feature vector. Multi-scale pyramid decomposition is an image processing technique that creates feature maps of different resolutions by downsampling layer by layer, which helps capture road condition information at different scales. Deep feature extraction uses deep learning models such as convolutional neural networks (CNNs) to extract high-level semantic features from multi-scale feature maps. The output of this step is a low-dimensional road condition feature vector that condenses the key information in the original data. The road condition feature vector is semantically parsed through feature mapping and knowledge graph enhancement to obtain a set of road condition attributes. Feature mapping uses a pre-trained deep learning model to map the feature vector to a semantic space. Knowledge graph enhancement introduces prior knowledge in the road field, such as pavement material properties, traffic rules, etc., to assist in interpreting the semantic meaning of the features. Through this step, the system can identify road condition attributes such as pavement type (such as asphalt, concrete), damage status (such as cracks, potholes), attachment distribution (such as water accumulation, snow accumulation) and traffic flow parameters (such as vehicle volume, average speed).

[0031] Then, multi-level risk quantification and spatiotemporal propagation simulation are performed on the road condition attribute set to obtain the graded warning indicators and risk distribution map. Multi-level risk quantification considers the impact of different attributes on road safety, and converts qualitative road condition attributes into quantitative risk indicators through methods such as hierarchical analysis method (AHP) or fuzzy comprehensive evaluation. Spatiotemporal propagation simulation uses models such as cellular automata or graph neural networks to simulate the risk propagation process in the road network and generate a dynamic risk distribution map. Finally, multi-objective trade-offs and causal chain reasoning are performed on the graded warning indicators and risk distribution map to obtain the adaptive traffic control instruction set. Multi-objective trade-offs consider multiple objectives such as safety, traffic efficiency, and maintenance cost, and find a balanced solution through multi-objective optimization algorithms such as NSGA-II. Causal chain reasoning is based on Bayesian networks or structural equation models to analyze the impact of different control measures on road conditions and traffic flow, thereby generating the optimal control strategy.

[0032] For example, a highway section uses this method for road condition monitoring. First, a high-definition video stream with a resolution of 1080p, an infrared thermal image with a resolution of 320×240 pixels, a laser point cloud with 1 million points per second, millimeter-wave radar data with an angular resolution of 0.1°, and environmental parameters such as temperature and humidity updated every second are collected. These data are aligned to a 1-second interval using the dynamic time window method and fused into a 512-dimensional spatiotemporal feature tensor using a multimodal variational autoencoder. This tensor is decomposed into a 4-layer Gaussian-Laplacian pyramid to obtain a multi-scale feature map, and then a 2048-dimensional road condition feature vector is extracted through the ResNet50 network. The feature vector is mapped to a 300-dimensional semantic space through a pre-trained Word2Vec model, and the road condition attribute set is parsed by combining it with a road knowledge graph containing 10,000 concept nodes. The attribute set includes pavement type (asphalt, 95% confidence), damage state (minor cracks, area of ​​about 0.5 square meters), attachment distribution (water accumulation, depth 3cm, coverage rate 2%) and traffic flow parameters (traffic volume 150 vehicles / hour, average speed 85km / h). Based on these attributes, a five-layer AHP model is used to quantify the risk, and a comprehensive risk index of 0.7 (full score 1) is obtained. Through the spatiotemporal propagation model based on graph convolutional networks, it is predicted that the risk may spread to the adjacent 5-kilometer section in the next 2 hours. Multi-objective optimization weighs safety and traffic efficiency, combined with Bayesian causal reasoning, and finally generates control instructions: reduce the speed limit to 70km / h, turn on the electronic warning sign, and arrange maintenance personnel to deal with water accumulation and cracks within 2 hours.

[0033] In the embodiments of the present application, multi-source data acquisition significantly improves the comprehensiveness and reliability of the data, including the comprehensive utilization of high-definition image streams, infrared thermal images, laser point clouds, millimeter-wave radar data and environmental parameters, so that road condition monitoring is no longer limited to a single data source, and the system's perception of complex road conditions is greatly enhanced. Dynamic time warping and multimodal feature fusion technology effectively solve the problem of spatiotemporal alignment of heterogeneous data, ensuring the consistency and comparability of data from different sources. Multiscale pyramid decomposition and deep feature extraction methods can capture the multi-scale characteristics of road conditions, from macroscopic road network structures to microscopic road surface details, which can be fully characterized, improving the accuracy and robustness of feature extraction. The introduction of feature mapping and knowledge graph enhancement combines machine learning with domain expert knowledge, greatly improving the accuracy and interpretability of semantic understanding of road conditions. Multi-level risk quantification and spatiotemporal propagation simulation technology can not only assess the current road condition risk, but also predict the evolution trend of risks, providing a scientific basis for active prevention and timely intervention. Finally, the application of multi-objective trade-offs and causal chain reasoning enables the system to find the optimal balance between multiple objectives such as safety, traffic efficiency and maintenance costs, and formulate more accurate and effective control strategies through causal analysis. Reduce road maintenance costs through accurate maintenance strategies.

[0034] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0035] (1) Perform high-resolution scanning of visible light scenes in the road environment to obtain a high-definition image stream, and perform infrared imaging of thermal radiation scenes in the road environment to obtain an infrared thermal map;

[0036] (2) Perform three-dimensional laser scanning on the road environment to obtain a laser point cloud, and use millimeter-wave radar to detect moving targets in the road environment to obtain millimeter-wave radar data;

[0037] (3) Collect multiple parameters of the meteorological conditions of the road environment to obtain environmental parameters;

[0038] (4) The high-definition image stream, infrared thermal image, laser point cloud, millimeter-wave radar data and environmental parameters are synchronized through the timestamp alignment algorithm to obtain time-aligned data, and the time-aligned data is converted into coordinates and processed in space through the spatial registration algorithm to obtain spatial consistency data;

[0039] (5) Perform data cleaning and outlier detection on the spatial consistency data to obtain preprocessed data, and then extract features from the preprocessed data using a multimodal feature extraction algorithm to obtain the original data set.

[0040] Specifically, the base first performs high-resolution scanning of the visible light scene in the road environment to obtain a high-definition image stream. A high-resolution camera, such as a 4K resolution (3840×2160 pixel) camera, is used to continuously collect road scene images at a rate of 30 frames per second. At the same time, infrared imaging of the thermal radiation scene in the road environment is performed to obtain an infrared thermal map. Infrared imaging uses a long-wave infrared thermal imager with an operating wavelength range of 8-14μm, a resolution of 640×480 pixels, and a temperature resolution of 0.05℃, which can effectively capture the temperature distribution and thermal anomalies on the road surface. A three-dimensional laser scan of the road environment is performed to obtain a laser point cloud. The laser scanner uses a 64-line laser radar with a scanning frequency of 10Hz, a ranging accuracy of ±2cm, a horizontal field of view of 360°, a vertical field of view of 26.9°, and can collect about 1.3 million points per second. At the same time, the millimeter-wave radar is used to detect moving targets in the road environment to obtain millimeter-wave radar data. The millimeter wave radar operates at a frequency of 77GHz, with a distance resolution of 0.2m and a velocity resolution of 0.1m / s, and can track up to 100 targets simultaneously.

[0041] Then, the meteorological conditions of the road environment are collected by multi-parameters to obtain environmental parameters. Meteorological parameter collection includes temperature (accuracy ±0.1℃), humidity (accuracy ±2%RH), wind speed (accuracy ±0.5m / s), precipitation (accuracy ±0.2mm) and light intensity (accuracy ±2%), etc., with a sampling frequency of 1Hz. Subsequently, the high-definition image stream, infrared thermal map, laser point cloud, millimeter wave radar data and environmental parameters are processed in time synchronization through the timestamp alignment algorithm to obtain time-aligned data. The timestamp alignment algorithm uses a linear interpolation method to align all data streams to a unified time axis with a time resolution of 0.1 seconds. The time-aligned data is converted into a coordinate system and processed in space by a spatial registration algorithm to obtain spatial consistency data. Spatial registration uses an ICP (iterative closest point) algorithm based on feature point matching to convert different sensor data into a unified world coordinate system, with a registration accuracy better than 5cm.

[0042] Finally, data cleaning and outlier detection are performed on the spatial consistency data to obtain preprocessed data. Data cleaning includes removing invalid data points, filling missing values, and smoothing noise. Outlier detection uses an algorithm based on local outlier factor (LOF) to identify and mark data points that deviate from the normal range. Feature extraction is performed on the preprocessed data using a multimodal feature extraction algorithm to obtain the original data set. Multimodal feature extraction uses deep learning models, such as multimodal autoencoders, to extract unified feature representations from different types of data.

[0043] For example, at a highway monitoring point, the specific implementation process of this method is as follows: First, a 4K high-definition camera collects road images at a frame rate of 30fps, while a 640×480 resolution infrared thermal imager captures the temperature distribution of the road surface. A 64-line laser radar scans the road environment at a frequency of 10Hz, and obtains about 130,000 spatial points each time. The 77GHz millimeter-wave radar outputs target detection results every 0.1 seconds, including distance, speed, and direction information. The weather station records environmental parameters once a second, including 22.5℃ temperature, 65% relative humidity, 3.2m / s wind speed, 0mm precipitation, and 15000lux light intensity. The timestamp alignment algorithm aligns these heterogeneous data to a time interval of 0.1 seconds. For example, the 30fps image stream is downsampled to 10fps, which is consistent with the sampling rate of the laser point cloud; the millimeter-wave radar data and environmental parameters are linearly interpolated to synchronize them with other data. During the spatial registration process, feature points (such as roadside lines, traffic signs, etc.) are first extracted from high-definition images and laser point clouds, and then the ICP algorithm is used to iteratively optimize the transformation matrix between different coordinate systems, and finally all data is unified into a world coordinate system based on the road centerline.

[0044] During the data cleaning phase, invalid points with reflectivity below 5% in the laser point cloud were removed (accounting for about 2% of the total points), local gaps in the infrared image caused by occlusion were filled (accounting for about 1% of the image area), and Gaussian filtering was applied to the high-definition image to reduce noise. Outlier detection identified an abnormally high temperature area (10°C higher than the surrounding temperature), which may indicate potential road problems. Multimodal feature extraction input the cleaned data into a pre-trained deep neural network to extract a 1024-dimensional feature vector, which contains comprehensive information such as road surface texture, temperature distribution, three-dimensional structure, and dynamic targets.

[0045] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0046] (1) Perform time series analysis on each data stream in the original data set to obtain a timestamp sequence, and align the timestamp sequence to obtain a regularized time series;

[0047] (2) interpolating the regularized time series to obtain a data series with a uniform sampling rate, and segmenting the data series with a uniform sampling rate to obtain a time window data set;

[0048] (3) Extract features from each data mode in the time window data set to obtain a multimodal feature set, and rescale the multimodal feature set to obtain a normalized feature set;

[0049] (4) Perform cross-modal feature fusion on the normalized feature set through a multimodal feature fusion algorithm to obtain a fused feature vector, and perform time series correlation analysis on the fused feature vector to obtain a time series feature matrix;

[0050] (5) The spatial relationship analysis of the time series feature matrix is ​​performed through the spatial correlation analysis algorithm to obtain the spatiotemporal feature tensor.

[0051] Specifically, a time series analysis is performed on each data stream to obtain a timestamp sequence. This involves extracting the acquisition time of each data point to form a discrete time series. Subsequently, these timestamp sequences are aligned to obtain a regularized time series. The alignment process uses the dynamic time warping (DTW) algorithm, which makes the time axes of different data streams correspond to each other by finding the optimal time warping path. Next, the regularized time series is interpolated to obtain a data series with a uniform sampling rate. The interpolation process uses the cubic spline interpolation method to create a smooth curve between the original data points to obtain data points with a fixed time interval. The uniform sampling rate is usually set to the common divisor of the highest sampling rate in all data streams to ensure that information is not lost. Then, the data series with a uniform sampling rate is segmented to obtain a time window data set. The segmentation uses a sliding window technology, and the window size is set according to the specific application requirements, usually 1-10 seconds, and the window overlap rate is 50% to capture local characteristics and changing trends in time.

[0052] Subsequently, feature extraction is performed on each data modality in the time window dataset to obtain a multimodal feature set. Feature extraction uses different methods for different types of data: for image data, a convolutional neural network (CNN) is used to extract spatial features; for time series data, such as radar signals, a long short-term memory network (LSTM) is used to extract temporal features; for point cloud data, a network such as PointNet is used to extract three-dimensional structural features. The extracted features are usually high-dimensional vectors, and each modality may have hundreds to thousands of dimensions. Next, the multimodal feature set is rescaled to obtain a normalized feature set. Normalization uses the Z-score standardization method to adjust the mean of each feature to 0 and the standard deviation to 1, ensuring that the features of different modalities are subsequently processed at the same scale. Then, the normalized feature set is cross-modal feature fusion is performed on the multimodal feature fusion algorithm to obtain a fused feature vector. The fusion algorithm uses a multimodal Transformer model with an attention mechanism, which can adaptively learn the correlation and importance weights between features of different modalities. The fused feature vector usually has a dimension between 1000 and 2000, and contains the key information of each modality. The fused feature vector is subjected to time series correlation analysis to obtain a time series feature matrix. Time series correlation analysis uses the autocorrelation function (ACF) and partial autocorrelation function (PACF) to identify periodic patterns and time dependencies in the feature sequence.

[0053] Finally, the spatial relationship analysis of the time series feature matrix is ​​performed through the spatial correlation analysis algorithm to obtain the spatiotemporal feature tensor. Spatial correlation analysis uses the spatial autocorrelation index (such as Moran's I) and the geographically weighted regression (GWR) method to capture the distribution law and local correlation of features in the spatial dimension. The final spatiotemporal feature tensor is a three-dimensional data structure that contains information in three dimensions: time, space, and features.

[0054] For example, at a certain highway monitoring point, the specific implementation process of this method is as follows: First, a 30fps 4K video stream, a 10Hz 64-line laser radar point cloud, a 100Hz millimeter-wave radar data, and a 1Hz environmental parameter data are collected. These data streams are aligned by the DTW algorithm to obtain a regular time series with an interval of 0.01 seconds. All data are adjusted to a uniform sampling rate of 100Hz using cubic spline interpolation, and then segmented with a window size of 5 seconds and an overlap of 2.5 seconds to obtain a time window data set. For each 5-second window, 150 frames of images are extracted from the 4K video, and 2048-dimensional image features are extracted through the pre-trained ResNet50 network; for laser point cloud data, 1024-dimensional three-dimensional structural features are extracted using the PointNet++ network; 512-dimensional time-frequency features are extracted from millimeter-wave radar data through 1D-CNN; and environmental parameters are directly used as 5-dimensional feature vectors. After these features are normalized by Z-score, they are input into the multimodal Transformer for fusion to obtain a 1536-dimensional fused feature vector.

[0055] ACF and PACF analysis of the fused feature vectors identified an obvious 15-minute periodicity in the feature sequence, which may correspond to the regular changes in traffic flow. Spatial correlation analysis showed that some features (such as road surface temperature) were highly positively correlated within a 2-kilometer range (Moran's I = 0.85), while other features (such as traffic density) were moderately negatively correlated (Moran's I = -0.42). The final generated spatiotemporal feature tensor dimension is 100×100×1536, where 100×100 represents the spatial grid of the monitoring area and 1536 is the feature dimension of each spatiotemporal point. This tensor comprehensively captures the changing characteristics of the road environment in the temporal and spatial dimensions, providing a rich information basis for subsequent road condition analysis and early warning. Through this multimodal, multi-scale data processing and feature extraction method, the accuracy and comprehensiveness of road condition monitoring are greatly improved, and potential road safety hazards and traffic anomalies can be better identified.

[0056] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0057] (1) Perform multi-scale decomposition of the spatiotemporal feature tensor using the Gaussian pyramid algorithm to obtain a multi-scale feature pyramid, and perform differential processing on the multi-scale feature pyramid using the Laplacian pyramid algorithm to obtain multi-scale differential features;

[0058] (2) The multi-scale differential features are enhanced through convolution operations to obtain enhanced feature maps, and the enhanced feature maps are reduced in dimension through pooling operations to obtain compressed feature representations;

[0059] (3) The compressed feature representation is transformed by a nonlinear activation function to obtain an activated feature map, and the activated feature map is fused with multiple layers of features to obtain a fused feature representation;

[0060] (4) The key information of the fused feature representation is extracted through the attention mechanism to obtain the attention-weighted feature, and the attention-weighted feature is reduced in dimension through the fully connected layer to obtain the road condition feature vector.

[0061] Specifically, the multi-scale decomposition is performed on it through the Gaussian pyramid algorithm to obtain a multi-scale feature pyramid. The Gaussian pyramid algorithm downsamples and smoothes the original tensor in an iterative manner, and the resolution of each layer is half of the previous layer. Specifically, for the original tensor T, the Gaussian pyramid of the i-th layer is represented as Gi(T), and the calculation process is to apply a 5×5 Gaussian kernel to Gi-1(T) for convolution, and then perform 2×2 downsampling. Usually 4 to 5 layers of pyramids are constructed to capture features of different scales.

[0062] Subsequently, the multi-scale feature pyramid is differentially processed by the Laplacian pyramid algorithm to obtain multi-scale differential features. The Laplacian pyramid is a differential representation of the Gaussian pyramid, and the calculation method is to subtract two adjacent layers of the Gaussian pyramid. For the Laplacian representation Li(T) of the i-th layer, its calculation formula is Li(T) = Gi(T)-upsample(Gi+1(T)), where upsample represents the upsampling operation. The Laplacian pyramid can highlight the feature edges and detail information at different scales. Next, the multi-scale differential features are enhanced by convolution operations to obtain enhanced feature maps. In this step, multiple convolution kernels are used to perform convolution operations on the differential features of each layer to extract richer feature representations. The size of the convolution kernel is usually 3×3 or 5×5, and the number ranges from 32 to 128, depending on specific needs. After the convolution operation, the enhanced feature map is reduced in dimension by pooling operations to obtain a compressed feature representation. The pooling operation commonly uses maximum pooling or average pooling, with a pooling window size of 2×2 and a step size of 2, which can significantly reduce the amount of data while retaining the main features.

[0063] Then, the compressed feature representation is transformed by a nonlinear activation function to obtain an activated feature map. Common nonlinear activation functions include ReLU (Rectified Linear Unit) or LeakyReLU, which can introduce nonlinear transformations and enhance the expressive power of the model. The activated feature map is subjected to multi-layer feature fusion to obtain a fused feature representation. Feature fusion adopts the idea of ​​Skip Connection to combine feature maps of different levels by element-level addition or concatenation to comprehensively utilize multi-scale information. Finally, the fused feature representation is subjected to key information extraction by the attention mechanism to obtain an attention-weighted feature. The attention mechanism calculates the importance weight of each feature position, focusing on the most critical areas and features for road condition recognition. The specific implementation can adopt the self-attention mechanism to calculate the attention weight by the query-key-value method. After obtaining the attention-weighted feature, it is subjected to dimensionality reduction processing by the fully connected layer to obtain the final road condition feature vector. The fully connected layer maps high-dimensional features to a lower-dimensional vector space, usually down to 256 or 512 dimensions, to facilitate subsequent classification or regression tasks.

[0064] For example, in a highway monitoring system, the specific implementation process of this method is as follows: First, a 5-layer Gaussian pyramid is constructed for the 100×100×1536 spatiotemporal feature tensor to obtain multi-scale feature representations with resolutions of 100×100, 50×50, 25×25, 13×13, and 7×7, respectively. Then, the Laplacian pyramid is calculated to obtain 4 layers of differential features, each containing 1536 channels. 64 3×3 convolution kernels are applied to each layer of differential features for feature enhancement to obtain 64 feature maps. Taking the first layer (100×100) as an example, the enhanced feature dimension is 100×100×64. Then, a 2×2 maximum pooling operation is performed to halve the size of the feature map to obtain a compressed feature representation of 50×50×64. The ReLU activation function is applied to the compressed features, f(x)=max(0,x), introducing nonlinear transformation. Then the activation feature maps of different levels are upsampled and concatenated to obtain a 100×100×256 fused feature representation. The self-attention mechanism is applied to the fused features to calculate the attention weight matrix A with a dimension of 100×100, which represents the importance of each spatial position. A is multiplied by the original feature map to obtain the attention weighted feature.

[0065] Finally, the 100×100×256 feature map is mapped to a 512-dimensional road condition feature vector through a two-layer fully connected network. The first layer of full connection reduces the input to 2048 dimensions, and the second layer further reduces it to 512 dimensions. The final 512-dimensional vector contains key information about road conditions, such as road surface type, degree of damage, traffic flow, etc., providing a highly concentrated feature representation for subsequent road condition analysis and decision-making. This multi-scale, multi-level feature extraction and fusion method can effectively capture the complex characteristics of the road environment, and can be comprehensively characterized from the macroscopic road network structure to the microscopic road surface details. By introducing the attention mechanism, the ability to identify key areas and features is further improved, greatly enhancing the accuracy and reliability of road condition monitoring.

[0066] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0067] (1) The road condition feature vector is projected into semantic space through a feature mapping algorithm to obtain a semantic feature representation, and the semantic feature representation is grouped to obtain preliminary semantic categories;

[0068] (2) Concept alignment of the preliminary semantic categories is performed using a knowledge graph matching algorithm to obtain a semantic concept set, and semantic relations are constructed for the semantic concept set to obtain a semantic relation network;

[0069] (3) Performing pavement type recognition on the semantic relationship network to obtain the pavement type, and performing pavement damage analysis on the semantic relationship network to obtain the damage status;

[0070] (4) Identify road attachments on the semantic relationship network to obtain the attachment distribution, and calculate traffic flow parameters on the semantic relationship network to obtain traffic flow parameters;

[0071] (5) The road surface type, damage status, attachment distribution, and traffic flow parameters are integrated through the attribute aggregation algorithm to obtain the road condition attribute set.

[0072] Specifically, after obtaining the road condition feature vector, the road condition monitoring and recognition method based on machine vision first performs semantic space projection through the feature mapping algorithm to obtain the semantic feature representation. This process uses a pre-trained word embedding model, such as Word2Vec or GloVe, to map the high-dimensional feature vector to a low-dimensional semantic space. Assuming that the road condition feature vector is 512-dimensional, it can be converted into a 300-dimensional semantic vector through feature mapping. The mapping process can be expressed as:

[0073] S=W·F+b

[0074] Among them, S is a 300-dimensional semantic vector, F is a 512-dimensional road condition feature vector, W is a 512×300 weight matrix, and b is a 300-dimensional bias vector. These parameters are obtained through pre-training and can convert the original features into representations with semantic meaning. Next, the semantic feature representations are feature grouped to obtain preliminary semantic categories. The feature grouping uses the K-means clustering algorithm to divide the vectors in the 300-dimensional semantic space into K clusters. The selection of cluster centers is based on prior knowledge in the road condition field, such as pavement type, damage degree, and other categories. Each cluster represents a preliminary semantic category, such as "asphalt pavement", "minor damage", etc.

[0075] Subsequently, the preliminary semantic categories are aligned with the knowledge graph matching algorithm to obtain a set of semantic concepts. The knowledge graph contains professional concepts and relationships in the road field, such as pavement materials, damage types, and traffic parameters. The concept alignment process uses cosine similarity to calculate the similarity between the preliminary semantic categories and the concepts in the knowledge graph, and selects the concept with the highest similarity as the matching result. For example, "asphalt pavement" may be aligned with the concept of "flexible pavement" in the knowledge graph. The semantic relationship set is constructed to obtain a semantic relationship network. The relationship construction is based on the predefined relationship types in the knowledge graph, such as "is a kind of", "includes", "leads to", etc., connecting the matching concepts into a network structure. Next, the semantic relationship network is used to identify the pavement type and obtain the pavement type. The pavement type identification uses a graph convolutional network (GCN) to process the semantic relationship network and classify it using node features and topological structures. The output of the GCN is passed through the softmax function to obtain the probability distribution of each pavement type, and the category with the highest probability is selected as the recognition result. At the same time, the pavement damage analysis is performed on the semantic relationship network to obtain the damage status. Damage analysis uses a multi-label classification model, such as LSTM (Long Short-Term Memory Network), which considers the temporal characteristics of the damage type and outputs multiple damage labels and their severity.

[0076] Then, the semantic relationship network is used to identify road attachments and obtain the attachment distribution. Attachment identification uses a variant of the regional convolutional neural network (R-CNN), such as Mask R-CNN, which can simultaneously perform target detection and semantic segmentation to identify the type, location, and coverage area of ​​the attachments. Traffic flow parameters are calculated on the semantic relationship network to obtain traffic flow parameters. Traffic flow parameter calculation uses time series analysis methods, such as ARIMA (autoregressive integrated moving average model) to predict indicators such as traffic volume and average speed in the short term. Finally, the road surface type, damage status, attachment distribution, and traffic flow parameters are integrated through the attribute aggregation algorithm to obtain a set of road condition attributes. Attribute aggregation uses a weighted summation method to assign weights to each attribute according to its importance to the road condition assessment. The aggregation formula is as follows:

[0077]

[0078] Among them, R is the comprehensive road condition index, A i is the quantitative value of the i-th attribute, w i is the corresponding weight, and n is the total number of attributes. The weight is determined based on expert experience and historical data analysis.

[0079] For example, a highway monitoring system uses this method to analyze a 1-kilometer-long section of road. First, the 512-dimensional road condition feature vector is mapped to a 300-dimensional semantic vector through a pre-trained Word2Vec model. K-means clustering (K=10) divides the semantic vector into 10 preliminary categories. Knowledge graph matching aligns these 10 categories with the road domain knowledge graph containing 5,000 nodes to obtain accurate semantic concepts. Semantic relationship construction forms a semantic relationship network containing 50 nodes and 200 edges. The GCN model processes the network and identifies the road surface type as "asphalt concrete" (confidence 0.92). The LSTM model analyzes the damage status as "minor cracks" (area 0.5 square meters, depth 2mm) and "ruts" (depth 5mm, length 10 meters). Mask R-CNN identifies the attachments as "water accumulation" (coverage 2%, average depth 1cm) and "gravel" (coverage 0.5%). The ARIMA model calculates that the current traffic volume is 1,200 vehicles / hour and the average speed is 85km / h.

[0080] When the attributes are aggregated, the weight of the road surface type is 0.3, the weight of the damage state is 0.25, the weight of the attachment distribution is 0.25, and the weight of the traffic flow parameter is 0.2. After the attributes are quantified, the comprehensive road condition index is calculated to be 0.82 (full score 1), indicating that the road condition is good but there are minor problems. This comprehensive index provides intuitive road condition assessment results for traffic management departments, which helps to timely identify potential problems and formulate maintenance plans.

[0081] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0082] (1) Identify risk factors for the road condition attribute set to obtain a risk factor set, and rank the risk factor set by importance to obtain a weighted risk factor;

[0083] (2) The weighted risk factors are processed hierarchically through a multi-level analysis method to obtain a risk hierarchy structure, and the risk hierarchy structure is divided into risk levels to obtain a graded risk indicator;

[0084] (3) The spatial distribution of the graded risk indicators is estimated by using a spatiotemporal interpolation algorithm to obtain a risk spatial distribution map, and the dynamic evolution of the risk spatial distribution map is simulated by a cellular automation algorithm to obtain a risk propagation model;

[0085] (4) Conduct multi-scenario simulations on the risk propagation model to obtain a risk evolution sequence, and calculate the probability distribution of the risk evolution sequence to obtain a risk probability map;

[0086] (5) Divide the risk regions of the risk probability map to obtain hierarchical warning indicators, and graphically represent the hierarchical warning indicators to obtain a risk distribution map.

[0087] Specifically, first identify its risk factors to obtain a risk factor set. A method combining principal component analysis (PCA) and random forest algorithm is adopted. PCA is used for dimensionality reduction and extraction of main features, while random forest is used to evaluate the importance of features. For n road condition attributes, PCA retains the principal components that explain 90% of the variance, usually reducing the dimension to m (m < n). Subsequently, the random forest algorithm scores the importance of these m principal components to obtain the importance weights of each factor. The formula for calculating the importance weights of risk factors is as follows:

[0088]

[0089] Among them, W i is the importance weight of the i-th risk factor, and E i is the importance score of this factor in the random forest, and m is the number of principal components. Then, hierarchically process the weighted risk factors through the analytic hierarchy process (AHP) to obtain a risk hierarchy structure. The AHP method first constructs a hierarchy model, including an objective layer (overall risk), a criterion layer (such as road surface condition, traffic flow, environmental factors), and a scheme layer (specific risk factors). Then, make pairwise comparisons to establish a judgment matrix. The element a ij of the judgment matrix A represents the importance of factor i relative to factor j, and its value range is usually an integer from 1 to 9. Calculate the eigenvector and the maximum eigenvalue, and conduct a consistency test. If the consistency test is passed, the eigenvector after normalization is the weight vector of each layer.

[0090] Conduct risk level division on the risk hierarchy structure to obtain hierarchical risk indicators. The risk level division adopts the fuzzy comprehensive evaluation method to convert the quantitative value of risk factors into qualitative risk levels. First, determine the evaluation factor set and the comment set, and then establish a fuzzy relation matrix R. The calculation formula for the fuzzy comprehensive evaluation result B is:

[0091]

[0092] Among them, W is the weight vector, R is the fuzzy relation matrix, represents the fuzzy composition operation, and b1, b2,..., b mThere are m elements in the B vector, each of which represents the membership of the final evaluation result corresponding to a certain comment level. Then, the spatial distribution of the graded risk indicators is estimated by the spatiotemporal interpolation algorithm to obtain the risk spatial distribution map. The spatiotemporal interpolation uses the Kriging method, which takes into account spatial autocorrelation and can provide the best linear unbiased estimate. The basic formula of Kriging interpolation is:

[0093]

[0094] Among them, Z*(x0) is the predicted value of the point to be estimated, Z(x i ) is the observed value of the known point, λ i is the weight coefficient. The weight coefficient is solved by the variogram and the Kriging equation system. Next, the dynamic evolution simulation of the risk space distribution map is performed by the cellular automaton algorithm to obtain the risk propagation model. The cellular automaton model defines a grid-like space, where each cell represents a road section. The state of each cell is updated in a discrete time step according to the state of the surrounding cells and predefined rules. The state update rule can be expressed as:

[0095] S t+1 (i, j) = f(S t (i, j), N t (i, j))

[0096] Among them, S t (i, j) is the state of the location unit (i, j) at time t, N t (i, j) is its neighborhood state, and f is the state transition function. Multi-scenario simulation is performed on the risk propagation model to obtain a risk evolution sequence. Multi-scenario simulation considers different initial conditions and parameter settings, such as weather changes, traffic flow fluctuations, etc. The cellular automaton model is run multiple times (e.g., 1000 times) in each scenario, and the risk evolution process of each simulation is recorded. The probability distribution of the risk evolution sequence is calculated to obtain a risk probability map. The probability distribution calculation uses the kernel density estimation method to estimate the probability density of the risk value of each spatial location at different time points.

[0097] Finally, the risk probability map is divided into risk areas to obtain graded warning indicators. The risk area division uses a clustering algorithm, such as DBSCAN (density-based spatial clustering algorithm), to cluster areas with similar risk levels. The graded warning indicators are graphically represented to obtain a risk distribution map. The graphical representation uses a combination of heat map and contour map to intuitively display the spatial distribution and intensity of the risk. For example, a certain highway section (length 50 kilometers) was risk assessed using this method. First, 15 major risk factors were identified from the road condition attribute set, including road damage, traffic flow, weather conditions, etc. PCA analysis reduced the 15 factors to 8 principal components, explaining 92% of the total variance. The random forest algorithm scored the importance of these 8 principal components and obtained a weight distribution of [0.25, 0.20, 0.15, 0.12, 0.10, 0.08, 0.06, 0.04]. The AHP method constructs a three-layer risk structure: overall risk (target layer), road conditions, traffic flow, environmental factors (criteria layer), and 8 specific risk factors (scheme layer). Through expert judgment and consistency test, the criterion layer weights [0.5, 0.3, 0.2] are obtained. The fuzzy comprehensive evaluation divides the risk level into low, medium, and high levels, and the risk level of each 500-meter road section is obtained.

[0098] The Kriging interpolation method generates a continuous risk spatial distribution map based on the risk level of these 500-meter sections. The cellular automaton model divides the 50-kilometer section into 100 units, each representing 500 meters. Define state transition rules, such as when the average risk level of the surrounding units is higher than the current unit, the risk level of the current unit has an 80% probability of rising. Perform 1,000 multi-scenario simulations, each simulating 24 hours of risk evolution. Through kernel density estimation, calculate the risk probability distribution of each unit at different time points. The DBSCAN algorithm (parameters: ε = 0.05, MinPts = 4) clusters the risk probability map and identifies 3 high-risk areas, 5 medium-risk areas, and 2 low-risk areas. The resulting risk distribution heat map clearly shows the spatial distribution of risk, with high-risk areas appearing red, mainly concentrated at 10-15 kilometers and 30-35 kilometers of the section.

[0099] In a specific embodiment, the process of executing step S106 may specifically include the following steps:

[0100] (1) Construct a multi-objective function for the hierarchical warning indicators to obtain a multi-objective optimization problem, and set constraints for the multi-objective optimization problem to obtain a constrained optimization problem;

[0101] (2) The multi-objective solution of the constrained optimization problem is solved by the Pareto optimization algorithm to obtain a set of candidate solutions, and the candidate solution set is evaluated to obtain weighted candidate solutions;

[0102] (3) Identify the causal relationship of the risk distribution map to obtain a causal map, and perform probabilistic reasoning on the causal map to obtain the event probability distribution;

[0103] (4) Evaluate the intervention effect of the event probability distribution through the causal intervention analysis algorithm to obtain the intervention plan set, and match the intervention plan set with the weighted candidate plans to obtain the control strategy set;

[0104] (5) Adaptively adjust the control strategy set to obtain an adaptive traffic control instruction set.

[0105] Specifically, a multi-objective function is constructed for the hierarchical warning indicators to obtain a multi-objective optimization problem. The multi-objective function usually includes aspects such as safety, traffic efficiency and maintenance cost. For example, the safety goal can be expressed as minimizing the length of high-risk sections, the traffic efficiency goal can be expressed as maximizing the average vehicle speed, and the maintenance cost goal can be expressed as minimizing the frequency of road maintenance. These goals are often conflicting, and a balance needs to be found between them. Constraints are set for the multi-objective optimization problem to obtain a constrained optimization problem. Constraints include budget constraints, human resource constraints, time constraints, etc. For example, the annual road maintenance budget does not exceed 10 million yuan, the emergency repair response time does not exceed 2 hours, and traffic control measures must not cause the average vehicle speed to decrease by more than 20%. These constraints ensure the feasibility of the optimization results in actual operations.

[0106] Next, the multi-objective solution of the constrained optimization problem is solved by the Pareto optimization algorithm to obtain a set of candidate solutions. Pareto optimization algorithms, such as NSGA-II (Non-dominated Sorting Genetic Algorithm II), can find the optimal trade-off between multiple objectives. The algorithm generates a series of non-dominated solutions, namely the Pareto frontier, through evolutionary computation methods. Each solution represents a possible combination of control strategies, such as adjusting speed limits, implementing traffic control, arranging road maintenance, etc. The candidate solution set is evaluated to obtain weighted candidate solutions. The evaluation process considers the performance of each solution on each objective and weights them according to the decision maker's preferences. Evaluation indicators may include the degree of risk reduction, traffic flow improvement effect, input-output ratio, etc. The weighted method uses the analytic hierarchy process (AHP) or fuzzy comprehensive evaluation method to comprehensively consider quantitative and qualitative factors.

[0107] Then, the risk distribution map is subjected to causal relationship identification to obtain a causal graph. Causal relationship identification uses structural equation models (SEM) or Bayesian networks to analyze the mutual influence between various risk factors. For example, it is identified how the slipperiness of the road surface affects the braking distance of the vehicle, which in turn affects the probability of an accident. The causal graph is represented in the form of a directed acyclic graph (DAG), where nodes represent risk factors and edges represent causal relationships. Probabilistic reasoning is performed on the causal graph to obtain the probability distribution of events. Probabilistic reasoning uses the Bayesian reasoning method to calculate the probability of each event occurring under given observation conditions. For example, based on the current weather conditions, traffic flow, and road conditions, the probability distribution of traffic accidents within the next 2 hours is calculated.

[0108] The intervention effect is evaluated on the event probability distribution through the causal intervention analysis algorithm to obtain the intervention plan set. Causal intervention analysis is based on the do-calculus theory and simulates the impact of different intervention measures on the system. For example, evaluate the effect of reducing the speed limit on reducing the probability of accidents, or analyze the effect of increasing the road friction coefficient on improving driving safety. The intervention plan set contains various possible control measures and their expected effects. The intervention plan set is matched with the weighted candidate plans to obtain the control strategy set. The matching process uses heuristic algorithms, such as simulated annealing or genetic algorithms, to find the best combination of intervention measures. Matching criteria include consistency between the intervention effect and the goals of the candidate plans, implementation difficulty, resource requirements, etc.

[0109] Finally, the control strategy set is adaptively adjusted to obtain the adaptive traffic control instruction set. The adaptive adjustment takes into account the real-time road condition changes and historical data feedback to dynamically optimize the control strategy. Reinforcement learning algorithms such as Q-learning or deep Q network (DQN) are used to continuously optimize the decision-making process and improve the control effect.

[0110] For example, a highway section (100 km long) uses this method for traffic control. First, a multi-objective function is constructed, including minimizing accident risk (f1), maximizing traffic efficiency (f2), and minimizing maintenance cost (f3). Constraints include an annual budget of 100 million yuan and an average speed of no less than 80 km / h. The NSGA-II algorithm generates 100 Pareto optimal solutions, each of which contains a different combination of control strategies. In the scheme evaluation stage, the AHP method is used to assign weights [0.5, 0.3, 0.2] to the three objectives. After evaluation, the top 10 weighted candidate schemes are selected. Causal relationship identification is analyzed through structural equation modeling, and a causal graph with 20 nodes and 35 edges is obtained. Bayesian reasoning calculates the accident probability distribution of each section in the next 4 hours based on real-time data (such as current traffic volume of 1500 vehicles / hour and rainfall of 5mm / h).

[0111] The causal intervention analysis evaluated a variety of measures, such as reducing the speed limit by 20 km / h in high-risk sections (10-20 km and 60-70 km), which is expected to reduce the accident probability by 30%. The matching process selected the optimal control strategy combination: a speed limit of 100 km / h on the 10-20 km section, temporary traffic control on the 60-70 km section, and 4 mobile speed measurement points along the entire line. Adaptive adjustment uses the DQN algorithm to dynamically adjust the strategy based on real-time traffic flow data and weather changes. For example, when the rainfall increases to 10 mm / h, the speed limit is automatically further reduced to 90 km / h, and the road drainage system is activated. This adaptive control method can flexibly adjust the strategy according to the actual situation, effectively improving road safety and traffic efficiency, while optimizing resource utilization.

[0112] The above describes the road condition monitoring and identification method based on machine vision in the embodiment of the present application. The following describes the road condition monitoring and identification device based on machine vision in the embodiment of the present application. Figure 2 In the embodiments of the present application, an embodiment of a road condition monitoring and identification device based on machine vision includes:

[0113] The acquisition module 201 is used to collect multi-source data of the road environment to obtain an original data set, wherein the original data set includes: high-definition image stream, infrared thermal map, laser point cloud, millimeter wave radar data and environmental parameters;

[0114] A fusion module 202 is used to perform dynamic time warping and multimodal feature fusion processing on the original data set to obtain a spatiotemporal feature tensor;

[0115] An extraction module 203 is used to perform multi-scale pyramid decomposition and deep feature extraction on the spatiotemporal feature tensor to obtain a road condition feature vector;

[0116] The parsing module 204 is used to perform semantic parsing on the road condition feature vector through feature mapping and knowledge graph enhancement to obtain a road condition attribute set, wherein the road condition attribute set includes road surface type, damage state, attachment distribution and traffic flow parameters;

[0117] A simulation module 205 is used to perform multi-level risk quantification and spatiotemporal propagation simulation on the road condition attribute set to obtain a graded warning index and a risk distribution map;

[0118] The reasoning module 206 is used to perform multi-objective weighing and causal chain reasoning on the graded warning indicators and risk distribution map to obtain an adaptive traffic control instruction set.

[0119] Through the collaboration of the above components, multi-source data collection significantly improves the comprehensiveness and reliability of data, including the comprehensive use of high-definition image streams, infrared thermal images, laser point clouds, millimeter-wave radar data and environmental parameters, so that road condition monitoring is no longer limited to a single data source, greatly enhancing the system's perception of complex road conditions. Dynamic time warping and multimodal feature fusion technology effectively solve the problem of spatiotemporal alignment of heterogeneous data, ensuring the consistency and comparability of data from different sources. Multiscale pyramid decomposition and deep feature extraction methods can capture the multi-scale characteristics of road conditions, from macroscopic road network structure to microscopic road surface details, which can be fully characterized, improving the accuracy and robustness of feature extraction. The introduction of feature mapping and knowledge graph enhancement combines machine learning with domain expert knowledge, greatly improving the accuracy and interpretability of semantic understanding of road conditions. Multi-level risk quantification and spatiotemporal propagation simulation technology can not only evaluate the current road condition risk, but also predict the evolution trend of risk, providing a scientific basis for active prevention and timely intervention. Finally, the application of multi-objective trade-offs and causal chain reasoning enables the system to find the optimal balance between multiple objectives such as safety, traffic efficiency and maintenance costs, and formulate more accurate and effective control strategies through causal analysis. Reduce road maintenance costs through accurate maintenance strategies.

[0120] Based on the same technical concept, the embodiment of the present application also provides an electronic device. Figure 3 As shown, it is a schematic diagram of the structure of the electronic device 300 provided in the embodiment of the present application, including a processor 301, a memory 302, and a bus 303. Among them, the memory 302 is used to store execution instructions, including a memory 3021 and an external memory 3022; the memory 3021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 301, and the data exchanged with the external memory 3022 such as a hard disk. The processor 301 exchanges data with the external memory 3022 through the memory 3021. When the electronic device 300 is running, the processor 301 communicates with the memory 302 through the bus 303.

[0121] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the machine vision-based road condition monitoring and identification method.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0124] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A road condition monitoring and identification method based on machine vision, characterized in that: The road condition monitoring and identification method based on machine vision includes: Perform multi-source data collection on the road environment to obtain an original data set, wherein the original data set includes: high-definition image stream, infrared thermal map, laser point cloud, millimeter wave radar data and environmental parameters; Performing dynamic time warping and multimodal feature fusion processing on the original data set to obtain a spatiotemporal feature tensor; Performing multi-scale pyramid decomposition and deep feature extraction on the spatiotemporal feature tensor to obtain a road condition feature vector; Performing semantic analysis on the road condition feature vector through feature mapping and knowledge graph enhancement to obtain a road condition attribute set, wherein the road condition attribute set includes road surface type, damage state, attachment distribution and traffic flow parameters; Performing multi-level risk quantification and spatiotemporal propagation simulation on the road condition attribute set to obtain graded warning indicators and risk distribution maps; Multi-objective trade-offs and causal chain reasoning are performed on the graded warning indicators and risk distribution diagram to obtain an adaptive traffic control instruction set.

2. The method for road condition monitoring and identification based on machine vision according to claim 1, characterized in that: The multi-source data collection of the road environment is performed to obtain an original data set, wherein the original data set includes: high-definition image stream, infrared thermal map, laser point cloud, millimeter wave radar data and environmental parameters, including: Performing high-resolution scanning on the visible light scene in the road environment to obtain a high-definition image stream, and performing infrared imaging on the thermal radiation scene in the road environment to obtain an infrared thermal map; Performing three-dimensional laser scanning on the road environment to obtain a laser point cloud, and detecting moving targets in the road environment by using a millimeter-wave radar to obtain millimeter-wave radar data; Performing multi-parameter collection on the meteorological conditions of the road environment to obtain environmental parameters; Performing time synchronization processing on the high-definition image stream, infrared thermal image, laser point cloud, millimeter-wave radar data and environmental parameters through a timestamp alignment algorithm to obtain time-aligned data, and performing coordinate system conversion and spatial correspondence processing on the time-aligned data through a spatial registration algorithm to obtain spatial consistency data; The spatial consistency data is cleaned and outlier detected to obtain preprocessed data, and features are extracted from the preprocessed data using a multimodal feature extraction algorithm to obtain the original data set.

3. The method for road condition monitoring and identification based on machine vision according to claim 1, characterized in that: The dynamic time warping and multimodal feature fusion processing are performed on the original data set to obtain a spatiotemporal feature tensor, including: Performing time series analysis on each data stream in the original data set to obtain a timestamp sequence, and aligning the timestamp sequence to obtain a regularized time series; Performing interpolation processing on the regularized time series to obtain a data sequence with a uniform sampling rate, and segmenting the data sequence with a uniform sampling rate to obtain a time window data set; Extracting features from each data modality in the time window data set to obtain a multimodal feature set, and rescaling the multimodal feature set to obtain a normalized feature set; Performing cross-modal feature fusion on the normalized feature set through a multimodal feature fusion algorithm to obtain a fused feature vector, and performing time series correlation analysis on the fused feature vector to obtain a time series feature matrix; The spatial relationship analysis of the time series feature matrix is ​​performed using a spatial correlation analysis algorithm to obtain the spatiotemporal feature tensor.

4. The method for road condition monitoring and identification based on machine vision according to claim 1, characterized in that: The step of performing multi-scale pyramid decomposition and deep feature extraction on the spatiotemporal feature tensor to obtain a road condition feature vector includes: Performing multi-scale decomposition on the spatiotemporal feature tensor by using a Gaussian pyramid algorithm to obtain a multi-scale feature pyramid, and performing differential processing on the multi-scale feature pyramid by using a Laplace pyramid algorithm to obtain a multi-scale differential feature; Performing feature enhancement on the multi-scale differential features through a convolution operation to obtain an enhanced feature map, and performing dimensionality reduction processing on the enhanced feature map through a pooling operation to obtain a compressed feature representation; Performing feature transformation on the compressed feature representation through a nonlinear activation function to obtain an activated feature map, and performing multi-layer feature fusion on the activated feature map to obtain a fused feature representation; The fused feature representation is represented by extracting key information through an attention mechanism to obtain an attention-weighted feature, and the attention-weighted feature is subjected to dimensionality reduction processing through a fully connected layer to obtain the road condition feature vector.

5. The method for road condition monitoring and identification based on machine vision according to claim 1, characterized in that: The road condition feature vector is semantically parsed through feature mapping and knowledge graph enhancement to obtain a road condition attribute set, wherein the road condition attribute set includes road surface type, damage state, attachment distribution and traffic flow parameters, including: Performing semantic space projection on the road condition feature vector by a feature mapping algorithm to obtain a semantic feature representation, and performing feature grouping on the semantic feature representation to obtain a preliminary semantic category; Performing concept alignment on the preliminary semantic categories through a knowledge graph matching algorithm to obtain a semantic concept set, and constructing semantic relationships on the semantic concept set to obtain a semantic relationship network; Performing pavement type recognition on the semantic relationship network to obtain a pavement type, and performing pavement damage analysis on the semantic relationship network to obtain a damage state; Performing road attachment identification on the semantic relationship network to obtain attachment distribution, and performing traffic flow parameter calculation on the semantic relationship network to obtain traffic flow parameters; The road surface type, damage status, attachment distribution and traffic flow parameters are integrated through an attribute aggregation algorithm to obtain the road condition attribute set.

6. The method for road condition monitoring and identification based on machine vision according to claim 1, characterized in that: The multi-level risk quantification and spatiotemporal propagation simulation of the road condition attribute set are performed to obtain a graded warning index and a risk distribution map, including: Identifying risk factors on the road condition attribute set to obtain a risk factor set, and ranking the risk factor set by importance to obtain a weighted risk factor; The weighted risk factors are hierarchically processed by a multi-level analysis method to obtain a risk hierarchy structure, and the risk hierarchy structure is divided into risk levels to obtain a graded risk indicator; The spatial distribution of the hierarchical risk indicators is estimated by a spatiotemporal interpolation algorithm to obtain a risk spatial distribution map, and the dynamic evolution of the risk spatial distribution map is simulated by a cellular automation algorithm to obtain a risk propagation model; Performing multi-scenario simulation on the risk propagation model to obtain a risk evolution sequence, and performing probability distribution calculation on the risk evolution sequence to obtain a risk probability graph; The risk probability map is divided into risk areas to obtain the graded warning indicators, and the graded warning indicators are graphically represented to obtain the risk distribution map.

7. The method for road condition monitoring and identification based on machine vision according to claim 1, characterized in that: The multi-objective weighing and causal chain reasoning of the hierarchical warning indicators and risk distribution map are performed to obtain an adaptive traffic control instruction set, including: Constructing a multi-objective function for the graded warning indicators to obtain a multi-objective optimization problem, and setting constraints for the multi-objective optimization problem to obtain a constrained optimization problem; Performing multi-objective solutions to the constrained optimization problem by using a Pareto optimization algorithm to obtain a candidate solution set, and performing scheme evaluation on the candidate solution set to obtain a weighted candidate solution; Performing causal relationship identification on the risk distribution map to obtain a causal map, and performing probability reasoning on the causal map to obtain event probability distribution; Performing intervention effect evaluation on the event probability distribution through a causal intervention analysis algorithm to obtain an intervention plan set, and matching the intervention plan set with the weighted candidate plans to obtain a control strategy set; Adaptively adjust the control strategy set to obtain the adaptive traffic control instruction set.

8. A road condition monitoring and identification device based on machine vision, used to implement the road condition monitoring and identification method based on machine vision as described in any one of claims 1 to 7, characterized in that: The road condition monitoring and identification device based on machine vision includes: An acquisition module is used to collect multi-source data of the road environment to obtain an original data set, wherein the original data set includes: high-definition image stream, infrared thermal map, laser point cloud, millimeter wave radar data and environmental parameters; A fusion module, used for performing dynamic time warping and multimodal feature fusion processing on the original data set to obtain a spatiotemporal feature tensor; An extraction module, used for performing multi-scale pyramid decomposition and deep feature extraction on the spatiotemporal feature tensor to obtain a road condition feature vector; A parsing module, used for performing semantic parsing on the road condition feature vector through feature mapping and knowledge graph enhancement to obtain a road condition attribute set, wherein the road condition attribute set includes road surface type, damage state, attachment distribution and traffic flow parameters; A simulation module, used for performing multi-level risk quantification and spatiotemporal propagation simulation on the road condition attribute set to obtain a graded warning index and a risk distribution map; The reasoning module is used to perform multi-objective weighing and causal chain reasoning on the graded warning indicators and risk distribution map to obtain an adaptive traffic control instruction set.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the machine vision-based road condition monitoring and identification method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the road condition monitoring and identification method based on machine vision as described in any one of claims 1 to 7 is implemented.

Images