Road inspection data processing and analyzing system based on multi-modal fusion
By integrating a variety of technical means and models in the road inspection system, a multi-modal integrated road inspection data processing and analysis system is designed, which solves the problem of poor identification results in harsh environments in existing systems, and achieves more efficient and accurate road inspections, meeting the needs of actual road inspections.
Patent Information
- Application Number
- CN202510092910.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-21
- Publication Date
- 2025-06-13
AI Technical Summary
The existing road inspection system has greatly reduced the recognition effect in severe weather or poor lighting conditions, and the multimodal fusion system has problems such as complex data processing processes, insufficient fusion algorithm optimization, and insufficient analysis results, which is difficult to meet the actual road inspection needs.
A road patrol data processing and analysis system based on multimodal fusion was designed, integrating technical means such as image data analysis, radar data analysis, and GPS positioning and map matching. Through convolutional neural networks, recurrent neural networks and graph theory algorithms, road damage recognition, object dynamic prediction and vehicle positioning are realized, and a comprehensive road patrol report is generated through multimodal fusion algorithm.
It has achieved all-round and multi-dimensional monitoring of the road environment, improved the accuracy and comprehensiveness of road inspections, maintained high inspection efficiency in complex and changeable road environments, reduced dependence on manual inspections, reduced inspection costs, and improved the timeliness and efficiency of inspections.
Smart Images
Figure CN120147793A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of road detection and analysis, and specifically to a road patrol data processing and analysis system based on multi-modal fusion. Background Art
[0002] With the rapid development of the transportation industry, road safety and maintenance have become the focus of social attention. Road patrol, as an important means to ensure road safety, promptly detect and handle road damages and obstacles, its efficiency and accuracy directly affect the smoothness and safety of road traffic. In recent years, with the rapid development of sensor technology, image recognition technology, and positioning technology, road patrol has gradually shifted towards intelligence and automation. However, a single technical means still has limitations when dealing with complex and changeable road environments. For example, although image data analysis technology can accurately identify road damages and obstacles, its recognition effect will be greatly reduced in bad weather or poor lighting conditions; radar data analysis technology can predict the dynamics of objects in front of the road, but its accuracy and reliability still need to be further improved when dealing with complex scenarios; while GPS positioning technology can provide the location information of vehicles, but in areas with severe occlusion such as urban canyons and tunnels, its positioning accuracy will be greatly affected.
[0003] In order to overcome the limitations of a single technical means and improve the accuracy and efficiency of road patrol, multi-modal fusion technology has emerged. Multi-modal fusion technology integrates the information of multiple sensors and data sources to achieve complementary advantages and improve the overall performance of the system. However, most of the current multi-modal fusion road patrol systems are still in the research stage and have not yet formed a mature and perfect solution. Existing systems often have problems such as complex data processing processes, insufficiently optimized fusion algorithms, and inaccurate analysis results, making it difficult to meet the needs of actual road patrol. Summary of the Invention
[0004] The purpose of the present invention is to provide a road patrol data processing and analysis system based on multi-modal fusion to solve the problems raised in the above background art.
[0005] To achieve the above purpose, the present invention provides the following technical solution: A road patrol data processing and analysis system based on multi-modal fusion, the system includes: An image data analysis unit, which constructs a road image analysis model for identifying road damages and obstacles; A radar data analysis unit, which constructs a radar data analysis model for predicting the dynamics of objects in front of the road; A positioning and map matching unit, which constructs a GPS positioning and map matching model based on graph theory algorithms. This model uses GPS data and map information to determine the precise location of the vehicle; A data acquisition unit that collects road image data, radar detection data, and GPS positioning data, and transmits the collected data to a road image analysis model, a radar data analysis model, and a GPS positioning and map matching model respectively to obtain the output results of the models; A multi-modal fusion unit that defines a multi-modal fusion algorithm to perform fusion analysis on the model output results to obtain a comprehensive road inspection report.
[0006] Preferably, the road image analysis model adopts a convolutional neural network (CNN) architecture, including: An input layer: receiving road image data; A convolutional layer: extracting image features through multiple convolutional kernels to generate a feature map; A pooling layer: performing downsampling on the feature map to reduce the computational amount; A fully connected layer: flattening the feature map and connecting it to the output layer; An output layer: outputting the road damage type and obstacle location for classification and detection tasks.
[0007] Preferably, for the classification task of the road image analysis model, the loss function adopts the cross-entropy loss function; for the detection task, the loss function adopts the Smooth L1 loss function.
[0008] Preferably, the radar data analysis model is constructed using a recurrent neural network (RNN) algorithm, including: An input layer: receiving radar detection data, with a sequence length of T and a feature dimension of R for each time step; A hidden layer: including multiple layers of LSTM units, with a Dropout layer connected after each layer to prevent overfitting; An output layer: outputting the dynamic prediction results of the objects in front of the road, including distance, speed, and acceleration.
[0009] Preferably, the structure of the GPS positioning and map matching model includes: An input layer: receiving GPS positioning data and map data, where the GPS data includes longitude, latitude, and altitude, and the map data includes road network and node information; A matching layer: matching the GPS data to the nearest node on the map through the Dijkstra algorithm; An output layer: outputting the accurate position information of the vehicle, including the road and node number where it is located.
[0010] Preferably, the steps for training the GPS positioning and map matching model include: Using the GPS positioning data and the corresponding map data as the training set, and selecting the accurate position of the vehicle as the label; Initializing the parameters of the graph theory algorithm, including the weights of the nodes and the lengths of the edges; Use the training set to train the graph theory algorithm and optimize the algorithm by minimizing the matching error; Use the validation set to verify the graph theory algorithm and evaluate the positioning accuracy and matching efficiency of the model; When the model performance reaches the preset standard, save the model parameters to obtain a trained GPS positioning and map matching model.
[0011] Preferably, the implementation steps of the multimodal fusion algorithm defined in the multimodal fusion unit include: Step 1: Define the road damage output by the road image analysis model as D_Image, the obstacle as O_Image, the object dynamics output by the radar data analysis model as D_Radar, and the vehicle position output by the GPS positioning and map matching model as P_GPS; Step 2: For D_Image and D_Radar, if there is an overlap in the results they output, fuse them according to the timestamp and spatial position to obtain the fused road condition result D_Fusion; Step 3: Combine O_Image with D_Fusion to form a comprehensive analysis result including road damage, obstacles, and object dynamics; Step 4: Combine P_GPS with the comprehensive analysis result to form a comprehensive road inspection report including road conditions and vehicle positions; Step 5: If there are multiple time periods or location points, perform the above fusion process for each time period or location point respectively to obtain their respective comprehensive reports; Step 6: According to the comprehensive reports of all time periods or location points, adopt data fusion and visualization technologies to further integrate and analyze the road inspection information to obtain the final comprehensive road inspection report.
[0012] Preferably, the specific steps of adopting data fusion and visualization technologies to further integrate and analyze the road inspection information to obtain the final comprehensive road inspection report include: Step 601: Integrate the comprehensive reports of each time period or location point to form a unified data format, including road damage types, obstacle positions, object dynamics information, and vehicle position information; perform interpolation processing on missing data and filter or correct abnormal data; Step 602: Adopt a spatio-temporal correlation analysis algorithm to perform spatio-temporal correlation analysis on the integrated data to identify the distribution laws and change trends of road damage, obstacles, and object dynamics in time and space; the spatio-temporal correlation analysis algorithm uses a spatio-temporal autocorrelation function; Step 603: Based on the spatio-temporal correlation analysis results, conduct risk assessments on road damages, obstacles, and object dynamics, and calculate their impact on road safety and traffic efficiency; according to the risk assessment results, rank the priorities of road inspection tasks to determine the road segments or location points that need to be processed first. Step 604: Use visualization technology to generate a final comprehensive road inspection report for the integrated and analyzed road inspection information, including road damage distribution maps, obstacle location maps, object dynamic change maps, as well as risk assessment and priority ranking results; the report is presented in various forms such as charts, maps, and animations.
[0013] Preferably, the multi-modal fusion algorithm further includes a weight assignment mechanism, which assigns different weights to the output results of the road image analysis model, radar data analysis model, and GPS positioning and map matching model according to the confidence or accuracy output by each model.
[0014] Preferably, the data acquisition unit further includes a data preprocessing module, which is responsible for cleaning, denoising, and normalizing the collected road image data, radar detection data, and GPS positioning data.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This system realizes all-round and multi-dimensional monitoring of the road environment by integrating various technical means such as image data analysis, radar data analysis, and GPS positioning and map matching. The image data analysis unit can accurately identify road damages and obstacles, the radar data analysis unit can predict the dynamics of objects ahead on the road, and the positioning and map matching unit ensures the accuracy of the vehicle's position. The multi-modal fusion technology enables the advantages of these single technical means to complement each other, effectively improving the accuracy and comprehensiveness of road inspections.
[0016] The effect of traditional road inspection methods will be greatly reduced in case of bad weather or poor lighting conditions. However, through the multi-modal fusion technology, this system can maintain a high inspection efficiency even in a complex and changeable road environment. The collaborative work of the image data analysis unit and the radar data analysis unit enables the system to accurately identify the road conditions and predict the dynamics of objects ahead even when the line of sight is restricted or the lighting is insufficient. This system designs a dedicated data acquisition unit, which is responsible for collecting road image data, radar detection data, and GPS positioning data, and transmitting these data into the corresponding analysis models respectively. This modular design makes the data processing flow more concise and efficient, avoiding redundancy and repetition in data processing and improving the overall processing efficiency of the system.
[0017] The multimodal fusion unit is the core part of this system. It defines an advanced multimodal fusion algorithm to perform fusion analysis on the output results from different analysis models. By continuously optimizing the fusion algorithm, this system can more accurately integrate various information, generate a more precise comprehensive road inspection report, and provide strong support for road maintenance and safety management. By integrating a variety of advanced technologies, the system can automatically complete road inspection tasks, reduce the dependence on manual inspections, lower the inspection cost, and improve the timeliness and efficiency of inspections. Brief Description of the Drawings
[0018] Figure 1 It is the working principle diagram of the road inspection data processing and analysis system based on multimodal fusion according to the present invention; Figure 2 It is the step diagram for training the GPS positioning and map matching model; Figure 3 It is the main step diagram for generating the comprehensive report. Detailed Embodiments
[0019] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
[0020] Please refer to Figures 1-3 , the present invention provides a technical solution: a road inspection data processing and analysis system based on multimodal fusion, and the system includes: Image data analysis unit: The image data analysis unit is responsible for constructing a road image analysis model. This model uses advanced image recognition algorithms to train and learn from road image data. Through a large number of labeled road damage and obstacle image samples, the model can learn the characteristics of road damage and obstacles, so as to accurately identify road damage and obstacles during actual inspections. This unit receives road image data from the data acquisition unit and outputs image analysis results, including information such as damage type, location, and size.
[0021] Radar data analysis unit: The radar data analysis unit is responsible for constructing a radar data analysis model. This model uses the dynamic data of objects in front of the road collected by radar sensors, and through signal processing and machine learning algorithms, classifies and predicts the radar data. The model can identify dynamic objects such as vehicles, pedestrians, and animals in front of the road and predict their movement trajectories. This unit receives radar detection data from the data acquisition unit and outputs radar analysis results, including information such as object type, location, speed, and movement direction.
[0022] Positioning and Map Matching Unit: The positioning and map matching unit is responsible for constructing a GPS positioning and map matching model based on graph theory algorithms. This model combines GPS positioning data and map information to determine the precise location of the vehicle. When GPS signals are blocked or interfered with, the model can utilize the road network information on the map and historical driving trajectories to correct and optimize the GPS positioning results. This unit receives GPS positioning data from the data acquisition unit and outputs positioning and map matching results, including information such as the precise location of the vehicle, driving direction, and the road it is on.
[0023] Data Acquisition Unit: The data acquisition unit is responsible for collecting road image data, radar detection data, and GPS positioning data. This unit uses cameras, radar sensors, and GPS receivers installed on the inspection vehicle to collect road environment data in real time. The collected data is preprocessed and then transmitted to the road image analysis model, radar data analysis model, and GPS positioning and map matching model respectively, providing data support for subsequent fusion analysis.
[0024] Multimodal Fusion Unit: The multimodal fusion unit is responsible for defining multimodal fusion algorithms and performing fusion analysis on the output results from the image data analysis unit, radar data analysis unit, and positioning and map matching unit. Through fusion analysis, the system can obtain a more comprehensive and accurate road inspection report, including road damage conditions, dynamics of objects ahead, vehicle position information, etc. The fused report can provide strong support for road maintenance and management.
[0025] The present invention will be further described below in conjunction with Embodiments 1 to 4:
[0026] Embodiment 1:
[0027] The road image analysis model adopts a convolutional neural network (CNN) architecture, aiming to accurately identify road damage types and obstacle positions. The specific structure of this model includes: Input Layer: Receives road image data from the data acquisition unit. After preprocessing the image data (such as scaling and normalization), it serves as the input to the model.
[0028] Convolutional Layer: Designs multiple convolutional kernels, each of which is responsible for extracting specific features in the image. Through convolutional operations, multiple feature maps are generated. These feature maps contain low-level features such as edges and textures of the image, as well as high-level features such as more complex shapes and structures.
[0029] Pooling Layer: After the convolutional layer, a pooling layer is set to perform downsampling on the feature maps. Adopts a max-pooling or average-pooling strategy to reduce the size of the feature maps, thereby reducing the computational amount while retaining important features.
[0030] Fully connected layer: Flatten the feature map output by the pooling layer into a one-dimensional vector and connect it to the fully connected layer. The fully connected layer maps the feature vector to the output space through linear transformation and non-linear activation functions (such as ReLU).
[0031] Output layer: Design the output layer according to the task requirements. For classification tasks, the output layer uses the softmax activation function to output the probability distribution of road damage types; for detection tasks, the output layer directly outputs the position coordinates of obstacles (such as the coordinates of the bounding box).
[0032] During the training process, for classification tasks, the cross-entropy loss function is used as the optimization objective to measure the difference between the probability distribution predicted by the model and the true label; for detection tasks, the Smooth L1 loss function is used to measure the position difference between the bounding box predicted by the model and the true bounding box. Through the backpropagation algorithm and the gradient descent optimizer, the model parameters are iteratively updated until the loss function converges.
[0033] For example, during the training process, for a road image dataset containing various damage types such as cracks, potholes, and foreign objects, the model can learn the features of various damages and accurately identify the damage types and locations on the test set.
[0034] The radar data analysis model is constructed using the Recurrent Neural Network (RNN) algorithm, especially the Long Short-Term Memory (LSTM) network, to predict the dynamics of objects in front of the road. The specific structure of the model is as follows: Input layer: Receive radar detection data from the data acquisition unit. The data is input in sequence, with a sequence length of T and a feature dimension of R at each time step, including initial information such as the distance and speed of the object.
[0035] Hidden layer: Construct multiple layers of LSTM units, and each layer of LSTM units is followed by a Dropout layer to prevent overfitting. The LSTM units can capture long-term dependencies in the sequence and perform temporal modeling on the radar data.
[0036] Output layer: After the hidden layer, set the output layer to output the dynamic prediction results of the objects in front of the road, including the distance, speed, and acceleration at future times. The output layer can use a linear activation function to directly output continuous values.
[0037] During the training process, the model learns the temporal patterns in the radar data sequence to predict the future motion state of the object. For example, for a moving vehicle, the model can predict its future driving path and speed changes based on the historical trajectory. Loss functions such as the Mean Squared Error (MSE) are used as the optimization objective to measure the difference between the predicted value and the true value of the model. Through the backpropagation algorithm and an appropriate optimizer (such as Adam), the model parameters are iteratively updated until the loss function reaches the minimum value.
[0038] Example 2:
[0039] The GPS positioning and map matching model is used to determine the precise position of a vehicle on a map. The structure and training process of this model will be described in detail below: The structure of the GPS positioning and map matching model includes: Input layer: The input layer receives two types of data: GPS positioning data and map data. The GPS data includes longitude, latitude, and altitude, which are provided in real time by an in-vehicle GPS receiver and represent the current position of the vehicle. The map data includes road network and node information. The road network consists of a series of nodes and the edges connecting these nodes. Each node represents a fixed position on the road, and the edges represent the connection relationship and distance between the nodes.
[0040] Matching layer: The matching layer uses the Dijkstra algorithm, which is a classic algorithm for finding the shortest path from a starting point to an ending point in a graph. In this model, the Dijkstra algorithm is used to match the GPS data to the nearest node on the map. The algorithm considers the distance between the GPS position and the map nodes, as well as the topological structure of the road network, to find the map node closest to the GPS position. During the matching process, the algorithm calculates the estimated distance from each map node to the GPS position and selects the node with the minimum distance as the matching result.
[0041] Output layer: The output layer outputs the precise position information of the vehicle, including the road and node number where it is located. This information can be used for subsequent multimodal fusion analysis and to provide accurate vehicle position indication for road inspection personnel. Steps for training the GPS positioning and map matching model Prepare the training set: Use historical GPS positioning data and the corresponding map data as the training set. Select the precise position of the vehicle (i.e., the actual node on the map) as the label to supervise the learning process of the model.
[0042] Initialize the algorithm parameters: Initialize the parameters of the Dijkstra algorithm, including the weights of the nodes (which can represent the importance or access cost of the nodes) and the lengths of the edges (representing the distances between the nodes). These parameters will be adjusted according to the optimization results during the training process.
[0043] Train the graph theory algorithm: Use the training set to train the Dijkstra algorithm. Optimize the algorithm parameters by minimizing the matching error (i.e., the actual distance between the GPS position and the matched node). During the training process, the algorithm will continuously adjust the node weights and edge lengths to improve the matching accuracy and efficiency.
[0044] Verify model performance: Use the validation set to verify the trained algorithm. Evaluate the positioning accuracy of the model (i.e., the consistency between the matched nodes and the actual positions) and the matching efficiency (i.e., the execution speed of the algorithm). If the model performance does not meet the preset standard, return to the training step to continue optimizing the algorithm parameters.
[0045] Save model parameters: When the model performance reaches the preset standard, save the parameters of the trained algorithm. Obtain the trained GPS positioning and map matching model, which can be used for actual road inspection data processing and analysis.
[0046] Specific example: Suppose there is a city road network, including multiple intersections and road segments. The on-vehicle GPS receiver provides a set of GPS positioning data indicating the current position of the vehicle in the city. The map data contains detailed information about these intersections and road segments. Through the trained GPS positioning and map matching model, the system can match the GPS data to the nearest node on the map, such as an intersection. The output layer will provide the accurate position information of the vehicle, including the road it is on (e.g., "XX Road") and the node number (e.g., "Intersection 123"). In this way, road inspection personnel can accurately understand the vehicle's position based on this information and thus carry out inspection tasks more effectively. At the same time, this model can also be fused and analyzed with other modality data (such as image data and radar data) to further improve the accuracy and efficiency of road inspection.
[0047] Example 3:
[0048] The multimodal fusion unit is responsible for effectively fusing data from different models to generate a comprehensive road inspection report. The following are the specific implementation steps and detailed descriptions of the multimodal fusion algorithm: Step 1: Define output variables: Define the road damage output by the road image analysis model as D_Image, and the obstacle as O_Image.
[0049] Define the object dynamics output by the radar data analysis model as D_Radar. Define the vehicle position output by the GPS positioning and map matching model as P_GPS. These output variables represent data of different modalities and are the basis of the fusion algorithm.
[0050] Step 2: Fuse road damage and object dynamics: For D_Image and D_Radar, check whether there is an overlap in the results output by both. The overlap judgment is based on the timestamp and spatial position. If the two results are within the same time period and the spatial positions are close, they are considered to overlap. According to the overlap situation, D_Image and D_Radar are fused to obtain the fused road condition result D_Fusion. The fusion process can consider factors such as the credibility and accuracy of different modality data, and adopt weighted average, voting mechanism or other fusion strategies.
[0051] Specific example: Suppose the road image analysis model detects a pothole in a certain section of the road, and the radar data analysis model also detects the movement of an object in that section (possibly a vehicle avoiding the pothole). According to the timestamp and spatial position, these two results are considered to overlap. After fusion, D_Fusion will contain the information of "pothole and possible vehicle avoidance".
[0052] Step 3: Combine obstacle information: Combine O_Image (obstacle information) with D_Fusion (fused road condition). The combination process considers factors such as the type, position, size of the obstacles, and their impact on the road condition. Form a comprehensive analysis result that includes road damage, obstacles, and object dynamics.
[0053] Specific example: If O_Image detects scattered gravel on the road surface, and D_Fusion already contains pothole information, then the comprehensive analysis result will contain the information of "pothole and scattered gravel" at the same time.
[0054] Step 4: Combine vehicle position information: Combine P_GPS (vehicle position) with the comprehensive analysis result. The combination process considers factors such as the position, driving direction, speed of the vehicle, and their relationship with the road condition. Form a comprehensive road inspection report that includes the road condition and vehicle position.
[0055] Specific example: If the vehicle's current position is close to an intersection, and the comprehensive analysis result shows that there are potholes and scattered gravel near the intersection, then the comprehensive report will contain this information and may indicate that the vehicle needs to slow down or detour.
[0056] Step 5: Process multiple time periods or position points: If there is data for multiple time periods or position points, the above fusion process is performed separately for each time period or position point. Obtain their respective comprehensive reports, and each report reflects the road condition and vehicle position at a specific time period or position point.
[0057] Step 6: Integrate and analyze the final report: Adopt data fusion and visualization techniques based on comprehensive reports for all time periods or location points. Further integrate and analyze road inspection information, considering the relevance and changing trends between different time periods or location points. Obtain the final comprehensive road inspection report, which provides a comprehensive, accurate, and real-time description of the road conditions.
[0058] By integrating data from different time periods, it is possible to detect the trend of a pothole gradually expanding or the traffic congestion at an intersection during peak hours. This information is of great significance for road maintenance and traffic management.
[0059] Embodiment 4: The present invention further integrates and analyzes road inspection information by using data fusion and visualization techniques to obtain the final comprehensive road inspection report. The specific steps include: Step 601: Data integration and preprocessing: Integrate the comprehensive reports for each time period or location point to form a unified data format. This data includes road damage types (such as potholes, cracks, etc.), obstacle positions (such as debris, construction signs, etc.), object dynamic information (such as vehicle avoidance, pedestrian crossing, etc.), and vehicle position information. Perform interpolation processing on missing data to estimate missing values; filter or correct abnormal data to eliminate the influence of errors or outliers. For example, for missing road damage data in a certain time period, linear interpolation can be performed based on data from adjacent time periods; for obviously incorrect obstacle position information, it can be corrected according to data from other modalities.
[0060] Step 602: Spatiotemporal correlation analysis: Perform spatiotemporal correlation analysis on the integrated data to identify the distribution patterns and changing trends of road damage, obstacles, and object dynamics in time and space. As the core of the spatiotemporal correlation analysis algorithm, the spatiotemporal autocorrelation function can measure the data correlation between different time points and spatial positions. By calculating the spatiotemporal autocorrelation function, it is possible to discover characteristics such as the clustering, diffusion, or periodicity of road damage, obstacles, and object dynamics in spatiotemporal domains.
[0061] Step 603: Risk assessment and prioritization: Based on the results of spatiotemporal correlation analysis, conduct risk assessment on road damage, obstacles, and object dynamics, and calculate their impact on road safety and traffic efficiency. For example, the risk level to road safety can be evaluated according to factors such as the severity of road damage, the type and position of obstacles, and the frequency of object dynamics. According to the risk assessment results, prioritize road inspection tasks to determine the road segments or location points that need to be processed first. The prioritization can consider factors such as risk level, road importance, and traffic flow to ensure the reasonable allocation and efficient utilization of road inspection resources.
[0062] Step 604: Visual Report Generation: Integrate and analyze the road inspection information, including road damage distribution maps, obstacle location maps, object dynamic change maps, and risk assessment and prioritization results. Use various forms such as charts (e.g., bar charts, pie charts, etc.), maps (e.g., electronic maps, heat maps, etc.), and animations to display the road inspection information. Through visualization technology, the distribution, change trends, and risk assessment results of road damage, obstacles, and object dynamics can be intuitively displayed, providing intuitive decision-making support for road maintenance and traffic management.
[0063] The multimodal fusion algorithm also includes a weight assignment mechanism, specifically including: Weight Assignment Mechanism: In the multimodal fusion algorithm, according to the confidence or accuracy of the output of each model, different weights are assigned to the output results of the road image analysis model, radar data analysis model, and GPS positioning and map matching model.
[0064] Confidence or Accuracy Evaluation: The confidence or accuracy of each model can be evaluated through methods such as historical data verification and model performance evaluation. For example, metrics such as accuracy and recall of each model on historical data can be calculated as the basis for weight assignment.
[0065] Weight Assignment: According to the evaluation results, corresponding weights are assigned to the output results of each model. The output results of models with higher confidence or accuracy will be given larger weights and play a greater role in the fusion process; while the output results of models with lower confidence or accuracy will be given smaller weights to reduce their impact on the fusion result.
[0066] Suppose during the inspection of a certain road section, the road image analysis model detects cracks on the road surface, the radar data analysis model detects vehicle avoidance behavior, and the GPS positioning and map matching model provides vehicle location information. During the data fusion process, first, the output results of each model are integrated and preprocessed to form a unified data format. Then, a spatio-temporal correlation analysis algorithm is used to analyze the integrated data, and it is found that the cracks show a diffusive characteristic in time and space, and there is a correlation between the vehicle avoidance behavior and the crack location. Next, according to the risk assessment results, it is determined that there is a high safety risk in this road section and it needs to be processed preferentially. Finally, a comprehensive road inspection report is generated through visualization technology, showing the crack distribution map, vehicle avoidance behavior change map, and risk assessment and prioritization results. At the same time, during the multimodal fusion process, different weights are assigned to the output results of the road image analysis model, radar data analysis model, and GPS positioning and map matching model according to the confidence or accuracy of each model to ensure the accuracy and reliability of the fusion result.
[0067] It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or apparatus comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or apparatus.
[0068] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. The road inspection data processing and analysis system based on multimodal fusion is characterized by: The system comprises: An image data analysis unit, which constructs a road image analysis model for identifying road damage and obstacles; A radar data analysis unit, which builds a radar data analysis model to predict the dynamics of objects ahead on the road; The positioning and map matching unit builds a GPS positioning and map matching model based on graph theory algorithms. The model uses GPS data and map information to determine the precise location of the vehicle; A data acquisition unit collects road image data, radar detection data, and GPS positioning data, and transmits the collected data to a road image analysis model, a radar data analysis model, and a GPS positioning and map matching model, respectively, to obtain the output results of the models; The multimodal fusion unit defines the multimodal fusion algorithm, performs fusion analysis on the model output results, and obtains a comprehensive road inspection report.
2. The road inspection data processing and analysis system based on multimodal fusion according to claim 1 is characterized in that: The road image analysis model adopts a convolutional neural network (CNN) architecture, including: Input layer: receives road image data; Convolution layer: extract image features through multiple convolution kernels to generate feature maps; Pooling layer: downsamples the feature map to reduce the amount of calculation; Fully connected layer: flattens the feature map and connects it to the output layer; Output layer: Outputs road damage type and obstacle location for classification and detection tasks.
3. The road inspection data processing and analysis system based on multimodal fusion according to claim 2 is characterized in that: The road image analysis model uses a cross entropy loss function as a loss function for classification tasks and a Smooth L1 loss function as a loss function for detection tasks.
4. The road inspection data processing and analysis system based on multimodal fusion according to claim 1 is characterized in that: The radar data analysis model is constructed using a recurrent neural network (RNN) algorithm, including: Input layer: receives radar detection data, the sequence length is T, and the feature dimension of each time step is R; Hidden layer: contains multiple layers of LSTM units, each layer is followed by a Dropout layer to prevent overfitting; Output layer: Outputs the dynamic prediction results of objects in front of the road, including distance, speed and acceleration.
5. The road inspection data processing and analysis system based on multimodal fusion according to claim 4 is characterized in that: The structure of the GPS positioning and map matching model includes: Input layer: receives GPS positioning data and map data. GPS data includes longitude, latitude and altitude. Map data includes road network and node information. Matching layer: Use Dijkstra algorithm to match GPS data to the nearest node on the map; Output layer: Outputs the precise location information of the vehicle, including the road and node number.
6. The road inspection data processing and analysis system based on multimodal fusion according to claim 5 is characterized in that: The steps to train the GPS positioning and map matching model include: Use GPS positioning data and corresponding map data as training sets, and select the precise location of the vehicle as the label; Initialize the parameters of the graph theory algorithm, including the weight of the node and the length of the edge; Use the training set to train the graph theory algorithm and optimize the algorithm by minimizing the matching error; Use the validation set to verify the graph theory algorithm and evaluate the positioning accuracy and matching efficiency of the model; When the model performance reaches the preset standard, the model parameters are saved to obtain the trained GPS positioning and map matching model.
7. The road inspection data processing and analysis system based on multimodal fusion according to claim 1 is characterized in that: The implementation steps of the multimodal fusion algorithm defined in the multimodal fusion unit include: Step 1: Define the road damage output by the road image analysis model as D_Image, the obstacle as O_Image, the object dynamics output by the radar data analysis model as D_Radar, and the vehicle position output by the GPS positioning and map matching model as P_GPS; Step 2: For D_Image and D_Radar, if the output results of the two overlap, they are fused according to the timestamp and spatial position to obtain the fused road condition result D_Fusion; Step 3: Combine O_Image with D_Fusion to form a comprehensive analysis result including road damage, obstacles and object dynamics; Step 4: Combine P_GPS with comprehensive analysis results to form a comprehensive road inspection report including road conditions and vehicle locations; Step 5: If there are multiple time periods or location points, the above fusion process is performed for each time period or location point to obtain a comprehensive report for each time period or location point; Step 6: Based on the comprehensive reports of all time periods or location points, data fusion and visualization technology are used to further integrate and analyze the road inspection information to obtain the final comprehensive road inspection report.
8. The road inspection data processing and analysis system based on multimodal fusion according to claim 7 is characterized in that: The data fusion and visualization technology is used to further integrate and analyze the road inspection information to obtain the final comprehensive road inspection report. The specific steps include: Step 601: Integrate the comprehensive reports of each time period or location point to form a unified data format, including road damage type, obstacle location, object dynamic information and vehicle location information; interpolate missing data and filter or correct abnormal data; Step 602: using a spatiotemporal correlation analysis algorithm to perform spatiotemporal correlation analysis on the integrated data to identify the distribution patterns and change trends of road damage, obstacles and object dynamics in time and space; the spatiotemporal correlation analysis algorithm uses a spatiotemporal autocorrelation function; Step 603: Based on the results of the spatiotemporal correlation analysis, a risk assessment is performed on the road damage, obstacles and object dynamics, and the degree of their impact on road safety and traffic efficiency is calculated; based on the risk assessment results, the road inspection tasks are prioritized to determine the road sections or locations that need to be processed first; Step 604: The integrated and analyzed road inspection information, including the road damage distribution map, obstacle location map, object dynamic change map, and risk assessment and priority ranking results, is used to generate a final comprehensive road inspection report using visualization technology; the report is displayed in multiple forms such as charts, maps, and animations.
9. The road inspection data processing and analysis system based on multimodal fusion according to claim 7 is characterized in that: The multimodal fusion algorithm also includes a weight allocation mechanism, which allocates different weights to the output results of the road image analysis model, the radar data analysis model, and the GPS positioning and map matching model according to the confidence or accuracy of the output of each model.
10. The road inspection data processing and analysis system based on multimodal fusion according to claim 1, characterized in that: The data acquisition unit also includes a data preprocessing module, which is responsible for cleaning, denoising and standardizing the collected road image data, radar detection data and GPS positioning data.