Intelligent road management and maintenance system and method based on multi-model fusion

Abstract

The invention provides an intelligent road management and maintenance system and method based on multi-model fusion, and relates to the technical field of road maintenance and intelligent traffic. The method comprises the following steps: acquiring road condition data and positioning data through a multi-source data acquisition module; the disease recognition module is combined with a neural network model to analyze the multi-modal data, so that the disease detection and recognition precision is improved; the big data processing early warning module mines a disease rule through a big data model, and realizes disease early warning and predictive maintenance; the digital twin modeling module is used for constructing a road digital twin model and realizing real-time monitoring and simulation analysis; and the decision scheduling module realizes automatic maintenance scheduling based on the output of the modules. Through multi-model deep fusion, the problems of high manual dependence, low precision, lack of predictability and scheduling lag in traditional road management and maintenance are solved, the road maintenance efficiency, quality and intelligent level are remarkably improved, and the maintenance cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of road engineering maintenance technology, specifically to an intelligent road management system and method based on multi-model fusion. Background Technology

[0002] As a core component of transportation infrastructure, the quality of road maintenance directly impacts traffic safety and transportation efficiency. Traditional road maintenance methods rely on manual inspections, experience-based judgment, and post-incident repairs, which have the following significant drawbacks:

[0003] The detection efficiency is low and the accuracy is poor: manual inspection requires maintenance personnel to observe and measure on-site, and each person can only inspect 5-10 kilometers of road per day, which is inefficient; moreover, the identification of defects is greatly affected by personnel experience, weather (visibility is obstructed on rainy or foggy days), and lighting. The missed detection rate of hidden defects such as cracks and small potholes is as high as 20% or more, and the misjudgment rate is over 15%.

[0004] Lack of predictive maintenance: Traditional maintenance adopts a passive "repair after damage occurs" model, without analyzing and predicting the development pattern of damage. For example, if road surface cracks are not treated in time, they may develop into potholes within 3 months, leading to an increase in maintenance costs of 30%-50% and affecting the service life of the road.

[0005] Lack of real-time monitoring capabilities: The road condition cannot be obtained in real time, and sudden defects (such as road subsidence caused by heavy rain) are difficult to detect in time, which can easily lead to traffic accidents; moreover, the maintenance effect lacks real-time evaluation methods, making it impossible to adjust maintenance strategies in a timely manner.

[0006] Low scheduling efficiency: The allocation of maintenance resources (teams, equipment, materials) relies on manual planning and does not take into account the urgency of the defects, the importance of the road, and the location of the resources, resulting in delays in the maintenance of high-priority defects (average delay time ≥ 4 hours) and a resource utilization rate of less than 60%.

[0007] While some existing technologies attempt to introduce single models (such as neural network-based defect identification and simple data statistics-based early warning), they fail to achieve deep integration of multiple models. For example, defect identification relies solely on image data, ignoring the influence of road structure and environmental factors; early warning models do not incorporate digital twins for visualized monitoring; and scheduling modules do not link prediction results with real-time resource status. Therefore, a multi-model fusion intelligent solution is urgently needed to address the challenges of intelligent, precise, and efficient traditional maintenance management. Summary of the Invention

[0008] The purpose of this application is to overcome the shortcomings of existing road maintenance technologies, such as strong reliance on manual labor, low accuracy, lack of predictability, and lag in scheduling, and to provide an intelligent road maintenance system and method based on multi-model fusion. Through the deep integration of models such as neural networks, big data, and digital twins, it can achieve intelligent maintenance throughout the entire process of accurate identification of road defects, early warning, real-time monitoring, and intelligent scheduling.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0010] Firstly, a smart road maintenance method based on multi-model fusion is provided, including the following steps:

[0011] Collect road surface condition data and location data to obtain multimodal raw data;

[0012] By combining a neural network model to analyze the multimodal raw data, the detection and identification of road defects can be achieved, and the defect identification results can be obtained.

[0013] By using big data models to mine the correlation patterns between the disease identification results and historical maintenance data, road disease early warning and predictive maintenance suggestions can be generated.

[0014] A digital twin model of the road is constructed, and the multimodal raw data, disease identification results and early warning information are mapped to the digital twin model to realize real-time monitoring and simulation analysis of road maintenance;

[0015] Based on the disease identification results, predictive maintenance suggestions, and real-time monitoring data from the digital twin model, automated scheduling of road maintenance is achieved.

[0016] As a further improvement to the technical solution of the present invention, the collection of road condition data and positioning data specifically includes:

[0017] Deploy data acquisition equipment, which includes a high-definition industrial camera, a laser road surface detector, a multi-parameter environmental sensor, and a GNSS dual-mode positioning module;

[0018] The high-definition industrial camera collects road surface appearance data, the laser road surface detector collects road surface structure data, the multi-parameter environmental sensor collects temperature, humidity and rainfall data, and the GNSS dual-mode positioning module collects latitude, longitude and elevation data of each monitoring point on the road surface.

[0019] The collected multi-source data is preprocessed, including denoising using a Gaussian filtering algorithm, aligning the multi-source data based on the positioning data, and normalizing the data to the [0,1] interval to obtain the multimodal raw data.

[0020] As a further improvement to the technical solution of the present invention, the analysis of the multimodal raw data using a neural network model specifically includes:

[0021] A multimodal fusion neural network model is constructed, which includes a convolutional neural network (CNN) sub-model, a Transformer sub-model, and an attention mechanism fusion sub-model.

[0022] The CNN sub-model extracts features from the road surface appearance data and outputs an appearance feature vector. The Transformer sub-model is used to extract features from the time-series data of the pavement structure (deflection values, crack depth time series), and output a structural feature vector. ;

[0023] F1 is calculated using the attention mechanism fusion sub-model. and The weights α are used to obtain the fused feature vector. , where α is the attention weight, which is optimized by training the model using historical disease data;

[0024] The fused feature vector is input into a fully connected layer and a Softmax classifier, and the output disease type (crack, pothole, rut, settlement) and disease level (Level IV) are obtained, which is the disease identification result, and the disease identification accuracy is ≥95%.

[0025] As a further improvement to the technical solution of the present invention, the step of mining the correlation between the disease identification results and historical maintenance data through a big data model specifically includes:

[0026] Construct a disease database to store the disease identification results, multimodal raw data, historical maintenance records (maintenance time, method, cost), and environmental data;

[0027] A disease development prediction model is constructed using a Long Short-Term Memory (LSTM) network. The prediction formula of the model is S(t+Δt) = S(t)·e^(k·Δt + ε), where S(t) is the severity of the disease at time t (based on disease level quantification, level I = 1, level V = 5), S(t+Δt) is the predicted severity at time t+Δt, k is the disease development rate coefficient, Δt is the prediction time interval, and ε is the error correction term with |ε|≤0.05.

[0028] The k is obtained through training on historical data and satisfies... , The basic rate of disease development (such as cracks) =0.02 / day), T is the average daily ambient temperature (°C), R is the average daily rainfall (mm), and a and b are environmental impact coefficients (trained through linear regression, such as a=-0.001, b=0.003).

[0029] Set an early warning threshold S_threshold (e.g., S_threshold=4 for level IV disease). When S (t+Δt)≥S_threshold, an early warning is triggered, and predictive maintenance suggestions (maintenance time, material selection, and work process) are generated.

[0030] As a further improvement to the technical solution of the present invention, the construction of the digital twin model of the road specifically includes:

[0031] Based on the integration technology of Building Information Modeling (BIM) and Geographic Information System (GIS), an initial digital twin model of the road is constructed. The model includes a road geometry sub-model (length, width, slope), a structural sub-model (base layer thickness, surface layer material parameters), and ancillary facility sub-models (manhole covers, road signs, drainage system).

[0032] A real-time data mapping channel is established, and the multimodal raw data, disease identification results and early warning information are transmitted to the digital twin model through the MQTT protocol to achieve synchronous updates of physical roads and digital models, with a synchronization delay of ≤10s;

[0033] Based on the aforementioned digital twin model, a maintenance simulation analysis was conducted. The simulation formula is σ_after = σ_before·(1+ β·D), where σ_before is the pavement strength before maintenance (MPa), σ_after is the pavement strength after maintenance, β is the repair effect coefficient (β=0.6-0.8 for asphalt repair, β=0.7-0.9 for concrete repair), and D is the maintenance operation depth (cm).

[0034] The digital twin model is displayed through a visual interface, allowing users to view disease details, environmental parameters, and simulation results by clicking with the mouse, enabling real-time monitoring.

[0035] As a further improvement to the technical solution of the present invention, the automated scheduling of road maintenance specifically includes: acquiring input data: the disease identification results (type, level, location), predictive maintenance suggestions, real-time road status of the digital twin model, and maintenance resource information (location of maintenance teams, equipment type, and material inventory).

[0036] Calculate maintenance priority using the following formula: ,in , , As weight and (Main road) =0.4, =0.4, =0.2; branch =0.2, =0.5, =0.3), L is the road importance level (level 1-5, main road = 5), S is the severity of the damage (level 1-5), and D is the distance between the damage and the maintenance base (normalized to 1-5, the closer = 1).

[0037] Based on priority, assign the nearest maintenance team to high-priority defects and generate scheduling instructions (including defect location, required equipment, material usage, and estimated duration).

[0038] Receive feedback from maintenance teams on their work (completion time, actual materials consumed, maintenance results), and update the disease database and digital twin model.

[0039] The second aspect of the present invention provides an intelligent road maintenance system based on multi-model fusion, including a data acquisition module, a disease identification module, a big data processing and early warning module, a digital twin modeling module, and a decision scheduling module;

[0040] The multi-source data acquisition module is used to collect road condition data and positioning data, and output multimodal raw data;

[0041] The disease identification module is used to load a neural network model, analyze the multimodal raw data, and output disease identification results;

[0042] The big data processing and early warning module is used to build a disease database, mine disease patterns through big data models, and output early warning information and predictive maintenance suggestions.

[0043] The digital twin modeling module is used to construct a digital twin model of the road, map real-time data, perform simulation analysis, and output real-time monitoring results.

[0044] The decision-making and scheduling module is used to receive the disease identification results, predictive maintenance suggestions, and real-time monitoring results to realize automated scheduling of road maintenance.

[0045] As a further improvement to the technical solution of the present invention, the multi-source data acquisition module includes an acquisition device submodule and a data preprocessing submodule;

[0046] The data acquisition equipment submodule includes a high-definition industrial camera (resolution ≥ 20 million pixels, frame rate ≥ 25fps), a laser road surface detector (detection accuracy ±0.1mm), a multi-parameter environmental sensor (temperature range -30~70℃, humidity range 0~100% RH), and a GNSS dual-mode positioning module (positioning accuracy ≤ 1m).

[0047] The data preprocessing submodule incorporates Gaussian filtering, data alignment, and normalization algorithms to perform noise reduction, correlation, and normalization on the collected data.

[0048] A third aspect of the present invention provides a computer device including a memory and a processor, the memory storing a computer program and the processor being configured to execute the computer program to implement the intelligent road maintenance method based on multi-model fusion as described above.

[0049] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the intelligent road maintenance method based on multi-model fusion as described above.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] This invention's technical solution achieves multi-dimensional efficiency improvements in road maintenance through deep multi-model fusion: It not only enhances the accuracy of defect identification to ≥95% and controls the false negative rate to ≤5% by combining road surface images, structural data, and environmental data with a multi-modal fusion neural network, but also increases the daily road inspection mileage from the traditional 5-10 kilometers to over 500 kilometers, a 50-fold increase in efficiency, thanks to automated data acquisition and identification processes. Furthermore, it can mine defect development patterns based on LSTM models, predicting defect evolution trends 7-90 days in advance, effectively preventing defect expansion, reducing maintenance costs by 30%-50%, and extending road lifespan by 2-3 years. In addition, it achieves real-time synchronization between the physical road and the digital model using a digital twin model (synchronization delay ≤10s), supporting visualized monitoring and maintenance scheme simulation, reducing the response time to sudden defects from the traditional 4 hours to within 10 minutes. Simultaneously, it optimizes maintenance resource allocation through automated priority calculation and greedy algorithms, increasing maintenance resource utilization from 60% to 85% and reducing the delay rate of high-priority defect maintenance from 30% to 5%. The following will ultimately construct a fully intelligent closed loop encompassing data collection, disease identification, early warning and prediction, real-time monitoring, intelligent scheduling, and feedback updates, significantly reducing reliance on manual labor, lowering the workload of maintenance personnel, and comprehensively improving the intelligence, precision, and efficiency of road maintenance. Attached Figure Description

[0052] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0053] Figure 1 This is a schematic diagram of the framework of an intelligent road maintenance method based on multi-model fusion according to an embodiment of the present invention;

[0054] Figure 2 This is a modular structure diagram of an intelligent road maintenance system based on multi-model fusion according to an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the composition of a computing device according to an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0057] The present invention will be further described in detail below with reference to the accompanying drawings.

[0058] Reference Figure 1 Firstly, a smart road maintenance method based on multi-model fusion is provided, comprising the following steps:

[0059] Collect road surface condition data and location data to obtain multimodal raw data;

[0060] By combining a neural network model to analyze the multimodal raw data, the detection and identification of road defects can be achieved, and the defect identification results can be obtained.

[0061] By using big data models to mine the correlation patterns between the disease identification results and historical maintenance data, road disease early warning and predictive maintenance suggestions can be generated.

[0062] A digital twin model of the road is constructed, and the multimodal raw data, disease identification results and early warning information are mapped to the digital twin model to realize real-time monitoring and simulation analysis of road maintenance;

[0063] Based on the disease identification results, predictive maintenance suggestions, and real-time monitoring data from the digital twin model, automated scheduling of road maintenance is achieved.

[0064] This invention provides the core framework of an intelligent road maintenance method based on multi-model fusion, achieving intelligent maintenance throughout the entire process through five key steps. First, road surface condition data and location data are collected, integrating multi-dimensional information to form multimodal raw data, providing the basic input for subsequent analysis. Second, a neural network model is introduced to fuse and analyze the multimodal raw data, leveraging the processing advantages of different models for different types of data to accurately identify road defects and their levels, outputting defect identification results. Third, based on a big data model, the inherent correlation between defect identification results and historical maintenance data is mined, and time-series analysis is used to predict defect development trends, triggering early warnings and generating predictive maintenance suggestions. Subsequently, a digital twin model of the road is constructed, mapping the multimodal raw data, defect identification results, and early warning information to the model in real time, achieving synchronization between the physical road and the digital model, and supporting simulation analysis of maintenance effects. Finally, combining defect identification results, predictive maintenance suggestions, and real-time monitoring data from the digital twin model, the importance of the road, the urgency of defects, and the distribution of maintenance resources are comprehensively considered to achieve automated scheduling of maintenance tasks, forming a closed-loop maintenance system.

[0065] This invention comprehensively addresses the problems of traditional road maintenance, such as reliance on manual labor, fragmented processes, low accuracy, and lack of predictability. By deeply integrating multiple models, it breaks down data silos and achieves full automation from data collection to scheduling and execution, significantly improving maintenance efficiency. Simultaneously, by combining predictive maintenance with real-time monitoring, it proactively avoids the risk of road damage escalation, reduces maintenance costs, extends road lifespan, and promotes the transformation of road maintenance from passive repair to proactive prevention, comprehensively enhancing the level of intelligent maintenance.

[0066] In some embodiments, the collection of road condition data and positioning data specifically includes:

[0067] Deploy data acquisition equipment, which includes a high-definition industrial camera, a laser road surface detector, a multi-parameter environmental sensor, and a GNSS dual-mode positioning module;

[0068] The high-definition industrial camera collects road surface appearance data, the laser road surface detector collects road surface structure data, the multi-parameter environmental sensor collects temperature, humidity and rainfall data, and the GNSS dual-mode positioning module collects latitude, longitude and elevation data of each monitoring point on the road surface.

[0069] The collected multi-source data is preprocessed, including denoising using a Gaussian filtering algorithm, aligning the multi-source data based on the positioning data, and normalizing the data to the [0,1] interval to obtain the multimodal raw data.

[0070] It should be noted that the detailed implementation process of multi-source data acquisition, through the deployment of specialized equipment and refined data preprocessing, ensures data quality. First, an acquisition system is deployed, including a high-definition industrial camera, a laser pavement detector, multi-parameter environmental sensors, and a GNSS dual-mode positioning module. The high-definition industrial camera captures pavement appearance details, the laser pavement detector acquires pavement structure parameters, the multi-parameter environmental sensors record environmental factors such as temperature, humidity, and rainfall, and the GNSS dual-mode positioning module determines the spatial location of each monitoring point. Second, various types of data are collected at preset frequencies to ensure data coverage of pavement appearance, structure, environment, and positioning dimensions. Finally, the collected data is preprocessed, using a Gaussian filtering algorithm to remove image noise and sensor errors, achieving spatiotemporal alignment of multi-source data based on positioning data, and using a normalization algorithm to unify the data to the [0,1] interval to eliminate dimensional influences, ultimately forming high-quality multimodal raw data. This invention solves the problems of traditional data acquisition methods, such as single data source, poor data quality, and weak spatiotemporal correlation. By deploying diverse professional equipment, comprehensive coverage of data dimensions is achieved, avoiding the limitations of single data types. The refined preprocessing process effectively improves data accuracy, ensures the consistency and availability of multi-source data, provides reliable data support for subsequent disease identification, early warning and prediction modules, reduces analysis errors caused by data problems, and ensures the operational accuracy of the entire maintenance system.

[0071] In some embodiments, the analysis of the multimodal raw data using a neural network model specifically includes:

[0072] A multimodal fusion neural network model is constructed, which includes a convolutional neural network (CNN) sub-model, a Transformer sub-model, and an attention mechanism fusion sub-model.

[0073] The CNN sub-model extracts features from the road surface appearance data and outputs an appearance feature vector. The Transformer sub-model is used to extract features from the time-series data of the pavement structure (deflection values, crack depth time series), and output a structural feature vector. ;

[0074] The attention mechanism is used to fuse the sub-model for calculation. and The weights α are used to obtain the fused feature vector. , where α is the attention weight, which is optimized by training the model using historical disease data;

[0075] The fused feature vector is input into a fully connected layer and a Softmax classifier, and the output disease type (crack, pothole, rut, settlement) and disease level (Level IV) are obtained, which is the disease identification result, and the disease identification accuracy is ≥95%.

[0076] In practice, a multimodal fusion neural network model is used to improve recognition accuracy. First, a multimodal fusion neural network model is constructed, consisting of a CNN sub-model, a Transformer sub-model, and an attention mechanism fusion sub-model. The CNN sub-model, targeting road surface appearance data, extracts appearance features such as crack textures and pothole contours through multi-layer convolution and pooling operations, outputting an appearance feature vector. The Transformer sub-model, based on time-series data of pavement structures, utilizes a self-attention mechanism to capture the temporal variation patterns of parameters such as deflection and crack depth, and outputs a structural feature vector. Attention mechanism fusion sub-model computation and The attention weight α is determined by the formula. Feature fusion is achieved, making the fused features more focused on information effective for disease identification. Finally, the fused feature vector is input into a fully connected layer and a Softmax classifier to output the disease type (cracks, potholes, ruts, settlement) and disease level (Level IV), thus completing disease identification.

[0077] This invention addresses the problems of traditional disease identification relying on single data sources, being susceptible to interference, and having low accuracy. By integrating multimodal data and a dedicated neural network model, it fully utilizes the spatial features of appearance data and the temporal features of structural data to improve the ability to identify subtle diseases and diseases in complex scenarios, achieving a disease identification accuracy of ≥95% and a false negative rate of ≤5%. Simultaneously, the model possesses strong anti-interference capabilities, reducing the impact of weather, lighting, and other factors on the identification results, providing accurate disease information for subsequent early warning and scheduling.

[0078] In some embodiments, the step of mining the correlation between the disease identification results and historical maintenance data using a big data model specifically includes:

[0079] Construct a disease database to store the disease identification results, multimodal raw data, historical maintenance records (maintenance time, method, cost), and environmental data;

[0080] A disease development prediction model is constructed using a Long Short-Term Memory (LSTM) network. The prediction formula of the model is S(t+Δt) = S(t)·e^(k·Δt + ε), where S(t) is the severity of the disease at time t (based on disease level quantification, level I = 1, level V = 5), S(t+Δt) is the predicted severity at time t+Δt, k is the disease development rate coefficient, Δt is the prediction time interval, and ε is the error correction term with |ε|≤0.05.

[0081] The k is obtained through training on historical data and satisfies... , The basic rate of disease development (such as cracks) =0.02 / day), T is the average daily ambient temperature (°C), R is the average daily rainfall (mm), and a and b are environmental impact coefficients (trained through linear regression, such as a=-0.001, b=0.003).

[0082] Set an early warning threshold S_threshold (e.g., S_threshold=4 for level IV disease). When S (t+Δt)≥S_threshold, an early warning is triggered, and predictive maintenance suggestions (maintenance time, material selection, and work process) are generated.

[0083] In practice, the big data processing early warning system is implemented through database construction and an LSTM prediction model to achieve disease early warning and predictive maintenance. First, a distributed disease database based on the Hadoop+HBase architecture is constructed to store multimodal raw data, disease identification results, historical maintenance records, and environmental data, providing a data foundation for data analysis. Second, an LSTM model is used to construct a disease development prediction model. The model input includes features such as the current disease level, temperature, humidity, rainfall, road age, and traffic volume. The model is trained using the Adam optimizer and mean squared error loss function to determine the disease development rate coefficient k, where k satisfies the formula... ( (Based on the rate, T is the average daily temperature, R is the average daily rainfall, and a and b are environmental impact coefficients); subsequently, using the formula... (S(t) represents the severity of the disease at time t, Δt is the prediction time interval, and ε is the error correction term) Predict the future development of the disease; finally, set an early warning threshold S_threshold. When the predicted severity of the disease is ≥ S_threshold, an early warning is triggered, and predictive maintenance suggestions are generated based on the disease type and development trend. This invention solves the problem of traditional management lacking predictability and passively repairing only after the disease has expanded. Through big data mining and time-series prediction models, the disease development trend can be predicted 7-90 days in advance, triggering early warnings in a timely manner, preventing the disease from evolving from mild to severe, and reducing maintenance costs by 30%-50%. At the same time, the generated predictive maintenance suggestions specify the maintenance time, material selection, and operation process, providing scientific guidance for maintenance work, reducing resource waste caused by blind maintenance, and improving the rationality of maintenance decisions.

[0084] In some embodiments, the digital twin model for constructing the road specifically includes:

[0085] Based on the integration technology of Building Information Modeling (BIM) and Geographic Information System (GIS), an initial digital twin model of the road is constructed. The model includes a road geometry sub-model (length, width, slope), a structural sub-model (base layer thickness, surface layer material parameters), and ancillary facility sub-models (manhole covers, road signs, drainage system).

[0086] A real-time data mapping channel is established, and the multimodal raw data, disease identification results and early warning information are transmitted to the digital twin model through the MQTT protocol to achieve synchronous updates of physical roads and digital models, with a synchronization delay of ≤10s;

[0087] Based on the aforementioned digital twin model, a maintenance simulation analysis was conducted. The simulation formula is σ_after = σ_before·(1+ β·D), where σ_before is the pavement strength before maintenance (MPa), σ_after is the pavement strength after maintenance, β is the repair effect coefficient (β=0.6-0.8 for asphalt repair, β=0.7-0.9 for concrete repair), and D is the maintenance operation depth (cm).

[0088] The digital twin model is displayed through a visual interface, allowing users to view disease details, environmental parameters, and simulation results by clicking with the mouse, enabling real-time monitoring.

[0089] It should be noted that the technical path of digital twin modeling is specifically explained, achieving real-time road monitoring and simulation analysis through the integration of BIM and GIS and real-time data mapping. First, a road BIM model is built based on Autodesk Revit, including the geometric and material parameters of the pavement, base course, and subbase. This model is then imported into ArcGIS to construct a GIS geographic scene, merging to form an initial digital twin model containing geometric sub-models, structural sub-models, and ancillary facility sub-models. Second, a real-time data mapping channel is established via the MQTT protocol to transmit multimodal raw data, defect identification results, and early warning information to the digital twin model, achieving synchronous updates between the physical road and the digital model with a synchronization delay of ≤10 seconds. Subsequently, maintenance simulation analysis is performed based on the model, using strength simulation formulas... (σ_before is the pavement strength before maintenance, β is the repair effectiveness coefficient, and D is the maintenance depth) Evaluate the maintenance effect using a cost simulation formula. (Q represents material usage, P represents material unit price, L represents maintenance length, and M represents labor and equipment cost per unit length) The maintenance cost is calculated. Finally, the model is displayed through a WebGL visualization interface, allowing users to view details of road defects, environmental parameters, and simulation results. This invention solves the problems of traditional management and maintenance, such as the difficulty in real-time monitoring of road conditions and the lack of simulation verification for maintenance plans. The digital twin model enables visualized real-time monitoring of road conditions, reducing the response time to sudden road defects from 4 hours to within 10 minutes, facilitating managers to quickly grasp road conditions. Maintenance simulation analysis can assess the effectiveness and cost of different plans in advance, optimize maintenance strategies, and avoid ineffective maintenance. The visualization interface improves information acquisition efficiency and provides intuitive support for management and maintenance decisions.

[0090] In some embodiments, the automated scheduling of road maintenance specifically includes: acquiring input data: the disease identification results (type, level, location), predictive maintenance suggestions, real-time road status of the digital twin model, and maintenance resource information (location of maintenance teams, equipment type, and material inventory).

[0091] Calculate maintenance priority using the following formula: ,in , , As weight and (Main road) =0.4, =0.4, =0.2; branch =0.2, =0.5, =0.3), L is the road importance level (level 1-5, main road = 5), S is the severity of the damage (level 1-5), and D is the distance between the damage and the maintenance base (normalized to 1-5, the closer = 1).

[0092] Based on priority, assign the nearest maintenance team to high-priority defects and generate scheduling instructions (including defect location, required equipment, material usage, and estimated duration).

[0093] Receive feedback from maintenance teams on their work (completion time, actual materials consumed, maintenance results), and update the disease database and digital twin model.

[0094] In practical implementation, the operational process of decision-making and scheduling needs to be clearly defined, and automated maintenance scheduling should be achieved through priority calculation and resource optimization allocation. First, obtain the disease identification results, predictive maintenance suggestions, real-time data from the digital twin model, and maintenance resource information (team location, equipment type, material inventory); second, use a weighted summation formula... ( , , Maintenance priorities are calculated using weights (L = road importance level, S = severity of road damage, and D = distance between the road damage and maintenance base), with weights dynamically adjusted based on road level. Subsequently, based on priority ranking, a greedy algorithm assigns the nearest maintenance team with the highest equipment matching degree to high-priority road damage, generating scheduling instructions that include road damage location, resource requirements, and time requirements. Finally, feedback from maintenance teams is received, and the road damage database and digital twin model are updated to complete the scheduling loop. This invention solves the problems of low efficiency, unreasonable resource allocation, and delays caused by high-priority road damage in traditional manual scheduling. Automated priority calculation ensures that high-urgency and high-importance road damage is handled first, reducing the maintenance delay rate of high-priority road damage from 30% to below 5%. The greedy algorithm optimizes resource allocation, improving maintenance resource utilization from 60% to 85%, reducing resource idleness and waste. The closed-loop management of scheduling instructions and feedback updates ensures that maintenance work is traceable and optimizable, improving overall scheduling efficiency.

[0095] Reference Figure 2 The second aspect of the present invention provides an intelligent road maintenance system based on multi-model fusion, including a data acquisition module, a disease identification module, a big data processing and early warning module, a digital twin modeling module, and a decision scheduling module;

[0096] The multi-source data acquisition module is used to collect road condition data and positioning data, and output multimodal raw data;

[0097] The disease identification module is used to load a neural network model, analyze the multimodal raw data, and output disease identification results;

[0098] The big data processing and early warning module is used to build a disease database, mine disease patterns through big data models, and output early warning information and predictive maintenance suggestions.

[0099] The digital twin modeling module is used to construct a digital twin model of the road, map real-time data, perform simulation analysis, and output real-time monitoring results.

[0100] The decision-making and scheduling module is used to receive the disease identification results, predictive maintenance suggestions, and real-time monitoring results to realize automated scheduling of road maintenance.

[0101] It should be noted that the overall architecture of the intelligent road maintenance system of this invention achieves its functions through the collaborative operation of five functional modules. The multi-source data acquisition module is responsible for collecting and preprocessing multi-dimensional data, outputting multimodal raw data; the disease identification module loads a multimodal fusion neural network model, analyzes the multimodal raw data, and outputs disease identification results; the big data processing and early warning module constructs a disease database, mines disease patterns through a big data model, and outputs early warning information and predictive maintenance suggestions; the digital twin modeling module constructs a digital twin model, maps real-time data, performs simulation analysis, and outputs real-time monitoring results; the decision-making and scheduling module receives the output data from each module, calculates maintenance priorities and allocates resources, generates scheduling instructions, and simultaneously receives feedback information to update system data. All modules achieve data flow through a data interaction bus, forming a collaborative system architecture.

[0102] This invention addresses the problems of fragmented functions and poor inter-module coordination in traditional road maintenance systems. Through modular design, the responsibilities of each module are clearly defined, and a data exchange bus ensures efficient data flow between modules, achieving system-level integration from data acquisition to scheduling and execution. The collaborative work of each module avoids process interruptions caused by functional fragmentation, ensuring the continuity and automation of the entire maintenance process, improving overall system efficiency, and providing an integrated solution for road maintenance.

[0103] In some embodiments, the multi-source data acquisition module includes an acquisition device submodule and a data preprocessing submodule;

[0104] The data acquisition equipment submodule includes a high-definition industrial camera (resolution ≥ 20 million pixels, frame rate ≥ 25fps), a laser road surface detector (detection accuracy ±0.1mm), a multi-parameter environmental sensor (temperature range -30~70℃, humidity range 0~100% RH), and a GNSS dual-mode positioning module (positioning accuracy ≤ 1m).

[0105] The data preprocessing submodule incorporates Gaussian filtering, data alignment, and normalization algorithms to perform noise reduction, correlation, and normalization on the collected data.

[0106] Specifically, this embodiment refines the internal structure of the multi-source data acquisition module, ensuring data acquisition quality through the cooperation of the equipment sub-module and the preprocessing sub-module. The acquisition equipment sub-module includes a high-definition industrial camera (resolution ≥ 20 million pixels, frame rate ≥ 25fps), a laser road surface detector (detection accuracy ±0.1mm), a multi-parameter environmental sensor (temperature range -30~70℃, humidity range 0~100% RH), and a GNSS dual-mode positioning module (positioning accuracy ≤ 1m). Each device acquires corresponding types of data at a preset frequency. The data preprocessing sub-module incorporates a Gaussian filtering algorithm, a data alignment algorithm, and a standardization algorithm. The Gaussian filtering algorithm removes data noise, the data alignment algorithm achieves spatiotemporal correlation of multi-source data based on positioning data, and the standardization algorithm normalizes the data to the [0,1] interval, ultimately outputting high-quality multimodal raw data. This invention solves the problem of unstable data quality caused by unclear data acquisition equipment parameters and missing preprocessing algorithms. Clearly defined equipment parameters ensure the accuracy and timeliness of collected data, avoiding data distortion caused by insufficient equipment performance; built-in preprocessing algorithms standardize the data processing flow, reduce human error, ensure the consistency and reliability of multimodal raw data, provide high-quality data input for subsequent modules, and further improve the operational accuracy of the entire maintenance system.

[0107] Reference Figure 3 A third aspect of this invention provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor is configured to execute the computer program to implement the intelligent road maintenance method based on multi-model fusion as described above. This invention provides a hardware carrier for implementing the above-described maintenance method. The computer device includes a memory and a processor. The memory uses a suitable storage medium (such as RAM, ROM, hard disk, etc.) to store the computer program and related data (such as multimodal raw data, disease identification results, model parameters, etc.) for implementing the above-described intelligent road maintenance method based on multi-model fusion. The processor uses a general-purpose CPU or a dedicated processor and is configured to read from the memory and execute the computer program. By calling instructions in the program, it sequentially executes steps such as data acquisition, disease identification, big data early warning, digital twin modeling, and decision scheduling to realize the intelligent road maintenance function. This computer device solves the problem of lack of hardware support for the above-described maintenance method. A standardized computer equipment architecture provides a stable operating environment for maintenance methods, ensuring that programs are executable and data is storable. The processor's computing power ensures the efficient operation of multiple models (neural networks, big data models, etc.), avoiding processing delays caused by insufficient hardware performance. The memory's storage capacity ensures the secure storage of large amounts of maintenance data and programs, providing hardware support for the practical application of maintenance methods.

[0108] In some embodiments, the intelligent road maintenance method based on multi-model fusion in the above embodiments can be implemented by a computer device, which includes at least one processor, a communication bus, a memory, and at least one communication interface.

[0109] A processor can be a general-purpose central processing unit (CPU) or an application-specific integrated circuit (ASIC).

[0110] A communication bus can be used to transmit information between the aforementioned components.

[0111] The memory can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, universal optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. The memory can exist independently and be connected to the processor via a communication bus. The memory can also be integrated with the processor.

[0112] The memory stores program code for executing the solution of this application, and its execution is controlled by a processor. The processor executes the program code stored in the memory. The program code may include one or more software modules. The intelligent road maintenance method based on multi-model fusion in the above embodiments can be implemented by a processor and one or more software modules in the program code in the memory.

[0113] A communication interface is a device that uses any transceiver or similar device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0114] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0115] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0116] A fourth aspect of this invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned intelligent road maintenance method based on multi-model fusion. This invention provides a program storage carrier for implementing the aforementioned maintenance method. The computer-readable storage medium is a computer-readable storage medium (such as a CD-ROM, USB flash drive, hard disk, flash memory, etc.) storing the computer program implementing the aforementioned intelligent road maintenance method based on multi-model fusion. When the computer program is read and executed by a processor (such as a computer device or server processor), the processor will complete all steps of multi-source data acquisition, defect identification, big data processing and early warning, digital twin modeling, and decision scheduling according to the program instructions, realizing the various functions of intelligent road maintenance. This computer-readable storage medium solves the problem of the difficulty in storing and porting the program for the aforementioned maintenance method. The computer-readable storage medium facilitates the saving, transmission, and deployment of the program, allowing it to be transferred between different computer devices, improving the portability of the maintenance method; simultaneously, the stability of the storage medium ensures that the program will not be easily lost or damaged, guaranteeing that the maintenance method can be called and executed stably for a long time, providing convenience for the promotion and application of the maintenance method.

[0117] Reference Figure 1 and Figure 2 To provide a clearer understanding of the invention, the following description further illustrates the invention:

[0118] A smart road maintenance method based on multi-model fusion, the method comprising the following steps:

[0119] Multi-source data acquisition steps: Collect road condition data and location data to obtain multimodal raw data.

[0120] Deploy data acquisition equipment: Deploy data acquisition terminals at intervals of 500-1000 meters along the road. Each terminal integrates a high-definition industrial camera (for capturing road surface images / videos), a laser road surface detector (for measuring crack depth, road surface smoothness, and deflection value), a multi-parameter environmental sensor (for collecting temperature, humidity, and rainfall data), and a GNSS dual-mode positioning module (supporting BeiDou and GPS to obtain the latitude, longitude, and elevation of the monitoring point).

[0121] Data acquisition: A high-definition industrial camera captures a road surface image every 30 minutes (resolution 2448×2048), a laser road surface detector measures structural parameters every hour, environmental sensors collect data in real time (sampling frequency 1Hz), and a positioning module synchronously records the location information of each monitoring point.

[0122] Data preprocessing:

[0123] Denoising: Gaussian filtering algorithm is used to denoise the road surface image and structure data. The Gaussian filtering formula is G(x,y) =(1 / (2πσ²))·e^(-(x²+y²) / (2σ²)), where σ is the standard deviation (taken as 0.8-1.2), which removes image noise and sensor error;

[0124] Data alignment: Based on GNSS positioning data, images, structural data, and environmental data of the same monitoring point are correlated to ensure temporal and spatial consistency;

[0125] Standardization: Normalize all data to the interval [0,1], with the formula x_norm = (x - x_min) / (x_max - x_min), where x is the original data, and x_min and x_max are the minimum and maximum values ​​of the data, respectively. This eliminates the influence of dimensions and yields the original multimodal data.

[0126] Disease identification steps: Combine neural network models to analyze multimodal raw data to obtain disease identification results.

[0127] Model Construction: Construct a multimodal fusion neural network model, which includes:

[0128] CNN Sub-model: Employing the ResNet50 architecture, it takes road surface image data as input and extracts appearance features (such as crack texture and pothole contours) through 5 layers of convolution and pooling operations, outputting a 256-dimensional appearance feature vector. ;

[0129] The Transformer sub-model employs an Encoder layer architecture, taking into input time-series data of pavement structure (such as deflection values ​​and crack depths over the past 7 days), capturing temporal correlations through a self-attention mechanism, and outputting a 128-dimensional structural feature vector. ;

[0130] Attention mechanism fusion sub-model: computation and The attention weight α (α∈[0,1]) is optimized by training the model with historical disease data (100,000+ sets of labeled data) to make the fused features more focused on effective information. The fusion formula is as follows: The output is a 384-dimensional fused feature vector F.

[0131] Disease Classification: The fused feature vector F is input into a fully connected layer (containing two hidden layers with 128 and 64 neurons respectively) and a Softmax classifier. The output is the disease type (cracks, potholes, ruts, settlement) and disease level (Level I: minor, Level II: mild, Level III: moderate, Level IV: severe, Level V: extremely severe), i.e., the disease identification result. Testing showed that the model achieved a crack identification accuracy of 97.2% and a pothole identification accuracy of 96.5%, meeting engineering requirements.

[0132] Big data processing and early warning steps: Discover disease patterns through big data models to achieve early warning and predictive maintenance.

[0133] Database Construction: Construct a distributed disease database (using Hadoop+HBase architecture) to store multimodal raw data, disease identification results, historical maintenance records (maintenance time, method, material usage, cost) and environmental data. The data storage period is ≥5 years, and it supports fast query and batch processing.

[0134] Disease Development Prediction: An LSTM model was used to predict disease development. The LSTM model consists of one input layer (10-dimensional features: current disease level, temperature, humidity, rainfall, road age, and traffic volume), two hidden layers (each with 128 neurons), and one output layer (predicting the disease level). The model was trained using the Adam optimizer with mean squared error (MSE) as the loss function, and the training set to test set ratio was 8:2. The prediction formula is as follows: ,in:

[0135] S(t): The severity of the disease at time t (Level I = 1, Level II = 2, Level III = 3, Level IV = 4, Level V = 5);

[0136] Δt: Prediction time interval (unit: days), which can be set to 7 days, 30 days, or 90 days;

[0137] k: Disease development rate coefficient, obtained from historical data training, satisfying... , Based on the rate (crack) =0.02 / day, pit =0.03 / day), T is the average daily temperature (°C), R is the average daily rainfall (mm), a=-0.001 (temperature increases inhibit crack development), b=0.003 (rainfall accelerates disease development);

[0138] ε: Error correction term, optimized through model training, |ε|≤0.05, ensuring prediction error ≤5%.

[0139] Early warning and maintenance recommendations: Set an early warning threshold S_threshold (e.g., S_threshold=4 for Class IV defects on main roads, S_threshold=5 for Class V defects on branch roads). When S (t+Δt)≥S_threshold, the system triggers an audible and visual early warning and generates predictive maintenance recommendations: For example, when the crack S (t+30)=4.2, it is recommended to use asphalt grouting within 15 days, with a grouting depth ≥2cm, and the material should be SBS modified asphalt.

[0140] Digital twin modeling steps: Construct a digital twin model to achieve real-time monitoring and simulation analysis.

[0141] Initial model construction: Based on the integration of BIM and GIS technologies, the road BIM model (including the geometric and material parameters of the road surface, base course, and subbase) was built using Autodesk Revit, and imported into ArcGIS to build a GIS geographic scene (including the terrain along the road and the location of ancillary facilities). The initial digital twin model was obtained by fusion, and the model accuracy reached the centimeter level.

[0142] Real-time data mapping: Multimodal raw data, disease identification results, and early warning information are transmitted to the digital twin model in real time via the MQTT protocol (message transmission delay ≤ 500ms), updating the road surface condition parameters in the model (such as crack location and depth, road surface strength), realizing the synchronization between the physical road and the digital model, with a synchronization delay ≤ 10s.

[0143] Simulation Analysis: Maintenance schemes are simulated based on digital twin models to evaluate the effectiveness and cost of different schemes.

[0144] Strength simulation: The finite element analysis algorithm is used, and the simulation formula is σ_after = σ_before·(1 + β·D), where σ_before is the pavement strength before curing (converted from deflection value, deflection value ≤0.2mm corresponds to σ_before=4MPa), σ_after is the strength after curing, β is the repair effect coefficient (asphalt grouting β=0.6, asphalt overlay β=0.8, concrete repair β=0.9), and D is the curing depth (grouting D=2cm, overlay D=5cm).

[0145] Cost simulation: Based on the simulation results and material unit price, the maintenance cost is calculated using the formula C = Q·P + L·M, where Q is the material usage (m³), P is the material unit price (yuan / m³), L is the maintenance operation length (km), and M is the labor and equipment cost per unit length (yuan / km), providing a cost reference for scheduling.

[0146] Real-time monitoring: The digital twin model is displayed through a WebGL visualization interface, which supports zooming, panning, and rotation operations. Clicking on the diseased area with the mouse can display detailed information (disease level, predicted development trend, environmental parameters), and supports access from multiple terminals (computer, mobile phone, tablet).

[0147] Decision-making and scheduling steps: Based on the output of the aforementioned modules, automated scheduling is achieved.

[0148] Data input: Obtain the results of disease identification (type, level, location), predictive maintenance suggestions based on big data early warning, real-time road status of digital twin model (such as whether there is traffic congestion), and maintenance resource information (location of maintenance team, equipment type (such as crack sealing machine, milling machine), and material inventory (asphalt, sand and gravel)).

[0149] Priority Calculation: Maintenance priority is calculated using a weighted summation formula. ,in:

[0150] , , Weighting coefficient, dynamically adjusted according to road grade, for main roads (such as urban expressways). =0.4、 =0.4、 =0.2, secondary arterial road =0.3、 =0.4、 =0.3, branch =0.2, 2 =0.5、 =0.3;

[0151] L: Road importance level, 1-5 (main road = 5, secondary road = 4, local road = 3, community road = 2, sidewalk = 1);

[0152] S: Severity of disease, 1-5 (I = 1, V = 5);

[0153] D: The distance between the disease and the maintenance base, normalized to 1-5 (distance ≤1km=1, 1-3km=2, ​​3-5km=3, 5-10km=4, >10km=5).

[0154] Resource allocation: Based on priority ranking (P-value from largest to smallest), a greedy algorithm is used to allocate maintenance resources: Maintenance teams with the closest proximity and highest equipment matching degree are prioritized for the defects with the highest P-value (e.g., teams with crack sealing machines are assigned to crack defects). Scheduling instructions are generated, including:

[0155] Basic information: Location of the disease (latitude and longitude), type and severity of the disease;

[0156] Resource requirements: Equipment (1 crack sealing machine, 1 transport vehicle), materials (0.5 tons of SBS modified asphalt);

[0157] Time requirements: estimated departure time, arrival time, and work duration (≤8 hours).

[0158] Feedback Updates: The maintenance team receives instructions via a mobile app, and after completing the work, uploads feedback information (actual work time, material usage, and pavement strength test values ​​after maintenance). The system updates the defect database (marking defects as "maintained") and the digital twin model (updating pavement strength parameters), forming a closed-loop management system.

[0159] (II) A multi-model fusion-based intelligent road maintenance system, comprising a multi-source data acquisition module, a disease identification module, a big data processing and early warning module, a digital twin modeling module, and a decision-making and scheduling module, with the functions of each module as follows:

[0160] Multi-source data acquisition module:

[0161] The data acquisition equipment submodule includes a high-definition industrial camera, a laser road surface detector, a multi-parameter environmental sensor, and a GNSS dual-mode positioning module, used to acquire data on the road surface appearance, structure, environment, and positioning.

[0162] Data preprocessing submodule: Built-in Gaussian filtering, data alignment and standardization algorithms are used to preprocess the collected data and output multimodal raw data to the disease identification module and digital twin modeling module.

[0163] Disease identification module:

[0164] Model loading submodule: Stores and loads multimodal fusion neural network models (CNN + Transformer + attention mechanism);

[0165] Feature extraction and fusion submodule: performs feature extraction and fusion on multimodal raw data, and outputs a fused feature vector;

[0166] Disease classification submodule: Through a fully connected layer and a Softmax classifier, the disease identification results are output to the big data processing and early warning module, the digital twin modeling module, and the decision scheduling module.

[0167] Big Data Processing Early Warning Module:

[0168] Database submodule: Adopts Hadoop+HBase architecture to store multi-source data, disease identification results, and historical maintenance records;

[0169] Model training submodule: Trains and updates the LSTM disease prediction model (retrains once per quarter and optimizes the k value).

[0170] Early warning and suggestion submodule: Triggers early warning based on prediction results, generates predictive maintenance suggestions, and outputs them to the digital twin modeling module and decision scheduling module.

[0171] Digital twin modeling module:

[0172] Initial Modeling Submodule: Constructs an initial digital twin model of the road based on BIM+GIS;

[0173] Data mapping submodule: Receives real-time data via the MQTT protocol and updates the digital twin model;

[0174] Simulation analysis submodule: Implements simulation of maintenance scheme intensity and cost, and outputs simulation results;

[0175] Visualized Monitoring Submodule: Provides a WebGL visualization interface to display real-time monitoring data.

[0176] Decision scheduling module:

[0177] Data integration submodule: Receives disease identification results, early warning suggestions, real-time monitoring data, and resource information;

[0178] Priority calculation submodule: Calculates maintenance priority based on a weighted formula;

[0179] Resource allocation submodule: Uses a greedy algorithm to allocate maintenance resources and generate scheduling instructions;

[0180] Feedback processing submodule: Receives maintenance feedback and updates system data.

[0181] Implementation Case: The following is a detailed description of the implementation process, using a real-world application scenario of a main urban road (10km long, 6 lanes in both directions, with a service life of 5 years):

[0182] I. System Deployment

[0183] Multi-source data acquisition equipment deployment: One acquisition terminal is deployed every 800 meters along the main road, for a total of 13 terminals; each terminal integrates:

[0184] High-definition industrial camera: Model Hikvision DS-2CD5A26EFWD-IZ, resolution 2560×1920, frame rate 30fps, installation height 3m, angle of depression 30°;

[0185] Laser road surface detector: Model Beijing Aerospace Keyu HY-3000, detection range 0-50mm, accuracy ±0.1mm;

[0186] Multi-parameter sensor: Model Sensirion SHT31, temperature range -40~125℃, humidity range 0~100%RH;

[0187] GNSS positioning module: Model: Beidou Navigation UM220-IV, positioning accuracy ≤1m.

[0188] Software system deployment:

[0189] The big data processing and early warning module is deployed on an Alibaba Cloud server (4 cores, 8GB memory, 1TB hard drive) and adopts a Hadoop 3.0 distributed architecture.

[0190] The digital twin modeling module uses Autodesk Revit 2024 to build the BIM model, ArcGIS Pro 2023 to build the GIS scene, and the WebGL visualization interface is deployed on an nginx server.

[0191] The decision scheduling module is deployed on a local server (8 cores and 16GB of memory) and supports access from mobile apps (Android / iOS).

[0192] II. Implementation Steps

[0193] Multi-source data acquisition:

[0194] Data acquisition frequency: High-definition camera takes one picture every 30 minutes, laser detector takes one measurement every hour, and sensor acquires data in real time;

[0195] Preprocessing: Gaussian filtering with σ=1.0 was used for noise reduction. Based on GNSS data, multi-source data were aligned and the deflection value (0-1mm), temperature (-10~40℃) and other data were normalized to the [0,1] interval to obtain multimodal raw data.

[0196] Disease identification:

[0197] Model loading: Load the pre-trained multimodal fusion neural network model (training set contains 50,000 labeled data sets with an accuracy of 96.8%).

[0198] Identification results: A Class III transverse crack with a length of 2.5m and a depth of 1.8mm was identified at K3+200 (3.2km from the starting point); a Class II pit with an area of ​​0.8m² and a depth of 2.2mm was identified at K7+500.

[0199] Big Data Processing Early Warning:

[0200] Predictive calculation: Crack S(t) at K3+200 = 3, current daily average temperature T = 25℃, rainfall R = 5mm, calculate k = 0.02 + (-0.001)×25 + 0.003×5 = 0.02 - 0.025 + 0.015 = 0.01; When the predicted Δt = 30 days, S(30) = 3×e^(0.01×30 + 0.02) = 3×e^0.32 ≈ 3×1.377 = 4.13 ≥ 4 (warning threshold), triggering a warning;

[0201] Maintenance Recommendation: It is recommended to use SBS modified asphalt to fill the cracks within 10 days, with a filling depth of ≥2cm, a material consumption of approximately 0.3 tons, and an estimated cost of 2000 yuan.

[0202] Digital twin modeling:

[0203] Model update: Crack information and early warning information at K3+200 are transmitted to the digital twin model via the MQTT protocol to update the crack parameters at that location in the model;

[0204] Simulation analysis: Simulating the strength of asphalt after crack sealing, σ_before=3.5MPa, β=0.6, D=2cm, the calculated σ_after=3.5×(1+0.6×0.02)=3.5×1.012=3.542MPa, which meets the road strength requirement (≥3MPa); the simulation cost C=0.3×4000 + 0.0025×5000=1200+12.5=1212.5 yuan (consistent with the suggested cost).

[0205] Decision-making and scheduling:

[0206] Priority calculation: The main road has L=5, crack S=3, and distance to the maintenance base D=2km (normalized D=2). =0.4, =0.4, =0.2, calculate P=0.4×5 + 0.4×3 + 0.2×2=2+1.2+0.4=3.6;

[0207] Resource allocation: The maintenance base has two teams, Team A (1.8km from K3+200, equipped with a crack sealing machine) and Team B (5km from K3, equipped with a milling machine). Team A is assigned, and a dispatch instruction is generated: departure time 9:00, arrival time 9:20, work period 4 hours, material 0.3 tons of SBS modified asphalt;

[0208] Feedback Update: Team A completed the work at 13:00, reporting an actual material usage of 0.28 tons and a post-curing strength of 3.55 MPa. The system updated the database and digital twin model, marking the crack as "cured".

[0209] In this implementation case, the system achieves:

[0210] The crack identification accuracy reaches 97%, and the time from data acquisition to identification completion is ≤5 minutes;

[0211] Predicting cracks to reach Class IV status 30 days in advance can prevent them from expanding into potholes (saving 8,000 yuan in subsequent pothole repair costs).

[0212] Maintenance dispatch response time is 20 minutes, and resource utilization rate is increased to 90%.

[0213] The digital twin model is updated in real time, and managers can view the maintenance progress and results in real time through a mobile app.

[0214] The technical solutions provided by the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the embodiments of the present invention. The descriptions of the embodiments above are only for helping to understand the principles of the embodiments of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A smart road maintenance method based on multi-model fusion, characterized in that, Includes the following steps: Collect road surface condition data and location data to obtain multimodal raw data; By combining a neural network model to analyze the multimodal raw data, the detection and identification of road defects can be achieved, and the defect identification results can be obtained. By using big data models to mine the correlation patterns between the disease identification results and historical maintenance data, road disease early warning and predictive maintenance suggestions can be generated. A digital twin model of the road is constructed, and the multimodal raw data, disease identification results and early warning information are mapped to the digital twin model to realize real-time monitoring and simulation analysis of road maintenance; Based on the disease identification results, predictive maintenance suggestions, and real-time monitoring data from the digital twin model, automated scheduling of road maintenance is achieved.

2. The intelligent road maintenance method based on multi-model fusion according to claim 1, characterized in that: The collected road condition data and location data specifically include: Deploy data acquisition equipment, which includes a high-definition industrial camera, a laser road surface detector, a multi-parameter environmental sensor, and a GNSS dual-mode positioning module; The high-definition industrial camera collects road surface appearance data, the laser road surface detector collects road surface structure data, the multi-parameter environmental sensor collects temperature, humidity and rainfall data, and the GNSS dual-mode positioning module collects latitude, longitude and elevation data of each monitoring point on the road surface. The collected multi-source data is preprocessed, including denoising using a Gaussian filtering algorithm, aligning the multi-source data based on the positioning data, and normalizing the data to the [0,1] interval to obtain the multimodal raw data.

3. The intelligent road maintenance method based on multi-model fusion according to claim 1, characterized in that: The analysis of the multimodal raw data using a neural network model specifically includes: Construct a multimodal fusion neural network model, which includes a convolutional neural network (CNN) sub-model, a Transformer sub-model, and an attention mechanism fusion sub-model; The CNN sub-model extracts features from the road surface appearance data and outputs an appearance feature vector. The Transformer sub-model is used to extract features from the time-series data of the road surface structure, outputting a structural feature vector. ; The attention mechanism is used to fuse the sub-model for calculation. and The weights α are used to obtain the fused feature vector. , where α is the attention weight, which is optimized by training the model using historical disease data; The fused feature vector is input into a fully connected layer and a Softmax classifier, and the disease type and disease level are output, which is the disease identification result, and the disease identification accuracy is ≥95%.

4. The intelligent road maintenance method based on multi-model fusion according to claim 1, characterized in that: The specific steps of mining the correlation between the disease identification results and historical maintenance data using big data models include: Construct a disease database to store the disease identification results, multimodal raw data, historical maintenance records, and environmental data; A disease development prediction model is constructed using a long short-term memory network. The prediction formula of the model is as follows: , where S(t) is the severity of the disease at time t, S(t+Δt) is the predicted severity at time t+Δt, k is the disease development rate coefficient, Δt is the prediction time interval, and ε is the error correction term and |ε|≤0.05; The k is obtained through training on historical data and satisfies... , The basic development rate of the disease is represented by T, the average daily ambient temperature is represented by R, the average daily rainfall is represented by a and b, and the environmental impact coefficients are represented by a and b. Set an early warning threshold S_threshold. When S(t+Δt)≥S_threshold, an early warning is triggered and predictive maintenance suggestions are generated.

5. The intelligent road maintenance method based on multi-model fusion according to claim 1, characterized in that: The digital twin model for constructing the road specifically includes: Based on the fusion technology of Building Information Modeling and Geographic Information System, an initial digital twin model of the road is constructed. The model includes a road geometric sub-model, a structural sub-model, and an ancillary facility sub-model. A real-time data mapping channel is established, and the multimodal raw data, disease identification results and early warning information are transmitted to the digital twin model through the MQTT protocol to achieve synchronous updates of physical roads and digital models, with a synchronization delay of ≤10s; Based on the digital twin model, a maintenance simulation analysis is performed. The simulation formula is σ_after = σ_before·(1 +β·D), where σ_before is the pavement strength before maintenance, σ_after is the pavement strength after maintenance, β is the repair effect coefficient, and D is the maintenance operation depth. The digital twin model is displayed through a visual interface, allowing users to view disease details, environmental parameters, and simulation results by clicking with the mouse, enabling real-time monitoring.

6. The intelligent road maintenance method based on multi-model fusion according to claim 1, characterized in that: The automated scheduling of road maintenance specifically includes: acquiring input data: the results of the road damage identification, predictive maintenance suggestions, real-time road status from the digital twin model, and maintenance resource information; Calculate maintenance priority using the following formula: ,in , , As weight and L represents the importance level of the road, S represents the severity of the damage, and D represents the distance between the damage site and the maintenance base. Based on priority, assign the nearest maintenance team to high-priority diseases and generate scheduling instructions; Receive feedback from the maintenance team and update the disease database and digital twin model.

7. An intelligent road maintenance system based on multi-model fusion, characterized in that: It includes a multi-source data acquisition module, a disease identification module, a big data processing and early warning module, a digital twin modeling module, and a decision scheduling module; The multi-source data acquisition module is used to collect road condition data and positioning data, and output multimodal raw data; The disease identification module is used to load a neural network model, analyze the multimodal raw data, and output disease identification results; The big data processing and early warning module is used to build a disease database, mine disease patterns through big data models, and output early warning information and predictive maintenance suggestions. The digital twin modeling module is used to construct a digital twin model of the road, map real-time data, perform simulation analysis, and output real-time monitoring results. The decision-making and scheduling module is used to receive the disease identification results, predictive maintenance suggestions, and real-time monitoring results to realize automated scheduling of road maintenance.

8. The intelligent road maintenance system based on multi-model fusion according to claim 7, characterized in that: The multi-source data acquisition module includes an acquisition device submodule and a data preprocessing submodule; The data acquisition equipment submodule includes a high-definition industrial camera, a laser road surface detector, a multi-parameter environmental sensor, and a GNSS dual-mode positioning module. The data preprocessing submodule incorporates Gaussian filtering, data alignment, and normalization algorithms to perform noise reduction, correlation, and normalization on the collected data.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program and the processor being configured to execute the computer program to implement the intelligent road maintenance method based on multi-model fusion as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the intelligent road maintenance method based on multi-model fusion as described in any one of claims 1-6.

Images