Road crack development prediction method and system based on time sequence images and environmental factors

Abstract

The application discloses a road crack development prediction method and system based on time sequence images and environmental factors, and belongs to the technical field of road maintenance monitoring and intelligent transportation. The method comprises the following steps: acquiring road crack multi-time sequence images of a prediction section and multi-source potential field environmental quantities such as traffic load, environmental climate and pavement material in a corresponding time period; calculating a comprehensive feature vector based on the images and environmental quantities; constructing a crack evolution potential field according to the comprehensive feature vector, and mapping the potential field features into a dynamic heterogeneous graph structure; and performing self-adaptive learning and time sequence modeling on the structure through a potential energy driven dynamic heterogeneous graph architecture search model (PE-DHGAS), and outputting road crack development prediction information and images. The application can not only improve the accuracy and stability of road crack evolution trend prediction, but also use the prediction results for risk assessment and maintenance decision-making, and provide scientific and forward-looking support for road maintenance departments.

Description

Technical Field

[0001] This invention belongs to the field of road maintenance monitoring and intelligent transportation technology, specifically relating to a method and system for predicting the development of road cracks based on time-series images and environmental factors. Background Technology

[0002] With the continuous development of transportation infrastructure, especially the widespread construction of important infrastructure such as roads, bridges, and airports, the detection and repair of road cracks has become an important task in road maintenance and management. Cracks are not only a manifestation of early road surface damage, but also a precursor to more serious defects such as potholes and slab breakage. If the development of cracks is not monitored in time and corresponding measures are not taken, further expansion of cracks may lead to a decline in road performance, increased maintenance costs, and compromised driving safety.

[0003] Currently, crack detection technology mainly relies on high-resolution road imagery, analyzing the images manually or through automated systems to identify crack areas. However, most existing technologies focus on static crack detection, meaning they detect and assess cracks based solely on single-period image data. These technologies cannot effectively predict future crack trends, leading to maintenance management decisions relying on human experience, resulting in problems such as delayed response and inefficient resource allocation.

[0004] While attempting to compare cracks using multiple image periods has alleviated this problem to some extent, common methods are mostly based on simple image differencing or local feature matching. These methods are not adaptable enough to changes in viewing angle, positioning deviations, and the topological evolution of the cracks themselves during the acquisition process, making it difficult to guarantee detection accuracy. Furthermore, existing methods are often limited to comparing only two periods of data, failing to utilize longer-term information to reveal the speed, direction, and future trends of crack propagation. Summary of the Invention

[0005] This application provides a method and system for predicting the development of road cracks based on time-series images and environmental factors. It can predict the future development trend of road cracks through time-series modeling, and by combining environmental factors such as pavement material and traffic load, it can more comprehensively, scientifically and accurately predict the evolution trend of road cracks, thereby helping road maintenance management departments to make intelligent and precise maintenance decisions.

[0006] Firstly, this application provides a method for predicting the development of road cracks based on temporal imagery and environmental factors, the method comprising:

[0007] The process involves acquiring time-series images of road cracks in the predicted road segment; acquiring multi-source potential energy field environmental quantities within the corresponding time period of the road crack time-series images; determining a comprehensive feature vector based on the road crack time-series images and environmental quantities; wherein the comprehensive feature vector is a comprehensive feature vector of at least one target feature under the environmental quantities; the target feature is a feature used to characterize road damage, including crack edge features and branch features; constructing a crack evolution potential energy field characterizing the energy distribution and potential expansion direction of the crack area based on the comprehensive feature vector, and mapping the potential energy field into a dynamic heterogeneous graph structure; adaptively learning and temporally modeling this structure using the potential energy-driven dynamic heterogeneous graph architecture search model (PE-DHGAS), and outputting a road crack development prediction image.

[0008] Therefore, road crack prediction can be achieved based on various data (including time-series images of road cracks and environmental parameters) and through a potential energy-driven dynamic heterogeneous graph architecture search model. This allows for more accurate prediction of road crack evolution trends, revealing the rate, direction, and bifurcation trends of road crack propagation. Based on the predicted road crack evolution trends, more comprehensive and precise maintenance and repair strategies can be provided, effectively improving road safety. Furthermore, by inputting different environmental parameters, the road crack prediction results can reflect the combined influence of multiple environmental and structural factors, improving the scientific rigor, universality, and reliability of the prediction.

[0009] In some possible implementations, obtaining time-series images of road cracks in the predicted road segment includes:

[0010] Acquire multiple road crack images of the predicted road segment, including at least two road crack images, with each road crack image acquired at a different time; based on the multiple road crack images, acquire a time-series image of the road cracks.

[0011] In some possible implementations, time-series images of road cracks are obtained based on multiple road crack images, including:

[0012] Multiple road crack images are matched using common feature points to map them to the same coordinate system, generating a time-series image of the road cracks. The time-series image is then matched using a geometric deformation field to generate an initial time-series image of the road cracks. The initial time-series image of the road cracks is then input into a deep learning network for detail enhancement. When the discriminability of crack edge features and branch features both reach the target threshold, the time-series image of the road cracks is output.

[0013] In the above implementation method, by acquiring time-series images of road cracks, preparation can be made for subsequent road crack prediction and processing.

[0014] In some possible implementations, environmental quantities include one or more of the following: traffic load information, environmental-climate factors, and pavement material factors; wherein, traffic load information includes traffic volume, vehicle speed, vehicle type, and vehicle weight; environmental-climate factors include temperature and humidity factors; and pavement material factors include pavement material type.

[0015] The above implementation method, by combining multiple types of environmental quantities, can provide more comprehensive and diverse data for subsequent road prediction processing, thereby improving the accuracy of road crack prediction results.

[0016] In some possible implementations, a comprehensive feature vector is determined based on time-series images of road cracks and environmental quantities, including:

[0017] The temporal images of road cracks are binarized. During binarization, it is determined whether target features exist in the temporal images of road cracks. If target features exist, a first binary image of the road crack is generated. Based on the first binary image of the road crack, crack skeleton features are extracted, and isolated short branches of the crack are removed to obtain a second binary image of the crack. Based on the second binary image of the crack, crack geometric features, crack topological features, and image temporal parameters are extracted. Among them, crack geometric features include crack length, average crack width, maximum crack width, crack area, and crack direction; crack topological features include a set of crack feature points, a set of crack segments, and a first connectivity relation; the set of crack feature points and the set of crack segments are obtained by connecting feature points based on the crack skeleton network, and the first connectivity relation is used to represent the connectivity between each crack feature point, and the first connectivity relation is recorded using a connected adjacency matrix; based on the crack geometric features and crack topological features, the crack is represented in a graph structure form to obtain a road crack graph structure vector; based on the image temporal parameters, the road crack graph structure vector at each time step is concatenated with the corresponding environmental quantities, and the concatenation result is normalized to obtain a comprehensive feature vector.

[0018] In some possible implementations, the road crack map structure vector at each time step is concatenated with the corresponding environmental variables. The concatenation process is the same at each time step; for example, the concatenation process at the first time step includes:

[0019] The process involves: acquiring the road surface material type from environmental parameters, encoding it into a categorical vector, and embedding it into the road crack map structure vector at the first moment; acquiring traffic load information from environmental parameters, performing statistical feature extraction and normalization on the traffic load information, and then concatenating the features of the traffic load information into the road crack map structure vector at the first moment; acquiring environmental-climate factors from environmental parameters, performing statistical feature extraction and normalization on the environmental-climate factors, and then concatenating the features of the environmental-climate factors into the road crack map structure vector at the first moment; and finally, obtaining the comprehensive feature vector at the first moment after completing the above concatenation process.

[0020] In the above implementation method, by determining the comprehensive feature vector, a data foundation can be provided for subsequent road crack prediction processing.

[0021] Among some possible implementations, the crack evolution potential field is constructed based on the comprehensive eigenvectors, including:

[0022] Based on the characteristics of crack morphology change rate, gray-level gradient change, traffic load intensity and environmental climate fluctuations in the comprehensive feature vector, the energy gradient distribution of crack pixels in space is calculated; based on the energy gradient distribution, the potential energy function of the crack region is established to characterize the degree of energy accumulation and expansion driving force of the crack region.

[0023] The purpose of constructing a crack evolution potential energy field is to characterize the energy distribution and potential propagation direction of the crack region from the physical meaning of energy change. The crack evolution potential energy field establishes a mapping relationship between energy gradient and crack growth trend by jointly modeling the crack morphology change rate, gray-scale gradient and environmental quantity change rate, thereby spatially depicting the stress concentration area and propagation driving force of the crack, and reflecting the dynamic characteristics of crack energy evolution in time.

[0024] In the above implementation method, by constructing the potential energy field, the complex crack evolution process can be transformed into a quantifiable energy distribution change problem, thereby enabling explicit characterization of crack growth direction, rate and potential merging region.

[0025] Furthermore, in the step of mapping the potential energy field to a dynamic heterogeneous graph structure after constructing the crack evolution potential energy field, the mapping process is used to transform continuous potential energy distribution data into a graph representation that can be modeled by a graph neural network, so as to simultaneously characterize the spatial correlation of the crack region, the direction of energy propagation, and the interaction between multiple environmental factors.

[0026] In this context, the pixel regions or grid cells in the crack evolution potential energy field can be regarded as nodes of the graph, and the potential energy gradient direction between nodes can be regarded as directed edges to represent the energy transfer path from high potential regions to low potential regions. At the same time, environmental information such as traffic load, temperature, humidity, and road surface material is added as heterogeneous features of nodes or edges, thereby forming a dynamic heterogeneous graph structure containing multiple types of nodes and multiple relational edges.

[0027] This mapping method can preserve the characteristics of crack energy distribution and stress concentration in the spatial dimension and reflect the dynamic evolution of potential energy with environmental changes in the temporal dimension, providing a unified data structure input for the subsequent potential energy driven dynamic heterogeneous graph architecture search model (PE-DHGAS).

[0028] Furthermore, after obtaining the dynamic heterogeneous graph structure, the present invention performs adaptive learning and temporal modeling on the structure through the potential energy driven dynamic heterogeneous graph architecture search model (PE-DHGAS) to output a road crack development prediction image.

[0029] Among them, the PE-DHGAS model is based on an automated neural network structure search mechanism, which can dynamically optimize the spatial and temporal modeling structure of the graph neural network under the physical constraints of the potential energy field, and realize the joint learning of the relationship between crack energy evolution and heterogeneous factor interaction.

[0030] The model takes the potential energy time series graph as input and models the spatial association, energy propagation path and environmental factor coupling effect between different types of nodes through an automatically constructed graph attention network. In the time dimension, it uses temporal convolution or cyclic structure to capture the dynamic change law of potential energy distribution, thereby learning the whole process of crack energy accumulation, release and expansion.

[0031] During the search process, the PE-DHGAS model simultaneously defines the localization space of the network topology and the parameterization space of the attention parameters. It uses a differentiable multi-stage search algorithm to automatically determine the optimal spatiotemporal fusion structure, achieving adaptive optimization of the model without manual intervention.

[0032] Through the self-learning and structural evolution mechanism of this model, this invention can characterize the driving force of crack propagation in the physical sense of energy change, and achieve high-precision prediction of crack propagation direction, growth rate and potential merging region, significantly improving the stability, generalization and physical interpretability of the prediction results.

[0033] In some possible implementations, after outputting the predicted image of road crack development, the method further includes:

[0034] Based on road crack development prediction images, the current crack status and predicted crack deterioration trend are determined; based on the current crack status, predicted crack deterioration trend and traffic load information, a risk index is calculated; based on the risk index, cracks are classified into high-risk, medium-risk or low-risk categories according to risk level; based on the risk level, treatment recommendations are generated; treatment recommendations include short-term repair, medium-term maintenance or long-term reinforcement.

[0035] In the above implementation method, by using the road crack development prediction image to further generate treatment suggestions, staff can prioritize and maintain high-risk road cracks more promptly based on the treatment suggestions, thereby improving road safety.

[0036] Secondly, this application provides a road crack development prediction system based on temporal imagery and environmental factors, including:

[0037] The multi-source data acquisition module is configured to acquire the time-series image of road cracks in the predicted road segment and the multi-source potential energy field environment quantity within the time period corresponding to the time-series image of road cracks.

[0038] The feature fusion processing module is configured to determine a comprehensive feature vector based on the road crack time-series image and the environmental quantity. The comprehensive feature vector is a comprehensive feature vector of at least one target feature under the environmental quantity. The target feature includes crack edge features and branch features characterizing road damage.

[0039] The crack evolution potential energy field construction module is configured to construct a crack evolution potential energy field that characterizes the energy distribution and potential propagation direction of the crack region based on the comprehensive feature vector, and map the potential energy field into a dynamic heterogeneous graph structure to form a potential energy time series graph containing multiple types of nodes and multiple relational edges.

[0040] The prediction module receives the output of the potential field-driven dynamic heterogeneous graph architecture search model and generates a road crack development prediction image based on the model's automatic structure search and time-series modeling results. The prediction image characterizes the crack's propagation direction, growth rate, and potential merging areas, and can further provide auxiliary decision-making support for road maintenance and risk assessment.

[0041] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0042] The beneficial effects of this invention are as follows: it can reveal the speed, direction and future trend of crack propagation using long-term time-series information, and improve the accuracy and comprehensiveness of predicting the evolution trend of road cracks by combining multi-source potential energy field environmental quantities. The prediction results can be used for risk assessment and maintenance decision-making, providing scientific and forward-looking support for road maintenance departments. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating a method for predicting road crack development based on temporal images and environmental factors, provided as an embodiment of the present invention.

[0045] Figure 2 A flowchart of a method for constructing time-series images of road cracks provided in an embodiment of the present invention.

[0046] Figure 3 This is a flowchart of a comprehensive feature vector determination method provided in an embodiment of the present invention.

[0047] Figure 4 A flowchart of a method for constructing a crack evolution potential energy field is provided in an embodiment of the present invention.

[0048] Figure 5 This is a schematic diagram of a road crack development prediction process based on the PE-DHGAS model, provided as an embodiment of the present invention.

[0049] Figure 6 This is a structural block diagram of a road crack development prediction system based on time-series images and environmental factors, provided in an embodiment of this application. Detailed Implementation

[0050] The method for predicting road crack development based on time-series imagery and environmental factors according to the present invention will be further described below with reference to the accompanying drawings and specific embodiments. This embodiment is a preferred implementation, but should not be construed as limiting the scope of protection of the present invention. Those skilled in the art should understand that adjustments to the order of steps, parameter selection, and module division without departing from the essence of the invention are all within the scope of protection of the present invention.

[0051] The road crack development prediction method based on temporal imagery and environmental factors provided in this application can be applied to road protection scenarios. In some embodiments, this method can be applied to various scenarios requiring road protection, such as highways, urban roads, and airport runways. This method helps road management departments proactively prevent and control road crack defects, optimize maintenance resource allocation, reduce maintenance costs, and improve road safety and service levels.

[0052] The road crack development prediction method based on time-series imagery and environmental factors provided in this application can be applied to electronic devices. The electronic device can be any device with display hardware and corresponding software support. For example, the electronic device can be a mobile phone, smart screen, tablet computer, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc. This application does not impose any limitations on the specific type of electronic device.

[0053] In another embodiment, the road crack development prediction method based on time-series images and environmental factors provided in this application can also be applied to a server, and this application does not limit this.

[0054] Given that existing technologies cannot effectively predict the future trends of road cracks, road crack maintenance and management rely heavily on manual experience, resulting in delays and hindering timely repairs. This application, compared to existing technologies, introduces multi-source potential energy field environmental quantities and constructs a crack evolution potential energy field, achieving a comprehensive characterization of the crack's stress state, environmental response, and developmental driving forces. Furthermore, this application maps the potential energy field to a dynamic heterogeneous graph structure and utilizes a potential energy-driven dynamic heterogeneous graph architecture search model for adaptive learning and temporal modeling to predict the future development trend of road cracks. Even further, by combining environmental quantities such as pavement material and traffic load, this application can more comprehensively, scientifically, and accurately predict the evolution trend of road cracks, thereby helping road maintenance management departments achieve intelligent and precise maintenance decisions, effectively improving road safety and operational efficiency.

[0055] Please see Figure 1 , Figure 1 A flowchart of a road crack development prediction method based on time-series images and environmental factors, provided in an embodiment of this application, is shown.

[0056] like Figure 1 As shown, the road crack development prediction method based on temporal images and environmental factors in this application embodiment includes the following steps:

[0057] S101, Obtain the time-series image of road cracks in the predicted road segment.

[0058] Optionally, in some embodiments, the predicted road segment is the target road segment for which road crack prediction needs to be performed. The predicted road segment includes at least one road segment. For example, the predicted road segment includes one road segment in a city; or several road segments in a city; or multiple road segments in multiple cities; or a connecting road segment between two adjacent cities.

[0059] It should be noted that the above explanations of the predicted road segments are for illustrative purposes only, and the embodiments of this application do not limit them.

[0060] Optionally, in some embodiments, road crack time-series images are used to represent images that are processed from multiple road crack images and have a temporal sequence.

[0061] Electronic devices can automatically acquire temporal images of road cracks in predicted road segments using acquisition devices (e.g., image sensors) or high-resolution imaging devices (e.g., industrial-grade digital cameras equipped with wide-angle, low-distortion lenses). This application embodiment, by acquiring temporal images of road cracks in predicted road segments, can provide a data foundation for subsequent road crack prediction.

[0062] It should be noted that the above-described method of obtaining time-series images of road cracks in predicted road sections is only an example. Electronic devices can also obtain time-series images of road cracks in predicted road sections in various ways, and this application embodiment does not limit this method.

[0063] Based on the above possible implementation methods, electronic devices can acquire time-series images of road cracks in the predicted road segment, thereby preparing for subsequent road crack prediction.

[0064] S102, acquire the multi-source potential energy field environment quantity within the time period corresponding to the road crack time series image.

[0065] Optionally, in some embodiments, environmental quantities include one or more of the following: traffic load information, environmental and climatic factors, and pavement material factors.

[0066] Optionally, in some embodiments, environmental quantities can be divided into static global inputs and temporal global inputs according to their properties. Static global inputs are used to describe the basic properties of the road itself, while temporal global inputs are used to characterize the external influences on the road during the prediction time period.

[0067] Optionally, in some embodiments, traffic load information is a time-series global input. Traffic load information includes traffic flow, vehicle speed, vehicle type, and vehicle weight.

[0068] Optionally, in some embodiments, environmental and climatic factors are included as time-series global inputs. Environmental and climatic factors include temperature and humidity. Temperature includes high temperatures, low temperatures, and temperature difference cycles. Humidity includes infiltration, freeze-thaw cycles, and drainage. Specifically, high temperature is used to characterize the number of days within an interval where the highest temperature exceeds a set threshold. Low temperature is used to characterize the number of days within an interval where the lowest temperature is below a set threshold. Temperature difference cycles are used to characterize the number of days where the daily temperature difference is greater than a set threshold. Humidity is used to characterize the average relative humidity within the interval.

[0069] It should be noted that when the temperature rises from below zero to above zero within the range, accompanied by the presence of moisture on the ground surface, it is counted as one freeze-thaw cycle.

[0070] Optionally, in some embodiments, the pavement material factor is a static global input. The pavement material factor includes the pavement material type. The pavement material type includes at least asphalt concrete, cement concrete, and composite pavement.

[0071] Electronic devices can acquire environmental data within the time frame corresponding to the road crack time-series images using external environmental monitoring equipment (such as image sensors, temperature sensors, and humidity sensors). This data acquisition provides a foundation for subsequent road prediction.

[0072] For example, electronic devices can analyze traffic flow, vehicle speed, vehicle type, and vehicle weight within a given time period using images acquired by image sensors. Electronic devices can also acquire temperature using temperature sensors and humidity information using humidity data.

[0073] It should be noted that the above-mentioned environmental data acquisition is only for illustrative purposes (for example, the type of road surface material can be obtained through manual observation or construction records in a municipal database), and this application embodiment does not limit this.

[0074] It should also be noted that the acquisition of other environmental quantities in this application can refer to relevant technologies and be obtained by those skilled in the art according to actual needs. This application does not limit this.

[0075] It should be understood that there is no specific time or order between S101 and S102; the order depends on the actual application scenario. For example, S101 can be executed before S102; or S101 can be executed before S102; or S101 and S102 can be executed simultaneously.

[0076] Based on the above possible implementation methods, electronic devices can acquire environmental quantities within the time period corresponding to the road crack time series image, thereby preparing for subsequent road crack prediction.

[0077] S103, based on the time-series images of road cracks and environmental quantities, a comprehensive feature vector is determined.

[0078] Optionally, in some embodiments, the composite feature vector is a composite feature vector of at least one target feature under environmental parameters. The composite feature vector may include one or more.

[0079] Optionally, in some embodiments, the composite feature vector is obtained by concatenating the normalized road crack map structure vector with external environmental features in chronological order. The composite feature vector maintains temporal alignment; that is, the road crack features at each moment and the environmental features within the corresponding time period together constitute a composite feature vector. .

[0080] Optionally, in some embodiments, the target features are features used to characterize road damage. Target features include crack edge features and branching features.

[0081] After obtaining time-series images and environmental parameters of road cracks, electronic equipment can determine a comprehensive feature vector based on these parameters. This provides a data foundation for subsequent road crack prediction.

[0082] S104, constructs the crack evolution potential field based on the comprehensive feature vector, and maps the potential field into a dynamic heterogeneous graph structure.

[0083] In this embodiment, the comprehensive feature vector is obtained by fusing the temporal image features of road cracks with the environmental quantity features of the corresponding time period.

[0084] Electronic devices can calculate the spatial energy gradient distribution of a crack region based on parameters such as crack morphology change rate, gray-scale gradient change, traffic load intensity, and environmental climate fluctuations in a comprehensive feature vector. Then, based on the spatial distribution of the energy gradient, a crack evolution potential energy field is constructed to characterize the energy accumulation state and potential expansion direction of the road crack region.

[0085] In practical implementation, electronic devices can transform crack time-series images into energy distribution maps by establishing a mapping relationship between energy gradient and crack geometric changes; where high potential energy regions correspond to concentrated driving forces for crack propagation, and low potential energy regions correspond to structurally stable regions.

[0086] Optionally, in some embodiments, the potential energy value of the crack evolution potential energy field It can be determined by the following formula:

[0087]

[0088] in, This represents the grayscale change rate of the crack time-series image. This represents the dynamic response characteristics of environmental quantities. and These are adjustable weighting coefficients used to balance the effects of image variation features and environmental driving forces.

[0089] The above calculations yield a three-dimensional potential energy field that varies with time, which can be used to reflect the accumulation and release of energy in the crack.

[0090] After obtaining the potential energy field of the crack evolution, the electronic device further maps the potential energy field into a dynamic heterogeneous graph structure.

[0091] During the mapping process, the electronic device treats the pixel blocks or grid cells in the crack region as nodes of the graph and the potential energy gradient direction between nodes as directed edges to describe the energy transfer relationship from high potential region to low potential region. At the same time, environmental factors such as traffic load, temperature, humidity and material characteristics are added as heterogeneous attributes of nodes or edges to form a potential energy time series graph containing multiple types of nodes and multiple relational edges.

[0092] This dynamic heterogeneous graph structure can preserve the characteristics of crack energy distribution and stress concentration in the spatial dimension and characterize the dynamic process of potential energy evolution in the temporal dimension, providing a unified structured input for subsequent time series modeling and prediction through the potential energy-driven dynamic heterogeneous graph architecture search model (PE-DHGAS).

[0093] S105 uses the Potential Energy Driven Dynamic Heterogeneous Graph Architecture Search Model (PE-DHGAS) to adaptively learn and temporally model the structure, outputting a road crack development prediction image.

[0094] In this embodiment, the electronic device inputs the potential energy timing diagram obtained in step S104 into the potential energy driven dynamic heterogeneous graph architecture search model (PE-DHGAS).

[0095] The model is based on an automated structure search mechanism, which can dynamically optimize the structural parameters of the graph neural network under the physical constraints of potential energy distribution, and realize the joint modeling of potential energy temporal characteristics and heterogeneous interaction relationships.

[0096] In a practical implementation, electronic devices can encode nodes and edges in the potential energy time series graph, where nodes represent crack regions or environmental factor units, and edges represent energy transfer paths or heterogeneous feature associations. The PE-DHGAS model captures the spatial energy associations between different nodes through automatically generated graph attention units, and learns the dynamic law of potential energy changing over time through temporal convolutional layers or recurrent structures.

[0097] Optionally, in some embodiments, the PE-DHGAS model includes a first sub-network for extracting spatial potential energy distribution features and a second sub-network for modeling the temporal variation law of potential energy; wherein, the first sub-network is used to identify the energy distribution pattern and heterogeneous interaction relationship between different regions, and the second sub-network is used to capture the evolution trend of potential energy in the time dimension.

[0098] Furthermore, the structural optimization process of the PE-DHGAS0 model adopts a multi-stage differentiable architecture search algorithm: in the first stage, the embedding position of the attention module is determined; in the second stage, the combination of attention weight parameters is optimized, thereby automatically discovering the optimal spatiotemporal fusion structure.

[0099] Through adaptive learning and temporal modeling of the model, the electronic device can output a predicted image of road crack development at future moments; this predicted image is used to characterize the crack's expansion direction, growth rate, and potential merging area.

[0100] By employing a potential energy-driven dynamic heterogeneous graph architecture search mechanism, this embodiment can characterize the driving force of crack propagation in the physical sense of energy evolution, achieve high-precision prediction of crack development trends, and significantly improve the stability, generalization and physical interpretability of the model.

[0101] Optionally, in some embodiments, after executing S105, the electronic device can further determine the current crack state and predict the crack deterioration trend based on the road crack development prediction image. Further, the electronic device can calculate a risk index based on the current crack state, the predicted crack deterioration trend, and traffic load information.

[0102] Furthermore, the risk index The calculation is performed using a comprehensive model that integrates the current state, evolution trend, and external loads. The calculation formula is as follows:

[0103]

[0104] Among them: The current severity is based on the current total length of the crack. With maximum width Calculated composite indicators, such as ,in This is a coefficient determined based on road grade. Crack type (e.g., reticular, longitudinal) can also be included as a correction factor in the calculation; The trend evolution threat score integrates information on the dynamic changes of cracks, such as... ,in The growth rate predicted by the PE-DHGAS model. This represents the recently observed rate of change in width. As weight; For load-environment sensitivity, it is determined by the normalized traffic load intensity. It is determined together with environmental impact factors. The weighting coefficients for each dimension sum to 1, and can be adaptively determined by analyzing the correlation between historical maintenance data and crack failure events.

[0105] Furthermore, electronic devices can classify cracks into high-risk, medium-risk, or low-risk categories based on a risk index. Further, based on the risk level, the electronic devices can generate treatment recommendations, including short-term repairs, medium-term maintenance, or long-term reinforcement. By generating risk levels and treatment recommendations, road crack repairs can be prioritized for predicted road sections with higher risk levels more accurately and promptly, thereby ensuring road safety.

[0106] Alternatively, in another embodiment, after obtaining the risk level, the electronic device can sort the risk levels of the predicted road segments from high to low and generate a sorting list. This allows staff to more clearly determine which or more predicted road segments should be prioritized for repair and maintenance based on the sorting list generated by the electronic device.

[0107] Alternatively, in another embodiment, after receiving the handling recommendations, the electronic device can combine the recommendations with traffic load information (e.g., traffic flow) and resource availability to further generate a maintenance plan for the predicted road segment. This allows for further optimization and supplementation of the handling recommendations.

[0108] The road crack development prediction method based on temporal imagery and environmental factors in this embodiment provides a data foundation for subsequent road crack prediction processing by acquiring temporal images of road cracks in the predicted road segment. Acquiring environmental parameters within the corresponding time period of the road crack temporal images further provides a data foundation for subsequent road crack prediction processing. Based on this, the system can extract crack geometric features, topological features, and temporal change features from the road crack temporal images and corresponding environmental parameters, and fuse them into a comprehensive feature vector. Further, after constructing the comprehensive feature vector, a crack evolution potential energy field can be generated based on this vector, and the potential energy field can be represented as a dynamic heterogeneous graph structure. Subsequently, the dynamic heterogeneous graph structure is input into a potential energy-driven dynamic heterogeneous graph architecture search model, and adaptive learning and temporal modeling are performed through the model to obtain a road crack development prediction image.

[0109] Therefore, crack evolution prediction based on multiple data sources (including temporal images of road cracks and environmental parameters) combined with a temporal modeling mechanism can be achieved. This enables more accurate prediction of road crack evolution trends, and based on the predicted trends, more comprehensive and precise maintenance and repair strategies can be provided, effectively improving road safety and maintenance efficiency. Furthermore, the input of different environmental parameters allows the road crack prediction results to reflect the combined influence of multiple environmental and structural factors, improving the scientific rigor, universality, and reliability of the prediction.

[0110] Based on the description in S101, the electronic device can acquire time-series images of road cracks. In this application embodiment, the electronic device can employ various possible implementation methods to acquire time-series images of road cracks. The following details one possible implementation method for the electronic device to acquire time-series images of road cracks.

[0111] Please see Figure 2 , Figure 2 A flowchart illustrating a method for constructing a time-series image of road cracks, as provided in an embodiment of this application, is shown.

[0112] like Figure 2 As shown, the method for constructing time-series images of road cracks in this embodiment includes the following steps:

[0113] S201, acquire multiple road crack images of the predicted road segment.

[0114] Optionally, in some embodiments, the multiple road crack images include at least two road crack images, and each road crack image was acquired at a different time. For example, when the multiple road crack images include more than two road crack images, the multiple road crack images can be represented as a single road crack image. Images of road cracks ... images of road cracks Among them, road crack images Images of road cracks ... images of road cracks The collection times were all different.

[0115] Alternatively, in another embodiment, the acquisition device can be a mobile mapping vehicle, equipped with an industrial-grade digital camera featuring a wide-angle, low-distortion lens and a precision positioning device. The mobile mapping vehicle is used for the automatic acquisition of road crack images. The acquisition device, through its industrial-grade digital camera equipped with a wide-angle, low-distortion lens, ensures coverage of the road surface of the predicted road segment and clearly records road crack details.

[0116] It should be noted that industrial-grade digital cameras can be fixed to a suitable position on the chassis of a mobile surveying vehicle using shock-absorbing brackets. The cameras can be installed vertically downwards or at a certain angle to ensure complete acquisition of road crack images.

[0117] Alternatively, in another embodiment, the mobile mapping vehicle can collect road crack images of a preset road segment at a preset acquisition frequency (e.g., one frame every 5 meters traveled) while traveling along the road. This ensures that adjacent road crack images have sufficient overlap in the field of view, avoiding missed detection of road cracks.

[0118] It should be noted that the preset acquisition frequency can be automatically adjusted based on the vehicle speed of the mobile mapping vehicle or manually set and adjusted by those skilled in the art, and this application embodiment does not limit this. For example, when the speed of the mobile mapping vehicle is 30 km / h, continuous shooting is performed at a preset acquisition frequency of approximately 10 frames / second.

[0119] Optionally, in some embodiments, the road crack images acquired by the mobile mapping vehicle are stored in real time on the vehicle's onboard computer, and timestamps and geographic coordinates are automatically added. This enables electronic devices to determine the acquisition time of each road crack image and the specific location of the road crack based on the road crack images transmitted by the acquisition equipment.

[0120] Optionally, in some embodiments, the mobile surveying vehicle can conduct roving inspections on the same road segment according to a set collection cycle (e.g., one month or one quarter) to obtain at least three road crack image data, and then obtain the time sequence of road crack evolution through at least three road crack image data.

[0121] After the mobile mapping vehicle collects multiple images of road cracks in the predicted road segment, they can be transmitted to electronic devices, enabling the electronic devices to obtain multiple images of road cracks in the same predicted road segment at different times, and providing a data basis for subsequent processing.

[0122] It should be noted that the mobile surveying vehicle can transmit multiple road crack images to electronic devices via wired transmission (e.g., data cable) or wireless transmission (e.g., Bluetooth), and this application embodiment does not limit this.

[0123] S202, based on multiple road crack images, constructs a time-series image of road cracks.

[0124] After obtaining multiple road crack images of the same predicted road segment at different times, electronic devices can construct a time-series image of road cracks based on the multiple road crack images, thereby providing a data foundation for subsequent road crack prediction processing.

[0125] The electronic device acquires temporal images of road cracks based on multiple road crack images, specifically including the following steps:

[0126] S2021, perform a first matching on multiple road crack images through common feature points, map them to the same coordinate system, and generate a time-series image of the road cracks.

[0127] Optionally, in some embodiments, after obtaining multiple road crack images of the same predicted road segment at different times, the electronic device can use the multiple road crack images as reference images and images to be registered, respectively. Further, the electronic device translates the image to be registered to the corresponding area of ​​the reference image based on the geographic coordinates (e.g., Global Positioning System (GPS) coordinates) carried by each road crack image, achieving a first match, and thereby generating a temporal image of the road cracks.

[0128] S2022, the time series images are subjected to a second matching through a geometric deformation field to generate an initial road crack time series image.

[0129] Optionally, in some embodiments, after obtaining a time-series image of the road crack, the electronic device performs a second matching of the time-series image through a geometric deformation field to generate an initial time-series image of the road crack.

[0130] Optionally, in some embodiments, the second matching includes initial matching and exact matching. For example, the electronic device may use robust feature detection algorithms such as Scale-Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF) to extract feature points and complete the initial matching based on descriptor similarity.

[0131] Furthermore, after the electronic device uses the Random Sample Consensus (RANSAC) algorithm to remove mismatched point pairs, it calculates the spatial geometric transformation relationship between the time-series images, mapping the image to be registered to the coordinate system of the reference image. The spatial geometric transformation relationship can be expressed using the perspective transformation homography matrix H, which satisfies the following equation:

[0132] ;

[0133] in, The pixel coordinates of the image to be registered. These are the pixel coordinates of the corresponding reference image.

[0134] Furthermore, the electronic device performs precise matching of the temporal images of the two road cracks through affine and perspective transformations to obtain the initial temporal image of the road cracks, thus preparing for subsequent processing.

[0135] S2023 inputs the initial road crack time series image into a deep learning network for detail enhancement, and outputs the road crack time series image when the recognition of crack edge features and branch features both reach the target threshold.

[0136] Optionally, in some embodiments, after obtaining the initial road crack time-series image, the electronic device inputs the initial road crack time-series image into a deep learning network for detail enhancement.

[0137] For example, the deep learning network is an augmentation network, which has an encoder-decoder symmetric structure. Electronic devices extract multi-scale features through the convolutions of the augmentation network and fuse them with skip connections during the decoding stage to preserve crack edge information, thereby enhancing the texture details of road cracks and suppressing noise.

[0138] Furthermore, after detail enhancement of the initial road crack time-series image, the electronic device determines whether the discriminability of crack edge features and branch features reaches a target threshold. When both the discriminability of crack edge features and branch features reach the target threshold, the electronic device outputs the road crack time-series image.

[0139] Based on the description in S103, the electronic device can determine a comprehensive feature vector based on the time-series image of road cracks and environmental quantities. In this embodiment, the electronic device can employ various possible implementation methods to determine the comprehensive feature vector. The following details one possible implementation method for the electronic device to determine the comprehensive feature vector.

[0140] Please see Figure 3 , Figure 3 A flowchart of a comprehensive feature vector determination method provided in an embodiment of this application is shown.

[0141] like Figure 3 As shown, the comprehensive feature vector determination method in this embodiment includes the following steps:

[0142] S301, perform binarization processing on the time-series images of road cracks, and determine whether the time-series images of road cracks have target features during the binarization process.

[0143] Optionally, in some embodiments, the electronic device can binarize the road crack time series image using the Otsu's Method (OTSU) and determine whether the road crack time series image has target features during the binarization process.

[0144] Alternatively, in another embodiment, the electronic device preprocesses the road crack time-series image before binarizing it, thereby preparing it for subsequent binarization.

[0145] S302, when the road crack time series image has target features, generate the first crack binary image.

[0146] When the time-series image of a road crack is determined to contain target features, the electronic device generates a first binary image of the crack. Specifically, when the time-series image of a road crack contains target features, the electronic device uses OTSU to automatically calculate the optimal threshold based on the grayscale histogram of the time-series image of the road crack, and divides the pixels of the time-series image of the road crack into foreground (or crack area) and background (or road surface area), obtaining a first binary image of the crack containing only 0 or 1, thus preparing for subsequent processing.

[0147] S303. Based on the first crack binary image, extract the crack skeleton features and remove isolated short branches of the crack to obtain the second crack binary image.

[0148] After obtaining the first crack binary image, the electronic device uses a thinning algorithm to extract the crack skeleton features of the first crack binary image and removes isolated short branches of the crack to generate the second crack binary image.

[0149] For example, after obtaining the first crack binary image, the electronic device uses a skeleton extraction algorithm (Zhang-Suen Thinning Algorithm) to convert the cracks in the first crack binary image into skeleton lines with a width of one pixel, and removes isolated short branches (or burr branches of cracks) with a length lower than a threshold to obtain the second crack binary image, and the crack skeleton network structure in the second crack binary image is simpler.

[0150] It should be noted that the skeleton extraction algorithm is one of the thinning algorithms. Those skilled in the art can use other thinning algorithms to extract the crack skeleton network, and the embodiments of this application are not limited to this.

[0151] S304. Based on the binary image of the second crack, extract the crack geometric features, crack topological features, and image temporal parameters.

[0152] Optionally, in some embodiments, the crack geometry features include crack length, crack average width, crack maximum width, crack area, and crack direction.

[0153] Optionally, in some embodiments, the crack topological features include a set of crack feature points, a set of crack segments, and a first connectivity relationship.

[0154] Optionally, in some embodiments, the set of crack feature points and the set of crack segments are obtained based on the crack skeleton network after connecting the feature points.

[0155] Optionally, in some embodiments, a first connectivity relation is used to represent the connectivity relationship between various crack feature points, and the first connectivity relation is recorded using a connected adjacency matrix.

[0156] Optionally, in another embodiment, the crack topology features further include the number of crack endpoints, the number of bifurcations, the number of branches, and the number of connected components. The specific meanings of each crack topology feature are shown in Table 1 below:

[0157] Table 1

[0158]

[0159] Optionally, in some embodiments, the image temporal parameter refers to the temporal order of multiple road crack images combined with timestamps.

[0160] After obtaining the binary image of the second crack, the electronic device extracts the crack's geometric features, topological features, and temporal parameters based on the image. Specifically, the electronic device extracts the set of crack feature points from the crack topological features based on the connection relationships of each feature point in the crack skeleton network of the second crack binary image. and crack segment set And using the connected adjacency matrix The connectivity between various feature points is described. At the segment level of the crack skeleton network in the second crack binary image, the electronic device uses crack segments as analysis units to calculate crack length and crack direction, thereby obtaining the crack geometric features in the second crack binary image.

[0161] Based on the above description, the electronic device can obtain the geometric features of the crack, the topological features of the crack, and the temporal parameters of the image, thereby providing a data foundation for subsequent processing.

[0162] S305, based on the geometric and topological features of cracks, uses a graph structure to represent cracks, resulting in a road crack graph structure vector.

[0163] After obtaining the geometric and topological features of the cracks, the electronic device represents the cracks in a graph structure, resulting in a road crack graph structure vector. This road crack graph structure vector provides the data foundation for the subsequent generation of a comprehensive feature vector.

[0164] S306, based on image time series parameters, concatenates the road crack map structure vector at each time step with the corresponding environmental quantity, and normalizes the concatenation result to obtain a comprehensive feature vector.

[0165] After obtaining the road crack image structure vectors at each time step, the electronic device concatenates these vectors with corresponding environmental variables based on image temporal parameters. The concatenated results are then normalized to obtain a comprehensive feature vector. This comprehensive feature vector provides a data foundation for subsequent road crack prediction.

[0166] Considering that the stitching process is the same at all times, the following explanation uses the stitching process at the first time point as an example. Here, the first time point can be any one of multiple time points. The stitching process at the first time point includes:

[0167] Optionally, in some embodiments, the electronic device acquires the road surface material type in the environmental data, encodes the road surface material type into a categorical vector, and embeds it into the road crack map structure vector at a first moment.

[0168] For example, the electronic device can pre-build a material category index table for different road surface material types. The material category index table categorizes common road surface materials such as asphalt concrete, cement concrete, and composite pavement, and assigns a unique index value to each category. The material category index table is shown in Table 2 below:

[0169] Table 2

[0170]

[0171] Furthermore, the electronic device utilizes an embedding layer to transform the index of the road material type into a low-dimensional learnable vector, which is then embedded into the road crack map structure vector at the first time step. This transformation of the road material type index into a low-dimensional learnable vector facilitates joint modeling of the road material type and the road crack map structure vector, thereby enhancing the model's adaptability to differences in various road material types.

[0172] Optionally, in some embodiments, the electronic device acquires traffic load information from the environment, performs statistical feature extraction and normalization on the traffic load information, and splices the features of the obtained traffic load information into the road crack map structure vector at the first moment.

[0173] For example, regarding traffic load information, electronic devices can obtain original records of traffic flow, vehicle speed, and vehicle weight from traffic load information through deployed traffic detection equipment within the time interval between two adjacent road crack image captures.

[0174] Furthermore, the electronic equipment extracts statistical features from the traffic load information, including:

[0175] Regarding traffic flow, electronic devices can extract statistical features of traffic flow by calculating the total number of vehicles passing through, the average throughput per unit time of the interval, the maximum flow rate per hour, and the standard deviation of traffic flow.

[0176] Regarding vehicle speed, electronic devices can extract statistical features of vehicle speed by calculating the average speed, standard deviation of speed, and peak and trough values ​​of speed within a range.

[0177] Regarding vehicle weight, electronic devices can extract statistical features of vehicle weight by calculating the average vehicle weight within a range, the total number of heavy vehicles, the proportion of heavy vehicles, and the average load rate per unit time.

[0178] Furthermore, based on the above description, after the electronic device completes the feature extraction of traffic flow, vehicle speed, and vehicle weight, it can normalize these multiple statistical feature indicators. After normalization, the electronic device can concatenate the normalized statistical feature indicators into the road crack map structure vector of the first frame at the first moment.

[0179] Optionally, in some embodiments, the electronic device acquires environmental and climatic factors from the environmental data, performs statistical feature extraction and normalization on the environmental and climatic factors, and splices the obtained features of the environmental and climatic factors into the road crack map structure vector at the first moment.

[0180] For example, in terms of environmental and climatic factors, electronic devices can also obtain indicators such as temperature and humidity through a meteorological monitoring platform, and extract statistical features from indicators such as temperature and humidity.

[0181] It should be noted that the relevant descriptions of indicators such as temperature and humidity can be found in S102, and will not be repeated here.

[0182] Furthermore, the electronic device extracts statistical features from indicators such as temperature and humidity, including:

[0183] The electronic device quantifies and normalizes indicators such as temperature and humidity, and then splices the quantified and normalized temperature and humidity indicators into the first frame of the road crack map structure vector at the first moment.

[0184] It should be noted that external factor features (including traffic load information, road surface material type, and environmental and climatic factors) are all given in numerical vector form. This allows the electronic device to obtain a preliminary multi-source fusion feature representation by concatenating these three types of features in the same feature space. Based on the above description, after completing the concatenation process, the electronic device can obtain the comprehensive feature vector at the first moment.

[0185] Alternatively, in another embodiment, the electronic device may normalize or standardize the fused integrated feature vector to avoid the impact of differences in the dimensions and numerical ranges of features from different sources on the modeling of subsequent models.

[0186] It should be noted that in this embodiment, the preferred normalization method is min-max normalization, which linearly scales each feature to the [0,1] interval according to its maximum and minimum values; when there are obvious differences in the distribution of each feature, zero-mean unit variance standardization (Z-score Standardization, Z-score) is used, which subtracts the mean from the feature value and divides it by the standard deviation to obtain the feature distribution with a mean of 0 and a variance of 1.

[0187] Based on the description in S104, the electronic device can construct a crack evolution potential energy field according to the comprehensive feature vector and map the potential energy field into a dynamic heterogeneous graph structure. The crack evolution potential energy field is used to characterize the energy distribution, stress concentration, and potential propagation direction of the crack region. In the embodiments of this application, the electronic device can adopt a variety of possible implementation methods to complete the construction of the potential energy field and the graph structure mapping. The following details one possible implementation method of the electronic device constructing the crack evolution potential energy field and its mapping.

[0188] Please see Figure 4 , Figure 4 A flowchart of a method for constructing a crack evolution potential energy field provided in an embodiment of this application is shown.

[0189] like Figure 4 As shown, the method for constructing the crack evolution potential energy field in this embodiment includes the following steps:

[0190] S401, calculate the energy gradient distribution in the crack region based on the comprehensive feature vector.

[0191] After obtaining the comprehensive feature vector, the electronic device can extract features such as crack morphology change rate, grayscale gradient change, traffic load intensity, and environmental climate fluctuations, and calculate the energy gradient distribution of crack pixels in space based on these features.

[0192] Optionally, in some embodiments, the electronic device can convert the geometric characteristics of the crack into corresponding energy changes by establishing a mapping relationship between the energy gradient and the crack's geometric changes. By calculating the energy gradient distribution, the degree of stress concentration and potential propagation direction within the crack region can be reflected, providing a physical basis for the subsequent construction of the potential energy field.

[0193] S402, construct the potential energy field for crack evolution based on the energy gradient distribution.

[0194] After obtaining the energy gradient distribution, the electronic device can establish a potential energy field model of the crack region based on the spatial distribution of the energy gradient.

[0195] Optionally, in some embodiments, the potential energy value of the crack evolution potential energy field It can be determined by the following formula:

[0196]

[0197] in, This represents the grayscale change rate of the crack time-series image. This represents the dynamic response characteristics of environmental quantities. and These are adjustable weighting coefficients.

[0198] Through the above calculations, the electronic device can obtain a three-dimensional potential energy field that varies with time, which can be used to characterize the energy distribution and evolution trend of the crack region.

[0199] S403 maps the crack evolution potential field into a dynamic heterogeneous graph structure.

[0200] After obtaining the potential energy field, the electronic device can transform the continuous potential energy distribution data into a graph representation that can be modeled by a graph neural network.

[0201] Specifically, electronic devices can treat pixel regions or local grid cells in the crack evolution potential energy field as nodes of a graph, and the energy gradient direction between nodes as directed edges to represent the energy transfer path from high potential regions to low potential regions. At the same time, environmental information such as traffic load, temperature, humidity, and road surface material is added as heterogeneous features of nodes or edges, thereby forming a potential energy time series graph structure containing multiple types of nodes and multiple relational edges.

[0202] Optionally, in some embodiments, the electronic device can establish a continuous mapping relationship between multiple frames of potential energy maps based on time-series changes, thereby realizing time-series modeling of dynamic heterogeneous maps. This mapping method can preserve the crack energy distribution and stress concentration characteristics in the spatial dimension and reflect the dynamic evolution of potential energy with environmental changes in the temporal dimension.

[0203] Through steps S401 to S403 described above, the electronic device can construct a physically meaningful crack evolution potential energy field based on the comprehensive feature vector, and express the coupling relationship between energy distribution and the environment in the form of a graph structure. This structure not only improves the physical interpretability of crack propagation modeling, but also provides a unified input format and structured data support for the subsequent potential energy driven dynamic heterogeneous graph architecture search model (PE-DHGAS).

[0204] Based on the description in S105, the electronic device can input the potential energy time-series graph structure obtained in S104 into the potential energy-driven dynamic heterogeneous graph architecture search model (PE-DHGAS) to achieve joint modeling of the temporal characteristics of the potential energy field and heterogeneous interaction relationships, and output a road crack development prediction image. In the embodiments of this application, the electronic device can adopt a variety of possible implementation methods to achieve crack development prediction. The following describes in detail one possible implementation method of the electronic device obtaining the prediction image based on the PE-DHGAS model.

[0205] Please see Figure 5 , Figure 5 This illustration shows a schematic diagram of a road crack development prediction process based on the PE-DHGAS model provided in an embodiment of this application.

[0206] like Figure 5 As shown, the electronic device can perform node encoding and structure search processing on the potential energy time series diagram, and achieve adaptive prediction of crack evolution trends through the PE-DHGAS model. Specifically, it includes the following steps:

[0207] S501, Dynamic Heterogeneous Graph Feature Encoding and Search Space Construction.

[0208] After obtaining the potential energy time series graph, the electronic device can first embed and encode the physical properties of the nodes in the graph (such as potential energy value, gradient direction, stress concentration, etc.). At the same time, different types of nodes (such as crack region nodes, environmental factor nodes, traffic load nodes) and their multiple relational edges (such as energy conduction, temperature influence, load action) are used as model inputs to form a multi-type heterogeneous graph feature set.

[0209] Optionally, in some embodiments, the PE-DHGAS model can automatically define two search spaces: a location space, used to determine the location where attention mechanisms and aggregation operations are applied in a heterogeneous graph structure; and a parameterization space, used to define parameter sharing strategies for different node and relation types.

[0210] S502, multi-stage differentiable structure search and adaptive temporal modeling.

[0211] The electronic device can execute a multi-stage differentiable structure search algorithm in the search space to automatically determine the optimal graph neural network topology and attention parameter combination.

[0212] In the first stage, the model determines the optimal layout of graph convolution or graph attention modules based on the spatial dependencies of different node types in the potential energy temporal graph. In the second stage, the model uses temporal convolution or recurrent units to model the dynamic changes in potential energy distribution in the time dimension, thereby achieving automatic extraction of temporal dependency features.

[0213] Optionally, in some embodiments, the model may introduce a potential energy-driven constraint term during the search process, so that the structural search direction is physically constrained by the potential energy distribution gradient and the crack propagation trend, thereby maintaining physical consistency and stability during the learning phase.

[0214] Through this multi-stage differentiable structure search mechanism, the model can complete adaptive learning of complex spatiotemporal features without human intervention, significantly improving the accuracy and generalization ability of prediction.

[0215] S503 outputs prediction results and generates feature indicators.

[0216] After completing adaptive modeling, the electronic device can generate a road crack development prediction image based on the output of the PE-DHGAS model. The prediction image can visually display the future expansion direction, growth rate, and potential merging areas of the cracks.

[0217] Optionally, in some embodiments, the electronic device can also calculate a crack evolution risk index based on the predicted image to assist in road maintenance and risk assessment; in addition, the system can execute an interpretability analysis module in the output stage to generate a crack propagation heat map based on the model attention weight or potential energy contribution to enhance the physical interpretability of the prediction results.

[0218] Through this step, electronic devices can characterize the driving force of crack propagation in the physical sense of energy changes, and achieve dynamic, continuous, and high-precision prediction of crack development.

[0219] Through processing steps S501 to S503, the electronic device utilizes the PE-DHGAS model to achieve automatic feature learning and spatiotemporal dependency modeling of the crack evolution potential energy field. This not only adaptively identifies the impact of different environmental factors on crack propagation but also optimizes the model structure and parameters under physical constraints. Compared to traditional manually designed time-series models, the PE-DHGAS model of this invention significantly improves structural flexibility, predictive stability, and physical consistency, thus providing more forward-looking support for road crack maintenance decisions.

[0220] The above combination Figures 1 to 5 This paper details the specific implementation process of a road crack development prediction method based on temporal imagery and environmental factors, as provided in the embodiments of this application. The following section combines... Figure 6 This application provides a detailed description of a road crack development prediction system based on time-series images and environmental factors, as provided in the embodiments of this application.

[0221] Please see Figure 6 , Figure 6 The diagram shows a structural block diagram of a road crack development prediction system based on time-series images and environmental factors, provided in an embodiment of this application.

[0222] like Figure 6 As shown, a road crack development prediction system 100 based on time-series images and environmental factors in this application embodiment includes a multi-source data acquisition module 101, a feature fusion processing module 102, a crack evolution potential energy field construction module 103, and a prediction module 104.

[0223] The multi-source data acquisition module 101 is used to acquire the time-series image of road cracks in the predicted road segment, as well as the multi-source potential energy field environmental quantities such as traffic load, environmental climate, and pavement materials within the time period corresponding to the time-series image of road cracks.

[0224] The feature fusion processing module 102 is used to determine a comprehensive feature vector based on the time-series image of the road crack and the environmental quantity; wherein, the comprehensive feature vector is a comprehensive feature vector of at least one target feature under the environmental quantity; the target feature is a feature used to characterize road damage, and the target feature includes crack edge features and branch features.

[0225] The crack evolution potential energy field construction module 103 is used to construct a crack evolution potential energy field based on the comprehensive feature vector, and map the potential energy field into a dynamic heterogeneous graph structure to form a potential energy time series graph containing multiple types of nodes and multiple relational edges; the crack evolution potential energy field is used to characterize the energy distribution and potential expansion direction of the crack region.

[0226] The prediction module 104 is used to receive the output results of the potential energy driven dynamic heterogeneous graph architecture search model (PE-DHGAS) and generate a road crack development prediction image based on the automatic structure search and time series modeling results of the model. The prediction image is used to characterize the crack extension direction, growth rate and potential merging area, and can further provide auxiliary decision-making basis for road maintenance and risk assessment.

[0227] This application discloses a road crack development prediction system based on temporal imagery and environmental factors. This system utilizes multi-source data (including temporal images of road cracks and corresponding environmental parameters) and a potential-driven dynamic heterogeneous graph architecture search model to model temporal characteristics, achieving accurate prediction of road crack evolution trends. Based on the predicted crack development direction, growth rate, and potential merging areas, more comprehensive and accurate maintenance and repair strategies can be generated, thereby effectively improving the safety and operational efficiency of road facilities.

[0228] For example, this application provides a readable storage medium storing a computer program and processor calling instructions, which cause an electronic device to implement the methods in the preceding embodiments.

[0229] For example, this application provides a chip system applied to an electronic device including a memory, a display screen, and sensors; the chip system includes: one or more interface circuits and one or more processors; the interface circuits and processors are interconnected via lines; the interface circuits are used to receive signals from the memory and send signals to the processors, the signals including computer code or instructions stored in the memory; the processors invoke instructions to cause the electronic device to execute the methods in the preceding embodiments.

[0230] For example, this application provides a computer program product that, when run on a computer, causes an electronic device to implement the methods described in the preceding embodiments.

[0231] In the above embodiments, all or part of the functionality can be implemented by software, hardware, or a combination of software and hardware. When implemented using software, it can be implemented wholly or partially in the form of a computer program product. A computer program product includes one or more computer codes or instructions. When the computer program code or instructions are loaded and executed on a computer, all or part of the flow or functionality according to this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer code or instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0232] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The above embodiments are only used to illustrate the technical solutions of the present invention, facilitating understanding and implementation by those skilled in the art. The scope of protection of the present invention is not limited to the above embodiments; any equivalent substitutions or improvements made under the guidance of the present invention should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for predicting road crack development based on temporal imagery and environmental factors, characterized in that, include: Obtain temporal images of road cracks in the predicted road segment; Acquire multi-source potential energy field environmental quantities within the time period corresponding to the road crack time-series image, wherein the environmental quantities include at least one of traffic load information, environmental-climate factors, and pavement material factors; A comprehensive feature vector is determined based on the time-series images of the road cracks and the environmental parameters. The comprehensive feature vector is a comprehensive feature vector of at least one target feature under the environmental parameters. The target feature includes crack edge features and branch features that characterize road damage. Based on the comprehensive feature vector, a potential energy field representing the energy distribution and potential propagation direction of the crack region is constructed. The construction of this potential energy field includes: calculating the spatial energy gradient distribution of crack pixels based on the crack morphology change rate, grayscale gradient change, traffic load intensity, and environmental climate fluctuation characteristics in the comprehensive feature vector; establishing a potential energy function for the crack region based on the energy gradient distribution to characterize the energy accumulation degree and propagation driving force of the crack region; and determining the potential energy value of the potential energy field. Determined by the following formula: in, This represents the grayscale change rate of the crack time-series image. This represents the dynamic response characteristics of environmental quantities. and The weighting coefficients are adjustable; by modeling the changes in the potential energy value over time, a potential energy time series diagram representing the energy evolution trend is formed. The potential energy field is then mapped into a dynamic heterogeneous graph structure to form a potential energy time series graph containing multiple types of nodes and multiple relational edges. The mapping of the potential energy field into a dynamic heterogeneous graph structure includes: mapping pixel regions or grid cells in the potential energy field into nodes, mapping the potential energy gradient direction into directed edges between nodes, and mapping environmental quantity features from different sources into heterogeneous nodes or edge attributes, thereby forming a potential energy time series graph structure containing multiple types of nodes and multiple relational edges. The dynamic heterogeneous graph structure is input into the potential energy field-driven dynamic heterogeneous graph architecture search model. Through automatic structure search, the joint modeling of the temporal characteristics of the potential energy field and the heterogeneous interaction relationship is realized, and the road crack development prediction image is output. The prediction image represents the crack's expansion direction, growth rate and potential merging area. The potential field-driven dynamic heterogeneous graph architecture search model includes: a first sub-network for extracting the spatial distribution features of the potential field, and a second sub-network for modeling the temporal variation law of the potential field; the first sub-network captures the spatial energy correlation between different nodes through graph convolution or attention mechanism, and the second sub-network learns the dynamic dependence of potential energy over time through temporal convolution or recurrent units. The automatic structure search includes: A localization space for describing the model topology and a parameterization space for describing the attention parameters are constructed, and a multi-stage differentiable search algorithm is used to optimize the model structure in the space. Specifically, the location of the attention module is determined in the first stage, and the parameter combination of the attention weights is optimized in the second stage to achieve joint modeling of the temporal characteristics of the potential energy field and the heterogeneous interaction relationship.

2. The road crack development prediction method according to claim 1, characterized in that, The acquisition of the time-series image of road cracks in the predicted road segment includes: Multiple road crack images of the predicted road segment are acquired, including at least two road crack images, and each road crack image is acquired at a different time. Based on the multiple road crack images, a time-series image of the road cracks is constructed, which includes: The multiple road crack images are matched using common feature points and mapped to the same coordinate system to generate a time-series image of the road cracks. The time-series images are subjected to a second matching process using a geometric deformation field to generate an initial time-series image of road cracks. The initial road crack time-series image is input into a deep learning network for detail enhancement, and the road crack time-series image is output when the recognizability of the crack edge features and the recognizability of the branch features both reach the target threshold.

3. The road crack development prediction method according to claim 1, characterized in that, The traffic load information includes traffic flow, vehicle speed, vehicle type, and vehicle weight; the environmental-climate factors include temperature and humidity; and the road material factors include road material type.

4. The method for predicting the development of road cracks according to claim 1, characterized in that, The determination of the comprehensive feature vector based on the time-series image of the road crack and the environmental parameters includes: The road crack time-series image is binarized, and during the binarization process, it is determined whether the target feature exists in the road crack time-series image. When the target feature is present in the time-series image of the road crack, a first binary image of the crack is generated; Based on the first binary image of the crack, the crack skeleton features are extracted and isolated short branches of the crack are removed to obtain the second binary image of the crack. Based on the second binary image of the crack, crack geometric features, crack topological features, and image temporal parameters are extracted. The crack geometric features include crack length, average crack width, maximum crack width, crack area, and crack direction. The crack topological features include a set of crack feature points, a set of crack segments, and a first connectivity relation. The set of crack feature points and the set of crack segments are obtained by connecting feature points based on a crack skeleton network. The first connectivity relation represents the connectivity between various crack feature points and is recorded using a connected adjacency matrix. Based on the geometric and topological features of the crack, the crack is represented in a graph structure form to obtain a road crack graph structure vector. Based on the image time series parameters, the road crack image structure vector at each time point is concatenated with the corresponding environmental quantity, and the concatenation result is normalized to obtain the comprehensive feature vector.

5. The road crack development prediction method according to claim 4, characterized in that, The method involves concatenating the road crack map structure vector at each time step with the corresponding environmental quantity, wherein the concatenation process is the same at each time step, including: Obtain the road surface material type from the environmental data, encode the road surface material type into a category vector, and embed it into the road crack map structure vector at each time point; The traffic load information in the environmental data is obtained, the traffic load information is subjected to statistical feature extraction and normalization, and the features of the obtained traffic load information are spliced ​​into the road crack map structure vector at each time point. The environmental-climate factors in the environmental quantities are obtained, and statistical features of the environmental-climate factors are extracted and normalized. The features of the obtained environmental-climate factors are then concatenated into the road crack map structure vector at each time point. After completing the above splicing process, the comprehensive feature vectors at each time point are obtained.

6. The road crack development prediction method according to claim 1, characterized in that, After outputting the predicted image of road crack development, the method further includes: Based on the road crack development prediction image, the current crack status and crack deterioration trend are determined; Calculate the risk index based on the current crack status, the predicted crack deterioration trend, and traffic load information; Based on the aforementioned risk index, cracks are classified into high-risk, medium-risk, or low-risk categories according to their risk level. Based on the risk level, treatment recommendations are generated; the treatment recommendations include short-term repair, medium-term maintenance, or long-term reinforcement.

7. A road crack development prediction system based on temporal imagery and environmental factors, characterized in that, include: The multi-source data acquisition module is configured to acquire a time-series image of road cracks in the predicted road segment and a multi-source potential energy field environmental quantity within a time period corresponding to the time-series image of road cracks. The environmental quantity includes at least one of traffic load information, environmental and climatic factors, and pavement material factors. The feature fusion processing module is configured to determine a comprehensive feature vector based on the road crack time-series image and the environmental quantity. The comprehensive feature vector is a comprehensive feature vector of at least one target feature under the environmental quantity. The target feature includes crack edge features and branch features characterizing road damage. The crack evolution potential energy field construction module is configured to construct a crack evolution potential energy field characterizing the energy distribution and potential expansion direction of the crack region based on the comprehensive feature vector, and map the potential energy field into a dynamic heterogeneous graph structure to form a potential energy time series graph containing multiple types of nodes and multiple relational edges. The construction of the potential energy field includes: calculating the spatial energy gradient distribution of crack pixels based on crack morphology change rate, gray-level gradient change, traffic load intensity, and environmental climate fluctuation characteristics in the comprehensive feature vector; establishing a potential energy function for the crack region based on the energy gradient distribution to characterize the energy accumulation degree and expansion driving force of the crack region; and defining the potential energy value of the potential energy field. Determined by the following formula: in, This represents the grayscale change rate of the crack time-series image. This represents the dynamic response characteristics of environmental quantities. and The weighting coefficients are adjustable; by modeling the changes in the potential energy value over time, a potential energy time series graph representing the energy evolution trend is formed; the potential energy field is mapped to a dynamic heterogeneous graph structure, which includes: mapping pixel regions or grid cells in the potential energy field to nodes, mapping the potential energy gradient direction to directed edges between nodes, and mapping environmental quantity features from different sources to heterogeneous nodes or edge attributes, thereby forming a potential energy time series graph structure containing multiple types of nodes and multiple relational edges; The prediction module is configured to receive the output of the dynamic heterogeneous graph architecture search model driven by the potential energy field, and generate a road crack development prediction image based on the automatic structure search and time series modeling results of the model; the prediction image is used to characterize the crack's expansion direction, growth rate, and potential merging area. The potential field-driven dynamic heterogeneous graph architecture search model includes: a first sub-network for extracting the spatial distribution features of the potential field, and a second sub-network for modeling the temporal variation law of the potential field; the first sub-network captures the spatial energy correlation between different nodes through graph convolution or attention mechanism, and the second sub-network learns the dynamic dependence of potential energy over time through temporal convolution or recurrent units. The automatic structure search includes: A localization space for describing the model topology and a parameterization space for describing the attention parameters are constructed, and a multi-stage differentiable search algorithm is used to optimize the model structure in the space. Specifically, the location of the attention module is determined in the first stage, and the parameter combination of the attention weights is optimized in the second stage to achieve joint modeling of the temporal characteristics of the potential energy field and the heterogeneous interaction relationship.

Images