Bridge ice and snow early warning method and system based on neural network and multi-modal data fusion

Abstract

The application provides a bridge deck ice and snow early warning method and system based on a neural network and multi-modal data fusion, which comprises the following steps: forming a multi-modal feature representation based on an image of a bridge to be evaluated, meteorological parameters and a bridge deck friction coefficient; the multi-modal feature representation comprises temperature-visual features and ice and snow texture features; fusing the multi-modal feature representation through a learnable dynamic weighting mechanism to obtain a unified feature embedding representation; inputting a time sequence of the feature embedding representation into a time series neural network model to predict the future ice layer thickness of the bridge deck; and generating early warning information based on the predicted ice layer thickness. The application can accurately warn the ice formation trend of the bridge deck, thereby reducing the accident rate and improving the snow melting and deicing efficiency.

Description

Technical Field

[0001] This invention belongs to the field of highway engineering technology, specifically relating to a bridge deck ice and snow early warning method and system based on neural network and multimodal data fusion. Background Technology

[0002] Among common accidents in the transportation sector, those occurring under severe weather conditions (such as rain, snow, and extreme cold) account for a relatively high proportion of traffic accidents and economic losses, seriously affecting vehicle traffic safety and road operational efficiency. In icy and snowy weather, low temperatures and high humidity cause moisture or ice fog in the air to easily condense on the road surface, forming an ice layer. Bridge surfaces, due to faster ventilation and heat dissipation, typically experience surface temperatures that drop below freezing more quickly than on ordinary road sections, leading to faster icing and longer ice retention. In some environments, further temperature drops at night can cause melted or partially melted snow and ice to refreeze, further increasing road slipperiness.

[0003] Studies have shown that icy roads significantly reduce the coefficient of friction between tires and the road surface: approximately 0.6 on dry asphalt, 0.4 in rainy weather, 0.28 on snow-covered roads, and as low as 0.18 on icy roads, leading to a substantial increase in vehicle braking distance. When bridge surfaces are icy, vehicle traction and handling decrease significantly, making it highly susceptible to skidding, rear-end collisions, and other accidents if driving at excessive speeds or with insufficient following distance. Although some vehicles improve traction by installing snow chains or using winter-specific tires, it is still difficult to completely eliminate the risk of accidents on severely icy bridge surfaces.

[0004] Existing solutions for dealing with snow and ice accumulation on bridges in winter typically involve measures such as spreading salt, spraying chemical de-icing agents, or mechanical snow removal. However, timely and accurate assessment of bridge surface icing conditions to determine when to initiate de-icing measures, and selecting appropriate de-icing solutions for different environmental conditions, remain significant challenges. In recent years, machine vision and meteorological sensing technologies have been gradually applied to snow and ice warning systems, but their practical deployment still faces the following major shortcomings:

[0005] Traditional solutions typically rely solely on temperature, humidity, or simple meteorological parameters, neglecting the fusion of multimodal information such as visual texture, wind speed changes, and road surface friction coefficient. This results in insufficient accuracy and timeliness in identifying ice and snow conditions, with false alarms or missed alarms being quite common.

[0006] Many systems are based on pre-set fixed thresholds or static models, making it difficult to adapt to different bridge deck structures (such as steel bridges and concrete bridges) and changing environmental conditions (such as aquatic environments and mountainous environments), and thus unable to accurately model and predict different seasons and real-time weather changes.

[0007] When choosing a de-icing solution, only a single objective (such as safety or cost) is often considered, while neglecting the comprehensive trade-off between multiple objectives such as economy, environmental protection and operational efficiency. This may result in the waste of de-icing resources or environmental pollution, and may also reduce safety under certain conditions. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of the aforementioned background technology by providing a bridge surface ice and snow early warning method and system based on neural network and multimodal data fusion. This method can provide accurate early warning of bridge surface icing trends, thereby reducing the accident rate and improving snow melting and de-icing efficiency.

[0009] The technical solution adopted in this invention is: a bridge surface ice and snow early warning method based on neural network and multimodal data fusion, comprising the following steps:

[0010] A multimodal feature representation is formed based on the image of the bridge surface to be evaluated, meteorological parameters, and the bridge surface friction coefficient; the multimodal feature representation includes temperature-visual features and ice and snow texture features;

[0011] By fusing the multimodal feature representations through a learnable dynamic weighting mechanism, a unified feature embedding representation is obtained;

[0012] The time series representation of the embedded features is input into a temporal neural network model to predict the future ice thickness on the bridge surface;

[0013] Early warning information is generated based on the predicted ice thickness.

[0014] The above technical solution also includes the following steps: calculating the normalized efficiency value of each decision objective of multiple candidate de-icing schemes based on the predicted ice thickness, and calculating the evaluation value of each candidate de-icing scheme based on the corresponding weight of the decision objective, so as to select the optimal de-icing scheme; the multiple decision objectives include at least safety, economy and environmental protection.

[0015] The above technical solution also includes the following steps: storing the fused feature embedding representation in a vector database to construct a historical ice and snow event feature library; using a graph-based approximate nearest neighbor indexing algorithm to construct a topological structure for the feature library, thereby achieving fast nearest neighbor retrieval of the current feature embedding representation; and inputting the retrieved historical feature vector and the time series of the feature embedding representation into a temporal neural network model to provide historical context reference for ice thickness prediction.

[0016] In the above technical solution, when the predicted ice thickness or bridge surface friction coefficient is lower than the corresponding set threshold, a warning message is output.

[0017] In the above technical solution, the generation process of the temperature-visual feature includes:

[0018] The bridge deck multispectral image and the corresponding temperature field matrix are stitched together by channel to form a multichannel input, which includes at least a visible light image data channel and a temperature data channel.

[0019] The multi-channel image is first normalized and then enhanced with a random mask before being input into a pre-trained visual encoder based on the Transformer structure. The parameters of the front layer of the visual encoder are partially frozen to maintain the pre-trained visual representation, and the temperature field attention mechanism is introduced in the back layer to give higher weight to temperature-sensitive regions.

[0020] The visual encoder outputs a high-dimensional feature vector that fuses temperature and visual information, which is then used for subsequent multimodal feature fusion processing.

[0021] In the above technical solution, the generation process of the ice and snow texture features includes:

[0022] A backbone model based on convolutional neural networks is adopted, with some convolutional layers or residual blocks frozen, to fuse bridge deck image data, meteorological data, and bridge deck friction coefficient data as input.

[0023] After the initial feature extraction, a spatial pyramid pooling layer is connected after the last residual block of the network. The spatial pyramid pooling layer uses multiple pooling windows of different sizes to pool the output features and concatenates the pooling results of each scale into a high-dimensional vector. After batch normalization and nonlinear activation function processing, multi-dimensional ice and snow texture features are output.

[0024] In the above technical solution, the input data preprocessing step of the backbone model includes:

[0025] Bridge deck image data, meteorological data, and bridge deck friction coefficient data are stored in a database or file system in a unified format. After reading the data from the database or file system, the bridge deck image data is subjected to data enhancement processing with random rotation and color jitter. At the same time, the read meteorological data and bridge deck friction coefficient data are matched and jointly preprocessed with the processed image data in tabular form, so that when inputting into the backbone model, all modal data are unified into a standardized format.

[0026] In the above technical solution, the process of fusing the multimodal feature representations through a learnable dynamic weighting mechanism includes:

[0027] First, calculate the cosine similarity between temperature-visual features and ice and snow texture features;

[0028] When the similarity reaches or exceeds the preset threshold, the original weights are directly used to perform weighted fusion of the two features to obtain the fused features.

[0029] When the similarity is lower than the preset threshold, the weight of temperature-visual features is reduced according to the difference between the cosine similarity and the threshold, based on the preset adjustment coefficient, while the weight of ice and snow texture features is increased accordingly.

[0030] Finally, the adjusted weights are used to perform weighted fusion of the two features to obtain the final fused features.

[0031] In the above technical solution, the fuzzy hierarchical analysis decision model constructs a fuzzy judgment matrix and uses the fuzzy eigenvalue method to calculate the comprehensive weight of each decision objective; the weights of multiple decision objectives in the fuzzy hierarchical analysis decision model are dynamically adjusted based on the predicted ice thickness.

[0032] This invention also provides a bridge surface ice and snow early warning system based on neural network and multimodal data fusion, used to implement the method described in the above technical solution, including:

[0033] A multimodal feature generation module is used to generate a multimodal feature representation based on the image of the bridge surface to be evaluated, meteorological parameters, and the bridge surface friction coefficient; the multimodal feature representation includes temperature-visual features and ice and snow texture features;

[0034] The multimodal feature fusion module is used to fuse the multimodal feature representations through a learnable dynamic weighting mechanism to obtain a unified feature embedding representation;

[0035] An ice thickness prediction module is used to input the time series of the embedded feature representation into a temporal neural network model to predict the future ice thickness of the bridge surface;

[0036] The early warning generation module is used to generate early warning information based on the predicted ice thickness.

[0037] The decision generation module is used to calculate the normalized efficiency value of each decision objective of multiple candidate de-icing schemes based on the predicted ice thickness, and to calculate the evaluation value of each candidate de-icing scheme based on the corresponding weight of the decision objective, so as to select the optimal de-icing scheme; the multiple decision objectives include at least safety, economy and environmental protection.

[0038] The beneficial effects of this invention are as follows: This invention comprehensively considers image data (such as visual texture), meteorological parameters, and bridge surface friction coefficient to achieve multi-angle characterization of bridge surface icing and improve prediction accuracy; it considers the dynamic process of bridge surface ice thickness changing over time, making it closer to the real icing and melting mechanism in practical applications; the effective fusion of different modal information can significantly reduce false alarms and missed alarms, providing a more reliable data foundation for bridge surface ice and snow early warning; from raw multimodal data to the final output early warning information, it forms a practically deployable end-to-end solution.

[0039] Furthermore, this invention not only considers safety but also integrates economic and environmental factors to avoid resource waste or secondary pollution caused by single-objective decision-making; it quantitatively evaluates different de-icing methods to reduce blindness and decision-making errors; the ice thickness prediction results directly affect the decision-making process and can be adjusted according to changes in actual conditions to improve de-icing efficiency and scientific rigor.

[0040] Furthermore, this invention compares the characteristics of historical ice and snow events with the current situation, which can quickly find similar scenarios and help the model incorporate richer prior information into the prediction; based on the approximate nearest neighbor algorithm, it can efficiently retrieve data from a large-scale feature library, and its combination with the time series prediction model improves the computational efficiency; historical context reference can make up for the deficiencies of single observation data and improve the accuracy and robustness of prediction.

[0041] Furthermore, the present invention can characterize the degree of road slipperiness by both ice layer thickness and friction coefficient, and the dual judgment can reduce missed or false alarms; the threshold judgment can trigger an early warning immediately after the model output, and the warning level can be distinguished according to different threshold levels; the prediction results are linked to actual operational safety requirements in a simple and easy-to-understand way, which is convenient for deployment and implementation.

[0042] Furthermore, this invention concatenates the bridge surface multispectral image with the temperature field matrix into a multi-channel input. The multispectral image provides richer visual information, while the temperature channel helps highlight local thermal features, improving the ability to identify bridge surface icing. Utilizing general features trained on large-scale data effectively avoids training from scratch, shortening the training cycle and improving accuracy. The later layers focus on temperature-sensitive areas, ensuring higher detection accuracy in low-temperature, potentially icy areas. Random masking enhancement improves the model's robustness, providing some tolerance to environmental noise or missing information.

[0043] Furthermore, by freezing some convolutional layers, this invention retains the pre-trained network's ability to extract general visual features, while reducing the difficulty of model training and the amount of parameter adjustment; by obtaining features under different receptive fields through spatial pyramid pooling layers, it can better identify the texture differences of ice and snow at different scales; the combination of batch normalization and nonlinear activation further enhances the model's learning ability, and the output ice and snow texture features have more refined discrimination of icy areas.

[0044] Furthermore, the present invention uses a database or file system for unified storage, which is beneficial for data consistency and batch retrieval, and avoids data disorder from multiple sources; image data is aligned with non-image data such as meteorological data and friction coefficient data, so that multimodal information is effectively integrated before being input into the network; random rotation and color dithering allow the model to adapt to different shooting angles and lighting conditions, further reducing errors in actual deployment.

[0045] Furthermore, this invention dynamically allocates weights based on feature similarity, ensuring automatic adjustment in cases of information conflict or significant differences, thereby reducing fusion errors. When a feature is unreliable (low similarity), the influence of other reliable features can be appropriately amplified to enhance model robustness. Thresholds and adjustment coefficients can be optimized during training or validation to match the fusion process with the data distribution, thereby improving prediction performance.

[0046] Furthermore, the fuzzy judgment matrix used in this invention can better characterize the fuzziness and uncertainty existing in experts or actual scenarios, and is more flexible than the traditional analytic hierarchy process. When calculating the weights of multiple objectives such as safety, economy, and environmental protection, it avoids the problem of excessive subjective bias, making the comprehensive weights more reasonable. Bridge icing and weather conditions often have randomness and fuzziness, and the introduction of fuzzy analytic hierarchy process is closer to the actual situation, improving the scientific nature of decision-making. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the overall process of an embodiment;

[0048] Figure 2 This is a flowchart of the multimodal feature fusion process in an embodiment.

[0049] Figure 3 The FAHP decision matrix diagram is shown in the example. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments to facilitate a clear understanding of the present invention, but these descriptions do not constitute a limitation on the present invention.

[0051] Example 1

[0052] like Figure 1 As shown, this invention provides a bridge surface ice and snow early warning method based on neural network and multimodal data fusion, including the following steps:

[0053] S1, a multimodal feature representation is formed based on the image of the bridge surface to be evaluated, meteorological parameters, and the bridge surface friction coefficient; the multimodal feature representation includes temperature-visual features and ice and snow texture features;

[0054] S2, by fusing the multimodal feature representations through a learnable dynamic weighting mechanism, a unified feature embedding representation is obtained;

[0055] S3, The time series of the embedded feature representation is input into a temporal neural network model to predict the future ice thickness on the bridge surface;

[0056] S4 generates early warning information based on the predicted ice thickness.

[0057] S5. Based on the predicted ice thickness, calculate the normalized efficiency value of each decision objective of multiple candidate de-icing schemes, and calculate the evaluation value of each candidate de-icing scheme based on the corresponding weight of the decision objective, so as to select the optimal de-icing scheme; the multiple decision objectives include at least safety, economy and environmental protection.

[0058] Specifically, in step S1, after the system starts, it synchronously activates the data acquisition layer to acquire information from various sensors on the bridge deck, including but not limited to: infrared thermal imagers, multispectral cameras, meteorological sensors (or external real-time meteorological data sources), and vibration sensors. The specific process is as follows:

[0059] When the system starts up, it first activates the infrared thermal imager and sets its sampling frequency to 5Hz, which means it can acquire 5 frames of infrared thermal images per second.

[0060] After acquiring the raw temperature field distribution of the bridge surface using an infrared thermal imager, a bicubic interpolation algorithm is employed to interpolate and correct the pixels, refining the spatial resolution of the infrared thermal image to approximately 0.05℃ / pixel (or better) to precisely depict the temperature differences in different areas of the bridge surface. The processed temperature field data is output in frame sequence format, along with timestamps and other identifying information, which can then be matched with data from other sensors.

[0061] The multispectral camera covers the visible light band (approximately 400-700nm) and the near-infrared band (approximately 850-1700nm), and synchronously acquires bridge surface images at 30fps (frames per second) in 4K resolution, enabling it to capture the reflection and absorption characteristics of snow and ice layers under different spectra.

[0062] The CLAHE algorithm was applied to the acquired multispectral images to enhance potential snow and ice textures. In the visible light images, conversion to the HSV (Hue-Saturation-Value) color space facilitated the separation or marking of high-reflectivity snow areas.

[0063] The final output is an enhanced multispectral image sequence, which can be further overlaid with corresponding timestamps and location information for subsequent feature extraction.

[0064] The system can be equipped with an independent meteorological module (sensors for temperature, humidity, wind speed, precipitation, etc.) or interface with an external real-time meteorological data source. In this embodiment, the sensor value is read once per second by default. If the external data source has a higher frequency or has its own timestamp, it can be aligned accordingly.

[0065] Kalman filtering is used to smooth continuous sensor readings such as temperature, humidity, and precipitation, reducing occasional fluctuations or interference. For wind speed data, a sliding window of approximately 30 seconds is used to calculate variance to identify sudden gusts (rapid and short-lived sharp increases in wind speed), facilitating feature weighting in subsequent predictions of bridge icing.

[0066] The output meteorological data includes wind speed (accuracy ±0.2m / s), humidity (±2%RH), precipitation (±0.1mm), etc., which can be written to the data storage module through a unified interface.

[0067] Vibration sensors distributed at key locations on the bridge deck or structure continuously collect vibration signals at a fixed sampling rate (e.g., 100Hz or higher, depending on actual needs). The collected vibration signals are then subjected to a Fourier Transform (FFT) to extract power spectrum features in key frequency bands such as 5-15Hz:

[0068]

[0069] Where X(f) is the frequency domain amplitude and N is the number of sampling points. All sensor data are stored in a circular buffer after being timestamped, and the time synchronization accuracy is controlled within ±10ms.

[0070] This embodiment combines the dynamic and historical calibration models of vehicle loads on the bridge deck to obtain the equivalent friction coefficient variation with an accuracy of approximately ±0.05.

[0071] The system updates the friction coefficient data to the database at set time intervals (e.g., per second) or by event triggering (e.g., when a large fluctuation is detected), and aligns it with the corresponding infrared thermal images, meteorological images, and multispectral images.

[0072] To ensure that multimodal data accurately correspond to the same time or the same observation window in subsequent feature fusion, the system uses a unified NTP (Network Time Protocol) or GPS timestamp for synchronization.

[0073] The data storage format of a database can take the following forms:

[0074] Image and infrared frame data: stored in an image library or video stream in time series, and can be indexed with metadata;

[0075] Meteorological and friction coefficient data: recorded in tabular or database format, with annotations indicating accuracy or filtering status.

[0076] The collected and preprocessed multimodal data (image frames, temperature fields, meteorological parameters, friction coefficients) are uniformly stored in the database, awaiting further processing by the subsequent feature generation and fusion modules.

[0077] Through the above steps, this embodiment can completely acquire multimodal information such as images, meteorological parameters, and friction coefficients of the bridge surface to be evaluated, and perform preliminary enhancement, denoising, and time synchronization to ensure data consistency, accuracy, and real-time performance. Subsequently, in the multimodal feature generation and fusion stages (such as the extraction and weighting of temperature-visual features and ice and snow texture features), higher-precision predictive analysis can be performed based on the high-quality data output by this embodiment.

[0078] Specifically, in step S1, the process of generating the temperature-visual feature includes:

[0079] The bridge deck multispectral image and the corresponding temperature field matrix are stitched together by channel to form a multichannel input, which includes at least a visible light image data channel and a temperature data channel.

[0080] The multi-channel image is first normalized and then enhanced with a random mask before being input into a pre-trained visual encoder based on the Transformer structure. The parameters of the front layer of the visual encoder are partially frozen to maintain the pre-trained visual representation, and the temperature field attention mechanism is introduced in the back layer to give higher weight to temperature-sensitive regions.

[0081] The visual encoder outputs a high-dimensional feature vector that fuses temperature and visual information, which is then used for subsequent multimodal feature fusion processing.

[0082] Preferably, after system startup, the multispectral image is stitched together with the temperature field matrix acquired by the infrared thermal imager to form a multi-channel input including at least visible light image channels (R, G, B) and temperature data channels (T). If hardware conditions permit, near-infrared or other band information can be added to expand to more channels and further enrich the input feature dimensions. The multi-channel image is first subjected to standardization processing such as LayerNorm to ensure that the data distribution of different channels is relatively consistent, which is beneficial for the subsequent network to extract stable features.

[0083] Meanwhile, to improve the robustness and generalization ability of the model, random masking enhancement can be performed on some regions of the input image during the training phase. This involves randomly occluding or noise-enhancing a certain proportion of pixels, thereby forcing the model to focus on the overall structure of the image and key temperature hotspots.

[0084] This embodiment uses the improved CLIP model as the main body of the visual encoder, and its structure includes a 12-layer Transformer coding network.

[0085] Freezing the first 8 layers: In order to preserve the generalization ability and general visual features obtained from pre-training on large-scale image data, the parameters of the first 8 layers are fixed in this embodiment.

[0086] The last four layers introduce a temperature field attention mechanism: In this part of the multi-head attention, attention weights for the temperature channel are added, that is, higher attention is given to areas with obvious temperature gradients, making it easier for the network to identify potential low-temperature freezing areas.

[0087] The attention weights for the temperature field can be calculated using the following formula:

[0088]

[0089] Among them, T i This represents the bridge deck temperature value (in °C) collected by the i-th temperature sensor, and τ = 3.0 is the temperature scaling factor. This mechanism allows the model to assign higher weight to the visual features of areas sensitive to temperature gradient changes.

[0090] After passing through the aforementioned 12 Transformer encoder layers, the model outputs a high-dimensional feature vector (e.g., 768-dimensional) that integrates visible light and temperature information. Compared to traditional visual encoders that only use RGB channels, this approach considers the bridge surface temperature distribution simultaneously, giving the model higher sensitivity and accuracy in recognizing icy and snowy environments. This high-dimensional feature vector will serve as the "temperature-visual feature" in the subsequent multimodal feature fusion stage, participating in the final fusion along with another "ice and snow texture feature."

[0091] Specifically, in step S1, the process of generating the ice and snow texture features includes:

[0092] A backbone model based on convolutional neural networks is adopted, with some convolutional layers or residual blocks frozen, to fuse bridge deck image data, meteorological data, and bridge deck friction coefficient data as input.

[0093] After the initial feature extraction, a spatial pyramid pooling layer is connected after the last residual block of the network. The spatial pyramid pooling layer uses multiple pooling windows of different sizes to pool the output features and concatenates the pooling results of each scale into a high-dimensional vector. After batch normalization and nonlinear activation function processing, multi-dimensional ice and snow texture features are output.

[0094] The input data preprocessing steps of the backbone model include:

[0095] Bridge deck image data, meteorological data, and bridge deck friction coefficient data are stored in a database or file system in a unified format. After reading the data from the database or file system, the bridge deck image data is subjected to data enhancement processing with random rotation and color jitter. At the same time, the read meteorological data and bridge deck friction coefficient data are matched and jointly preprocessed with the processed image data in tabular form, so that when inputting into the backbone model, all modal data are unified into a standardized format.

[0096] Preferably, during system deployment, bridge deck image data (visible light images), meteorological data (such as temperature, humidity, wind speed, etc.), and bridge deck friction coefficient data are stored in a database or file system in a unified format.

[0097] When training the model or performing online inference, the required image data is first read from the database or file system; at the same time, meteorological data and friction coefficient data within the corresponding time or time window are read so that multimodal alignment can be completed in subsequent network input or fusion processing.

[0098] The read bridge surface image data is randomly rotated within a certain angle range (e.g., ±10°) to simulate the actual shooting angle difference and the viewpoint shift that may occur during vehicle movement; the hue (H), saturation (S), or brightness (V) of the image is slightly perturbed (e.g., within ±0.1) in the HSV color space to enhance the robustness of the model to different lighting conditions, camera settings, and changes in ambient light. Note that data augmentation is only used in the training phase, and random augmentation is disabled in the inference phase.

[0099] The acquired meteorological data and bridge surface friction coefficient data are tabulated and matched with image data of corresponding timestamps or frame numbers to achieve accurate alignment of multimodal data at the same time or within the same observation window. After alignment, a set of standardized multimodal input samples is formed (e.g., image + meteorological array + friction coefficient).

[0100] After completing the above preprocessing, the image is fed into the ResNet50 backbone network as input. In order to make full use of meteorological data and friction coefficient information, the data is fused in the network input or intermediate layer by splicing or embedding, depending on the design requirements.

[0101] To preserve the basic visual features trained on large-scale general data, this embodiment freezes the parameters of the first 5 convolutional blocks of ResNet50 (corresponding to layer indices 0-4), and only performs fine-tuning for ice and snow scenes in subsequent layers and fully connected layers to balance model accuracy and training cost. After passing through the last residual block of ResNet50, an SPP layer (Spatial Pyramid Pooling Layer) is connected, configuring multiple pooling windows of different sizes (such as 6×6, 3×3, 1×1) to perform multi-scale pooling on the output feature map; the feature vectors obtained from different pooling scales are concatenated together to obtain a high-dimensional feature description (e.g., 1024 dimensions, the specific dimension is determined by the number of network channels and design requirements), which can more comprehensively depict the texture distribution of ice and snow on the bridge surface at both local and global scales.

[0102] The concatenated high-dimensional vector is batch normalized (BatchNorm) to reduce the differences in data distribution between different batches and improve the model's convergence stability. Then, a non-linear activation function (such as GeLU or ReLU) is used to further enhance the network's ability to express complex features, outputting the final ice and snow texture features. In this embodiment, the output can be set to a 512-dimensional vector (which can also be adjusted depending on the network design).

[0103] The ice and snow texture features obtained through the above process effectively reflect the local texture, edge morphology, and potential slippery area distribution of ice and snow cover on the bridge surface. Combined with the meteorological and friction coefficient information read and matched with the image data in the steps, the model can more accurately capture the impact of changes in temperature, humidity, or vehicle load on ice and snow morphology when extracting local features. The features output here can be directly or through further normalization, mapping, etc., and weighted and combined with other branches (such as temperature-visual features) in the subsequent multimodal feature fusion module to provide multi-dimensional information support for ice and snow warning or thickness prediction.

[0104] This embodiment improves the parallel processing of features with different emphases ("temperature-visual" vs. "ice and snow texture") through two paths: the CLIP model and ResNet50. This enables a more comprehensive representation of bridge icing conditions. Both branches retain parameters from the first few layers of the pre-trained model, balancing general visual features with domain-specific features, reducing training costs and improving recognition accuracy in icy and snowy scenes. The SPP layer provides a multi-scale receptive field for the ResNet50 branch, exhibiting good detection capabilities for both large-area and local features that may appear in ice and snow-covered areas. Random masking, random rotation, and color jitter all enhance the model's robustness in complex real-world environments and reduce the risk of overfitting. The feature vectors output by the two branches complement each other in subsequent steps (usually through dynamic fusion using learnable weights), helping the overall model more accurately predict ice thickness or identify icy and snowy areas.

[0105] In this embodiment, the temperature-visual feature extraction and ice and snow texture feature extraction described in step S1 are performed in parallel or alternately in the computation layer. Once completed, they can be input into the feature fusion module of the next stage. This enables the formation of multimodal, multi-scale, and multi-level feature representations, laying a solid data foundation for the final bridge surface ice and snow early warning and de-icing decision-making. For different hardware configurations or demand scenarios, appropriate cropping or expansion can be made in terms of image resolution, number of channels, and CNN backbone network type, but the overall idea and technical effect can still remain consistent.

[0106] Specifically, step S2 includes the following steps: First, calculate the cosine similarity between temperature-visual features and ice and snow texture features;

[0107] Dynamic weight calculation formula:

[0108] α new =α old -γ·(θ-similarity)

[0109] Where γ is the learning rate and θ is the similarity threshold.

[0110] When the similarity reaches or exceeds the preset threshold, the original weights are directly used to perform weighted fusion of the two features to obtain the fused features.

[0111] When the similarity is lower than the preset threshold, the weight of temperature-visual features is reduced according to the difference between the cosine similarity and the threshold, based on the preset adjustment coefficient, while the weight of ice and snow texture features is increased accordingly.

[0112] Finally, the adjusted weights are used to perform weighted fusion of the two features to obtain the final fused features.

[0113] like Figure 2 As shown, preferably, in this embodiment, in order to fully combine temperature-visual features (output by the improved CLIP model) and snow texture features (output by the ResNet50+SPP branch), the system introduces a dynamic feature fusion module in step S2, the specific implementation process of which is as follows:

[0114] Assume the visual-temperature joint feature output by the CLIP branch is F visual ∈R 768 The thermodynamic (ice and snow texture) feature output by the ResNet50 branch is F. thermal ∈R 512 .

[0115] To facilitate subsequent integration, first, F... visual and F thermal Perform L2 normalization separately; where F thermal It also needs to be extended to F through a fully connected layer or other mapping layer. visual The same 768-dimensional space (which can be L2 normalized again after the fully connected layer).

[0116] Two learnable parameters λ were introduced during the fusion stage. v and λ t The initial values ​​are set to 0.6 and 0.4 respectively; both can be automatically optimized through backpropagation, thereby gradually converging to a more suitable ratio during the training process based on different samples.

[0117] Without triggering the weight rebalancing mechanism, the feature fusion in this embodiment can be expressed as:

[0118] F fusion =λ v ·Fvisual +λ t ·F thermal

[0119] Among them, F visual and F thermal After L2 normalization, both are mapped to a unified 768-dimensional feature space through a fully connected layer.

[0120] To determine the degree of similarity or complementarity between two features, this embodiment uses cosine similarity to measure the correlation between the two features. When the similarity is lower than the threshold θ = 0.65, a weight rebalancing mechanism is triggered.

[0121] When the cosine similarity is greater than or equal to 0.65, it is considered that the temperature-visual features and the ice and snow texture features have a high degree of similarity or complementarity in the current sample. In this case, the current λ value is directly adopted. v and λ t To merge:

[0122] If the cosine similarity is less than 0.65, it indicates that there are certain differences or conflicts between the two features in the current sample, and the weight rebalancing mechanism needs to be triggered for dynamic adjustment.

[0123] λ' v =λ v -0.1×(0.65-S cos )

[0124] λ' t =λ t +0.1×(0.65-S cos )

[0125] Among them, S cos The cosine similarity is calculated in real time; the preset adjustment coefficient is 0.1, which can be optimized during training.

[0126] The larger the difference between the cosine similarity and its set threshold, the more obvious the difference between temperature-visual features and ice and snow texture features, and the more it needs to favor ice and snow texture features in this training iteration.

[0127] Updated λ v and λ t The sum is approximately equal to 1 (which can be fine-tuned or normalized during implementation) to prevent excessive shifts in weight values.

[0128] After the aforementioned possible rebalancing operations, the system recalculates the fused output features. This vector is then compared with the true labels (such as ice thickness, snow and ice classification, etc.) in the subsequent loss function, generating a backpropagation gradient used to update λ. v and λ t And the trainable parameters of the preceding network.

[0129] Similar to the traditional neural network training process, this embodiment simultaneously updates the following through gradient backpropagation: the parameters of the CLIP model and the unfrozen layers in the ResNet50 branch that have been improved; and the learnable weights, fully connected mapping layers, and other subsequent network structures in the dynamic fusion module. By iterating on large-scale training data or a certain amount of bridge surface ice and snow scene data, the multimodal feature fusion module can learn the optimal weight allocation strategy under different scene or environmental conditions (such as temperature differences, light intensity, and snow thickness).

[0130] In this embodiment, based on cosine similarity, the model can adaptively adjust the weights of temperature-visual and snow / ice texture features, ensuring that when the two feature paths conflict, the more discriminative features are utilized more effectively. When the similarity is high, fusion is performed directly according to the current weights, reducing additional computational overhead and maintaining stability; when the differences are large, weight rebalancing better uncovers the potential value of snow / ice texture features. The learnable weights are not fixed constants, but rather automatically find the optimal fusion ratio suitable for the overall network performance during training. For snow and ice weather scenarios, temperature plays a crucial role in icing tendency, while texture features help distinguish between snow or ice cover patterns; this fusion mechanism can significantly improve the accuracy of identification or prediction in actual bridge surface icing warnings and de-icing decisions.

[0131] Through the above process, this embodiment successfully introduces cosine similarity and learnable weights in the multimodal feature fusion stage, and can adaptively adjust when the similarity is below a threshold, ultimately enabling the model to have flexible discrimination capabilities and robustness for various bridge surface ice and snow scenarios. This feature fusion mechanism can be closely integrated with the aforementioned temperature-visual feature generation and ice and snow texture feature generation processes, providing higher-dimensional and more reliable feature representations for subsequent time-series prediction and early warning decisions.

[0132] Specifically, step S3 further includes the following steps: storing the fused feature embedding representation in a vector database to construct a historical ice and snow event feature library; using a graph-based approximate nearest neighbor indexing algorithm to construct a topological structure for the feature library, thereby achieving fast nearest neighbor retrieval for the current feature embedding representation; and inputting the retrieved historical feature vector and the time series of the feature embedding representation into a temporal neural network model to provide historical context reference for ice thickness prediction.

[0133] Preferably, after multimodal features (such as temperature-visual features and snow texture features) are fused through a learnable dynamic weighting mechanism, a unified high-dimensional feature vector is obtained. The system stores these fused feature vectors, along with corresponding timestamps, geographic locations, or other optional identification information, in the FAISS vector database to form a historical snow event feature library. This feature library is continuously updated over time or as the system collects new bridge surface snow-related data, and its storage size gradually expands.

[0134] In order to quickly retrieve the few records most similar to the current bridge surface features from a massive amount of historical feature vectors, this embodiment uses the HNSW (Hierarchical Navigable Small World) approximate nearest neighbor indexing algorithm to construct the index topology.

[0135] When a new fused feature vector is inserted into the database, the FAISS engine calls the HNSW algorithm to add the vector to the index graph and maintains adjacency relationships based on cosine similarity or Euclidean distance between vectors (chosen according to actual needs). This graph-based indexing method can significantly reduce retrieval time while maintaining high retrieval accuracy, ensuring the availability of the system in real-time or near-real-time environments.

[0136] The current ice and snow features of the bridge surface (also obtained after multimodal fusion) are input into the FAISS vector database as a query vector. The database calls the HNSW indexing algorithm to quickly find neighboring nodes in the high-dimensional vector space and outputs the Top-K historical feature items most similar to the query vector (in this embodiment, K=5 is set, that is, the 5 records most similar to the current bridge surface features are returned).

[0137] After testing, the retrieval latency of this embodiment is stable at less than 50ms, which can complete the nearest neighbor retrieval within milliseconds, meeting the needs of most real-time scenarios.

[0138] The retrieved Top-5 historical feature vectors, along with the time series of the current bridge surface feature vectors, are input into the temporal neural network model. By leveraging the feature distribution of these historically similar scenarios, the temporal neural network model can better learn the evolution of ice thickness under similar weather conditions, similar bridge surface conditions, or similar friction coefficients. In this way, the model not only relies on the data at the current moment but also makes full use of information from historically similar scenarios, thereby further improving the accuracy and stability of ice thickness prediction.

[0139] Thanks to the efficient implementation of the HNSW approximate nearest neighbor algorithm combined with the FAISS database, massive historical features can be retrieved in a very short time, ensuring system response speed. Top-K similar features provide reference samples that are closer to the current bridge surface environment for time series prediction, making the neural network's learning of potential icing trends more targeted and helping to reduce prediction errors caused by environmental differences. The FAISS vector database and HNSW index in this embodiment are not limited to bridge surface ice and snow data. If it is necessary to introduce features of a larger scale or different modalities (such as radar or lidar data) in the future, they can also be incorporated in a similar way and retrieved quickly.

[0140] Specifically, in step S3, before making a prediction, the system collects the time series of the current bridge surface features (such as the fused feature embedding representation within the last 30 minutes) and merges the retrieved Top-K historical feature vectors (and their label information) into additional input as a priori reference for "how ice layer evolves under similar weather / bridge surface conditions".

[0141] There are various specific integration methods:

[0142] Directly concatenate the mean or weighted vector of historical features to the features at the current time step; add encoding of historical information to the initial hidden state or an intermediate layer of the time series model;

[0143] Alternatively, before using bidirectional LSTM, historical retrieval features can be mapped to the same dimension as the current time series before being mixed.

[0144] This embodiment uses a bidirectional LSTM network to process temporal feature sequences, with 64 hidden layer units and an input sequence length of 30 minutes (corresponding to 180 5Hz sampling points), combined with the supplementary information of the aforementioned historical feature vectors;

[0145] In the network output layer, the model makes multi-step predictions of future ice thickness (e.g., 15 minutes, 30 minutes) and calculates key indicators such as formation time or ice growth rate. After the LSTM hidden layer, a fully connected layer is set to map to the scalar output, representing the predicted future ice thickness (e.g., the value of the bridge surface ice thickness 15 minutes or longer in the future).

[0146] This embodiment chooses Huber loss instead of the traditional mean squared error (MSE) or mean absolute error (MAE) to balance sensitivity to large errors with smoothness for small errors, thereby improving the model's convergence stability. An additional regularization based on the ice thickness gradient is introduced, as shown below:

[0147]

[0148] Where h(t) is the predicted ice thickness at time t.

[0149] During training, in addition to using Huber loss and ice thickness gradient penalty terms, historical feature retrieval accuracy (such as similarity distribution) can also be included in the regularization objective as needed to further enhance the model's attention to high similarity scenes;

[0150] Backpropagation not only updates the parameters of the bidirectional LSTM network, but also fine-tunes historical retrieval strategies (such as thresholds and weights), thereby continuously improving the overall prediction accuracy.

[0151] Specifically, in step S4, an early warning is triggered once the predicted future ice thickness exceeds 2 mm (as set in this embodiment) or the friction coefficient is monitored to drop below 0.25 in real time.

[0152] By combining similar historical cases, the system can provide auxiliary information such as "the risk level that may evolve to" or "how long it may take to reach the safety threshold", providing more intuitive reference for maintenance personnel or vehicle drivers.

[0153] Specifically, in step S5, the fuzzy hierarchical analysis decision model constructs a fuzzy judgment matrix and uses the fuzzy eigenvalue method to calculate the comprehensive weight of each decision objective; and dynamically adjusts the weights of multiple decision objectives in the fuzzy hierarchical analysis decision model based on the predicted ice thickness.

[0154] like Figure 3 As shown, preferably, in this embodiment, after completing the snow and ice prediction (e.g., ice thickness prediction), in order to achieve a comprehensive balance among multiple decision-making objectives such as economy, safety, and environmental protection, a multi-objective decision-making model is constructed using the fuzzy hierarchical analysis method (FAHP). Its main process and key implementation are as follows:

[0155] 1. Constructing a 3D target matrix and a fuzzy judgment matrix

[0156] The system sets three main objectives for de-icing decision-making: economy (cost, resource consumption, etc.), safety (vehicle traffic safety, reduction of accident risk, etc.), and environmental protection (impact on the environment or bridge structure, emissions, etc.).

[0157] In real-world scenarios, more sub-goals or detailed indicators can be added based on application requirements, such as snow melting efficiency and time cost.

[0158] First, based on expert judgment, w = (0.45, 0.37, 0.18) is obtained, which correspond to the preliminary weight values ​​of economy, safety and environmental protection, respectively.

[0159] For each objective (such as economy vs. safety, economy vs. environmental friendliness, and safety vs. environmental friendliness), fuzzy comparison values ​​of relative importance are provided by experts or system experience bases.

[0160] The FAHP decision model constructs a three-dimensional objective matrix C = [c ij ] 3×3 , where c ij This represents the importance of objective i relative to objective j. Triangular fuzzy numbers are used to represent the elements of the judgment matrix. For example, when experts determine that the relative importance between "economic efficiency" and "safety" is moderately high, their triangular fuzzy numbers can be set to... The triangular fuzzy number for safety relative to environmental protection is set to (0.4, 0.6, 0.8); the triangular fuzzy number for economic efficiency relative to environmental protection is set to (0.7, 0.9, 1.0).

[0161] By synthesizing, defuzzifying, and normalizing the triangular fuzzy numbers, a set of relatively clear ratios is obtained for subsequent eigenvalue solving.

[0162] After converting the fuzzy judgment matrix into the corresponding fuzzy relative weight matrix, the eigenvalue method (generally the largest eigenvalue λ) is used. max The weight vector w = (w1, w2, w3) for each objective is obtained by using the corresponding eigenvectors of the objective and their respective features.

[0163]

[0164] To ensure that the sum of the weight vectors is 1, a suitable normalization process can be performed to ensure that ∑w i =1.

[0165] In the preceding steps, a temporal neural network or other prediction model has already provided a predicted value y for the ice thickness over a future time period (e.g., 15 minutes, 30 minutes). t .

[0166] When the predicted ice thickness exceeds a certain safety threshold (e.g., 2 mm), the system will increase its relative focus on the "safety" objective; if the predicted ice thickness is low, it can prioritize economy or environmental protection while ensuring basic safety.

[0167] Preferably, the system can set one or more thresholds and corresponding weighting strategies:

[0168] w 安全性 ←w 安全性 +Δ w ×(y t -threshold)

[0169] Where Δw This is an adjustable coefficient; threshold represents the safety threshold value. Similarly, the weight of economic or environmental objectives can be reduced accordingly to maintain overall harmony.

[0170] Thus, when the predicted ice thickness is detected to be at a dangerous level, the weight of "safety" will be appropriately amplified; conversely, "economic efficiency" or "environmental friendliness" may take on a larger proportion.

[0171] By normalizing the results of the above dynamic weighting process again, we can obtain the updated final weight vector w. i = (w1′, w2′, w3′), used to generate the next multi-objective decision evaluation. If other real-time information (such as sudden wind, sudden temperature drop, etc.) needs to be considered, the weights can also be fine-tuned in parallel during this process to achieve more refined dynamic decision-making.

[0172] For various predefined de-icing or snow melting schemes (such as mechanical de-icing, de-icing agent de-icing, microwave de-icing, etc.), normalized efficiency values ​​are calculated under the three objectives of economy, safety, and environmental protection.

[0173] In this embodiment, the system maintains a solution library, which contains several commonly used or feasible de-icing / snow melting techniques, such as:

[0174] Mechanical de-icing: using mechanical equipment to remove snow and ice;

[0175] De-icing agents (salt-based de-icing): Spraying or spreading chemical agents (salt, calcium chloride, etc.) to accelerate the melting of ice and snow;

[0176] Microwave de-icing (hot water spray): Melts or softens ice layers by heating with electromagnetic waves, in conjunction with mechanical assistance.

[0177] Based on the three main objectives of the system—economic efficiency, safety, and environmental friendliness—a set of evaluation indicators is set for each objective to quantify the performance of each solution. For example:

[0178] Economic indicators include:

[0179] Direct costs: De-icing costs per unit time or per unit area, including equipment rental fees, chemical costs, labor costs, etc.

[0180] Cost = BaseCost + k ice ×y t ,

[0181] Among them, y t For the predicted ice thickness (e.g., mm), BaseCos is the start-up or fixed cost, and k is the base value. iceThe variable cost factor corresponding to the icing thickness (which can be derived from historical operation records or equipment manuals).

[0182] Time cost: The total time (minutes or hours) from start to finish of the de-icing operation, which can be converted into indirect economic losses or operational impacts.

[0183] Time = Tsetup + r × y t ,

[0184] Where TsetupT is the preparation time (mechanical installation, personnel arrival, etc.), and r is the thickness of the ice layer that the equipment or work unit can melt within 1 hour (or minute) (mm / h or mm / min).

[0185] Safety indicators include:

[0186] Residual Ice: The maximum thickness (in millimeters) of ice or snow that may remain on the bridge surface after de-icing operations are completed;

[0187] If we assume that in cases of high ice thickness, a single operation may not be able to completely remove all the ice layers, then we can estimate based on an empirical model:

[0188] ResidualIce=max(0,y t -E(Method,t)),,

[0189] Where E(Method,t) represents the thickness of ice that can be removed after time t (or one job cycle) under the specified de-icing method Method. If y t If the value is large, but the de-icing efficiency is limited, then ResidualIce may be significantly greater than 0.

[0190] Traffic accident risk coefficient (Risk): Based on historical accident statistics or expert scores, it comprehensively measures the safety and reliability of different de-icing solutions during operation (the higher the value, the higher the risk).

[0191] Generally, the thicker the ice layer, the more slippery the road surface, and the higher the risk of vehicle travel. Related functions can be used to map the "predicted ice thickness" to a range of "accident risk coefficients," such as:

[0192] Risk(y t )=y t / (y t +c),

[0193] Where c is a constant (such as 1 mm or other reference value), when y t When >>c, Risk approaches 1 (high risk); conversely, it approaches 0 (low risk).

[0194] Environmental indicators:

[0195] Chemical Emission: For de-icing agent types, estimate their potential pollution or corrosion to the environment (rivers, land);

[0196] When using de-icing agents, the dosage is typically proportional to the total thickness and area of ​​the ice layer that needs to be melted. Example formula:

[0197] ChemicalEmi=α*A*y t ,

[0198] Where A represents the bridge deck working area, and α represents the chemical dosage and emission coefficient required per unit thickness of ice layer. The value of α will vary depending on the type of chemical used or the method of application. The higher the predicted ice thickness, the more chemical substances are usually required.

[0199] Energy consumption: such as the electricity demand of microwave de-icing, or the fuel consumption of mechanical equipment;

[0200] For example, microwave de-icing or mechanical equipment (fuel- or electric) operates for longer periods and at higher power levels when the thickness is greater, resulting in increased energy consumption. This can be described using linear or nonlinear functions.

[0201] Energy = P × T(y) t )

[0202] Where P is the equipment power or power factor, T(y) t The time taken is calculated based on the thickness of the ice layer.

[0203] Byproduct impact: such as the degree of corrosion of bridge metal structures by residues after de-icing agent spraying or the assessment of damage to surrounding vegetation.

[0204] The greater the ice thickness, the more chemical reagents or more frequent operations may be required, resulting in more byproducts such as "waste liquid" or "metal corrosion products." If historical data or experiments indicate that each millimeter of ice corresponds to a certain amount of corrosion or pollution residue, then the amount can be estimated relatively directly proportionally from the ice thickness.

[0205] ByproductImpact=β*y t

[0206] Where β is the amount of byproducts or environmental damage coefficient corresponding to "unit thickness of ice layer".

[0207] To facilitate comprehensive comparison, the original indicators mentioned above need to be standardized and normalized to obtain an effectiveness value (or score) in the range of [0,1]. The larger the value, the better the performance under the objective.

[0208] The normalization formula for economic objectives is as follows:

[0209]

[0210] Cost (k) Let Time be the unit de-icing cost of scheme k. (k) For the required time (after appropriate normalization or logarithm taking), w C and w T The weights of each sub-indicator under this objective.

[0211] The formula for normalizing the safety target is as follows:

[0212]

[0213] Where Norm(·) represents the normalization function that maps the original numerical value to [0,1], w res and w risk These refer to the level of concern regarding residual ice volume and risk factor, respectively. (k) Indicates the amount of residual ice, Risk (k) This represents the risk coefficient.

[0214] The formula for normalizing environmental protection targets is as follows:

[0215]

[0216] Among them, Chemical (k) Represents the chemical pollution emission intensity of scheme k, Energy (k) This refers to energy consumption per unit time or per unit area. (ByproductScore) (k) The impact of byproducts can be scored [0,1] by experts or historical data, w chem ,w energy ,w byproduct The weights are for each item.

[0217] The final comprehensive score S for each scheme k k It can be represented as:

[0218]

[0219] in, Let be the normalized performance value of the k-th scheme under the i-th objective.

[0220] After calculating the comprehensive score for all candidate snow melting schemes, they are sorted from highest to lowest. Based on years of practical experience and data accumulation, this embodiment selects the following three de-icing schemes and scores them in terms of economy, safety, and environmental protection (the score ranges from 0 to 1, with higher scores indicating better performance in that indicator):

[0221] Mechanical de-icing: Economy: 0.82; Safety: 0.75; Environmental friendliness: 0.63

[0222] Salt-based snow melting: Economic efficiency: 0.91; Safety: 0.68; Environmental friendliness: 0.42

[0223] Hot water spraying: Economy: 0.73; Safety: 0.85; Environmental friendliness: 0.79

[0224] A comprehensive score is calculated for each option using the weighted average method. The weights and scores of each option are substituted into the calculation:

[0225] Mechanical de-icing:

[0226] 0.45×0.82+0.37×0.75+0.18×0.63≈0.78

[0227] Salt-induced snow melting:

[0228] 0.45×0.91+0.37×0.68+0.18×0.42≈0.72

[0229] Hot water spray:

[0230] 0.45×0.73+0.37×0.85+0.18×0.79≈0.81

[0231] Based on the scoring results, the hot water spraying solution achieved a better balance between economy, safety and environmental protection, and had the highest overall score. Therefore, in this embodiment, the "hot water spraying" method should be given priority for bridge de-icing.

[0232] If the difference in the overall score is within a certain range (e.g., less than 0.05), the system can select a multi-scheme collaborative operation mode to accelerate de-icing by combining multiple methods; otherwise, the system will directly schedule the single scheme with the highest score or a few optimal schemes for rapid execution.

[0233] Once a decision is generated, the corresponding execution mechanism (such as mechanical de-icing equipment, automatic de-icing agent spreading device, microwave preheating device, etc.) can be triggered through the system's decision engine. During the de-icing operation, the friction coefficient and ice thickness changes are monitored in real time, and new observations are fed back to the system so that the decision process can be further corrected or deactivated, forming a closed-loop control.

[0234] The FAHP decision model of this system incorporates dynamic weight adjustment and can be applied to different bridge types such as steel bridges and concrete bridges, as well as various environmental conditions such as mountainous areas, water areas, and cities. According to statistics, within the range of -15℃ to 5℃, the system's ice layer prediction error can be controlled within ±1.2mm (confidence level 95%). After combining with FAHP dynamic decision-making, a snow melting plan can be generated within 8 seconds, which is more efficient than traditional methods.

[0235] In step S5 of this embodiment, the fuzzy hierarchical analysis decision model (FAHP) is closely integrated with the real-time predicted ice thickness. A basic weight vector is calculated using a fuzzy judgment matrix and the fuzzy eigenvalue method, and dynamic weight adjustments are made when a bridge surface icing risk is detected. Finally, the system comprehensively scores various snow melting schemes based on the updated multi-objective weights, selecting the highest priority or implementing a collaborative operation strategy. This achieves a balance among multiple indicators such as safety, economy, and environmental protection, significantly improving the intelligence and effectiveness of bridge surface ice and snow early warning and de-icing decision-making.

[0236] Example 2

[0237] This invention also provides a bridge surface ice and snow early warning system based on neural network and multimodal data fusion, used to implement the method described in the above technical solution, including:

[0238] A multimodal feature generation module is used to generate a multimodal feature representation based on the image of the bridge surface to be evaluated, meteorological parameters, and the bridge surface friction coefficient; the multimodal feature representation includes temperature-visual features and ice and snow texture features;

[0239] The multimodal feature fusion module is used to fuse the multimodal feature representations through a learnable dynamic weighting mechanism to obtain a unified feature embedding representation;

[0240] An ice thickness prediction module is used to input the time series of the embedded feature representation into a temporal neural network model to predict the future ice thickness of the bridge surface;

[0241] The early warning generation module is used to generate early warning information based on the predicted ice thickness.

[0242] The decision generation module is used to calculate the normalized efficiency value of each decision objective of multiple candidate de-icing schemes based on the predicted ice thickness, and to calculate the evaluation value of each candidate de-icing scheme based on the corresponding weight of the decision objective, so as to select the optimal de-icing scheme; the multiple decision objectives include at least safety, economy and environmental protection.

[0243] To test the system's performance during the initial deployment phase (when there is limited historical data), this embodiment selected three representative bridges in northern China (two steel bridges and one concrete bridge), continuously monitored them for 20 days during winter, and collected the following multimodal data:

[0244] Visual data: Deploy high dynamic range (HDR) cameras (4096×2160 resolution, 30fps) to simultaneously acquire visible light (400-700nm) and short-wave infrared (900-1700nm) images; acquire approximately 800 images per day and store them on an edge server for further processing.

[0245] Temperature data: Distributed fiber optic temperature sensors are used with an accuracy of ±0.05℃, a spatial resolution of 0.1m, and a sampling frequency of 10Hz; the temperature gradient of the bridge deck is monitored in real time, and the sensors are evenly distributed along the bridge deck to obtain temperature changes in different areas.

[0246] Meteorological data: Wind speed (accuracy ±0.1m / s), relative humidity (±1%RH), and precipitation intensity (±0.05mm / h) are collected from local meteorological stations and transmitted to edge computing nodes via LoRa wireless method;

[0247] After being aligned with temperature data and visual data via timestamps, they are integrated to form a multimodal input source.

[0248] In this embodiment, the continuous monitoring period is 20 days, with data collected at different times each day (including nighttime and extreme weather conditions) to provide diverse scenarios for subsequent model training and testing.

[0249] Two certified engineers used a majority vote to annotate the collected images and corresponding sensor data, focusing primarily on ice and snow type and thickness.

[0250] Define three types of ice and snow forms:

[0251] Frost ice: thickness ≤1mm, surface is granular;

[0252] Compacted snow: 3-5mm thickness, density >0.3g / cm³ 3 ;

[0253] Mixed state: The ice layer contains salt particles or anti-slip materials and is distributed heterogeneously.

[0254] The following data augmentation methods were used:

[0255] Cross-modal enhancement: Applying a random temperature offset (±1.5℃) to the infrared image to simulate potential measurement biases of the sensor under extreme conditions;

[0256] Spatiotemporal alignment: Optical flow algorithm is used to compensate for the spatial misalignment of the camera and temperature sensor in the installation position and to synchronize the time (accuracy ≤5ms) to ensure that different modal data correspond to the same target area and time.

[0257] The basic training set in this embodiment includes 300 sets of data (120 sets of frost and ice, 100 sets of compacted snow, and 80 sets of mixed states); the extreme test set includes 50 sets of data (including 10 sets of ice formation under inversion layers and 20 sets of snow and ice morphology variations after traffic compaction).

[0258] Through the above steps, a set of labeled data, which is relatively small in scale but rich in scenarios, is obtained in the initial deployment phase, providing a foundation for testing the cold start performance of the system.

[0259] In an edge computing environment, an extreme test set (50 sets of data in total) is injected into the system frame by frame to simulate the real operation process; two typical scenarios are designed for key verification:

[0260] Scenario A: Ice formation due to temperature inversion on steel bridge surface: The ambient temperature gradually rises from -5℃ to 0℃, and an ice formation due to temperature inversion occurs locally on the bridge surface; observe the detection accuracy and response speed of the system under rapid temperature gradient changes.

[0261] Scenario B: Changes in the morphology of compacted snow after peak traffic hours: Due to the high-frequency rolling of vehicles, the surface friction coefficient changes abruptly, and the snow layer gradually compacts and mixes with mud and sand; the focus is on examining the system's ability to identify complex snow surface textures and abnormal friction coefficients.

[0262] The following evaluation metrics are used in this embodiment:

[0263] First frame recognition accuracy: The accuracy of the system in recognizing signs of ice and snow for the first time (the correct classification rate of frost, compacted snow, and mixed conditions);

[0264] Steady-state error: Root mean square error (in mm) of ice thickness prediction during the continuous monitoring phase;

[0265] False alarm decay rate: The percentage of false alarms that decrease after 24 hours of self-correction due to model updates or threshold adjustments.

[0266] Cross-material consistency: The degree of difference in test results for the same type of ice and snow conditions between steel bridges and concrete bridges (the smaller the difference, the better the consistency).

[0267] Anti-interference capability: The suppression rate of interference factors such as vehicle headlight reflection and bridge surface cracks, that is, the difference in recognition accuracy under conditions of interference.

[0268] Then perform cold start performance optimization:

[0269] The subsystem is equipped with a dual-buffered storage area, including: storing high-confidence positive samples (typical ice and snow features) and storing difficult negative samples (easily confused features, such as cracks, oil stains, etc.).

[0270] The update rules introduce a dynamic threshold. When the feature similarity is higher or lower than a certain standard, the sample is automatically included in the corresponding buffer so that it can be focused on during subsequent incremental learning.

[0271] The model is updated weekly, with only the parameters of the last layer of the network unlocked for fine-tuning (the remaining layers are frozen) to avoid catastrophic amnesia.

[0272] Under small sample conditions, the model can gradually adapt to extreme or novel scenarios by learning from new samples in the buffer.

[0273] The specific results obtained by comparing our system with a baseline system (traditional fusion strategy) on extreme test sets and basic training sets are shown in Table 1:

[0274] First frame recognition accuracy 83.6％ 67.2％ +16.4％ Steady-state error (mm) ±0.9 ±1.8 —50％ False alarm decay rate (24h) 72.3％ 45.1％ +27.2％ Cross-material consistency 91.5％ 78.4％ +13.1％ Anti-interference capability 88.2％ 63.7％ +24.5％

[0275] Table 1 Performance Evaluation of This Embodiment and Existing Technologies

[0276] Table 1 shows that our system achieved an accuracy of 83.6%, significantly higher than the baseline system's 67.2%, indicating a faster identification of ice and snow types in the initial detection stage. The root mean square error of ice thickness prediction decreased from ±1.8 mm to ±0.9 mm, a reduction of half, demonstrating the high precision of the fusion algorithm. After 24 hours of self-correction, our system reduced false alarms by 72.3%, compared to only 45.1% for the baseline system, indicating a stronger ability to correct its own predictions. The detection difference between the two materials (steel bridge vs. concrete bridge) decreased by 13.1%, indicating enhanced adaptability of the algorithm to differences in bridge deck materials. Interference such as headlight reflections and cracks were better suppressed, with our system showing a 24.5% improvement over the baseline system.

[0277] With only 300 sets of training data, the system achieved a first-frame recognition accuracy of 83.6%, demonstrating strong generalization ability in the data-scarce stage. The GWU (Gated Weighted Update) mechanism allows the feature fusion weights to be automatically adjusted according to the scene (frost, compacted snow, etc.), improving the accuracy by 12.8% compared with the fixed weight strategy. After 4 weeks of incremental optimization (fine-tuning the last layer of the network every week), the extreme case recognition rate increased from the initial 71.5% to 89.3%, significantly enhancing the adaptive ability.

[0278] To verify the system's effectiveness in a real-world scenario, a steel box girder section of a cross-sea bridge in northern China was selected, and icing in the inversion layer was monitored at night. The specific process is as follows:

[0279] On-site conditions: Ambient temperature approximately -3℃, wind speed 6m / s, icing was detected on the bridge surface due to temperature inversion, with an initial ice thickness of 1.2mm;

[0280] System response: The first frame recognition took 1.8 seconds, which means that ice formation was detected and a level 2 warning was triggered;

[0281] Thickness prediction: If the ice layer is predicted to thicken to 2.5 mm in 30 minutes using an LSTM model, the system will upgrade the warning to Level 1.

[0282] The priority of the FAHP model output schemes is as follows:

[0283] Option B (liquid de-icing agent): Overall score 0.81

[0284] Option C (Microwave De-icing): Overall score 0.76

[0285] Option A (Mechanical De-icing): Overall score 0.69

[0286] After selecting Option B (liquid de-icing agent), the ice layer completely melted within 20 minutes, and the bridge surface friction coefficient returned to 0.48 (higher than the safety threshold of 0.35). Compared with the traditional spraying amount, the amount of de-icing agent used was reduced by 24%, resulting in better economic and environmental benefits.

[0287] This field verification demonstrates that the system possesses industrial deployment capabilities even during the cold start phase, and can quickly approach optimal performance through mechanisms such as incremental learning and dynamic weighting. For potentially more complex weather conditions in the future, the system can also continuously improve model accuracy and decision-making efficiency through self-learning.

[0288] In summary, the verification results of this embodiment show that, under the condition of limited initial data scale, the proposed bridge surface ice and snow early warning system can still effectively complete ice and snow identification and thickness prediction. Moreover, through elastic memory pool and progressive fine-tuning, the identification accuracy of extreme cases is continuously improved, providing reliable technical support for road safety in northern winters.

[0289] Example 3

[0290] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the bridge surface ice and snow early warning method based on neural network and multimodal data fusion described in the above technical solution.

[0291] Example 4

[0292] The present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the bridge surface ice and snow early warning method based on neural network and multimodal data fusion described above.

[0293] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0294] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0295] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0296] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0297] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

[0298] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A bridge deck ice and snow early warning method based on a neural network and multi-modal data fusion, characterized in that: Includes the following steps: A multimodal feature representation is formed based on the image of the bridge surface to be evaluated, meteorological parameters, and the bridge surface friction coefficient; the multimodal feature representation includes temperature-visual features and ice and snow texture features; By fusing the multimodal feature representations through a learnable dynamic weighting mechanism, a unified feature embedding representation is obtained; The time series representation of the embedded features is input into a temporal neural network model to predict the future ice thickness on the bridge surface; Early warning information is generated based on the predicted ice thickness; The generation process of the temperature-visual feature includes: The bridge deck multispectral image and the corresponding temperature field matrix are stitched together by channel to form a multichannel input, which includes at least a visible light image data channel and a temperature data channel. After normalization and random masking enhancement of the multi-channel image, it is then input into a pre-trained visual encoder based on the Transformer structure. The parameters of the front layer of the visual encoder are partially frozen to maintain the pre-trained visual representation, and the temperature field attention mechanism is introduced in the back layer to give higher weight to temperature-sensitive areas. The high-dimensional feature vector that fuses temperature and visual information is output after processing by the visual encoder and is used for subsequent multimodal feature fusion processing. The generation process of the ice and snow texture features includes: A backbone model based on convolutional neural networks is adopted, with some convolutional layers or residual blocks frozen, to fuse bridge deck image data, meteorological data, and bridge deck friction coefficient data as input. After the initial feature extraction, a spatial pyramid pooling layer is connected after the last residual block of the network. The spatial pyramid pooling layer uses multiple pooling windows of different sizes to pool the output features and concatenates the pooling results of each scale into a high-dimensional vector. After batch normalization and nonlinear activation function processing, multi-dimensional ice and snow texture features are output. The process of fusing the multimodal feature representations through a learnable dynamic weighting mechanism includes: First, calculate the cosine similarity between temperature-visual features and ice and snow texture features; When the similarity reaches or exceeds the preset threshold, the original weights are directly used to perform weighted fusion of the two features to obtain the fused features. When the similarity is lower than the preset threshold, the weight of temperature-visual features is reduced according to the difference between the cosine similarity and the threshold, based on the preset adjustment coefficient, while the weight of ice and snow texture features is increased accordingly. Finally, the adjusted weights are used to perform weighted fusion of the two features to obtain the final fused features.

2. The method of claim 1, wherein: It also includes the following steps: Based on the predicted ice thickness, the normalized efficiency values ​​of each decision objective of multiple candidate de-icing schemes are calculated, and the evaluation values ​​of each candidate de-icing scheme are calculated based on the corresponding weights of the decision objectives, so as to select the optimal de-icing scheme; the decision objectives include at least safety, economy and environmental protection.

3. The method according to claim 1, characterized in that: It also includes the following steps: The fused feature embeddings are stored in a vector database to construct a historical ice and snow event feature library. A graph-based approximate nearest neighbor indexing algorithm is used to construct a topological structure for the feature library, thereby enabling fast nearest neighbor retrieval of the current feature embedding. The retrieved historical feature vectors and the time series of the feature embeddings are input into a temporal neural network model to provide historical context reference for ice thickness prediction.

4. The method according to claim 1, characterized in that: When the predicted ice thickness or bridge surface friction coefficient is lower than the corresponding set threshold, an early warning message is output.

5. The method according to claim 1, characterized in that, The input data preprocessing steps of the backbone model include: Bridge deck image data, meteorological data, and bridge deck friction coefficient data are stored in a database or file system in a unified format. After reading the data from the database or file system, the bridge deck image data is subjected to data enhancement processing with random rotation and color jitter. At the same time, the read meteorological data and bridge deck friction coefficient data are matched and jointly preprocessed with the processed image data in tabular form, so that when inputting into the backbone model, all modal data are unified into a standardized format.

6. The method according to claim 2, characterized in that: A multi-objective decision-making model is constructed using fuzzy hierarchical analysis. The multi-objective decision-making model calculates the comprehensive weight of each decision objective by constructing a fuzzy judgment matrix and using the fuzzy eigenvalue method.

7. A bridge surface ice and snow early warning system based on neural network and multimodal data fusion, characterized in that: To implement the method according to any one of claims 1-6, comprising: A multimodal feature generation module is used to generate a multimodal feature representation based on the image of the bridge surface to be evaluated, meteorological parameters, and the bridge surface friction coefficient; the multimodal feature representation includes temperature-visual features and ice and snow texture features; The multimodal feature fusion module is used to fuse the multimodal feature representations through a learnable dynamic weighting mechanism to obtain a unified feature embedding representation; An ice thickness prediction module is used to input the time series of the embedded feature representation into a temporal neural network model to predict the future ice thickness of the bridge surface; The early warning generation module is used to generate early warning information based on the predicted ice thickness. The decision generation module is used to calculate the normalized efficiency value of each decision objective of multiple candidate de-icing schemes based on the predicted ice thickness, and to calculate the evaluation value of each candidate de-icing scheme based on the corresponding weight of the decision objective, so as to select the optimal de-icing scheme; the decision objective includes at least safety, economy and environmental protection.

Images