Vehicle load identification method based on multi-modal large model

Abstract

The invention discloses a vehicle load identification method based on a multi-modal large model, and the method comprises the following steps: selecting a reference bridge in a region for testing, and obtaining a reference bridge case library; analyzing the new bridge through a multi-modal large model, and matching a plurality of reference bridges most similar to the core characteristics of the new bridge from a reference bridge case library; the multi-modal large model migrates a load response model and model parameters of the reference bridge case to a new bridge through a case reasoning mechanism; and according to the structural difference between the new bridge and the matched reference bridge and the actual working condition of the new bridge, the multi-modal large model carries out architecture reconstruction on the migrated load response model. The method has the beneficial effects that the new bridge can directly reuse the model parameters and the load response rule of the reference bridge through the constructed reference bridge case library in combination with the retrieval enhancement generation and bridge-crossing knowledge migration mechanism of the multi-modal large model, so that the time cost and the economic cost of bridge-crossing deployment can be greatly reduced.

Description

Technical Field

[0001] This application relates to the field of intelligent traffic monitoring technology, and in particular to a vehicle load identification method based on a multimodal large model. Background Technology

[0002] Currently, vehicle load identification for regional bridge clusters mainly relies on two methods: traditional physical sensor systems and deep learning-based intelligent recognition models. Existing technologies employ the following methods to address these two types of approaches: Static weighing method: This method involves weighing vehicles at low speeds or while stationary by installing weighbridges, strain gauges, piezoelectric sensors, or other weighing devices on the bridge deck. While this method offers high accuracy, it requires traffic interruption, has high construction and maintenance costs, and is difficult to implement for continuous, dynamic monitoring of bridges.

[0003] Dynamic weighing systems: Sensor arrays are installed on the bridge deck or under the road surface to collect axle load and total weight data in real time during vehicle movement. Such systems are widely used on highways, but they rely on fixed sensor networks, which present problems such as equipment vulnerability, poor environmental adaptability, and complex calibration; and they are difficult to implement on old bridges or in scenarios where there are no pre-embedded sensors.

[0004] A deep learning-based vehicle load identification method identifies vehicle types on bridge decks using high-definition cameras and object detection algorithms such as YOLO, and then infers vehicle mass from bridge deflection response data. While possessing a certain level of intelligence, this method places extremely high demands on data quality and model generalization ability. Current methods require separate data collection and model training for each bridge, making rapid deployment across large-scale bridge groups difficult, resulting in high training costs, low efficiency, and poor model transferability. Summary of the Invention

[0005] One objective of this application is to provide a vehicle load identification method based on a multimodal large model that can solve at least one of the defects in the aforementioned background art.

[0006] To achieve at least one of the above objectives, the technical solution adopted in this application is: a vehicle load identification method based on a multimodal large model, comprising the following steps: S100: Select benchmark bridges with different years of construction, structural types and stiffness levels in the region, conduct tests under multiple working conditions, and obtain a benchmark bridge case library. S200: Analyze the core feature parameters of new bridges deployed in the region using a multimodal large model, and based on the analysis results, use an enhanced generation mechanism to match multiple benchmark bridges from the benchmark bridge case library that are most similar to the core features of the new bridge. S300: Based on the matching benchmark bridge case, the multimodal large model transfers the load response model and model parameters of the benchmark bridge case to the new bridge through the case reasoning mechanism; S400: Based on the structural differences between the new bridge and the matching benchmark bridge, as well as the actual working conditions of the new bridge, the multimodal large model reconstructs the architecture of the migrated load response model to obtain the load response model architecture with the highest matching degree to the current working conditions and performs load identification.

[0007] Preferably, step S200 includes the following processes: analyzing the core characteristic parameters of the new bridge based on a multimodal large model, vectorizing the obtained structured data and unstructured data based on real-time traffic flow to obtain the feature vector of the new bridge; extracting multimodal features from the benchmark bridge case library through the multimodal large model, vectorizing the obtained structured and unstructured data to obtain the feature vector of the benchmark bridge corresponding to different benchmark bridges; calculating the cosine similarity between the feature vector of the new bridge and the feature vector of each benchmark bridge, and matching the new bridge with multiple benchmark bridges that meet the cosine similarity requirement.

[0008] Preferably, for the structured data of the new bridge and the benchmark bridge, a difference enhancement strategy is adopted when constructing the feature vector, which specifically includes the following process: based on the structural feature type of the bridge, the structural features of the new bridge and the benchmark bridge are clustered to obtain structural clusters corresponding to different structural feature types; principal component analysis is performed on each structural cluster, and the first two principal components are taken as the structural commonality vector to obtain the structural deviation vector; the obtained structural deviation vector is amplified based on the channel attention mechanism to obtain the structural deviation enhancement vector; the structural commonality vector and the structural deviation enhancement vector are re-concatenated to obtain the structural feature vector of the bridge after difference enhancement.

[0009] Preferably, the structural feature vector of the bridge after differential enhancement The expression is as follows: ; ; ; in, Represents structural commonality vectors. This represents the structural difference enhancement vector, where μ represents the difference amplification factor. Represents the gain vector. Represents the structural deviation vector. σ(·) represents element-wise multiplication, W1 represents the weighting matrix, W2 represents the weighting matrix, and ReLU(·) represents the linear rectified activation function.

[0010] Preferably, step S300 includes the following process: using the load response model that appears most frequently among the matched multiple benchmark bridges as the initial template for new bridge load identification; using the case data corresponding to the matched multiple benchmark bridges as anchor points, and combining the real-time operating characteristics of the new bridge, the multimodal large model generates a structured reasoning basis; the initial template is corrected according to the obtained structured reasoning basis, and the corrected new bridge data is marked and added to the benchmark bridge case library.

[0011] Preferably, during step S400, the multimodal large model, based on the real-time traffic flow conditions of the new bridge and the matching hyperparameters of the benchmark bridge, and combined with the performance feedback data of different architecture models in the benchmark bridge case library under similar scenarios, restructures the neural network architecture of the structural response model of the new bridge. This includes the following processes: defining a candidate pool of suitable neural network architectures based on the data in the benchmark bridge case library; designing a grid structure based on grid depth, number of neurons, and feature fusion method for the structural features and traffic flow conditions of the new bridge; and verifying and selecting the optimal neural network architecture with the highest matching pair with the current scenario through simulation training.

[0012] Preferably, the process of defining the candidate pool for neural network architectures includes the following steps: based on the obtained structured reasoning, clarify the core constraints of the neural network architecture design; combine the traffic flow characteristics of the new bridge with the model experience data of the matched benchmark bridge, define the candidate pool for neural network architectures in a targeted manner, and exclude neural network architectures with poor performance feedback in the same scenario in the benchmark bridge case library.

[0013] Preferably, when designing the neural network architecture, it is necessary to adaptively map the feature dimensions of the new bridge to the input layer. Specifically, this includes the following process: preprocessing the input features of the new bridge; if the stiffness of the new bridge and the reference bridge is less than a set value, linearly correcting and adjusting the deflection response data of the new bridge; to address the interference of environmental factors, calling the "environment-deflection correction curve" of the matching reference bridge from the reference bridge case library to correct the response data of the new bridge; and combining the corrected data features and the input layer design experience of similar data from the reference bridge, using the optimal sequence length of the time series data of the reference bridge as the input layer parameter.

[0014] Preferably, when designing the neural network architecture, the generation of the mesh structure includes the following process: based on the complexity of the mechanical properties of the new bridge, the complexity of the working conditions, and the structural differences between the new bridge and the benchmark bridge, multiple sets of candidate mesh structures are dynamically generated; if inference indicates that the complexity of the new bridge is higher than that of the benchmark bridge, the mesh depth is increased by 1 to 2 layers based on the optimal depth of the benchmark bridge in the same scenario; for scenarios with a mesh depth greater than 5 layers, residual connections are automatically added; the number of neurons in the input layer is strongly bound to the corrected feature dimension, and the number of neurons in the hidden layer is adaptively reduced layer by layer.

[0015] Preferably, the process of fine-tuning the load response model of the new bridge is as follows: establish the spatiotemporal correlation of vehicle travel between the reference bridge and the new bridge through license plate recognition technology; if there are vehicles that continuously drive across the reference bridge and the new bridge, automatically associate the load response data of the vehicles on the reference bridge with the new bridge dataset for training the load response model.

[0016] Compared with the prior art, the beneficial effects of this application are as follows: (1) The technical solution of this application constructs a benchmark bridge case library covering multiple types of bridges and multiple working conditions. Combined with the retrieval enhancement generation of multimodal large models and the cross-bridge knowledge transfer mechanism, the new bridge can directly reuse the model parameters and load response laws of the benchmark bridge without repeating the experiment. This can greatly reduce the time and economic costs of cross-bridge deployment and realize the rapid promotion and application of dozens to hundreds of bridges in the region.

[0017] (2) The technical solution of this application relies on the multimodal large model to accurately analyze the working conditions such as single vehicle, multi-vehicle parallel driving, and following vehicle. Combined with the adaptation mechanism of the special model for each working condition, it solves the problem of identification confusion in the multi-vehicle coupling scenario of traditional methods and meets the high precision requirements of bridge safety monitoring for load data.

[0018] (3) The technical solution of this application utilizes the automatic structural search capability driven by multimodal large model, which can dynamically adjust the model architecture according to the structural characteristics and traffic flow characteristics of different bridges, replacing the traditional manual parameter adjustment mode that relies on expert experience.

[0019] (4) The technical solution of this application, through the bridge data supplementation mechanism associated with license plate recognition and the dynamic update capability of the case library, enables the system to autonomously accumulate real operation data without the need for periodic shutdown for calibration or full retraining of the model. In long-term operation and maintenance, the labor cost of data collection and model updates is significantly reduced, improving the sustainability and economy of the system in the full life cycle monitoring of bridges. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall working steps of this application. Detailed Implementation

[0021] The present application will now be further described in conjunction with specific embodiments. It should be noted that, in the description of this specification, the use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicates that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0022] In the description of this application, it should be noted that the terms "center", "lateral", "longitudinal", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., which indicate the orientation and positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and should not be construed as limiting the specific protection scope of this application.

[0023] It should be noted that the terms "first," "second," etc., in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0024] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0025] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0026] The terms “comprising” and “having”, and any variations thereof, in the specification and claims of this application are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0027] One preferred embodiment of this application, such as Figure 1 As shown, a vehicle load identification method based on a multimodal large model includes the following steps: S100: Select benchmark bridges with different years of construction, structural types and stiffness levels within the region, conduct tests under multiple working conditions, and obtain a benchmark bridge case library.

[0028] It is understandable that before the load response model is deployed on the new bridge in the area, there are several historical bridges in the area that have already been deployed. For some of the historical bridges in the same area, their structural design and working conditions are similar to those of the new bridge. Therefore, it is possible to conduct experiments on the historical bridges in the area where load response models have already been deployed, and use the data obtained from the experiments as the data basis for the new bridge.

[0029] S200: Analyze the core feature parameters of new bridges deployed in the region using a multimodal large model, and based on the analysis results, use an enhanced generation mechanism to match multiple benchmark bridges from the benchmark bridge case library that are most similar to the core features of the new bridge.

[0030] It is understandable that the benchmark bridges selected within the region cover all bridge structure types, meaning that there are benchmark bridges whose structure is inconsistent with that of the new bridge. Due to the inconsistency in bridge structure, the theoretically corresponding load response models have poor universality. Therefore, by comparing the characteristics of the new bridge and the benchmark bridge, benchmark bridges with more similar characteristics to the new bridge are selected as the basis for subsequent knowledge transfer.

[0031] S300: Based on the matching benchmark bridge case, the multimodal large model transfers the load response model and model parameters of the benchmark bridge case to the new bridge through the case reasoning mechanism.

[0032] S400: Based on the structural differences between the new bridge and the matching benchmark bridge, as well as the actual working conditions of the new bridge, the multimodal large model reconstructs the architecture of the migrated load response model to obtain the load response model architecture with the highest matching degree to the current working conditions and performs load identification.

[0033] Understandably, since there are still some structural differences between the new bridge and the reference bridge, the actual working conditions caused by these structural differences may also differ. Therefore, the load response model of the migrated reference bridge needs to be adaptively adjusted and reconstructed based on the actual situation of the new bridge to ensure that the obtained load response model can match the actual working conditions of the new bridge well, thereby ensuring the accuracy of vehicle load identification.

[0034] Compared to traditional methods that require separate data collection and model training for each new bridge in a regional bridge network, the technical solution in this application constructs a benchmark bridge case library covering multiple bridge types and working conditions. Combined with multimodal large-scale model retrieval enhancement generation and a bridge knowledge transfer mechanism, new bridges can directly reuse the model parameters and load response patterns of benchmark bridges without repeated experiments. This significantly reduces the time and economic costs of bridge deployment, enabling rapid deployment and application across dozens to hundreds of bridges within a region.

[0035] In this embodiment, regarding the establishment of the benchmark bridge case library in step S100, representative bridges can be selected from the regional bridge group as benchmark bridges. First, representative bridges covering different years of construction (newly built bridges, bridges with 10 years of service, bridges with more than 20 years of service), structural types (simply supported beam bridges, continuous beam bridges, arch bridges, etc.) and stiffness levels are selected as benchmark bridges. Then, a multi-condition load test scheme is designed for each benchmark bridge: systematic tests are carried out using a calibration weight vehicle (equipped with standard load modules of known weight), covering different load levels (10t, 20t, 30t, etc.), driving speeds (20km / h, 40km / h, 60km / h, etc.), and traffic conditions (single-vehicle driving, two-vehicle driving side by side, multiple-vehicle following (vehicle distance 5m / 10m)) to ensure coverage of typical scenarios in actual traffic.

[0036] For each benchmark bridge, multi-dimensional data, vehicle load and operating condition data, load response data, and visual auxiliary data were collected simultaneously during the testing process. Multi-dimensional data included bridge name, geographical location, year of construction, structural type, span combination, main beam material, design load level, measured stiffness parameters (such as mid-span bending stiffness), and bridge deck pavement type, obtained and quantified through drawing analysis and on-site testing. Vehicle load and operating condition data included the actual total weight of the weighted vehicle, its lane position, and the number of vehicles in multi-vehicle conditions, recorded synchronously using vehicle-mounted positioning equipment and bridge-deck cameras. Load response data was collected using dynamic deflection time-series response data under vehicle loads, obtained from targets placed at key sections of the benchmark bridge using digital image correlation (DIC) algorithms, optical flow methods, or millimeter-wave radar and other dynamic deflection monitoring technologies. Environmental interference parameters (such as temperature) were recorded simultaneously for later data correction. Visual auxiliary data included vehicle driving images / videos collected by high-definition cameras on the bridge deck, used to extract vehicle characteristics, driving status, and visual labels for traffic flow conditions.

[0037] It is understandable that the core data of the benchmark bridge case library obtained through the above process includes three key elements: first, bridge metadata, covering basic attributes such as construction year, structural type, and common traffic flow conditions on the bridge deck; second, load-response correlation data, i.e., structured mapping relationships collected under different test conditions (such as the correlation between the total weight of the weighted vehicle and the corresponding peak dynamic deflection at mid-span, and the correlation between the load application location and deflection distribution characteristics); and third, model experience data, including the validated and adapted load-response model structure for the benchmark bridge (the most suitable network architecture for the bridge, such as BiLSTM) and historical optimization results of model parameters (such as the optimal hyperparameter combination and the weight matrix for training convergence). For the above multi-source heterogeneous data, numerical structural parameters can be preprocessed, and features (such as structural type and load condition type) can be embedded and vectorized. By vectorizing and encoding the cases and storing them in a vector database that supports semantic retrieval, a benchmark bridge case library covering multiple bridge types and multiple load conditions can be constructed, providing core data support for subsequent multimodal large-scale model retrieval, enhanced reasoning, and cross-bridge knowledge transfer.

[0038] In this embodiment, step S200 includes the following process: S210: Based on the analysis of the core characteristic parameters of the new bridge using a multimodal large model, the obtained structured data and unstructured data based on real-time traffic flow are vectorized to obtain the feature vector of the new bridge.

[0039] Understandably, the structured data for the new bridge includes its construction year, structural type, span components, design load level, and measured stiffness, while the unstructured data includes real-time traffic flow visual data.

[0040] S220: Multimodal feature extraction is performed on the benchmark bridge case library using a multimodal large model. The obtained structured and unstructured data are vectorized to obtain the benchmark bridge feature vectors corresponding to different benchmark bridges.

[0041] Understandably, the structured data in the benchmark bridge case library includes construction year, structural type, span components, design load level, and measured stiffness, which are converted into numerical feature vectors through standardized coding. The unstructured data includes dynamic deflection time-series response curves and traffic flow video frame features, which can be converted into dimensional semantic vectors through a multimodal large model. After obtaining the feature vectors of the structured data and the feature vectors of the unstructured data, the multimodal large model can map all the feature vectors to the same semantic space to obtain the benchmark bridge feature vectors corresponding to different benchmark bridges. All the benchmark bridge feature vectors can form a benchmark bridge case vector library and be stored in a vector database.

[0042] S230: Calculate the cosine similarity between the feature vector of the new bridge and the feature vectors of each reference bridge, and match the new bridge with multiple reference bridges that meet the cosine similarity requirement.

[0043] Understandably, a similarity threshold can be set to determine the similarity between the baseline bridge and the new bridge. If the cosine similarity calculated between the feature vectors of the new bridge and the baseline bridge is greater than or equal to the set similarity threshold, then the cosine similarity requirement is met, and the baseline bridge can be matched with the new bridge. The specific value of the similarity threshold can be set according to the actual needs of those skilled in the art; for example, the similarity threshold can be 0.8 to 0.9. Generally, there may be multiple baseline bridges that meet the similarity threshold. These baseline bridges can be arranged in descending order of their cosine similarity values, and then the top few baseline bridges can be matched with the new bridge. For example, the top 3 to top 5 baseline bridges can be selected for matching. The specific calculation process of cosine similarity is well known to those skilled in the art and will not be described in detail here.

[0044] It is important to understand that, based on the above, both the reference bridge feature vector and the new bridge feature vector include structural and non-structural feature terms. For non-structural feature terms, they are mainly influenced by structural features and external environmental features. Since external environmental features can be considered consistent within the region, the similarity between the reference bridge and the new bridge primarily lies in their structural features. However, the differences in structural parameters such as bridge stiffness and span components are often between 5% and 15%. When high-dimensional vectors are overwhelmed by a large amount of redundant information, cosine similarity may relatively reduce the structural differences, leading to errors in the matching results between the reference bridge and the new bridge, resulting in lower accuracy of the subsequent transferred load response model. Therefore, before calculating the cosine similarity, this embodiment needs to enhance the differences in the structured data of the new bridge and the reference bridge, amplifying the mechanical deviations that determine the load response relationship between the new bridge and the reference bridge.

[0045] Specifically, for the structured data of the new bridge and the benchmark bridge, a difference enhancement strategy is adopted when constructing the feature vectors, which includes the following process: S201: Based on the structural feature types of bridges, cluster the structural features of new bridges and benchmark bridges to obtain structural clusters corresponding to different structural feature types.

[0046] As can be understood from the foregoing, the structural characteristics of bridges include their year of construction, structural type, span components, design load level, and measured stiffness. These structural characteristic data differ significantly from each other; therefore, clustering can be used to categorize data of the same type, resulting in corresponding structural clusters. These structural clusters are preserved in vector form.

[0047] S202: Perform principal component analysis on each structural cluster, and obtain the structural deviation vector by taking the first two principal components as the structural commonality vector.

[0048] It is understood that the specific process of principal component analysis is well known to those skilled in the art, and therefore will not be described in detail here. Based on principal component analysis, the earlier the dimension features, the stronger the commonality. Therefore, in this embodiment, the first two principal components are preferred as structural commonality vectors, and the remaining components are used as structural deviation vectors. It should be noted that the dimension of the structural deviation vector is consistent with the dimension of the structural cluster, and the dimension corresponding to the commonality vector can be replaced by a unit vector.

[0049] S203: The obtained structural deviation vector is amplified based on the channel attention mechanism to obtain the structural deviation enhancement vector; the structural commonality vector and the structural deviation enhancement vector are then re-concatenated to obtain the bridge's structural feature vector after differential enhancement. The structural feature vector of the bridge after differential enhancement... The expression is as follows: .

[0050] .

[0051] .

[0052] in, Represents structural commonality vectors. represents the structural difference enhancement vector; μ represents the difference amplification factor, and the specific value can be selected according to the actual needs of those skilled in the art, for example, it can be 2 to 4; Represents the gain vector. Represents the structural deviation vector. σ(·) represents element-wise multiplication, W1 represents the weighting matrix, W2 represents the weighting matrix, and ReLU(·) represents the linear rectified activation function.

[0053] In this embodiment, when executing step S300, the multimodal large model uses a case-based reasoning mechanism to transfer knowledge across bridges to obtain model experience data of the benchmark bridge, including the load response model structure and model parameters of the benchmark bridge that have been verified and adapted. The case-based reasoning mechanism is based on four parts: "retrieval, reuse, correction, and retention", which will be described in detail below.

[0054] Specifically, for case retrieval, the case data corresponding to multiple benchmark bridges matched by the retrieval enhancement generation mechanism based on step S200 are used as source cases. For case reuse, the load response model that appears most frequently in the source cases is extracted as the initial template for new bridge load identification. For case correction, the multimodal large model uses the retrieved source cases as anchors, combines them with the real-time operating characteristics of the new bridge, generates structured reasoning basis, and corrects the initial template based on the obtained structured reasoning basis; for example, if the stiffness difference ΔK≤10%, the multimodal large model will infer "it is recommended to use linear correction (correction coefficient = new bridge stiffness / source case bridge stiffness)". For case retention, the corrected new bridge data is also added to the benchmark bridge case library, but its bridge metadata annotation is a newly deployed bridge with model parameters intelligently generated by multimodal large model reasoning, rather than a bridge obtained through actual driving by a weighted vehicle. Although currently only benchmark bridges are retrieved, their inclusion in the retrieval scope will be considered in the future.

[0055] In this embodiment, after completing the knowledge transfer of the load response model, the new bridge, in order to meet the demand for accurate load identification in multi-vehicle mixed traffic scenarios in actual traffic, utilizes the high-precision visual understanding and scene reasoning capabilities of a multimodal large model, relying on real-time visual data collected by bridge traffic flow monitoring cameras, to construct an intelligent response mechanism of "condition identification - model adaptation": First, the multimodal large model accurately analyzes the real-time visual data for traffic flow scenarios, automatically identifying different traffic flow condition types such as single-vehicle driving, multi-vehicle parallel driving, and multi-vehicle following, and extracting key features of the condition, such as the number of vehicles, relative positions, and driving distances; then, based on the identification results, it intelligently matches the corresponding adapted deflection-traffic load deep learning model. If it is a single-vehicle condition, the load response model specifically trained on single-vehicle data is directly called to accurately output the load data of a single vehicle; if it is a complex condition such as multi-vehicle parallel driving or following, the load response model trained on multi-vehicle coupled data is automatically called, and the model accurately models the coupling relationship between multi-vehicle load and deflection response, realizing the synchronous and accurate output of the load data of multiple vehicles. Compared with traditional methods, the technical solution of this application relies on the multimodal large model to accurately analyze working conditions such as single vehicle, multi-vehicle parallel driving, and following vehicle, and combines the adaptation mechanism of the special model for each working condition to solve the problem of identification confusion in multi-vehicle coupled scenarios in traditional methods, and meet the high-precision requirements of bridge safety monitoring for load data.

[0056] Understandably, the multimodal large model performs deep analysis of real-time visual data to identify key information such as vehicle type, license plate, and lane, and combines time-series analysis techniques to determine the traffic flow condition type. If there is only a single vehicle in the image without interference from other vehicles, it is determined to be a "single-vehicle driving condition"; if two or more vehicles are parallel in adjacent lanes with a lateral distance ≤ 2m, it is determined to be a "multi-vehicle driving condition"; if multiple vehicles are traveling sequentially in the same lane with a longitudinal distance ≤ 5m, it is determined to be a "multi-vehicle following condition". At the same time, the multimodal large model extracts key features of the condition: in the single-vehicle condition, it records vehicle speed and lane position; in the multi-vehicle condition, it additionally records parameters such as the number of vehicles, relative distance (lateral / longitudinal), and vehicle type combination (such as trucks and buses mixed), forming a structured condition feature vector, providing accurate scene labels for subsequent model adaptation.

[0057] It is important to note that, based on the structural characteristics of the benchmark bridges within the region, various neural network architectures are required to design load response models for different working conditions. Commonly used neural network architectures include CNN, BiLSTM, Transformer, and CNN-BiLSTM hybrid architectures. Furthermore, model performance optimization relies on manual adjustment of structural parameters such as the number of network layers and neurons, as well as hyperparameters such as learning rate and batch size. Since the initial template of the new bridge's load response model is transferred from the benchmark bridge, although the benchmark and new bridges are structurally similar after cosine similarity matching, certain differences still exist. Therefore, under the same working conditions based on these structural differences, the neural network architecture of the new bridge's load response model using the benchmark bridge as the initial template may not be optimal. Thus, in this embodiment, after obtaining the initial template through knowledge transfer from a multimodal large model, the new bridge automatically performs intelligent search and optimization design of the neural network architecture. Simultaneously, the multimodal large model can automatically test various deep learning model architectures and hyperparameter combinations based on the measured data characteristics of different bridges. By comparing the performance indicators such as recognition accuracy and convergence speed of different schemes, the optimal model configuration is adaptively selected. Compared to traditional methods, the technical solution of this application, through the automatic structure search and code generation capabilities driven by multimodal large models, can dynamically adjust the model architecture according to the structural characteristics and traffic flow features of different bridges, replacing the traditional manual parameter tuning mode that relies on expert experience. This enables rapid iteration and dynamic adaptation of the model structure, significantly improving development efficiency and reducing maintenance costs.

[0058] Specifically, during step S400, the multimodal large model, based on the real-time traffic flow conditions of the new bridge and the matching hyperparameters of the benchmark bridge, and combined with the performance feedback data of different architecture models in the benchmark bridge case library under similar scenarios, restructures the neural network architecture of the new bridge's structural response model. This includes the following processes: defining a candidate pool of suitable neural network architectures based on the data in the benchmark bridge case library; designing a grid structure based on grid depth, number of neurons, and feature fusion method for the structural features and traffic flow characteristics of the new bridge; and verifying and selecting the optimal neural network architecture with the highest matching pair with the current scenario through simulation training.

[0059] The process of defining the candidate pool for neural network architectures includes the following steps: First, based on the structured reasoning evidence obtained in the preceding steps (such as "structural difference analysis between the new bridge and the matching benchmark bridge"), intelligent modeling prompts are used to clarify the core constraints of the neural network architecture design (for example, if the reasoning indicates that "the stiffness of the new bridge is slightly lower than that of the benchmark bridge but the consistency of operating conditions is high", then the architecture type that has been verified as valid by the benchmark bridge is retained first). Combining the traffic flow operating conditions characteristics of the new bridge (such as the spatial locality of single-vehicle operating conditions and the temporal coupling of multi-vehicle following) and the model experience data of the matching benchmark bridge, the candidate architecture pool is defined in a targeted manner. Among them, the single-vehicle operating conditions focus on BiLSTM and CNN-BiLSTM architectures, while the multi-vehicle operating conditions focus on Transformer and hybrid architectures. Architectures with poor performance feedback in similar scenarios in the benchmark bridge case library are excluded (such as pure CNN architectures with accuracy below 85% in multi-vehicle operating conditions), thus reducing the invalid search space.

[0060] In designing the neural network architecture, an adaptive mapping between the feature dimensions of the new bridge and the input layer is required. This process includes the following steps: Preprocessing the input features of the new bridge; if the stiffness difference ΔK between the new bridge and the benchmark bridge is ≤10%, linear correction is used to adjust the deflection response data. To address environmental interference, the "environment-deflection correction curve" of the matching benchmark bridge is retrieved from the benchmark bridge case library to correct the new bridge response data. For example, using temperature interference, the "temperature-deflection correction curve" of the benchmark bridge is retrieved to correct the new bridge response data, eliminating environmental bias. Combining the corrected data features (such as the dynamic deflection time series length) and the input layer design experience of similar data from the benchmark bridge, the optimal sequence length of the benchmark bridge's time series data is used as the input layer parameter. In other words, the multimodal large model automatically uses the optimal time step obtained by the benchmark bridge during the experiment as the input layer parameter of the reconstructed neural network architecture.

[0061] In designing the neural network architecture, the generation of the mesh structure includes the following process: Based on the complexity of the new bridge's mechanical properties (e.g., nonlinear response of a long-span arch bridge > linear response of a simply supported beam bridge), the complexity of the operating conditions (multiple vehicles following > two vehicles driving side-by-side > single vehicle), and the degree of structural difference between the new bridge and the benchmark bridge, multiple candidate mesh structures are dynamically generated. When designing the mesh depth (i.e., the number of mesh layers), if inference indicates that the complexity of the new bridge is higher than that of the benchmark bridge, the mesh depth is increased by 1 to 2 layers, using the optimal depth of the benchmark bridge in similar scenarios as a benchmark; for example, the mesh depth for a simply supported beam bridge under single-vehicle conditions is 3-5 layers, and the mesh depth for a complex arch bridge under multi-vehicle conditions is 6-8 layers. For scenarios with a mesh depth greater than 5 layers, residual connections are automatically added to ensure training stability; the specific method for automatically adding residual connections can be referenced from the deep network anti-gradient vanishing scheme of the benchmark bridge, and therefore will not be described here. The number of neurons in the input layer is strongly bound to the corrected feature dimension, and the number of neurons in the hidden layer decreases adaptively layer by layer; that is, the number of neurons in the input layer is equal to the corrected feature dimension, and the percentage by which the number of neurons in the hidden layer decreases layer by layer can be set according to the actual needs of those skilled in the art, for example, by decreasing by half layer by layer.

[0062] It's important to understand that the network structure of the load response model used on the new bridge is derived from the retrieval and inference of a multimodal large model, and it currently uses the model weights of the benchmark bridge as its initial weights. However, to achieve better model results, fine-tuning with real data from the new bridge is necessary. Since the new bridge does not undergo weighted vehicle testing, this embodiment achieves dynamic supplementation of the new bridge's real-world operating data by spatiotemporally associating the benchmark bridge with the new bridge, eliminating the need for separate weighted vehicle testing for the new bridge. For ease of understanding, a detailed description will follow.

[0063] In this embodiment, the process of fine-tuning the load response model of the new bridge is as follows: a spatiotemporal association of vehicle travel between the reference bridge and the new bridge is established through license plate recognition technology; if there are vehicles that continuously drive across the reference bridge and the new bridge, the load response data of the vehicles on the reference bridge is automatically associated with the new bridge dataset for training the load response model.

[0064] It's understandable that both the new bridge and the reference bridge are located within the same bridge cluster, meaning they belong to the same geographical area. Therefore, the spatiotemporal relationship of vehicle movement on the new and reference bridges can be constructed by using license plate recognition from cameras on both bridges. For example, if a large truck passes over the reference bridge at time t1, the multimodal large model can identify the truck's license plate using a camera on the reference bridge. Since the reference bridge has undergone a weighted vehicle test, its load response model data is relatively accurate. The multimodal large model can then record the load response data of the truck passing over the reference bridge and identify it using the license plate. When the truck reaches the new bridge at time t2, the multimodal large model can identify the truck's license plate using a camera on the new bridge and retrieve its load response data from the reference bridge based on the identified license plate, adding it to the new bridge's dataset for training the new bridge's load response model. It is important to note that, to ensure the accuracy of load response models under different working conditions, the dataset for supplementing data from the benchmark bridge through spatiotemporal correlation for new bridges needs to include data from different working conditions, and the load response data under different working conditions should be used separately to train the load response model for each working condition. Compared with traditional methods, the technical solution of this application, through the bridge data supplementation mechanism associated with license plate recognition and the dynamic update capability of the case library, can autonomously accumulate real-world operational data without the need for periodic shutdowns for calibration or full model retraining. In long-term operation and maintenance, the labor costs of data collection and model updates are significantly reduced, improving the sustainability and economy of bridge lifecycle monitoring.

[0065] Specifically, when training the load response model of the new bridge under different working conditions, for the single-vehicle working condition, the multimodal large model automatically loads the single-vehicle load-response dataset sample library and uses the MSE loss function to optimize the model's analytical ability for a single load. For the multi-vehicle working condition, a multi-vehicle coupled dataset is used, and the cross-entropy loss function is introduced to enhance the learning of the load distribution ratio between vehicles. At the same time, the model converges quickly and its accuracy is tuned by dynamically adjusting the learning rate. For ease of understanding, the training of the load response model under different working conditions will be described in detail below.

[0066] For training the load response model under single-vehicle conditions: Leveraging the code generation capabilities of the multimodal large model, a supplementary single-vehicle load-response dataset is automatically invoked, and the training and validation sets are divided in an 8:2 ratio. In data preprocessing, based on the inference results of the multimodal large model, code is automatically generated for tasks such as stiffness correction, linear correction of deflection data, and normalization of vehicle weight labels (mapping to the [0, 1] interval), improving model convergence efficiency. The loss function used is the MSE (mean squared error) loss function, and the optimizer is Adam. The initial learning rate is taken from the optimal hyperparameters of the benchmark bridge single-vehicle model, set to 0.001 by default; the weight decay coefficient is set to 1e-5 to suppress overfitting. Training stops when the total weight error of the validation set is ≤1% or after 50 rounds, and the optimal model parameters are saved.

[0067] For training the load response model under multi-vehicle conditions: Based on the code generation capability of the multimodal large model, the supplementary multi-vehicle load-response dataset is automatically invoked, and the training set and validation set are divided in an 8:2 ratio; at the same time, the data is grouped according to sub-conditions (two-vehicle parallel, two-vehicle following (vehicle distance 5m / 10m), three-vehicle following, etc.) to ensure that the samples of each sub-scenario are balanced. In terms of data preprocessing, based on the inference results of the multimodal large model (such as stiffness differences, temperature interference, etc.), code is automatically generated to perform linear correction (stiffness correction coefficient = new bridge stiffness / reference bridge stiffness) and temperature compensation (calling the temperature-deflection correction curve of the reference bridge), etc., for different sub-conditions, and the weight label of each vehicle is normalized (mapped to the [0,1] interval) to enhance the load response model's ability to identify differences in conditions. The loss function adopts a weighted cross-entropy (60%) + MSE (40%) hybrid loss. Cross-entropy loss focuses on learning the load distribution ratio among vehicles, while MSE loss optimizes the absolute accuracy of single-vehicle weight. The optimizer uses AdamW, with the initial learning rate taken from the optimal hyperparameters of the subdivided working condition model corresponding to the benchmark bridge, set to 0.002 by default. The weight decay coefficient is set to 5e-5 to suppress overfitting. The training stops when the total weight error of the validation set is ≤2% or when 80 training iterations are completed, and the optimal model parameters are saved.

[0068] The basic principles, main features, and advantages of this application have been described above. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely the principles of this application. Various changes and modifications can be made to this application without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claims. The scope of protection claimed by this application is defined by the appended claims and their equivalents.

Claims

1. A vehicle load identification method based on a multimodal large model, characterized in that, Includes the following steps: S100: Select benchmark bridges with different years of construction, structural types and stiffness levels in the region, conduct tests under multiple working conditions, and obtain a benchmark bridge case library. S200: Analyze the core feature parameters of new bridges deployed in the region using a multimodal large model, and based on the analysis results, use an enhanced generation mechanism to match multiple benchmark bridges from the benchmark bridge case library that are most similar to the core features of the new bridge. S300: Based on the matching benchmark bridge case, the multimodal large model transfers the load response model and model parameters of the benchmark bridge case to the new bridge through the case reasoning mechanism; S400: Based on the structural differences between the new bridge and the matching benchmark bridge, as well as the actual working conditions of the new bridge, the multimodal large model reconstructs the architecture of the migrated load response model to obtain the load response model architecture with the highest matching degree to the current working conditions and performs load identification.

2. The vehicle load identification method based on a multimodal large model as described in claim 1, characterized in that, Step S200 includes the following process: Based on the analysis of the core characteristic parameters of the new bridge using a multimodal large model, the obtained structured data and unstructured data based on real-time traffic flow are vectorized to obtain the feature vector of the new bridge. Multimodal feature extraction is performed on the benchmark bridge case library using a multimodal large model. The obtained structured and unstructured data are then vectorized to obtain benchmark bridge feature vectors corresponding to different benchmark bridges. Calculate the similarity between the feature vector of the new bridge and the feature vectors of each reference bridge, and match the new bridge with the reference bridges that meet the similarity requirements.

3. The vehicle load identification method based on a multimodal large model as described in claim 2, characterized in that, For the structured data of the new bridge and the benchmark bridge, a difference enhancement strategy is adopted when constructing the feature vectors, which specifically includes the following process: Based on the structural feature types of bridges, the structural features of new bridges and benchmark bridges are clustered to obtain structural clusters corresponding to different structural feature types; Principal component analysis is performed on each structural cluster, and the structural deviation vector is obtained by taking the first two principal components as the structural commonality vector. The obtained structural deviation vector is amplified based on the channel attention mechanism to obtain the structural deviation enhancement vector. After re-concatenating the structural commonality vector and the structural deviation enhancement vector, the structural feature vector of the bridge based on the difference enhancement is obtained.

4. The vehicle load identification method based on a multimodal large model as described in claim 3, characterized in that, The structural feature vector of the bridge after differential enhancement The expression is as follows: ； ； ； in, Represents structural commonality vectors. This represents the structural difference enhancement vector, where μ represents the difference amplification factor. Represents the gain vector. Represents the structural deviation vector. σ(·) represents element-wise multiplication, W1 represents the weighting matrix, W2 represents the weighting matrix, and ReLU(·) represents the linear rectified activation function.

5. The vehicle load identification method based on a multimodal large model as described in any one of claims 1-4, characterized in that, Step S300 includes the following process: The load response model that appears most frequently among the matched benchmark bridges is used as the initial template for load identification of the new bridge. The multimodal large model uses case data corresponding to multiple benchmark bridges as anchor points, and combines the real-time operating characteristics of the new bridge to generate structured reasoning basis. The initial template was modified based on the obtained structured reasoning, and the modified new bridge data was added to the benchmark bridge case library after being marked.

6. The vehicle load identification method based on a multimodal large model as described in claim 5, characterized in that, During step S400, the multimodal large model, based on the real-time traffic flow conditions of the new bridge and the matched hyperparameters of the benchmark bridge, and combined with the performance feedback data of different architecture models in the benchmark bridge case library under similar scenarios, restructures the neural network architecture of the new bridge's structural response model, including the following process: Based on data from the benchmark bridge case library, a candidate pool of suitable neural network architectures is defined; for the structural features and traffic flow characteristics of the new bridge, a grid structure based on grid depth, number of neurons, and feature fusion method is designed; and the optimal neural network architecture with the highest matching pair with the current scenario is selected through simulation training and verification.

7. The vehicle load identification method based on a multimodal large model as described in claim 6, characterized in that, The process of defining the candidate pool for neural network architectures includes the following steps: Based on the obtained structured reasoning, the core constraints of neural network architecture design are clarified; By combining the traffic flow characteristics of the new bridge with the model experience data of the matched benchmark bridge, a candidate pool of neural network architectures is defined in a targeted manner, and neural network architectures with poor performance feedback in similar scenarios in the benchmark bridge case library are excluded.

8. The vehicle load identification method based on a multimodal large model as described in claim 6, characterized in that, When designing a neural network architecture, it is necessary to adaptively map the feature dimensions of the new bridge to the input layer, which includes the following process: The input characteristics of the new bridge are preprocessed. If the stiffness of the new bridge and the reference bridge is less than the set value, the deflection response data of the new bridge is linearly corrected and adjusted. To address the interference from environmental factors, the "environment-deflection correction curve" of a matching benchmark bridge is retrieved from the benchmark bridge case library to correct the response data of the new bridge. Based on the corrected data characteristics and the input layer design experience of similar data from the benchmark bridge, the optimal sequence length of the benchmark bridge time series data is used as the input layer parameter.

9. The vehicle load identification method based on a multimodal large model as described in claim 6, characterized in that, When designing a neural network architecture, the generation of the mesh structure includes the following process: Based on the complexity of the mechanical properties of the new bridge, the complexity of the working conditions, and the structural differences between the new bridge and the reference bridge, multiple sets of candidate mesh structures are dynamically generated. If the reasoning indicates that the complexity of the new bridge is higher than that of the benchmark bridge, the mesh depth is increased by 1 to 2 layers based on the optimal depth of the benchmark bridge in similar scenarios; for scenarios with a mesh depth greater than 5 layers, residual connections are automatically added. The number of neurons in the input layer is strongly bound to the corrected feature dimension, and the number of neurons in the hidden layer is adaptively reduced layer by layer.

10. The vehicle load identification method based on a multimodal large model as described in claim 6, characterized in that, The process of fine-tuning the load response model of the new bridge is as follows: Establish a spatiotemporal correlation of vehicle travel between the benchmark bridge and the new bridge using license plate recognition technology; If there are vehicles that continuously cross the reference bridge and the new bridge, the load response data of the vehicles on the reference bridge will be automatically associated with the new bridge dataset for training the load response model.

Images