A multi-modal data processing method and system for smart traffic

Abstract

The application belongs to the technical field of intelligent transportation and artificial intelligence, and discloses a multi-modal data processing method and system for intelligent transportation. The method comprises the following steps: obtaining initial features of each mode corresponding to multi-modal traffic data; mapping the initial features of each mode to a unified dimension semantic space to obtain initial semantic features of each mode; performing semantic alignment on the initial semantic features of each mode to obtain target semantic features of each mode after alignment; fusing a traffic domain knowledge graph to obtain knowledge-enhanced semantic features of each mode; and predicting the knowledge-enhanced semantic features of each mode through a special prediction head to obtain task prediction results and confidence corresponding to each special prediction head, and a final decision result corresponding to the semantic features. The method realizes intelligent transportation scene adaptation, integrates domain knowledge and advanced fusion technology, and realizes multi-modal data semantic-level fusion and unified management.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and artificial intelligence technology, and in particular to a multimodal data processing method and system for intelligent transportation. Background Technology

[0002] Modern intelligent transportation systems are centered on "data-driven, collaborative perception, and intelligent decision-making." Their efficient operation heavily relies on the comprehensive collection, effective integration, and standardized management of multi-source heterogeneous traffic data. Currently, data sources in the field of intelligent transportation cover multiple dimensions, including roadside perception, vehicle-mounted terminals, management platforms, and third-party services. Specifically, this includes various modalities of data such as video surveillance, radar sensing, ETC gantries, internet-based travel information, meteorological information, and traffic management texts. These data collectively form the foundation for intelligent transportation decision-making.

[0003] However, existing multimodal traffic data fusion and governance technologies still face many insurmountable bottlenecks, severely restricting the improvement of the intelligence level of intelligent transportation systems. Although recent research has attempted to apply large-scale models and knowledge graph technologies to cross-modal data fusion, they still have significant limitations in the vertical field of intelligent transportation: First, the general semantic representation of large-scale models is difficult to adapt to the professional scenarios in the transportation field, lacking domain-specific fine-tuning and optimization; second, the fusion of knowledge graphs and multimodal features mostly remains at the surface level, failing to achieve deep knowledge propagation and feature enhancement through technologies such as graph neural networks; third, a complete pipeline of "feature extraction - semantic alignment - knowledge enhancement - multi-layer fusion - quality governance" has not been formed, making it impossible to systematically solve the aforementioned technical bottlenecks. Therefore, those skilled in the art are committed to developing a systematic method and system that can adapt to intelligent transportation scenarios, integrate domain knowledge and advanced fusion technologies, and achieve semantic-level fusion and unified governance of multimodal data. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is to provide a multimodal data processing method and system for intelligent transportation.

[0005] To achieve the above objectives, this invention provides a multimodal data processing method for intelligent transportation, comprising: S1, acquiring multimodal traffic data and extracting features from different modal traffic data to obtain initial features for each modality; S2, mapping the initial features of each modality to a semantic space of a unified dimension through a modality-specific projection network to obtain initial semantic features for each modality; S3, performing semantic alignment on the initial semantic features of each modality through a dual-mechanism alignment layer to obtain aligned target semantic features for each modality. The dual-mechanism alignment layer includes a contrastive learning alignment mechanism and a cross-modal attention alignment mechanism. The contrastive learning alignment mechanism is used to optimize the projection network through backpropagation of a contrastive learning loss function during the model training phase to achieve global semantic alignment of the initial semantic features of each modality. The cross-modal attention alignment mechanism is used to perform fine-grained association on the initial semantic features of each modality to obtain aligned target semantic features for each modality; S4, performing knowledge enhancement on the aligned target semantic features for each modality through a traffic domain knowledge graph to obtain knowledge-enhanced semantic features for each modality; S5, predicting the corresponding task decision results based on the knowledge-enhanced semantic features for each modality.

[0006] On the other hand, this application also provides a multimodal data processing system for intelligent transportation, including: a multi-source data access module for acquiring and caching multimodal traffic data in traffic scenarios, including visual modal data, text modal data, and numerical signal modal data; a feature extraction module for extracting features from the corresponding modal traffic data through a visual feature extraction module, a text feature extraction module, and a temporal encoder to obtain initial features for each modality, including initial visual features, initial text features, and initial numerical signal features; a semantic projection module for mapping the initial features of each modality to a unified semantic space through a modality-specific projection network to obtain initial semantic features for each modality; and a semantic alignment and knowledge enhancement module for semantically aligning the initial semantic features of each modality through a contrastive learning alignment mechanism and a cross-modal attention alignment mechanism, and combining traffic domain knowledge. The graph is enhanced by entity embedding to obtain knowledge-enhanced semantic features for each modality; a multi-level fusion module is used to perform feature-level fusion and semantic-level fusion on the knowledge-enhanced semantic features of each modality to generate a unified semantic representation; a dynamic decision module is used to obtain the prediction results and corresponding confidence scores of each modality based on the knowledge-enhanced semantic features of each modality, and generate the final decision result based on the prediction results and corresponding confidence scores of each modality; a data governance module is used to perform quality assessment, anomaly detection and automatic repair on multimodal traffic data and intermediate features, and feed back the repair results to the multimodal traffic data acquisition stage or the initial feature extraction stage of each modality; a data storage module is used to store the governed multimodal traffic data, knowledge-enhanced semantic features of each modality, unified semantic representation and final decision result of the task; a monitoring and management module is used to monitor the system operation status, configure parameters and issue anomaly alarms.

[0007] The beneficial effects of this invention are as follows: By extracting features from visual, textual, and numerical signal modal traffic data, and through unified semantic space projection, dual-mechanism cross-modal semantic alignment, traffic domain knowledge graph enhancement, and deep multimodal fusion, unified semantic representation and collaborative decision-making of multi-source heterogeneous traffic data are achieved; at the same time, combined with a data quality governance mechanism, online evaluation, anomaly detection, automatic repair, and feedback optimization of data quality are carried out, which improves the accuracy, robustness, and real-time performance of multimodal traffic data processing results, providing high-quality data support and decision-making basis for intelligent transportation applications such as traffic situation analysis, event detection and early warning, and road network collaborative scheduling. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of a multimodal data processing method for intelligent transportation provided in an embodiment of this application;

[0009] Figure 2 This is a diagram of a multi-level semantic fusion architecture provided in an embodiment of this application;

[0010] Figure 3 This is a schematic diagram of a multimodal data processing system structure for intelligent transportation provided in an embodiment of this application. Detailed Implementation

[0011] First, current multimodal data fusion and governance technologies in the field of intelligent transportation face numerous insurmountable bottlenecks, severely restricting the improvement of the intelligence level of intelligent transportation systems. Specific manifestations include:

[0012] The problems of data heterogeneity and semantic fragmentation are prominent: different modalities of traffic data have fundamental differences in structure, feature space, and semantic expression. Specifically, visual modal data uses pixel matrices as its carrier and focuses on the intuitive presentation of scenes; textual modal data uses natural language as its carrier and focuses on event description and attribute description; numerical signal modal data uses time series as its carrier and focuses on quantitative indicator monitoring. Traditional fusion methods mostly stay at the level of shallow feature splicing or simple weighted summation, without deeply exploring the semantic relationships behind different modalities of data. They cannot achieve semantic alignment and deep understanding of cross-modal data, resulting in a lack of semantic consistency in the fusion results, which makes it difficult to support high-level decision-making.

[0013] The "data silo" phenomenon is difficult to resolve: Traffic data is collected and managed by different departments and entities, such as traffic police, road administration, meteorology, and internet companies. The data standards across these departments are inconsistent, their semantic representations are incompatible, and there is a lack of a unified integration and governance mechanism. This results in data from different sources being unable to be effectively linked and shared, creating information barriers. For example, congestion information monitored by roadside videos cannot be effectively linked and analyzed with rainfall data from meteorological departments, making it difficult to determine the core causes of congestion and hindering accurate perception and collaborative decision-making regarding the overall traffic situation.

[0014] The data quality governance system is inadequate: Traffic data is prone to quality problems such as noise, missing data, anomalies, and redundancy during collection (e.g., sensor malfunctions, video obstruction), transmission (e.g., network latency, signal interference), and storage (e.g., data redundancy, format errors). Existing technologies mostly employ offline, singular quality verification methods, lacking online quality assessment, real-time anomaly detection, and automatic repair mechanisms across the entire "collection-transmission-fusion-application" process. This results in low-quality data entering downstream fusion and decision-making stages, severely impacting model reliability and decision accuracy.

[0015] Insufficient utilization and disconnected integration of domain knowledge: Existing multimodal fusion methods are mostly data-driven, lacking systematic integration and utilization of prior knowledge in the transportation domain. This means that knowledge from areas such as traffic regulations, road network topology, historical traffic incident cases, and traffic flow characteristics is not effectively integrated into the entire data fusion process. Furthermore, existing methods for fusing knowledge representation and data features are relatively simple, often supplementing information after the fact, failing to achieve deep coupling between knowledge and data. This results in poor interpretability of the fusion model's decisions and weak adaptability to complex traffic scenarios (such as extreme weather and special road conditions).

[0016] Insufficient adaptability of the converged architecture: Smart transportation scenarios have high requirements for the real-time and accuracy of data processing. Existing multi-level converged architectures are mostly general designs that have not been optimized for the time-series, correlation, and sudden characteristics of traffic data, resulting in problems such as high convergence latency, large resource consumption, and untimely response to sudden traffic events.

[0017] Therefore, this invention aims to overcome the aforementioned shortcomings of existing intelligent transportation multimodal data fusion and governance technologies, and solve core technical problems such as difficulties in semantic fusion of multi-source heterogeneous data, the inability to break down "data silos," lack of data quality governance, insufficient utilization of domain knowledge, and insufficient adaptability of fusion architecture. It provides a method and system for semantic-level fusion and unified governance of multimodal data in intelligent transportation, achieving semantic-level alignment of multimodal traffic data, knowledge-enhanced deep fusion, and full-process quality governance. This constructs a high-quality, unified intelligent transportation data foundation, providing reliable data support for upper-level traffic decision-making applications and improving the intelligence and collaboration level of intelligent transportation systems. The invention will be further described below with reference to the accompanying drawings and embodiments. It should be noted that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0018] like Figure 1 As shown in the figure, this application provides a multimodal data processing method for intelligent transportation, the method including:

[0019] S1, acquire multimodal traffic data, and extract features from different modal traffic data to obtain initial features for each modality;

[0020] Specifically, standardized interfaces can be used to connect to energy data nodes such as roadside sensing devices (including video surveillance, radar sensors, and loop detectors), traffic management platforms (including ETC gantries, alarm receiving systems, and violation records), and third-party services (including internet travel platforms and meteorological departments). This enables real-time access and caching of visual modal data, text modal data, and numerical signal modal data. Visual modal data can include video streams and image frames, text modal data can include alarm texts, construction notices, and weather reports, and numerical signal modal data can include data such as vehicle speed, traffic flow, rainfall, and visibility. It should be noted that multimodal traffic data can be preprocessed before being connected to the data processing network. Preprocessing steps may include data format standardization (converting to JSON / TFRecord format), data redundancy removal (deleting duplicate and invalid data), and data normalization (normalizing numerical signals to the [0,1] interval) to lay the foundation for subsequent feature extraction. Timestamps should be added to each modality data, with timestamps accurate to milliseconds. Event-level alignment should be performed using a sliding time window (e.g., 10 seconds). At the same time, the corresponding modality data within the window should be marked as missing. No restrictions are imposed here.

[0021] This allows for the acquisition of multimodal traffic data for extracting initial features of each modality. After obtaining the multimodal traffic data, the corresponding initial features of each modality are extracted through the visual feature extraction module, text feature extraction module, and temporal encoder, which are fine-tuned by the traffic scene.

[0022] Therefore, prior to S1, the present invention further includes: acquiring traffic scene annotation data, which includes visual annotation data, text annotation data, and temporal annotation data; training a visual model using the cross-entropy loss function based on the visual annotation data to obtain a visual feature extraction module; training a text model using masked language modeling and domain classification based on the text annotation data to obtain a text feature extraction module; and training an untuned temporal encoder using temporal annotation data combined with mean squared error to obtain a temporal encoder.

[0023] Specifically, multiple sets (e.g., 20,000 images) of accident, congestion, and other scenarios can be acquired as visual annotation data. The visual annotation data can be labeled traffic images / video frames containing tags such as accident, congestion, and construction scenarios. The visual model is trained using this visual annotation data in conjunction with the cross-entropy loss function (e.g., training for 20 or 50 rounds) to obtain a trained visual feature extraction module. This visual model can use a convolutional neural network or a visual Transformer network as the backbone network, such as ResNet-50 or Vision Transformer (ViT-B / 16). There are no restrictions here.

[0024] Traffic alarm records, construction notices, and other textual data can be acquired as text annotation data. This text annotation data can be structured traffic text corpus, that is, text information such as alarm records, construction notices, and weather reports that have already completed scene classification and annotation. By using masked language modeling and domain classification as loss functions, the text model is trained to learn traffic language rules and judge traffic event categories. The text model is trained (e.g., 10 rounds) to obtain the text feature extraction module. This text model can use BERT-based as the backbone network, which is not restricted here. It should be noted that masked language modeling can be understood as randomly masking some words in a sentence during model training, and letting the model guess the masked words based on the context. This yields the masked language model loss. At the same time, the domain classification loss can be used to train the model's ability to judge the category of traffic events, and the domain classification loss can be obtained (which can be calculated using cross-entropy). By jointly optimizing the masked language model loss and the domain classification loss, the text model can simultaneously learn the semantic representation of the traffic domain and the ability to distinguish traffic event categories. It can learn entity relationships and semantic associations in traffic event descriptions based on traffic domain corpora, such as the relationships between vehicle accidents, road construction, severe weather, and traffic congestion.

[0025] Similarly, a time-series labeled dataset in the transportation domain can be obtained. This dataset can include continuous time-series information such as vehicle speed data, traffic flow data, road occupancy data, and meteorological environmental data. Appropriate labels are set according to the corresponding traffic scenario or prediction task. This time-series dataset is then input into a time-series encoder for training. The encoder learns the local variation patterns and long-term temporal dependencies in the traffic time-series data. The time-series encoder can employ a network structure combining a Temporal Convolutional Network (TCN) and a Long Short-Term Memory Network (LSTM). The TCN extracts local temporal patterns from the traffic data to capture abrupt changes in the time-series data within a short timeframe. The LSTM learns the gradual trend features formed by traffic states changing over a long period, thereby improving the encoder's ability to represent complex traffic change patterns. It should also be noted that during the training of the time-series encoder, the error between the predicted time-series features output by the model and the actual labeled data can be calculated using a loss function. The network parameters are then updated based on backpropagation of the error. The loss function can be the Mean Squared Error (MSE). The time-series error (MSE) loss function measures the deviation between the model's predictions and the actual traffic conditions, enabling the model to progressively improve its ability to learn traffic temporal changes. During model optimization, the AdamW optimization algorithm is used to update the parameters of the time-series encoder. In one implementation, the time-series encoder is trained using a joint smart traffic flow-speed-weather time-series dataset for 15 training epochs, employing the AdamW optimizer with an initial learning rate of 1e-4 and a batch size of 64. The model parameters are updated end-to-end during training, allowing the time-series encoder to learn traffic state change patterns.

[0026] This allows for the aggregation and training of traffic data from different departments, breaking down information barriers and enabling effective correlation and sharing of data from different sources.

[0027] After completing the traffic domain training of the visual feature module, text feature extraction module, and temporal encoder, the trained model is used as the feature extraction network for the corresponding modality. This network is applied to encode the features of the input multimodal traffic data, which includes visual modal data, text modal data, and numerical signal modal data. The feature extraction in S1 includes: extracting features from video frames and image data in the visual modal data through the visual feature extraction module to obtain spatial visual features; extracting spatiotemporal visual features from the video stream in the visual modal data through a 3D convolutional neural network, and processing the spatiotemporal visual features through a temporal attention module to obtain processed spatiotemporal visual features. The aforementioned spatial visual features and the processed spatiotemporal visual features constitute the initial visual features; performing word segmentation and encoding on the text modal data to obtain text word codes, and inputting the text word codes into the text feature extraction module to extract text semantic information to obtain initial text features; and extracting features from the numerical signal modal data through the temporal encoder to obtain initial numerical signal features. The initial features for each modality include initial visual features, initial text features, and initial numerical signal features.

[0028] Specifically, the visual feature extraction module encodes traffic entities, road environments, and event-related areas in the input image data or video frames through a multi-layer feature extraction network. It can extract spatial visual features containing information on vehicle targets, road structures, traffic conditions, and event categories. Of course, for continuous traffic video stream data, it can be processed using a three-dimensional convolutional neural network (3D CNN), that is, feature extraction is performed simultaneously in the spatial and temporal dimensions to obtain spatiotemporal visual features containing the dynamic changes of traffic events. Furthermore, in order to enhance the model's ability to focus on key traffic time stages, the spatiotemporal features output by the 3D CNN can be input into the temporal attention module. By calculating the attention weights corresponding to different time frames, the model can adaptively focus on key time segments such as accident occurrence frames, congestion formation frames, and construction impact frames, reducing the influence of irrelevant background frames on feature expression, and obtaining processed spatiotemporal visual features. The spatial visual features and the processed spatiotemporal visual features are used together as the initial features of the high-dimensional visual modality.

[0029] Furthermore, the text modal data can be divided into multiple units, and each unit can be encoded. Invalid words (such as modal particles and neutral particles) can be filtered out to obtain text unit encodings. Here, a pre-defined vocabulary can be used to encode each unit; no restrictions are placed on this. The text unit encodings are then input into the BERT-base model, which has been fine-tuned using traffic domain corpora, and processed by the text feature extraction model to obtain the initial text features.

[0030] Furthermore, when extracting the temporal variation patterns and trend features of numerical signal modal data through a temporal encoder, in order to improve the model's ability to perceive abnormal traffic conditions and key change nodes, a temporal attention mechanism can be used to calculate the weights of key time nodes, such as nodes where vehicle flow suddenly drops or vehicle speed suddenly increases, so that the model focuses on key time nodes where traffic conditions change significantly and outputs high-dimensional features of numerical signal modalities, i.e., initial features of numerical signals.

[0031] According to the technical solutions provided in the above embodiments, corresponding feature extraction mechanisms are designed for different modal data, which can fully explore the spatial features, semantic features and dynamic change features of images, text and time series data in traffic scenarios, improve the multimodal feature expression capability and cross-modal fusion effect, thereby improving the accuracy and stability of traffic event recognition and decision analysis in intelligent transportation scenarios.

[0032] It should be noted that each modality described in this application includes visual modality, text modality and numerical signal modality. Undoubtedly, the initial semantic features of each modality are visual initial semantic features, text initial semantic features and numerical signal initial semantic features. Other analogies will not be elaborated further.

[0033] S2 maps the initial features of each modality to a semantic space of a unified dimension through a modality-specific projection network to obtain the initial semantic features of each modality.

[0034] To reduce feature heterogeneity between different modalities, this embodiment first maps the initial features of different modalities to a semantic space of a unified dimension through a modality-specific projection network to obtain the corresponding initial semantic features of the modality.

[0035] Specifically, since different modal data have different data distributions and semantic representation characteristics, this embodiment sets up modality-specific projection networks with independent parameters for each modality. , m∈{v,t,s}, Corresponding visual modal projection network; Corresponding text modal projection network; This corresponds to a numerical signal modal projection network. Each modality-specific projection network can adopt the same or different network structures. The parameters of each projection network are independent of each other and are trained according to the data distribution of the corresponding modal features to learn the mapping relationship of different modal features from the original feature space to the unified semantic space.

[0036] Modality-specific projection networks are used to map initial features from different modalities to a semantic space of uniform dimension, thereby obtaining initial semantic features for each modality. = ( ),in ∈R N×D, ∈R N×D , ∈R N×D .

[0037] in, Represents the initial semantic features of vision. Indicates the initial semantic features of the text. Let N represent the initial semantic features of the numerical signal, D represent the number of samples, and D represent the dimension of the unified semantic space. In one implementation, the dimension of the unified semantic space is set to 256, but it is not limited to 256 dimensions. It can also be set to 128 dimensions, 512 dimensions, or other dimensions suitable for cross-modal alignment, depending on the actual application requirements. Through the above mapping process, the feature representations of different modalities can be placed in the same semantic space, thereby reducing the semantic gap between different modalities caused by differences in data type, feature dimension, and feature distribution, and providing a unified feature foundation for subsequent cross-modal semantic alignment and deep fusion.

[0038] In some embodiments, the projection network includes a first fully connected layer, a nonlinear activation layer, a second fully connected layer, and a normalization layer connected in sequence.

[0039] Specifically, in this embodiment, the specific projection networks for each modality adopt the same network structure. The nonlinear activation layer can use the ReLU activation function, and the normalization layer can use the Batch Normalization (BatchNorm) layer, thus forming a projection structure of "fully connected layer - activation layer - fully connected layer - normalization layer". Furthermore, the projection network can also be implemented using a multilayer perceptron, convolutional projection network, etc., and the normalization layer can also use normalization methods such as Layer Normalization. No restrictions are imposed here.

[0040] This approach allows for unified dimensional mapping of different modal features by setting up modality-specific projection networks. This effectively eliminates modal differences caused by variations in feature dimensions and distributions among visual, textual, and numerical signal modalities, improving the effectiveness of subsequent cross-modal semantic alignment and knowledge enhancement fusion. It also enhances the consistency and comparability of multimodal feature representations, thereby improving the accuracy and stability of multimodal joint modeling in intelligent transportation scenarios.

[0041] S3 uses a dual-mechanism alignment layer to semantically align the initial semantic features of each modality, resulting in aligned target semantic features for each modality. The contrastive learning alignment mechanism optimizes the projection network through backpropagation of the contrastive learning loss function during the model training phase to achieve global semantic alignment of the initial semantic features of each modality. The cross-modal attention alignment mechanism performs fine-grained association of the initial semantic features of each modality, resulting in aligned target semantic features for each modality.

[0042] It's important to note that the dual-mechanism alignment layer is not a traditional synchronous or continuous processing structure. Instead, it's a network structure that operates separately during the model training and inference phases. Specifically, the contrastive learning mechanism optimizes the projection network during training, at which point cross-modal attention can be frozen or processed to achieve fine-grained feature association. During inference, the projection network is already optimized, the contrastive learning alignment mechanism is frozen, and the cross-modal attention mechanism applies the optimized projection network output. In this embodiment, the contrastive learning alignment mechanism is used for training and optimizing the projection network parameters during training and does not directly output semantic features for subsequent fusion.

[0043] Specifically, the initial semantic features of each modality, mapped by a modality-specific projection network, are input into a dual-mechanism alignment layer. During model training, the contrastive learning alignment mechanism constructs the associations between semantic features of different modalities, calculates the corresponding contrastive learning loss function, and updates the projection network parameters through backpropagation. This enables the projection network to learn shared semantic information between different modalities. During training, initial semantic features of different modalities belonging to the same traffic event are constructed as positive sample pairs, and initial semantic features of different modalities belonging to different traffic events are constructed as negative sample pairs. The projection network parameters are updated through backpropagation using the contrastive learning loss function, thereby achieving higher consistency among initial semantic features of different modalities belonging to the same traffic event in a unified semantic space. In other words, the semantic features of different modalities output by the projection network exhibit cross-modal consistency within a unified semantic space.

[0044] In some embodiments, the contrastive learning loss function adopts the InfoNCE-based cross-modal contrastive loss function, which optimizes the model's ability to distinguish the relationship between positive and negative samples by calculating the similarity between semantic features of different modalities.

[0045] For example: Where M is the training batch size (e.g., 64). Anchor modal features (initial semantic features of any modality). For positive sample pairs of features that describe the same traffic event as the anchor features (initial semantic features of another modality). These are the negative sample features, i.e., the initial semantic features of traffic events that differ from the anchor modal features. The temperature coefficient (selectable value range is 0.1-0.5, in this embodiment it can be 0.3), sim(), is the similarity calculation function, in this embodiment it is cosine similarity, where for each batch of anchor samples ( , , For the same traffic event, the same modal features of other events in the same batch are used as negative samples, and the positive sample pairs are ( , (), , (), , By training with this loss function, the projection network can be driven to learn cross-modal semantic consistency, thereby improving semantic alignment accuracy. Of course, in other embodiments, the contrastive learning loss function can be based on cosine distance, triplet loss, center loss, or other loss functions that can constrain cross-modal feature consistency; no restrictions are imposed here.

[0046] Even after modality-specific projection network mapping, local semantic inconsistencies may still exist between modalities. Therefore, this invention introduces a cross-modal attention mechanism to achieve precise entity-level and attribute-level association of semantic features of different modalities during the model inference stage. In some embodiments, the cross-modal attention alignment mechanism is used to perform fine-grained association of initial semantic features of each modality to obtain aligned target semantic features of each modality. This includes: for each target modality initial semantic feature among the initial semantic features of each modality, calculating the attention weight matrix between the target modality initial semantic feature and the initial semantic features of each other modality; based on the attention weight matrix, performing weighted aggregation on the initial semantic features of each other modality to obtain the enhancement information of each other modality to the target modality; and fusing the enhancement information of each target modality with the corresponding initial semantic feature of the target modality to obtain the target semantic feature corresponding to each target modality.

[0047] Specifically, taking the visual modality as the target modality as an example, the calculation of the initial visual semantic features is explained. To the initial semantic features of the text Attention weight matrix ,in To unify the dimensions of the semantic space (e.g., 256). The relevance score matrix represents the relationship between the initial visual semantic features and the initial textual semantic features. It measures the degree of matching between each semantic element in the initial visual semantic features and each semantic element in the textual features. The softmax function is used to normalize the attention weights. Similarly, the initial visual semantic features can be calculated. Initial characteristics of numerical signals Attention weight matrix Based on attention weight matrix Initial semantic features of the text We perform weighted aggregation to obtain the first enhancement information of the text modality to the visual modality. Based on attention weight matrix Initial semantic features of numerical signals Weighted aggregation is performed to obtain the second enhancement information of the numerical signal mode on the visual mode. The first enhancement information is fused with the second enhancement information (e.g., element-wise addition, splicing, or gated fusion) to obtain the comprehensive enhancement information of the visual modality. This comprehensive enhancement information is then fused with the initial semantic features of the visual modality to obtain the corresponding semantic features of the visual target.

[0048] Similarly, text modality and numerical signal modality are used as target modalities respectively, and the text target semantic features and numerical information target semantic features are calculated in the same way as above, thereby achieving bidirectional information enhancement and fine-grained semantic alignment between different modalities.

[0049] S4. Knowledge enhancement is performed on the aligned target semantic features of each modality using a knowledge graph in the transportation domain to obtain knowledge-enhanced semantic features for each modality.

[0050] In order to address the problem of insufficient utilization of domain knowledge, this application embodiment constructs a knowledge graph of the transportation domain and realizes knowledge embedding and propagation through graph neural networks, deeply integrating domain knowledge into the multimodal semantic fusion process, thereby improving the interpretability and robustness of the fusion results.

[0051] First, a traffic domain knowledge graph G=(E,R,T) containing "entity-relationship-attribute" triples can be established, where E is the entity set, including road entities (intersections, road segments), vehicle entities (license plates, vehicle types), event entities (traffic accidents, congestion, construction), weather entities (rainfall, icing), and management entities (traffic police departments, construction units), etc.; R is the relation set, including "located at" (intersection-road segment), "caused by" (rainfall-congestion), "belongs to" (small cars-vehicles), "occurred at" (traffic accidents-intersections), etc.; T is the entity attribute set, including the location of the intersection, the speed limit of the road segment, the vehicle type, the duration of the event, etc. The knowledge graph can be stored in the Neo4j database, supporting real-time updates and retrieval of entities and relations, without any restrictions here.

[0052] In some embodiments, S4 includes: using a knowledge representation learning algorithm to represent entities and relations in the traffic domain knowledge graph as vectors, obtaining entity embedding vectors and relation embedding vectors; performing multi-hop message propagation on the entity embedding vectors through a graph neural network and aggregating information of neighboring nodes of entity nodes to obtain knowledge-enhanced entity embedding vectors; determining a first fusion weight based on the modality type corresponding to each modality target semantic feature and the association relationship between entities in the knowledge graph; and fusing the knowledge-enhanced entity embedding vectors with each modality target semantic feature based on the first fusion weight to obtain knowledge-enhanced semantic features for each modality.

[0053] Specifically, the TransE knowledge representation learning algorithm can be used to embed entities E and relations R in the knowledge graph into low-dimensional vectors (e.g., 256), resulting in entity embedding vectors and relation embedding vectors. A graph neural network (e.g., GNN, preferably a graph attention network GAT) is then introduced to perform multi-hop information propagation on the entity embedding vectors. This multi-hop information propagation can be represented by GAT stacking two layers, where each entity aggregates information from its two-hop neighborhood nodes, updating the embedding layer by layer, effectively enhancing the context of entity semantics. Specifically, multi-hop neighbor sampling is achieved through adjacency matrix exponentiation. Aggregating the information of entity neighbor nodes and updating the entity embedding vector improves the completeness of entity semantics; no restrictions are imposed here.

[0054] The updated entity embedding vector is then weighted and fused with the aligned target semantic features of each modality. A first fusion weight α is set based on the correlation between the modality type and the entity (as an example, for visual modalities). =0.3, text modality =0.4, numerical signal mode =0.3), thus obtaining knowledge-enhanced semantic features:

[0055]

[0056] in, To enhance the semantic features of the m-th modality knowledge, m∈{v,t,s} GNN (Genius Neural Network) represents the semantic features of the target in each modality. G) is the entity embedding vector output by the graph neural network that is associated with the m-th modality feature. Let G represent the set of knowledge graph entities associated with the m-th modality, and let G represent the entire transportation domain knowledge graph. Through entity-modality association mapping, the semantics of entities in the knowledge graph are integrated into the target semantic features of the corresponding modality, achieving knowledge-driven feature enhancement and obtaining the corresponding knowledge-enhanced semantic features. Further explanation is omitted here.

[0057] This approach integrates the target semantic features, which have undergone cross-modal semantic alignment, with the entity knowledge in the knowledge graph, which has been enhanced through graph neural network propagation. This results in knowledge-enhanced semantic features that include prior knowledge and entity relationship information in the transportation domain, thereby improving the model's ability to understand and reason about complex traffic scenarios.

[0058] S5, based on the modal knowledge, enhance the semantic features to predict the corresponding task decision results.

[0059] Traffic task identification can be performed by enhancing semantic features of knowledge from various modalities based on prediction models or output heads, and task decision results can be obtained. There are no restrictions here. The task decision results correspond to traffic tasks and traffic events. For example, the output may be decision results such as car accidents or congestion. In this way, the corresponding traffic conditions can be quickly identified through multimodal traffic data, and traffic control can be carried out flexibly.

[0060] In some embodiments, S5 includes: performing task prediction on the knowledge-enhanced semantic features of each modality using dedicated prediction heads to obtain task prediction results and confidence levels corresponding to each dedicated prediction head; calculating a second fusion weight based on the task prediction results and confidence levels; and fusing the task prediction results output by each dedicated prediction head based on the second fusion weight to obtain a task decision result.

[0061] Specifically, for different intelligent transportation tasks (such as event detection and situation assessment), semantic features of each modality of knowledge are enhanced. Each modality's knowledge-enhanced semantic features correspond to a dedicated prediction head, which outputs the prediction result for the current task. Different tasks share the same set of prediction head structures, but the parameters between tasks are independent, meaning each task has its own modality prediction head. This dedicated prediction head can be a fully connected layer plus a softmax classifier, based on which the task prediction result is output. (such as event type, situation level) and corresponding confidence level (Value range [0,1]); Design a dynamic weight allocation network g(·) to... and Using the input as input, the normalized dynamic fusion weights (i.e., the second fusion weights) for each modality are calculated through a fully connected layer and a sigmoid activation function. (satisfies Σ) =1), the final task decision result y is obtained by weighted summation of the prediction results of each modality: The dynamic weight allocation network can adaptively adjust the weights based on the confidence and semantic relevance of different modalities. For example, when the confidence of the text modality is high (such as when the alarm record clearly indicates a traffic accident), the weight of the text modality is increased to improve the accuracy of decision-making. No restrictions are imposed here.

[0062] This application embodiment also introduces a deep fusion mechanism based on a Transformer encoder, and achieves deep semantic interaction between knowledge-enhanced semantic features of different modalities through cascaded self-attention and cross-attention mechanisms to construct a unified global semantic representation. In some embodiments, after S4, the method further includes: processing the knowledge-enhanced semantic features of each modality through a multi-head attention mechanism to obtain semantic association information between modalities; inputting the knowledge-enhanced semantic features of each modality and the semantic association information between modalities into a cross-modal fusion encoder for processing to obtain a unified semantic representation, which is used to store in a target database for performing downstream tasks in the transportation domain.

[0063] Specifically, a multi-head cross-attention module is constructed within a unified semantic space. Through querying, matching, and information aggregation between different modal features, dynamic interaction and complementary enhancement between different modal information are achieved. Let the m-th modal knowledge-enhanced semantic feature be used as the query vector. Key vector ′ and value vector Then, the multi-head cross-attention process can be represented as MultiHead( , ′, ′), This represents the query vector used by the current modality to actively obtain information from other modalities. ' represents a key vector used to describe the semantic content of other modalities. ′ indicates that the corresponding value vector is used to provide supplementary semantic information to the current modality.

[0064] Through this mechanism, different modalities can query, complement, and enhance each other across multiple semantic subspaces, thereby uncovering complex cross-modal semantic relationships hidden in traffic scenarios. In this embodiment, the multi-head cross-attention module can employ eight attention heads, each corresponding to a 32-dimensional semantic subspace, i.e., 8 × 32 = 256, thus ensuring consistency in the semantic space dimensions. Subsequently, the three modal knowledge-enhanced semantic features are concatenated into a sequence of length 3 according to modal order. , , Each position has a dimension of 256, which is used as the input sequence of the Transformer encoder. After interaction with the Transformer encoder, the output is a deeply fused unified semantic representation. The unified semantic representation is stored in the target database for performing downstream tasks in the transportation field. There are no restrictions here. It should be noted that it can be set as an independent feature branch, which does not participate in single-layer modal task prediction, but is only used to build a global unified data foundation to support global downstream tasks such as global traffic situation analysis and road network collaborative scheduling, and complements the dedicated task prediction branch.

[0065] Furthermore, this embodiment of the application also employs a feature concatenation method for shallow concatenation of highly synchronized and closely related modal target semantic features, preserving the original feature information of each modality to obtain feature layer fusion features. These features can be input together with knowledge-enhanced semantic features into the target encoder for deep semantic fusion to obtain a unified semantic representation. No limitations are imposed here.

[0066] This forms a multi-level fusion architecture that coordinates the feature layer, semantic layer, and decision layer, taking into account information integrity, semantic interaction depth, and task adaptability, and achieving deep fusion of multimodal semantic features after knowledge enhancement.

[0067] Figure 2 This is a schematic diagram of a multi-level semantic fusion architecture provided in an embodiment of this application. The following is in conjunction with... Figure 2 This section will provide further explanation of the multi-level converged architecture.

[0068] like Figure 2 As shown, the feature fusion layer concatenates the dimensionality of the aligned target semantic features of each modality, fully preserving the original modal details and achieving shallow information aggregation; the semantic layer fusion uses three types of knowledge-enhanced semantic features as input to construct a 3-sequence, 2-layer, 8-head Transformer encoding structure, and achieves multi-dimensional semantic complementarity, association mining, and global semantic reconstruction between modalities through multi-head cross-attention, outputting a unified semantic representation; the decision layer fusion uses the independent prediction heads of each modality to output individual task results and confidence scores, and the dynamic weight network adaptively allocates the second fusion weights according to the modality reliability, and the weighted calculation yields the final intelligent transportation task decision result, taking into account both global semantic consistency and task adaptability, without any restrictions here.

[0069] In some embodiments, the method further includes a data quality governance step: acquiring at least one of multimodal traffic data, initial features of each modality, aligned semantic features of each modality, or knowledge-enhanced semantic features of each modality as data to be governed; performing an online quality assessment of the data to be governed based on a multidimensional online data quality assessment to obtain a quality score for the data to be governed, and determining that a quality score less than a preset score is low-quality data; automatically repairing abnormal data detected in the low-quality data; and feeding back the repair results to the multimodal traffic data acquisition stage or the initial feature extraction stage.

[0070] In other embodiments, the online quality assessment of the data to be governed is performed based on a multi-dimensional online data quality assessment to obtain a quality score for the data to be governed. This includes: calculating four-dimensional indicators of the data to be governed and weighting and fusing the indicators to obtain a quality score; the four-dimensional indicators include completeness, accuracy, consistency, and timeliness.

[0071] Specifically, a full-process data quality governance mechanism of "online evaluation - anomaly detection - automatic repair - feedback optimization" is established to ensure the reliability of input data, intermediate features, and output fused data, including:

[0072] Online data quality assessment: Establish a multi-dimensional data quality assessment index system, which may include completeness, accuracy, consistency, and timeliness. In some implementations, completeness can be determined by the ratio of valid data to total data, i.e., the proportion of missing values, completeness = number of valid data / total number of data; accuracy can be determined by the deviation between the predicted deviation value and the true value, and is assessed by the deviation between the model output data and the corresponding true value, i.e., accuracy = 1 - predicted deviation value / true value; consistency can be assessed by the semantic consistency of cross-modal data, i.e., consistency = cross-modal semantic matching similarity; timeliness can be determined by the delay in data collection and processing; after calculating the corresponding values, a weighted summation method is used to calculate the data quality score Q (value range [0,100]). As an example, the four assessment indicators can be fixed with weights of 0.3 for completeness, 0.3 for accuracy, 0.2 for consistency, and 0.2 for timeliness, and the weighted summation is used to obtain the final quality score Q. If Q ≥ 80, the corresponding data is high-quality data; if 60 ≤ Q < 80, it is average-quality data; if Q < 60, it is low-quality data. The preset score is 60 points, but there are no restrictions here. The specific threshold value can be set according to actual needs. Data quality can also be evaluated by other data evaluation methods or parameters, such as calculating the data anomaly rate and other parameters to evaluate data quality in reverse. There are no restrictions here.

[0073] Anomaly detection: Based on the characteristics of different modal data, targeted anomaly detection algorithms can be adopted. For example, visual modal data uses anomaly detection based on reconstruction error (such as autoencoders) to detect abnormal frames caused by video occlusion and sensor failure; text modal data uses anomaly detection based on semantic similarity to detect meaningless text and erroneous descriptions; numerical signal modal data uses an algorithm combining Isolation Forest (IForest) and DBSCAN clustering to detect abnormal data such as sudden increases or decreases in traffic flow and abnormal vehicle speed (such as exceeding the speed limit by 2 times).

[0074] Automatic repair: For detected low-quality abnormal data, an appropriate repair method is adopted. For example, for missing values, a Transformer-based generative model (such as STTN, E2FGVI) is used to fill in the missing values ​​based on the temporal information of the preceding and following frames, and the missing values ​​are predicted using contextual semantics and historical data. Noisy data is denoised using Gaussian filtering and median filtering. Abnormal data is corrected by combining the historical average of the same period with domain knowledge (such as rate limits and traffic peaks). Data that cannot be repaired is marked as invalid data and removed.

[0075] Feedback optimization: The data quality assessment results and repair status are fed back to the data access and feature extraction stages. The model can dynamically adjust the multimodal data preprocessing parameters (such as the denoising threshold) and the attention weight of the feature extraction model based on the repair results, so as to continuously improve data quality and feature effectiveness.

[0076] Finally, the unified semantic representation, decision results, and high-quality multimodal data after multi-level fusion can be stored on a unified data platform (using a distributed database HBase + Redis cache), providing standardized query and service interfaces to support calls from upper-layer intelligent transportation applications (traffic situation analysis, signal optimization, event alarms), thereby achieving data reuse and sharing.

[0077] Figure 3 This is a schematic diagram of a multimodal data processing system structure for intelligent transportation provided in an embodiment of this application, as shown below. Figure 3 As shown, the system includes:

[0078] The multi-source data access module is used to acquire and cache multimodal traffic data in traffic scenarios, including visual modal data, text modal data, and numerical signal modal data.

[0079] The feature extraction module is used to extract features from multimodal traffic data through the visual feature extraction module, the text feature extraction module, and the time encoder to obtain initial features for each modality, including visual initial features, text initial features, and numerical signal initial features.

[0080] The semantic projection module is used to map the initial features of each modality to a unified semantic space through a modality-specific projection network to obtain the initial semantic features of each modality.

[0081] The semantic alignment and knowledge enhancement module is used to semantically align the initial semantic features of each modality through a contrastive learning alignment mechanism and a cross-modal attention alignment mechanism, and to enhance entity embedding by combining the knowledge graph of the transportation domain to obtain knowledge-enhanced semantic features of each modality.

[0082] The multi-level fusion module is used to perform feature-level fusion and semantic-level fusion on the semantic features of knowledge enhancement from various modalities to generate a unified semantic representation.

[0083] The dynamic decision-making module is used to enhance semantic features based on knowledge of each modality, obtain prediction results and corresponding confidence levels for each modality, and generate final decision results based on the prediction results and corresponding confidence levels for each modality.

[0084] The data governance module is used to perform quality assessment, anomaly detection, and automatic repair on the multimodal traffic data and intermediate processing features, and to feed back the repair results to the multimodal traffic data acquisition stage or the initial feature extraction stage of each modality.

[0085] The data storage module is used to store the processed multimodal traffic data, the knowledge-enhanced semantic features of each modality, the unified semantic representation, and the final decision-making results of the task.

[0086] The monitoring and management module is used to monitor the system's operating status, configure parameters, and issue anomaly alarms.

[0087] Specifically, the monitoring and management module can be deployed on a cloud management platform, responsible for the overall system operation monitoring, parameter configuration, and fault alarms; real-time monitoring of the operating status of each module; support for online adjustment of fusion parameters (such as temperature coefficient τ, fusion weight α) and governance parameters (such as quality scoring thresholds); and timely issuance of alarm signals when a module malfunctions or data quality is abnormal to ensure stable system operation. The remaining modules in this embodiment correspond to those in the above method embodiments and have been fully explained, so they will not be elaborated upon here. Furthermore, the connection relationships between the various modules are clearly understood through the descriptions of the above embodiments, and will not be elaborated upon here.

[0088] The technical problems described herein have been solved through the above embodiments, and the following significant beneficial effects can be achieved:

[0089] 1. It achieves semantic-level deep fusion of multimodal traffic data, breaking through the limitations of traditional shallow feature splicing: by constructing a unified semantic space, contrastive learning and fine-grained attention alignment, semantic-level alignment of heterogeneous data such as visual, text, and numerical signals is achieved; by combining traffic domain knowledge graphs and graph neural networks, domain knowledge is deeply integrated into the fusion process, forming a knowledge-enhanced fusion architecture, which significantly improves the accuracy, semantic consistency and interpretability of the fusion results.

[0090] 2. It effectively addresses the "data silo" phenomenon and constructs a unified intelligent transportation data foundation: Through standardized data access, unified semantic representation and fusion mechanisms, it achieves effective association and sharing of multi-source heterogeneous data from different departments and systems; the full-process data quality governance mechanism ensures the high quality and robustness of the data foundation, providing reliable and unified data support for upper-level intelligent transportation applications and breaking down information barriers.

[0091] 3. A full-process, intelligent data quality governance system has been constructed: adopting a closed-loop governance model of "online assessment - anomaly detection - automatic repair - feedback optimization", and designing targeted governance algorithms for the characteristics of different modalities of data, the system realizes real-time control and dynamic optimization of data quality, effectively solves problems such as noise, missing data and anomalies in traffic data, and significantly improves the reliability of downstream decision-making models.

[0092] 4. Improved system adaptability and decision-making ability for complex traffic scenarios: The multi-level fusion architecture takes into account both information integrity and semantic interaction depth. The dynamic weight allocation network can adaptively adjust the fusion weights of each modality to adapt to different intelligent transportation tasks and complex scenarios (such as extreme weather and sudden traffic events). The deep integration of domain knowledge enables the model to have the ability to reason about traffic common sense, improving the interpretability and robustness of decision-making.

[0093] 5. The system architecture is modular and scalable, with good implementability and reusability: The system adopts a distributed modular design, and each module can be deployed independently and flexibly expanded. It supports the rapid access of new modal data (such as vehicle terminal data) and new algorithm models. The standardized interface design enables it to adapt to intelligent transportation systems of different regions and scales. It can be widely used in various scenarios such as urban traffic, highways, and intercity road networks, and has strong practicality and promotion value.

[0094] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multimodal data processing method for intelligent transportation, characterized in that, include: S1, acquire multimodal traffic data, and extract features from different modal traffic data to obtain initial features for each modality; By using a visual feature extraction module, a text feature extraction module, and a temporal encoder finely tuned for traffic scenarios, features are extracted from the corresponding modal traffic data to obtain initial modal features that include visual initial features, text initial features, and numerical signal initial features. S2, the initial features of each modality are mapped to a semantic space of a unified dimension through a modality-specific projection network to obtain the initial semantic features of each modality; S3, through a dual-mechanism alignment layer, the initial semantic features of each modality are semantically aligned to obtain the aligned target semantic features of each modality. The dual-mechanism alignment layer includes a contrastive learning alignment mechanism and a cross-modal attention alignment mechanism. The contrastive learning alignment mechanism is used to optimize the projection network through backpropagation of the contrastive learning loss function during the model training stage to achieve global semantic alignment of the initial semantic features of each modality. The cross-modal attention alignment mechanism is used to perform fine-grained association of the initial semantic features of each modality to obtain the aligned target semantic features of each modality. S4, knowledge enhancement is performed on the aligned target semantic features of each modality through the knowledge graph of the transportation domain to obtain knowledge-enhanced semantic features of each modality; S5, based on the modal knowledge, enhance the semantic features to predict the corresponding task decision results.

2. The method according to claim 1, characterized in that, The multimodal traffic data includes visual modal data, text modal data, and numerical signal modal data. The feature extraction in S1 includes: The visual feature extraction module extracts features from the video frames and image data in the visual modality data to obtain spatial visual features; the three-dimensional convolutional neural network extracts spatiotemporal visual features from the video stream in the visual modality data, and the spatiotemporal visual features are processed by the temporal attention module to obtain the processed spatiotemporal visual features. The spatial visual features and the processed spatiotemporal visual features are the initial visual features. The text modal data is segmented and encoded to obtain text word encoding. The text word encoding is then input into the text feature extraction module to extract text semantic information and obtain the initial text features. The initial features of the numerical signal are obtained by extracting features from the modal data of the numerical signal using a time encoder. The initial features of each modality include the initial visual features, the initial text features, and the initial numerical signal features.

3. The method according to claim 1, characterized in that, The projection network includes a first fully connected layer, a nonlinear activation layer, a second fully connected layer, and a normalization layer connected in sequence.

4. The method according to claim 1, characterized in that, The contrastive learning loss function is a cross-modal contrastive loss function based on InfoNCE.

5. The method according to claim 1, characterized in that, The cross-modal attention alignment mechanism is used to perform fine-grained association of the initial semantic features of each modality to obtain aligned target semantic features for each modality, including: For each target modal initial semantic feature in each modal initial semantic feature, calculate the attention weight matrix between the target modal initial semantic feature and each other modal initial semantic feature; Based on the attention weight matrix, the initial semantic features of each other modality are weighted and aggregated to obtain the enhancement information of each other modality on the target modality; The enhanced information of each target modality is fused with the corresponding initial semantic features of the target modality to obtain the target semantic features corresponding to each target modality.

6. The method according to claim 1, characterized in that, The S4 includes: A knowledge representation learning algorithm is used to represent entities and relations in the traffic domain knowledge graph as vectors, resulting in entity embedding vectors and relation embedding vectors. The entity embedding vector is subjected to multi-hop message propagation through a graph neural network, and the neighbor node information of the entity node is aggregated to obtain the knowledge-enhanced entity embedding vector. The first fusion weight is determined based on the modality type corresponding to the semantic features of each modality target and the relationship between entities in the knowledge graph. Based on the first fusion weight, the knowledge-enhanced entity embedding vector is fused with the target semantic features of each modality to obtain the knowledge-enhanced semantic features of each modality.

7. The method according to claim 1, characterized in that, Following S4, it also includes: The semantic features enhanced by the knowledge of each modality are processed through a multi-head attention mechanism to obtain semantic association information between modalities; The enhanced semantic features of each modality knowledge and the semantic association information between modalities are input into the cross-modal fusion encoder for processing to obtain a unified semantic representation. The unified semantic representation is stored in the target database for executing downstream tasks in the transportation field.

8. The method according to claim 1, characterized in that, The S5 includes: By using dedicated prediction heads to perform task prediction on the knowledge-enhanced semantic features of each modality, the task prediction results and confidence scores corresponding to each dedicated prediction head are obtained. Based on the task prediction results and the confidence level, a second fusion weight is calculated. Based on the second fusion weight, the task prediction results output by each dedicated prediction head are fused to obtain the task decision result.

9. The method according to claim 1, characterized in that, The method also includes data quality governance steps: At least one of the following is obtained as the data to be governed: the multimodal traffic data, the initial features of each modality, the aligned semantic features of each modality, or the knowledge-enhanced semantic features of each modality. Based on the online assessment of multi-dimensional data quality, the data to be governed is assessed online to obtain a quality score for the data to be governed, and data with a quality score less than a preset score is identified as low-quality data. The abnormal data detected in the low-quality data is automatically repaired, and the repair results are fed back to the multimodal traffic data acquisition stage or the initial feature extraction stage of each mode.

10. A multimodal data processing system for intelligent transportation, characterized in that, include: The multi-source data access module is used to acquire and cache multimodal traffic data in traffic scenarios, including visual modal data, text modal data, and numerical signal modal data. The feature extraction module is used to extract features from multimodal traffic data through the visual feature extraction module, the text feature extraction module, and the time encoder to obtain initial features for each modality, including visual initial features, text initial features, and numerical signal initial features. The semantic projection module is used to map the initial features of each modality to a unified semantic space through a modality-specific projection network to obtain the initial semantic features of each modality. The semantic alignment and knowledge enhancement module is used to perform semantic alignment on the initial semantic features of each modality through a contrastive learning alignment mechanism and a cross-modal attention alignment mechanism, and to perform entity embedding enhancement by combining the knowledge graph of the transportation domain to obtain knowledge-enhanced semantic features of each modality. A multi-level fusion module is used to perform feature-level fusion and semantic-level fusion on the semantic features of the knowledge enhancement of each modality to generate a unified semantic representation; The dynamic decision-making module is used to enhance semantic features based on the modal knowledge, obtain prediction results and corresponding confidence levels for each modality, and generate final decision results based on the prediction results and corresponding confidence levels for each modality. The data governance module is used to perform quality assessment, anomaly detection, and automatic repair on the multimodal traffic data and intermediate processing features, and to feed back the repair results to the multimodal traffic data acquisition stage or the initial feature extraction stage of each modality. The data storage module is used to store the processed multimodal traffic data, the knowledge-enhanced semantic features of each modality, the unified semantic representation, and the final decision-making results of the task. The monitoring and management module is used to monitor the system's operating status, configure parameters, and issue anomaly alarms.

Images