Highway multi-modal toll data fusion governance and analysis engine system

Abstract

This invention relates to a multimodal toll data fusion governance and analysis engine system for highways. Its architecture includes a data access and buffer layer, a core computing engine layer, a data service layer, and a unified management and control layer. Its core modules include a multimodal data standardization and preprocessing module, a data quality governance and enhancement module, a spatiotemporal alignment and data association fusion module, an intelligent analysis model and algorithm library module, a fused data asset management and service interface module, and a platform operation and monitoring management module. This system functions as both a "refinery" and an "assembly workshop" in the highway operation data value chain, and the quality of its output directly improves the intelligence level and application effectiveness of the entire existing highway smart operation platform.

Description

Technical Field

[0001] This invention relates to a multimodal toll data fusion governance and analysis engine system for highways, belonging to the field of intelligent highway operation platforms. Background Technology

[0002] As my country's expressway network moves towards the goal of "all-weather access, full-process monitoring, and all-round service," especially with the promotion and application of "quasi-free flow" and "booth-less" toll collection models, expressway operation and management is undergoing a profound digital transformation. The core driving force and key challenges of this transformation both stem from data. Traditional toll collection systems are mostly built around a single business logic, forming numerous isolated data silos, such as independent transaction log databases, video surveillance systems, license plate recognition logs, and mobile law enforcement recorders. These data sources differ significantly in format, timing, granularity, and semantics, constituting a typical "multimodal" data environment, including structured transaction records, semi-structured logs and JSON messages, and unstructured video streams, audio streams, and high-definition images.

[0003] In the "booth-less" operation scenario, the physical removal of toll booths and the mobile operation of toll collectors make the traditional supervision and service model, which relies on fixed locations, manual patrols, and post-event review, unsustainable. The effectiveness of operation management increasingly depends on real-time perception, intelligent analysis, and collaborative application of data across the entire road segment, all time period, and all elements. However, raw, isolated, and unmanaged multimodal data cannot directly support upper-level intelligent applications. The lack of correlation between data points, like scattered puzzle pieces, prevents the formation of a complete business view; data quality varies, with issues such as missing, abnormal, and inconsistent data, directly affecting the reliability of analysis results; and the value inherent in massive amounts of unstructured data cannot be extracted using traditional methods. Therefore, there is an urgent need for a powerful middleware software system specifically responsible for the deep management, organic integration, and preliminary analysis of the aggregated multimodal raw data, refining it into high-quality, high-value "data fuel" and "knowledge components" that can directly serve business decision-making. The applicant has conducted research and development to address this urgent need and has achieved results, and now applies for this patent. Summary of the Invention

[0004] The main objective of this invention is to address the problems existing in the prior art by providing a multimodal toll data fusion governance and analysis engine system for highways. This system can fill the key technological gap between raw data aggregation and intelligent applications, and become the core hub of existing highway smart operation platforms.

[0005] The technical solution of this invention to solve its technical problem is as follows:

[0006] A highway multimodal toll data fusion governance and analysis engine system includes an architecture layer and a core module; the architecture layer includes:

[0007] The data access and buffering layer is used to receive raw data streams from the upstream data aggregation platform and distribute or store different types of data.

[0008] The core computing engine layer is used for stream or batch processing of data, as well as data quality assessment and model management.

[0009] The data service layer is used to provide data access and analysis services to external parties;

[0010] A unified management and control layer is used to provide users with a management and control interface;

[0011] The core module includes:

[0012] The multimodal data standardization and preprocessing module is used to initially organize the format of the raw data stream into a standardized format;

[0013] The data quality governance and enhancement module is used to configure data quality verification rules, fix existing data problems, and enhance data quality.

[0014] The spatiotemporal alignment and data association fusion module is used to integrate data from different sources, times, and spatial granularities into a coherent business scenario.

[0015] The intelligent analysis model and algorithm library module provides a series of analysis algorithms and models specifically for charging operation scenarios;

[0016] The integrated data asset management and service interface module is used to manage data assets formed after being processed by the data quality governance and enhancement module and the spatiotemporal alignment and data association fusion module, and provides a unified data service interface.

[0017] The platform operation and maintenance and monitoring management module is used to provide users with platform operation and maintenance and monitoring management functions.

[0018] The further improved technical solution of this invention is as follows:

[0019] Preferably, the data access and buffer layer is configured with a message queue and a storage system; for consuming real-time data streams (such as transaction logs and alarm information with extremely high real-time requirements), the data is distributed through streaming data topics; for massive historical data streams (such as video clips, image packages, and other large-capacity data), the data is stored in the storage system and the downstream processing unit is notified through the message queue.

[0020] Preferably, the core computing engine layer is configured with a series of interconnected microservice clusters and a dedicated metadata repository; the microservice clusters include a stream processing engine cluster, a batch processing engine cluster, a data quality assessment service, and a model management service; wherein, the stream processing engine cluster is used to perform data cleaning, formatting, real-time correlation, and simple risk control rule calculations on the consumed real-time data stream; the batch processing engine cluster is used to run trajectory reconstruction algorithms, large-scale correlation analysis, periodic feature calculations, and machine learning model training on massive historical data; the data quality assessment service is used to continuously monitor the quality indicators of input and output data and trigger predefined quality governance workflows; the model management service is used to manage the lifecycle of analysis models (such as license plate recognition enhancement models and abnormal behavior detection models); the processing results of each microservice cluster are recorded in the dedicated metadata repository.

[0021] Preferably, the data service layer provides the following services: query service, real-time subscription service, model API service, and data export service; wherein, the query service is used to support flexible multi-dimensional queries on the fused vehicle trajectory, event details, and statistical results; the real-time subscription service is used to subscribe to specific types of events or vehicle status changes; the model API service is used to package the trained analysis model into a service that can be called in real time; and the data export service is used to push the analysis results to a specified location on demand.

[0022] Preferably, the unified management and control layer is configured with a visual console for configuring data source mapping rules, defining data quality verification rules, editing data fusion logic, managing the deployment and scheduling strategies of analysis models, monitoring the health status and performance indicators of each component of the entire platform, viewing data lineage reports and quality dashboards, and managing user permissions and audit logs.

[0023] More preferably, in the data access and buffering layer, the message queue is a high-performance message queue that uses metadata messages, and the storage system is an object storage system and / or a distributed file storage system;

[0024] In the core computing engine layer, the stream processing engine cluster and the batch processing engine cluster use the same core data processing logic library; the lifecycle management of the analysis model includes version control, deployment, A / B testing and performance monitoring; the processing results of each microservice cluster include metadata, lineage, and task status;

[0025] In the data service layer, the model API service provides RESTful API services or gRPC services; the data export service pushes data to a specified database, data warehouse, or message queue.

[0026] By adopting the above preferred solutions, the specific technical details of the architecture layer can be further optimized.

[0027] Preferably, the workflow of the multimodal data standardization and preprocessing module includes:

[0028] S1. After the data access and buffer layer distributes or stores the original data stream, the original data stream is identified.

[0029] S2. If the recognition result is structured data (such as transaction records), then perform field mapping, type conversion, encoding unification, and basic information completion on the data.

[0030] If the identification result is unstructured data (such as video), the integrated lightweight AI algorithm is invoked to perform initial analysis on the data, extract key metadata, and generate a structured description file; the structured description file includes timestamps, location, and license plate number;

[0031] If the recognition result is audio data, the data is preprocessed (such as noise reduction, track splitting, etc.) for subsequent speech recognition.

[0032] If the recognition result is image data, then the data is compressed, formatted, and subjected to basic quality filtering.

[0033] S3. Add a unified timestamp and watermark to each data point for subsequent spatiotemporal alignment.

[0034] Preferably, the data quality governance and enhancement module is configured with a rule engine; the workflow of the data quality governance and enhancement module includes:

[0035] T1. The rule engine supports user-defined quality verification rules for different data types, including value range checks, logical consistency checks, and integrity checks. The data quality assessment service of the core computing engine layer performs real-time or periodic scanning on the raw data stream distributed or stored by the data access and buffer layer, and the data stream obtained by the modal data standardization and preprocessing module, according to the quality verification rules, and triggers the predefined quality governance workflow to generate quality scores and problem reports.

[0036] T2. Based on the quality score and problem report, fix the problems in the corresponding data; if the problem is one that can be automatically fixed (such as format errors in certain fields of the data), then perform automatic repair on the problem; if the problem is one that is missing important information (such as missing gantry signage due to weather), then perform intelligent repair based on context (such as front and rear gantry records, synchronous video).

[0037] T3. Implement data augmentation processing as needed (e.g., use super-resolution technology to improve the quality of blurry license plate images, or use data synthesis technology to expand training data in small sample scenarios).

[0038] The data quality governance and enhancement module adopts a trajectory repair model that integrates road network topology and vehicle kinematics. For trajectory repair with missing gantry records, it learns the historical traffic patterns of vehicles on the one hand, and uses the road network topology of the highway and basic vehicle kinematic constraints as hard conditions on the other hand, so as to achieve intelligent repair.

[0039] The data quality governance and enhancement module encapsulates each quality verification rule and each data enhancement operation as an independent microservice. The rule engine dynamically assembles the microservices according to the characteristics and quality objectives of the predetermined data stream to form a processing pipeline for the predetermined data stream.

[0040] Preferably, the spatiotemporal alignment and data association fusion module is configured with a preset high-precision road network spatiotemporal coordinate system; the workflow of the spatiotemporal alignment and data association fusion module includes:

[0041] V1. Select a vehicle and perform time synchronization calibration on the vehicle's identification information on different devices to eliminate clock deviations;

[0042] V2. Based on vehicle identifiers (such as license plate number, OBU-ID) and spatiotemporal proximity, all data of the vehicle (such as transaction records, gantry passage events, video capture clips, audio recordings, etc.) are associated and clustered to form a "data cluster" of the vehicle in a single passage process;

[0043] V3. Using trajectory interpolation and map matching algorithms, the "data clusters" are fused into a smooth and continuous spatiotemporal trajectory.

[0044] V4. Select the next vehicle and repeat V1 to V3 above;

[0045] When encountering time-series discrepancies between different data of a reserved vehicle, the spatiotemporal alignment and data association fusion module adopts a dynamic time warping algorithm to perform multimodal time-series alignment by finding the optimal alignment path of different modal data sequences in the change mode.

[0046] The spatiotemporal alignment and data association fusion module is configured with a vehicle identifier confidence list and adopts incremental entity association parsing technology. For newly added identification records, on the one hand, it matches them with vehicle identifiers with high confidence in the list, and on the other hand, it analyzes the spatiotemporal continuity between them and vehicles with low confidence in the list, dynamically merging or splitting vehicle identity assumptions. By combining the two aspects, the association relationship of vehicle entities is gradually built and corrected in the data flow process.

[0047] Preferably, the intelligent analysis model and algorithm library module is configured with a vehicle trajectory reconstruction and completion model, a traffic behavior pattern recognition model, a group association and gang analysis model, a video content structured analysis model, and an audio semantic analysis model. The vehicle trajectory reconstruction and completion model, based on Hidden Markov Chains or Graph Neural Networks, is used to intelligently infer and complete trajectory interruptions caused by equipment failure or recognition failure, ensuring the integrity of the path. The traffic behavior pattern recognition model utilizes temporal pattern recognition technology to learn normal vehicle traffic patterns from historical trajectories and identify abnormal behaviors in real time (such as turning around after prolonged stops at service areas, slow lingering on road sections, following too closely, etc.). (Similar to toll evasion or security risk behaviors); Group association and gang analysis model, used to analyze the co-occurrence patterns of vehicles in time and space, and the similarity of traffic habits, to discover potential accompanying vehicle groups in order to combat organized toll evasion; Video content structured analysis model, integrating target detection and behavior recognition algorithms, used to automatically identify semantic information (such as vehicle color, model, whether there is littering, whether there are pedestrians running in, toll collector gestures, etc.) from videos; Audio semantic analysis model, used to convert the dialogue between toll collectors and drivers into text through speech recognition and natural language processing technology, and further analyze the key information (such as disputed amount, special situation type) and sentiment in the dialogue content.

[0048] Preferably, the data assets of the fusion data asset management and service interface module include a "vehicle-trip" fusion fact table, a "vehicle long-term profile" tag library, and an "event knowledge graph"; wherein, the "vehicle-trip" fusion fact table is used to record complete multimodal information for each trip; the "vehicle long-term profile" tag library is used to accumulate vehicle tags based on historical behavior, including risk level, credit score, and frequently traveled routes; and the "event knowledge graph" is used to store various abnormal events, handling results, and their associated vehicle, personnel, and location information in a graph structure;

[0049] The data service interfaces of the integrated data asset management and service interface module include a query interface, a subscription interface, an analysis interface, and a data push interface. The query interface supports combined retrieval based on multi-dimensional conditions; the subscription interface allows business users to subscribe in real-time to the appearance of vehicles with specific tags or the occurrence of specific types of events; the analysis interface receives data fragments uploaded by users and performs real-time analysis using intelligent analysis models and algorithm libraries; and the data push interface synchronizes processed data assets to a predetermined data warehouse or business database according to a pre-defined plan.

[0050] Preferably, the platform operation and maintenance monitoring management module is configured with a unified graphical management portal; the platform operation and maintenance monitoring management module includes a resource monitoring submodule, a task scheduling and orchestration submodule, an alarm center submodule, and a log aggregation and auditing submodule; wherein, the resource monitoring submodule is used to collect real-time health status indicators of CPU, memory, disk, network, and various microservice clusters; the task scheduling and orchestration submodule is responsible for managing the dependencies, priorities, and scheduling strategies of batch processing jobs; the alarm center submodule is used to issue alarms for abnormal situations according to preset thresholds; the log aggregation and auditing submodule is used to collect all operation logs and processing logs, and provide retrieval and auditing functions.

[0051] The highway multimodal toll data fusion governance and analysis engine system of the present invention can be positioned as the core capability layer in the existing highway smart operation platform. On the one hand, it connects to the "all-element data aggregation platform" in the platform and receives the raw multimodal data stream provided by it. On the other hand, it serves a series of intelligent applications in the platform such as "risk intelligent alarm engine", "service quality inspection and evaluation system" and "intelligent handling workbench". The system can achieve: (1) Transforming data into information: Through cleaning, aligning and structuring multimodal data (such as video behavior recognition and speech to text), the raw data that is difficult to use directly is transformed into structured information with clear semantics. (2) Transforming information into knowledge: Through correlation fusion, spatiotemporal modeling, feature extraction and analysis, discrete information fragments are integrated into deep knowledge that reflects vehicle behavior, event causality and operation rules, such as complete vehicle travel trajectory map, accompanying vehicle group identification, abnormal event evidence chain, etc. (3) Transforming knowledge into services: By providing standardized analysis models, data service interfaces and real-time computing capabilities, the accumulated knowledge is empowered to upper-level business applications in a "plug and play" manner, which greatly improves the efficiency and intelligence level of application development. In summary, this system serves as both a "refinery" and an "assembly workshop" in the highway operation data value chain, and the quality of its output directly enhances the intelligence level and application effectiveness of the entire existing highway smart operation platform. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the overall architecture of Embodiment 1 of the present invention.

[0053] Figure 2 This is a schematic diagram of the core module of Embodiment 1 of the present invention. Detailed Implementation

[0054] The present invention will be further described in detail below with reference to the embodiments. However, the present invention is not limited to the examples given.

[0055] Example 1

[0056] The highway multimodal toll data fusion governance and analysis engine system of this embodiment follows the following principles when it is built: (1) Unified semantic layer principle: Construct a unified business semantic layer that is higher than the original data model, define standard data models for core entities (such as vehicles, toll stations, events) and business events (such as passage, special situations, alarms), and shield the heterogeneity of the underlying data source. (2) Integrated stream and batch processing principle: Design a unified processing logic that can simultaneously support real-time data stream processing (for immediate monitoring and response) and historical data batch processing (for in-depth analysis and model training), reducing the complexity of the technology stack and maintenance costs. (3) Pluggable architecture principle: Data analysis algorithms, quality verification rules, and data fusion rules are all modular and pluggable, supporting dynamic loading, updating and configuration without restarting the core service, adapting to rapid changes in business rules. (4) Data lineage and traceability principle: Completely record every step of the data processing process from access, fusion, governance to analysis, forming a clear data lineage graph, ensuring that any output result can be traced back to the original source data, and meeting audit and compliance requirements. (5) Resource elasticity and stability principle: computing and storage resources support elastic scaling to cope with traffic peaks such as holidays; the system has high availability and fault tolerance mechanism, key service nodes have no single point of failure, and ensure continuous and stable operation 24 / 7.

[0057] This embodiment of the highway multimodal toll data fusion governance and analysis engine system adopts a layered, microservice-based, and cloud-native friendly overall technical architecture, aiming to achieve high cohesion, loose coupling, and easy scalability. Figure 1As shown, the entire architecture is divided into a data access and buffering layer, a core computing engine layer, a data service layer, and a unified management and control layer from bottom to top. Among them, (1) Data access and buffering layer: receives raw data streams from the upstream data aggregation platform through a high-performance message queue and a distributed file storage system; establishes independent topics or storage paths for different types of data; distributes transaction streams and alarm information with extremely high real-time requirements through streaming data topics; stores large-capacity data such as video clips and image packages in object storage or a distributed file system, and notifies downstream processing units through metadata messages; this layer plays a key role in decoupling upstream and downstream, buffering peak traffic, and ensuring data reliability. (2) Core computing engine layer: It is configured with a series of microservice clusters that work together; the stream processing engine cluster is responsible for consuming real-time data streams and performing low-latency data cleaning, formatting, real-time correlation and simple risk control rule calculation; the batch processing engine cluster is oriented towards massive historical data and runs complex trajectory restoration algorithms, large-scale correlation analysis, periodic feature calculation and machine learning model training tasks; the stream processing engine cluster and the batch processing engine cluster share the same core data processing logic library to ensure the consistency of processing results; a dedicated data quality assessment service continuously monitors the quality indicators of input and output data and triggers predefined quality governance workflows; the model management service is responsible for the lifecycle management of analysis models (such as license plate recognition enhancement models and abnormal behavior detection models), including version control, deployment, A / B testing and performance monitoring; all metadata, lineage and task status generated in the processing process are recorded in a dedicated metadata repository. (3) Data Service Layer: Provides standardized data access and analysis services to the outside world; specifically including: query service, which supports flexible multi-dimensional queries on the fused vehicle trajectory, event details, and statistical results; real-time subscription service, which allows upstream applications to subscribe to specific types of events or vehicle status changes; model API service, which packages the trained analysis model into a RESTful API or gRPC service for business systems to call in real time; data export service, which is responsible for pushing the analysis results to the specified database, data warehouse, or message queue as needed for use by downstream BI tools or special systems. (4) Unified Management and Control Layer: Provides a comprehensive management and control interface for system administrators and data analysts; through the visual console, users can configure data source mapping rules, define data quality verification rules, edit data fusion logic, manage the deployment and scheduling strategies of analysis models, monitor the health status and performance indicators of each component of the entire platform, view data lineage reports and quality dashboards, and manage user permissions and audit logs; this layer makes complex back-end technical systems observable, configurable, and manageable.

[0058] The core modules of the highway multimodal toll data fusion governance and analysis engine system in this embodiment include: a multimodal data standardization and preprocessing module, a data quality governance and enhancement module, a spatiotemporal alignment and data association fusion module, an intelligent analysis model and algorithm library module, a fused data asset management and service interface module, and a platform operation and maintenance and monitoring management module. Details are as follows.

[0059] (1) Multimodal data standardization and preprocessing module:

[0060] This module serves as the "outpost" for data processing, responsible for initially organizing raw, chaotic, heterogeneous data into a standardized format that the engine can process. For structured data, such as transaction logs, the module performs field mapping, type conversion, and encoding standardization, and completes necessary basic information. For unstructured data, such as video, the module calls integrated lightweight AI algorithms for initial analysis, extracting key metadata, such as extracting segments of vehicle passage from video streams, and generating structured description files containing timestamps, locations, and license plate numbers (if identifiable). For audio data, the module performs preprocessing such as noise reduction and track splitting to prepare for subsequent speech recognition. For image data, it performs compression, format standardization, and basic quality filtering. This module is also responsible for adding a unified timestamp and watermark to all input data, laying the foundation for subsequent spatiotemporal alignment.

[0061] (2) Data Quality Governance and Enhancement Module:

[0062] This module aims to enhance data credibility and value density. Specifically, it includes a configurable rules engine that allows users to define quality verification rules for different data types, such as value range checks, logical consistency checks (e.g., entry time earlier than exit time), and integrity checks. The data quality assessment service scans the data stream in real-time or periodically based on these rules, generating quality scores and issue reports. For automatically resolvable issues, such as format errors in certain fields, the system performs automatic repair. For missing critical information, such as missing gantry identification due to weather conditions, the module can trigger an intelligent repair process based on context (records from preceding and following gantry points, concurrent video). Furthermore, the module includes data augmentation features, such as using super-resolution technology to improve the quality of blurry license plate images, or using data synthesis technology to augment training data in small-sample scenarios.

[0063] This module employs a trajectory restoration model that integrates road network topology and vehicle kinematics. For trajectory restoration with missing gantry records, a combination of physical constraints and statistical learning is used. The model not only learns historical vehicle traffic patterns but also incorporates highway network topology (ramps, interchanges, service area entrances) and basic vehicle kinematic constraints (maximum reasonable speed, acceleration) as hard conditions into the restoration algorithm. For example, when determining possible vehicle paths, it excludes path assumptions that require vehicles to complete impossible speed changes or movements within a very short time. This ensures that the trajectory restoration results not only conform to data patterns but also to the laws of the physical world, significantly improving the accuracy of path reconstruction at key nodes (such as interchanges).

[0064] This module also employs a distributed data quality pipeline technology based on microservice collaboration: data quality governance is designed as a distributed, orchestratable pipeline. Each quality verification rule or enhancement operation is encapsulated as an independent microservice. The rule engine dynamically assembles these microservices based on data characteristics and quality objectives, forming a processing pipeline for a specific data stream. For example, for image data, the pipeline can sequentially execute "clarity filtering service," "stain detection service," and "super-resolution enhancement service." This design makes the quality governance process highly flexible and customizable, and each service can be independently scaled and upgraded, greatly improving the overall processing capacity and adaptability of the system.

[0065] (3) Spatiotemporal alignment and data association fusion module:

[0066] This module is one of the most core innovative modules of the system. Its task is to unify and integrate data from different sources, times, and spatial granularities into a coherent business scenario. This module maintains a high-precision road network spatiotemporal coordinate system. For the identification information of each vehicle, the module first performs time synchronization calibration to eliminate clock deviations between different devices. Then, based on vehicle identifiers (such as license plate number, OBU-ID) and spatiotemporal proximity, it clusters transaction records, gantry events, video capture clips, audio recordings, etc., to form a "data cluster" for the vehicle during a single passage. Through algorithms such as trajectory interpolation and map matching, these discrete point records are fused into a smooth, continuous spatiotemporal trajectory. Furthermore, this module can identify the same vehicle video stream across cameras, performing cross-camera tracking to form a complete visual trajectory. Finally, it constructs a "digital twin" object that integrates all multimodal evidence for each vehicle passage, serving as the basis for subsequent in-depth analysis.

[0067] This module employs a multimodal temporal alignment technique based on dynamic time warping (DTW). Addressing the challenge of temporal asynchrony between video frame sequences, audio streams, and discrete transaction events, an improved DTW algorithm is used. This algorithm is not limited to strict clock alignment but rather seeks the optimal alignment path for different modal data sequences across changing patterns. For example, by performing feature extraction and DTW matching between the action sequence of a vehicle starting to enter a lane in the video and the sound wave sequence of a conversation starting in the audio, even without a unified millisecond-level timestamp, the cross-modal events of "vehicle coming to a stop" and "greeting beginning" can be accurately aligned, providing a solid foundation for subsequent semantic association.

[0068] This module employs incremental entity association resolution technology for streaming data: in real-time data streams, vehicle identity information may be incomplete or ambiguous. Incremental association resolution maintains a list of vehicle identifier confidence levels within a short time window. When a new identification record arrives, the system not only matches it with high-confidence identifiers but also considers its spatiotemporal continuity with low-confidence records in the window, dynamically merging or splitting vehicle identity assumptions. This technology effectively handles complex situations such as damaged license plates and temporary plate changes, progressively building and correcting the association relationships of vehicle entities during the data stream process, improving the accuracy of real-time fusion.

[0069] (4) Intelligent Analysis Model and Algorithm Library Module:

[0070] This module encapsulates a series of specialized analysis algorithms and models for charging operation scenarios, and provides them in a service-oriented manner. The specific core models are as follows.

[0071] Vehicle trajectory reconstruction and completion model: Based on hidden Markov chains or graph neural networks, it intelligently infers and completes trajectory interruptions caused by equipment failure or recognition failure, ensuring the integrity of the path.

[0072] Traffic behavior pattern recognition model: Utilizing temporal pattern recognition technology, it learns the normal traffic patterns of vehicles from historical trajectories and can identify abnormal behaviors in real time, such as turning around after a long stay in a service area, lingering at low speed on a road segment, following too closely, and other behaviors that are suspected of evading tolls or posing safety risks.

[0073] Group association and gang analysis model: By analyzing the co-occurrence patterns of vehicles in time and space, and the similarity of their travel habits, potential accompanying vehicle groups can be identified, providing clues for combating gang-related toll evasion.

[0074] Video content structured analysis model: Integrating target detection and behavior recognition algorithms, it can automatically identify rich semantic information from videos, such as vehicle color, vehicle type, whether there is littering, whether there are pedestrians running in, and toll collector gestures.

[0075] Audio semantic analysis model: Through speech recognition and natural language processing technology, the dialogue between toll collectors and drivers is converted into text, and further analyzed for key information (such as disputed amount, special situation type) and sentiment in the dialogue content.

[0076] These models exist as "plugins" and can be dynamically updated and loaded through the model management service.

[0077] (5) Integration of data asset management and service interface module:

[0078] This module manages the high-value data assets formed after governance and integration, and provides a unified access point. The core data assets maintained by this module include: a "Vehicle-Trip" fused fact table, recording complete multimodal information for each trip; a "Long-Term Vehicle Profile" tag library, accumulating tags such as risk level, credit score, and frequently used routes based on historical behavior; and an "Event Knowledge Graph," storing various abnormal events, their handling results, and associated vehicle, personnel, and location information in a graph structure. This module provides rich data service interfaces: a query interface supporting combined searches based on multi-dimensional conditions; a subscription interface allowing business users to subscribe in real-time to the occurrence of vehicles with specific tags or the occurrence of specific types of events; an analysis interface allowing the upload of a data fragment and requesting the system to perform real-time analysis using intelligent analysis models and algorithm libraries (e.g., analyzing anomalies in a given video); and a data push interface that synchronizes processed data assets to the data warehouse or business database according to a predetermined plan.

[0079] (6) Platform Operation and Monitoring Management Module:

[0080] This module ensures the stable and efficient operation of the entire system. It includes: a resource monitoring submodule, which collects real-time health indicators for CPU, memory, disk, network, and various microservices; a task scheduling and orchestration submodule, responsible for managing the dependencies, priorities, and scheduling strategies of batch processing jobs; an alarm center submodule, which issues timely alarms based on preset thresholds for system anomalies, sudden drops in data quality, and model performance degradation; and a log aggregation and auditing submodule, which collects all operation and processing logs and provides powerful search and auditing functions. This module integrates all these functional submodules into a unified graphical management portal, providing operations and maintenance personnel and data administrators with a one-stop management experience.

[0081] This embodiment of the highway multimodal toll data fusion governance and analysis engine system supports a hybrid cloud deployment mode, with core components running in private data centers or on cloud platforms. The system's basic operating environment requires servers running mainstream Linux operating systems. The system relies on a container orchestration platform for deploying and managing all microservice instances, enabling rapid service start / stop, rolling updates, and elastic scaling. The underlying layer requires a distributed computing framework to support stream processing and batch processing tasks. State data and metadata are stored in a relational database, while massive amounts of intermediate results, model files, and unstructured data assets are stored in distributed object storage or file systems. To ensure real-time data exchange, a high-throughput distributed message queue needs to be deployed. Internal communication between all services prioritizes a high-performance RPC framework. The entire system is uniformly configured and managed through a configuration management center, and centrally managed through a log collection system. To meet data security and compliance requirements, the system supports integration with existing enterprise-level identity authentication and authorization systems.

[0082] Based on the specific implementation methods represented by the above embodiments, the highway multimodal toll data fusion governance and analysis engine system of the present invention has the following characteristics:

[0083] (1) Multimodal fusion: Achieve unified spatiotemporal alignment and semantic association of heterogeneous data such as transaction flow, gantry identification, video sequence, audio stream, and image file, and construct a fusion data model with "vehicle-time-space" as the main axis.

[0084] (2) Data governance: Establish a sound data quality control system to automatically clean, verify, repair and enhance multi-source data to ensure the accuracy, integrity, consistency and timeliness of data used by downstream applications.

[0085] (3) Intelligent analysis: Integrates lightweight machine learning and deep learning models to provide real-time and offline analysis capabilities, such as vehicle trajectory reconstruction, behavior pattern recognition, anomaly detection, feature profile generation, etc.

[0086] (4) Service-oriented: Provide highly cohesive and loosely coupled data and analysis service interfaces with a microservice architecture, which can be flexibly and efficiently called by upper-layer business systems to support real-time decision-making and batch analysis scenarios.

[0087] (5) High efficiency and scalability: It adopts a distributed computing and batch processing architecture to ensure that it can handle PB-level massive historical data and tens of thousands of real-time data streams per second; it has good horizontal scalability and supports online hot updates and expansion of analysis models.

[0088] In addition to the embodiments described above, the present invention may have other implementations. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.

Claims

1. A multimodal toll data fusion governance and analysis engine system for highways, comprising an architecture layer and core modules; characterized in that, The architecture layer includes: The data access and buffering layer is used to receive raw data streams from the upstream data aggregation platform and distribute or store different types of data. The core computing engine layer is used for stream or batch processing of data, as well as data quality assessment and model management. The data service layer is used to provide data access and analysis services to external parties; A unified management and control layer is used to provide users with a management and control interface; The core module includes: The multimodal data standardization and preprocessing module is used to initially organize the format of the raw data stream into a standardized format; The data quality governance and enhancement module is used to configure data quality verification rules, fix existing data problems, and enhance data quality. The spatiotemporal alignment and data association fusion module is used to integrate data from different sources, times, and spatial granularities into a coherent business scenario. The intelligent analysis model and algorithm library module provides a series of analysis algorithms and models specifically for charging operation scenarios; The integrated data asset management and service interface module is used to manage data assets formed after being processed by the data quality governance and enhancement module and the spatiotemporal alignment and data association fusion module, and provides a unified data service interface. The platform operation and maintenance and monitoring management module is used to provide users with platform operation and maintenance and monitoring management functions.

2. The highway multimodal toll data fusion governance and analysis engine system according to claim 1, characterized in that, The data access and buffering layer is configured with a message queue and a storage system; for consuming real-time data streams, distribution is carried out through streaming data topics; for massive historical data streams, storage is carried out in the storage system and downstream processing units are notified through the message queue.

3. The highway multimodal toll data fusion governance and analysis engine system according to claim 2, characterized in that, The core computing engine layer is configured with a series of interconnected microservice clusters and a dedicated metadata repository. The microservice clusters include a stream processing engine cluster, a batch processing engine cluster, a data quality assessment service, and a model management service. Specifically, the stream processing engine cluster performs data cleaning, formatting, real-time correlation, and simple risk control rule calculations on the consumed real-time data stream; the batch processing engine cluster performs trajectory reconstruction algorithms, large-scale correlation analysis, periodic feature calculations, and machine learning model training on massive amounts of historical data; the data quality assessment service continuously monitors the quality indicators of input and output data and triggers predefined quality governance workflows; and the model management service is responsible for the lifecycle management of the analysis models. The processing results of each microservice cluster are recorded in the dedicated metadata repository.

4. The highway multimodal toll data fusion governance and analysis engine system according to claim 3, characterized in that, The data service layer provides the following services: query service, real-time subscription service, model API service, and data export service. The query service supports flexible, multi-dimensional queries of the fused vehicle trajectories, event details, and statistical results. The real-time subscription service allows subscription to specific types of events or vehicle status changes. The model API service packages trained analysis models into services that can be called in real-time. The data export service pushes analysis results to designated locations on demand.

5. The highway multimodal toll data fusion governance and analysis engine system according to claim 4, characterized in that, The unified management and control layer is equipped with a visual console for configuring data source mapping rules, defining data quality verification rules, editing data fusion logic, managing the deployment and scheduling strategies of analysis models, monitoring the health status and performance indicators of all components of the platform, viewing data lineage reports and quality dashboards, and managing user permissions and audit logs.

6. The highway multimodal toll data fusion governance and analysis engine system according to claim 4, characterized in that, In the data access and buffering layer, the message queue is a high-performance message queue that uses metadata messages, and the storage system is an object storage system and / or a distributed file storage system. In the core computing engine layer, the stream processing engine cluster and the batch processing engine cluster use the same core data processing logic library; the lifecycle management of the analysis model includes version control, deployment, A / B testing and performance monitoring; the processing results of each microservice cluster include metadata, lineage, and task status; In the data service layer, the model API service provides RESTful API services or gRPC services. The data export service pushes data to a specified database, data warehouse, or message queue.

7. A highway multimodal toll data fusion governance and analysis engine system according to any one of claims 1 to 6, characterized in that, The workflow of the multimodal data standardization and preprocessing module includes: S1. After the data access and buffer layer distributes or stores the original data stream, the original data stream is identified. S2. If the recognition result is structured data, then perform field mapping, type conversion, encoding unification, and basic information completion on the data. If the identification result is unstructured data, the integrated lightweight AI algorithm is invoked to perform initial analysis on the data, extract key metadata, and generate a structured description file; the structured description file includes timestamps, location, and license plate number; If the recognition result is audio data, the data is preprocessed, including noise reduction and track splitting, in preparation for subsequent speech recognition. If the recognition result is image data, then the data is compressed, formatted, and subjected to basic quality filtering. S3. Add a unified timestamp and watermark to each data point for subsequent spatiotemporal alignment.

8. The highway multimodal toll data fusion governance and analysis engine system according to claim 7, characterized in that, The data quality governance and enhancement module is equipped with a rule engine; The workflow of the data quality governance and enhancement module includes: T1. The rule engine supports user-defined quality verification rules for different data types, including value range checks, logical consistency checks, and integrity checks. The data quality assessment service of the core computing engine layer performs real-time or periodic scanning on the raw data stream distributed or stored by the data access and buffer layer, and the data stream obtained by the modal data standardization and preprocessing module, according to the quality verification rules, and triggers the predefined quality governance workflow to generate quality scores and problem reports. T2. Based on the quality score and issue report, fix the problems in the corresponding data; if the problem is one that can be automatically fixed, then perform automatic fix; if the problem is one that is missing important information, then perform context-based intelligent fix. T3. Implement data augmentation processing as needed; The data quality governance and enhancement module adopts a trajectory repair model that integrates road network topology and vehicle kinematics. For trajectory repair with missing gantry records, it learns the historical traffic patterns of vehicles on the one hand, and uses the road network topology of the highway and basic vehicle kinematic constraints as hard conditions on the other hand, so as to achieve intelligent repair. The data quality governance and enhancement module encapsulates each quality verification rule and each data enhancement operation as an independent microservice. The rule engine dynamically assembles the microservices according to the characteristics and quality objectives of the predetermined data stream to form a processing pipeline for the predetermined data stream.

9. The highway multimodal toll data fusion governance and analysis engine system according to claim 8, characterized in that, The spatiotemporal alignment and data association fusion module is configured with a preset high-precision road network spatiotemporal coordinate system; The workflow of the spatiotemporal alignment and data association fusion module includes: V1. Select a vehicle and perform time synchronization calibration on the vehicle's identification information on different devices to eliminate clock deviations; V2. Based on vehicle identification and spatiotemporal proximity, all data of the vehicle are associated and clustered to form a "data cluster" of the vehicle during a single passage. V3. Using trajectory interpolation and map matching algorithms, the "data clusters" are fused into a smooth and continuous spatiotemporal trajectory. V4. Select the next vehicle and repeat V1 to V3 above; When encountering time-series discrepancies between different data of a reserved vehicle, the spatiotemporal alignment and data association fusion module adopts a dynamic time warping algorithm to perform multimodal time-series alignment by finding the optimal alignment path of different modal data sequences in the change mode. The spatiotemporal alignment and data association fusion module is configured with a vehicle identifier confidence list and adopts incremental entity association parsing technology. For newly added identification records, on the one hand, it matches them with vehicle identifiers with high confidence in the list, and on the other hand, it analyzes the spatiotemporal continuity between them and vehicles with low confidence in the list, dynamically merging or splitting vehicle identity assumptions. By combining the two aspects, the association relationship of vehicle entities is gradually built and corrected in the data flow process.

10. The highway multimodal toll data fusion governance and analysis engine system according to claim 9, characterized in that, The intelligent analysis model and algorithm library module includes a vehicle trajectory reconstruction and completion model, a traffic behavior pattern recognition model, a group association and gang analysis model, a video content structured analysis model, and an audio semantic analysis model. The vehicle trajectory reconstruction and completion model, based on Hidden Markov Chains or Graph Neural Networks, is used to intelligently infer and complete trajectory interruptions caused by equipment failure or recognition failure, ensuring path integrity. The traffic behavior pattern recognition model utilizes temporal pattern recognition technology to learn normal vehicle traffic patterns from historical trajectories and identify abnormal behaviors in real time. The group association and gang analysis model analyzes the co-occurrence patterns and traffic habit similarities of vehicles in time and space to uncover potential accompanying vehicle groups, in order to combat organized toll evasion. The video content structured analysis model integrates object detection and behavior recognition algorithms to automatically identify semantic information from videos. The audio semantic analysis model uses speech recognition and natural language processing technology to convert the dialogue between toll collectors and drivers into text and further analyze key information and sentiment within the dialogue content.

11. The highway multimodal toll data fusion governance and analysis engine system according to claim 10, characterized in that, The data assets of the integrated data asset management and service interface module include a "vehicle-trip" integrated fact table, a "vehicle long-term profile" tag library, and an "event knowledge graph." The "vehicle-trip" integrated fact table records complete multimodal information for each trip; the "vehicle long-term profile" tag library accumulates vehicle tags based on historical behavior, including risk level, credit score, and frequently used routes; and the "event knowledge graph" stores various abnormal events, their handling results, and associated vehicle, personnel, and location information in a graph structure. The data service interfaces of the integrated data asset management and service interface module include a query interface, a subscription interface, an analysis interface, and a data push interface. The query interface supports combined retrieval based on multi-dimensional conditions; the subscription interface allows business users to subscribe in real-time to the appearance of vehicles with specific tags or the occurrence of specific types of events; the analysis interface receives data fragments uploaded by users and performs real-time analysis using intelligent analysis models and algorithm libraries; and the data push interface synchronizes processed data assets to a predetermined data warehouse or business database according to a pre-defined plan.

12. The highway multimodal toll data fusion governance and analysis engine system according to claim 11, characterized in that, The platform operation and maintenance monitoring management module is configured with a unified graphical management portal. This module includes a resource monitoring submodule, a task scheduling and orchestration submodule, an alarm center submodule, and a log aggregation and auditing submodule. The resource monitoring submodule collects real-time health status indicators for CPU, memory, disk, network, and various microservice clusters. The task scheduling and orchestration submodule manages the dependencies, priorities, and scheduling strategies of batch processing jobs. The alarm center submodule issues alarms for abnormal situations based on preset thresholds. The log aggregation and auditing submodule collects all operation logs and processing logs, providing retrieval and auditing functions.

Images