A unified storage and management method for highway data center data

Abstract

The present invention discloses a unified data storage and management method for highway data centers, including: step S1, constructing a multi-source heterogeneous data acquisition channel, using edge computing nodes to process various data sources to generate standardized metadata packages; step S2, establishing a dynamic sharding storage matrix, slicing the data three-dimensionally from the time, space, and business dimensions; step S3, designing an event-driven data association engine, establishing a time-space mapping relationship, and achieving automatic associated storage through event ID chain binding; step S4, deploying a multimodal storage cluster, using different storage units to store different data, and tracking data lineage through distributed indexing services; step S5, implementing dynamic vectorized permission management, calculating access permission levels based on cosine similarity, and establishing a fine-grained data access control model; step S6, building a service-oriented data interface gateway. The present invention has significant advantages in highway data acquisition, storage, association, management, and sharing.

Description

Technical Field

[0001] The present invention relates to the field of highway data management and storage technology, and in particular to a method for unified storage and management of highway data middle platform data. Background Art

[0002] After years of development and construction, my country's expressway mileage continues to increase. During daily operations, various management systems generate rapid, large-scale, diverse, and heterogeneous data from multiple sources. As expressway mileage grows, the generation of big data is also rapidly increasing, and the amount of data that needs to be processed is becoming increasingly massive.

[0003] However, the current routine operation and management of expressways across China is relatively extensive, and this massive amount of data is not fully mined and utilized. The application of big data analysis technology is limited to exploring solutions to hot issues in specific segments. The informatization of expressways is plagued by information silos and inefficient management.

[0004] The massive amount of data generated presents significant challenges for its storage and analysis. Traditional storage methods struggle to cope with the demands of massive data volumes and are unable to efficiently and scientifically store this large amount of data. Furthermore, the heterogeneous nature of the data complicates storage and management. Efficiently and scientifically storing this large amount of data, while enabling rapid access and analysis to better serve transportation, has become a crucial issue. Summary of the Invention

[0005] In view of this, the present invention proposes a unified storage and management method for highway data middle platform data to solve the problem in the existing technology that multi-source heterogeneous data is difficult to efficiently collect, process and store.

[0006] The specific technical solutions of the present invention are as follows:

[0007] A method for unified storage and management of highway data in a middle platform, comprising:

[0008] Step S1: Build a dedicated data collection channel for the multi-source heterogeneous data generated by highway operations; design standardized data templates to connect to various data sources, use edge computing nodes for protocol conversion and semantic cleaning, and generate standardized metadata packages with timestamps;

[0009] Step S2: Establish a dynamic sharding storage matrix to slice and store data in three dimensions: time, space, and business;

[0010] Step S3: Design an event-driven data association engine to establish a spatiotemporal mapping relationship between device status data and video stream data based on a dynamic weight allocation algorithm; and implement automatic association and storage of abnormal event data packets and related data through an event ID chain binding mechanism;

[0011] Step S4: deploy a multimodal storage cluster, use different storage units to store different data, and implement data lineage tracking across storage units through distributed indexing services;

[0012] Step S5: Implement dynamic vectorized permission management to determine the access rights level of business roles to data areas, establish a fine-grained data access control model, and achieve real-time verification and violation interception;

[0013] Step S6: Build a service-oriented data interface gateway to support concurrent data access for multiple types of applications.

[0014] Furthermore, in step S2, in the time dimension, a sliding time window mechanism is used to implement data update, divide data according to set time intervals, dynamically adjust the time window size and sliding step size, and aggregate and compress the data within the time window; in the spatial dimension, the highway space is divided into raster storage units based on the section pile number coordinate system, and the data is stored according to its geographic location information; in the business dimension, the data is separated into a basic data layer and an event feature layer to form a two-layer storage structure.

[0015] Furthermore, in step S2, in the time dimension, let W be the time window, t be the time, is the standardized metadata package of the i-th data source at time t, is the aggregation result of the data in the time window W, then

[0016] ,

[0017] in, and are the start and end time of time window W respectively; is the weight of the i-th data source at time t.

[0018] Furthermore, in step S2, in terms of spatial dimension, the road section stake coordinate system is set to (x, y), and the highway space is divided into M×N grid cells, and the coordinates of each grid cell are ( , )(1≤ ≤M,1≤ ≤N); Assume that the geographical location information of data D is , then the grid cell coordinates where the data should be stored ( , )satisfy: , ,in 、 is the minimum coordinate of the highway space, 、 is the side length of the grid cell in the x and y directions.

[0019] Furthermore, in step S3, when the event-driven data association engine performs data association storage, it first performs data preprocessing to extract key information from device status data and video stream data, and then establishes a spatiotemporal mapping relationship based on a dynamic weight allocation algorithm. The system monitors the operating status of the highway in real time, generates a unique event ID when an abnormal event is detected, and performs chain binding storage using the event ID as a clue.

[0020] Furthermore, in step S3, the dynamic weight allocation algorithm comprehensively considers the highway operation scene, road section traffic flow and equipment status factors, dynamically adjusts the weights of equipment status data and video stream data, and establishes a spatiotemporal mapping relationship; let S be the highway operation scene, Q be the road section traffic flow, is the device status, 、 are the weights of device status data and video stream data respectively, then the dynamic weight allocation algorithm formula is:

[0021]

[0022] in, and are the impact coefficients of the highway operation scenario S on the weights of device status data and video stream data, respectively; and Device status and the characteristic function of traffic flow Q; and They are traffic flow Q and equipment status Correction factor for weights.

[0023] Furthermore, in step S4, in the multimodal storage cluster, different storage units are used to store different data, including a column storage unit for storing structured business data, an object storage unit for storing video stream data, and a time series database for storing equipment status monitoring data; wherein, the column storage unit stores structured business data such as charging transaction records and equipment management information, the object storage unit uses block storage and redundant backup to store video stream data, and the time series database stores equipment status monitoring data to achieve monitoring and predictive maintenance of equipment operating conditions; the distributed index service records data storage locations and associations to achieve data lineage tracking across storage units.

[0024] Furthermore, in step S5, dynamic vectorized permission management is implemented, and the key features of the business role are analyzed to form a business role feature vector; the data space is divided into different data areas, and a corresponding data space coordinate vector is generated for each data area; and the access permission level of the business role to the data area is determined by calculating the cosine similarity between the two.

[0025] Furthermore, in step S5, it is assumed that the business role R has n key features, which are , then the business role feature vector ,in It is a quantitative representation of various information of business roles;

[0026] Set data area There are m key information, which are , then the data space coordinate vector ,in It is a quantitative representation of various information in the data space;

[0027] Business role feature vector and the data space coordinate vector Cosine similarity of , determine the access permission level based on similarity , where f is a monotonically increasing function that maps similarity to different access rights levels.

[0028] Furthermore, in step S6, a service-oriented data interface gateway is constructed to provide real-time data push services through the HTTP / 2 protocol; video stream proxy services are provided through the WebSocket protocol to optimize video stream transmission; and seamless docking with IoT devices is achieved through the MQTT protocol conversion module. The service-oriented data interface gateway adopts load balancing technology and caching mechanism to reasonably allocate system resources, support concurrent data access of three types of applications: large-screen visualization system, mobile inspection terminal, and AI analysis engine, and perform security reinforcement.

[0029] The beneficial effects of the present invention are:

[0030] (1) By building multi-source heterogeneous data acquisition channels and standardized data templates, unified access and processing of five types of data sources, including video surveillance streams, event detection messages, charging transaction records, device status heartbeat packets, and infrastructure spatial coordinates, are achieved.

[0031] Edge computing nodes are used for real-time protocol conversion and semantic cleaning to generate standardized metadata packages with timestamps, providing high-quality basic data for subsequent storage and analysis.

[0032] (2) A dynamic sharding storage matrix based on the three dimensions of time, space, and business is established to achieve comprehensive and detailed sharding storage of data. A sliding time window mechanism is used in the time dimension to ensure the timeliness of data and optimize storage space. In the spatial dimension, raster storage units are divided according to the road section stake coordinate system to achieve a close integration of data and geographic location. In the business dimension, the basic data layer and the event feature layer are separated to improve the efficiency of data retrieval and analysis.

[0033] (3) Design an event-driven data association engine based on a dynamic weight allocation algorithm to establish a spatiotemporal mapping relationship between device status data and video stream data. Through the event ID chain binding mechanism, the abnormal event data packet and related data are automatically associated and stored, providing a complete data chain for subsequent analysis.

[0034] (4) Deploy a highly scalable multimodal storage cluster, using different storage units to store different data to meet the storage needs of a large number of different types of data. Columnar storage units, object storage units, and time series databases are used to store structured business data, video streaming data, and device status monitoring data, respectively, to improve storage and query efficiency. Distributed indexing services enable data lineage tracking across storage units, improving the transparency and efficiency of data management.

[0035] (5) Implement dynamic vectorized permission management, determining access rights levels by calculating the cosine similarity between the business role feature vector and the data space coordinate vector. Establish a fine-grained data access control model to accurately limit the access of different business roles to different data areas, preventing data leakage and illegal access.

[0036] (6) Build a real-time data push channel based on HTTP / 2, a WebSocket video streaming proxy service, and an MQTT IoT protocol conversion module to support concurrent data access for multiple applications. Optimize push strategies and video stream processing to ensure low-latency, high-bandwidth transmission and smooth data flow. Integrate load balancing, caching mechanisms, and security reinforcement measures to improve the performance and security of the interface gateway. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 This is a simple flow chart of the unified storage and management method for highway data middle platform data of the present invention. DETAILED DESCRIPTION

[0039] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] The present invention proposes a unified storage and management method for highway data middle platform, which includes: step S1, building a multi-source heterogeneous data acquisition channel, connecting five types of data sources such as video surveillance stream, event detection message, toll transaction record, equipment status heartbeat packet, and infrastructure spatial coordinate through standardized data template, using edge computing nodes to perform protocol conversion and semantic cleaning on real-time data stream, and generating standardized metadata packets with timestamps; step S2, establishing a dynamic sharding storage matrix, slicing and storing data from three dimensions of time, space, and business; step S3, designing an event-driven data association engine based on dynamic weights. The allocation algorithm establishes a spatiotemporal mapping relationship between device status data and video stream data; and through the event ID chain binding mechanism, it realizes the automatic associated storage of abnormal event data packets and related data; Step S4, deploys a multimodal storage cluster, uses different storage units to store different data, and realizes data lineage tracking across storage units through distributed indexing services; Step S5, implements dynamic vectorized permission management, determines the access permission level of business roles to data areas, establishes a fine-grained data access control model, and realizes real-time verification and violation interception; Step S6, builds a service-oriented data interface gateway to support concurrent data access for multiple types of applications.

[0041] In step S1, highway operation generates data of various types and formats. These data come from a wide range of sources and have varying structures, known as multi-source heterogeneous data. To effectively collect and process this data, a dedicated data collection channel must be built. Standardized data templates are used to connect to these multiple data sources. Edge computing nodes are then used to process these five data sources and generate standardized metadata packages.

[0042] Specifically, to ensure unified and standardized data integration from diverse data sources, a comprehensive and detailed standardized data template was designed based on the data structure, format, and semantics of five data sources: video surveillance streams, event detection messages, toll transaction records, device status heartbeat packets, and infrastructure spatial coordinates. This template should be extensible and adaptable to future data source types. For example, for video surveillance streams, the template specifies the video frame format, resolution, and timestamp recording method; for event detection messages, the template specifies the format for fields such as event type, occurrence time, and location information; for toll transaction records, the template specifies the recording format for fields such as transaction time, transaction amount, and vehicle model; for device status heartbeat packets, the template specifies the format for fields such as device ID, status code, and reporting time; and for infrastructure spatial coordinates, the template specifies the recording method for information such as road section stake number and geographic coordinates. Through this standardized data template, the five data sources—video surveillance streams, event detection messages, toll transaction records, device status heartbeat packets, and infrastructure spatial coordinates—are integrated into the data collection pipeline.

[0043] To ensure rapid and efficient collection and processing of real-time data streams from various data sources, edge computing nodes are deployed along highways at the front end of data collection. These nodes are close to data sources and can access data streams in real time. Furthermore, edge computing nodes possess powerful real-time processing capabilities, enabling efficient protocol conversion for the continuous flow of real-time data streams. Since different data sources may use different communication protocols, protocol conversion unifies these diverse formats into a standardized protocol that can be recognized and processed within the system. Furthermore, the nodes perform in-depth semantic cleansing to precisely remove noise, erroneous information, duplicate content, and data that does not conform to business logic, ensuring data accuracy and consistency. After protocol conversion and semantic cleansing, edge computing nodes add timestamps accurate to the second to the processed data, ensuring accurate tracing of data generation during subsequent storage and analysis. Finally, this carefully curated data is packaged into a standardized, timestamped metadata package. This package, containing the curated and tagged data, provides a high-quality foundation for subsequent data storage and analysis.

[0044] Regarding step S2, in highway data management, faced with massive and continuously growing data, the present invention proposes a solution to establish a dynamic sharding storage matrix, aiming to perform comprehensive and detailed slicing storage of data through three dimensions: time, space, and business.

[0045] First, in the temporal dimension, the present invention employs a sliding time window mechanism. The time window slides along the time axis, partitioning the data according to a set time interval (e.g., every second). As time passes, the time window continuously slides, new standardized metadata packets enter the storage matrix, and old data is processed according to the set rules, ensuring that the system can update data in seconds. This mechanism can flexibly adapt to real-time changes in highway data, thereby always storing the latest and valid information. The size of the time window and the sliding step size are dynamically adjusted based on the actual frequency of highway data changes and subsequent analysis requirements, ensuring data timeliness while optimizing storage space usage. Furthermore, data within the time window is rationally aggregated and compressed to further improve data storage efficiency.

[0046] Here, one embodiment is given, assuming W is the time window, t is the time, is the standardized metadata package of the i-th data source at time t, is the aggregation result of the data in the time window W, then

[0047] ,

[0048] in, and are the start and end time of time window W respectively; is the weight of the i-th data source at time t, which is dynamically adjusted according to the importance of the data and subsequent analysis requirements.

[0049] Secondly, in terms of spatial dimensions, this invention precisely divides the entire spatial extent of the highway into a number of uniformly sized gridded storage units based on the road section stake coordinate system. Each grid unit corresponds to a specific area on the highway, tightly integrating data storage with the highway's actual geographic location. When storing data, it is accurately stored in the corresponding gridded storage unit based on its corresponding geographic location, ensuring data accuracy and traceability. This facilitates the subsequent rapid location and query of specific road section data, allowing for quick access to data surrounding that section in the event of an accident, for example.

[0050] Here, one embodiment is given. Assume that the road section stake number coordinate system is (x, y), and the highway space is divided into M×N grid cells. The coordinates of each grid cell are ( , )(1≤ ≤M,1≤ ≤N). Assume that the geographical location information of data D is , then the grid cell coordinates where the data should be stored ( , )satisfy: , ,in 、 is the minimum coordinate of the highway space, 、 is the side length of the grid cell in the x and y directions.

[0051] Finally, in terms of the business dimension, the present invention separates the data into a basic data layer and an event feature layer to form a two-layer storage structure. The basic data layer stores various relatively fixed basic information related to the operation of the highway, such as basic parameters of the equipment, charging standards, basic attributes of road facilities, etc. This information usually does not change frequently, so it can be stored in a relatively stable storage medium. The event feature layer is specifically used to store data features closely related to various abnormal events or key events, such as vehicle speed, location, event type, etc. when a traffic accident occurs. This hierarchical storage structure can improve the efficiency of data retrieval and analysis, and facilitates the rapid acquisition of relevant data for different business needs. When an event needs to be analyzed, relevant data can be directly obtained from the event feature layer, and at the same time, combined with the information in the basic data layer for a more comprehensive analysis.

[0052] Regarding step S3, during highway operation, inherent connections exist between different types of data, particularly between equipment status data and video stream data, as well as between abnormal events and various related records. To better utilize these connections and improve the efficiency and accuracy of abnormal event processing, the present invention designs an event-driven data association engine. The core of this engine is to establish a spatiotemporal mapping relationship between equipment status data and video stream data based on a dynamic weight allocation algorithm. Through an event ID chain binding mechanism, it automatically associates and stores abnormal event data packets with corresponding video clips, equipment logs, maintenance records, and other related data.

[0053] Based on a dynamic weighting algorithm, the weights of device status data and video stream data in the spatiotemporal dimensions are dynamically adjusted, taking into account various factors, including different highway operating scenarios (such as peak and off-peak hours, and weather conditions), traffic flow in different sections, and equipment status. This approach establishes a precise spatiotemporal mapping relationship. In other words, the algorithm continuously optimizes the dynamic weighting algorithm by thoroughly studying the inherent connections and mutual influences between device status data and video stream data in different scenarios. For example, during periods of high traffic volume, video stream data may be more important for analyzing traffic conditions, so it is assigned a higher weight. During periods of equipment maintenance, the weight of device status data is increased accordingly. This dynamic weighting method more accurately reflects the degree of correlation between data.

[0054] Here, one of the embodiments is given. Let S be the highway operation scene, Q be the traffic flow of the road section, is the device status, 、 are the weights of device status data and video stream data respectively, then the dynamic weight allocation algorithm formula is:

[0055]

[0056] in, and are the impact coefficients of the highway operation scenario S on the weights of device status data and video stream data, respectively; and Device status and the characteristic function of traffic flow Q; and They are traffic flow Q and equipment status Correction factor for weights.

[0057] At the same time, the present invention also designs an event ID chain binding mechanism to establish an index and association relationship for event IDs. When the system detects the occurrence of an abnormal event, it will immediately generate a unique event ID for the abnormal event. Through this event ID, the abnormal event data packet is quickly and accurately chain-bound with the corresponding video clips, equipment logs, maintenance records and other related data, realizing automatic associated storage. Automatic associated storage of data related to specific events is achieved through event identifier chain association. This mechanism ensures the accuracy of the chronological order and logical relationships of the data, providing a complete data chain for subsequent comprehensive analysis of abnormal events. In this way, when subsequently analyzing abnormal events, all data related to the event can be easily obtained, and the background, process and possible causes of the event can be fully understood.

[0058] In terms of implementation steps, the present invention first performs data preprocessing to extract key information from device status data and video stream data, providing a foundation for the subsequent establishment of spatiotemporal mapping relationships. Next, a spatiotemporal mapping relationship is established based on a dynamic weight allocation algorithm, accurately reflecting the degree of correlation between data based on the assigned weights. The system then monitors the operating status of the highway in real time and immediately generates a unique event ID when an abnormal event is detected. Finally, using the event ID as a clue, the abnormal event data packet is automatically associated and stored with the video clip at the corresponding moment, the log records of the relevant equipment before and after the event, and the relevant maintenance records through chain binding.

[0059] In step S4, to meet the storage needs of the large amount of different types of data generated during highway operation, the present invention deploys a highly scalable multimodal storage cluster, using different storage units to store different data. For structured business data, column-based storage units with high-performance query and analysis capabilities are selected, and storage resources are rationally allocated based on the size and growth trend of the data. For video streaming data, object storage units that can efficiently manage and store large amounts of unstructured data are selected, and storage strategies are optimized to improve data read and write performance. For equipment status monitoring data, a dedicated time series database is used to ensure accurate storage and analysis of time-series data. Data lineage tracking across storage units is achieved through distributed indexing services.

[0060] The cluster uses column-based storage units to specifically store structured business data, such as billing transaction records and equipment management information. Column-based storage is characterized by storing data in columns. When performing queries and analyses, the required column data can be quickly located and read, greatly improving query efficiency and making it suitable for processing complex business queries. At the same time, object storage units are used to store video stream data. Video stream data is typically large in volume and unstructured. The object storage architecture can efficiently manage and store this type of data, providing stable read and write performance. To ensure smooth reading and storage of video data, the present invention optimizes the storage strategy of the object storage unit, such as using block storage and redundant backup technologies to further improve data reliability and security. In addition, a time series database is specifically used to store equipment status monitoring data. This type of data is arranged in chronological order. The time series database is optimized for time series data and can efficiently store, query, and analyze changes in equipment status over time. Through real-time storage and historical tracing of equipment status data, comprehensive monitoring and predictive maintenance of equipment operating conditions can be achieved.

[0061] To effectively manage and trace data across different storage units, a unified index structure is established across these units, deploying a distributed index service. This service acts like a data map, recording the storage location, data identifier, and relationships of each piece of data across different storage units. This service enables cross-storage data tracing, understanding the entire data generation and storage process. This allows for clear and traceable data sources and flows, facilitating data management, maintenance, quality control, and troubleshooting. The distributed index service should be scalable and fault-tolerant, adapting to the ever-expanding storage cluster size and data volume. Users can easily query and trace data sources, processing steps, and storage locations, improving data management efficiency and transparency. During data storage and query processes, the distributed index service updates and maintains index information in real time, ensuring accurate and timely data traceability.

[0062] For step S5, dynamic vectorized permission management is implemented. First, a detailed analysis is conducted on the various business roles involved in the highway data center, including management personnel, operation and maintenance personnel, data analysts, etc., and their key features are extracted to form an accurate business role feature vector. This vector contains information such as the scope of responsibilities, work authority, and operation history of the business role. At the same time, the data space is divided into different data areas according to factors such as the jurisdiction of the road section and the geographical fence of the service area, and a corresponding data space coordinate vector is generated for each data area. These vectors contain key information such as the geographical location and business attributes of the data area, so as to effectively match and calculate with the business role feature vector. Then, by calculating the cosine similarity between the business role feature vector and the data space coordinate vector, the degree of match between the business role and the data area is measured, thereby determining the level of access permission for the business role to the data area. The higher the similarity, the higher the degree of match between the business role and the data area, and therefore the higher the level of access permission.

[0063] Based on the calculated access rights level, a fine-grained data access control model is established based on road section jurisdictions and service area geofencing. This precisely restricts access to different data areas by different business roles, ensuring that different personnel can only access the data areas required and authorized for their work, preventing data leakage and unauthorized access. For example, managers of a specific road section can only access data for that section and its associated service areas, and cannot access data for other sections. This ensures that data access rights align with the responsibilities and needs of business roles, thus ensuring data security. The system also implements real-time verification during the data access process. Each access is rigorously verified by the system to ensure compliance with permission management requirements. Any illegal access will be promptly intercepted and recorded by the system for subsequent audit and resolution.

[0064] Here, we give one of the embodiments. Assume that the key features of business role R are n, which are , then the business role feature vector ,in It can be a quantitative representation of information such as scope of responsibilities, work authority, and operating history.

[0065] Set data area There are m key information, which are , then the data space coordinate vector ,in It can be a quantitative representation of information such as geographic location and business attributes.

[0066] Business role feature vector and the data space coordinate vector Cosine similarity of , determine the access permission level based on similarity , where f is a monotonically increasing function that maps similarity to different access rights levels.

[0067] For step S6, the highway data middle platform needs to provide data support for a variety of different applications. These applications have different ways of accessing and requiring data, so it is necessary to build a unified interface gateway to achieve efficient sharing and distribution of data. The present invention constructs a service-oriented data interface gateway, which provides a real-time data push channel based on HTTP / 2, and uses the high performance characteristics of the HTTP / 2 protocol to ensure that real-time data can be pushed to the required application system in a low-latency, high-bandwidth manner. At the same time, a WebSocket video stream proxy service is provided to optimize the transmission and processing of video streams and ensure the smoothness of video data. In addition, it is also equipped with an MQTT Internet of Things protocol conversion module to facilitate communication with various Internet of Things devices. Through these functions, the service-oriented data interface gateway can support concurrent data access for three types of applications: large-screen visualization systems, mobile inspection terminals, and AI analysis engines, providing comprehensive data support for highway monitoring, management, and decision-making.

[0068] The real-time data push channel based on HTTP / 2 fully leverages the efficiency of the HTTP / 2 protocol. Through features such as multiplexing and header compression, it provides fast and stable data push services for applications that require real-time data. For example, it can provide real-time traffic flow, accident information, and other information to the large-screen visualization system of a traffic control center, enabling it to display the latest road conditions. Furthermore, push strategies are optimized based on the specific real-time data requirements of different applications, such as setting different push frequencies and data formats.

[0069] The gateway also provides a WebSocket video stream proxy service. Leveraging the bidirectional communication capabilities of the WebSocket protocol, this service provides stable and smooth video proxying for applications that require access to video stream data. During the proxy process, the gateway optimizes the video stream, adjusting the video resolution and controlling the bit rate to ensure that applications such as mobile inspection terminals can obtain clear and smooth video surveillance images, meeting the needs of real-time on-site viewing.

[0070] The service-oriented data interface gateway also integrates an MQTT IoT protocol conversion module. This module converts the different protocols used by various IoT devices into a unified standard protocol within the system, enabling seamless integration with various IoT devices. Through the MQTT protocol's publish / subscribe messaging mechanism, the gateway efficiently collects and transmits device data, supporting functions such as device status monitoring and remote control, and meeting the processing requirements of AI analysis engines for IoT device data.

[0071] To support concurrent data access for three types of applications: large-screen visualization systems, mobile inspection terminals, and AI analysis engines, the service-oriented data interface gateway has been optimized in terms of architecture and performance. By employing load balancing technologies and caching mechanisms, the gateway rationally allocates system resources, ensuring that even in high-concurrency scenarios, each application can quickly and stably access the required data, providing strong data support for highway monitoring, management, and decision-making. Furthermore, the interface gateway has been reinforced with security measures such as identity authentication and data encryption to ensure the security and integrity of data transmission.

[0072] The beneficial effects of the present invention are:

[0073] (1) The constructed multi-source heterogeneous data collection channel and standardized data template effectively solved the problem of difficulty in uniformly collecting and processing multi-type and multi-format data during highway operation.

[0074] (2) The sliding time window mechanism in the time dimension ensures that the system can update data in seconds and flexibly adapt to real-time data changes; the spatial dimension rasterizes the highway space to achieve a close integration of data and geographic location, facilitating the rapid location and query of data on specific road sections; the two-layer storage structure in the business dimension separates basic data and event feature data, improving data retrieval and analysis efficiency and meeting different business needs.

[0075] (3) The event-driven data association engine uses a dynamic weight distribution algorithm and an event ID chain binding mechanism to effectively explore the intrinsic connections between different types of data.

[0076] (4) Column-based storage improves query efficiency for structured business data, object-based storage efficiently manages video stream data, and time-series databases accurately store and analyze device status monitoring data. Distributed indexing services enable data lineage tracking across storage units, facilitating data management, maintenance, quality control, and troubleshooting.

[0077] (5) Dynamic vectorized permission management accurately limits the access of different business roles to different data areas, preventing data leakage and illegal access. The system's real-time verification function can intercept and record illegal access to ensure data security.

[0078] (6) The constructed service-oriented data interface gateway realizes the efficient sharing and distribution of data.

[0079] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for unified storage and management of highway data in a middle platform, characterized in that: include: Step S1: building a dedicated data acquisition channel for the multi-source heterogeneous data generated by highway operation; Design standardized data templates to connect to various data sources, use edge computing nodes for protocol conversion and semantic cleaning, and generate standardized metadata packages with timestamps; Step S2: Establish a dynamic sharding storage matrix to slice and store data in three dimensions: time, space, and business. In the time dimension, a sliding time window mechanism is used to update data, partition data according to set time intervals, dynamically adjust the time window size and sliding step size, and aggregate and compress the data within the time window. Step S3: Design an event-driven data association engine to establish a spatiotemporal mapping relationship between device status data and video stream data based on a dynamic weight allocation algorithm; and implement automatic association and storage of abnormal event data packets and related data through an event ID chain binding mechanism; Step S4: deploy a multimodal storage cluster, use different storage units to store different data, and implement data lineage tracking across storage units through distributed indexing services; Step S5: Implement dynamic vectorized permission management to determine the access rights level of business roles to data areas, establish a fine-grained data access control model, and achieve real-time verification and violation interception; Step S6: Build a service-oriented data interface gateway to support concurrent data access for multiple types of applications.

2. The method for unified storage and management of highway data in the middle platform according to claim 1, characterized in that: In step S2, in the spatial dimension, the highway space is divided into rasterized storage units according to the road section pile number coordinate system, and the data is stored according to the geographic location information; in the business dimension, the data is separated into a basic data layer and an event feature layer to form a two-layer storage structure.

3. The method for unified storage and management of highway data in the middle platform according to claim 2, characterized in that: In step S2, in the time dimension, let W be the time window, t be the time, P i (t) is the standardized metadata package of the i-th data source at time t, A W is the aggregation result of the data in the time window W, then Among them, t start and t end are the start and end time of the time window W; α i (t) is the weight of the i-th data source at time t.

4. The method for unified storage and management of highway data in the middle platform according to claim 2, characterized in that: In step S2, in terms of spatial dimension, the section stake coordinate system is set as (x, y), and the highway space is divided into M×N grid cells, and the coordinates of each grid cell are (x0, y0) (1≤x0≤M, 1≤y0≤N); ​​the geographical location information of data D is set as (x D ,y D ), the grid cell coordinates (x0, y0) where the data should be stored satisfy: where x min 、y min is the minimum coordinate of the highway space, Δx and Δy are the side lengths of the grid unit in the x and y directions.

5. The method for unified storage and management of highway data in the middle platform according to claim 1, characterized in that: In step S3, when the event-driven data association engine performs data association storage, it first performs data preprocessing to extract key information from device status data and video stream data, and then establishes a spatiotemporal mapping relationship based on a dynamic weight allocation algorithm. The system monitors the operating status of the highway in real time, generates a unique event ID when an abnormal event is detected, and performs chain binding storage using the event ID as a clue.

6. The method for unified storage and management of highway data in the middle platform according to claim 5, characterized in that: In step S3, the dynamic weight allocation algorithm comprehensively considers the highway operation scene, road section traffic flow and equipment status factors, dynamically adjusts the weights of equipment status data and video stream data, and establishes a spatiotemporal mapping relationship; let S be the highway operation scene, Q be the road section traffic flow, E s is the device state, ω1 and ω2 are the weights of device state data and video stream data respectively, then the dynamic weight allocation algorithm formula is: Among them, β1(S) and β2(S) are the influence coefficients of highway operation scenario S on the weight of equipment status data and video stream data respectively; f1(E s ) and f2(Q) are the device states E s and the characteristic function of traffic flow Q; γ1(Q) and γ2(E s ) are the traffic flow Q and the equipment status E respectively s Correction factor for weights.

7. The method for unified storage and management of highway data in the middle platform according to claim 1, characterized in that: In step S4, in the multimodal storage cluster, different storage units are used to store different data, including a column storage unit for storing structured business data, an object storage unit for storing video stream data, and a time series database for storing equipment status monitoring data; wherein, the column storage unit stores structured business data such as charging transaction records and equipment management information, the object storage unit uses block storage and redundant backup to store video stream data, and the time series database stores equipment status monitoring data to achieve monitoring and predictive maintenance of equipment operating conditions; the distributed index service records data storage locations and associations to achieve data lineage tracking across storage units.

8. The method for unified storage and management of highway data in the middle platform according to claim 1, characterized in that: In step S5, dynamic vectorized permission management is implemented, key features of business roles are analyzed to form business role feature vectors; the data space is divided into different data areas, and a corresponding data space coordinate vector is generated for each data area; and the access permission level of the business role to the data area is determined by calculating the cosine similarity between the two.

9. The method for unified storage and management of highway data in the middle platform according to claim 8, characterized in that: In step S5, it is assumed that the business role R has n key features, namely r1, r2, ..., r n , then the business role feature vector where r i (i=1, 2, ..., n) is a quantitative representation of various information of business roles; Assume data area D a There are m key information, namely d1, d2, ..., d m , then the data space coordinate vector where d i (i=1, 2, ..., m) is the quantitative representation of various information in the data space; Business role feature vector and the data space coordinate vector Cosine similarity of Determine access rights based on similarity Where f is a monotonically increasing function that maps similarity to different access rights levels.

10. The highway data middle platform unified storage and management method according to claim 1, characterized in that: In step S6, a service-oriented data interface gateway is constructed to provide real-time data push services through the HTTP / 2 protocol; a video stream proxy service is provided through the WebSocket protocol to optimize video stream transmission; Through the MQTT protocol conversion module, seamless connection with IoT devices is achieved; the service-oriented data interface gateway adopts load balancing technology and caching mechanism to rationally allocate system resources, support concurrent data access of three types of applications: large-screen visualization system, mobile inspection terminal, and AI analysis engine, and perform security reinforcement.

Images