Unified storage management method for data in expressway data

Abstract

The invention discloses a unified storage management method for station data in expressway data, which comprises the following steps of: S1, constructing a multi-source heterogeneous data acquisition channel, and processing various data sources by using edge computing nodes to generate a standardized metadata packet; s2, establishing a dynamic fragment storage matrix, and performing three-dimensional slicing on the data from time, space and business dimensions; s3, designing an event-driven data association engine, establishing a space-time mapping relation, and realizing automatic association storage through event ID chain type binding; s4, deploying a multi-modal storage cluster, storing different data by using different storage units, and tracking data consanguinity through a distributed index service; s5, implementing dynamic vectorization authority management, calculating an access authority level according to cosine similarity, and establishing a fine-grained data access control model; and step S6, constructing a service data interface gateway. The method has remarkable advantages in the aspects of highway data acquisition, storage, association, management, sharing and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of highway data management and storage, and in particular to a method for unified storage management of highway data in a data center. Background Art

[0002] After years of development and construction, the mileage of my country's expressways has continued to increase. In the daily operation process, various management systems generate fast, large, diverse and multi-source heterogeneous data. With the increase in the mileage of expressways, the generation of big data is also increasing rapidly, and the amount of data that needs to be processed is becoming increasingly huge.

[0003] However, the daily operation and management of highways in various provinces in China is relatively extensive, and these massive data have not been fully mined and utilized. The application of big data analysis technology is limited to the exploration of solutions to hot issues in specific segments, and the informatization construction in the highway field has the problems of widespread information islands and low management efficiency.

[0004] The large-scale generation of data has brought huge challenges to data storage and analysis. On the one hand, traditional storage methods are difficult to cope with the storage needs of massive data, and cannot store these big data efficiently and scientifically; on the other hand, the heterogeneity of data also increases the difficulty of storage and management. How to efficiently and scientifically store these big data, and to quickly access and analyze them in order to better serve transportation, has become a crucial issue. Summary of the invention

[0005] In view of this, the present invention proposes a unified storage management method for highway data center data to solve the problem in the prior art that multi-source heterogeneous data is difficult to efficiently collect, process and store.

[0006] The specific technical solutions of the present invention are as follows:

[0007] A method for unified storage and management of highway data in a data center, comprising:

[0008] Step S1: construct a dedicated data collection channel for multi-source heterogeneous data generated by highway operation; design standardized data templates to connect various data sources, use edge computing nodes to perform protocol conversion and semantic cleaning, and generate standardized metadata packages with timestamps;

[0009] Step S2, establish a dynamic sharding storage matrix to slice and store data from three dimensions: time, space, and business;

[0010] Step S3, designing an event-driven data association engine, establishing a spatiotemporal mapping relationship between device status data and video stream data based on a dynamic weight allocation algorithm; and realizing automatic association storage of abnormal event data packets and related data through an event ID chain binding mechanism;

[0011] Step S4, deploying a multimodal storage cluster, using different storage units to store different data, and implementing data lineage tracking across storage units through a distributed index service;

[0012] Step S5, implement dynamic vectorized permission management, determine the level of access rights of business roles to data areas, establish a fine-grained data access control model, and achieve real-time verification and violation interception;

[0013] Step S6, construct a service-oriented data interface gateway to support concurrent data access for multiple types of applications.

[0014] Furthermore, in step S2, in the time dimension, a sliding time window mechanism is used to implement data update, divide data according to set time intervals, dynamically adjust the time window size and sliding step size, and aggregate and compress data within the time window; in the spatial dimension, the highway space is divided into raster storage units according to the section pile number coordinate system, and the data is stored according to its geographic location information; in the business dimension, the data is separated into a basic data layer and an event feature layer to form a two-layer storage structure.

[0015] Furthermore, in step S2, in the time dimension, let W be the time window, t be the time, is the standardized metadata package of the i-th data source at time t, is the aggregation result of the data in the time window W, then

[0016] ,

[0017] in, and are the start and end time of time window W respectively; is the weight of the i-th data source at time t.

[0018] Furthermore, in step S2, in the spatial dimension, the road section stake number coordinate system is set to (x, y), and the highway space is divided into M×N grid cells, and the coordinates of each grid cell are ( , )(1≤ ≤M,1≤ ≤N); Assume that the geographical location information of data D is , then the grid cell coordinates where the data should be stored ( , )satisfy: , ,in , is the minimum coordinate of the highway space, , is the side length of the grid cell in the x and y directions.

[0019] Furthermore, in step S3, when the event-driven data association engine performs data association storage, it first performs data preprocessing to extract key information from device status data and video stream data, and then establishes a spatiotemporal mapping relationship based on a dynamic weight allocation algorithm. The system monitors the operation status of the highway in real time, generates a unique event ID when an abnormal event is detected, and performs chain binding storage using the event ID as a clue.

[0020] Furthermore, in step S3, the dynamic weight allocation algorithm comprehensively considers the highway operation scenario, road section traffic flow and device status factors, dynamically adjusts the weights of device status data and video stream data, and establishes a spatiotemporal mapping relationship; let S be the highway operation scenario, Q be the road section traffic flow, is the device status, , are the weights of device status data and video stream data respectively, then the dynamic weight allocation algorithm formula is:

[0021]

[0022] in, and are the influence coefficients of the highway operation scenario S on the weights of the equipment status data and the video stream data, respectively; and The device status and the characteristic function of traffic flow Q; and are the traffic flow Q and the equipment status respectively Correction factor for weights.

[0023] Furthermore, in step S4, in the multimodal storage cluster, different storage units are used to store different data, including a column storage unit for storing structured business data, an object storage unit for storing video stream data, and a time series database for storing equipment status monitoring data; wherein, the column storage unit stores structured business data such as charging transaction records and equipment management information, the object storage unit uses block storage and redundant backup to store video stream data, and the time series database stores equipment status monitoring data to realize monitoring and predictive maintenance of equipment operating conditions; the distributed index service records data storage locations and associations to realize data lineage tracking across storage units.

[0024] Furthermore, in step S5, dynamic vectorized permission management is implemented, and the key features of the business role are analyzed to form a business role feature vector; the data space is divided into different data areas, and a corresponding data space coordinate vector is generated for each data area; and the access permission level of the business role to the data area is determined by calculating the cosine similarity between the two.

[0025] Further, in step S5, it is assumed that the key features of the business role R are n, which are , then the business role feature vector ,in It is a quantitative representation of various information of business roles;

[0026] Set data area There are m key information, which are , then the data space coordinate vector ,in It is a quantitative representation of various information in the data space;

[0027] Business Role Characteristic Vector With the data space coordinate vector The cosine similarity of , determine the access permission level based on similarity , where f is a monotonically increasing function that maps similarities to different levels of access rights.

[0028] Furthermore, in step S6, a service-oriented data interface gateway is constructed to provide real-time data push service through the HTTP / 2 protocol; a video stream proxy service is provided through the WebSocket protocol to optimize video stream transmission; and a seamless connection with IoT devices is achieved through the MQTT protocol conversion module. The service-oriented data interface gateway uses load balancing technology and caching mechanism to reasonably allocate system resources, support concurrent data access of three types of applications: large-screen visualization system, mobile inspection terminal, and AI analysis engine, and perform security reinforcement.

[0029] The beneficial effects of the present invention are:

[0030] (1) By building multi-source heterogeneous data collection channels and standardized data templates, unified access and processing of five types of data sources are achieved: video surveillance streams, event detection messages, charging transaction records, device status heartbeat packets, and infrastructure spatial coordinates.

[0031] Edge computing nodes are used to perform real-time protocol conversion and semantic cleaning to generate standardized metadata packages with timestamps, providing high-quality basic data for subsequent storage and analysis.

[0032] (2) A dynamic sharding storage matrix in the three dimensions of time, space, and business is established to achieve comprehensive and detailed sharding storage of data. In the time dimension, a sliding time window mechanism is used to ensure the timeliness of data and the optimization of storage space. In the spatial dimension, raster storage units are divided according to the road section stake coordinate system to achieve a close integration of data and geographic location. In the business dimension, the basic data layer and the event feature layer are separated to improve the efficiency of data retrieval and analysis.

[0033] (3) Design an event-driven data association engine based on a dynamic weight allocation algorithm to establish a spatiotemporal mapping relationship between device status data and video stream data. Through the event ID chain binding mechanism, the abnormal event data packet and related data are automatically associated and stored, providing a complete data chain for subsequent analysis.

[0034] (4) Deploy a highly scalable multimodal storage cluster and use different storage units to store different data to meet the storage needs of a large number of different types of data. Column storage units, object storage units, and time series databases are used to store structured business data, video streaming data, and device status monitoring data, respectively, to improve storage and query efficiency. Distributed indexing services enable data lineage tracking across storage units, improving the transparency and efficiency of data management.

[0035] (5) Implement dynamic vectorized permission management, and determine the access permission level by calculating the cosine similarity between the business role feature vector and the data space coordinate vector. Establish a fine-grained data access control model to accurately limit the access of different business roles to different data areas, and prevent data leakage and illegal access.

[0036] (6) Build a real-time data push channel based on HTTP / 2, a WebSocket video stream proxy service, and an MQTT IoT protocol conversion module to support concurrent data access for multiple applications. Optimize push strategies and video stream processing to ensure low latency, high bandwidth transmission, and smoothness of data. Integrate load balancing, caching mechanisms, and security reinforcement measures to improve the performance and security of the interface gateway. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0038] Figure 1 It is a simple flow chart of the unified storage management method of highway data middle platform of the present invention. DETAILED DESCRIPTION

[0039] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] The present invention proposes a unified storage management method for highway data middle platform, which includes: step S1, constructing a multi-source heterogeneous data acquisition channel, connecting five types of data sources including video surveillance stream, event detection message, toll transaction record, equipment status heartbeat packet, and infrastructure space coordinate through standardized data template, using edge computing nodes to perform protocol conversion and semantic cleaning on real-time data stream, and generating standardized metadata packets with timestamps; step S2, establishing a dynamic sharding storage matrix, slicing and storing data from three dimensions of time, space, and business; step S3, designing an event-driven data association engine, based on dynamic weights The allocation algorithm establishes the spatiotemporal mapping relationship between device status data and video stream data; and through the event ID chain binding mechanism, the automatic associated storage of abnormal event data packets and related data is realized; step S4, deploy a multimodal storage cluster, use different storage units to store different data, and realize data lineage tracking across storage units through distributed indexing services; step S5, implement dynamic vectorized permission management, determine the access permission level of business roles to data areas, establish a fine-grained data access control model, and realize real-time verification and violation interception; step S6, build a service-oriented data interface gateway to support concurrent data access for multiple types of applications.

[0041] For step S1, during the operation of the highway, data of various types and formats are generated. These data come from a wide range of sources and have different structures, which are called multi-source heterogeneous data. In order to effectively collect and process these data, it is necessary to build a dedicated data collection channel, connect multiple data sources with the help of standardized data templates, and use edge computing nodes to process five types of data sources to generate standardized metadata packages.

[0042] Specifically, in order to ensure that data from different data sources can be uniformly and standardizedly accessed to the system, the characteristics of five types of data sources, namely video surveillance stream, event detection message, toll transaction record, equipment status heartbeat packet, and infrastructure spatial coordinates, are analyzed in depth, and a set of comprehensive and detailed standardized data templates are designed. This template should have good scalability and be able to adapt to the types of data sources that may be added in the future. For example, for video surveillance stream, the format, resolution, and timestamp recording method of the video frame are specified; for event detection message, the format of the event type, occurrence time, and location information fields are specified; for toll transaction record, the recording format of the transaction time, transaction amount, and vehicle model fields is specified; for equipment status heartbeat packet, the format of the equipment ID, status code, and reporting time fields is specified; for infrastructure spatial coordinates, the recording method of information such as road section pile number and geographic coordinates is specified. Through this standardized data template, the five types of data sources, namely video surveillance stream, event detection message, toll transaction record, equipment status heartbeat packet, and infrastructure spatial coordinates, are accessed to the data collection channel.

[0043] In order to ensure the rapid and effective collection and processing of real-time data streams from various data sources, edge computing nodes are deployed at the front end of data collection along the highway. These nodes are close to the data source and can obtain data streams in real time. In addition, the edge computing nodes have strong real-time processing capabilities and can efficiently convert the protocols of the continuous real-time data streams. Because different data sources may use different communication protocols, these protocols in different formats are converted into standard protocols that can be recognized and processed within the system through protocol conversion. At the same time, the nodes also perform in-depth semantic cleaning to accurately remove noise, error information, duplicate content, and data that does not conform to business logic in the data to ensure data accuracy and consistency. After protocol conversion and semantic cleaning, the edge computing nodes will add timestamps accurate to seconds to the processed data so that the generation time of the data can be accurately traced during subsequent storage and analysis. Finally, these carefully sorted data are encapsulated into standardized metadata packets with timestamps. This packet contains sorted and labeled data, providing a high-quality foundation for subsequent data storage and analysis.

[0044] For step S2, in highway data management, faced with massive and continuously growing data, the present invention proposes a solution to establish a dynamic slicing storage matrix, aiming to perform comprehensive and detailed slicing storage of data through three dimensions: time, space, and business.

[0045] First, in the time dimension, the present invention adopts a sliding time window mechanism. The time window slides on the time axis, and the data is divided according to the set time interval (such as every second). As time goes by, the time window continues to slide, new standardized metadata packets enter the storage matrix, and old data are processed according to the set rules to ensure that the system can update the data in seconds. This mechanism can flexibly adapt to the real-time changes of highway data, so as to always store the latest and valid information. The size of the time window and the sliding step are dynamically adjusted according to the actual change frequency of the highway data and the subsequent analysis requirements, which not only ensures the timeliness of the data, but also optimizes the occupancy of storage space. At the same time, the data in the time window is reasonably aggregated and compressed to further improve the efficiency of data storage.

[0046] Here, one embodiment is given, assuming W is the time window, t is the time, is the standardized metadata package of the i-th data source at time t, is the aggregation result of the data in the time window W, then

[0047] ,

[0048] in, and are the start and end time of time window W respectively; is the weight of the i-th data source at time t, which is dynamically adjusted according to the importance of the data and subsequent analysis requirements.

[0049] Secondly, in the spatial dimension, the present invention accurately divides the entire spatial range of the highway into a number of gridded storage units of uniform size based on the road section stake number coordinate system. Each grid unit corresponds to a specific area on the highway, realizing the close integration of data storage and the actual geographical location of the highway. When storing data, according to the geographical location information corresponding to the data, it is accurately stored in the corresponding gridded storage unit to ensure the accuracy and traceability of the data, and facilitate the subsequent rapid positioning and query of specific road section data. For example, when an accident occurs, the data around the road section can be quickly queried.

[0050] Here, one embodiment is given. Assume that the road section stake number coordinate system is (x, y), and the highway space is divided into M×N grid cells. The coordinates of each grid cell are ( , )(1≤ ≤M,1≤ ≤N). Assume that the geographical location information of data D is , then the grid cell coordinates where the data should be stored ( , )satisfy: , ,in , is the minimum coordinate of the highway space, , is the side length of the grid cell in the x and y directions.

[0051] Finally, in terms of the business dimension, the present invention separates the data into a basic data layer and an event feature layer to form a two-layer storage structure. The basic data layer stores various relatively fixed basic information related to the operation of highways, such as basic parameters of equipment, charging standards, basic attributes of road facilities, etc. This information usually does not change frequently, so it can be stored in a relatively stable storage medium. The event feature layer is specifically used to store data features closely related to various abnormal events or key events, such as vehicle speed, location, event type, etc. when a traffic accident occurs. This hierarchical storage structure can improve the efficiency of data retrieval and analysis, and facilitates the rapid acquisition of relevant data for different business needs. When an event needs to be analyzed, relevant data can be directly obtained from the event feature layer, and a more comprehensive analysis can be performed in combination with the information in the basic data layer.

[0052] For step S3, during the operation of the highway, there are inherent connections between different types of data, especially between equipment status data and video stream data, as well as between abnormal events and various related records. In order to better utilize the association between these data and improve the efficiency and accuracy of abnormal event processing, the present invention designs an event-driven data association engine. The core of the engine is to establish a spatiotemporal mapping relationship between equipment status data and video stream data based on a dynamic weight allocation algorithm, and through an event ID chain binding mechanism, realize the automatic association storage of abnormal event data packets and corresponding video clips, equipment logs, maintenance records and other related data.

[0053] Based on the dynamic weight allocation algorithm, the weights of the equipment status data and video stream data in the time and space dimensions are dynamically adjusted by comprehensively considering various factors such as different operation scenarios of highways (such as peak hours, off-peak hours, special weather, etc.), traffic flow in different sections, and equipment status, so as to establish an accurate time and space mapping relationship. In other words, the inherent connection and mutual influence of equipment status data and video stream data in different scenarios are deeply studied, and the dynamic weight allocation algorithm is continuously optimized. For example, during periods of heavy traffic, video stream data may be more important for analyzing traffic conditions, and a higher weight is assigned to video stream data at this time; and during equipment maintenance, the weight of equipment status data will be increased accordingly. By dynamically adjusting the weight in this way, the degree of correlation between data can be more accurately reflected.

[0054] Here, one of the embodiments is given. Let S be the highway operation scenario, Q be the traffic flow of the road section, is the device status, , are the weights of device status data and video stream data respectively, then the dynamic weight allocation algorithm formula is:

[0055]

[0056] in, and are the influence coefficients of the highway operation scenario S on the weights of the equipment status data and the video stream data, respectively; and The device status and the characteristic function of traffic flow Q; and are the traffic flow Q and the equipment status respectively Correction factor for weights.

[0057] At the same time, the present invention also designs an event ID chain binding mechanism to establish an index and association relationship of the event ID. When the system detects the occurrence of an abnormal event, it will immediately generate a unique event ID for the abnormal event. Through this event ID, the abnormal event data packet is quickly and accurately chain-bound with the corresponding video clips, equipment logs, maintenance records and other related data to achieve automatic associated storage. Automatic associated storage of specific event-related data is achieved through event identifier chain association. This mechanism ensures that the time sequence and logical relationship of the data are accurate and correct, and provides a complete data chain for subsequent comprehensive analysis of abnormal events. In this way, when analyzing abnormal events in the future, all data related to the event can be easily obtained, and the background, process and possible causes of the event can be fully understood.

[0058] In terms of implementation steps, the present invention first performs data preprocessing to extract key information from equipment status data and video stream data, providing a basis for the subsequent establishment of space-time mapping relationships. Next, a space-time mapping relationship is established based on a dynamic weight allocation algorithm, accurately reflecting the degree of association between data according to the assigned weights. Then, the system monitors the operating status of the highway in real time, and immediately generates a unique event ID when an abnormal event is detected. Finally, using the event ID as a clue, the abnormal event data packet is automatically associated and stored with the video clip at the corresponding moment, the log records of the relevant equipment before and after the event, and the maintenance records involved through chain binding.

[0059] For step S4, in order to meet the storage requirements of a large amount of different types of data generated during the operation of the highway, the present invention deploys a highly scalable multimodal storage cluster, using different storage units to store different data. For structured business data, select a column storage unit with high-performance query and analysis capabilities, and reasonably configure storage resources according to the size and growth trend of the data volume; for video streaming data, select an object storage unit that can efficiently manage and store a large amount of unstructured data, and optimize the storage strategy to improve the data read and write performance; for equipment status monitoring data, use a dedicated time series database to ensure that the data generated in time series can be accurately stored and analyzed; and realize data lineage tracking across storage units through distributed index services.

[0060] The cluster uses a column storage unit to store structured business data, such as charging transaction records, equipment management information, etc. The characteristic of column storage is that data is stored in columns. When querying and analyzing, the required column data can be quickly located and read, which greatly improves the query efficiency and is suitable for processing complex business queries. At the same time, the object storage unit is used to store video stream data. Video stream data usually has the characteristics of large data volume and unstructured. The object storage architecture can efficiently manage and store such data and provide stable read and write performance. In order to ensure the smooth reading and storage of video data, the present invention optimizes the storage strategy of the object storage unit, such as using block storage, redundant backup and other technologies to further improve the reliability and security of data. In addition, the time series database is specifically used to store equipment status monitoring data. This type of data has the characteristics of being arranged in chronological order. The time series database is optimized for time series data and can efficiently store, query and analyze the changes of equipment status over time. Through the real-time storage and historical tracing of equipment status data, comprehensive monitoring and predictive maintenance of equipment operation status can be achieved.

[0061] In order to achieve effective management and traceability of data in different storage units, a unified index structure is established between different storage units, and a distributed index service is deployed. This service is like a data map, which records the storage location, data identification, and association relationship of each data in different storage units with other related data. Through it, data lineage tracking across storage units can be achieved, that is, the entire process from data generation to storage can be understood, so that the source and flow of data can be clearly checked, which is convenient for data management, maintenance, quality control, and troubleshooting. The distributed index service should have good scalability and fault tolerance, and be able to adapt to the continuous expansion of the storage cluster scale and the continuous growth of data volume. Users can easily query and trace the source, processing process, and storage location of data, and improve the efficiency and transparency of data management. During the data storage and query process, the distributed index service updates and maintains the index information in real time to ensure the accuracy and timeliness of data traceability.

[0062] For step S5, dynamic vectorized permission management is implemented. First, a detailed analysis is conducted on various business roles involved in the highway data center, including management personnel, operation and maintenance personnel, data analysts, etc., and their key features are extracted to form an accurate business role feature vector. This vector contains information such as the scope of responsibilities, work authority, and operation history of the business role. At the same time, the data space is divided into different data areas according to factors such as the road section jurisdiction and the service area geographic fence, and a corresponding data space coordinate vector is generated for each data area. These vectors contain key information such as the geographical location and business attributes of the data area, so as to effectively match and calculate with the business role feature vector. Then, by calculating the cosine similarity between the business role feature vector and the data space coordinate vector, the matching degree between the business role and the data area is measured, thereby determining the access permission level of the business role to the data area. The higher the similarity, the higher the matching degree between the business role and the data area, and therefore the higher the access permission level.

[0063] According to the calculated access rights level, based on the road section jurisdiction and service area geographic fence, a fine-grained data access control model is established to accurately limit the access of different business roles to different data areas, ensuring that different personnel can only access the data areas required and authorized for their work, preventing data leakage and illegal access. For example, the manager of a certain road section can only access the data of the road section and its related service areas, but not the data of other roads. It ensures that the access rights of the data match the responsibilities and requirements of the business roles and guarantees the security of the data. During the data access process, the system also implements real-time verification. Each access will be strictly verified by the system to ensure compliance with the requirements of permission management. For illegal access behaviors, the system will intercept and record them in a timely manner for subsequent audit and processing.

[0064] Here, one embodiment is given. Assume that the key features of the business role R are n, which are , then the business role feature vector ,in It can be a quantitative representation of information such as scope of responsibilities, work authority, and operating history.

[0065] Set data area There are m key information, which are , then the data space coordinate vector ,in It can be a quantitative representation of information such as geographic location and business attributes.

[0066] Business Role Characteristic Vector With the data space coordinate vector The cosine similarity of , determine the access permission level based on similarity , where f is a monotonically increasing function that maps similarities to different levels of access rights.

[0067] For step S6, the highway data middle station needs to provide data support for a variety of different applications. These applications have different access methods and requirements for data, so it is necessary to build a unified interface gateway to achieve efficient sharing and distribution of data. The present invention constructs a service-oriented data interface gateway, which provides a real-time data push channel based on HTTP / 2, and utilizes the high performance characteristics of the HTTP / 2 protocol to ensure that real-time data can be pushed to the required application system in a low-latency, high-bandwidth manner. At the same time, a WebSocket video stream proxy service is provided to optimize the transmission and processing of video streams and ensure the fluency of video data. In addition, it is also equipped with an MQTT Internet of Things protocol conversion module to facilitate communication with various Internet of Things devices. Through these functions, the service-oriented data interface gateway can support concurrent data access for three types of applications: large-screen visualization system, mobile inspection terminal, and AI analysis engine, providing comprehensive data support for highway monitoring, management, and decision-making.

[0068] The real-time data push channel based on HTTP / 2 makes full use of the high efficiency of the HTTP / 2 protocol. Through multiplexing, header compression and other features, it provides fast and stable data push services for applications that need to obtain data in real time. For example, it provides real-time traffic flow, accidents and other information to the large-screen visualization system of the traffic command center, so that it can display the latest road conditions in a timely manner. In addition, according to the characteristics of different applications' demand for real-time data, the push strategy is optimized, such as setting different push frequencies and data formats.

[0069] At the same time, the gateway also provides WebSocket video stream proxy service. Through the two-way communication capability of the WebSocket protocol, this service provides a stable and smooth video proxy for applications that need to access video stream data. During the proxy process, the gateway optimizes the video stream, such as adjusting the video resolution and controlling the bit rate, to ensure that applications such as mobile inspection terminals can obtain clear and smooth video surveillance images, thereby meeting the needs of real-time viewing on site.

[0070] In addition, the service-oriented data interface gateway also integrates the MQTT IoT protocol conversion module. This module can convert the different protocols used by various IoT devices into a unified standard protocol within the system, achieving seamless connection with various IoT devices. Through the message publishing / subscribing mechanism of the MQTT protocol, the gateway can efficiently collect and transmit device data, support functions such as device status monitoring and remote control, and meet the AI ​​analysis engine's processing requirements for IoT device data.

[0071] In order to support concurrent access to data by three types of applications: large-screen visualization system, mobile inspection terminal, and AI analysis engine, the service-oriented data interface gateway has been optimized in terms of architecture and performance. By adopting load balancing technology, caching mechanism and other means, the gateway can reasonably allocate system resources to ensure that under high concurrency, system resources are reasonably allocated, and each application can quickly and stably obtain the required data, providing strong data support for highway monitoring, management and decision-making. At the same time, the interface gateway is reinforced with security measures such as identity authentication and data encryption to ensure the security and integrity of the data transmission process.

[0072] The beneficial effects of the present invention are:

[0073] (1) The constructed multi-source heterogeneous data collection channel and standardized data template effectively solved the problem of the difficulty in uniformly collecting and processing multi-type and multi-format data during highway operation.

[0074] (2) The sliding time window mechanism in the time dimension ensures that the system can update data in seconds and flexibly adapt to real-time data changes; the spatial dimension grids the highway space to achieve a close integration of data and geographic location, facilitating the rapid location and query of data on specific sections; the two-layer storage structure in the business dimension separates basic data and event feature data, improving data retrieval and analysis efficiency and meeting different business needs.

[0075] (3) The event-driven data association engine uses a dynamic weight allocation algorithm and event ID chain binding mechanism to effectively mine the intrinsic connections between different types of data.

[0076] (4) Column storage units improve the query efficiency of structured business data, object storage units efficiently manage video stream data, and time series databases accurately store and analyze equipment status monitoring data. Distributed indexing services enable data lineage tracking across storage units, facilitating data management and maintenance, quality control, and troubleshooting.

[0077] (5) Dynamic vectorized permission management accurately limits the access of different business roles to different data areas, preventing data leakage and illegal access. The system's real-time verification function can intercept and record illegal access to ensure data security.

[0078] (6) The service-oriented data interface gateway constructed realizes the efficient sharing and distribution of data.

[0079] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for unified storage and management of highway data in a data center, characterized in that: include: Step S1, building a dedicated data collection channel for multi-source heterogeneous data generated by highway operation; Design standardized data templates to connect to various data sources, use edge computing nodes for protocol conversion and semantic cleaning, and generate standardized metadata packages with timestamps; Step S2, establish a dynamic sharding storage matrix to slice and store data from three dimensions: time, space, and business; Step S3, designing an event-driven data association engine, and establishing a spatiotemporal mapping relationship between device status data and video stream data based on a dynamic weight allocation algorithm; And through the event ID chain binding mechanism, the abnormal event data packet and related data are automatically associated and stored; Step S4, deploying a multimodal storage cluster, using different storage units to store different data, and implementing data lineage tracking across storage units through a distributed index service; Step S5, implement dynamic vectorized permission management, determine the level of access rights of business roles to data areas, establish a fine-grained data access control model, and achieve real-time verification and violation interception; Step S6, construct a service-oriented data interface gateway to support concurrent data access for multiple types of applications.

2. The method for unified storage and management of highway data in the middle platform according to claim 1, characterized in that: In step S2, in the time dimension, a sliding time window mechanism is used to implement data update, data is divided according to set time intervals, the time window size and sliding step length are dynamically adjusted, and data in the time window is aggregated and compressed; in the spatial dimension, the highway space is divided into raster storage units according to the section pile number coordinate system, and the data is stored according to the geographic location information; in the business dimension, the data is separated into a basic data layer and an event feature layer to form a double-layer storage structure.

3. The method for unified storage and management of highway data in the middle platform according to claim 2, characterized in that: In step S2, in the time dimension, let W be the time window, t be the time, is the standardized metadata package of the i-th data source at time t, is the aggregation result of the data in the time window W, then ， in, and are the start and end time of time window W respectively; is the weight of the i-th data source at time t.

4. The method for unified storage and management of highway data in the middle platform according to claim 2, characterized in that: In step S2, in terms of spatial dimension, the road section stake number coordinate system is set to (x, y), and the highway space is divided into M×N grid cells, and the coordinates of each grid cell are ( )(1≤ ≤M,1≤ ≤N); Assume that the geographical location information of data D is , then the grid cell coordinates where the data should be stored ( , )satisfy: ,in , is the minimum coordinate of the highway space, is the side length of the grid cell in the x and y directions.

5. The method for unified storage and management of highway data in the middle platform according to claim 1, characterized in that: In step S3, when the event-driven data association engine performs data association storage, it first performs data preprocessing to extract key information from device status data and video stream data, and then establishes a spatiotemporal mapping relationship based on a dynamic weight allocation algorithm. The system monitors the operation status of the highway in real time, generates a unique event ID when an abnormal event is detected, and performs chain binding storage using the event ID as a clue.

6. The highway data center unified storage management method according to claim 5, characterized in that: In step S3, the dynamic weight allocation algorithm comprehensively considers the highway operation scenario, road section traffic flow and device status factors, dynamically adjusts the weights of device status data and video stream data, and establishes a spatiotemporal mapping relationship; let S be the highway operation scenario, Q be the road section traffic flow, is the device status, , are the weights of device status data and video stream data respectively, then the dynamic weight allocation algorithm formula is: ； in, and are the influence coefficients of the highway operation scenario S on the weights of the equipment status data and the video stream data, respectively; and The device status and the characteristic function of traffic flow Q; and are the traffic flow Q and the equipment status respectively Correction factor for weights.

7. The highway data center unified storage management method according to claim 1, characterized in that: In step S4, in the multimodal storage cluster, different storage units are used to store different data, including a column storage unit for storing structured business data, an object storage unit for storing video stream data, and a time series database for storing equipment status monitoring data; wherein, the column storage unit stores structured business data such as charging transaction records and equipment management information, the object storage unit uses block storage and redundant backup to store video stream data, and the time series database stores equipment status monitoring data to realize monitoring and predictive maintenance of equipment operating conditions; the distributed index service records data storage locations and associations to realize data lineage tracking across storage units.

8. The highway data center unified storage management method according to claim 1, characterized in that: In step S5, dynamic vectorized permission management is implemented, key features of business roles are analyzed to form business role feature vectors; the data space is divided into different data areas, and a corresponding data space coordinate vector is generated for each data area; and the access permission level of the business role to the data area is determined by calculating the cosine similarity between the two.

9. The method for unified storage and management of highway data in the data center according to claim 8, characterized in that: In step S5, assume that the business role R has n key features, which are , then the business role feature vector ,in It is a quantitative representation of various information of business roles; Set data area There are m key information, which are , then the data space coordinate vector ,in It is a quantitative representation of various information in the data space; Business Role Characteristic Vector With the data space coordinate vector The cosine similarity of , determine the access permission level based on similarity , where f is a monotonically increasing function that maps similarity to different levels of access rights.

10. The highway data center unified storage management method according to claim 1, characterized in that: In the step S6, a service-oriented data interface gateway is constructed to provide real-time data push service through HTTP / 2 protocol; a video stream proxy service is provided through WebSocket protocol to optimize video stream transmission; Through the MQTT protocol conversion module, seamless connection with IoT devices is achieved; the service-oriented data interface gateway adopts load balancing technology and caching mechanism to reasonably allocate system resources, support concurrent data access of three types of applications: large-screen visualization system, mobile inspection terminal, and AI analysis engine, and perform security reinforcement.

Images