Rail transit multi-source heterogeneous data integration and sharing management method and system

Abstract

The invention discloses a rail transit multi-source heterogeneous data integration and sharing management method and system. The method comprises the following steps: S1, acquiring and integrating rail transit multi-source heterogeneous data; for different types of data, data acquisition is carried out by adopting corresponding interfaces and acquisition modes; the different types of data comprise vehicle-mounted Internet of Things equipment data, business system operation data and statistical analysis report file data; s2, processing the multi-source heterogeneous data of the rail transit, supporting a last custom function operator at the same time, and generating a statistical analysis result through the data to realize a configurable data management function; and S3, performing configuration resource sharing on the real-time stream and the offline data by using a data security sharing technology. According to the invention, integration, data management and secure sharing of multi-source heterogeneous data can be realized.

Description

Technical Field

[0001] This invention mainly relates to the field of information processing technology, specifically to a method and system for integrating and sharing multi-source heterogeneous data in rail transit. Background Technology

[0002] In today's rapidly developing information age, enterprises and organizations face the challenges of diversified data sources and heterogeneous data formats. To address these challenges, from early manual data migration to the emergence of ETL tools, the rise of data warehouses, and the challenges of the big data era and the integration of cloud and AI, data integration technologies are constantly adapting to the new demands of data processing and analysis. Simultaneously, to reduce data silos, low data utilization, and promote innovation and development, data sharing technologies have also been further promoted with the rapid development of computer network technology. The emergence of technologies such as data platforms, cloud computing, and big data provides a broader platform and more efficient ways to unlock the potential value of data, drive scientific decision-making, improve user experience, and promote product innovation. Summary of the Invention

[0003] To address the technical problems existing in the prior art, this invention provides a method and system for integrating and sharing multi-source heterogeneous data in rail transit, enabling the integration, data governance, and secure sharing of multi-source heterogeneous data.

[0004] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0005] A method for integrating and sharing heterogeneous data from multiple sources in rail transit includes the following steps:

[0006] S1. Acquire and integrate multi-source heterogeneous data from rail transit; adopt corresponding interfaces and acquisition methods for different types of data; different types of data include data from on-board IoT devices, operational data from business systems, and statistical analysis report files.

[0007] S2. Process multi-source heterogeneous data of rail transit, and support custom function operators to generate statistical analysis results through data, so as to realize configurable data governance functions.

[0008] S3. Utilize data security sharing technology to enable configurable resource sharing of real-time streaming and offline data.

[0009] Preferably, the multi-source heterogeneous data includes databases, API interfaces, data protocol streams, files, and video streams.

[0010] Preferably, in step S1, the main data integration process for integrating database type data is as follows:

[0011] S1.1.1 Creating a data source: First, configure the driver packages of various database software into the driver directory. By defining the database source component, configure various data sources and configure the connection information of the data sources, you can access the database.

[0012] S1.1.2 Constructing a data synchronization model: Through data-driven packages and connection information, metadata of the source database is collected, which assists in establishing a data synchronization table model in the collection platform;

[0013] S1.1.3, Constructing Data Synchronization Scripts: The database integration mode uses SQL data scripts to obtain data from the data source. At the same time, the extracted database tables, synchronization data source and target tables, incremental fields and insert / update strategies are determined through configuration to complete the writing of the data synchronization scripts.

[0014] S1.1.4 Build a data synchronization mechanism: By configuring a data synchronization mechanism, synchronized data is stored in the database, and the program calls the script periodically to achieve incremental data synchronization.

[0015] Preferably, in the integration of HTTP interface data, the HTTP interface data is acquired, the HTTP interface component is integrated, and data acquisition is performed through configurable interface parameters. This achieves a configurable parameterized approach, integrating the data requested from the HTTP interface into the destination source. The main data integration process is as follows:

[0016] S1.2.1 Configuration Parameters: Define the connection information for the data source;

[0017] S1.2.2 Build a data synchronization model: Establish a data synchronization table model by returning the data format through the interface;

[0018] S1.2.3 Constructing the data synchronization script: By configuring the script, determine the data source and target table for synchronization, the mapping relationship between interface data items and data items in the synchronization table, the incremental fields, and the insertion / update strategy, and complete the writing of the data synchronization script;

[0019] S1.2.4 Build a data synchronization mechanism: By configuring a data synchronization mechanism, synchronized data is stored in the database, and the program calls the script periodically to achieve incremental data synchronization.

[0020] Preferably, for interfaces of network protocol packet types, a combination of a general data parsing engine and a dynamically configurable database signal point table is designed. Only the data protocol information needs to be configured and the signal point table created according to the data protocol; then the data parsing engine automatically completes data parsing based on the signal point table. The main data interface process is as follows:

[0021] S1.3.1 Configuration Parameters: Define the connection information for the data source;

[0022] S1.3.2 Configuring the Data Protocol Point Table: Using data protocol information, configure the data protocol point table in the database. The point table stores the data content of various data flow protocols to prepare for subsequent data parsing.

[0023] S1.3.3 Constructing a data synchronization script: Using signal point information, the data parsing engine completes data parsing based on the signal point table, converting the raw binary data into signal values ​​with specific meanings, and importing them into the data warehouse.

[0024] Preferably, for file data integration, by integrating the data interface capabilities of FTP, SFTP, and MINIO, the integration of file type data is provided, and various types of files are stored in the file repository, and file information is recorded in the database table; at the same time, if the file is a binary protocol file, the data stream is read out through a method similar to a network protocol packet interface, the binary protocol is parsed, and stored in the database.

[0025] Preferably, for video stream data integration, the video stream interface uses a general video plugin for video playback, and records video stream metadata in the database, which is then used by the asset platform for permission allocation.

[0026] Preferably, in step S2, the data governance process is as follows:

[0027] S2.1 Extract the data from the source system, transform it, and then load it into the target data warehouse;

[0028] S2.2 Develop rule-based function operator components using code, and simultaneously upload custom code function JAR files to the platform to support custom data operations;

[0029] S2.3. Load the data into the destination data source through the data export component for storage, and provide it to downstream applications for access and use.

[0030] Preferably, the specific process of step S2.1 is as follows:

[0031] S2.1.1 Extraction: Extracting data from one or more source systems; source systems include databases, HTTP interfaces, and data in network protocol packets;

[0032] S2.1.2 Transformation: During the transformation phase, the data undergoes cleaning, filtering, formatting, and merging operations to conform to the data model and quality standards of the target data warehouse.

[0033] S2.1.3 Loading: The loading phase loads the transformed data into a specific table or data structure in the target data warehouse.

[0034] Preferably, in step S3, the data security sharing process for offline historical data is as follows:

[0035] S3.1.1 Encapsulates the main relational database driver package;

[0036] S3.1.2. Based on the database driver package, implement a general database access JDBC template;

[0037] S3.1.3 Implement a general access backend interface, and enable free access to database tables through interface data configuration parameters;

[0038] S3.1.4. IP management based on Spring Boot backend requests: Set up blacklist and whitelist permission configurations in the platform interface to enable and deny IP access.

[0039] S3.1.5 Based on the Spring Boot backend token generation mechanism, user information is configured in the platform interface, and non-users cannot generate access tokens;

[0040] S3.1.6 Users can access data normally by using the access token and request information.

[0041] Preferably, in step S3, the process for the secure sharing service of real-time streaming data is as follows:

[0042] S3.2.1 Configure user permission information and MQ message queue name;

[0043] S3.2.2 Based on configuration information, dynamically create users and authorize queue security authentication permissions;

[0044] S3.2.3 After authorization, users can access queue data normally.

[0045] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, the computer program performing the steps of the method described above when run by a processor.

[0046] The present invention further discloses a multi-source heterogeneous data integration and sharing management system for rail transit, including a memory and a processor connected to each other. The memory stores a computer program, and the computer program executes the steps of the method described above when run by the processor.

[0047] Compared with the prior art, the advantages of the present invention are as follows:

[0048] This method is based on a multi-source data integration and management approach. It integrates heterogeneous data from multiple sources by collecting data from relational and non-relational databases, API interfaces, data streams, files (images, documents, binary files, etc.), and video streams. Through a user-friendly process configuration interface and a visualized data processing workflow, it enables data governance functions such as data quality management, anomaly monitoring, and statistical analysis of the collected data streams. By controlling data access permissions and enabling customized and rapid construction of data asset services, it ensures data security and flexibility, and achieves secure data sharing in multiple ways. Attached Figure Description

[0049] Figure 1 This diagram illustrates an embodiment of the data integration and sharing management method of the present invention in a specific application.

[0050] Figure 2 This is a diagram illustrating an embodiment of the network protocol packet interface in this invention during a specific application.

[0051] Figure 3 This is a diagram illustrating an embodiment of the data sharing security platform of the present invention in a specific application. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0053] In the field of big data integration for rail transit information systems, actual data mainly includes data from onboard IoT devices, operational data from business systems, and statistical analysis report files. Device data is real-time, business system data is semi-real-time or non-real-time, and report files are non-real-time. Their generation methods and interface methods differ, requiring different data acquisition methods for data access. Furthermore, after data acquisition, it needs to be provided to third parties for data development. Therefore, designing a configurable, unified data acquisition and sharing platform is essential. This platform enables configurable data acquisition, processing, and sharing services.

[0054] like Figure 1 As shown in the figure, the method for integrating and sharing heterogeneous data from multiple sources in rail transit provided by this invention includes the following steps:

[0055] S1. In terms of data integration, the platform integrates operational data from business systems using either databases or HTTP interfaces. IoT device data mainly consists of real-time status data from various vehicle sensors or detection devices. Data is collected via network protocol interfaces and file data interfaces. Files are uploaded via FTP / SFTP / MinIO protocols, parsed and aggregated using parsing tools, and stored in the database. Video stream data is accessed using general video call interfaces, and video stream links are stored in the database.

[0056] S2. In terms of data processing, the platform can perform data conversion, cleaning and quality inspection on the data stream during the collection process. It also supports custom function operators and can generate statistical analysis results through data to achieve configurable data governance functions.

[0057] S3. In terms of data security sharing, the platform can utilize data security sharing technology to enable configurable, rapid, and efficient resource sharing of real-time streaming and offline data, providing users with multiple data sharing methods, promoting application and system integration, breaking down information protection barriers, and facilitating cooperation and communication.

[0058] Specifically, in the multi-source heterogeneous data integration process of step S1, the data in the big data integration systems of the rail transit field currently includes various types and formats of data. Multi-source heterogeneous data includes databases, API interfaces, data protocol streams, files (images, documents, binary files, etc.), video streams, and other data. The platform has developed a unified collection and management method to address the characteristics of these data sources, thereby achieving the integration of multi-source heterogeneous data. Specifically, this includes the following data integration:

[0059] 1. Database Integration

[0060] Regarding the integration of database types, multi-source data catalog functionality is used to extract and access data from mainstream data lakes and databases such as MySQL, Oracle, SQL Server, and Postgres. The main data integration process is as follows:

[0061] S1.1.1 Creating a data source

[0062] The platform first configures the driver packages for various database software into the driver directory. By defining the database source component, various data sources can be configured. Once the connection information of the data source (data source username, data source password, data source connection URL, port, etc.) is configured, the database can be accessed.

[0063] S1.1.2, Constructing a data synchronization model

[0064] By using data-driven packages and connection information, metadata from the source database can be collected, which helps to establish a data synchronization table model in the collection platform.

[0065] S1.1.3, Constructing the data synchronization script

[0066] The database integration mode uses SQL data scripts to retrieve data from the data source. At the same time, the extracted database tables, synchronization data source and target tables, incremental fields and insert / update strategies are determined through configuration to complete the data synchronization script writing.

[0067] S1.1.4, Constructing a data synchronization mechanism

[0068] By configuring a data synchronization mechanism, synchronized data is stored in the database, and the program periodically calls the script to achieve incremental data synchronization.

[0069] 2. HTTP Interface Data Integration

[0070] To obtain data from HTTP interfaces, the platform integrates an HTTP interface component. Data is acquired through configurable interface parameters, enabling a configurable parameterized approach to integrate data from HTTP interface requests into the destination source. The main data integration process is as follows:

[0071] S1.2.1 Configuration Parameters

[0072] Define the data source connection information (data source connection URL, input parameters, HTTP request method (POST / GET), etc.)

[0073] S1.2.2, Constructing a data synchronization model

[0074] By returning data in the interface format, a data synchronization table model is established in the data warehouse of the data acquisition platform.

[0075] S1.2.3, Constructing the data synchronization script

[0076] By configuring the script, the data synchronization script is written to determine the data source and target table, the mapping relationship between interface data items and data items in the synchronization table, the incremental fields, and the insertion / update strategy.

[0077] S1.2.4, Constructing a data synchronization mechanism

[0078] By configuring a data synchronization mechanism, synchronized data is stored in the database, and the program periodically calls the script to achieve incremental data synchronization.

[0079] 3. Network protocol packet interface

[0080] like Figure 2 As shown, for interfaces of network protocol packet types, the platform designs a combination of a general data parsing engine and a dynamically configurable database signal point table. Users only need to configure the data protocol information and create a signal point table according to the data protocol, and then the data parsing engine automatically completes the data parsing based on the signal point table; the main data interface process is as follows:

[0081] 1) Configuration parameters

[0082] Define the connection information for the data source (data protocol TCP / UDP / MODUBUS / IEC, port, etc.).

[0083] 2) Configure the data protocol point table

[0084] By using data protocol information, a data protocol point table is configured in the database. The point table stores the data content of various data flow protocols, preparing for subsequent data parsing.

[0085] 3) Build a data synchronization script

[0086] Using signal point information, the data parsing engine completes data parsing based on the signal point table, converting the raw binary data into signal values ​​with specific meanings, and then importing them into the data warehouse.

[0087] 4. File data integration

[0088] By integrating data interface capabilities with FTP, SFTP, and MINIO, it provides integrated file type data, allowing various types of files to be stored in a file repository and their information recorded in a database table. Furthermore, if a file is a binary protocol file, the data stream can be read through a network protocol packet interface for binary protocol parsing and storage in the database.

[0089] 5. Video stream data integration

[0090] The video stream interface uses a general video plugin for video playback and records video stream metadata in the database, which is then used by the asset platform for permission allocation.

[0091] In the data governance step S2, to improve data processing efficiency, enhance data value, and address issues related to data quality and security, this method implements visualized data processing workflows through a graphical process configuration interface. By performing ETL operations such as cleaning and transformation on the data streams collected during data acquisition, and by creating various operators and adding the use of custom functions, it achieves data quality governance, function transformation, and statistical analysis operations for the collected data streams.

[0092] The main data processing flow in step S2 includes:

[0093] S2.1 Data ETL Module: ETL (Extract, Transform, Load) is a key process used to extract data from the source system, transform it, and then load it into the target data warehouse.

[0094] a. Extraction: Extracting data from one or more source systems. Source systems can be databases, HTTP interfaces, and data in network protocol packets, etc.

[0095] b. Transformation: During the transformation phase, data undergoes cleaning, filtering, formatting, and merging operations to conform to the data model and quality standards of the target data warehouse. Transformation operations typically include data cleaning (such as handling missing values ​​and outliers), data integration (merging data from multiple sources into a single entity), and data transformation (such as data format conversion and date conversion).

[0096] c. Load: The load phase involves loading the transformed data into specific tables or data structures in the target data warehouse. Loading operations need to consider data integrity and consistency, and typically include data validation and verification steps.

[0097] S2.2 Operator Module: After the data passes through the ETL module, basic processing has been performed. However, considering data processing efficiency and flexibility, this method utilizes code development to implement rule-based function operator components (addition, subtraction, multiplication, division, AND, OR, NOT, average, variance, summation, cumulative value, sine / cosine, tangent / cotangent, logarithm, square, duration, greater than / less than, equal to, not equal to, remainder, threshold setting, etc.). Simultaneously, considering complex data scenarios, this method allows uploading custom code function JAR packages to the platform to support custom data operations. The operator component functionality is integrated into the platform interface, fulfilling functions such as functional statistical analysis, threshold judgment, and anomaly detection for the collected data stream and fields. For example, in train vehicle energy consumption statistics, the cumulative value operator can be used to statistically analyze traction energy consumption and air conditioning, while threshold, average, and variance operators can be used to detect anomalies in energy consumption data.

[0098] S2.3 Data Export Module: After passing through the ETL and operator modules, the data can be loaded into the destination data source for storage via the data export component, making it accessible and usable by downstream applications. This method is consistent with database interface integration in data integration. The destination data source is the final storage location for the exported data, which can be a relational database (such as MySQL, PostgreSQL, etc.), a NoSQL database (such as MongoDB, Cassandra, etc.), or a file storage system (such as Hadoop HDFS, Amazon S3, etc.). Downstream applications include report generation, business analysis, data mining, machine learning model training, etc. The exported data can be a complete, integrated dataset or a specific subset of data, customized according to the needs of the downstream applications in terms of content and format.

[0099] Each module in the data processing flow is implemented through code, integrating various components. Users can perform configuration operations on the platform interface to form the data processing flow. The order and configuration of the data flow are stored in the database table.

[0100] like Figure 3 As shown, in the data security sharing in step S3, after the data is integrated into the data platform, in order to meet the needs of third-party systems for real-time and offline data, a set of data security sharing technology needs to be developed. This method realizes the configuration of user permissions for offline and real-time data on the interface and realizes the custom generation of data sharing interface.

[0101] For offline data, it supports users with read-only database access. It is also developed based on the Spring Boot technology framework and supports the deployment of API services for various data sources, such as configurable relational databases and non-relational databases.

[0102] For real-time streaming data, based on the MQ message queue framework, dynamic user creation and permission management (subscription and publication) are achieved by controlling the security authentication mechanism attributes of the queue, enabling rapid resource allocation. It features automatic API generation, user management, resource permission management, blacklists and whitelists, and access frequency control, realizing secure data access and sharing.

[0103] Specifically as follows:

[0104] 1. Database sharing service

[0105] For offline historical data stored in the database, the platform provides an interface-executable database read-only permission configuration. Database DDL commands can be executed on the interface to create database users with read-only permissions for data tables, providing data access for third-party systems.

[0106] 2. Offline API Data Security Sharing Service

[0107] Offline historical data is typically stored in a data warehouse, which can use databases such as MySQL, Oracle, SQL Server, and Postgres. By encapsulating and configuring their driver packages and leveraging the Spring Boot framework, a general API interface can be developed. The specific steps are as follows:

[0108] S3.1.1 Encapsulates the main relational database driver package.

[0109] S3.1.2 Implement a general database access JDBC template based on the database driver package.

[0110] S3.1.3 Implement a general access backend interface, and enable free access to database tables through interface data configuration parameters.

[0111] S3.1.4. IP management based on Spring Boot backend requests: Set up blacklist and whitelist permission configurations in the platform interface to enable and deny IP access.

[0112] S3.1.5 Based on the Spring Boot backend token generation mechanism, user information is configured in the platform interface, and non-users cannot generate access tokens.

[0113] S3.1.6 Users can access data normally by using the access token and request information.

[0114] 3. Real-time streaming data secure sharing service

[0115] Adopting the Spring Boot architecture and based on the MQ message queue framework, the platform interface implements SASL_SCRAM management of queue TOPICs, enabling dynamic user creation and permission control. This facilitates data exchange and management control with third-party systems, improving the efficiency of data exchange and management and enabling rapid resource allocation.

[0116] S3.2.1 Configure user permission information and MQ message queue name.

[0117] S3.2.2 Based on configuration information, dynamically create users and authorize queue security authentication permissions.

[0118] S3.2.3 After authorization, users can access queue data normally.

[0119] This method is based on a multi-source data integration and management approach. It integrates heterogeneous data from multiple sources by collecting data from relational and non-relational databases, API interfaces, data streams, files (images, documents, binary files, etc.), and video streams. Through a user-friendly process configuration interface and a visualized data processing workflow, it enables data governance functions such as data quality management, anomaly monitoring, and statistical analysis of the collected data streams. By controlling data access permissions and enabling customized and rapid construction of data asset services, it ensures data security and flexibility, and achieves secure data sharing in multiple ways.

[0120] This invention supports automated data acquisition from real-time / offline sources and multiple data sources. It replaces the previous manual development model with configurable data acquisition, improving the efficiency and accuracy of data acquisition. For parsing network binary stream data, it utilizes a combination of a general data parsing engine and a dynamically configurable signal point table; automatic parsing of network protocol packets can be completed simply by configuring data protocol information.

[0121] This invention addresses abnormal and non-compliant data by providing data monitoring / detection, data cleaning, and data repair functions, ensuring the compliance and accuracy of the integrated data. It implements a configurable data flow operation interface and allows for the creation of custom data operators, enabling data governance and statistical analysis of the collected data.

[0122] This invention achieves rapid, efficient, and secure sharing of offline and real-time data by studying technologies such as rapid generation of RestAPI data interfaces, real-time queue management, and access control.

[0123] This invention also discloses a computer-readable storage medium storing a computer program thereon, which, when run by a processor, executes the steps of the method described above. This invention further discloses a multi-source heterogeneous data integration and sharing management system for rail transit, including an interconnected memory and a processor, wherein the memory stores a computer program, which, when run by a processor, executes the steps of the method described above. The medium and system of this invention, corresponding to the methods described above, also possess the advantages described above.

[0124] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be implemented by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium includes: any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. The memory is used to store computer programs and / or modules. The processor implements various functions by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory. The memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0125] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for integrating and sharing heterogeneous data from multiple sources in rail transit, characterized in that, Including the following steps: S1. Acquire and integrate multi-source heterogeneous data from rail transit; adopt corresponding interfaces and acquisition methods for different types of data; different types of data include data from on-board IoT devices, operational data from business systems, and statistical analysis report files. S2. Process multi-source heterogeneous data of rail transit, and support custom function operators to generate statistical analysis results through data, so as to realize configurable data governance functions. S3. Utilize data security sharing technology to enable configurable resource sharing of real-time streaming and offline data.

2. The method for integrating and sharing multi-source heterogeneous data in rail transit according to claim 1, characterized in that, The multi-source heterogeneous data includes databases, API interfaces, data protocol streams, files, and video streams.

3. The method for integrating and sharing heterogeneous data from multiple sources in rail transit according to claim 2, characterized in that, In step S1, the main data integration process for integrating database types is as follows: S1.1.1 Creating a data source: First, configure the driver packages of various database software into the driver directory. By defining the database source component, configure various data sources and configure the connection information of the data sources, you can access the database. S1.1.2 Constructing a data synchronization model: Through data-driven packages and connection information, metadata of the source database is collected, which assists in establishing a data synchronization table model in the collection platform; S1.1.3, Constructing Data Synchronization Scripts: The database integration mode uses SQL data scripts to obtain data from the data source. At the same time, the extracted database tables, synchronization data source and target tables, incremental fields and insert / update strategies are determined through configuration to complete the writing of the data synchronization scripts. S1.1.4 Build a data synchronization mechanism: By configuring a data synchronization mechanism, synchronized data is stored in the database, and the program calls the script periodically to achieve incremental data synchronization.

4. The method for integrating and sharing heterogeneous data from multiple sources in rail transit according to claim 2, characterized in that, In integrating HTTP interface data, the process involves acquiring HTTP interface data, integrating HTTP interface components, and using configurable interface parameters for data acquisition. This allows for a configurable parameterized approach to integrate data from HTTP interface requests into the destination source. The main data integration process is as follows: S1.2.1 Configuration Parameters: Define the connection information for the data source; S1.2.2 Build a data synchronization model: Establish a data synchronization table model by returning the data format through the interface; S1.2.3 Constructing the data synchronization script: By configuring the script, determine the data source and target table for synchronization, the mapping relationship between interface data items and data items in the synchronization table, the incremental fields, and the insertion / update strategy, and complete the writing of the data synchronization script; S1.2.4 Build a data synchronization mechanism: By configuring a data synchronization mechanism, synchronized data is stored in the database, and the program calls the script periodically to achieve incremental data synchronization.

5. The method for integrating and sharing heterogeneous data from multiple sources in rail transit according to claim 2, characterized in that, For interfaces using network protocol packets, a combination of a general data parsing engine and a dynamically configurable database signal point table is designed. Users only need to configure the data protocol information and create the signal point table based on the data protocol; the data parsing engine then automatically completes the data parsing based on the signal point table. The main data interface process is as follows: S1.3.1 Configuration Parameters: Define the connection information for the data source; S1.3.2 Configuring the Data Protocol Point Table: Using data protocol information, configure the data protocol point table in the database. The point table stores the data content of various data flow protocols to prepare for subsequent data parsing. S1.3.3 Constructing a data synchronization script: Using signal point information, the data parsing engine completes data parsing based on the signal point table, converting the raw binary data into signal values ​​with specific meanings, and importing them into the data warehouse.

6. The method for integrating and sharing heterogeneous data from multiple sources in rail transit according to claim 2, characterized in that, For file data integration, by integrating the data interface capabilities of FTP, SFTP, and MINIO, we provide integration of file type data, store various types of files in the file repository, and record file information in the database table; at the same time, if the file is a binary protocol file, we read the data stream through a method similar to network protocol packet interface, perform binary protocol parsing, and store it in the database.

7. The method for integrating and sharing heterogeneous data from multiple sources in rail transit according to claim 2, characterized in that, For video stream data integration, the video stream interface uses a general video plugin for video playback and records video stream metadata in the database, which is then used by the asset platform for permission allocation.

8. The method for integrating and sharing multi-source heterogeneous data in rail transit according to any one of claims 1-7, characterized in that, In step S2, the data governance process is as follows: S2.1 Extract the data from the source system, transform it, and then load it into the target data warehouse; S2.2 Develop rule-based function operator components using code, and simultaneously upload custom code function JAR files to the platform to support custom data operations; S2.

3. Load the data into the destination data source through the data export component for storage, and provide it to downstream applications for access and use.

9. The method for integrating and sharing heterogeneous data from multiple sources in rail transit according to claim 8, characterized in that, The specific process of step S2.1 is as follows: S2.1.1 Extraction: Extracting data from one or more source systems; source systems include databases, HTTP interfaces, and data in network protocol packets; S2.1.2 Transformation: During the transformation phase, the data undergoes cleaning, filtering, formatting, and merging operations to conform to the data model and quality standards of the target data warehouse. S2.1.3 Loading: The loading phase loads the transformed data into a specific table or data structure in the target data warehouse.

10. The method for integrating and sharing multi-source heterogeneous data in rail transit according to any one of claims 1-7, characterized in that, In step S3, the data security sharing process for offline historical data is as follows: S3.1.1 Encapsulates the main relational database driver package; S3.1.

2. Based on the database driver package, implement a general database access JDBC template; S3.1.3 Implement a general access backend interface, and enable free access to database tables through interface data configuration parameters; S3.1.

4. IP management based on Spring Boot backend requests: Set up blacklist and whitelist permission configurations in the platform interface to enable and deny IP access. S3.1.5 Based on the Spring Boot backend token generation mechanism, user information is configured in the platform interface, and non-users cannot generate access tokens; S3.1.6 Users can access data normally by using the access token and request information.

11. The method for integrating and sharing multi-source heterogeneous data in rail transit according to any one of claims 1-7, characterized in that, In step S3, the process for the real-time streaming data secure sharing service is as follows: S3.2.1 Configure user permission information and MQ message queue name; S3.2.2 Based on configuration information, dynamically create users and authorize queue security authentication permissions; S3.2.3 After authorization, users can access queue data normally.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a processor, performs the steps of the method as described in any one of claims 1-11.

13. A multi-source heterogeneous data integration and sharing management system for rail transit, comprising an interconnected memory and a processor, wherein the memory stores a computer program, characterized in that, The computer program, when run by a processor, performs the steps of the method as described in any one of claims 1-11.

Images