Railway full life cycle data and model integration method and system and storage medium

Abstract

The invention provides a railway full life cycle data and model integration method and system and a storage medium. Relates to the technical field of railway data management. The method comprises the following steps: acquiring and storing railway data in real time from each railway data source accessed to a platform through a data lake module; the data lake module comprises a data uptake sub-module and a data storage sub-module; the data uptake sub-module is used for importing railway data from each railway business data source in real time based on a data source address and storing the railway data into the data storage sub-module; each railway data source comprises a data source corresponding to each service in the full life cycle of the railway; obtaining railway equipment facility models from each railway data source in real time through the model library module, and storing the railway equipment facility models in the model library module in a document form; the model library module comprises an access interface sub-module, and the access interface sub-module is used for receiving and executing a model library operation request acting on the model library module; according to the invention, the problem of difficulty in integration and sharing of railway full-life-cycle data can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of railway data management, and in particular to a method, system and storage medium for integrating data and models throughout the entire life cycle of a railway. Background Art

[0002] With the continuous advancement of science and technology, high-speed railways have gradually become an important pillar of modern transportation with their fast, convenient and efficient characteristics. As an upgraded version of high-speed railways, intelligent railways are leading the future development trend of the transportation field. In the entire life cycle of intelligent railways, each link such as survey and design, construction and operation and maintenance involves a large amount of data. These data are massive, multi-source and heterogeneous, which is of great significance to improving the design level, construction quality and operation efficiency of intelligent railways.

[0003] Traditional data storage methods can usually only store data at a specific stage or type, and lack an in-depth understanding and comprehensive consideration of the characteristics of data throughout the entire life cycle of intelligent railways. In addition, due to the wide range of data sources and diverse formats, there are large differences and incompatibilities between data, which brings great challenges to the organization and management of data, making it difficult to achieve data integration and sharing. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method, system and storage medium for integrating data and models throughout the entire life cycle of a railway, so as to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present invention provides a method for integrating data and models throughout the life cycle of a railway, which is applied to a data and model integration platform for the entire life cycle of a railway, wherein the platform includes a data lake module and a model library module; the method includes the following steps:

[0006] The data lake module is used to obtain and store railway data in real time from various railway data sources accessing the platform; the data lake module includes a data acquisition submodule and a data storage submodule; the data acquisition submodule is used to obtain the data source address corresponding to each railway data source from the preset configuration information, import railway data from various railway business data sources in real time based on the data source address, load the railway business data in its original form, and store it in the data storage submodule; each railway data source includes the data source corresponding to each business in the entire life cycle of the railway; railway data includes structured data, semi-structured data and unstructured data; the data storage submodule adopts a distributed storage mode to store railway data;

[0007] The railway equipment and facility models are acquired from various railway data sources in real time through the model library module, and the railway equipment and facility models are stored in the model library module in the form of documents; the model library module includes an access interface submodule, which is used to accept and execute model library operation requests acting on the model library module.

[0008] In some embodiments of the present invention, the data storage submodule includes a native business data repository and a metadata repository; the railway data is stored in the native business data repository; after the railway data is acquired and stored in real time from various railway data sources of the access platform through the data lake module, the following is further included:

[0009] The data ingestion submodule identifies the structure and format of the data from the semi-structured data and extracts metadata;

[0010] Store metadata in a metadata repository.

[0011] In some embodiments of the present invention, storing metadata in a metadata repository includes:

[0012] Based on the frequency of use and update, metadata is partitioned and stored; each metadata partition has different importance, and the importance is positively correlated with the frequency of use and update;

[0013] In each metadata partition, the metadata in each metadata partition is divided based on the preset data domain and the preset data scenario to obtain the scenario data set corresponding to each data scenario; the preset data domain and the preset data scenario are obtained based on the pre-division of each business in the entire life cycle of the railway;

[0014] Construct a data domain and hierarchical organization table based on the scene data set.

[0015] In some embodiments of the present invention, the data lake module further includes an interactive application submodule; the interactive application submodule is used to provide a data retrieval function for users, so that users can generate formatted query requests through stored metadata to retrieve data; after obtaining and storing railway data in real time from various railway data sources of the access platform through the data lake module, it also includes:

[0016] In response to receiving the query request, the interactive application submodule obtains a query result corresponding to the query request;

[0017] A visualization interface is generated based on the query results to display the query results; the visualization interface includes an annotation sub-interface and / or a filtering sub-interface; the annotation sub-interface is used to receive data annotation information input by a user; the filtering sub-interface is used to receive filtering conditions input by a user, and the query results displayed in the visualization interface are filtered based on the filtering conditions;

[0018] The metadata generated during the data retrieval is stored in the metadata repository through the interactive application submodule.

[0019] In some embodiments of the present invention, the data lake module also includes a data conversion submodule; the data conversion submodule is used to provide a data conversion engine; the data conversion engine is used to convert railway data into a predefined data structure; the data conversion engine is also used to publish the railway equipment and facility model stored in the model library module as a model service.

[0020] In some embodiments of the present invention, each railway data source includes a satellite navigation system data source, and the railway data includes railway position information data provided by the satellite navigation system data source; the digital conversion submodule is also used to perform real-time calculation of the railway position information data.

[0021] In some embodiments of the present invention, before converting the railway data into a predefined data structure by a data conversion engine, the method further includes:

[0022] The railway data is cleaned by the data conversion submodule, and the data cleaning includes noise data detection based on a detection algorithm and / or data repair based on a data repair algorithm.

[0023] In some embodiments of the present invention, the model library module further includes a view submodule; after obtaining the railway equipment and facility model from various railway data sources in real time through the model library module and storing the railway equipment and facility model in the form of a document in the model library module, it also includes:

[0024] The view submodule organizes and indexes the document data corresponding to each model based on the preset model mapping strategy and the model attributes corresponding to each model, and displays the document data in the model display interface provided by the view submodule.

[0025] Another aspect of the present invention provides a data and model integration system for the entire life cycle of railways, including a processor, a memory, and a computer program / instructions stored in the memory, characterized in that the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the aforementioned data and model integration method for the entire life cycle of railways.

[0026] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the aforementioned method for integrating data and models throughout the life cycle of a railway.

[0027] The data and model integration method, system and storage medium of the railway life cycle of the present invention can solve the problem of difficult data integration and sharing of the railway life cycle; through the data lake module and the automated ingestion process, data is extracted from various railway data sources in real time. Standardized data source address configuration is adopted to ensure the efficiency and accuracy of data source access; at the same time, the distributed storage architecture is used to realize flexible storage of different types of data (structured, semi-structured and unstructured data), ensuring high availability and scalability; through the model library module, the railway equipment and facility model is obtained from each data source in real time to ensure the latest and accuracy of the model, and the model is stored in the form of a document to facilitate subsequent query and call, so as to realize the integration of railway data and models; through the design of a flexible access interface, various model library operation requests are supported, which is convenient for efficient management of data and models, reduces data movement, and realizes efficient sharing of railway data and models; at the same time, the data lake module can simplify the data ingestion process by loading the original form of railway data, without defining its structure when storing data, and improves the flexibility and efficiency of data loading to cope with various complex data scenarios.

[0028] In addition, by acquiring railway data and models in real time through the data lake module and model library module, it is possible to update and maintain data storage organization in real time according to dynamic changes in data, ensuring the timeliness and accuracy of railway data and models.

[0029] In addition, the data lake module will extract metadata and control data quality for the acquired railway data; at the same time, according to the data conversion engine, the original data in the data lake module will be converted into a predefined data structure through data cleaning, conversion, integration, etc., and the original data in the data storage submodule will be effectively integrated according to the user's specific application requests, so that the railway data can be organically integrated at different stages, which is convenient for cross-stage analysis and application of data, ensuring data accuracy, consistency and integrity, facilitating user understanding and use, improving data readability and operability, and at the same time reducing decision-making risks caused by data problems.

[0030] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0031] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0033] Figure 1 A flowchart of a method for integrating data and models for the entire life cycle of a railway provided by an embodiment of the present invention.

[0034] Figure 2 A schematic diagram of the system architecture of a data lake module provided in one embodiment of the present invention.

[0035] Figure 3 A schematic diagram of data partitioning provided by an embodiment of the present invention.

[0036] Figure 4 A schematic diagram of a model library provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0038] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0039] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0040] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0041] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0042] The following is a detailed introduction to the data and model integration method for the entire railway life cycle provided by this application.

[0043] like Figure 1As shown, the embodiment of the present application provides a method for integrating data and models of the entire life cycle of a railway. The implementation of the method can rely on a computer program, which can be run on a computer device such as a smart phone, a tablet computer, a personal computer, or a server. This embodiment does not limit the operating subject of the method. The method includes at least the following steps:

[0044] Step S101, obtain and store railway data in real time from various railway data sources connected to the platform through the data lake module.

[0045] like Figure 2 As shown, the data lake module includes a data ingestion submodule and a data storage submodule.

[0046] Among them, the data ingestion submodule is used to import multi-source heterogeneous data from heterogeneous data sources.

[0047] In some embodiments of the present invention, since railway data comes from a variety of heterogeneous data sources, including sensor data, historical records, and real-time monitoring information, the data formats and structures are different. Based on this, in order to acquire and load railway data of various data types from heterogeneous data sources at the lowest cost, the data ingestion submodule avoids format conversion during railway data loading and directly loads the railway data in its original form, thereby simplifying the railway data ingestion process.

[0048] like Figure 2 As shown, the data source addresses corresponding to various heterogeneous data sources are configured through data source configuration, configuration information is generated, and the configuration information is pre-stored in the data storage submodule. The data ingestion submodule realizes the automatic ingestion of railway data through the data source address in the configuration information pre-stored in the data storage submodule.

[0049] Specifically, the data acquisition submodule is used to obtain the data source address corresponding to each railway data source from the preset configuration information, import railway data from each railway business data source in real time based on the data source address, load the railway business data in its original form, and store it in the data storage submodule.

[0050] Among them, various railway data sources include data sources corresponding to various businesses in the entire life cycle of the railway.

[0051] For example, the entire life cycle of a railway includes the survey and design stage, the construction stage, and the operation and maintenance stage. In the survey and design stage, the railway data obtained from the data source includes but is not limited to terrain data, geological condition data, climate and environmental data, and other data to provide a basis for railway line planning, station layout, etc. In the construction stage, the railway data obtained from the data source includes but is not limited to real-time construction progress data, material usage data, quality inspection data, etc. to ensure project quality and safety. In the operation and maintenance stage, the railway data obtained from the data source includes but is not limited to train operation data, equipment maintenance records, etc. to achieve real-time monitoring and optimization of the operation status of the intelligent railway.

[0052] In actual implementation, the data lake module can also obtain railway data through offline batch import. At this time, the data source includes but is not limited to historical databases, local file systems, sensor device logs, etc.

[0053] Since the railway data extracted by the data ingestion submodule is in the original format, including structured data, semi-structured data and unstructured data, different storage systems are needed to store various types of railway data, and a distributed storage mode is used to store railway data.

[0054] In some embodiments of the present invention, Figure 2 As shown, after the data ingestion submodule imports the railway data, the data ingestion submodule is also used to extract metadata from the railway data. Metadata extraction refers to obtaining and extracting information about the railway data itself from the railway data.

[0055] Correspondingly, the data storage submodule includes a native business data repository and a metadata repository. Among them, the railway data is stored in the native business data repository. The native business data repository includes but is not limited to diversified spatiotemporal databases such as MySQL, PostgreSQL, MongoDB, and Redis to achieve efficient organization and storage of structured, semi-structured, and unstructured railway data; by leveraging the technical advantages of different storage formats, the efficiency of spatiotemporal data storage access and query is improved.

[0056] The metadata repository is used to store metadata extracted by the data ingestion submodule. Metadata extraction can identify patterns in semi-structured data sources such as JSON format or XML format, and the extracted metadata is then stored and managed in the metadata repository of the data storage submodule. In some embodiments of the present invention, the metadata repository stores structured, semi-structured, and unstructured metadata based on the Hadoop Distributed File System (HDFS).

[0057] Specifically, after obtaining and storing railway data in real time from various railway data sources accessing the platform through the data lake module, it also includes: identifying the structure and format of data from semi-structured data through the data ingestion submodule, extracting metadata; and storing the metadata in the metadata repository.

[0058] In addition, in order to provide users with a unified query and access method, in some embodiments of the present invention, the data storage submodule uses the rich SQL query language and data model provided by the Hbase database to implement a unified data access interface to encapsulate the hybrid storage architecture. The interface provides query language and data model, and has sufficiently effective expressions to handle the complex query logic and corresponding complex data structures managed by the data lake module.

[0059] At the same time, in order to support users to access and analyze the data stored in the data storage submodule, it is necessary to partition, domain, and organize the data stored in the data storage submodule for storage and encapsulation, and manage the data into clean data according to the data domain, data level, actual data scenarios and needs to support user access analysis.

[0060] After the data ingestion submodule extracts metadata from the railway data, the metadata is partitioned according to the data application status corresponding to each business in the entire life cycle of the intelligent railway. First, the metadata is partitioned according to the frequency of data use and update.

[0061] For example: Reference Figure 3, taking railway data including asset management data, EMU intelligent operation and maintenance data, infrastructure intelligent operation and maintenance data, intelligent comprehensive dispatching data, operation diagram intelligent compilation data, intelligent travel data, intelligent passenger data and intelligent monitoring data, the metadata is divided into cold data area, warm data area and hot data area according to the frequency of use and update frequency as an example; in asset management data, the cold data area includes land use data and land development data, the warm data area includes project documents and other data, and the hot data area includes station commodity statistics data, land asset statistics data, etc.; in EMU intelligent operation and maintenance data, the cold data area includes EMU history, the warm data area includes basic diagrams and road network capacity data, and the hot data area includes EMU condition-based maintenance statistics data, etc.; in infrastructure intelligent operation and maintenance data, the cold data area includes fixed facility history, the warm data area includes power supply maintenance data and engineering maintenance data, and the hot data area includes Facility status evaluation data, etc.; intelligent comprehensive dispatching data, the cold data area includes historical road network capacity diagrams, the warm data area includes route marshalling data and other data, and the hot data area includes train operation performance data, etc.; in the intelligent compilation data of operation diagrams, the cold data area includes historical operation diagrams, the warm data area includes operation diagrams and planned statistical data and other data, and the hot data area includes congestion analysis data, etc.; in the intelligent travel data, the cold data area includes service feedback data, the warm data area includes ticket data and other data, and the hot data area includes arrival and departure passenger flow situation prediction data, etc.; in the intelligent passenger transport data, the cold data area includes historical passenger flow data, the warm data area includes passenger station management data and comprehensive traffic data and other data, and the hot data area includes personnel abnormal behavior identification data, etc.; in the intelligent monitoring data, the cold data area includes historical monitoring data, the warm data area includes electrical detection and monitoring data and other data, and the hot data area includes video alarm data and other data.

[0062] Afterwards, the data in each partition is divided according to the preset data domain and preset data scenario, wherein the preset data domain and preset data scenario are obtained based on the pre-division of various businesses in the entire life cycle of the railway.

[0063] In some embodiments of the present invention, the preset data domain includes a primary subject domain and a secondary subject domain. The primary subject domain is the main business field or stage in the entire life cycle of the railway, covering the core functions and activities of railway operations, etc. The secondary subject domain is a further subdivision of the primary subject domain, and is classified in detail for specific businesses or activities, so as to manage and analyze data more deeply.

[0064] For example: as shown in Table 1, the first-level subject domain includes the safety management data domain, equipment and facility data domain, operation management data domain, station and train service data domain, and business development data domain; the second-level subject domain corresponding to the safety management data domain includes external access data, video alarm data, foreign object alarm data, and external hidden danger identification data; the second-level subject domain corresponding to the equipment and facility data domain includes comprehensive detection data, disease identification result data, maintenance decision recommendations, and personnel behavior warnings; the second-level subject domain corresponding to the operation management data domain includes driving data, meteorological data, dispatching data, and emergency response plans; the second-level subject domain corresponding to the station and train service data domain includes ticket data, comprehensive traffic data, station equipment status, and inbound and outbound passenger statistics; the second-level subject domain corresponding to the business development data domain includes station commercial data, advertising data, land value assessment, and advertising placement strategies; the corresponding data scenarios include natural disaster monitoring and early warning application scenarios, satellite infrastructure monitoring application scenarios, comprehensive dispatching application scenarios for regional road networks, full-trip service application scenarios, and asset intelligent management application scenarios.

[0065]

[0066]

[0067] Table 1

[0068] In actual implementation, the preset data domain may also include other subject domains to further divide the railway data, such as intelligent detection and monitoring data domain, intelligent passenger transport data domain, intelligent ticketing data domain, intelligent comprehensive dispatching data domain, operation diagram intelligent compilation data domain, infrastructure intelligent operation and maintenance data domain, EMU intelligent operation and maintenance data domain, etc. This embodiment does not limit the implementation method of the preset data domain.

[0069] After partitioning, domaining and grading the data and adopting distributed storage technology for hybrid architecture storage, the data set content required for the data scenario is mapped to the data table. According to the requirements of the data scenario, a unified data access interface is used to uniformly encapsulate the data required for the same data scenario, so as to provide users with a unified query and access method.

[0070] Specifically, the metadata is stored in a metadata repository, including: storing the metadata in partitions based on usage frequency and update frequency; each metadata partition has a different importance, and the importance is positively correlated with the usage frequency and update frequency; in each metadata partition, the metadata in each metadata partition is divided based on a preset data domain and a preset data scenario to obtain a scenario data set corresponding to each data scenario; and a data domain hierarchical organization table is constructed based on the scenario data set.

[0071] In actual implementation, the metadata repository is also used to store metadata manually added by users during the use of the data lake module. These metadata manually added by users include historical feedback information about the use of the data lake module, such as the properties used to connect datasets, the transformation methods applied to datasets (such as integration, cleaning, etc.), and analysis reports of related datasets. These historical feedback information constitutes a knowledge base for subsequent use of datasets, providing valuable reference and assistance for the effective use of datasets.

[0072] In some embodiments of the present invention, in order to avoid the data swamp phenomenon, railway data governance and railway data quality management become crucial in the data ingestion stage. Among them, railway data quality control ensures that the ingested railway data meets basic quality standards.

[0073] Due to the huge amount of railway data and its diverse types, it is impractical to manually set data quality rules. Figure 2 As shown, the data lake module also includes a data conversion submodule, through which dynamic and static data of different sources and standards are reviewed to achieve data conversion.

[0074] In some embodiments of the present invention, the data conversion submodule is used to provide a data conversion engine, through which the railway data is converted into a predefined data structure. The data conversion engine includes but is not limited to a Geographic Information System (GIS) engine, a Building Information Modeling (BIM) engine or a solution software.

[0075] In actual implementation, in order to further realize the effective integration of railway data and provide effective data for user access and analysis, the railway data needs to be cleaned before converting it into a predefined data structure.

[0076] The data cleaning process includes abnormal data detection and abnormal data repair. Abnormal data detection refers to noise data detection based on detection algorithms; abnormal data repair refers to data repair based on data repair algorithms.

[0077] Specifically, before converting the railway data into a predefined data structure through the data conversion engine, it also includes: performing data cleaning on the railway data through the data conversion submodule, and the data cleaning includes noise data detection based on a detection algorithm and / or data repair based on a data repair algorithm.

[0078] There are different detection algorithms for different types of railway data, including but not limited to manual detection, statistical methods, pattern recognition methods, clustering methods, proximity methods, association rule methods, etc.

[0079] In some embodiments of the present invention, the detection algorithms include a standard score (Z-Score) statistical method, an interquartile range (IQR) statistical method, a density-based spatial clustering algorithm applied to noise (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) in machine learning methods, and an isolation forest algorithm.

[0080] Among them, the Z-Score algorithm is mainly used for data sets that conform to or approximate normal distribution; the IQR algorithm is used to identify abnormal traffic flow data in the process of railway station traffic monitoring data processing. For example, a sudden increase or decrease in passenger flow due to an accident or special event, etc., to help optimize train schedules; DBSCAN is a density-based clustering algorithm used to process data sets with complex shapes and sizes; the Isolation Forest algorithm can effectively identify abnormal monitoring images or environmental indicator readings in safety monitoring and environmental monitoring projects. For example, unusual temperature changes or sudden changes in air quality, etc., to indicate fires, pollution incidents or equipment failures.

[0081] Accordingly, different data repair algorithms are required for different types of railway data, including but not limited to median filling algorithm, majority filling algorithm, nearest neighbor interpolation filling algorithm, forward / backward filling algorithm and regression filling algorithm.

[0082] Among them, the median filling algorithm is suitable for processing missing data due to lack of complete records or inconsistent data collection. For example, if there are missing values ​​in the material prices of a certain area, given that the price data may fluctuate due to extreme market fluctuations, using the median filling can avoid the deviation caused by extreme values, thereby providing a stable basis for price estimation. The mode filling algorithm is suitable for processing missing categorical data in projects. In urban traffic flow research, if the type data of some traffic nodes (such as stations, intersections, etc.) are missing, using the mode filling can ensure the integrity of the data while maintaining the accuracy of the analysis of urban traffic patterns. The nearest neighbor interpolation (K-nearest neighbor, KNN) filling algorithm is applied to the processing of spatial data. The forward / backward filling algorithm uses the previous or next data of the missing data for processing. Forward filling and backward filling are suitable for processing time series data in engineering projects, such as construction progress reports, material supply records, etc. When processing this type of data, if the records of a certain day are missing, it can be assumed that the construction progress or material demand will not change much in the short term, so using the data of the previous day or the next day for filling is a practical method. The regression filling algorithm is suitable for situations where there is an obvious linear or nonlinear relationship between data and is used in data analysis situations, especially when there is an obvious correlation between data.

[0083] In some embodiments of the present invention, data cleaning also includes missing value detection and repair. The goal of missing value detection is to identify missing values ​​in a data set, which can be achieved through the isnull() and notnull() functions in Python, or the custom if(value == null) code in Java; the missing value repair algorithm is the same as the data repair algorithm for outliers, which will not be repeated here.

[0084] In some embodiments of the present invention, data cleaning also includes duplicate data detection and repair, wherein duplicate data refers to data generated due to data entry errors or overlaps in the data integration process. Duplicate data processing methods include rule-based duplicate detection algorithms and similarity-based algorithms.

[0085] Among them, the rule-based duplicate detection algorithm refers to a rule-based method that identifies duplicate records by defining a set of rules. These rules are usually based on the matching of key fields. For example, name, address or other unique identifiers. A record is uniquely identified by precisely defining a combination of fields, and duplicates in the data set are checked accordingly. For example, the application scenario of the rule-based duplicate detection algorithm is the application scenario of supply chain management as an example; in the application scenario of supply chain management, projects usually involve a large number of material suppliers and subcontractors. By defining clear rules (such as supplier name, contract number) to identify and merge duplicate records in the supply chain database, the accuracy of material procurement and supply chain management is ensured, and duplicate orders or payments are avoided.

[0086] Similarity-based algorithms can identify records that are not identical on the surface but are essentially duplicates. Text similarity algorithms are usually used to calculate the similarity scores between records and determine whether the records are duplicates based on the similarity scores. Among them, text similarity algorithms include but are not limited to Jaccard Similarity, Cosine Similarity, and Levenshtein Distance.

[0087] The cleaned data is integrated into the data mart driven by the data scenario through the data integration workflow. In addition, the knowledge created when defining the data mart (for example, how to transform, integrate, and analyze the relevant information of the data set) needs to be maintained in the data transformation submodule and stored in the metadata repository.

[0088] In addition, in some embodiments of the present invention, each railway data source includes a satellite navigation system data source, and the railway data includes railway location information data provided by the satellite navigation system data source; in this case, the digital conversion submodule is also used to perform real-time solution on the railway location information data.

[0089] In addition, in some embodiments of the present invention, the data conversion submodule is also used to implement operations such as data comparison, coordinate system conversion, vectorization, association of spatial elements and attribute information, topological relationship check, and data compression of railway data. This is achieved through a process-based and batch processing tool built on a feature processing engine (FME) or a geographic information system (GIS). This tool implements data customization, rule configuration, batch processing, data tracking and backtracking, which can improve data management efficiency.

[0090] In some embodiments of the present invention, Figure 2As shown, the data lake module also includes an interactive application submodule. The interactive application submodule focuses on the interaction between users and the data lake module. Users will query the data categories they can access through metadata. Data retrieval and metadata management are closely related. The metadata generated during data retrieval (for example, semantic annotations, newly discovered relationships, etc.) will be stored in the metadata repository through schema management. At the same time, the interactive application submodule is also used to provide data retrieval based on query formulas to support users in creating formatted query requests that can express their information needs. Since users cannot directly access the native functions of various data storage systems, the interactive application submodule also provides common functions related to data operations, including data visualization, annotation, selection, filtering, and basic data analysis capabilities.

[0091] Specifically, the interactive application submodule is used to provide users with a data retrieval function so that users can generate formatted query requests for data retrieval through stored metadata; after obtaining and storing railway data in real time from various railway data sources accessing the platform through the data lake module, it also includes: obtaining query results corresponding to the query requests in response to the query requests received through the interactive application submodule; generating a visualization interface based on the query results to display the query results; the visualization interface includes an annotation subinterface and / or a filtering subinterface; the annotation subinterface is used to receive data annotation information input by the user; the filtering subinterface is used to receive filtering conditions input by the user, and filter the query results displayed in the visualization interface based on the filtering conditions; the metadata generated during data retrieval is stored in the metadata repository through the interactive application submodule.

[0092] In some embodiments of the present invention, facing the physical world in the entire life cycle of the railway, the data sources corresponding to each business that is accessed are clearly aggregated, and multimodal data collection, business system data access, external data integration and other methods are adopted to realize the aggregation of dynamic business data and static business data in the entire life cycle of the railway into the data lake module. Through the data lake module, a comprehensive breakthrough in the dynamic collection, governance and sharing of multi-source, heterogeneous and cross-domain data in the field of high-speed rail assets and operations is achieved, effectively solving the problems of data island phenomenon and insufficient data mining; and achieving real-time monitoring and comprehensive mastery of massive heterogeneous data, thereby providing strong support for railway operation management and decision-making. At the same time, by effectively governing these data, the quality of the data can be improved and the accuracy and reliability of the data can be ensured.

[0093] Step S102: obtaining railway equipment and facility models from various railway data sources in real time through the model library module, and storing the railway equipment and facility models in the model library module in the form of documents.

[0094] In some embodiments of the present invention, in order to better manage and utilize geographic information systems (GIS), building information models (BIM), satellite navigation systems, etc., a comprehensive representation model library module is established to integrate various railway equipment and facility models such as the GIS model of intelligent high-speed rail, BIM model, national basic geographic information model, station 3D model, industrial power supply 3D model, and BIM model of key facilities. By classifying, storing and organizing these railway equipment and facility models, a comprehensive railway equipment and facility model library is constructed.

[0095] In the model library module, various railway equipment and facility models are usually stored in the form of documents for easy management and retrieval. In order to make these railway equipment and facility models more widely used, the data conversion engine is also used to publish the railway equipment and facility models stored in the model library module as model services. In this way, users can access and use these models through the network, thereby improving work efficiency and collaboration capabilities.

[0096] In some embodiments of the present invention, the model library module uses the open source, document-oriented CouchDB data as the core database management system, and supports JSON format as the basis for data storage and exchange. This data format has high flexibility and scalability, making the operation of the model library module more flexible and convenient.

[0097] In some embodiments of the present invention, the model library module includes an access interface submodule, which is used to receive and execute model library operation requests acting on the model library module. Among them, the access interface submodule includes an interface based on Representational State Transfer (REST) ​​or Simple Object Access Protocol (SOAP), and the administrator implements operations such as adding, deleting, modifying, and checking the model library module through model library operation requests based on Hypertext Transfer Protocol (HTTP), which simplifies the development process, improves the scalability and interoperability of the system, and makes data interaction between different systems and platforms smoother.

[0098] In addition, the model library module also includes a view submodule. The model library module manipulates the organization and presentation of documents through the view submodule, such as Figure 4 As shown, Figure 4 The "Name" column is the model name, the "Size" column is the size of the model, the "#ofDocs" column indicates the number of documents corresponding to the model, and the "Actions" column indicates the operations on the model, including deletion or locking operations.

[0099] Administrators can organize and index document data based on model attribute rules and related mapping functions. By defining different views to meet different needs and application scenarios, you can flexibly display and manage the content in the model library. For example, you can create a view to arrange documents in chronological order, or group and sort documents according to certain specific fields in the document.

[0100] Specifically, after obtaining the railway equipment and facility models from various railway data sources in real time through the model library module and storing the railway equipment and facility models in the model library module in the form of documents, it also includes: organizing and indexing the document data corresponding to each model based on the preset model mapping strategy and the model attributes corresponding to each model through the view sub-module, and displaying the document data in the model display interface provided by the view sub-module.

[0101] It is worth noting that in some embodiments of the present invention, step S101 may be executed before step S102, or after step S102, or step S101 and step S102 may be executed synchronously. This embodiment does not limit the execution order of step S101 and step S102.

[0102] In summary, the data and model integration method for the entire life cycle of railways provided by the present invention is applied to the data and model integration platform for the entire life cycle of railways. The platform includes a data lake module and a model library module; the data lake module is used to obtain and store railway data in real time from various railway data sources connected to the platform; the data lake module includes a data acquisition submodule and a data storage submodule; the data acquisition submodule is used to obtain the data source address corresponding to each railway data source from the preset configuration information, import railway data from various railway business data sources in real time based on the data source address, load the railway business data in its original form, and store it in the data storage submodule; each railway data source includes a railway The data source corresponding to each business in the whole life cycle; railway data includes structured data, semi-structured data and unstructured data; the data storage submodule adopts a distributed storage mode to store railway data; the railway equipment and facility models are obtained from various railway data sources in real time through the model library module, and the railway equipment and facility models are stored in the model library module in the form of documents; the model library module includes an access interface submodule, which is used to accept and execute model library operation requests acting on the model library module; it can solve the problem of difficult integration and sharing of railway data throughout the whole life cycle; through the data lake module and the automated ingestion process, data can be extracted from various railway data sources in real time. Standardized data source address configuration is adopted to ensure the efficiency and accuracy of data source access. At the same time, the distributed storage architecture is used to realize flexible storage of different types of data (structured, semi-structured and unstructured data) to ensure high availability and scalability. Through the model library module, railway equipment and facility models are obtained from various data sources in real time to ensure the latest and accuracy of the models. At the same time, the models are stored in the form of documents to facilitate subsequent queries and calls, so as to realize the integration of railway data and models. By designing flexible access interfaces, various model library operation requests are supported, which facilitates the efficient management of data and models, reduces data movement, and realizes the efficient sharing of railway data and models. At the same time, the data lake module loads railway data in its original form, and there is no need to define its structure when storing data. It can simplify the data ingestion process, improve the flexibility and efficiency of data loading, and cope with various complex data scenarios.

[0103] In addition, by acquiring railway data and models in real time through the data lake module and model library module, it is possible to update and maintain data storage organization in real time according to dynamic changes in data, ensuring the timeliness and accuracy of railway data and models.

[0104] In addition, the data lake module will extract metadata and control data quality for the acquired railway data; at the same time, according to the data conversion engine, the original data in the data lake module will be converted into a predefined data structure through data cleaning, conversion, integration, etc., and the original data in the data storage submodule will be effectively integrated according to the user's specific application requests, so that the railway data can be organically integrated at different stages, which is convenient for cross-stage analysis and application of data, ensuring data accuracy, consistency and integrity, facilitating user understanding and use, improving data readability and operability, and at the same time reducing decision-making risks caused by data problems.

[0105] Corresponding to the above method, the present invention also provides a data and model integration system for the entire life cycle of railways, the system comprising a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps of the data and model integration method for the entire life cycle of railways as described above.

[0106] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data and model integration method for the entire life cycle of the railway are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0107] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0108] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0109] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for integrating data and models throughout the railway life cycle, characterized in that: A data and model integration platform for the entire life cycle of a railway, the platform comprising a data lake module and a model library module; the method comprising: The data lake module is used to obtain and store railway data in real time from various railway data sources connected to the platform; the data lake module includes a data acquisition submodule and a data storage submodule; the data acquisition submodule is used to obtain the data source address corresponding to each railway data source from the preset configuration information, import the railway data from each railway business data source in real time based on the data source address, load the railway business data in original form, and store it in the data storage submodule; the various railway data sources include data sources corresponding to various businesses in the entire life cycle of the railway; the railway data include structured data, semi-structured data and unstructured data; the data storage submodule adopts a distributed storage mode to store the railway data; The model library module is used to obtain railway equipment and facility models from the various railway data sources in real time, and the railway equipment and facility models are stored in the model library module in the form of documents; the model library module includes an access interface submodule, and the access interface submodule is used to accept and execute model library operation requests acting on the model library module.

2. The method according to claim 1, characterized in that The data storage submodule includes a native business data repository and a metadata repository; the railway data is stored in the native business data repository; After obtaining and storing railway data in real time from various railway data sources connected to the platform through the data lake module, the method further includes: Identifying the structure and format of data from the semi-structured data through the data ingestion submodule, and extracting metadata; The metadata is stored in the metadata repository.

3. The method according to claim 2, characterized in that The storing the metadata in the metadata repository comprises: The metadata is partitioned and stored according to the metadata based on the usage frequency and the update frequency; each metadata partition has a different importance, and the importance is positively correlated with the usage frequency and the update frequency; In each metadata partition, the metadata in each metadata partition is divided based on a preset data domain and a preset data scenario to obtain a scenario data set corresponding to each data scenario; the preset data domain and the preset data scenario are obtained based on the pre-division of each business in the entire life cycle of the railway; A data domain-based hierarchical organization table is constructed based on the scene data set.

4. The method according to claim 2, characterized in that: The data lake module also includes an interactive application submodule; the interactive application submodule is used to provide a data retrieval function for the user, so that the user generates a formatted query request through the stored metadata to perform data retrieval; After obtaining and storing railway data in real time from various railway data sources connected to the platform through the data lake module, the method further includes: In response to receiving the query request, the interactive application submodule obtains a query result corresponding to the query request; Generate a visualization interface based on the query results to display the query results; the visualization interface includes an annotation sub-interface and / or a filtering sub-interface; the annotation sub-interface is used to receive the data annotation information input by the user; the filtering sub-interface is used to receive the filtering conditions input by the user, and filter the query results displayed in the visualization interface based on the filtering conditions; The metadata generated during the data retrieval is stored in the metadata repository through the interactive application submodule.

5. The method according to claim 1, characterized in that The data lake module also includes a data conversion submodule; the data conversion submodule is used to provide a data conversion engine; the data conversion engine is used to convert the railway data into a predefined data structure; the data conversion engine is also used to publish the railway equipment and facility model stored in the model library module as a model service.

6. The method according to claim 5, characterized in that The various railway data sources include a satellite navigation system data source, and the railway data include railway position information data provided by the satellite navigation system data source; the digital conversion submodule is also used to perform real-time calculation on the railway position information data.

7. The method according to claim 5, characterized in that Before converting the railway data into a predefined data structure by the data conversion engine, the method further includes: The railway data is cleaned by the data conversion submodule, and the data cleaning includes noise data detection based on a detection algorithm and / or data repair based on a data repair algorithm.

8. The method according to claim 1, characterized in that The model library module also includes a view submodule; after obtaining the railway equipment and facility model from each railway data source in real time through the model library module and storing the railway equipment and facility model in the form of a document in the model library module, it also includes: The view submodule organizes and indexes the document data corresponding to each model based on the preset model mapping strategy and the model attributes corresponding to each model, and displays the document data in the model display interface provided by the view submodule.

9. A data and model integration system for the entire life cycle of a railway, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images