A Data Governance System and Method Based on Big Data Technology in the Transportation Industry

Abstract

This application proposes a data governance system and method in the transportation industry based on big data technology. The system includes a data security module, a data integration module, a data governance module, a data development module, a data service module, an analysis and modeling module, an operation and maintenance monitoring module, and a data application module. Each module can be used independently or in any combination according to different user requirements. Considering the situation of multiple operators in the transportation industry and the need for data isolation and security, the system provides targeted data governance solutions for the transportation industry in terms of data security, data integration, data governance, data development, data service, analysis and modeling, operation and maintenance monitoring, and data application. It can effectively govern the data in the transportation industry, improve data processing efficiency, ensure data security and quality, and reduce governance costs.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a data governance system and method based on big data technology in the transportation industry. Background Art

[0002] Data is an important asset, and the reasonable use of data can bring huge value to enterprises.

[0003] Currently, the following main defects exist in data governance:

[0004] (1) Due to different focuses and perspectives of data governance, the overall architecture methods of data governance projects in relevant industries at home and abroad are also different. Currently, there is no personalized data governance solution for transportation industry data.

[0005] (2) When data quality problems occur, timely alarm notifications cannot be sent. It is only possible to discover, then manage, and then rework tasks, which affects data output and decision-making.

[0006] (3) The leakage of data will pose great risks to enterprises. There is a lack of consideration in terms of data security, resulting in the system being unable to meet data security requirements. Summary of the Invention

[0007] The main purpose of the embodiments of this application is to propose a data governance system and method based on big data technology in the transportation industry. Considering the situation of multiple operators in the transportation industry and the need for data isolation and security, a targeted data governance solution for the transportation industry is given in terms of data security, data integration, data governance, data development, data service, analysis and modeling, operation and maintenance monitoring, and data application, which can effectively govern transportation industry data, improve data processing efficiency, ensure data security and quality, and reduce governance costs.

[0008] To achieve the above object, the first aspect of the embodiments of this application proposes a data governance system for the transportation industry, and the system includes:

[0009] A data security module, a data integration module, a data governance module, a data development module, a data service module, an analysis and modeling module, an operation and maintenance monitoring module, and a data application module. Each module can be used independently or in any combination according to different user requirements;

[0010] The data security module is used to perform module deployment management, user level management, permission management for each level, and project management according to user requirements. The project management includes managing project basic information, project configuration personnel, project roles, and project data permissions;

[0011] The data integration module is used to integrate traffic industry data sources together through data access, data synchronization, and scheduling configuration to support secure data access;

[0012] The data governance module is used to perform metadata management, data asset management, data metric management, and data quality management on traffic industry data;

[0013] The data development module is used to develop traffic industry data according to project requirements through offline development and real-time computing development;

[0014] The data service module is used to configure the interface sets under different projects, the interfaces under the interface sets, and perform application authorization management on the interfaces;

[0015] The analysis and modeling module is used to develop models, deploy models, and evaluate models for the traffic industry data;

[0016] The operation and maintenance monitoring module is used to monitor the running status of current system tasks and the general running situation of instances;

[0017] The data application module is used to perform business intelligence analysis and algorithm application on the traffic industry data.

[0018] In some embodiments, the user hierarchy management includes:

[0019] Construct a user hierarchy, which includes super administrators, system administrators, project administrators, and ordinary users;

[0020] Set and manage the permissions corresponding to each user hierarchy.

[0021] In some embodiments, metadata includes business metadata and technical metadata, and the metadata management includes:

[0022] Manage the basic information, field information, lineage, and DDL changes of the business metadata and technical metadata. The basic information includes data table information, business information, and storage information. The field information includes field name, field type, field size, whether it is null, precision, field comment, calculation method, whether it is indexed, and partition field. The lineage includes the upstream and downstream generation relationships between system access data nodes, where the data nodes include upstream nodes, intermediate nodes, and downstream nodes. The DDL changes include operation information on data tables.

[0023] In some embodiments, the data asset management includes:

[0024] Query various types of offline data and real-time data in the traffic industry by searching system application programming interfaces, accessing collaborative data tables, or real-time data Topics.

[0025] In some embodiments, the data metrics include industry operation metrics, energy consumption metrics, and business operation metrics, and the data metric management includes:

[0026] Managing atomic metrics, derived metrics, time periods, and modifiers for the industry operation metrics, energy consumption metrics, and business operation metrics, wherein the derived metrics are automatically generated individually or in batches by means of the atomic metrics, time periods, and modifiers.

[0027] In some embodiments, the data quality management includes quality rule management and quality monitoring management;

[0028] The quality rule management is used to set quality rules for the transportation industry data, and the quality rules include uniqueness verification, field length verification, regular expression, enumerated value verification, table row count verification, null value verification, multi-table accuracy verification, custom SQL, timeliness verification, and two-table value comparison;

[0029] The quality monitoring management is used to issue a quality warning when the data to be verified in the transportation industry data triggers a preset threshold condition.

[0030] In some embodiments, the data service module includes interface management and application management;

[0031] The interface management is used to configure the interface set under different projects and the interfaces under the interface set, including:

[0032] Creating an interface set for the system application program;

[0033] Under the selected interface set, creating the interfaces in the selected interface set and configuring the basic information, where the basic information includes interface name, interface belonging set, interface path, version number, and interface description;

[0034] Determining the data source information, where the data source information includes interface data generation method, data source type, data source name, database name, and table name;

[0035] Configuring the interface parameters, where the interface parameters include input parameter definition, return parameters, request parameters, and sorting parameters;

[0036] After the interface parameters are configured, publishing the interface so that the interface is authorized for application;

[0037] The application management is used to perform application authorization management on the published interfaces, including:

[0038] Adding interface authorization applications and editing application information, where the application information includes application name, application belonging project, and application description;

[0039] Query all interfaces associated with each application and the authorization information corresponding to all interfaces.

[0040] In some embodiments, the interface management is further configured to perform the following operations:

[0041] When the target interface set to which the interface to be newly added belongs already exists, select the target interface set and add an interface under the target interface set;

[0042] When the target interface set to which the interface to be newly added belongs does not exist, create the target interface set and add an interface under the newly created target interface set.

[0043] In some embodiments, the system further includes:

[0044] A data storage module, configured to store the transportation industry data in a corresponding database as needed.

[0045] To achieve the above object, a second aspect of the embodiments of the present application proposes a method for governing transportation industry data, including:

[0046] Collect transportation industry data sources;

[0047] Integrate the transportation industry data sources together through a data integration module based on data access, data synchronization, and scheduling configuration to support secure data access;

[0048] Through a data security module, perform module deployment management, user level management, permission management for each level, and project management according to user requirements, and the project management includes managing project basic information, project configuration personnel, project roles, and project data permissions;

[0049] Through a data governance module, perform metadata management, data asset management, data indicator management, and data quality management on transportation industry data;

[0050] Through a data development module, perform transportation industry data development according to project requirements based on the methods of offline development and real-time computing development;

[0051] Through an analysis and modeling module, perform model development, model deployment, and model evaluation on the transportation industry data;

[0052] Through a data service module, configure interface sets and interfaces under the interface sets for different projects and perform application authorization management on the interfaces;

[0053] Through an operation and maintenance monitoring module, monitor the running status of current system tasks and the running overview of instances;

[0054] The traffic industry data is subjected to business intelligence analysis and algorithm application through the data application module.

[0055] A data governance system and method for the traffic industry based on big data technology proposed in this application. The system includes a data security module, a data integration module, a data governance module, a data development module, a data service module, an analysis and modeling module, an operation and maintenance monitoring module, and a data application module. Each module can be used independently or in any combination according to different user requirements. The system takes into account the situation of multiple operators in the traffic industry and the need for data isolation and security, and provides a targeted data governance solution for the traffic industry in terms of data security, data integration, data governance, data development, data service, analysis and modeling, operation and maintenance monitoring, and data application, which can effectively govern the traffic industry data, improve data processing efficiency, ensure data security and quality, and reduce governance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is the architecture diagram of the traffic industry data governance system provided in the embodiment of this application;

[0057] Figure 2 is the example diagram of user level management provided in the embodiment of this application;

[0058] Figure 3 is the example diagram of project member information provided in the embodiment of this application;

[0059] Figure 4 is the example diagram of basic information provided in the embodiment of this application;

[0060] Figure 5 is the example diagram of lineage provided in the embodiment of this application;

[0061] Figure 6 is the example diagram of DDL change provided in the embodiment of this application;

[0062] Figure 7 is the example diagram of the display page of the search API provided in the embodiment of this application;

[0063] Figure 8 is the example diagram of a single newly added derived indicator provided in the embodiment of this application;

[0064] Figure 9 is the example diagram of batch newly added derived indicators provided in the embodiment of this application;

[0065] Figure 10 is the example diagram of the energy consumption indicator table provided in the embodiment of this application;

[0066] Figure 11 is the example diagram of the front-end interface including data quality management provided in the embodiment of this application;

[0067] Figure 12 It is an example diagram of the interface management display interface provided by the embodiment of the present application;

[0068] Figure 13 It is an example diagram of the display interface of the newly added interface set provided by the embodiment of the present application;

[0069] Figure 14 It is an example diagram of the display interface for querying the newly added interface set provided by the embodiment of the present application;

[0070] Figure 15 It is an example diagram of the basic information display interface provided by the embodiment of the present application;

[0071] Figure 16 It is an example diagram of the data source display interface provided by the embodiment of the present application;

[0072] Figure 17 It is an example diagram of the input parameter definition display interface provided by the embodiment of the present application;

[0073] Figure 18 It is an example diagram of the return parameter display interface provided by the embodiment of the present application;

[0074] Figure 19 It is an example diagram of the request parameter display interface provided by the embodiment of the present application;

[0075] Figure 20 It is an example diagram of the request parameter display interface provided by the embodiment of the present application;

[0076] Figure 21 It is an example diagram of the interface authorization display interface provided by the embodiment of the present application;

[0077] Figure 22 It is an example diagram of the newly added application display interface provided by the embodiment of the present application;

[0078] Figure 23 It is an example diagram of the view authorization display interface provided by the embodiment of the present application;

[0079] Figure 24 It is one of the interface display example diagrams of the BI analysis provided by the embodiment of the present application;

[0080] Figure 25 It is an example diagram of the operation side rights and interests pricing algorithm interface provided by the embodiment of the present application;

[0081] Figure 26 It is a flowchart of the traffic industry data governance method provided by the embodiment of the present application. Specific implementation manner

[0082] In order to make the objectives, technical solutions, and advantages of this application more clearly understood, the following further elaborates on this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0083] It should be noted that although functional module division is carried out in the device schematic diagram and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be executed in a different module division from that in the device or a different order from that in the flowchart. Terms such as "first" and "second" in the description, claims, and the above-mentioned drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0084] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0085] The report "China Data Governance Market Share 2021" officially released by the Internet Data Center (IDC) shows that the scale of the Chinese data governance platform market reached 2.39 billion yuan in 2021. More and more industries and fields are beginning to recognize the value and necessity of data governance. The market for data governance will usher in a period of rapid growth.

[0086] It can be seen that the data governance industry has been widely accepted and recognized. At the same time, after decades of development in the Chinese data governance market, the market demand has undergone a major transformation. The internal data architecture of enterprises has become increasingly complex, posing higher requirements for the methods and approaches of data governance work.

[0087] In a data governance project, formulating a data governance architecture is the most core task. A good data governance architecture can ensure the integrity of data governance, achieve thorough and perfect data governance, and better achieve the expected effects of data governance.

[0088] However, due to different focuses and perspectives of data governance, the overall architecture methods of data governance projects in related industries at home and abroad are also different. At present, there is no personalized data governance solution for traffic industry data.

[0089] Based on this, the embodiments of this application propose a data governance system for the traffic industry, which can adapt to the needs of the traffic industry and give a targeted data governance solution for the traffic industry in terms of data security, data integration, data governance, data development, data services, analysis and modeling, operation and maintenance monitoring, and data application. It can effectively govern traffic industry data, improve data processing efficiency, ensure data security and quality, and reduce governance costs.

[0090] Data Governance is a set of management behaviors in an organization related to data usage. Initiated and promoted by the enterprise's data governance department, it involves a series of policies and processes for formulating and implementing commercial applications and technical management of internal data across the entire enterprise.

[0091] The Data Management Capability Maturity Model (DCMM) defines eight core capability areas of data governance: data strategy, data governance, data architecture, data application, data security, data quality, data standards, and data lifecycle.

[0092] From the perspective of technical implementation, data governance includes five steps: "sorting", "acquisition", "storage", "management", and "usage", namely business and data resource sorting, data acquisition and cleaning, database design and storage, data management, and data usage.

[0093] Refer to Figure 1 , based on the eight core capability areas of the data governance standards specified by DCMM, the embodiments of this application propose a data governance system for the transportation industry, taking into account elements such as multiple operators in the transportation industry and the need for data isolation and security, and giving a targeted data governance solution for the transportation industry. As Figure 1 shown, the data governance system mainly includes eight modules: data security module, data integration module, data governance module, data development module, data service module, analysis and modeling module, operation and maintenance monitoring module, and data application module. Each module can be used independently or in any combination according to different user requirements, connecting all links of data governance and quickly meeting various data governance scenarios of different operators. The super administrator of this governance system can select different modules for deployment according to the requirements of different operators. During the deployment process, according to the requirements of the operators, their respective transportation data can be isolated or shared, supporting diverse deployment forms.

[0094] Next, each module of the system will be described in detail.

[0095] Data Security Module:

[0096] It is used for module deployment management, user hierarchy management, and permission management at each level according to user requirements. Specifically, user hierarchy management includes constructing user hierarchies, which include super administrators, system administrators, project administrators, and ordinary users; setting and managing the permissions corresponding to each user hierarchy. The data security module is also used for project management, including managing project basic information, project configuration personnel, project roles, and project data permissions.

[0097] Data security is mainly reflected in the following two aspects:

[0098] (1) Module deployment: During the deployment process, the traffic data corresponding to different operators can be isolated, that is, the data between different operators will not be disclosed to each other, which can improve data security.

[0099] (2) Through user level management and level authority management, data isolation within the operator enterprise and functional isolation of management levels can be achieved. That is, according to the user's level and corresponding authority, the user can only view, manage and edit the data content corresponding to the level to which he belongs, which can also improve data security.

[0100] Reference Figure 2 , Figure 2 is an example diagram of user level management provided by the embodiment of the present application. Figure 2 As shown in the figure, users are divided into four levels, namely super administrator, system administrator, project administrator and ordinary user. The corresponding permissions of each role are shown in Table 1.

[0101] Table 1 Comparison table of permissions corresponding to different roles

[0102]

[0103] In the embodiment of the present application, project management supports adding new projects, querying and managing projects. In querying and managing projects, you can view and manage basic project information, configure project members, project roles and project data permissions in the project. For example, you can unbind project members. Project members can come from Figure 2 Any user in the system user list shown, there is a many-to-many relationship between users and projects. Figure 3 , Figure 3 This is an example diagram of project member information provided in an embodiment of the present application.

[0104] Data Integration Module:

[0105] Used to integrate transportation industry data sources through data access, data synchronization and scheduling configuration to support secure data access.

[0106] In the embodiment of the present application, the data integration module supports data access, data synchronization and scheduling configuration. Data access supports access to common data source types in the transportation industry. For example, relational databases, message queues and semi-structured storage. Among them, relational databases support MySQL databases, Dor is databases and Oracle databases; message queues support kafka, Rocketmq, etc.; semi-structured storage supports SFTP data collection, etc. Data synchronization can use Datax to synchronize offline data, and Flink CDC can be used to synchronize real-time data.

[0107] Data governance module:

[0108] The data governance module includes metadata management, data asset management, data metric management, and data quality management.

[0109] Among them, metadata management mainly manages the metadata information of the accessed systems. Metadata mainly includes business metadata and technical metadata. Metadata management mainly includes management in aspects such as basic information, field information, lineage, and DDL changes. Among them, as Figure 4 shown, Figure 4 is an example diagram of basic information provided by an embodiment of this application. The basic information shows the database table information, business information, and storage information of the system-accessed library tables. In the field information, the field name, field type, field size, whether it is nullable, precision digits, field comment, calculation method, whether it is indexed, partition field, etc. are shown. Refer to Figure 5 , Figure 5 is an example diagram of lineage provided by an embodiment of this application. As Figure 5 shown, the lineage shows the upstream and downstream generation relationships between the system-accessed data nodes. The nodes include upstream nodes, intermediate nodes, and downstream nodes. Refer to Figure 6 , Figure 6 is an example diagram of DDL changes provided by an embodiment of this application. As Figure 6 shown, in the DDL changes, the operations on the table in the past three months can be viewed. The change types include creating a table, deleting a table, renaming a table, replacing the table structure, modifying the table remarks, adding a column, modifying a column, modifying the partition value, deleting a partition, and other changes.

[0110] Among them, data asset management is mainly implemented based on the data map. The data map supports searching for the collaborative data tables, real-time data Topics, and all APIs in the system that are accessed. In this way, various types of offline and real-time data in the transportation industry can be viewed and managed. Refer to Figure 7 , Figure 7 is an example diagram of the display page for searching APIs provided by an embodiment of this application.

[0111] Among them, data metric management includes atomic metrics, derived metrics, time periods, and modifier management. Data metrics include three categories: industry operation metrics, energy consumption metrics, and commercial operation metrics. Among them, derived metrics can be automatically generated by combining atomic metrics, time periods, and modifiers. Currently, single addition and batch addition methods are supported. As Figure 8 shown, Figure 8 is an example diagram of single addition of a derived metric provided by an embodiment of this application. For batch addition, the atomic metric, modifier, time period, and associated dimension need to be selected respectively. As Figure 9 shown, Figure 9It is an example diagram of batch adding derived indicators provided by the embodiments of the present application. Through the management of data indicators, the statistical calibers can be converged and uniformly managed, improving the accuracy of indicators; at the same time, the control of data security and permissions will be more refined, and the management of resources will be more standardized. To a certain extent, it can also reduce the cost of data development.

[0112] Exemplarily, referring to Table 2, Table 2 is an industry operation indicator table. Table 2 shows the classification, indicator definitions, units, and calculation methods of some indicators in industry operation. The embodiments of the present application formulate corresponding industry operation indicators according to the data characteristics and operation purposes of the transportation industry, which can play a certain guiding role in aspects such as operators changing operation strategies and increasing enterprise benefits. For example, according to the indicator of "passenger satisfaction" shown in Table 2, the service quality and level of the enterprise can be understood to a certain extent. Thus, based on the results presented by this indicator, it can be used to guide whether to continue maintaining the current service level or whether to formulate corresponding systems to standardize and enhance the service level, etc.

[0113] Table 2 Industry Operation Indicator Table

[0114]

[0115]

[0116] It should be noted that only some indicators in the industry operation indicators are exemplarily shown in Table 2. The industry operation indicators also include line network density, number of line networks per 10,000 people, station density, station coverage rate, number of stations per 10,000 people, urban rail transit passenger sharing rate, etc., which are not enumerated in the embodiments of the present application.

[0117] Exemplarily, referring to Figure 10 , Figure 10 is an example diagram of the energy consumption indicator table provided by the embodiments of the present application. As Figure 10 shown, the energy consumption indicator table shows some indicators of urban rail transit and the corresponding categories and units of each indicator. The embodiments of the present application formulate corresponding energy consumption indicators according to the data characteristics and operation purposes of the transportation industry, which can standardize the data while also being able to understand the energy consumption situation through the energy consumption indicators, and can upgrade and improve the energy consumption reduction strategy according to the energy consumption situation and guide the cause investigation when the energy consumption is high, etc.

[0118] It should be noted that Figure 10 only shows some indicators in the energy consumption indicators exemplarily. The energy consumption indicators may also include other energy consumptions, which are not enumerated in the embodiments of the present application.

[0119] Exemplarily, referring to Table 3, which is a commercial operation indicator table. As shown in Table 3, the commercial operation indicator table shows the content of traffic data label classification, data fields, field definitions, and value requirements. Exemplarily, in the embodiments of the present application, corresponding commercial operation indicators are formulated according to the data characteristics and operation purposes of the transportation industry, etc. Passenger travel information can be understood through the commercial operation indicators, and passenger flow prediction can be carried out based on the passenger travel information, and the adjustment of relevant systems can be assisted, etc.

[0120] It should be noted that only some of the commercial operation indicators are exemplarily shown in Table 3. The commercial operation indicators may also include the non-riding payment amount in 7 days, the payment method distribution in 30 days, the number of times of using the travel rights issued by the platform in 30 days, etc. The embodiments of the present application do not enumerate them here.

[0121] Table 3 Commercial Operation Indicator Table

[0122]

[0123]

[0124] Among them, data quality management is mainly implemented based on DolphinScheduler. The data quality in the transportation industry focuses on aspects such as data accuracy, uniqueness, integrity, consistency, timeliness, and effectiveness. Referring to Figure 11 , Figure 11 is an example diagram of the front-end interface including data quality management provided by the embodiments of the present application. As shown by Figure 11 , data quality includes 4 modules: quality rules, quality tasks, task monitoring, and quality dashboard. Among them, quality rules include uniqueness verification, field length verification, regular expression, enumerated value verification, table row number verification, null value verification, multi-table accuracy verification, custom SQL, timeliness verification, two-table value comparison, etc. The rule types include single-table detection, multi-table detection, two-table value comparison, and custom SQL verification, etc. Data quality management mainly sets quality rules for transportation industry data through quality rule management. When the data to be verified in the transportation industry data triggers the preset threshold condition, a quality alarm is issued through quality monitoring management.

[0125] Exemplarily, in the quality rule setting, after selecting the data source to be verified, selecting the verification rule, setting the expected value, operation method, and threshold, the system will automatically select the corresponding SQL according to the selected verification rule to calculate the actual value, and then compare the actual value and the expected value in the set operation method. If the comparison result triggers the threshold comparison condition, a quality alarm will be issued for the original scheduling task in the quality monitoring. The alarm rules are divided into two types: strong rules and weak rules. Strong rules will stop the original scheduling task and no longer calculate regularly; weak rules will give an alarm but do not stop the previous task.

[0126] Data Development Module:

[0127] It is used to develop transportation industry data according to project requirements through offline development and real-time computing development. Among them, offline development supports development in multiple language environments, including commonly used development languages such as Python, Java, and SQL. The offline development level includes projects and workflows. In the workflow, it supports configuring task nodes and DAG scheduling, and supports viewing task execution logs and setting alarms. There are multiple projects corresponding to one operator, and one project can contain multiple workflows. Data between operators is completely isolated. System users can join different development projects, and different projects can access the same data source. It meets the different requirements of multi-layer data permission scopes for subway stations, lines, and line networks. For example, the line network needs to see the data permissions of all lines, that is, projects can be established for each line, and various data development tasks can be established under the project, corresponding to multiple workflows. The system administrator can view different projects, that is, the line network administrator can correspond to the system administrator role and view the development task status of different lines.

[0128] Data Service Module:

[0129] The data service module includes interface management and application management. Among them, interface management supports configuring interface sets under different projects and interfaces under the set. After the interface is configured, in application management, it supports authorizing different applications for different interfaces. When authorizing an interface, it supports setting the start and end times of interface calls, the frequency of interface calls, the row and column permissions of interface calls. The data service module provides the database tables accessed by the system as data assets, quickly generates APIs, and can configure input parameters, return parameters, sorting parameters, etc. through the interface.

[0130] Specifically, taking the wizard mode as an example, the usage process of interface management is described. The usage process of interface management includes:

[0131] (1) Create an API set;

[0132] (2) Create an API under the selected set and configure basic information: including API name, API set, interface path, version number, and API description;

[0133] (3) Select the data source, including generation method, data source type, data source name, database name, and table name;

[0134] (4) Configure API parameters, including input parameters, return parameters, request parameters, and sorting parameters;

[0135] (5) After the API parameters are configured, click "Publish" in the API management list to publish the API;

[0136] (6) Binding application: Only APIs in the published state can be authorized for application. After the release, the API cannot be edited and can only be edited after it is offline.

[0137] (7) Unbinding applications: To unbind an API from an application, click the API name, find the list of bound applications, and unbind each one individually.

[0138] For example, refer to Figure 12 , Figure 12 This is an example diagram of the interface management display interface provided by the embodiment of the present application. Figure 12 As shown, the interface management display interface includes interface name, request path, collection name, publishing status, update time, creation time and other contents.

[0139] In the embodiment of the present application, when a new interface needs to be added, if the interface set to which the new interface belongs already exists, select the interface set and add the new interface under the interface set; if the interface set to which the new interface belongs does not exist, click "Add new interface set" to add it. Figure 13 , Figure 13 : is an example diagram of the display interface of the newly added interface set provided in the embodiment of the present application. Figure 13 As shown in the figure, when adding an interface set, you need to fill in the interface set name, interface set path, belonging project and interface set description. Figure 14 , Figure 14 This is an example diagram of the query new interface set display interface provided by the embodiment of the present application. Figure 14 As shown in the figure, after adding an interface set, select the interface set name to view all interfaces under the interface set.

[0140] In the embodiment of the present application, a new interface under the interface set can be added through "Add interface", and the new interface includes filling in basic information, data source, input parameter definition, return parameter, request parameter, and sorting parameter information.

[0141] Among them, refer to Figure 15 , Figure 15 is an example diagram of the basic information display interface provided in the embodiment of the present application, such as Figure 15 As shown, the basic information needs to fill in the API name, API collection, interface path, version number and API description.

[0142] Reference Figure 16 , Figure 16 is an example diagram of the data source display interface provided in the embodiment of the present application, such as Figure 15As shown, the data source needs to fill in the API generation method (wizard mode, SQL mode), data source type, data source name, database name, and table name. Specifically, in the data source, the interface data generation method (wizard model / SQL mode), data source type (MySQL database, Doris database), data source name (the specific name of the data source for accessing the system, such as user label data), database name (the specific database name under the selected data source), and table name (the data table under the selected database) can be selected.

[0143] Refer to Figure 17 , Figure 17 is an example diagram of the input parameter definition display interface provided by the embodiment of the present application. As Figure 17 shown, the input parameter definition needs to fill in the parameter name, parameter type, select whether it is required, and give the parameter default value. After the input parameter definition, there are two switches for result caching and apiToken authentication. Among them, after the data result caching switch is turned on, the API query result will be cached by default for 5 minutes. The API supports two authentication methods: APItoken and application. If the apiToken authentication switch is turned on, the API can be called through apiToken, and only the fields apiToken, appkey, and version need to be added in the header (this type of authentication method is applicable to API usage scenarios with low security requirements such as data reports and data dashboards).

[0144] Refer to Figure 18 , Figure 18 is an example diagram of the return parameter display interface provided by the embodiment of the present application. As Figure 18 shown, the return parameter supports selecting the fields that need to be used as return parameters within the range of the data source - database - table configured in the interface, which is equivalent to the result of a SQL statement query.

[0145] Refer to Figure 19 , Figure 19 is an example diagram of the request parameter display interface provided by the embodiment of the present application. As Figure 19 shown, the request parameter supports selecting the fields that need to be used as request parameters within the range of the data source - database - table configured in the interface. The parameter name, bound field, parameter type, operator, description, and operation can be displayed in the request parameter.

[0146] Refer to Figure 20 , Figure 20 is an example diagram of the request parameter display interface provided by the embodiment of the present application. As Figure 20 shown, the sorting parameter supports selecting the fields that need to be used as sorting parameters within the range of the data source - database - table configured in the interface, which is equivalent to the fields added after order by in a SQL statement.

[0147] In the embodiments of the present application, after saving an interface, the interface can be published in a list. After the interface is published, it can be authorized for use by specific applications. Refer to Figure 21 , Figure 21 which is an example diagram of the interface authorization display interface provided by the embodiments of the present application. As shown in Figure 21 , the interface call deadline can be set to permanent or within a certain time range; the number of interface calls can be set to limit the number of calls per day; row permissions are to select the range of the number of rows returned by the interface within the range of the interface return parameters, and support selecting equal to, not equal to, less than or equal to, greater than or equal to, greater than, less than the current return fields; column permissions are to select the column range of the interface return parameters within the range of the interface return parameters, and only some fields can be returned.

[0148] In the embodiments of the present application, refer to Figure 22 , Figure 22 which is an example diagram of the new application display interface provided by the embodiments of the present application. As shown in Figure 22 , through application management, application information can be added and edited. When adding, the application name, application attribution project, and application description need to be filled in. After adding an application, an appkey and appsecret will be generated for authentication of interface calls.

[0149] Refer to Figure 23 , Figure 23 which is an example diagram of the view authorization display interface provided by the embodiments of the present application. As shown in Figure 23 , one application can be associated with multiple APIs. Clicking "View Authorization" can view all the APIs associated with the current application.

[0150] In the embodiments of the present application, operations such as editing application information and deleting applications can also be achieved through application management.

[0151] Analysis and Modeling Module:

[0152] The analysis and modeling module includes model development, model deployment, and model evaluation of algorithm models. The algorithms support passenger flow prediction, parking fee estimation algorithms, etc., and these algorithms have extensive application scenarios in the transportation industry.

[0153] The analysis and modeling module provides a rich algorithm component library and a convenient operation framework, covering the entire development process of data processing, feature engineering, model training, and model prediction.

[0154] The data source types include: reading and writing local files, distributed file systems (HDFS, OSS), and at the same time support four databases of Hive, MySQL, Derby, and SQLite. Because it is based on the Flink algorithm platform, it supports Flink Table as input and output. At the same time, when performing streaming calculations, it supports reading and writing the message queue of Kafka.

[0155] Operation and maintenance monitoring module:

[0156] Referring to Table 4, the operation and maintenance monitoring module can monitor the number of scheduling tasks, total tasks, planned execution tasks, unfinished tasks, completion rate, completion time, average completion time, planned execution instances, failed instances, failed instances, and unprocessed failed instances of the current system. At the same time, it can view the overview of the instance operation of the current system tasks and generate corresponding overview diagrams.

[0157] Table 4 Current system task operation situation table

[0158]

[0159]

[0160] Data application module:

[0161] The data application module supports BI analysis and algorithm application.

[0162] Referring to Figure 24 , Figure 24 is one of the interface display example diagrams of the BI analysis provided by the embodiments of the present application. As shown by Figure 24 , BI analysis can customize the selection of required rows and columns and data filtering ranges, support querying and exporting data, and reduce duplicate development of reports.

[0163] The algorithm application includes an algorithm parameter input interface, which mainly supports the calculation of algorithm parameters on the operation side and industry side. Referring to Figure 25 , Figure 25 is the interface example diagram of the operation side rights and interests pricing algorithm provided by the embodiments of the present application. As shown by Figure 25 , the rights and interests configuration includes inputting rights and interests scenarios, target systems, coupon characteristics, user factors, fare factors, etc.

[0164] In the embodiments of the present application, as shown by Figure 1 , the system further includes a data storage module, and the data storage module is used to store traffic industry data into corresponding databases as needed.

[0165] Referring to Figure 25 , the embodiments of the present application also propose a traffic industry data governance method, including but not limited to steps S2601 to S2609.

[0166] Step S2601, collecting traffic industry data sources;

[0167] Step S2602, integrating traffic industry data sources together through the data integration module based on data access, data synchronization, and scheduling configuration to support secure data access;

[0168] Step S2603: Through the data security module, perform module deployment management, user hierarchy management, permission management for each level, and project management according to user requirements. Project management includes managing basic project information, project configured personnel, project roles, and project data permissions.

[0169] Step S2604: Through the data governance module, perform metadata management, data asset management, data metric management, and data quality management on transportation industry data.

[0170] Step S2605: Through the data development module, perform transportation industry data development based on the offline development and real-time computing development methods according to project requirements.

[0171] Step S2606: Through the analysis and modeling module, perform model development, model deployment, and model evaluation on transportation industry data.

[0172] Step S2607: Through the data service module, configure the interface set under different projects, the interfaces under the interface set, and perform application authorization management on the interfaces.

[0173] Step S2608: Through the operation and maintenance monitoring module, monitor the running status of the current system tasks and the running overview of instances.

[0174] Step S2609: Through the data application module, perform business intelligence analysis and algorithm application on transportation industry data.

[0175] In the embodiment of the present application, after collecting the transportation industry data source, through Figure 1 the data governance system shown can effectively govern transportation industry data, improve data processing efficiency, ensure data security and quality, and reduce governance costs.

[0176] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings. However, this does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall fall within the scope of the rights of the embodiments of the present application.

Claims

1. A traffic industry data governance system, characterized in that, The system realizes various different data governance scenarios for different subway operators, and isolates the traffic industry data corresponding to different subway operators. The system includes: a data security module, a data integration module, a data governance module, a data development module, an analysis and modeling module, a data service module, an operation and maintenance monitoring module, and a data application module. Each module can be used independently or in any combination according to different user requirements; The data security module is used for module deployment management, user level management, permission management at each level, and project management according to user requirements. The project management includes managing project basic information, project configuration personnel, project roles, and project data permissions; Among them, the data security module is further used for data isolation between subway operators and functional isolation of each level management of subway operators through user level management and permission management at each level under the subway operator; The data integration module is used to integrate traffic industry data sources together through data access, data synchronization, and scheduling configuration to support secure data access; The data governance module is used for metadata management, data asset management, data index management, and data quality management of traffic industry data; the data development module is used for traffic industry data development according to project requirements through offline development and real-time computing development; Among them, the offline development includes projects and workflows. Task nodes and DAG scheduling are supported in the workflow; there are multiple projects corresponding to the subway operator, and multiple workflows are included in one project; To meet the different needs of multi-layer data of subway stations, lines, and line networks, the line network needs to see the data permissions of all lines. Projects are established for each line, and various data development tasks are established under the project. The data development tasks correspond to multiple workflows; The line network administrator corresponding to the system administrator can view different projects, that is, the development task status of different lines; the data service module is used to configure the interface set under different projects, the interfaces under the interface set, and perform application authorization management on the interfaces; Among them, the data service module includes interface management and application management; the interface management is used to configure the interface set under different projects and the interfaces under the interface set, including: Creating an interface set for the system application program; Under the selected interface set, creating an interface in the selected interface set and configuring basic information, where the basic information includes interface name, interface belonging set, interface path, version number, and interface description; Determining data source information, where the data source information includes interface data generation method, data source type, data source name, database name, and table name; Configuring interface parameters, where the interface parameters include input parameter definition, return parameter, request parameter, and sorting parameter; After the interface parameters are configured, publishing the interface so that the interface can be authorized for application; The application management is used for application authorization management of each published interface, including: Adding interface authorization applications and editing application information, where the application information includes application name, application belonging project, and application description; Query all interfaces associated with each application and the authorization information corresponding to all interfaces; The analysis and modeling module is used to develop models, deploy models, and evaluate models for the transportation industry data; The operation and maintenance monitoring module is used to monitor the running status of the current system tasks and the general running status of instances; The data application module is used to perform business intelligence analysis and algorithm applications on the transportation industry data.

2. The system according to claim 1, wherein The user level management includes: Constructing user levels, which include super administrators, system administrators, project administrators, and ordinary users; Setting and managing the permissions corresponding to each user level.

3. The system according to claim 1, wherein Metadata includes business metadata and technical metadata, and the metadata management includes: Managing the basic information, field information, lineage, and DDL changes of the business metadata and technical metadata. The basic information includes data table information, business information, and storage information. The field information includes field name, field type, field size, whether it is null, precision, field comment, calculation method, whether it is indexed, and partition fields. The lineage includes the upstream and downstream generation relationships between system access data nodes, where the data nodes include upstream nodes, intermediate nodes, and downstream nodes. The DDL changes include the operation information of data tables.

4. The system according to claim 1, wherein The data asset management includes: Querying various types of offline data and real-time data in the transportation industry by searching system application programming interfaces, or accessing collaborative data tables, or real-time data Topics.

5. The system according to claim 1, wherein Data metrics include industry operation metrics, energy consumption metrics, and business operation metrics, and the data metric management includes: Managing atomic metrics, derived metrics, time periods, and modifiers for the industry operation metrics, energy consumption metrics, and business operation metrics. Among them, the derived metrics are automatically generated individually or in batches through the atomic metrics, time periods, and modifiers.

6. The system according to claim 1, wherein The data quality management includes quality rule management and quality monitoring management; The quality rule management is used to set quality rules for the transportation industry data. The quality rules include uniqueness verification, field length verification, regular expressions, enumeration value verification, table row count verification, null value verification, multi-table accuracy verification, custom SQL, timeliness verification, and two-table value comparison; The quality monitoring management is used to issue a quality warning when the data to be verified in the transportation industry data triggers a preset threshold condition.

7. The system according to claim 1, wherein The interface management is also used to perform the following operations: When the target interface set to which the interface to be added belongs already exists, select the target interface set and add an interface under the target interface set; When the target interface set to which the interface to be added belongs does not exist, create the target interface set and add an interface under the newly created target interface set.

8. The system according to claim 1, wherein The system further includes: A data storage module for storing the transportation industry data in the corresponding database as needed.

9. A method for data governance in the transportation industry, characterized in that, The method realizes various different data governance scenarios for different subway operators and isolates the transportation industry data corresponding to different subway operators, including: Collecting the transportation industry data source; Integrate the transportation industry data sources together through a data integration module based on data access, data synchronization and scheduling configuration to support secure data access; Through the data security module, module deployment management, user level management, authority management at each level and project management are performed according to user needs. The project management includes the management of basic project information, project configuration personnel, project roles and project data permissions; The data security module is further used to perform data isolation between the subway operators and functional isolation of management at each level of the subway operator through user level management and authority management at each level under the subway operator; Through the data governance module, metadata management, data asset management, data indicator management and data quality management are carried out on transportation industry data; Develop transportation industry data based on project requirements through data development modules based on offline development and real-time computing development; The offline development includes projects and workflows, and the workflow supports configuration of task nodes and DAG scheduling; the subway operator corresponds to multiple projects, and one project contains multiple workflows; To meet the different needs of subway stations, lines, and network multi-layer data, the network needs to see the data permissions of all lines, establish a project for each line, and establish various data development tasks under the project. The data development tasks correspond to multiple workflows; The line network administrator corresponding to the system administrator can view different projects, that is, the development tasks of different lines; Perform model development, model deployment and model evaluation on the transportation industry data through analytical modeling modules; Configure the interface sets under different projects and the interfaces under the interface sets through the data service module and perform application authorization management on the interfaces; The data service module includes interface management and application management; the interface management is used to configure interface sets under different projects and interfaces under the interface sets, including: Create an interface set for system applications; Under the selected interface set, create the interface in the selected interface set and configure basic information, the basic information including the interface name, the set to which the interface belongs, the interface path, the version number and the interface description; Determine data source information, the data source information includes interface data generation method, data source type, data source name, database name and table name; Configure interface parameters, including input parameter definition, return parameter, request parameter, and sort parameter; After the interface parameter configuration is completed, the interface is released so that the interface is authorized for application; The application management is used to perform application authorization management on each of the published interfaces, including: Added a new interface to authorize applications and edit application information, including application name, application belonging project and application description; Query all interfaces associated with each application and the authorization information corresponding to all interfaces; Monitor the current system task operation status and instance operation overview through the operation and maintenance monitoring module; The data application module is used to perform business intelligence analysis and algorithm application on the transportation industry data.

Images