Urban rail transit network data processing method, electronic device and storage medium

Abstract

The application provides a method for processing urban rail transit network data, an electronic device and a storage medium, wherein the method comprises: obtaining unstructured demand information input by a user, extracting data governance information from the unstructured demand information, and determining a governance demand semantic tuple; determining a current scene feature tuple according to a governance object, a business scenario and a pre-constructed global data asset directory, and generating a data governance scheme document based on the current scene feature tuple and the governance demand semantic tuple by a central large model; determining task arrangement information and a database operation statement according to the unstructured demand information, the data governance scheme document and the global data asset directory; generating an executable task package according to the task arrangement information, the database operation statement, the global data asset directory and the data governance scheme document, and executing the executable task package. The application can realize automatic arrangement and closed-loop execution of data governance tasks.

Description

Technical Field

[0001] This application relates to the field of data governance technology, and more specifically, to a method for processing urban rail transit network data, electronic equipment, and storage medium. Background Technology

[0002] Urban rail transit has entered the stage of network-based operation. As the core hub for data aggregation, processing, and services, network-level data centers undertake important functions such as supporting key businesses like operation scheduling, passenger flow forecasting, equipment maintenance, and clearing and settlement. With the continuous expansion of the network scale and the continuous improvement of its intelligence level, how to efficiently, intelligently, and securely achieve automated data governance of network-level data centers has become a key issue that urgently needs to be addressed in the digital transformation of the urban rail transit industry.

[0003] Current urban rail transit network data center construction technologies generally adopt a data platform, business requirements, and data governance development model. The data platform provides data access, processing, and analysis capabilities, while data governance development relies on business inventory and is driven by business needs.

[0004] However, existing technologies typically rely on manual processes to complete tasks such as data standard setting, quality rule configuration, and data model design, which suffers from problems such as rigid rules, excessive reliance on manual intervention, semantic fragmentation, difficulties in collaboration, and fragmented governance tasks. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a method, electronic device, and storage medium for urban rail transit network data processing, in order to solve the problems of rigid rules, excessive reliance on manual intervention, semantic fragmentation, difficulty in collaboration, and fragmented governance tasks in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides a method for processing urban rail transit network data, the method comprising: The system acquires unstructured requirement information input by the user, extracts data governance information from the unstructured requirement information, and determines governance requirement semantic tuples based on the data governance information and the pre-trained central big model. The governance requirement semantic tuples include: business scenario, governance object, governance indicator, quantitative target and implicit constraint. Based on the governance object, the business scenario, and the pre-constructed global data asset catalog, the current scenario feature tuple is determined, and the central big model generates a data governance solution document based on the current scenario feature tuple and the governance requirement semantic tuple. Based on the unstructured requirement information, the data governance solution document, and the global data asset catalog, task orchestration information and database operation statements are determined. The task orchestration information includes: business semantic structured data, field mapping recommendation information, and logical model description information. Based on the task orchestration information, the database operation statements, the global data asset catalog, and the data governance scheme document, an executable task package is generated and executed.

[0007] Optionally, determining the current scenario feature tuple based on the governance object, the business scenario, and a pre-constructed global data asset catalog includes: Based on the governance object, a set of data sources corresponding to the governance object is obtained by matching from a pre-constructed global data asset catalog, and the set of data sources includes at least one data source. Based on the governance object and the data source set, at least one metadata information corresponding to the governance object is read from the global data asset catalog; Based on the business scenario, a current scenario feature tuple is extracted from each of the metadata information. The current scenario feature tuple includes at least: data volume, data source, data change type, and security level.

[0008] Optionally, the step of generating a data governance scheme document by the central large model based on the current scene feature tuple and the governance requirement semantic tuple includes: The central big model determines at least one candidate data governance strategy based on the current scene feature tuples and a pre-constructed governance strategy candidate pool. The central big model determines the target data governance strategy from among the candidate data governance strategies based on the governance indicators, the quantitative objectives, and the implicit constraints. The central big model generates a data governance scheme document based on the target data governance strategy and the semantic tuple of governance requirements.

[0009] Optionally, determining the task orchestration information and database operation statements based on the unstructured requirement information, the data governance solution document, and the global data asset catalog includes: The unstructured requirement information is subjected to business semantic extraction to obtain business semantic structured data; Based on the business semantic structured data, the global data asset catalog, and the pre-constructed industry knowledge base, determine the field mapping recommendation information; Based on the business semantic structured data, the field mapping recommendation information, and the target data governance strategy in the data governance solution document, determine the logical model description information and database operation statements.

[0010] Optionally, determining the field mapping recommendation information based on the business semantic structured data, the global data asset catalog, and the pre-constructed industry knowledge base includes: Based on the business semantic structured data, generate at least one query vector; Based on each of the query vectors, at least one first candidate field mapping information is retrieved from the global data asset catalog; Based on each of the query vectors, at least one second candidate field mapping information is retrieved from the industry knowledge base; The central large model generates field mapping recommendation information based on the mapping information of each of the first candidate fields and the mapping information of each of the second candidate fields.

[0011] Optionally, determining the logical model description information and database operation statements based on the business semantic structured data, the field mapping recommendation information, and the target data governance strategy in the data governance solution document includes: Based on the business semantic structured data, a modeling pattern is determined, and based on the modeling pattern, a target logical model description template and a target database statement template are determined from multiple preset template structures. Based on the field mapping recommendation information and the target data governance strategy in the data governance scheme document, the target logical model description template and the target database statement template are populated to obtain the logical model description information and the database operation statement.

[0012] Optionally, generating an executable task package based on the task orchestration information, the database operation statements, the global data asset catalog, and the data governance solution document includes: Based on the task orchestration information, the database operation statements, the global data asset catalog, and the data governance scheme document, the central big model obtains a sub-task list and sub-task dependencies. The sub-task list includes at least one sub-task and a task description corresponding to each sub-task. Obtain the script templates corresponding to each subtask, and the central big model generates the scripts corresponding to each subtask based on the task descriptions and script templates corresponding to each subtask. Based on the task orchestration information, the database operation statements, the global data asset catalog, and the data governance solution document, a configuration file is generated; An executable task package is generated based on the subtask list, the subtask dependencies, the scripts corresponding to each subtask, and the configuration file.

[0013] Optionally, after executing the executable task package, the method further includes: Obtain execution status and quality indicators; A governance score is determined based on the execution status, the quality indicators, and the target data governance strategy in the data governance solution document. The governance outcome is determined based on the governance score and the quantitative target. If the governance result is satisfactory, the central large model will generate a governance effect report. If the governance result is unsatisfactory, the central big model will adjust the executable task package based on the governance score and re-execute the adjusted executable task package.

[0014] Secondly, another embodiment of this application provides an urban rail transit network data processing device, the device comprising: The acquisition module is used to acquire unstructured requirement information input by the user, extract data governance information from the unstructured requirement information, and determine the governance requirement semantic tuple based on the data governance information and the pre-trained central big model. The governance requirement semantic tuple includes: business scenario, governance object, governance indicator, quantitative target and implicit constraint. The first determining module is used to determine the current scenario feature tuple based on the governance object, the business scenario, and the pre-constructed global data asset catalog, and the central big model generates a data governance scheme document based on the current scenario feature tuple and the governance requirement semantic tuple. The second determining module is used to determine task orchestration information and database operation statements based on the unstructured requirement information, the data governance scheme document, and the global data asset catalog. The task orchestration information includes: business semantic structured data, field mapping recommendation information, and logical model description information. The generation module is used to generate an executable task package based on the task orchestration information, the database operation statements, the global data asset catalog, and the data governance scheme document, and to execute the executable task package.

[0015] Optionally, the first determining module is specifically used for: Based on the governance object, a set of data sources corresponding to the governance object is obtained by matching from a pre-constructed global data asset catalog, and the set of data sources includes at least one data source. Based on the governance object and the data source set, at least one metadata information corresponding to the governance object is read from the global data asset catalog; Based on the business scenario, a current scenario feature tuple is extracted from each of the metadata information. The current scenario feature tuple includes at least: data volume, data source, data change type, and security level.

[0016] Optionally, the first determining module is specifically used for: The central big model determines at least one candidate data governance strategy based on the current scene feature tuples and a pre-constructed governance strategy candidate pool. The central big model determines the target data governance strategy from among the candidate data governance strategies based on the governance indicators, the quantitative objectives, and the implicit constraints. The central big model generates a data governance scheme document based on the target data governance strategy and the semantic tuple of governance requirements.

[0017] Optionally, the second determining module is specifically used for: The unstructured requirement information is subjected to business semantic extraction to obtain business semantic structured data; Based on the business semantic structured data, the global data asset catalog, and the pre-constructed industry knowledge base, determine the field mapping recommendation information; Based on the business semantic structured data, the field mapping recommendation information, and the target data governance strategy in the data governance solution document, determine the logical model description information and database operation statements.

[0018] Optionally, the second determining module is specifically used for: Based on the business semantic structured data, generate at least one query vector; Based on each of the query vectors, at least one first candidate field mapping information is retrieved from the global data asset catalog; Based on each of the query vectors, at least one second candidate field mapping information is retrieved from the industry knowledge base; The central large model generates field mapping recommendation information based on the mapping information of each of the first candidate fields and the mapping information of each of the second candidate fields.

[0019] Optionally, the second determining module is specifically used for: Based on the business semantic structured data, a modeling pattern is determined, and based on the modeling pattern, a target logical model description template and a target database statement template are determined from multiple preset template structures. Based on the field mapping recommendation information and the target data governance strategy in the data governance scheme document, the target logical model description template and the target database statement template are populated to obtain the logical model description information and the database operation statement.

[0020] Optionally, the generation module is specifically used for: Based on the task orchestration information, the database operation statements, the global data asset catalog, and the data governance scheme document, the central big model obtains a sub-task list and sub-task dependencies. The sub-task list includes at least one sub-task and a task description corresponding to each sub-task. Obtain the script templates corresponding to each subtask, and the central big model generates the scripts corresponding to each subtask based on the task descriptions and script templates corresponding to each subtask. Based on the task orchestration information, the database operation statements, the global data asset catalog, and the data governance solution document, a configuration file is generated; An executable task package is generated based on the subtask list, the subtask dependencies, the scripts corresponding to each subtask, and the configuration file.

[0021] Optionally, the generation module is further configured to: Obtain execution status and quality indicators; A governance score is determined based on the execution status, the quality indicators, and the target data governance strategy in the data governance solution document. The governance outcome is determined based on the governance score and the quantitative target. If the governance result is satisfactory, the central large model will generate a governance effect report. If the governance result is unsatisfactory, the central big model will adjust the executable task package based on the governance score and re-execute the adjusted executable task package.

[0022] Thirdly, another embodiment of this application provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the methods described in the first aspect above.

[0023] Fourthly, another embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the methods described in the first aspect above.

[0024] The beneficial effects of this application are as follows: By acquiring unstructured requirement information input by users, data governance information is extracted from the unstructured requirement information. Based on the data governance information and a pre-trained central big model, governance requirement semantic tuples are determined. Based on the governance object, business scenario, and a pre-constructed global data asset catalog, current scenario feature tuples are determined. The central big model generates a data governance solution document based on the current scenario feature tuples and governance requirement semantic tuples. This can automatically generate urban rail transit network data governance solutions, eliminating the heavy reliance on expert experience. Based on the unstructured requirement information, data governance solution document, and global data asset catalog, task orchestration information and database operation statements are determined. Based on the task orchestration information, database operation statements, global data asset catalog, and data governance solution document, an executable task package is generated and executed. This can automatically eliminate cross-system and cross-departmental data semantic conflicts, realize intelligent recommendation design of data standards and models, thereby achieving automated orchestration and closed-loop execution of data governance tasks, forming a self-optimizing governance link of problem identification, strategy generation, task execution, and effect verification. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram of a system architecture for an urban rail network data processing system provided in this application embodiment; Figure 2 A flowchart illustrating a method for processing urban rail network data provided in an embodiment of this application; Figure 3 A flowchart illustrating the process of determining feature tuples in the urban rail network data processing method provided in this application embodiment; Figure 4 This is a flowchart illustrating the process of generating a data governance scheme document in the urban rail network data processing method provided in this application embodiment. Figure 5 This is a flowchart illustrating the process of determining task scheduling information and database operation statements in the urban rail network data processing method provided in this application embodiment. Figure 6 This is a flowchart illustrating the process of determining field mapping recommendation information in the urban rail network data processing method provided in this embodiment of the application. Figure 7This is a flowchart illustrating the process of determining logical model description information and database operation statements in the urban rail network data processing method provided in this application embodiment. Figure 8 This is a schematic flowchart illustrating the generation of an executable task package in the urban rail network data processing method provided in this application embodiment. Figure 9 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0028] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0029] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0030] Current urban rail transit network data center construction technologies generally adopt a data platform, business requirements, and data governance development model. The data platform provides data access, processing, and analysis capabilities, while data governance development relies on business inventory and is driven by business needs.

[0031] However, existing technologies typically rely on manual processes to complete tasks such as data standard setting, quality rule configuration, and data model design, which suffers from problems such as rigid rules, excessive reliance on manual intervention, semantic fragmentation, difficulties in collaboration, and fragmented governance tasks.

[0032] Based on the aforementioned problems, this application proposes a method for processing urban rail transit network data. It acquires unstructured demand information input by users, extracts data governance information from this information, and determines governance demand semantic tuples based on the data governance information and a pre-trained central big model. Furthermore, it determines current scenario feature tuples based on the governance object, business scenario, and a pre-constructed global data asset catalog. The central big model then generates a data governance scheme document based on the current scenario feature tuples and the governance demand semantic tuples. This method can automatically generate urban rail transit network data governance schemes, reducing reliance on expert experience. Based on the unstructured demand information, the data governance scheme document, and the global data asset catalog, it determines task orchestration information and database operation statements. Finally, it generates and executes executable task packages based on these information, automatically eliminating cross-system and cross-departmental data semantic conflicts and enabling intelligent recommendation design of data standards and models. This achieves automated orchestration and closed-loop execution of data governance tasks, forming a self-optimizing governance chain encompassing problem identification, strategy generation, task execution, and effect verification.

[0033] First, the system architecture of the urban rail network data processing system involved in the urban rail network data processing method provided in the embodiments of this application will be described.

[0034] Figure 1 This is a schematic diagram of a system architecture for an urban rail network data processing system provided in an embodiment of this application, with reference to... Figure 1 As shown, the system includes a central large model engine, at least one edge actuator, at least one secure data agent, and an interaction layer.

[0035] The central large model engine is deployed on the management network. The central large model engine includes: the central large model, the industry knowledge base, the prompt word engineering module, the task orchestrator, and the effect evaluator.

[0036] For example, the central large model can be a large model pre-adapted using urban rail industry data, or a general large model, such as GPT-4 or Claude 3 Opus.

[0037] Specifically, the large model that is pre-adapted with urban rail transit industry data refers to selecting a general large model as a base, and then using supervised fine-tuning (SFT) and low-rank adaptation (LoRA) techniques to perform lightweight customization specific to the urban rail transit industry, so that the output content (code, standards, solutions) conforms to the professional context of the urban rail transit field.

[0038] For example, the base model is fine-tuned in advance using urban rail industry standard documents (such as GB / T 40484-2021, CJJ / T 315-2022, etc.), historical governance requirements documents and corresponding expert processing solutions, and industry best practice code. This enables the model to master the terminology system and business logic of the urban rail field. Based on the above fine-tuning, a lightweight LoRA adapter is trained for enterprise-specific rules (such as internal terminology abbreviations and specific desensitization requirements) and scenario-level rules (such as specific data granularity for passenger flow scenarios and fault code classification dictionaries for equipment scenarios). This achieves low-cost customization of a single base for multiple scenarios.

[0039] Each edge executor is deployed in a network region (e.g., production network, management network, external service network) and is responsible for local metadata scanning, task execution, and status reporting.

[0040] Each secure data agent is deployed in a different network area. The secure data agent is used to provide de-identification, encryption, and digest extraction functions between different network areas to ensure that the original data does not leave the domain.

[0041] The interaction layer provides a natural language interface, allowing business personnel to input requirements and view governance reports.

[0042] The following describes in detail the urban rail network data processing method provided in this application embodiment with reference to several embodiments.

[0043] Figure 2 This is a flowchart illustrating a method for processing urban rail network data provided in an embodiment of this application, with reference to... Figure 2 As shown, the executing entity of this method can be any electronic device with processing capabilities, such as the aforementioned urban rail network data processing system. The method includes: S201. Obtain unstructured requirement information input by the user, extract data governance information from the unstructured requirement information, and determine the governance requirement semantic tuple based on the data governance information and the pre-trained central big model.

[0044] Optionally, unstructured requirement information input by the user can be obtained. This unstructured requirement information can include natural language descriptions, Word / PDF requirement documents, meeting minutes, etc.

[0045] Optionally, after cleaning the unstructured demand information, data governance information can be extracted. This data governance information consists of governance objectives expressed in natural language.

[0046] For example, taking the passenger flow scenario as an example, the data governance information is "governing passenger flow data to achieve 99.5% availability".

[0047] Optionally, governance requirement prompts are generated based on data governance information and preset governance requirement understanding prompt templates. These prompts are then input into a pre-trained central big model, which infers the governance requirement semantic tuples based on the prompts.

[0048] For example, the prompt template for understanding governance requirements could be: "You are a senior data governance expert in the urban rail transit industry, proficient in analyzing data governance requirements of network-level data centers."

[0049]

Task Objective

Field Definitions and Data Types

Output Example

[0050] [Data Governance Information] Data governance information population field [Terminology Processing Guidelines] It is necessary to identify professional abbreviations in the urban rail transit industry (such as AFC, SCADA, clearing system, etc.) and expand them into complete Chinese terms; ambiguous business expressions should be standardized according to industry common sense.

[0051] [Implicit Constraint Reasoning Guidelines] In addition to the user's explicitly stated requirements, you also need to combine common sense in the urban rail transit industry and business scenarios to deduce the implicit constraints that the user has not explicitly stated but that must be met objectively. For example: the "passenger flow prediction" scenario implies that the data source comes from the AFC system; the "ticket clearing" scenario implies that passenger card numbers need to be anonymized; the description of "real-time" or "near real-time" implies that the data delay should be within minutes.

[0052]

Field Extraction Instructions

[0053]

Field Definitions and Data Types

[0054]

Output Example

[0055] The semantic tuple of governance requirements includes: business scenarios, governance objects, governance indicators, quantitative goals, and implicit constraints.

[0056] Specifically, the governance requirement semantic tuple is used to indicate macro-level governance requirements in the data governance process, describing information at the business objective level. The governance requirement semantic tuple can be presented in structured JSON format. `business_scenario` refers to the business domain to which this governance pertains. `governance_target` refers to the specific data object targeted by this governance. `governance_metric` refers to the metric dimension that this governance focuses on. `quantitative_goal` refers to the specific value that this governance metric needs to achieve. `implicit_constraints` refers to the technical limitations that are not explicitly stated but are implicit in the data governance information.

[0057] S202. Based on the governance object, business scenario, and pre-constructed global data asset catalog, determine the current scenario feature tuples, and the central big model generates a data governance solution document based on the current scenario feature tuples and governance requirement semantic tuples.

[0058] Optionally, after obtaining the semantic tuple of governance requirements, the feature tuple of the current scenario can be determined from the pre-constructed global data asset catalog based on the governance object and business scenario.

[0059] Among them, the current scene feature tuple describes information at the data status level, which is used to indicate the objective state description of the data to be governed.

[0060] The global data asset catalog is a unified metadata index list of all data resources in the urban rail transit network-level data center. It does not store specific data rows, but records all descriptive information about what the data looks like, where it is, and what its characteristics are.

[0061] Specifically, the global data asset catalog may include technical metadata, business metadata, and management metadata.

[0062] For example, technical metadata may include: data source information such as network region, business system, and database type; table structure information such as table name, field name, field type, field length, primary key / foreign key; statistical information such as approximate values, data size, and update time; and partition information such as partition fields and partitioning strategies.

[0063] For example, business metadata may include: business descriptions such as table comments, field comments, and business meanings; category tags for the business domain; and data owner information such as the responsible department / contact person.

[0064] For example, managing metadata may include: sensitivity tags such as field-level sensitivity levels; quality tags; and lineage.

[0065] Optionally, the central large model can infer and generate a data governance solution document based on the current scenario feature tuples and governance requirement semantic tuples.

[0066] S203. Based on unstructured requirements information, data governance solution documents, and the global data asset catalog, determine task orchestration information and database operation statements.

[0067] Optionally, after obtaining the data governance solution document, data integration and governance tasks can be orchestrated based on unstructured requirements information, the data governance solution document, and the global data asset catalog, generating task orchestration information and database operation statements.

[0068] The task orchestration information includes: business semantic structured data, field mapping recommendation information, and logical model description information.

[0069] Specifically, business semantic structured data is used to indicate metrics in unstructured requirement information, how those metrics are calculated, and the dimensions by which they are viewed. Business semantic structured data can include: metrics, the dimensions corresponding to those metrics, the granularity of those metrics, and clues about the data source.

[0070] The field mapping recommendation information consists of conversion rules between the original fields of the source table and the target standard fields. This information includes: source field, target field, data type, conversion type, and standard source.

[0071] The logical model description information is used to describe the structural blueprint of the data warehouse, that is, how many tables need to be created, what fields each table has, and how the tables are related.

[0072] Among them, the database operation statements are the executable SQL statements used to create the table structures in the above logical model description information.

[0073] S204. Based on the task orchestration information, database operation statements, global data asset catalog, and data governance solution document, generate an executable task package and execute the executable task package.

[0074] Optionally, the task orchestration information, database operation statements, global data asset catalog, and data governance solution documents can be orchestrated and processed to convert them into an executable task package for execution.

[0075] The executable task package is a code package that can be independently distributed and run, including all the contents required to complete this data governance task on the edge executor, such as task orchestration logic, executable scripts, and runtime configuration.

[0076] For example, the executable task package is sent to the edge executor in the corresponding network region for execution, depending on where the data is located.

[0077] In this embodiment, by acquiring unstructured requirement information input by the user, data governance information is extracted from the unstructured requirement information. Based on the data governance information and a pre-trained central big model, governance requirement semantic tuples are determined. Based on the governance object, business scenario, and a pre-constructed global data asset catalog, current scenario feature tuples are determined. The central big model generates a data governance solution document based on the current scenario feature tuples and governance requirement semantic tuples. This can automatically generate urban rail network data governance solutions, eliminating the heavy reliance on expert experience. Based on the unstructured requirement information, data governance solution document, and global data asset catalog, task orchestration information and database operation statements are determined. Based on the task orchestration information, database operation statements, global data asset catalog, and data governance solution document, an executable task package is generated and executed. This can automatically eliminate cross-system and cross-departmental data semantic conflicts, realize intelligent recommendation design of data standards and models, thereby achieving automated orchestration and closed-loop execution of data governance tasks, forming a self-optimizing governance link of problem identification, strategy generation, task execution, and effect verification.

[0078] In one possible implementation, Figure 3 This is a flowchart illustrating the process of determining the current scene feature tuple in the urban rail network data processing method provided in this application embodiment, with reference to... Figure 3 As shown, in S202 above, based on the governance object, business scenario, and pre-constructed global data asset catalog, the current scenario feature tuple is determined, including: S301. Based on the governance object, match the data source set corresponding to the governance object from the pre-constructed global data asset catalog.

[0079] Optionally, the governance object in the governance requirement semantic tuple is used as the search keyword to perform a matching search in a pre-constructed global data asset catalog to find all data sources associated with the governance object, thus forming a data source set. The data source set includes at least one data source.

[0080] For example, matching retrieval may include: exact matching, semantic matching, and relational matching.

[0081] For example, each data source in the data source set contains at least the following information: data source name, network area (production network / management network / external service network), connection method (JDBC / API / file), and business system (AFC / SCADA / signal / clearing, etc.).

[0082] S302. Based on the governance object and the data source set, read at least one metadata information corresponding to the governance object from the global data asset catalog.

[0083] Optionally, based on the data source set and the specific data table or field range that the governance object points to, at least one metadata information corresponding to the governance object can be read from the global data asset catalog.

[0084] For example, depending on the category, metadata information includes: structural metadata information, statistical metadata information, descriptive metadata information, administrative metadata information, and relational metadata information.

[0085] For example, structural metadata information may include: table name, field name, field type, field length, and primary / foreign key definition. Statistical metadata information may include: approximate number of table records, data size, partition information, and update timestamp. Descriptive metadata information may include: table comments, field comments, and business descriptions. Management metadata information may include: network region, business system, and sensitivity tags. Relational metadata information may include: data lineage, such as upstream source table and downstream target table.

[0086] S303. Based on the business scenario, extract the current scenario feature tuple from the metadata information.

[0087] Optionally, the metadata information can be integrated, and the key features of the current governance scenario can be extracted from the metadata information in combination with the business scenario to form the current scenario feature tuple.

[0088] The current scenario feature tuple includes at least: data volume, data source, data change pattern, and security level.

[0089] For example, approximate values ​​and data sizes are extracted from various metadata information to obtain the data volume.

[0090] For example, the business system and data source type are extracted from the metadata information to obtain the data source.

[0091] For example, the update frequency and data change pattern (e.g., new / update / full overwrite) are extracted from various metadata information, and combined with the preset data update and data change mapping relationship to obtain the data change type.

[0092] For example, if the business scenario itself exhibits obvious abrupt changes in time patterns (such as morning and evening rush hours), and the timestamp distribution in the metadata shows a clear peak characteristic, the data change type is high volatility. If the data update frequency is stable, the timestamp distribution is uniform, and there are no obvious sudden peaks, the data change type is low volatility. If the data changes are related to specific events (such as holidays or large-scale events), the volatility is between the above two types, and the data change type is medium volatility.

[0093] For example, sensitivity tags and field-level desensitization markers are extracted from various metadata information, and the security level is obtained by combining them with a preset sensitivity and security mapping relationship.

[0094] For example, a data source containing highly sensitive tagged fields (such as ID card numbers, bank card numbers, and mobile phone numbers) has a security level of high sensitivity. A data source containing moderately sensitive tagged fields (such as device numbers, station codes, and other business identifiers) has a security level of moderate sensitivity. A data source where all fields are either low-sensitivity or not marked as sensitive has a security level of low sensitivity.

[0095] For example, continuing with the passenger flow scenario, the current scenario feature tuple can be {"data_volume": "Daily hundreds of millions of records", "data_source": "AFC system (single source)", "data_change_pattern": "High volatility (sudden changes during morning and evening peak hours)", "security_level": "High sensitivity (including passenger card number)"}.

[0096] By using governance objects, the system matches the corresponding set of data sources from a pre-constructed global data asset catalog. Based on the governance object and the set of data sources, it reads at least one metadata information corresponding to the governance object from the global data asset catalog. Then, based on the business scenario, it extracts the current scenario feature tuples from each metadata information. This allows for precise anchoring of vague business objectives to specific physical data sources. This enables subsequent strategy recommendations to be quantitatively evaluated based on objective facts such as the actual data volume, the degree of heterogeneity of sources, update fluctuation patterns, and the distribution of sensitive fields, rather than relying on subjective assumptions based on expert experience. This achieves automated mapping from business semantics to physical assets, eliminating the high communication costs and information delays caused by manual data source inventory. It also ensures that the governance solution has a high degree of scenario adaptability and execution feasibility in resource estimation and strategy selection, fundamentally improving the scientific nature and accuracy of decision-making.

[0097] In one possible implementation, Figure 4 This is a flowchart illustrating the process of generating a data governance scheme document in the urban rail network data processing method provided in this application embodiment, with reference to... Figure 4 As shown, in S202 above, the central large model generates a data governance scheme document based on the current scene feature tuples and governance requirement semantic tuples, including: S401. The central large model determines at least one candidate data governance strategy based on the current scene feature tuples and the pre-constructed governance strategy candidate pool.

[0098] Optionally, the central big model selects structurally applicable candidate strategies from the governance strategy candidate pool based on objective facts such as data volume, timeliness requirements, and security level in the scenario feature tuple, thereby achieving a preliminary match between the scenario and the strategy.

[0099] The governance strategy candidate pool can store multiple governance strategies and their applicable conditions. These strategies can include: full governance, incremental governance, partitioned governance, real-time streaming governance, micro-batch governance, sampling governance, etc.

[0100] Specifically, full-volume governance is suitable for scenarios with small data volumes, stable structures, and slow business changes. Incremental governance only processes changed data and is suitable for scenarios with large data volumes but moderate change frequencies. Partition governance splits governance by business domain, time, or region and is suitable for scenarios with multi-source heterogeneity and cross-domain collaboration. Real-time streaming governance verifies and cleans data instantly upon writing and is suitable for scenarios with extremely high timeliness requirements. Micro-batch governance processes data in batches at minute-level windows, representing a compromise between real-time and incremental approaches. Sampling governance processes only representative samples and is suitable for exploratory analysis or resource-constrained scenarios.

[0101] For example, the central big model first identifies the hard constraints in the scene feature tuples, and then compares the hard constraints in the scene feature tuples with the applicable conditions of each strategy in the governance strategy candidate pool one by one, eliminating strategies that are clearly not met, and obtaining at least one candidate data governance strategy.

[0102] For example, when the data volume is "extremely high", full data processing is excluded; when the timeliness requirement is ≤5 minutes, full data processing and sampling processing are excluded; when the data source is "single", partitioned processing is excluded.

[0103] S402. The central big model determines the target data governance strategy from among the candidate data governance strategies based on governance indicators, quantitative objectives, and implicit constraints.

[0104] Optionally, the central big model uses the quantitative goals and implicit constraints in the semantic tuple of governance requirements as hard benchmarks, and combines the quantitative scores of each candidate strategy in four dimensions—business fit, resource consumption, implementation complexity, and scalability—for weighted ranking. Through hard constraint filtering and comprehensive comparison, the optimal target strategy is selected, transforming qualitative experience into quantitative decision-making, and thus obtaining the target data governance strategy.

[0105] For example, the central big model first performs a quantitative evaluation of the candidate strategies according to the preset evaluation dimensions and evaluation weights to obtain the weighted total score of each candidate data governance strategy, and then obtains the target data governance strategy based on the weighted total score of each candidate data governance strategy.

[0106] For example, evaluation dimensions may include: business fit, resource consumption, implementation complexity, and scalability and adaptability. Business fit accounts for 30% of the weight, and its evaluation point is whether the strategy can meet core business metrics. Resource consumption accounts for 25% of the weight, and its evaluation point is the level of computing, storage, and network resource consumption of the strategy. Implementation complexity accounts for 20% of the weight, and its evaluation point is the difficulty of implementing the strategy, including technology stack adaptation, rule configuration workload, and the degree of modification to existing systems. Scalability and adaptability account for 25% of the weight, and its evaluation point is whether the strategy can scale smoothly when business grows or requirements change.

[0107] S403. The central big model generates a data governance solution document based on the target data governance strategy and the semantic tuple of governance requirements.

[0108] Optionally, the large model fills in the business scenario, governance object, governance indicator and other information in the semantic tuple of target strategy and governance requirements into the preset scheme template to generate a data governance scheme document with a complete structure.

[0109] In one possible implementation, Figure 5This is a flowchart illustrating the process of determining task scheduling information and database operation statements in the urban rail network data processing method provided in this application embodiment, with reference to... Figure 5 As shown, in S203 above, based on unstructured requirement information, data governance solution documents, and the global data asset catalog, task orchestration information and database operation statements are determined, including: S501. Extract business semantics from unstructured requirement information to obtain structured business semantic data.

[0110] Optionally, after cleaning the unstructured requirement information, the chapter structure is preserved by paragraph segmentation, the table content is identified and preserved, and the preset business semantic extraction prompt word template is filled in to generate complete business semantic extraction prompt words. The business semantic extraction prompt words are then input into the central large model to obtain business semantic structured data.

[0111] For example, the business semantic extraction prompt could be: "[Role Setting] You are a senior data governance expert in the urban rail transit industry, proficient in data warehouse modeling and business definition."

[0112] [Business Context] The business scenario for this governance is: network-level passenger flow prediction; the governance object is: passenger flow prediction data.

[0113]

Task Objective

Field Definitions and Data Types

Output Example

[0114] [Input Materials] The Metro Network Operations Center needs to conduct passenger flow forecasting data management. Currently, the daily passenger flow data volume is approximately 8 million records, mainly from the AFC gate systems of various lines. Entry volume is counted based on gate entry records, including normal card swipes and mobile NFC payments; duplicate swipes and test card records need to be removed. Exit volume is counted based on gate exit records, including all passengers exiting from the paid area. Transfer volume is currently defined inconsistently across lines: Line 1 uses the transfer identification algorithm of the clearing system, while Line 2 uses a 30-minute continuous card swipe determination. All passenger flow data needs to be aggregated by station and hour for passenger flow forecasting during morning peak (7:00-9:00) and evening peak (17:00-19:00), with minute-level detail required. Existing passenger flow data suffers from inconsistent standards and manually configured quality rules, resulting in a forecast accuracy rate of less than 70%.

[0115] [Terminology Processing Guidelines] It is necessary to identify professional abbreviations in the urban rail transit industry (such as AFC, SCADA, clearing system, etc.) and expand them into complete Chinese terms; to standardize ambiguous business expressions according to industry common sense; when the same indicator has multiple descriptions in different paragraphs, all of them should be extracted and the differences should be marked.

[0116] [Field Extraction Instructions] Business Scenario: Directly reference the given business scenario; Governance Object: Directly reference the given governance object; Core Metrics: Extract all metrics containing measurement meaning from the document, recording their name, definition, data type, and unit; Dimensions: Extract all analysis dimensions and their hierarchical relationships from the document; Granularity: Determined based on the time and spatial precision in the metric description (second-level details / minute-level details / hour-level summary / day-level summary); Data Source Clues: Identify the clues describing the data source from the document and match them with the asset catalog.

[0117]

Field Definitions and Data Types

Output Example

[0118] S502. Based on the business semantic structured data, the full-domain data asset catalog, and the pre-constructed industry knowledge base, determine the field mapping recommendation information.

[0119] Optionally, multiple query vectors can be constructed based on business semantic structured data, and each query vector can be used to search and process data in the global data asset catalog and a pre-constructed industry knowledge base to obtain field mapping recommendation information, thereby converting business semantics into technical mapping.

[0120] S503. Based on the business semantic structured data, field mapping recommendation information, and the target data governance strategy in the data governance solution document, determine the logical model description information and database operation statements.

[0121] Optionally, after obtaining the business semantic structured data and field mapping recommendation information, the business semantic structured data, field mapping recommendation information, and target data governance strategies from the data governance solution document can be filled into the preset target logical model description template and target database statement template to obtain logical model description information and database operation statements.

[0122] By extracting business semantics from unstructured requirement information, structured business semantic data is obtained. Based on this structured business semantic data, the global data asset catalog, and a pre-constructed industry knowledge base, recommended field mapping information is determined. Then, based on the structured business semantic data, the recommended field mapping information, and the target data governance strategy in the data governance solution document, logical model description information and database operation statements are determined. This transforms the vague and fragmented natural language expressions of business personnel into precise and calculable structured indicator definitions, thereby eliminating ambiguities in understanding caused by different expression methods at the business description level. Furthermore, it enables fields with the same meaning but different names or the same name but different meanings in different lines and systems to be accurately identified and mapped to unified standard fields, eliminating cross-system semantic conflicts. This allows vague requirements from business personnel to directly reach the physical database table structure without relying on manual design by data architects, thus achieving fully automated conversion from business semantics to technical models. This reduces the data warehouse model design cycle from several days to weeks in the traditional model to minutes.

[0123] In one possible implementation, Figure 6 This is a flowchart illustrating the process of determining field mapping recommendation information in the urban rail network data processing method provided in this application embodiment, with reference to... Figure 6 As shown, in S502 above, based on business semantic structured data, the full-domain data asset catalog, and a pre-constructed industry knowledge base, the field mapping recommendation information is determined, including: S601. Generate at least one query vector based on the business semantic structured data.

[0124] Optionally, key retrieval elements can be extracted from the business semantic structured data, and an independent query vector can be generated for each core indicator. At the same time, corresponding query vectors can also be generated for dimensional information.

[0125] For example, the central large model transforms textual descriptions in business semantic structured data into semantic vector representations. By mapping words, semantic relationships, and contextual information in the text to numerical values ​​in the vector space, the central large model ensures that semantically similar texts are close in distance within the vector space, thereby achieving semantic retrieval.

[0126] S602. Based on each query vector, retrieve at least one first candidate field mapping information from the global data asset catalog.

[0127] Optionally, for each field in the global data asset catalog, its field name, field comment, table name, and data type are combined into a descriptive text, which is then converted into a vector through an embedding model to build a vector index library for the asset catalog.

[0128] Optionally, the similarity (e.g., cosine similarity) of each query vector with all vectors in the asset catalog vector index is calculated, and the vectors are sorted from high to low similarity to retrieve the Top-K most matching asset fields, thus obtaining at least one first candidate field mapping information. Here, K is a positive integer.

[0129] S603. Based on each query vector, retrieve at least one second candidate field mapping information from the industry knowledge base.

[0130] Optionally, a vector index is pre-established for the content in the industry knowledge base, and the similarity between each query vector and the standard definitions of each type in the industry knowledge base is calculated to recall the standard definition that best matches the semantics of the business indicators, thereby obtaining at least one second candidate field mapping information.

[0131] S604. The central large model generates field mapping recommendation information based on the mapping information of each first candidate field and the mapping information of each second candidate field.

[0132] Optionally, the mapping information of each first candidate field and the mapping information of each second candidate field are filled into a preset field mapping prompt word template to obtain field mapping prompt words, which are then input into the central big model. The central big model groups the mapping information of each first candidate field and the mapping information of each second candidate field according to the associated indicators / dimensions to form a combination of multiple business indicators and candidate mapping schemes, and performs voting-style reasoning decision-making on each candidate mapping scheme to generate field mapping recommendation information.

[0133] For example, field mapping prompts could include: "[Role Setting] You are a senior data governance expert in the urban rail transit industry, proficient in data standard setting and data warehouse modeling."

[0134] [Task Objective] The following provides candidate field mapping schemes for multiple business metrics, sourced from the "Global Data Asset Catalog" and the "Industry Knowledge Base." Please group these schemes by related metrics / dimensions, and use a voting-based reasoning approach to determine the best candidate field mappings for each metric, generating the final recommended field mapping information.

[0135] Decision-making rules Priority rule: National standards (GB) > Industry standards (industry standards) > Enterprise standards (enterprise standards) > Historical governance experience.

[0136] Conflict resolution rules: If a national standard definition exists, the national standard shall be adopted first, and other sources shall be downgraded to "alternatives" and conflict descriptions shall be recorded; if an industry standard exists but there is no national standard, the industry standard shall be adopted; if there is a definition in the candidate source that contradicts the definition of the high-priority standard, it shall be marked as "definition deviation".

[0137] Mapping relationship generation rules: When the asset catalog field has the same semantics as the standard field, a "direct mapping" is generated; when the asset catalog field has a different format / encoding method than the standard field, a "code table mapping" or "conversion mapping" is generated.

[0138] Confidence rating rules: The confidence rating (0-1) is given based on a combination of similarity score and standard authority.

[0139]

Field Definitions and Data Types

[0140] [Candidate mapping information pending decision] I. Candidate mapping schemes for the "transfer volume" indicator: Candidate Solution A (from the Global Data Asset Catalog): { "Candidate Source": "Asset Catalog - Line 1 Clearing System", "Related Indicators": "Transfer Volume", "Source Table": "ods_transfer_l1", "Source Field": "transfer_count", "Source Field Comments": "Number of transfer records (statistics based on the clearing system's transfer identification algorithm)", "Data Type": "BIGINT", "Similarity": 0.88, "Definition": "A transfer is defined as a continuous card swiping time difference ≤ 30 minutes at the transfer station gate"} Candidate Solution B (from the global data asset catalog): { "Candidate Source": "Asset Catalog - Line 2 Operation Documents", "Related Indicators": "Transfer Volume", "Source Table": "ods_transfer_l2", "Source Field": "transfer_volume", "Source Field Comments": "Transfer Passengers (Difference between Passenger Flow Entering and Exiting at Transfer Stations)", "Data Type": "BIGINT", "Similarity": 0.75, "Definition": "Difference between Passenger Flow Exiting and Entering at Transfer Stations"} Candidate Solution C (from industry knowledge base): { "Candidate Source": "Industry Standard - GB / T 40484-2021", "Related Indicators": "Transfer Volume", "Standard Entry": "Article 4.2.3", "Target Field": "transfer_passenger_flow", "Target Data Type": "BIGINT", "Similarity": 0.93, "Definition": "Number of passengers transferring from one line to another at a transfer station"}

Decision Requirements

[0141] By using business semantic structured data, at least one query vector is generated. Based on each query vector, at least one first candidate field mapping information is retrieved from the global data asset catalog. Based on each query vector, at least one second candidate field mapping information is retrieved from the industry knowledge base. Then, the central big model generates field mapping recommendation information based on the first and second candidate field mapping information. This ensures that no data fields actually owned by the enterprise are omitted from the asset catalog side, and that the industry knowledge base side does not deviate from national standard definitions. The parallel retrieval of two independent information sources achieves knowledge complementarity, thereby improving both recall and precision. This provides a complete candidate information foundation for subsequent decision-making, making the source and judgment logic of each mapping relationship traceable, greatly improving the transparency of the governance process and the convenience of compliance auditing. In addition, the vector retrieval mechanism enables the central big model to identify fields with the same name but different meanings, breaking through the limitations of traditional exact matching. This allows existing fields in the asset catalog to be fully discovered and reused, avoiding duplicate definitions and resource waste.

[0142] In one possible implementation, Figure 7 This is a flowchart illustrating the process of determining logical model description information and database operation statements in the urban rail network data processing method provided in this application embodiment, with reference to... Figure 7 As shown, in S503 above, based on business semantic structured data, field mapping recommendation information, and the target data governance strategy in the data governance solution document, the logical model description information and database operation statements are determined, including: S701. Based on the business semantic structured data, determine the modeling mode, and based on the modeling mode, determine the target logical model description template and the target database statement template from multiple preset template structures.

[0143] Optionally, a modeling pattern can be obtained by judging the modeling pattern based on the granularity and indicator update method characteristics in the business semantic structured data.

[0144] Optionally, after obtaining the modeling pattern, the corresponding target logical model description template and target database statement template are obtained from multiple preset template structures.

[0145] For example, the modeling schema can be a transaction fact table. The target logical model description template can be a transaction fact table logical model template, and the target database statement template can be a transaction fact table database statement template.

[0146] S702. Based on the field mapping recommendation information and the target data governance strategy in the data governance solution document, populate the target logical model description template and the target database statement template to obtain the logical model description information and database operation statements.

[0147] Optionally, based on the field mapping recommendation information and the target data governance strategy in the data governance solution document, all placeholders in the target logical model description template and the target database statement template are filled one by one to obtain the logical model description information and the database operation statement.

[0148] For example, the fact table name is derived from the business domain plus a fact table suffix; the fact table granularity description is derived from the granularity definition in the business semantic structured data; the primary key field is automatically generated by the system as a surrogate key according to the transaction fact table specification; the metric field is generated one by one by traversing the core indicators in the business semantics and combining the corresponding target standard field names and data types in the field mapping recommendation information; the foreign key field is generated by traversing the dimension list in the business semantics and reading the standard field names and data types corresponding to each dimension from the field mapping recommendation information; the dimension table list extracts all dimension table definitions and their primary keys and attribute fields from the field mapping recommendation information; the partitioning strategy is determined according to the target governance strategy; and the distribution strategy is determined according to the evaluation data volume in the business semantics. After all placeholders are filled, the logical model description information and database operation statements are obtained.

[0149] In one possible implementation, Figure 8 This is a flowchart illustrating the process of generating an executable task package in the urban rail network data processing method provided in this application embodiment, with reference to... Figure 8 As shown, in S204 above, an executable task package is generated based on task orchestration information, database operation statements, the global data asset catalog, and the data governance solution document. This package includes: S801. The central big model obtains a list of subtasks and their dependencies based on task orchestration information, database operation statements, the global data asset catalog, and data governance solution documents.

[0150] Optionally, based on task orchestration information, database operation statements, global data asset catalog, and data governance solution documents, subtask decomposition prompts are constructed and input into the central big model. The central big model then decomposes the tasks based on the subtask decomposition prompts to obtain a list of subtasks and determine the subtask dependencies.

[0151] The subtask list includes at least one subtask and a task description for each subtask.

[0152] S802. Obtain the script templates corresponding to each subtask. The central big model generates the scripts corresponding to each subtask based on the task descriptions and script templates corresponding to each subtask.

[0153] Optionally, obtain the script template corresponding to each subtask, generate prompts for each subtask, and input the prompts into the central big model. The central big model then fills in the specific information such as task description, processing rules, and input / output paths into the script template to generate the script corresponding to each subtask.

[0154] S803. Generate configuration files based on task orchestration information, database operation statements, global data asset catalog, and data governance solution documents.

[0155] Optionally, task orchestration information, database operation statements, global data asset catalog, and data governance solution documents are categorized by purpose according to the preset mapping relationship between purpose and configuration file, and then matched and populated to generate configuration files.

[0156] For example, the configuration file may include multiple configuration files in JSON format.

[0157] For example, by using the global data asset catalog and data governance solution documents, data source connection information and incremental conditions are generated, thereby generating source_config.json.

[0158] S804. Generate an executable task package based on the subtask list, subtask dependencies, scripts corresponding to each subtask, and configuration files.

[0159] Optionally, the subtask list, subtask dependencies, scripts corresponding to each subtask, and configuration files are packaged according to a preset structure to generate an executable task package.

[0160] The executable task package includes a task package manifest (manifest.json), task dependency definitions (dag_definition.json), executable scripts for each subtask (scripts / directory), and configuration files for each subtask (config / directory).

[0161] In one possible implementation, after executing the executable task package in S204 above, the method further includes: Obtain execution status and quality indicators; determine the governance score based on the execution status, quality indicators, and target data governance strategy in the data governance plan document; determine the governance result based on the governance score and quantitative targets; if the governance result is satisfactory, the central big model generates a governance effect report; if the governance result is unsatisfactory, the central big model adjusts the executable task package based on the governance score and re-executes the adjusted executable task package.

[0162] Optionally, after the executable task package is executed by the edge executor, the edge executor will send the complete result of this execution back to the central large model engine through a secure channel.

[0163] The returned information includes: the execution status of each subtask, the corresponding execution time and exception log summary, processing statistics, and quality indicator calculation results.

[0164] The execution status includes: success, failure, or partial failure. Processing statistics include: the number of records extracted, the number of records cleaned, the number of records loaded, and the number of records rejected due to quality issues. Quality metrics calculation results include core quality indicators such as data non-empty rate, format compliance rate, deduplication rate, and business availability.

[0165] Optionally, the central large model conducts a comprehensive quantitative evaluation from three dimensions: technical quality (50% weight), scored based on KPIs such as the non-empty rate of returned data, format compliance rate, and data consistency; execution efficiency (20% weight), scored based on whether all sub-tasks were successfully completed and whether the execution time was within the expected range; and business availability (30% weight), scored based on the degree to which the governed data supports business scenarios. The comprehensive score for this governance is obtained by weighting and summing the three dimensions.

[0166] Optionally, the governance score is compared with the quantitative target in the governance requirement semantic tuple to determine whether the governance result meets the standard. If the governance score reaches or exceeds the quantitative target, it is determined to meet the standard. The central big model fills the governance score, the achievement of various KPIs, the comparison of data quality before and after governance, and the experience summary of this governance into the preset governance effect report template, generates a complete governance effect report and archives it. The process ends successfully at this point.

[0167] Optionally, if the governance score fails to meet the quantitative target, it is deemed substandard, and the system enters a closed-loop correction process. First, the central big data model, combined with execution logs, anomaly log summaries, and detailed data of each KPI, investigates the specific reasons for the substandard performance from three dimensions: technical quality, execution efficiency, and business availability.

[0168] For example, technical quality might manifest as an excessive null value rate in a field or a time format that does not meet expectations; execution efficiency might manifest as a subtask timing out or insufficient resources causing some tasks to fail; and business availability might manifest as a discrepancy between the data definition after governance and the actual business needs.

[0169] Optionally, after identifying the specific reasons for non-compliance, the central large model makes targeted modifications to the original executable task package based on different root cause types, and then re-issues the adjusted executable task package to the edge executor for execution through a secure channel. After completion, it enters the evaluation again.

[0170] For example, if the null value rate exceeds the standard, the threshold parameter in the cleaning rules is adjusted or a null value filling step is added; if the time format does not meet expectations, the time parsing format string in the time zone standardization logic is corrected; if the data volume is significantly insufficient, the time window for incremental extraction is adjusted or the extraction range is expanded; if the business caliber is inaccurate, the code table mapping relationship is corrected according to the actual data characteristics; if the subtask execution times out, the parallelism is increased or the data volume for batch processing is adjusted.

[0171] Based on the same inventive concept, this application also provides an urban rail network data processing device corresponding to the urban rail network data processing method. Since the principle of the device in this application is similar to the urban rail network data processing method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0172] The urban rail network data processing device includes: The acquisition module is used to acquire unstructured requirement information input by users, extract data governance information from the unstructured requirement information, and determine the governance requirement semantic tuple based on the data governance information and the pre-trained central big model. The governance requirement semantic tuple includes: business scenario, governance object, governance indicator, quantitative target and implicit constraint. The first determination module is used to determine the feature tuples of the current scenario based on the governance object, business scenario and pre-constructed full-domain data asset catalog, and the central big model generates a data governance solution document based on the feature tuples of the current scenario and the semantic tuples of governance requirements. The second determination module is used to determine task orchestration information and database operation statements based on unstructured requirement information, data governance solution documents, and the full-domain data asset catalog. The task orchestration information includes: business semantic structured data, field mapping recommendation information, and logical model description information. The generation module is used to generate executable task packages based on task orchestration information, database operation statements, global data asset catalog, and data governance solution documents, and then execute the executable task packages.

[0173] Optionally, the first determining module is specifically used for: Based on the governance object, the data source set corresponding to the governance object is obtained from the pre-constructed global data asset catalog. The data source set includes at least one data source. Based on the governance object and the data source set, at least one metadata information corresponding to the governance object is read from the global data asset catalog; Based on the business scenario, the current scenario feature tuple is extracted from the metadata information. The current scenario feature tuple includes at least: data volume, data source, data change type, and security level.

[0174] Optionally, the first determining module is specifically used for: The central big model determines at least one candidate data governance strategy based on the current scene feature tuples and the pre-constructed governance strategy candidate pool. The central big model determines the target data governance strategy from among the candidate data governance strategies based on governance indicators, quantitative objectives, and implicit constraints. The central big model generates a data governance solution document based on the target data governance strategy and semantic tuples of governance requirements.

[0175] Optionally, the second determining module is specifically used for: Business semantics are extracted from unstructured requirement information to obtain structured business semantic data; Based on the business semantic structured data, the full-domain data asset catalog, and the pre-constructed industry knowledge base, determine the field mapping recommendation information; Based on the business semantic structured data, field mapping recommendation information, and the target data governance strategy in the data governance solution document, determine the logical model description information and database operation statements.

[0176] Optionally, the second determining module is specifically used for: Generate at least one query vector based on the business semantic structured data; Based on each query vector, at least one first candidate field mapping information is retrieved from the global data asset catalog; Based on each query vector, at least one second candidate field mapping information is retrieved from the industry knowledge base; The central large model generates field mapping recommendation information based on the mapping information of each first candidate field and each second candidate field.

[0177] Optionally, the second determining module is specifically used for: Based on the business semantic structured data, determine the modeling pattern, and based on the modeling pattern, determine the target logical model description template and the target database statement template from multiple preset template structures; Based on the field mapping recommendations and the target data governance strategy in the data governance solution document, the target logical model description template and the target database statement template are populated to obtain the logical model description information and database operation statements.

[0178] Optionally, a generation module is specifically used for: Based on task orchestration information, database operation statements, global data asset catalog, and data governance solution documents, the central big model obtains a list of subtasks and subtask dependencies. The list of subtasks includes at least one subtask and a task description for each subtask. Obtain the script templates corresponding to each subtask, and the central big model generates the scripts corresponding to each subtask based on the task descriptions and script templates corresponding to each subtask. Generate configuration files based on task orchestration information, database operation statements, global data asset catalog, and data governance solution documents; An executable task package is generated based on the subtask list, subtask dependencies, scripts corresponding to each subtask, and configuration files.

[0179] Optionally, the generation module is also used for: Obtain execution status and quality indicators; The governance score is determined based on the execution status, quality indicators, and the target data governance strategy in the data governance solution document; The governance outcome is determined based on the governance score and quantitative targets; If the governance results are satisfactory, the central big data model will generate a governance effectiveness report. If the governance result is unsatisfactory, the central big model will adjust the executable task package based on the governance score and re-execute the adjusted executable task package.

[0180] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0181] This application also provides an electronic device, such as... Figure 9 As shown, Figure 9 The schematic diagram of the electronic device structure provided in this application embodiment includes: a processor 901 and a memory 902, and optionally, a bus 903. The memory 902 stores machine-readable instructions executable by the processor 901. When the electronic device is running, the processor 901 and the memory 902 communicate through the bus 903, and the processor 901 executes the machine-readable instructions to perform the steps of the above-described urban rail network data processing method.

[0182] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described urban rail network data processing method.

[0183] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0184] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0185] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for processing urban rail transit network data, characterized in that, include: The system acquires unstructured requirement information input by the user, extracts data governance information from the unstructured requirement information, and determines governance requirement semantic tuples based on the data governance information and the pre-trained central big model. The governance requirement semantic tuples include: business scenario, governance object, governance indicator, quantitative target and implicit constraint. Based on the governance object, the business scenario, and the pre-constructed global data asset catalog, the current scenario feature tuple is determined, and the central big model generates a data governance solution document based on the current scenario feature tuple and the governance requirement semantic tuple. Based on the unstructured requirement information, the data governance solution document, and the global data asset catalog, task orchestration information and database operation statements are determined. The task orchestration information includes: business semantic structured data, field mapping recommendation information, and logical model description information. Based on the task orchestration information, the database operation statements, the global data asset catalog, and the data governance scheme document, an executable task package is generated and executed.

2. The urban rail network data processing method according to claim 1, characterized in that, The step of determining the current scenario feature tuple based on the governance object, the business scenario, and the pre-constructed global data asset catalog includes: Based on the governance object, a set of data sources corresponding to the governance object is obtained by matching from a pre-constructed global data asset catalog, and the set of data sources includes at least one data source. Based on the governance object and the data source set, at least one metadata information corresponding to the governance object is read from the global data asset catalog; Based on the business scenario, a current scenario feature tuple is extracted from each of the metadata information. The current scenario feature tuple includes at least: data volume, data source, data change type, and security level.

3. The urban rail network data processing method according to claim 1, characterized in that, The process of generating a data governance scheme document based on the current scene feature tuple and the governance requirement semantic tuple by the central large model includes: The central big model determines at least one candidate data governance strategy based on the current scene feature tuples and a pre-constructed governance strategy candidate pool. The central big model determines the target data governance strategy from among the candidate data governance strategies based on the governance indicators, the quantitative objectives, and the implicit constraints. The central big model generates a data governance scheme document based on the target data governance strategy and the semantic tuple of governance requirements.

4. The urban rail network data processing method according to claim 1, characterized in that, The step of determining task orchestration information and database operation statements based on the unstructured requirement information, the data governance solution document, and the global data asset catalog includes: The unstructured requirement information is subjected to business semantic extraction to obtain business semantic structured data; Based on the business semantic structured data, the global data asset catalog, and the pre-constructed industry knowledge base, determine the field mapping recommendation information; Based on the business semantic structured data, the field mapping recommendation information, and the target data governance strategy in the data governance solution document, determine the logical model description information and database operation statements.

5. The urban rail network data processing method according to claim 4, characterized in that, The step of determining field mapping recommendation information based on the business semantic structured data, the global data asset catalog, and the pre-constructed industry knowledge base includes: Based on the business semantic structured data, generate at least one query vector; Based on each of the query vectors, at least one first candidate field mapping information is retrieved from the global data asset catalog; Based on each of the query vectors, at least one second candidate field mapping information is retrieved from the industry knowledge base; The central large model generates field mapping recommendation information based on the mapping information of each of the first candidate fields and the mapping information of each of the second candidate fields.

6. The urban rail network data processing method according to claim 4, characterized in that, The step of determining logical model description information and database operation statements based on the business semantic structured data, the field mapping recommendation information, and the target data governance strategy in the data governance solution document includes: Based on the business semantic structured data, a modeling pattern is determined, and based on the modeling pattern, a target logical model description template and a target database statement template are determined from multiple preset template structures. Based on the field mapping recommendation information and the target data governance strategy in the data governance scheme document, the target logical model description template and the target database statement template are populated to obtain the logical model description information and the database operation statement.

7. The urban rail network data processing method according to claim 1, characterized in that, The step of generating an executable task package based on the task orchestration information, the database operation statements, the global data asset catalog, and the data governance solution document includes: Based on the task orchestration information, the database operation statements, the global data asset catalog, and the data governance scheme document, the central big model obtains a sub-task list and sub-task dependencies. The sub-task list includes at least one sub-task and a task description corresponding to each sub-task. Obtain the script templates corresponding to each subtask, and the central big model generates the scripts corresponding to each subtask based on the task descriptions and script templates corresponding to each subtask. Based on the task orchestration information, the database operation statements, the global data asset catalog, and the data governance solution document, a configuration file is generated; An executable task package is generated based on the subtask list, the subtask dependencies, the scripts corresponding to each subtask, and the configuration file.

8. The urban rail network data processing method according to claim 1, characterized in that, After executing the executable task package, the method further includes: Obtain execution status and quality indicators; A governance score is determined based on the execution status, the quality indicators, and the target data governance strategy in the data governance solution document. The governance outcome is determined based on the governance score and the quantitative target. If the governance result is satisfactory, the central large model will generate a governance effect report. If the governance result is unsatisfactory, the central big model will adjust the executable task package based on the governance score and re-execute the adjusted executable task package.

9. An electronic device, characterized in that, include: The device includes a processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when the electronic device is in operation, are executed by the processor to perform the steps of the urban rail network data processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the urban rail network data processing method as described in any one of claims 1 to 8.

Images