Data development method and system based on data pipeline

Abstract

The invention discloses a data development method and system based on a data pipeline. The method comprises the following steps: connecting an external system through a business demand management module and intelligently analyzing a demand; the data demand management module is disassembled into specific data demands; the task hierarchy management module confirms the task hierarchy and associates the task hierarchy to a unified demand batch; the development task management module automatically generates a development task, creates a development state structure definition in a production environment, and supports graphical SQL construction, one-key benchmarking, automatic testing and dynamic desensitization; the test task management module sequentially executes a system integration test, a user acceptance test and a scheduling test; and the production task management module executes one-key production and automatically synchronizes to a production environment. The system correspondingly comprises six functional modules. According to the method, the whole-process online, standardized and automatic management of the data project is realized, parallel development and three-state resource dynamic allocation are supported, the development efficiency and the data quality are remarkably improved, and the data security is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of data project development process management technology, and in particular to a data development method and system based on a data pipeline, used to optimize the entire lifecycle management of data projects from requirement proposal to production launch. Background Technology

[0002] Against the backdrop of digital transformation in the financial industry, the development and management of banking data projects still face systemic challenges. Traditional data project management models generally suffer from fragmented process phases, inconsistent toolchains, and extensive management practices, resulting in insufficient release of data value, long project delivery cycles, and difficulties in quality control. Specifically, different management systems are used for business, development, testing, and operations, and data and status rely on manual transmission, creating de facto "data silos" and "process breakpoints." This not only causes delays and distortions in information transmission but also makes it difficult to achieve end-to-end automated and transparent control throughout the development lifecycle. Specifically, existing technologies mainly have the following shortcomings:

[0003] 1. The business requirement transmission chain is lengthy and distorted: Due to the lack of a unified and intelligent requirement docking hub, the original requirements put forward by the business department need to go through multiple manual translations and confirmations in the process of being transmitted to the technology department, which is very easy to cause misunderstandings and result in significant differences between the final delivered data products and the original business intentions.

[0004] 2. High technical barriers and insufficient standardization in the development process: Data development work relies heavily on technical personnel manually writing complex structured query languages ​​or Python scripts. The development process lacks unified standardized control, and different developers use different tools and coding standards, resulting in inconsistent code quality, low development efficiency, high system maintenance costs, and difficulty in reusing data assets.

[0005] 3. Low automation and high risk in testing and deployment: First, the testing phase heavily relies on manual writing of test cases and reports, which is time-consuming and labor-intensive. Second, configuration differences exist between the testing and production environments, causing many data issues to go undetected during testing, resulting in a high rollback rate in the production environment. Finally, version management is chaotic, with inconsistent version identifiers between development, testing, and production environments, frequently leading to situations where "what is tested is not what is deployed." The deployment process relies on manual configuration synchronization, posing a high operational risk.

[0006] In conclusion, existing technologies are insufficient to support efficient, high-quality, secure, and controllable full lifecycle management of data projects. There is an urgent need for a new data development and management solution that can connect all aspects and achieve standardization and automation. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a data development method and system based on a data pipeline. It aims to realize the online, standardized and automated management of the entire process of data projects from requirement proposal to final production, so as to significantly improve development efficiency, ensure data quality, strengthen security control and shorten project delivery cycle.

[0008] The technical solution for achieving the objective of this invention is as follows:

[0009] A data development method based on data pipelines includes the following steps:

[0010] Business requirements management steps: Connect with external project management platforms, synchronize business requirements, and identify and decompose data-related business requirements through intelligent parsing;

[0011] Data requirement management steps: Based on the data-related business requirements, conduct preliminary research and analysis, apply for resources, and break them down into specific data requirement tasks. Manage the tasks that need to be put into production in the same batch under one data requirement batch.

[0012] Task hierarchy management steps: Based on the data requirement task, determine the data hierarchy to which the task belongs, output the task structure with hierarchy information, and associate the development task with a unified data requirement batch number;

[0013] Development task management steps: Based on task hierarchy information, automatically generate development tasks, create development state structure definitions in the production environment, build a structured query language in a graphical way, execute unit tests and automated data preview and desensitization, generate test reports and submit them for review, and generate test tasks after the review is approved.

[0014] Test task management steps: For approved development tasks, system integration testing, user acceptance testing, and scheduling testing are executed sequentially. The corresponding test state structure definition is automatically created, automated testing is performed based on production environment data, and reports are generated.

[0015] Production task management steps: Conduct production review for tasks that have passed testing, execute one-click production operation, automatically synchronize development results to the production environment, and push job information to the scheduling system.

[0016] Furthermore, the development task management steps, specifically including the graphical method for constructing a structured query language, include:

[0017] Select table resources by dragging and dropping, and retrieve field information from the metadata system;

[0018] The one-click benchmarking function automatically matches the standard platform based on the Chinese name of the field and supplements the standard information.

[0019] Obtain recommended inter-table relationships from the metadata system;

[0020] Choose a built-in function and write a conditional statement;

[0021] The processing steps are assembled into mapping data and stored.

[0022] A complete structured query language is generated based on the mapping data.

[0023] Furthermore, the development task management steps also include:

[0024] After the development task at each level is completed, its metadata is registered to the draft state of the metadata system;

[0025] Downstream development tasks query and reference draft metadata from upstream layers based on data requirement batch numbers, enabling parallel development across layers.

[0026] Furthermore, in the development task management steps and test task management steps, corresponding structure definitions are automatically created in the production environment cluster according to the task status, including development state, system integration test state and user acceptance test state structure definitions, to realize dynamic allocation and logical isolation of resources in the three states.

[0027] Furthermore, in the test task management step, the extraction, transformation, and loading statements are automatically parsed during data preview, the query fields are identified, and dynamic desensitization is performed according to the desensitization function mapped in their standard information.

[0028] This invention also proposes a data development system based on a data pipeline, comprising:

[0029] The business requirements management module is used to connect with external project management platforms, synchronize business requirements, and identify and decompose data-related business requirements through intelligent parsing.

[0030] The data requirement management module is used to conduct preliminary research and analysis based on the data-related business requirements, apply for form resources, and break them down into specific data requirement tasks, and manage the tasks that need to be put into production in the same batch under one data requirement batch.

[0031] The task hierarchy management module is used to determine the data hierarchy to which the task belongs based on the data requirement task, output the task structure with hierarchy information, and associate the development task with a unified data requirement batch number.

[0032] The development task management module is used to automatically generate development tasks based on task hierarchy information, create development state structure definitions in the production environment, build a structured query language in a graphical way, execute unit tests and automated data preview and desensitization, generate test reports and submit them for review, and generate test tasks after the review is approved.

[0033] The test task management module is used to execute system integration testing, user acceptance testing and scheduling testing sequentially for approved development tasks, automatically create corresponding test state structure definitions, execute automated tests based on production environment data and generate reports.

[0034] The production task management module is used to review tasks that have passed testing for production deployment, execute one-click production deployment operations, automatically synchronize development results to the production environment, and push job information to the scheduling system.

[0035] Furthermore, the development task management module also includes a graphical development unit, used for:

[0036] Select table resources by dragging and dropping, and retrieve field information from the metadata system;

[0037] The one-click benchmarking function automatically matches the standard platform based on the Chinese name of the field and supplements the standard information.

[0038] Obtain recommended inter-table relationships from the metadata system;

[0039] Choose a built-in function and write a conditional statement;

[0040] The processing steps are assembled into mapping data and stored.

[0041] A complete structured query language is generated based on the mapping data.

[0042] Furthermore, the development task management module is also configured as follows:

[0043] After the development task at each level is completed, its metadata is registered to the draft state of the metadata system;

[0044] Downstream development tasks query and reference draft metadata from upstream layers based on data requirement batch numbers, enabling parallel development across layers.

[0045] Furthermore, the development task management module and the test task management module are configured to automatically create corresponding structure definitions in the production environment cluster according to the task status, including structure definitions for development state, system integration test state and user acceptance test state, so as to realize the dynamic allocation and logical isolation of resources in the three states.

[0046] Furthermore, the development task management module and the test task management module are configured to automatically parse the extraction, transformation, and loading statements during data preview, identify the query fields, and perform dynamic desensitization based on the desensitization function mapped in their standard information.

[0047] Compared with the prior art, the significant advantages and beneficial effects of this invention are as follows:

[0048] 1. Achieved online and integrated management of the entire data project process: By connecting six core modules, this invention integrates previously fragmented processes such as requirements analysis, task development, testing and verification, and production deployment into a continuous online process. This eliminates errors and delays caused by manual offline handover, achieves full traceability and auditability, and shortens the project delivery cycle by an average of 40%.

[0049] 2. Breaking through the bottleneck of serial development and realizing parallel development of hierarchical tasks: This invention innovatively introduces a "draft-state metadata" mechanism. Once an upstream development task is completed, its metadata is registered in the metadata system for downstream tasks to reference. This allows downstream tasks to begin development without waiting for the upstream task to be officially launched, breaking the efficiency barrier of the traditional serial model and improving development efficiency by approximately 60%.

[0050] 3. The "three-state dynamic resource allocation" mechanism ensures environmental consistency and data security: the system automatically creates corresponding schemas in the production environment cluster based on the task status (development, system integration testing, user acceptance testing) to achieve logical isolation. This method avoids problems caused by environmental differences and ensures the authenticity and security of test data, fundamentally eliminating the risk of "testing the wrong thing."

[0051] 4. The system promotes data asset standardization through "one-click benchmarking" and graphical development: During development, the system can automatically match and fill in standard information (such as English names, data types, etc.) from the standard platform based on the Chinese names of fields, achieving "one-click benchmarking." Combined with graphical drag-and-drop development, this greatly reduces the technical threshold for development, ensuring the standardization and consistency of data assets from the source, and laying a solid foundation for data asset management.

[0052] 5. Enhanced quality and efficiency through comprehensive automation: The system incorporates configurable automated testing points, enabling automatic execution of tests and report generation during unit testing, system integration testing, and user acceptance testing. This significantly reduces the workload of testers while ensuring the comprehensiveness and objectivity of the tests. The final "one-click production deployment" function automatically synchronizes configuration information from the test state to the production state and schedules job execution, simplifying the deployment process and avoiding human error.

[0053] 6. A comprehensive data security management system has been established: In key stages such as data preview, the system can automatically parse, extract, transform, and load statements, identify sensitive fields, and dynamically call the corresponding de-identification functions for data display, effectively preventing the leakage of sensitive information during development and testing, and meeting the strict compliance requirements of the financial industry.

[0054] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0055] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0056] Figure 1 This is a schematic diagram of the operation-dependent implementation method in this invention. Detailed Implementation

[0057] It is readily understood that, based on the technical solution of this invention, various embodiments of the invention can be conceived by those skilled in the art without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention. Rather, these embodiments are provided to enable those skilled in the art to gain a more thorough understanding of the invention. Preferred embodiments of the invention are described below in conjunction with the accompanying drawings, which form part of this application and, together with the embodiments of the invention, serve to illustrate the innovative concept of the invention.

[0058] This invention relates to a data development method and system based on a data pipeline, which aims to achieve online management and control of the entire data project process from requirement to production through modularization and automation.

[0059] System functional module composition

[0060] like Figure 1 As shown, the data development system based on the data pipeline consists of six main functional modules, which work together to form a complete and coherent data development pipeline.

[0061] 1. Business Requirements Management Module

[0062] This module serves as the system entry point, responsible for interfacing with external project management platforms (such as DevOps) and automatically synchronizing original project requirements from business departments. Subsequently, it intelligently parses the received requirements, automatically identifying and breaking them down into business requirements falling under the data development category, and outputting structured data-related business requirements for use by downstream data requirement analysts.

[0063] 2. Data Requirement Management Module

[0064] This module receives data-related business requests from upstream suppliers. Within this module, the data management department's requirements analysts request the necessary table resources based on the business needs, conduct in-depth preliminary research and analysis, and ultimately break down the macro-level business requirements into specific, actionable data requirements.

[0065] The data requirements management module also includes the following key sub-modules:

[0066] The table resource allocation submodule is responsible for dynamically allocating and managing access permissions for table resources involved in data requirements. It supports dynamically scaling up or down resources as needed during analysis and ensures that requirements, development, and testing personnel can only access the table resources they are authorized to, thus protecting data asset security.

[0067] Task-level generation submodule: assists requirements analysts in generating corresponding data requirement tasks based on the results of business requirements analysis.

[0068] 3. Task Hierarchy Management Module

[0069] The task hierarchy management module analyzes and determines the corresponding task hierarchy based on data requirement tasks, outputting a task structure with hierarchy information. This module manages multiple tasks that need to be deployed in the same batch under a single data requirement batch. It is responsible for confirming the data hierarchy to which a development task belongs, such as the standard layer, model layer, indicator layer, and application layer. These layers are dynamically configurable and can be added or removed according to the actual project situation. All automatically created development tasks are associated with a unified data requirement batch number to ensure traceability.

[0070] 4. Develop a task management module

[0071] The development task management module generates development tasks based on the input task hierarchy information, constructs SQL through a graphical interface, executes unit tests and data previews (with automatic data masking), generates automated test reports, submits them to the development team leader for review, and generates test tasks after the development tasks are completed through hierarchical development task decomposition.

[0072] The core functionalities of the task management module include:

[0073] (1) Development task generation and resource allocation

[0074] Based on the confirmed task hierarchy information, the system automatically generates corresponding development tasks. After the development tasks at the corresponding level are created, a schema definition for the development hierarchy is created on the big data platform. The naming convention is: Task Hierarchy Number_Data Requirement Batch Number_DEV, for example, FDM_YWYH202508080101_DEV. These schema definitions are established in the production environment, thus directly using production environment data. This overcomes the limitations of traditional methods that require importing production data into the test environment, enabling dynamic allocation of resources across System Integration Testing (SIT), User Acceptance Testing (UAT), and Scheduled Testing.

[0075] (2) Graphical Development Process

[0076] Developers assign tasks based on a task list using a bidding process. After receiving a task, developers construct SQL queries graphically. This graphical development interface renders draggable tables, fields, and function components using a front-end framework. User actions are recorded in real-time and converted into a structured mapping of data. Specifically:

[0077] Drag and drop the corresponding table resources to retrieve the field information of the table resources from the metadata system;

[0078] With its one-click benchmarking feature, the system automatically matches and recommends the most relevant standard field information based on the Chinese field names entered or selected by the user, by calculating the textual semantic similarity with the standard terms defined in the standard platform. After user confirmation, the system automatically supplements the standard information such as the English field name, field length, precision, and type, ensuring that data standardization is implemented from the source.

[0079] Obtain recommended inter-table relationships from the metadata system;

[0080] Choose built-in functions and write conditional statements such as WHERE and GROUP BY;

[0081] The source table of the processing process, the relationships between tables, the English and Chinese names of the fields, the processing rules, etc. are assembled into mapping data;

[0082] The mapping data is stored in the database's job information table, field mapping information table, and inter-table relationship information table.

[0083] Based on the mapped data, the system automatically generates complete SQL statements that conform to the dialect of the production environment database through its built-in SQL rendering engine.

[0084] (3) Testing and Auditing Process

[0085] Unit testing is performed after development is complete, specifically as follows:

[0086] Online preview of table creation and ETL statements;

[0087] Execute table creation and ETL scripts;

[0088] The ETL statement is automatically parsed during data preview, the query fields are identified and dynamically desensitized;

[0089] The system provides a configurable library of automated test points, including data quality (such as field non-null rate and value range compliance), business logic (such as the correctness of indicator derivation logic), and performance (such as job execution time). During testing, the system automatically generates and executes corresponding test scripts based on the configured test points to complete batch testing.

[0090] Generate automated test reports;

[0091] Submit to the development team leader for review (the review should include the ETL script and test report);

[0092] After the review is approved, the system integration test (SIT) will begin.

[0093] (4) Parallel development support

[0094] Upon completion of each development task at each level, its metadata is registered in a specific namespace of the metadata system in "draft" form. The "draft" state is a temporary state relative to the "production" state metadata, indicating that the development task corresponding to the metadata is completed but has not yet undergone final production review. Downstream development can query and reference these upstream draft metadata based on the requirement batch number. This allows downstream tasks to begin development without waiting for upstream tasks to be officially put into production, achieving parallel development across levels. For example, after the model layer development is completed, even if it is not yet in production, the indicator layer task can still be developed based on the development-state model layer task, achieving parallel development across levels.

[0095] The development task management module also includes the following sub-modules:

[0096] Automatic order grabbing submodule: Developers can obtain development tasks through an order grabbing mode, which supports subsequent performance statistics.

[0097] Automated Testing Submodule: By configuring automated testing projects and writing test scripts, test reports can be automatically generated.

[0098] The table creation and ETL submodule allows you to select an ETL date and generate table creation and ETL statements online. During execution, you can view logs in real time and preview anonymized data. During data preview, the ETL statement is automatically parsed, and after identifying the query fields, it dynamically anonymizes the data using the anonymization functions in the standard information mapped to those fields, ensuring that sensitive data is not leaked.

[0099] 5. Test Task Management Module

[0100] The test task management module is responsible for handling the testing phase after the development team leader approves the test. This module first requests test table resources, then executes system integration testing (SIT), user acceptance testing (UAT), and scheduling testing in sequence, and finally generates production-ready tasks.

[0101] (1) System Integration Testing (SIT) Phase

[0102] Automatically create test state structure definitions (schema), with the naming convention: Task level number_Data requirement batch number_SIT;

[0103] The test data originated from the production environment and was directly imported into the SIT test environment.

[0104] Perform SIT automated testing and write test scripts based on preset test points;

[0105] Generates SIT automated test reports, supporting test pipeline feedback and manual difference comparison tests.

[0106] (2) User Acceptance Testing (UAT) Phase

[0107] Automatically create test state structure definitions (schema), with the naming convention: Task level number_Data requirement batch number_UAT;

[0108] The test data is based on production environment data and imported into the UAT test environment;

[0109] Execute UAT automated tests and write test scripts according to the test requirements;

[0110] Generates UAT automated test reports, supporting test pipeline feedback and manual difference comparison tests.

[0111] (3) Scheduling test phase

[0112] Perform job dependency testing to verify batch processing between the parent and current levels, ensuring the integrity and correctness of job dependencies.

[0113] The test task management module also includes the following sub-modules:

[0114] Test table resource allocation submodule: responsible for allocating corresponding table resources and supporting dynamic table resource expansion function.

[0115] Pre-test report submodule: Provides the function of viewing pre-test reports.

[0116] Test pipeline feedback submodule: Supports viewing the automated test pipeline of SIT and UAT.

[0117] Manual testing submodule: Supports data comparison and dynamic desensitization preview functions.

[0118] The scheduling test submodule verifies the correctness of dependency configurations.

[0119] 6. Production Task Management Module

[0120] The production launch task management module prepares for production launch based on the tasks completed during testing. This module analyzes the impact between operations, reviews the corresponding requirements, and determines the scope of production launch according to the granularity of task or data requirements.

[0121] After the review is completed, the production task management module can perform a one-click production deployment operation for all development tasks awaiting deployment. An example is as follows:

[0122] Automatically synchronize the mapping information of development results to the production environment;

[0123] Automatically push batch job information to the job scheduling management system (Control-M);

[0124] End-of-day batch processing starts automatically at 8 p.m., and the entire process requires no manual intervention.

[0125] How the various functional modules of the system cooperate

[0126] 1. The business requirements management module interfaces with the project management platform, handling data-related requirements broken down by the platform. After receiving the requirements, AI technology is used to analyze them and generate corresponding data-related business requirements.

[0127] 2. The data requirements management module interfaces with the business requirements management module. In this module, tasks at the same hierarchical level are managed together under a single data requirements batch. Requirements analysts apply for the corresponding table resources to conduct preliminary business requirements research; once the research is approved, the corresponding data requirements are broken down.

[0128] 3. The task hierarchy management module, based on the data requirements confirmed by the data requirements management module, is responsible for breaking down development task levels. The task hierarchy includes a standard layer, a model layer, a metric layer, and an application layer, with a dynamic confirmation mechanism. Once each task level is confirmed, the automatically created development tasks will be associated with a data requirement batch number.

[0129] 4. After the task hierarchy is confirmed, the development task management module automatically splits the development tasks. Once the corresponding level of development tasks is created, a schema for that level is created on the big data platform, named as follows: Task Hierarchy Number_Data Requirement Batch Number.

[0130] _DEV. These schemas are established in the production environment, directly using production environment data to achieve dynamic allocation of three-state resources. After developers obtain development tasks through a bidding model, they construct SQL using a graphical interface, and after completing development, they perform unit testing and an approval process.

[0131] 5. After the task development is completed, the test task management module enters the testing phase. Testers first apply for test table resources, and then conduct SIT testing, UAT testing and scheduling testing in sequence.

[0132] 6. After the test task passes, the production task management module enters the production stage, conducts expert production review, executes one-click production operation, and realizes the task production.

[0133] The following example of a specific data development task illustrates the data development process based on a data pipeline.

[0134] Step 1: The project management platform receives the retail personal loan tag request from the retail credit department, project number: YWYH20250808.

[0135] Step 2: The project management platform synchronizes business requirements to the data pipeline system. It uses AI to analyze retail personal loan tag requirements and breaks down data-related business requirements into the data pipeline system. The business requirement number is YWYH2025080801.

[0136] Step 3: Upon receiving the retail personal loan tag requirement YWYH2025080801, the business requirements analyst conducts a preliminary research. After the preliminary research is completed, the following information is filled in: research period, researchers, research results (pass / fail), research location, project manager, and requirements analyst.

[0137] Step 4: After the preliminary research is completed, business requirements are broken down. The data requirement is: YWYH202508080101. The table resources FDM_TO1_B1_PR_CUSTOMER, FDM_T01_B1_PR_EBNK_USERS_INFO, and FDM_T11_B1_EBNK_PARTY_CHNL_REAL are applied for.

[0138] Step 5: Based on the data requirement YWYH202508080101 retail personal loan tag requirement, confirm the task level, split it into indicator layers, and automatically create two indicator layer tasks: the indicator task for the current day of personal customer mobile banking activation date, and the derived indicator task for the current day of personal customer mobile banking activation days.

[0139] Step Six: After creating the indicator-level development tasks, create the corresponding development-level schema on the big data platform, following the rule: Task Level Number_Data Requirement Batch Number_DEV, IDM_YWYH202508080101_DEV. These schemas are established in the production environment, directly using production environment data to achieve dynamic allocation of resources for System Integration Testing (SIT), User Acceptance Testing (UAT), and Scheduled Testing.

[0140] Step 7: Developers confirm task assignments based on the development task list using a bidding process. Developers request development resources: FDM_TO1_B1_PR_CUSTOMER, FDM_T01_B1_PR_EBNK_USERS_INFO, and FDM_T11_B1_EBNK_PARTY_CHNL_REAL, and select the task related to the current date of mobile banking activation for individual clients to access the development page.

[0141] Step 8: Perform one-click benchmarking based on the indicator's Chinese name, and automatically translate the standard English name using the standard platform: PRVT_CUST_MBLBNK_DRDG_DT. Automatically query the standard using the field's Chinese name to supplement standard information such as the field's English name, field length, precision, and type.

[0142] Step 9: On the page, select the relevant parameters by dragging and dropping to generate the complete SQL statement. On the page, select the value range: greater than or equal to 0, the unit of measurement: none, and the data source type: inline. Select the source table.

[0143] Perform an INNER JOIN operation on FDM_TO1_B1_PR_CUSTOMER, FDM_T01_B1_PR_EBNK_USERS_INFO, and FDM_T11_B1_EBNK_PARTY_CHNL_REAL, configure the WHERE and GROUP BY conditions, and generate a complete SQL statement.

[0144] SELECT'{TX_DATE}'as ETL_DATE,T1.PRVT_CUST_NO as

[0145] PRVT_CUST_NO,'PRVT_CUST_MBLBNK_DRDG_DT'as

[0146] INDEX_ENAME,min(DATE(t3.OPEN_TIME))as INDEX_RESULT

[0147] from FDM.FDM_TO1_B1_PR_CUSTOMER T1

[0148] INNER JOIN FDM.FDM_T01_B1_PR_EBNK_USERS_INFO T2 on t1.orvt_cust_no=t2.cust_no and t2.etl_date='{TX_DATE}'AND t2.user_status='1'

[0149] INNER JOIN FDM.FDM_T11_B1_EBNK_PARTY_CHNL_REAL T3 on t2.party_no=

[0150] t3.party_no AND t3.status='1'AND t3.etl_date='{TX_DATE}'AND t3.chnl_no='020401'

[0151] WHERE T1.etl_date='{TX_DATE}'

[0152] GROUP BY t1.prvt_cust_no

[0153] Step 10: Configure the corresponding job dependencies. Configure the following job dependencies: FDM_TO1_B1_PR_CUSTOMER, FDM_T01_B1_PR_EBNK_USERS_INFO, FDM_T11_B1_EBNK_PARTY_CHNL_REAL.

[0154] Step 11: After development is complete, perform unit testing, generate an automated test report, and submit it to the development team leader for review. After development is complete, perform unit testing, preview the table creation and ETL statements online, and execute the table creation and ETL scripts. During data preview, automatically parse the ETL statements, identify query fields, and dynamically mask them (data security control). Execute tests using Python scripts, generate an automated test report, and submit it to the development team leader for review.

[0155] For the task of quantifying the daily progress of mobile banking activation for individual customers, unit testing is performed after development is completed. Online preview of table creation and ETL scripts is conducted, and the corresponding table creation and ETL scripts are executed, displaying the result data preview. During data preview, the ETL statements are automatically parsed, and after identifying the query fields, dynamic desensitization is performed based on the desensitization function in the standard information mapped to those fields to ensure that sensitive data is not leaked (data security control). Then, test points are configured through configuration, batch runs are performed using Python scripts, and an automated test report is generated for result display (automated testing). After the test report is generated, it is submitted to the development team leader for review. The review includes the ETL script and the automated test report. After the team leader's review is completed, the task enters the SIT testing phase.

[0156]

[0157] Automated Test Report:

[0158] Table level:

[0159]

[0160] Field level:

[0161]

[0162]

[0163] Step 12: The task enters SIT testing, automatically creating a test schema and executing test requirements. The task enters SIT testing, automatically creating the test schema: IDM_YWYH202508080101_SIT. Test data is based on production environment data. SIT testers are assigned, and they view the pre-test report, execute SIT test requirements, and perform test pipeline feedback and dynamic data anonymization preview.

[0164] The task for the daily mobile banking activation date target for private clients enters SIT testing. First, in SIT testing, a test schema is automatically created with the following rules: Task Level Number_Data Requirement Batch Number_SIT, schema name: IDM_YWYH202508080101_SIT. The test data is based on production environment data and is pumped into the production environment's SIT test schema (dynamic allocation of three-state resources). Then, a corresponding SIT tester, Zhang XX, is assigned, with a SIT test start time of 2025-08-20 and an SIT test end time of 2025-08-26. The tester already has the corresponding developer table permissions. Special test table permissions are granted by... For this test case, no additional form access is required upon special request from the tester. The SIT tester first reviews the pre-test report: ETL script, automated test report, and corresponding job dependencies. Then, they set the corresponding ETL date and run the corresponding SIT test points. The tester provides test flow feedback based on the corresponding test flow. The SIT tester can preview the dynamic data anonymization. During the data preview, the ETL statement is automatically parsed, and after identifying the query fields, the anonymization is automatically performed dynamically according to the anonymization function in the standard information mapped to that field, ensuring that sensitive data is not leaked (data security control). After the SIT test passes, it proceeds to the UAT test. If the test fails, the reason is noted and it is returned to the development team leader for review.

[0165]

[0166]

[0167] Automated Test Report: Table Level:

[0168]

[0169] Field level:

[0170]

[0171]

[0172] Step Thirteen: After the SIT test passes, proceed to the UAT test for manual testing and data comparison. The UAT test schema is automatically created after the SIT test passes: IDM_YWYH202508080101_UAT. Testers review the unit test and SIT test pre-test reports, execute the UAT test procedures, perform manual testing, and compare the MINUS results to ensure data consistency.

[0173] The test data is based on production environment data, which is then fed into the UAT test state (dynamic allocation of three-state resources) in the production environment. UAT testing is then conducted. First, a UAT tester, Li XX, is assigned the test start time: 2025-08-27, and the test end time: 2025-08-30. The tester already has the corresponding table permissions from the developers; special test table permissions are requested separately by the tester. The UAT tester first reviews the unit tests, SIT test pre-test reports (ETL scripts, automated test reports), and corresponding job dependencies, and then sets up the appropriate ETL. The corresponding UAT test points are run on the specified dates. Testers provide test pipeline feedback based on the corresponding test pipeline. UAT testers can conduct manual testing. In this stage, testers can first inject data into the target table of the test state according to the developer's processing rules, and then verify whether the preview data meets the requirements through data filtering and other methods. In addition, testers can also create a comparison table TMP_YWYH202508080101.PRVT_CUST_MBLBNK_DRDG_DT_0001 using their own processing methods, and compare it with the target table processed by the data developer's processing rules.

[0174] Perform left join, right join, and inner join on IDM_YWYH202508080101_UAT.IDM_I01_PRVT_CUST_ATTRI_INDX_RES_INFO respectively to check for differences in data, i.e., MINUS result comparison. If no differences are found, further verification of the reliability of the indicator development results is performed. UAT testers can preview the dynamic data anonymization. During the data preview, the ETL statement is automatically parsed, and after identifying the query fields, dynamic anonymization is performed automatically according to the anonymization function in the standard information mapped to the field to ensure that sensitive data is not leaked (data security control). After the UAT test passes, it enters the dependency scheduling test. If the test fails, the reason is marked and it is returned to the development team leader for review.

[0175]

[0176] Automated Test Report:

[0177] Table level:

[0178]

[0179] Field level:

[0180]

[0181]

[0182] MINUS vs. SQL:

[0183]

[0184] Step Fourteen: Conduct scheduling tests to verify job dependencies. After successful testing, proceed to production review. Analyze the impact at the job, table, and field levels using metadata to assess downstream impact. See details for job dependencies. Figure 1 .

[0185] Step 15: Utilize draft metadata from the metadata system to achieve parallel development. Develop the task related to the number of days since the individual customer activated mobile banking as a derived metric within the same requirement batch as the first task. Utilize the draft metadata of the first task in the metadata system to achieve parallel development (hierarchical task parallel development).

[0186] The test task is transferred to the corresponding production task and enters the expert production review stage. The impact analysis of the corresponding task at the job, table and field level is pulled out through metadata to analyze the downstream data usage. After the review is completed, the task management of the indicator of the date of opening mobile banking for private customers can perform a one-click production operation on all development tasks to be put into production. It will automatically synchronize the development results mapping information to the production state and automatically push the batch run information to Control-M. The end-of-day batch run will start automatically at 8 pm without manual intervention, realizing the task production. After the entire task is put into production, the corresponding data requirements are also put into production.

[0187] This task involves developing a derived indicator for the number of days since the activation of mobile banking for individual customers. This task and the task for the same indicator are within the same data requirement batch (batch number: YWYH202508080101). During development, the indicator is registered in the draft state of the metadata system. This allows the current task to retrieve the upstream draft metadata based on the batch number, enabling parallel development of tasks at the same and different levels within the same batch (parallel development of hierarchical tasks). The development, testing, and deployment process for this task is similar to that of the task for the same customer's mobile banking activation date.

[0188] Step Sixteen: After all tasks are developed and deployed, the entire requirement batch is deployed. Once both tasks are developed and deployed, the entire requirement batch is deployed, and the entire data pipeline project process is complete.

[0189] Based on the above implementation methods, this invention proposes a data development method based on a data pipeline, comprising the following steps:

[0190] Business requirements management steps: Connect with external project management platforms, synchronize business requirements, and identify and decompose data-related business requirements through intelligent parsing;

[0191] Data requirement management steps: Based on the data-related business requirements, conduct preliminary research and analysis, apply for resources, and break them down into specific data requirement tasks. Manage the tasks that need to be put into production in the same batch under one data requirement batch.

[0192] Task hierarchy management steps: Based on the data requirement task, determine the data hierarchy to which the task belongs, output the task structure with hierarchy information, and associate the development task with a unified data requirement batch number;

[0193] Development task management steps: Based on task hierarchy information, automatically generate development tasks, create development state structure definitions in the production environment, build a structured query language in a graphical way, execute unit tests and automated data preview and desensitization, generate test reports and submit them for review, and generate test tasks after the review is approved.

[0194] Test task management steps: For approved development tasks, system integration testing, user acceptance testing, and scheduling testing are executed sequentially. The corresponding test state structure definition is automatically created, automated testing is performed based on production environment data, and reports are generated.

[0195] Production task management steps: Conduct production review for tasks that have passed testing, execute one-click production operation, automatically synchronize development results to the production environment, and push job information to the scheduling system.

[0196] Furthermore, the development task management steps, specifically including the graphical method for constructing a structured query language, include:

[0197] Select table resources by dragging and dropping, and retrieve field information from the metadata system;

[0198] The one-click benchmarking function automatically matches the standard platform based on the Chinese name of the field and supplements the standard information.

[0199] Obtain recommended inter-table relationships from the metadata system;

[0200] Choose a built-in function and write a conditional statement;

[0201] The processing steps are assembled into mapping data and stored.

[0202] A complete structured query language is generated based on the mapping data.

[0203] Furthermore, the development task management steps also include:

[0204] After the development task at each level is completed, its metadata is registered to the draft state of the metadata system;

[0205] Downstream development tasks query and reference draft metadata from upstream layers based on data requirement batch numbers, enabling parallel development across layers.

[0206] Furthermore, in the development task management steps and test task management steps, corresponding structure definitions are automatically created in the production environment cluster according to the task status, including development state, system integration test state and user acceptance test state structure definitions, to realize dynamic allocation and logical isolation of resources in the three states.

[0207] Furthermore, in the test task management step, the extraction, transformation, and loading statements are automatically parsed during data preview, the query fields are identified, and dynamic desensitization is performed according to the desensitization function mapped in their standard information.

[0208] This invention also proposes a data development system based on a data pipeline, comprising:

[0209] The business requirements management module is used to connect with external project management platforms, synchronize business requirements, and identify and decompose data-related business requirements through intelligent parsing.

[0210] The data requirement management module is used to conduct preliminary research and analysis based on the data-related business requirements, apply for form resources, and break them down into specific data requirement tasks, and manage the tasks that need to be put into production in the same batch under one data requirement batch.

[0211] The task hierarchy management module is used to determine the data hierarchy to which the task belongs based on the data requirement task, output the task structure with hierarchy information, and associate the development task with a unified data requirement batch number.

[0212] The development task management module is used to automatically generate development tasks based on task hierarchy information, create development state structure definitions in the production environment, build a structured query language in a graphical way, execute unit tests and automated data preview and desensitization, generate test reports and submit them for review, and generate test tasks after the review is approved.

[0213] The test task management module is used to execute system integration testing, user acceptance testing and scheduling testing sequentially for approved development tasks, automatically create corresponding test state structure definitions, execute automated tests based on production environment data and generate reports.

[0214] The production task management module is used to review tasks that have passed testing for production deployment, execute one-click production deployment operations, automatically synchronize development results to the production environment, and push job information to the scheduling system.

[0215] Furthermore, the development task management module also includes a graphical development unit, used for:

[0216] Select table resources by dragging and dropping, and retrieve field information from the metadata system;

[0217] The one-click benchmarking function automatically matches the standard platform based on the Chinese name of the field and supplements the standard information.

[0218] Obtain recommended inter-table relationships from the metadata system;

[0219] Choose a built-in function and write a conditional statement;

[0220] The processing steps are assembled into mapping data and stored.

[0221] A complete structured query language is generated based on the mapping data.

[0222] Furthermore, the development task management module is also configured as follows:

[0223] After the development task at each level is completed, its metadata is registered to the draft state of the metadata system;

[0224] Downstream development tasks query and reference draft metadata from upstream layers based on data requirement batch numbers, enabling parallel development across layers.

[0225] Furthermore, the development task management module and the test task management module are configured to automatically create corresponding structure definitions in the production environment cluster according to the task status, including structure definitions for development state, system integration test state and user acceptance test state, so as to realize the dynamic allocation and logical isolation of resources in the three states.

[0226] Furthermore, the development task management module and the test task management module are configured to automatically parse the extraction, transformation, and loading statements during data preview, identify query fields, and dynamically de-identify them according to the de-identification functions mapped in their standard information.

[0227] This invention achieves fully online management of data projects from business requirement creation to production deployment through the interaction of six modules: business requirement management, data requirement management, task hierarchy management, development task management, testing task management, and production task management. The system stores documentation materials for all stages and records a complete operation log, ensuring that all stages are controllable and auditable. The data project development process is 100% sequential, eliminating manual offline handover and mitigating the risk of handover errors; development efficiency is significantly improved, and project delivery cycle is shortened by 40%.

[0228] This invention enables parallel development of hierarchical tasks. The data requirements management module manages tasks within the same batch, associating development tasks at each task level with data requirements batch numbers. Completed task metadata is registered in the draft state of the metadata system, allowing downstream tasks to query upstream draft state metadata based on the requirement batch number for development. This breaks through the traditional serial development model, achieving cross-level parallel development and improving development efficiency by 60%.

[0229] This invention enables dynamic allocation of resources across three states. It automatically creates structural schemas for development, testing, and production environments based on task status. Tasks then execute ETL statements under the corresponding schema according to their status. This achieves automatic resource allocation and isolation, ensuring data security and mitigating the risks associated with manual operations.

[0230] This invention enables one-click benchmarking. The development task management module automatically retrieves field information from the metadata system, matches the Chinese field names with the standard system, and automatically fills in standard information (Chinese and English names, data types, length, precision, etc.). This ensures the standardization of data outputs, simplifies data asset inventory work, and lays the foundation for data asset management.

[0231] This invention enables automated testing. The test task management module maintains the key points for automated testing, automatically executing tests and generating reports during the unit testing, SIT, and UAT phases. The test points support dynamic maintenance. It provides reliable verification criteria for development deliverables, significantly reducing testing and review workload and ensuring development quality.

[0232] This invention enables one-click production deployment. The production deployment task management module supports one-click deployment, automatically synchronizing mapping data to the production environment and pushing job information to the Control-M scheduling system for scheduled automatic operation. This simplifies the production deployment process, avoids manual configuration errors, and improves production deployment efficiency.

[0233] This invention enables secure data management. Data is processed through a graphical interface, and the processed information is stored as structured mapped data. During data preview, ETL statements are automatically parsed, and dynamic anonymization is performed based on standard field information. This effectively protects sensitive data and enhances data security and compliance.

[0234] At the business requirements management level, this invention connects to the DevOps platform through a business requirements management module, automatically synchronizing business requirements and documents to achieve unified requirements management and avoid errors and misunderstandings caused by manual handover.

[0235] This invention provides a graphical development interface for task management, reducing manual SQL errors through drag-and-drop functionality and standardized mapped data storage. It also supports version comparison, facilitating developers' verification and optimization.

[0236] This invention provides dynamically configurable automated testing points and automatically generates test reports at the testing and production management level. It enables one-click production deployment, automatically synchronizes configuration information, ensures environmental consistency, and avoids human error.

[0237] This invention employs a three-state data isolation mechanism in the production environment, which ensures data security and solves the problem of inconsistency between development and testing environments.

[0238] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0239] It should be understood that, in order to simplify the present invention and help those skilled in the art understand its various aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or with reference to a single figure. However, the present invention should not be construed as including all features in the exemplary embodiments as essential technical features of the claims of this patent.

[0240] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0241] It should be understood that the modules, units, components, etc., included in the device of one embodiment of the present invention can be adaptively changed to be placed in a device different from that embodiment. Different modules, units, or components included in the device of the embodiment can be combined into a single module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

[0242] The modules, units, or components in the embodiments of the present invention can be implemented in hardware, in software running on one or more processors, or in a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement the embodiments of the present invention. The present invention can also be implemented on a computer program product or computer-readable medium for performing some or all of the methods described herein.

Claims

1. A data development method based on data pipelines, characterized in that, Includes the following steps: Business requirements management steps: Connect with external project management platforms, synchronize business requirements, and identify and decompose data-related business requirements through intelligent parsing; Data requirement management steps: Based on the data-related business requirements, conduct preliminary research and analysis, apply for resources, and break them down into specific data requirement tasks. Manage the tasks that need to be put into production in the same batch under one data requirement batch. Task hierarchy management steps: Based on the data requirement task, determine the data hierarchy to which the task belongs, output the task structure with hierarchy information, and associate the development task with a unified data requirement batch number; Development task management steps: Based on task hierarchy information, automatically generate development tasks, create development state structure definitions in the production environment, build a structured query language in a graphical way, execute unit tests and automated data preview and desensitization, generate test reports and submit them for review, and generate test tasks after the review is approved. Test task management steps: For approved development tasks, system integration testing, user acceptance testing, and scheduling testing are executed sequentially. The corresponding test state structure definition is automatically created, automated testing is performed based on production environment data, and reports are generated. Production task management steps: Conduct production review for tasks that have passed testing, execute one-click production operation, automatically synchronize development results to the production environment, and push job information to the scheduling system.

2. The method according to claim 1, characterized in that, The development task management steps, specifically the use of a graphical approach to construct a structured query language, include: Select table resources by dragging and dropping, and retrieve field information from the metadata system; The one-click benchmarking function automatically matches the standard platform based on the Chinese name of the field and supplements the standard information. Obtain recommended inter-table relationships from the metadata system; Choose a built-in function and write a conditional statement; The processing steps are assembled into mapping data and stored. A complete structured query language is generated based on the mapping data.

3. The method according to claim 1, characterized in that, The development task management steps also include: After the development task at each level is completed, its metadata is registered to the draft state of the metadata system; Downstream development tasks query and reference draft metadata from upstream layers based on data requirement batch numbers, enabling parallel development across layers.

4. The method according to claim 1, characterized in that, In the development task management steps and test task management steps, corresponding structure definitions are automatically created in the production environment cluster according to the task status, including development state, system integration test state and user acceptance test state structure definitions, to realize dynamic allocation and logical isolation of resources in the three states.

5. The method according to claim 1, characterized in that, In the test task management steps, the extraction, transformation, and loading statements are automatically parsed during data preview, the query fields are identified, and dynamic desensitization is performed according to the desensitization function mapped in their standard information.

6. A data development system based on a data pipeline, characterized in that, include: The business requirements management module is used to connect with external project management platforms, synchronize business requirements, and identify and decompose data-related business requirements through intelligent parsing. The data requirement management module is used to conduct preliminary research and analysis based on the data-related business requirements, apply for resource forms, and break them down into specific data requirement tasks, and manage the tasks that need to be put into production in the same batch under one data requirement batch. The task hierarchy management module is used to determine the data hierarchy to which the task belongs based on the data requirement task, output the task structure with hierarchy information, and associate the development task with a unified data requirement batch number. The development task management module is used to automatically generate development tasks based on task hierarchy information, create development state structure definitions in the production environment, build a structured query language in a graphical way, execute unit tests and automated data preview and desensitization, generate test reports and submit them for review, and generate test tasks after the review is approved. The test task management module is used to execute system integration testing, user acceptance testing and scheduling testing sequentially for approved development tasks, automatically create corresponding test state structure definitions, execute automated tests based on production environment data and generate reports. The production task management module is used to review tasks that have passed testing for production deployment, execute one-click production deployment operations, automatically synchronize development results to the production environment, and push job information to the scheduling system.

7. The system according to claim 6, characterized in that, The development task management module also includes a graphical development unit, used for: Select table resources by dragging and dropping, and retrieve field information from the metadata system; The one-click benchmarking function automatically matches the standard platform based on the Chinese name of the field and supplements the standard information. Obtain recommended inter-table relationships from the metadata system; Choose a built-in function and write a conditional statement; The processing steps are assembled into mapping data and stored. A complete structured query language is generated based on the mapping data.

8. The system according to claim 6, characterized in that, The development task management module is also configured as follows: After the development task at each level is completed, its metadata is registered to the draft state of the metadata system; Downstream development tasks query and reference draft metadata from upstream layers based on data requirement batch numbers, enabling parallel development across layers.

9. The system according to claim 6, characterized in that, The development task management module and the test task management module are configured to automatically create corresponding structure definitions in the production environment cluster according to the task status, including structure definitions for development state, system integration test state and user acceptance test state, so as to realize the dynamic allocation and logical isolation of resources in the three states.

10. The system according to claim 6, characterized in that, The development task management module and the test task management module are configured to automatically parse, extract, transform, and load statements during data preview, identify query fields, and dynamically desensitize them according to the desensitization functions mapped in their standard information.

Images