A data governance link construction method supporting multi-tasking

Through multi-task linkage mechanism and modular design, the problems of tight resources and insufficient collaboration capabilities in the existing data governance processing methods are solved, efficient and flexible data governance processing are achieved, system performance and stability are improved, and complex business needs are adapted to the needs of complex business .

CN118819787BActive Publication Date: 2024-11-26BEIJING HUITONG JINCAI INFORMATION TECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411293166.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-14
Publication Date
2024-11-26
Estimated Expiration
2044-09-14

AI Technical Summary

Technical Problem

The existing data governance processing methods have problems such as resource shortage, inability to split scheduling and lack of coordination capabilities, resulting in performance degradation and system crashes, making it difficult to flexibly utilize resources, and processing efficiency and accuracy are low.

Method used

Adopt multi-task linkage mechanism to build multiple task nodes, support multi-task processing, dynamically adjust task execution order and priority according to priority and dependency scheduling, adopt modular design to achieve flexible input and output, and support fast function customization and extension.

Benefits of technology

Effectively utilize system resources, improve processing efficiency, avoid performance bottlenecks, ensure the stability and reliability of data governance processes, reduce operational costs, and adapt to complex data governance needs and business changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118819787B_ABST
    Figure CN118819787B_ABST
Patent Text Reader

Abstract

The present invention discloses a data governance link construction method supporting multi-task processing, which relates to the technical field of data governance link construction, and includes the following steps: constructing multiple task nodes, each task node including a data input module, a data processing module and a data output module; the data input module is used to obtain input data from multiple data sources, and a SQL script is used to describe the way of obtaining input data from the data source, and supports multi-table joint input, result nested query and dynamic configuration of data source. The present invention processes multiple data governance tasks simultaneously through a multi-task linkage mechanism, flexibly schedules according to priority and dependency, improves resource utilization efficiency, and dynamically adjusts the execution order and priority by splitting governance requirements into multiple tasks to avoid performance bottlenecks. It adopts modular and configurable design, supports rapid customization and expansion of task nodes, and adapts to various complex data governance scenarios and requirements.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data governance link construction, and in particular to a data governance link construction method supporting multi-tasking processing. Background Art

[0002] When dealing with data governance tasks, existing methods usually adopt a one-time processing approach, such as Figure 3 A complete processing node is completed in the order shown. In this mode, each processing node is a closed system with one input and one output, that is, data enters from the data input module, and the metadata management module maintains the data table and its field information. The data management module maintains the processor description, scheduling, input, output, and mapping information, and provides start-stop control, execution control and other functions. The security review module then reviews and records the data governance tasks. The data processing module has built-in multiple predefined data mapping algorithms, which can perform data encryption, decryption, desensitization, dictionary mapping, data formatting and other operations, and finally outputs the results through the data output module. This one-time, sequential processing method has the following obvious disadvantages:

[0003] 1. It is easy to generate sudden resource demands. Since all tasks are completed in one processing, when the amount of data processed increases suddenly or encounters complex processing tasks, the system needs to provide a large amount of computing resources and computing power instantly. This may lead to sudden resource shortages and bottlenecks, especially when processing critical business data, which may cause performance degradation or even system crashes.

[0004] 2. Unable to split the scheduling. Since the processing nodes are completed at one time, it is difficult for the system to split the task into smaller parts and schedule them on different time periods or resources. This limits the system's flexible use and optimization of resources and increases the difficulty of processing complex tasks.

[0005] 3. Lack of coordination between processing nodes. Each processing node works independently and lacks coordination and communication capabilities. This leads to difficulties in data transmission, task coordination and result integration between multiple processing nodes, reducing the overall processing efficiency and accuracy.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a data governance link construction method that supports multi-task processing. By introducing a multi-task linkage mechanism, it supports the simultaneous processing of multiple data governance tasks, and can be flexibly scheduled according to the priority and dependency of the tasks, so as to effectively utilize system resources and improve processing efficiency. By splitting the data governance needs into multiple coordinated tasks, the performance bottleneck caused by centralized processing is avoided, and the execution order and priority of tasks are dynamically adjusted according to the real-time usage of system resources to achieve balanced utilization of resources. The method adopts a modular and configurable design, so that each task node has flexible input and output, supports rapid function customization and expansion, adapts to complex data governance needs, and can be continuously optimized and evolved with business changes and data growth to solve the problems in the above-mentioned background technology.

[0008] In order to achieve the above object, the present invention provides the following technical solution: a data governance link construction method supporting multi-task processing, comprising the following steps:

[0009] Construct multiple task nodes, each task node includes a data input module, a data processing module and a data output module;

[0010] The data input module is used to obtain input data from multiple data sources. It uses SQL scripts to describe the method of obtaining input data from data sources. It supports multi-table joint input, nested query of results, and dynamic configuration of data sources. The data sources include Oracle, Mysql, MongoDb, strongly isolated Oracle, and strongly isolated Mysql.

[0011] The data processing module is used to establish the correspondence between input data and output data. It includes two parts: data correspondence and data conversion. It supports multiple predefined data mapping operators, including data encryption, decryption, desensitization, dictionary mapping, data formatting, and custom operators implemented through extended interfaces.

[0012] The data output module is used to output processed data. It supports insert, update, merge, and delete operations of Oracle, Mysql, MongoDb, strongly isolated Oracle, and strongly isolated Mysql, and performs CRUD operations according to usage requirements.

[0013] Preferably, task nodes can be dynamically allocated and adjusted, support multi-task linkage mechanism, and process multiple data governance tasks at the same time; the input and output configuration of link nodes can be dynamically adjusted according to actual needs to adapt to different data processing requirements;

[0014] The input and output configurations of the link nodes are dynamically adjusted according to actual needs to adapt to different data processing requirements. The specific steps are as follows:

[0015] According to business needs, we identify the data sets and their characteristics that need to be processed, determine parameters such as data type, field size, data change rate, and allowed processing delay time, and calculate the priority of data processing. The calculation expression is: , where is the data type, is the field size, is the data change rate, is the maximum allowed processing delay time, f is a multivariable function used to determine the data processing priority;

[0016] Based on the results of data demand analysis, adjust the configuration of the data input module, including data source connection and data format conversion, and determine the optimized input channel capacity. The calculation expression is: , where is the input channel capacity, indicating the maximum data processing capability of the input module. is the optimized input channel capacity;

[0017] According to the adjustment of input configuration, optimize the operator chain configuration in the data processing module, define and adjust the order and combination of operators, and calculate the execution efficiency of the optimized operator chain. The calculation expression is: , where is the execution efficiency of the operator chain, is the ith operator, is the resource requirement of the ith operator, is the weight of each operator;

[0018] Based on the optimized data processing flow, adjust the configuration of the data output module to ensure that the output meets business requirements and determine the optimized output channel capacity. The calculation expression is: , where is the output channel capacity, indicating the maximum data output capability of the output module. is the optimized output channel capacity.

[0019] Preferably, the data processing module adopts an operator chain approach to combine or nest multiple operators to implement complex data processing processes. The operator chain can reduce communication overhead, improve execution efficiency and reduce resource usage, support sequential and parallel processing of data, and optimize data processing performance and resource utilization efficiency;

[0020] The data processing module uses an operator chain to combine or nest multiple operators to implement complex data processing processes. The specific steps are as follows:

[0021] In the first step of the data processing module, data preprocessing mainly includes data cleaning and format conversion to ensure data consistency and integrity. The main task of this step is to screen valid data and perform necessary type conversion to prepare for subsequent data processing. The preprocessing expression is: , where D is the original input data set, which contains multiple data fields and records, and S is the filter function used to filter the data records that meet the conditions, defined as , where x is the data record, returns 1 to indicate valid data, returns 0 to indicate invalid data, and T is a conversion function used to convert data types, such as string to integer, defined as , where y is the original data field, is the converted data field, It is a preprocessed data set, which contains the data that has been filtered and format converted;

[0022] In the data preprocessing stage, the filter function S is first applied to filter out all valid data records. Then, the conversion function T is applied to the fields of each data record to convert the data into the required format and type. This step ensures that all input data meets the processing requirements.

[0023] The data mapping step mainly maps the input data to the target output data according to the specific needs of data processing. By applying the mapping function, a one-to-one or one-to-many mapping relationship between fields is achieved. The mapping expression is: , where M is the mapping result set, which contains all the mapped data records. is the data record after preprocessing, records, is a mapping function that maps input data to output data and is defined as , where y is the input field, k is the output field, F is the mapping rule set, which defines the specific logic and rules of field mapping, including field renaming, data grouping, etc., and n is the total number of records in the preprocessed data set;

[0024] Data conversion and calculation are the core steps of the data processing module. In this stage, the operator chain is applied to the mapped data to perform multiple atomic operations or compound operations, such as data encryption, decryption, data desensitization, data formatting, etc., to generate the final output data. The expression of data conversion and calculation is: , where O is the output result set, which contains all the data records after conversion and calculation. is the data record after mapping, the jth record, is an operator function, defined as , where z is the converted data field, P is the operator parameter set, which contains the specific parameter configuration of each operator in the operator chain, such as encryption key, desensitization strategy, data format, etc., and m is the total number of records in the mapping result set.

[0025] Preferably, the data governance link includes a data pipeline, which is used to transmit data between task nodes to ensure that the output data format and structure of each node match the input requirements of subsequent nodes; the core rule of the data pipeline is front output matching, and it supports defining the data transmission method between nodes through task configuration. Task configuration includes field mapping, data type conversion, data filtering and other operations.

[0026] Preferably, the method includes a global node management module, which is used to schedule and orchestrate data links between task nodes, supports three task triggering strategies: planned triggering, manual direct triggering, and post-triggering, and supports four task scheduling strategies: planned scheduling, single scheduling, automatic scheduling, and slice scheduling.

[0027] Preferably, the method further comprises a plurality of support modules, the support modules comprising:

[0028] The metadata management module is used to maintain data tables and their field information, and manage the permissions between users and table-level and field-level metadata;

[0029] The security review module is used to review and record data governance tasks, provide table-level and field-level read and write permission control, and automatically block unauthorized access and dangerous operations;

[0030] The status query module is used to provide status information such as task progress, associated personnel, and table-level task numbers.

[0031] Preferably, the method supports real-time monitoring and management of task execution, and realizes dynamic scheduling and resource optimization of data governance tasks through the start-stop control, execution control and other functions provided by the task management module.

[0032] Preferably, the method supports multiple security measures, including encrypted data transmission, identity authentication, and permission control, to ensure security and compliance during data processing.

[0033] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0034] The present invention supports the simultaneous processing of multiple data governance tasks by introducing a multi-task linkage mechanism. These tasks can be flexibly scheduled and combined according to factors such as priority and dependency, thereby making full use of system resources and improving overall processing efficiency. The multi-task processing capability can ensure that the data governance process will not be interrupted due to the blocking of a single task, thereby improving the stability and reliability of data governance.

[0035] The present invention divides data governance requirements into multiple tasks that run in coordination with each other. Compared with the method of concentrating all governance operations on one governance node, it can avoid sudden performance requirements and dynamically adjust the execution order and priority of tasks according to the real-time usage of system resources, and evenly utilize system software and hardware resources to achieve continuous and complex data governance operations. It ensures that each task can obtain sufficient resource support, thereby improving resource utilization and overall system performance. This flexible resource management method not only improves resource utilization efficiency, but also reduces operating costs, because enterprises do not need to purchase and maintain a large number of unused resources.

[0036] The present invention adopts a modular and configurable design concept, dividing task integration into different task nodes, each of which has clear inputs and outputs, and can be quickly customized and modified according to actual needs. It can also be flexibly configured and expanded according to different business needs and data characteristics, and flexibly configure various parameters and options of the system. This allows the present invention to adapt to various complex data governance scenarios and continue to evolve with the development of business and the growth of data volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0038] Figure 1 The present invention is a method flow chart of a data governance link construction method supporting multi-tasking processing.

[0039] Figure 2 This is a schematic diagram of an example of the present invention.

[0040] Figure 3 A schematic diagram of the modules of the prior art method of the present invention when processing data management tasks. DETAILED DESCRIPTION

[0041] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art.

[0042] The present invention provides Figure 1 A data governance link construction method supporting multi-tasking processing is shown, comprising the following steps:

[0043] Construct multiple task nodes, each task node includes a data input module, a data processing module and a data output module;

[0044] The data input module is used to obtain input data from multiple data sources. It uses SQL scripts to describe the method of obtaining input data from data sources. It supports multi-table joint input, nested query of results, and dynamic configuration of data sources. The data sources include Oracle, Mysql, MongoDb, strongly isolated Oracle, and strongly isolated Mysql.

[0045] The data processing module is used to establish the correspondence between input data and output data. It includes two parts: data correspondence and data conversion. It supports multiple predefined data mapping operators, including data encryption, decryption, desensitization, dictionary mapping, data formatting, and custom operators implemented through extended interfaces.

[0046] The data output module is used to output processed data. It supports insert, update, merge, and delete operations of Oracle, Mysql, MongoDb, strongly isolated Oracle, and strongly isolated Mysql, and performs CRUD operations according to usage requirements.

[0047] A single task node contains three main logics: data input, data processing, and data output. The data input logic provides the task node with data to be processed; the data processing module establishes the correspondence between input data and output data, and specifies the operators required to convert the data to be processed into processed data; the data output logic outputs the data processed by the task node.

[0048] The data input logic is described using SQL scripts, that is, the input data is obtained from the data source in what way according to the usage requirements. It supports reading, connecting, aggregating and other operations on the original data according to the usage requirements, supports multi-table joint input, and supports nested query of results (responsible for executing various data governance tasks defined in the task management module, including association, aggregation, mapping and other data operations to generate result data for use by the result output module.). At the same time, it supports dynamic configuration of data sources. The data source implicitly includes information such as database instances, databases, users, etc., and clearly defines the access address that carries the table data, so as to distinguish between tables with the same name in different libraries. It supports 5 types of data sources: Oracle, Mysql, MongoDb, strongly isolated Oracle, and strongly isolated Mysql. (Note: Strong isolation is a network boundary firewall device, and the application interacts with data through the database).

[0049] The input and output configurations of the link nodes are dynamically adjusted according to actual needs to adapt to different data processing requirements. The specific steps are as follows:

[0050] According to business needs, we identify the data sets and their characteristics that need to be processed, determine parameters such as data type, field size, data change rate, and allowed processing delay time, and calculate the priority of data processing. The calculation expression is: , where is the data type, is the field size, is the data change rate, is the maximum allowed processing delay time, f is a multivariable function used to determine the data processing priority;

[0051] Based on the results of data demand analysis, adjust the configuration of the data input module, including data source connection and data format conversion, and determine the optimized input channel capacity. The calculation expression is: , where is the input channel capacity, indicating the maximum data processing capability of the input module. is the optimized input channel capacity;

[0052] According to the adjustment of input configuration, optimize the operator chain configuration in the data processing module, define and adjust the order and combination of operators, and calculate the execution efficiency of the optimized operator chain. The calculation expression is: , where is the execution efficiency of the operator chain, is the ith operator, is the resource requirement of the ith operator, is the weight of each operator;

[0053] Based on the optimized data processing flow, adjust the configuration of the data output module to ensure that the output meets business requirements and determine the optimized output channel capacity. The calculation expression is: , where is the output channel capacity, indicating the maximum data output capability of the output module. is the optimized output channel capacity.

[0054] The data processing logic consists of two parts: data correspondence and data conversion. The data correspondence is defined by dynamically configurable task nodes, which maintain a one-to-one relationship between input data fields and output data fields, and is used to establish a connection between input data and output data. Data conversion is defined by dynamically configurable task nodes, which maintain the operators and operator chains required to complete data conversion, and is used to clarify what kind of transformation the input data undergoes to obtain the output data.

[0055] The present invention implements a variety of predefined data mapping operators, including but not limited to data encryption, decryption, desensitization, dictionary mapping and data formatting, and also provides an operator extension interface to implement personalized conversion logic.

[0056] A single operator generally implements atomic operations. For transformations that require the cooperation of multiple atomic operations, the present invention proposes the concept of operator chain, which allows operators to be combined or nested to flexibly implement complex data processing flows and meet diverse data processing needs.

[0057] Operator chains can reduce communication overhead, improve execution efficiency, and reduce resource usage by combining multiple operators into a single task unit. This is because when multiple operators are executed in the same task unit, they can share resources and reduce the transmission and copying of data between different operators. Suppose we have a data processing process that first needs to decrypt the data and then desensitize the plaintext data. By using operator chains, we can combine these two operators together to form a continuous processing flow. The data first enters the decryption operator for processing, and then is handed over to the desensitization operator for processing. The whole process is completed in local memory, reducing the transmission and copying of data between different operators. Especially for decryption scenarios, the copying and transmission of sensitive plaintext data is avoided, which improves the security of the data.

[0058] The data processing module uses an operator chain to combine or nest multiple operators to implement complex data processing processes. The specific steps are as follows:

[0059] In the first step of the data processing module, data preprocessing mainly includes data cleaning and format conversion to ensure data consistency and integrity. The main task of this step is to screen valid data and perform necessary type conversion to prepare for subsequent data processing. The preprocessing expression is: , where D is the original input data set, which contains multiple data fields and records, and S is the filter function used to filter the data records that meet the conditions, defined as , where x is the data record, returns 1 to indicate valid data, returns 0 to indicate invalid data, and T is a conversion function used to convert data types, such as string to integer, defined as , where y is the original data field, is the converted data field, It is a preprocessed data set, which contains the data that has been filtered and format converted;

[0060] In the data preprocessing stage, the filter function S is first applied to filter out all valid data records. Then, the conversion function T is applied to the fields of each data record to convert the data into the required format and type. This step ensures that all input data meets the processing requirements.

[0061] The data mapping step mainly maps the input data to the target output data according to the specific needs of data processing. By applying the mapping function, a one-to-one or one-to-many mapping relationship between fields is achieved. The mapping expression is: , where M is the mapping result set, which contains all the mapped data records. is the data record after preprocessing, records, is a mapping function that maps input data to output data and is defined as , where y is the input field, k is the output field, F is the mapping rule set, which defines the specific logic and rules of field mapping, including field renaming, data grouping, etc., and n is the total number of records in the preprocessed data set;

[0062] Data conversion and calculation are the core steps of the data processing module. In this stage, the operator chain is applied to the mapped data to perform multiple atomic operations or compound operations, such as data encryption, decryption, data desensitization, data formatting, etc., to generate the final output data. The expression of data conversion and calculation is: , where O is the output result set, which contains all the data records after conversion and calculation. is the data record after mapping, the jth record, is an operator function, defined as , where z is the converted data field, P is the operator parameter set, which contains the specific parameter configuration of each operator in the operator chain, such as encryption key, desensitization strategy, data format, etc., and m is the total number of records in the mapping result set.

[0063] The result output uses SQL scripts to describe the data output logic, that is, to describe how to process the result data according to the usage requirements. Support CURD operations on the result data according to the usage requirements. Support insert, update, merge, and delete operations for 5 data sources: Oracle, Mysql, MongoDb, strongly isolated Oracle, and strongly isolated Mysql.

[0064] The data pipeline is the data transmission channel between nodes, which is responsible for combining various task nodes. In essence, it refers to the data matching process between nodes. Its core rule is front-output matching, that is, the output data format and structure of each node should match the input requirements of the subsequent nodes. When data flows from one node to the next, no additional data format conversion or structure adjustment is required manually. This design makes the flow of data intuitive and easy to manage. At the same time, the input configuration of the subsequent nodes is dynamic and can be flexibly adjusted according to actual needs. Users can define multiple nodes in the data pipeline according to business needs and data characteristics, and connect these nodes through front-output matching to form a complete data processing link.

[0065] When building a data pipeline, we create a task configuration for each data transfer between nodes. This configuration specifies how to extract data from the output of the previous node and convert it into the input format required by the next node. The configuration may include field mapping (for example, renaming field A to field B), data type conversion (for example, converting a string to an integer), data filtering (transferring only records that meet certain conditions), etc.

[0066] The present invention performs global node management and is responsible for scheduling and arranging overall data links between nodes.

[0067] To improve the flexibility and efficiency of data processing, node management supports three task triggering strategies:

[0068] 1. Schedule trigger: Automatically execute data processing tasks according to the schedule defined by the Corn expression.

[0069] 2. Manual direct triggering: Users can manually trigger the data processing flow without modifying any input logic.

[0070] 3. Post-triggering: passively triggered by other tasks. For example, if there are three tasks A, B, and C, and B and C are set as post-tasks of A, then tasks B and C will be automatically started after task A is executed. Recursive calls are supported.

[0071] At the same time, node management provides four task scheduling strategies:

[0072] 1. Scheduling: Use Corn expressions to describe the task running schedule to ensure that data processing tasks are executed accurately to the minute at the scheduled time.

[0073] 2. Single scheduling: directly specify the running time through the date control, accurate to the minute, to achieve single task triggering.

[0074] 3. Automatic scheduling: This strategy allows data processing tasks to be triggered by other tasks or manually, improving the level of automation of data processing.

[0075] 4. Sharding Scheduling: Under this strategy, you can specify the current shard value, end shard value, and step size. The data processing task will run according to the predefined shard value calculation logic, and increase the current shard value by the step size after each run. This strategy is suitable for large-scale data processing scenarios, and can effectively disperse processing pressure and improve processing efficiency.

[0076] The above method is carried out by the task scheduling module, input module, data processing module, output module, and supporting modules and executed on demand.

[0077] The task management module performs global node management and is responsible for scheduling and arranging the overall data link between nodes. The task management module maintains processor description, scheduling, input, output, and mapping information, and provides start-stop control, execution control and other functions. It is responsible for three task triggering strategies and four task scheduling strategies between nodes.

[0078] The input module, data processing module and output module exist in each task node. The input and output modules are mainly responsible for the input and output of data in each node. The data processing module is composed of the combination and nesting of operator chains to perform the main data mapping.

[0079] The supporting modules include: metadata management module, security review module and status query module.

[0080] The metadata management module uses the output data of the data input module as input data. The metadata management function maintains the data table and its field information. The table information includes: table name, remarks, business name, business remarks; the field information includes: field name, remarks, business name, data type, data length. The above information can be used for the next step of the task management module configuration. The management maintains the authority relationship between users and table-level and field-level metadata, and does not allow unauthorized use of tables and fields in subsequent steps.

[0081] The security review module is responsible for reviewing and recording data governance tasks. It supports table-level and field-level read and write permission control for binding independent users, automatically blocks unauthorized access and dangerous operations, and ensures data security.

[0082] The status query module provides status information such as task progress, associated personnel, and table-level task numbers by reading the status data of the data processing module.

[0083] Here is an example of the above method running to facilitate understanding: Figure 2 As shown:

[0084] Among them, task A consists of a task chain consisting of 2 task nodes. The input of node A1 is used as the output of node A2, and the data output by node A2 is the output governance data.

[0085] Based on task model A, if different governance operations need to be performed based on a set of original data, a forked task chain can be constructed, as shown in task B. Several B3 nodes that perform different governance operations can use the output of the B2 node as input at the same time.

[0086] Additional complexity is introduced to the task node, making it inappropriate to use a single task node to complete it. Taking B2 as an example, its governance logic can be split arbitrarily. Some nodes in the task chain can be flexibly replaced while keeping the original input and final output unchanged.

[0087] If ABC are three task chains that exist simultaneously, they can be scheduled in parallel.

[0088] The present invention supports the simultaneous processing of multiple data governance tasks by introducing a multi-task linkage mechanism. These tasks can be flexibly scheduled and combined according to factors such as priority and dependency, thereby making full use of system resources and improving overall processing efficiency. The multi-task processing capability can ensure that the data governance process will not be interrupted due to the blocking of a single task, thereby improving the stability and reliability of data governance.

[0089] The present invention divides data governance requirements into multiple tasks that run in coordination with each other. Compared with the method of concentrating all governance operations on one governance node, it can avoid sudden performance requirements and dynamically adjust the execution order and priority of tasks according to the real-time usage of system resources, and evenly utilize system software and hardware resources to achieve continuous and complex data governance operations. It ensures that each task can obtain sufficient resource support, thereby improving resource utilization and overall system performance. This flexible resource management method not only improves resource utilization efficiency, but also reduces operating costs, because enterprises do not need to purchase and maintain a large number of unused resources.

[0090] The present invention adopts a modular and configurable design concept, dividing task integration into different task nodes, each of which has clear inputs and outputs, and can be quickly customized and modified according to actual needs. It can also be flexibly configured and expanded according to different business needs and data characteristics, and flexibly configure various parameters and options of the system. This allows the present invention to adapt to various complex data governance scenarios and continue to evolve with the development of business and the growth of data volume.

[0091] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A data governance link construction method supporting multi-tasking processing, characterized in that: The following steps are involved: Construct multiple task nodes, each task node includes a data input module, a data processing module and a data output module; The data input module is used to obtain input data from multiple data sources. It uses SQL scripts to describe the method of obtaining input data from data sources. It supports multi-table joint input, nested query of results, and dynamic configuration of data sources. The data sources include Oracle, Mysql, MongoDb, strongly isolated Oracle, and strongly isolated Mysql. The data processing module is used to establish the correspondence between input data and output data. It includes two parts: data correspondence and data conversion. It supports multiple predefined data mapping operators, including data encryption, decryption, desensitization, dictionary mapping, data formatting, and custom operators implemented through extended interfaces. The data processing module adopts the operator chain method to combine or nest multiple operators to realize complex data processing processes. The operator chain can reduce communication overhead, improve execution efficiency and reduce resource usage, support sequential and parallel processing of data, and optimize data processing performance and resource utilization efficiency; The data processing module uses an operator chain to combine or nest multiple operators to implement complex data processing processes. The specific steps are as follows: In the first step of the data processing module, data preprocessing includes data cleaning and format conversion to ensure the consistency and integrity of the data and prepare for subsequent data processing. The expression of preprocessing is: ′ =f(D, S, T), where D is the original input data set, which contains multiple data fields and records. S is a filter function used to filter qualified data records, defined as S(x)→{0, 1}, where x is a data record, returns 1 to indicate valid data, and returns 0 to indicate invalid data. T is a conversion function used to convert data types, defined as T(y)→y ′ , where y is the original data field, y ′ is the converted data field, D ′ It is a preprocessed data set, which contains the data that has been filtered and format converted; The data mapping step is to map the input data to the target output data according to the specific needs of data processing. By applying the mapping function, a one-to-one or one-to-many mapping relationship between fields is achieved. The mapping expression is: Where M is the mapping result set, which contains all the mapped data records, and D u ′ is the preprocessed data record, the uth record, m u is a mapping function that maps input data to output data, defined as m u (y, F)→k, where y is the input field, k is the output field, F is the mapping rule set, which defines the specific logic and rules of field mapping, and n is the total number of records in the preprocessed data set; Data transformation and calculation,In this stage, the operator chain is applied to the mapped data,,performing multiple atomic operations or compound operations to generate the final,output data. The expression of data transformation and calculation is: Where O is the output result set, which contains all the converted and calculated data records, and M is j is the data record after mapping, the jth record, g j is an operator function, defined as g j (k, P)→z, where z is the converted data field, P is the operator parameter set, which contains the specific parameter configuration of each operator in the operator chain, and m is the total number of records in the mapping result set; The data output module is used to output processed data. It supports insert, update, merge, and delete operations of Oracle, Mysql, MongoDb, strongly isolated Oracle, and strongly isolated Mysql, and performs CRUD operations according to usage requirements.

2. A data governance link construction method supporting multi-tasking according to claim 1, characterized in that: Task nodes can be dynamically allocated and adjusted, support multi-task linkage mechanism, and process multiple data governance tasks at the same time; the input and output configurations of link nodes can be dynamically adjusted according to actual needs to adapt to different data processing requirements.

3. A method for constructing a data governance link supporting multi-tasking according to claim 2, characterized in that: The input and output configurations of the link nodes are dynamically adjusted according to actual needs to adapt to different data processing requirements. The specific steps are as follows: According to business needs, we identify the data set and its characteristics that need to be processed, determine the data type, field size, data change rate, and allowable processing delay time, and calculate the priority of data processing. The calculation expression is: P data =f(D type , F size , V rate , L latency ), where D type is the data type, F size is the field size, V rate is the data change rate, L latency is the maximum allowed processing delay time, f is a multivariable function used to determine the data processing priority; Based on the results of data demand analysis, adjust the configuration of the data input module, including data source connection and data format conversion, and determine the optimized input channel capacity. The calculation expression is: In the formula, C input is the input channel capacity, indicating the maximum data processing capability of the input module, C opt is the optimized input channel capacity; According to the adjustment of input configuration, optimize the operator chain configuration in the data processing module, define and adjust the order and combination of operators, and calculate the execution efficiency of the optimized operator chain. The calculation expression is: In the formula, O eff is the execution efficiency of the operator chain, O i is the ith operator, R exec (O i ) is the resource requirement of the ith operator, W i is the weight of each operator; Based on the optimized data processing flow, adjust the configuration of the data output module to ensure that the output meets business requirements and determine the optimized output channel capacity. The calculation expression is: In the formula, E output E is the output channel capacity, which indicates the maximum data output capability of the output module. opt is the optimized output channel capacity.

4. A method for constructing a data governance link supporting multi-tasking according to claim 1, characterized in that: The data governance link includes a data pipeline, which is used to transmit data between task nodes to ensure that the output data format and structure of each node matches the input requirements of subsequent nodes; the core rule of the data pipeline is front output matching, and it supports defining the data transmission method between nodes through task configuration.

5. A method for constructing a data governance link supporting multi-tasking according to claim 1, characterized in that: It also includes a global node management module, which is used to schedule and orchestrate data links between task nodes. It supports three task triggering strategies: planned triggering, manual direct triggering, and post-triggering, and supports four task scheduling strategies: planned scheduling, single scheduling, automatic scheduling, and shard scheduling.

6. A method for constructing a data governance link supporting multi-tasking according to claim 1, characterized in that: It also includes multiple supporting modules, including: The metadata management module is used to maintain data tables and their field information, and manage the permissions between users and table-level and field-level metadata; The security review module is used to review and record data governance tasks, provide table-level and field-level read and write permission control, and automatically block unauthorized access and dangerous operations; The status query module is used to provide status information.