Data management method and device, equipment, storage medium and computer program product

Abstract

The invention discloses a data governance method and device, equipment, a storage medium and a computer program product, which are used for solving the problems that an existing data governance scheme cannot dynamically adjust a governance process, the execution efficiency of each stage is low, and parallel execution of governance tasks is difficult to realize. The method comprises the following steps: constructing a data governance graph structure according to acquired configuration information; determining a data governance task corresponding to each processing node in the data governance graph structure, and determining a dependency relationship among the data governance tasks according to edges in the data governance graph structure; according to the dependency relationship, performing task arrangement on each data governance task, and generating a governance task scheduling strategy; and executing each data governance task in an asynchronous parallel mode according to a governance task scheduling strategy to obtain a task processing result, and storing the task processing result and the corresponding data governance stage in a shared storage area in an associated manner.

Description

Data governance methods, devices, equipment, storage media, and computer program products Technical Field

[0001] This application relates to the field of wireless communication technology, and in particular to a data governance method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] With the continuous growth of enterprise data scale and the increasing complexity of data application scenarios, data governance has become a key link in enhancing data value and ensuring data quality and security. Traditional data governance methods mostly rely on manual serial processing or static process orchestration, which is difficult to adapt to dynamically changing business needs and the processing requirements of massive amounts of data.

[0003] Currently, common data governance solutions mainly include the following three categories: 1. Manual serial governance solution: Each stage of this governance solution is performed manually, relying on manual coordination and document transmission, which leads to low data governance efficiency and difficulty in ensuring data consistency and traceability of the governance process.

[0004] 2. Automated data governance solutions based on static graphs: This type of governance solution uses a predefined fixed graph structure to organize processes. Although it can achieve partial automation, it lacks dynamic adjustment capabilities, has poor scalability, and is difficult to adapt to complex governance scenarios with multiple stages and dependencies.

[0005] 3. Data governance scheme based on simple task orchestration: Each governance stage of this scheme runs independently and automatically, resulting in a lack of a unified intelligent orchestration mechanism. Furthermore, task execution is mainly synchronous and serial, leading to low system resource utilization and difficulty in inter-stage coordination.

[0006] It is evident that common data governance solutions currently suffer from the following common problems: inability to dynamically adjust the governance process, low execution efficiency at each stage, difficulty in achieving parallel optimization, and reliance on manual or simple interfaces for data transfer between stages, which easily leads to data inconsistency.

[0007] Therefore, how to implement a data governance method that can support dynamic process construction and intelligent task orchestration has become a technical problem that needs to be solved by existing technologies. Summary of the Invention

[0008] This application provides a data governance method to address the problems of existing data governance solutions being unable to dynamically adjust the governance process, having low execution efficiency at each stage, and being unable to achieve parallel execution of governance tasks.

[0009] This application also provides a data governance device to solve the problems of existing data governance solutions being unable to dynamically adjust the governance process, having low execution efficiency at each stage, and being unable to achieve parallel execution of governance tasks.

[0010] This application also provides a data governance device to solve the problems of existing data governance solutions being unable to dynamically adjust the governance process, having low execution efficiency at each stage, and being unable to achieve parallel execution of governance tasks.

[0011] This application also provides a computer-readable storage medium to address the problems of existing data governance solutions being unable to dynamically adjust the governance process, having low execution efficiency at each stage, and being unable to achieve parallel execution of governance tasks.

[0012] A computer program product designed to address the problems of existing data governance solutions being unable to dynamically adjust governance processes, having low execution efficiency at each stage, and being unable to achieve parallel execution of governance tasks.

[0013] This application adopts the following technical solution: a data governance method, comprising: constructing a data governance graph structure based on acquired configuration information, wherein each processing node in the data governance graph structure represents a data governance stage in the data governance process, and the edges in the data governance graph structure represent the dependencies between the data governance stages; determining the data governance tasks corresponding to each processing node in the data governance graph structure, and determining the dependencies between the data governance tasks based on the edges in the data governance graph structure; orchestrating the data governance tasks according to the dependencies to generate a governance task scheduling strategy; executing each data governance task in an asynchronous and parallel manner according to the governance task scheduling strategy to obtain task processing results, and associating the task processing results with the corresponding data governance stages in a shared storage area, so that when executing the data governance task corresponding to the second type of data governance stage, the processing results of the preceding task corresponding to the preceding data governance stage are obtained based on the shared storage area, and the data governance task corresponding to the second type of data governance stage is executed according to the preceding task processing results.

[0014] A data governance apparatus includes: a graph structure construction unit, configured to construct a data governance graph structure based on acquired configuration information, wherein each processing node in the data governance graph structure represents a data governance stage in the data governance process, and the edges in the data governance graph structure represent dependencies between the data governance stages; a dependency determination unit, configured to determine the data governance tasks corresponding to each processing node in the data governance graph structure, and determine the dependencies between the data governance tasks based on the edges in the data governance graph structure; a task orchestration unit, configured to orchestrate the data governance tasks according to the dependencies, and generate a governance task scheduling strategy; and a data governance unit, configured to execute each data governance task in an asynchronous and parallel manner according to the governance task scheduling strategy, obtain task processing results, and associate the task processing results with the corresponding data governance stages in a shared storage area, so that when executing the data governance task corresponding to a second type of data governance stage, the processing results of the preceding task corresponding to the preceding data governance stage are obtained based on the shared storage area, and the data governance task corresponding to the second type of data governance stage is executed based on the preceding task processing results.

[0015] A data governance device includes: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the following operations: constructing a data governance graph structure based on acquired configuration information, wherein each processing node in the data governance graph structure represents a data governance stage in a data governance process, and the edges in the data governance graph structure represent dependencies between data governance stages; determining data governance tasks corresponding to each processing node in the data governance graph structure, and determining dependencies between the data governance tasks based on the edges in the data governance graph structure; orchestrating the data governance tasks according to the dependencies to generate a governance task scheduling strategy; executing each data governance task in an asynchronous and parallel manner according to the governance task scheduling strategy to obtain task processing results, and associating the task processing results with the corresponding data governance stages in a shared storage area, so that when executing a data governance task corresponding to a second type of data governance stage, the processing results of the preceding task corresponding to the preceding data governance stage are obtained based on the shared storage area, and the data governance task corresponding to the second type of data governance stage is executed based on the preceding task processing results.

[0016] A computer-readable storage medium stores one or more programs that, when executed by an electronic device including multiple applications, cause the electronic device to perform the following operations: constructing a data governance graph structure based on acquired configuration information, wherein each processing node in the data governance graph structure represents a data governance stage in a data governance process, and edges in the data governance graph structure represent dependencies between data governance stages; determining data governance tasks corresponding to each processing node in the data governance graph structure, and determining dependencies between the data governance tasks based on the edges in the data governance graph structure; orchestrating the data governance tasks according to the dependencies to generate a governance task scheduling strategy; executing each data governance task in an asynchronous and parallel manner according to the governance task scheduling strategy, obtaining task processing results, and associating the task processing results with the corresponding data governance stages in a shared storage area, so that when executing a data governance task corresponding to a second type of data governance stage, the processing results of the preceding task corresponding to the preceding data governance stage are obtained based on the shared storage area, and the data governance task corresponding to the second type of data governance stage is executed based on the preceding task processing results.

[0017] A computer program product includes a computer program that, when executed by a processor, implements the following: constructing a data governance graph structure based on acquired configuration information, wherein each processing node in the data governance graph structure represents a data governance stage in a data governance process, and the edges in the data governance graph structure represent dependencies between data governance stages; determining the data governance tasks corresponding to each processing node in the data governance graph structure, and determining the dependencies between the data governance tasks based on the edges in the data governance graph structure; orchestrating the data governance tasks according to the dependencies to generate a governance task scheduling strategy; executing each data governance task in an asynchronous and parallel manner according to the governance task scheduling strategy to obtain task processing results, and associating the task processing results with the corresponding data governance stages in a shared storage area, so that when executing the data governance task corresponding to a second type of data governance stage, the processing results of the preceding tasks corresponding to the preceding data governance stage are obtained based on the shared storage area, and the data governance task corresponding to the second type of data governance stage is executed based on the preceding task processing results.

[0018] The above-mentioned at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects: Using the data governance method provided in the embodiments of this application, a data governance graph structure can be constructed based on the obtained configuration information. Based on the constructed data governance graph structure, the data governance tasks corresponding to each processing node in the data governance graph structure are determined, and the dependencies between the data governance tasks are determined based on the edges in the data governance graph structure. Then, based on the determined dependencies, each data governance task is orchestrated to generate a governance task scheduling strategy. According to the governance task scheduling strategy, each data governance task is executed asynchronously and in parallel to obtain the task processing results. The task processing results are associated with the corresponding data governance stage and stored in a shared storage area, so that when executing the data governance task corresponding to the second type of data governance stage, the processing results of the preceding task corresponding to the preceding data governance stage are obtained based on the shared storage area, and the data governance task corresponding to the second type of data governance stage is executed based on the preceding task processing results. The data governance method provided in this application has several advantages. First, by constructing a data governance graph structure and determining task dependencies based on it, the system can automatically determine the logic between different data governance stages and tasks, replacing the existing method of manually coordinating data governance stages and greatly improving the efficiency of data governance. Second, the method provided in this application allows for asynchronous and parallel data governance based on a governance task scheduling strategy. When system resources are sufficient, data governance tasks without dependencies can be executed concurrently, significantly reducing the overall time consumption of the entire data governance process. Finally, by associating the task processing results of each governance stage with the stage identifier and persisting them to a unified shared storage area, this application establishes a centralized and reliable data results repository. When subsequent stages are executed, all required preliminary results can be directly and accurately obtained from the shared storage area, avoiding data version errors, loss, or inconsistent formats that may occur through manual transmission or scattered interface calls in traditional methods, thus ensuring the integrity and traceability of data in the data governance process. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and constitute a part of this application, illustrate exemplary embodiments of this application and, together with their descriptions, serve to explain this application and do not constitute an undue limitation thereof. In the drawings: Figure 1 is a schematic diagram of the specific structure of a data governance system provided in an embodiment of this application; Figure 2 is a schematic diagram of the specific flow of a data governance method provided in an embodiment of this application; Figure 3 is a schematic diagram of the specific structure of a data governance device provided in an embodiment of this application; Figure 4 is a schematic diagram of the specific structure of a data governance apparatus provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] This application provides a data governance method to address the problems of existing data governance solutions being unable to dynamically adjust the governance process, having low execution efficiency at each stage, and being unable to achieve parallel execution of governance tasks.

[0022] The execution subject of the data governance method provided in this application embodiment may be, but is not limited to, at least one of a data management server, a data governance server, and a task management server; in addition, the execution subject of the method may also be the system or application (APP) itself running on these servers.

[0023] For ease of description, the following description uses a data governance system as the execution subject to illustrate the implementation of this method. It should be understood that using a data governance system as the execution subject is merely an illustrative example and should not be construed as a limitation of the method.

[0024] In one implementation, the specific system architecture of the data governance system is shown in Figure 1, which mainly consists of a supervision layer, a processing layer, a service layer, a user interaction layer, and a data storage layer.

[0025] The Supervisor Layer is primarily responsible for process orchestration, intelligent routing, and state coordination. Its core functions include: 1. State inspection and reasoning: monitoring the current system state and analyzing user intent; 2. Intelligent routing decision: selecting the next processing stage based on the context; 3. Process orchestration: coordinating the execution order of each processing agent; 4. State synchronization: maintaining global state consistency.

[0026] Processing Layer (Agent Layer): Contains multiple business processing agents, each agent is used to handle a specific business stage, realizing professional division of labor, such as demand research agent, data inventory agent, standard design agent, etc.

[0027] Service Layer: Provides shared service modules, such as task management, status management, and file processing.

[0028] User interaction layer: Provides users with the interface and API for front-end operation and interaction.

[0029] Based on the above data governance system, the specific implementation flow diagram of the data governance method provided in this application is shown in Figure 2, which mainly includes the following steps: Step 11, construct a data governance graph structure according to the obtained configuration information; wherein, each processing node in the constructed data governance graph structure is used to represent each data governance stage in the data governance process, and the edges in the data governance graph structure are used to represent the dependency relationship between each data governance stage.

[0030] In this embodiment, the data governance system can obtain basic information of each data governance stage of the data governance process from a configuration source (e.g., a specific configuration table, configuration file, or configuration center of a relational database), and obtain the configuration information of each data governance stage by parsing the JSON configuration in the basic information (such as the phase_info field).

[0031] In this application embodiment, the configuration information obtained by the data governance system may include, but is not limited to, the following: 1. Data governance stage identifier: Specifically, the data governance stage identifier can uniquely identify a governance stage in the data governance process.

[0032] 2. Data Governance Stage Dependency Information: This section indicates the preceding data governance stages that this data governance stage depends on. For example, the standard design stage may depend on two preceding data governance stages: requirements gathering and data inventory.

[0033] Type 3, conditional dependencies determined based on runtime conditional assessment results; for example, the conditional dependency could be: if the data quality assessment finds a problem, then perform a data cleaning task; or if the data volume exceeds a threshold, then perform a data preprocessing task first.

[0034] Type 4, resource dependency relationship, which is used to indicate the system resource requirements when performing the data governance task. 3. Mapping information corresponding to the data governance stage: In this embodiment of the application, the mapping information can be used to indicate which specific intelligent agent or processing class will perform the data governance stage. In one implementation, the mapping information is usually a class name or service identifier. The data governance system can determine the intelligent agent class name and module path corresponding to the data governance stage through the mapping information.

[0035] It should be noted that, in this embodiment of the application, the data governance system can adopt a multi-level caching strategy to obtain configuration information. Specifically, the data governance system can first check whether the configuration information corresponding to the data governance stage to be obtained exists in the memory cache. When the configuration information is not found in the memory cache, it will then obtain the configuration information through a database query, thereby avoiding repeated database queries.

[0036] After obtaining the configuration information, the data governance system can create the data governance graph structure according to the following sub-steps, which may include: Sub-step 1101, initializing each processing node in the original graph structure according to the data governance stage identifier; specifically, in this embodiment of the application, the data governance system can initialize an empty graph structure using a StateGraph object according to the obtained configuration information.

[0037] Next, the data governance system can add nodes to the empty graph structure. Specifically, the data governance system can create a corresponding processing node in the empty graph structure according to each data governance stage identifier defined in the obtained configuration information, and bind each processing node to the specific business processing logic or intelligent agent pointed to by the mapping information in the above configuration information. For example, create a processing node for the data inventory stage and bind the processing logic of the DataInventoryAgent class.

[0038] Sub-step 1102: Add a supervisory node; In this embodiment of the application, the data governance system can also add a supervisory node to the empty graph structure. In one implementation, the supervisory node does not perform specific business governance tasks, but serves as the scheduling center of the entire data governance process. It is used to monitor the global status, understand user or system intent, and determine the next execution node based on multi-dimensional information according to preset rules or intelligent reasoning. It can also identify abnormal system states, trigger corresponding recovery mechanisms, and perform system status synchronization updates to ensure data synchronization between processing nodes.

[0039] Sub-step 1103: Based on the data governance stage dependency information, generate edges between the processing nodes in the original graph structure.

[0040] Specifically, the data governance system can establish directed edges between processing nodes in the graph structure based on the data governance stage dependency information obtained from the configuration information. These directed edges define the flow direction of each data governance task in the data governance process.

[0041] Furthermore, it should be noted that the edges generated between processing nodes in this embodiment are not simple sequential connections, but conditional edges. For example, the edge from the monitoring node to the data inventory stage node is set with the trigger condition "when the user triggers the start data inventory command and the status of the requirements survey stage is completed". Through this design of conditional edges, the execution path of the process is no longer static, but can change dynamically according to the runtime status.

[0042] Sub-step 1104, Compilation and Persistence: After adding processing nodes and edges through sub-steps 1101-1103, the data governance system can call the compilation method of the graph computing framework to compile the above logical graph structure into an executable data governance graph structure. During the compilation process, structural verification is performed to check for unreachable nodes. Simultaneously, the data governance system can configure state persistence hooks for this graph structure so that after any node is executed, the current execution context can be saved to the shared memory area, thereby ensuring that even if the process is interrupted, related tasks can continue to be executed based on the data stored in the shared memory area.

[0043] Step 12: Determine the data governance tasks corresponding to each processing node in the data governance graph structure, and determine the dependencies between each data governance task based on the edges in the data governance graph structure; In this embodiment, the data governance system can determine the data governance tasks corresponding to each data governance stage based on the data governance graph structure.

[0044] It should be noted that, since each processing node (i.e., the data governance stage) is not a single operation but rather consists of a series of finer-grained data governance tasks, in this embodiment, the data governance system can determine the specific list of tasks to be executed in each stage by parsing the agent configuration or task definition file corresponding to that stage. For example, the data access stage may include a series of tasks such as access scheme design, access script generation, access task execution, and access result verification.

[0045] Next, the data governance system can determine the dependencies between various data governance tasks. It should be noted that since the dependencies between data governance tasks are more complex than those between data governance stages, in one implementation, the data governance system can determine these dependencies using the following methods: Method 1, based on the stage graph structure: Since data governance tasks belong to their corresponding data governance stages, the dependencies between stages—that is, the edges in the data governance graph structure—are propagated to the data governance tasks they contain. For example, if stage B depends on stage A, then all tasks in stage B depend on the completion of stage A. Therefore, in this embodiment, the dependencies between data governance tasks can be determined based on the dependencies between stages in the data governance graph structure.

[0046] Method 2, based on the task's own dependency declaration: It should be noted that each data governance task declares its specific dependencies in its definition. These dependencies can be divided into four types, which can include: Type 1, intra-stage dependencies between data governance tasks within the same data governance stage. This dependency is used to indicate the execution order of data governance tasks within the same data governance stage.

[0047] Type 2: Inter-stage dependencies between different data governance stages. This data governance stage dependency information indicates the preceding data governance stages that the current data governance stage depends on. For example, the standard design stage may depend on two preceding data governance stages: requirements gathering and data inventory.

[0048] Type 3, conditional dependencies determined based on runtime conditional assessment results; for example, the conditional dependency could be: if the data quality assessment finds a problem, then perform a data cleaning task; or if the data volume exceeds a threshold, then perform a data preprocessing task first.

[0049] Type 4, resource dependency, which represents the system resource requirements when performing the data governance task.

[0050] Step 13: Based on the dependencies determined by executing Step 12, orchestrate the data governance tasks and generate a governance task scheduling strategy.

[0051] In this embodiment of the application, the data governance system can generate a governance task scheduling strategy according to the following sub-steps, including: sub-step 1301, generating a task dependency directed graph based on the determined dependency relationship; specifically, the data governance system can model a directed acyclic graph (DAG) based on all data governance tasks determined by executing step 12 and the dependency relationship between each data governance task. In this directed acyclic graph, vertices represent data governance tasks, and directed edges represent the dependency relationship between data governance tasks. For example, a directed edge between node A and node B indicates that data governance task B depends on data governance task A.

[0052] Sub-step 1302: Determine the task execution order based on the task dependency directed graph determined by executing sub-step 1301; specifically, the data governance system can perform topological sorting on the task dependency DAG to obtain a linear task execution sequence.

[0053] Sub-step 1303: Determine the set of tasks to be executed in parallel and generate a scheduling strategy.

[0054] By executing sub-step 1302 above, the resulting topological sorting naturally divides the data governance tasks into multiple levels. Tasks at the same level have no dependencies and can therefore be executed in parallel. The data governance system can then determine the set of tasks to be executed in parallel based on this result, and generate the following types of final governance task scheduling strategies based on this set: Type a, Task Concurrency Control Strategy: This governance task scheduling strategy explicitly specifies the maximum parallelism of each task batch. For example, the access script generation batch allows a maximum of 10 tasks to run in parallel.

[0055] Type b, resource constraint integration strategy: This governance task scheduling strategy can refer to the resource dependency information of the task and monitor the global resource usage in real time through the resource manager to ensure that the total amount of resources occupied by concurrently executed tasks does not exceed the preset threshold.

[0056] Type c, priority scheduling strategy: This governance task scheduling strategy can assign higher priority to high-priority tasks and prioritize scheduling when there is resource contention.

[0057] Step 14: Based on the governance task scheduling strategy obtained by executing Step 13, execute each data governance task in an asynchronous and parallel manner to obtain the task processing results.

[0058] In this embodiment of the application, the data governance system can submit the governance task scheduling strategy obtained by executing step 13 above to the asynchronous execution queue, and perform asynchronous processing in a non-blocking execution mode to ensure that the main process is not blocked by long-running tasks.

[0059] Specifically, in this embodiment of the application, the data governance system can perform asynchronous data governance tasks according to the following sub-steps, including: Sub-step 1401, dependency check; before performing asynchronous execution, the data governance system can check in real time whether the dependencies of all data governance tasks to be executed are satisfied. If it is determined that the dependencies are not satisfied, the corresponding data governance task is added to the waiting queue and will not be executed this time.

[0060] Sub-step 1402, resource allocation: The data governance system determines the execution resources (e.g., database connection, memory, CPU, etc.) required by each data governance task and allocates execution resources to these data governance tasks.

[0061] Sub-step 1403: Execute the task asynchronously in the background and update the task execution status in real time through the event mechanism. After the task is completed, trigger the subsequent task. Sub-step 1404: After the task is completed, update the stage status with the processing result and release the execution resources used during the task execution.

[0062] Specifically, during task execution, the data governance system can perform asynchronous parallel orchestration in the following ways: 1. For tasks at different stages: Under the premise of satisfying stage dependencies, independent tasks in different stages can be orchestrated for parallel execution.

[0063] 2. Tasks in the same stage: Under the premise of satisfying task dependencies, identify task groups that can be executed in parallel through topological sorting.

[0064] 3. Resource limit control: When the number of parallel tasks reaches the resource limit, subsequent tasks enter the waiting queue and are automatically woken up after the resources are released.

[0065] 4. Priority scheduling: High-priority tasks are executed first, and task preemption is supported.

[0066] It's important to note that after each task is completed, the output results are structured and stored in the shared storage area. In this embodiment, the data governance system can store these results using the AgentState data structure, which includes a dictionary field called `process_outputs`. This dictionary field uses the "stage identifier" as the key and the aggregated output of all tasks in that stage as the value. For example, after executing tasks in the data inventory stage, the results of that stage, such as the data asset list and quality report, will be integrated and stored in the corresponding `process_outputs` dictionary field. The entire AgentState is persisted to the shared storage area, thus achieving centralized and versioned storage of governance results. Subsequently, the data governance system can access and collaboratively execute data across stages based on the data stored in the shared storage area.

[0067] Specifically, in this application embodiment, when it is necessary to execute the second type of data governance stage, that is, the data governance task of the stage that depends on the output of other stages, the data governance system does not need complex interface calls or file transfers. Its specific execution flow is shown in the following sub-steps: Sub-step 14a, obtain the pre-results; the data governance system loads the current AgentState from the shared storage area.

[0068] In sub-step 14b, the data governance system directly reads the task processing results of the preceding data governance stage that it depends on through the AgentState.process_outputs dictionary.

[0069] Sub-step 14c involves inputting the obtained results of the preceding task processing into the corresponding agent for task execution.

[0070] The above steps enable automatic and accurate data flow between stages, achieving multi-stage intelligent collaboration. Furthermore, based on persistent AgentState, the entire governance process supports interruption and recovery at any time, demonstrating robust fault tolerance.

[0071] The data governance method provided in this application can construct a data governance graph structure based on the obtained configuration information. Based on the constructed data governance graph structure, the data governance tasks corresponding to each processing node in the data governance graph structure are determined, and the dependencies between the data governance tasks are determined based on the edges in the data governance graph structure. Then, based on the determined dependencies, the data governance tasks are orchestrated to generate a governance task scheduling strategy. According to the governance task scheduling strategy, each data governance task is executed asynchronously and in parallel to obtain the task processing results. The task processing results are associated with the corresponding data governance stages and stored in a shared storage area. This allows the execution of the data governance tasks corresponding to the second type of data governance stage when the data governance tasks corresponding to the previous data governance stage are executed, based on the shared storage area, obtaining the processing results of the previous data governance stages, and then executing the data governance tasks corresponding to the second type of data governance stage according to the processing results of the previous data governance stages. The data governance method provided in this application has several advantages. First, by constructing a data governance graph structure and determining task dependencies based on it, the system can automatically determine the logic between different data governance stages and tasks, replacing the existing method of manually coordinating data governance stages and greatly improving the efficiency of data governance. Second, the method provided in this application allows for asynchronous and parallel data governance based on a governance task scheduling strategy. When system resources are sufficient, data governance tasks without dependencies can be executed concurrently, significantly reducing the overall time consumption of the entire data governance process. Finally, by associating the task processing results of each governance stage with the stage identifier and persisting them to a unified shared storage area, this application establishes a centralized and reliable data results repository. When subsequent stages are executed, all required preliminary results can be directly and accurately obtained from the shared storage area, avoiding data version errors, loss, or inconsistent formats that may occur through manual transmission or scattered interface calls in traditional methods, thus ensuring the integrity and traceability of data in the data governance process.

[0072] In one embodiment, this application also provides a data governance device to address the problems of existing data governance solutions being unable to dynamically adjust the governance process, having low execution efficiency at each stage, and being unable to achieve parallel execution of governance tasks. A schematic diagram of the specific structure of this data governance device is shown in Figure 3, including: a graph structure construction unit 31, a dependency determination unit 32, a task orchestration unit 33, and a data governance unit 34.

[0073] The system includes a graph structure construction unit 31, which constructs a data governance graph structure based on the acquired configuration information. Each processing node in the graph structure represents a data governance stage in the data governance process, and the edges in the graph structure represent dependencies between these stages. A dependency determination unit 32 determines the data governance tasks corresponding to each processing node in the graph structure and determines the dependencies between these tasks based on the edges. A task orchestration unit 33 orchestrates the data governance tasks according to the dependencies, generating a governance task scheduling strategy. A data governance unit 34 executes each data governance task asynchronously and in parallel according to the governance task scheduling strategy, obtains task processing results, and associates these results with the corresponding data governance stages in a shared storage area. This allows the execution of data governance tasks corresponding to the second type of data governance stage when the shared storage area is used to retrieve the processing results of the preceding data governance stages and execute the data governance tasks corresponding to the second type of data governance stage based on these results.

[0074] In one implementation, the configuration information includes a data governance stage identifier, data governance stage dependency information, and mapping information corresponding to the data governance stage. The graph structure construction unit 31 is specifically used for: initializing each processing node in the original graph structure according to the data governance stage identifier, and adding a supervisor node, wherein the supervisor node is used to make routing decisions based on the task processing results corresponding to the current data governance stage; generating edges between each processing node in the original graph structure according to the data governance stage dependency information to obtain the data governance graph structure. In one implementation, the supervisor node is specifically used for: obtaining the task processing results of the data governance task corresponding to the current data governance stage; and selecting the processing node corresponding to the data governance stage to be executed based on the task processing results.

[0075] In one implementation, the dependencies specifically include: intra-stage dependencies between data governance tasks within the same data governance phase; and / or inter-stage dependencies between different data governance phases; and / or conditional dependencies determined based on runtime condition evaluation results; and / or resource dependencies, wherein the resource dependencies are used to represent the system resource requirements when performing the data governance tasks.

[0076] In one implementation, the task orchestration unit 33 is specifically configured to: generate a task dependency directed graph based on the dependency relationship; determine the task execution order based on the task dependency directed graph; determine a set of tasks to be executed in parallel based on the task execution order; and generate a governance task scheduling strategy that includes a task concurrency control strategy based on the set of tasks to be executed in parallel.

[0077] In one implementation, the data governance unit 34 is specifically configured to: acquire current system resource usage data, wherein the resource usage data includes at least one of database connection count, memory usage rate, or processor utilization rate; set the number of concurrently executed tasks according to the governance task scheduling strategy and the resource usage data; and asynchronously execute each data governance task in parallel according to the number of tasks.

[0078] The data governance apparatus provided in this application embodiment can construct a data governance graph structure based on the acquired configuration information. Based on the constructed data governance graph structure, it determines the data governance tasks corresponding to each processing node in the data governance graph structure and determines the dependencies between the data governance tasks based on the edges in the data governance graph structure. Then, based on the determined dependencies, it orchestrates each data governance task, generates a governance task scheduling strategy, and executes each data governance task in an asynchronous and parallel manner according to the governance task scheduling strategy to obtain task processing results. The task processing results are then associated with the corresponding data governance stage and stored in a shared storage area. This allows the apparatus to retrieve the processing results of the preceding tasks corresponding to the preceding data governance stage from the shared storage area when executing the data governance task corresponding to the second type of data governance stage, and to execute the data governance task corresponding to the second type of data governance stage based on the preceding task processing results. The data governance method provided in this application has several advantages. First, by constructing a data governance graph structure and determining task dependencies based on it, the system can automatically determine the logic between different data governance stages and tasks, replacing the existing method of manually coordinating data governance stages and greatly improving the efficiency of data governance. Second, the method provided in this application allows for asynchronous and parallel data governance based on a governance task scheduling strategy. When system resources are sufficient, data governance tasks without dependencies can be executed concurrently, significantly reducing the overall time consumption of the entire data governance process. Finally, by associating the task processing results of each governance stage with the stage identifier and persisting them to a unified shared storage area, this application establishes a centralized and reliable data results repository. When subsequent stages are executed, all required preliminary results can be directly and accurately obtained from the shared storage area, avoiding data version errors, loss, or inconsistent formats that may occur through manual transmission or scattered interface calls in traditional methods, thus ensuring the integrity and traceability of data in the data governance process.

[0079] Figure 4 is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Referring to Figure 4, at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include RAM, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk storage device. Of course, the electronic device may also include other hardware required for other services.

[0080] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in Figure 4, but this does not imply that there is only one bus or one type of bus.

[0081] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0082] The processor reads the corresponding computer program from non-volatile memory into memory and runs it, forming a data governance device at the logical level. The processor executes the program stored in memory and specifically performs the following operations: Based on the acquired configuration information, it constructs a data governance graph structure, where each processing node in the data governance graph structure represents a data governance stage in the data governance process, and the edges in the data governance graph structure represent the dependencies between the data governance stages; it determines the data governance tasks corresponding to each processing node in the data governance graph structure, and determines the dependencies between the data governance tasks based on the edges in the data governance graph structure; it orchestrates the data governance tasks according to the dependencies, generating a governance task scheduling strategy; it executes each data governance task in an asynchronous and parallel manner according to the governance task scheduling strategy, obtains the task processing results, and associates the task processing results with the corresponding data governance stages in a shared storage area, so that when executing the data governance task corresponding to the second type of data governance stage, it obtains the processing results of the preceding tasks corresponding to the preceding data governance stage based on the shared storage area, and executes the data governance task corresponding to the second type of data governance stage based on the processing results of the preceding tasks.

[0083] The data governance electronic device method disclosed in the embodiment shown in Figure 4 of this application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0084] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0085] This application also proposes a computer-readable storage medium that stores one or more programs, each program including instructions. When executed by a portable electronic device including multiple applications, these instructions enable the portable electronic device to perform the data governance method of the embodiment shown in FIG1, specifically for performing the following operations: constructing a data governance graph structure based on acquired configuration information, wherein each processing node in the data governance graph structure represents a data governance stage in the data governance process, and the edges in the data governance graph structure represent dependencies between data governance stages; and determining the data governance task corresponding to each processing node in the data governance graph structure. Based on the edges in the data governance graph structure, the dependencies between the data governance tasks are determined; based on the dependencies, the data governance tasks are orchestrated to generate a governance task scheduling strategy; based on the governance task scheduling strategy, each data governance task is executed asynchronously and in parallel to obtain the task processing results, and the task processing results are associated with the corresponding data governance stages and stored in a shared storage area, so that when executing the data governance tasks corresponding to the second type of data governance stage, the processing results of the preceding tasks corresponding to the preceding data governance stage are obtained based on the shared storage area, and the data governance tasks corresponding to the second type of data governance stage are executed based on the processing results of the preceding tasks.

[0086] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0090] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0091] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0092] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0093] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0094] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data governance method, characterized in that, include: Based on the obtained configuration information, a data governance graph structure is constructed, wherein each processing node in the data governance graph structure represents a data governance stage in the data governance process, and the edges in the data governance graph structure represent the dependencies between the data governance stages; the data governance tasks corresponding to each processing node in the data governance graph structure are determined, and the dependencies between the data governance tasks are determined based on the edges in the data governance graph structure; based on the dependencies, the data governance tasks are orchestrated to generate a governance task scheduling strategy; According to the governance task scheduling strategy, each data governance task is executed asynchronously and in parallel to obtain the task processing results. The task processing results are associated with the corresponding data governance stages and stored in the shared storage area. This allows the data governance task corresponding to the second type of data governance stage to be executed when the data governance task is executed, based on the shared storage area, to obtain the processing results of the preceding data governance stage, and to execute the data governance task corresponding to the second type of data governance stage based on the processing results of the preceding data governance stage.

2. The method according to claim 1, characterized in that, The configuration information includes a data governance stage identifier, data governance stage dependency information, and mapping information corresponding to the data governance stage. The step of constructing a data governance graph structure based on the obtained configuration information specifically includes: initializing each processing node in the original graph structure according to the data governance stage identifier, and adding a supervisor node, wherein the supervisor node is used to make routing decisions based on the task processing results corresponding to the current data governance stage; and generating edges between each processing node in the original graph structure according to the data governance stage dependency information to obtain the data governance graph structure.

3. The method according to claim 2, characterized in that, The supervisory node makes routing decisions, specifically including: obtaining the task processing results of the data governance task corresponding to the current data governance stage; and selecting the processing node corresponding to the data governance stage to be executed based on the task processing results.

4. The method according to claim 1, characterized in that, The dependencies specifically include: intra-stage dependencies between data governance tasks within the same data governance phase; and / or inter-stage dependencies between different data governance phases; and / or conditional dependencies determined based on runtime condition evaluation results; and / or resource dependencies, wherein the resource dependencies are used to represent the system resource requirements when executing the data governance task.

5. The method according to claim 4, characterized in that, The step of orchestrating the data governance tasks according to the dependencies and generating a governance task scheduling strategy specifically includes: generating a task dependency directed graph based on the dependencies; determining the task execution order based on the task dependency directed graph; determining a set of tasks to be executed in parallel based on the task execution order; and generating a governance task scheduling strategy that includes a task concurrency control strategy based on the set of tasks to be executed in parallel.

6. The method according to claim 1, characterized in that, The method of executing each data governance task in an asynchronous and parallel manner includes: obtaining the current system resource usage data, wherein the resource usage data includes at least one of database connection count, memory usage rate, or processor utilization rate; setting the number of tasks to be executed concurrently according to the governance task scheduling strategy and the resource usage data; and executing each data governance task asynchronously and in parallel according to the number of tasks.

7. A data governance device, characterized in that, include: A graph structure construction unit is used to construct a data governance graph structure based on the acquired configuration information. Each processing node in the data governance graph structure represents a data governance stage in the data governance process, and the edges in the data governance graph structure represent the dependencies between the data governance stages. A dependency determination unit is used to determine the data governance tasks corresponding to each processing node in the data governance graph structure and to determine the dependencies between the data governance tasks based on the edges in the data governance graph structure. A task orchestration unit is used to orchestrate the data governance tasks according to the dependencies and generate a governance task scheduling strategy. A data governance unit is used to execute each data governance task in an asynchronous and parallel manner according to the governance task scheduling strategy, obtain the task processing results, and associate the task processing results with the corresponding data governance stages in a shared storage area. This allows the execution of the data governance task corresponding to the second type of data governance stage when the data governance task is executed, by obtaining the processing results of the preceding task corresponding to the preceding data governance stage from the shared storage area and executing the data governance task corresponding to the second type of data governance stage based on the preceding task processing results.

8. A data governance device, comprising: processor; And a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following operations: constructing a data governance graph structure based on acquired configuration information, wherein each processing node in the data governance graph structure represents a data governance stage in the data governance process, and the edges in the data governance graph structure represent dependencies between the data governance stages; determining the data governance tasks corresponding to each processing node in the data governance graph structure, and determining the dependencies between the data governance tasks based on the edges in the data governance graph structure; and orchestrating the data governance tasks according to the dependencies to generate a governance task scheduling strategy. According to the governance task scheduling strategy, each data governance task is executed asynchronously and in parallel to obtain the task processing results. The task processing results are associated with the corresponding data governance stages and stored in the shared storage area. This allows the data governance task corresponding to the second type of data governance stage to be executed when the data governance task is executed, based on the shared storage area, to obtain the processing results of the preceding data governance stage, and to execute the data governance task corresponding to the second type of data governance stage based on the processing results of the preceding data governance stage.

9. A computer-readable storage medium storing one or more programs that, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the data governance method as described in any one of claims 1-6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data governance method as described in any one of claims 1-6.

Images