Data processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention relates to a data processing method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a knowledge graph constructed for a data warehouse; on the basis of a preset analysis target and a preset path requirement, path processing is executed on the knowledge graph, a target path corresponding to the data warehouse is determined, and the target path is associated with the preset analysis target and meets the preset path requirement; updating a data processing flow associated with the data warehouse based on the target path, wherein the data processing flow is configured to execute data processing on a data table stored in the data warehouse; and in response to the detected updating of the data table, executing data processing on the updated data table according to the updated data processing flow. Therefore, the data processing efficiency for the data warehouse can be improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to a method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product for data processing. Background Art

[0002] With the rapid development of Internet technology, more and more platforms or applications use data warehouses for data management and data analysis. However, with the continuous development of platforms or applications, the data and tasks in the data warehouse are increasing, and the data links in the data warehouse become redundant, intertwined, and long, which seriously affects the data processing efficiency and data processing cost of the data warehouse. Summary of the invention

[0003] In the first aspect of the present disclosure, a method for data processing is provided. The method includes: obtaining a knowledge graph constructed for a data warehouse, wherein the knowledge graph includes a plurality of nodes and directed edges connecting the nodes, each node represents a data table stored in the data warehouse or an indicator obtained after data processing is performed on the data table, and each directed edge connecting the nodes represents the data flow relationship between the data tables or indicators corresponding to the connected nodes; performing path processing on the knowledge graph based on a predetermined analysis target and a preset path requirement, determining a target path corresponding to the data warehouse, wherein the target path is associated with a preset analysis target and meets the preset path requirement; updating a data processing flow associated with the data warehouse based on the target path, wherein the data processing flow is configured to perform data processing on a data table stored in the data warehouse; and in response to detecting an update to the data table, performing data processing on the updated data table according to the updated data processing flow.

[0004] In the second aspect of the present disclosure, a device for data processing is provided. The device includes: a knowledge graph acquisition module, configured to obtain a knowledge graph constructed for a data warehouse, wherein the knowledge graph contains multiple nodes and directed edges connecting the nodes, each node represents a data table stored in the data warehouse or an indicator obtained after data processing is performed on the data table, and each directed edge connecting the nodes represents the data flow relationship between the data tables or indicators corresponding to the connected nodes; a target path determination module, configured to perform path processing on the knowledge graph based on a predetermined analysis target and a preset path requirement, and determine the target path corresponding to the data warehouse, wherein the target path is associated with a preset analysis target and meets the preset path requirement; a data processing flow update module, configured to update a data processing flow associated with the data warehouse based on the target path, wherein the data processing flow is configured to perform data processing on a data table stored in the data warehouse; and a data processing module, configured to perform data processing on the updated data table according to the updated data processing flow in response to detecting an update to the data table.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which includes a computer program, wherein when the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A flowchart showing a process for data processing according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram of a metadata knowledge graph according to some embodiments of the present disclosure is shown;

[0013] Figure 4A A schematic diagram showing a graph to be traversed according to some embodiments of the present disclosure;

[0014] Figure 4B A schematic diagram showing a process of graph traversal according to some embodiments of the present disclosure;

[0015] Figure 5 A schematic diagram showing path length calculation according to some embodiments of the present disclosure is shown;

[0016] Figure 6 A flowchart showing a process of path optimization according to some embodiments of the present disclosure is shown;

[0017] Figure 7 Schematic diagram of simulation experiment results according to some embodiments of the present disclosure;

[0018] Figure 8 A schematic structural block diagram showing an apparatus for data processing according to some embodiments of the present disclosure; and

[0019] Fig. 9 A block diagram of a computing device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0022] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0023] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0024] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0025] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation to the embodiments of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the embodiments of the present disclosure.

[0027] As used in this article, the term "data warehouse" can be understood as a data management system. A "data warehouse" is a data management system that can realize functions such as data storage, data management, data analysis, and data query. A "data warehouse" can usually store data from multiple data sources. Based on the data management, data analysis, data query and other functions of the "data warehouse", users can achieve more efficient data decision-making, process improvement, etc.

[0028] "Data link" can be understood as the data flow link in the data warehouse. The data link can include the entire process of generating another data unit from one data unit, such as data processing tasks, intermediate data units, etc.

[0029] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, the example environment 100 may include an electronic device 110 . The electronic device 110 may be configured to perform data processing on data in a data repository 120 .

[0030] In the example environment 100, the electronic device 110 can process the data obtained from the data warehouse 120 and store the processing results in the data warehouse. The electronic device 110 can also use the processing results for other platforms or applications, and can also send the processing results to other electronic devices, etc. The data warehouse 120 can be deployed in the electronic device 110 or in other electronic devices. The data warehouse 120 can store data such as data tables, data tasks, indicators, etc., and can also store data attribute information or other related data information, etc., which will not be described in detail here.

[0031] The electronic device 110 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, server devices, etc. The terminal device may be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a handheld computer, a portable game terminal, a VR / AR device, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a game device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof.

[0032] The server-side device can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. Server-side devices can include computing systems / servers such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0033] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0034] As briefly mentioned above, a platform or application can utilize a data warehouse (e.g. Figure 1 The data warehouse 120 in the data warehouse is used for data management, data analysis, etc. For example, the logistics operation platform can use the logistics operation data warehouse as an important infrastructure for storing and managing data, and carry the flow data in the logistics operation process. Due to the development of enterprise project operations, the growth of data tables, data tasks, etc. in the data warehouse is rapid, for example, showing an exponential growth trend. The data links in the data warehouse become redundant, intertwined, and the links are too long, which not only brings great difficulties to the use of data, but also increases the data calculation cost and data storage cost of logistics operations.

[0035] To this end, the platform or application can build a golden link for the data warehouse. The golden link can be understood as an authoritative and accurate data processing process record created through data quality management and other methods during the multi-source data integration process of the data warehouse. For example, the golden link can include data processing processes such as data matching, data fusion, data cleaning, and data deduplication.

[0036] Specifically, the golden link has the following advantages:

[0037] (1) Simplify the data model: By creating authoritative data records and reducing duplicate and redundant data in the data warehouse, the data model can be simplified, making it easier to develop and use the data in the data warehouse;

[0038] (2) Improve data quality: Golden Link emphasizes data cleaning, fusion, and deduplication. Therefore, Golden Link can improve the accuracy and completeness of data and reduce the probability of errors and inconsistencies.

[0039] (3) Reduce storage costs: Golden links can reduce duplicate data, effectively reducing the space requirements for data storage, and thus reducing data storage costs;

[0040] (4) Optimize query performance: Golden links have clear and accurate data structures, which can speed up queries, improve data warehouse performance, and reduce computing costs;

[0041] (5) Support better decision-making: Golden links enable data warehouses to provide more reliable data support, thereby helping platforms or applications perform more accurate business analysis and decision-making.

[0042] Based on this, by building the golden link of the data warehouse, the platform or application can improve the efficiency and quality of data processing, and the golden link can more deeply explore the value of data and provide more accurate and powerful decision support. With the development of technologies such as automated data modeling, data lakes, real-time data processing, and cloud computing, the design and management of data warehouses have become more flexible and efficient, and can better adapt to the rapidly changing business environment and data needs.

[0043] In some embodiments, the platform or application can find data links in the data warehouse through manual analysis, and build the golden link of the data warehouse accordingly. However, building the golden link based on manual analysis faces many disadvantages, such as the following:

[0044] (1) Manual processing is cumbersome: It relies on manual definition of data models and relationships, which is not only time-consuming but also prone to errors;

[0045] (2) Difficulty in relationship discovery: When the amount of data is large and the data sources are diverse, it is very difficult to manually discover and optimize the relationships between data in the data warehouse;

[0046] (3) Poor scalability: As data increases, the existing golden links may be difficult to effectively and quickly expand to meet the same type of data processing needs;

[0047] (4) High maintenance cost: Manually maintaining data relationships and golden links requires a lot of time and expertise, resulting in high maintenance costs;

[0048] (5) Poor data real-time performance: Manually updating the golden link is slow, which may lead to insufficient data real-time performance and affect the timeliness of subsequent decisions that rely on data from the data warehouse.

[0049] In view of this, an embodiment of the present disclosure provides an improved scheme for data processing. In this scheme, a knowledge graph constructed for a data warehouse is obtained, wherein the knowledge graph includes multiple nodes and directed edges connecting the nodes, each node represents a data table stored in the data warehouse or an indicator obtained after data processing is performed on the data table, and each directed edge connecting the nodes represents the data flow relationship between the data tables or indicators corresponding to the connected nodes; based on a predetermined analysis target and a preset path requirement, the knowledge graph is subjected to path processing to determine the target path corresponding to the data warehouse, wherein the target path is associated with the preset analysis target and meets the preset path requirement; based on the target path, a data processing flow associated with the data warehouse is updated, wherein the data processing flow is configured to perform data processing on a data table stored in the data warehouse; and in response to detecting an update to the data table, data processing is performed on the updated data table according to the updated data processing flow.

[0050] In this way, data links in the data warehouse can be automatically found and the golden links of the data warehouse can be constructed, which can improve the efficiency of analyzing the relationship between the data in the data warehouse, increase the update speed of data in the data warehouse, and reduce the operation and maintenance costs of the data warehouse. In addition, the target path can be automatically determined according to the predetermined analysis target, and then the golden link corresponding to the predetermined analysis target can be determined and the data processing flow can be automatically updated, which can significantly improve the scalability and expansion speed of the data warehouse.

[0051] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0052] Figure 2 1 shows a flowchart of a process 200 for data processing according to some embodiments of the present disclosure. The process 200 may be implemented at the electronic device 110. For ease of discussion, reference will be made to Figure 1 The process 200 is described with reference to the environment 100 of FIG.

[0053] At box 210, a knowledge graph constructed for the data warehouse is obtained, wherein the knowledge graph includes multiple nodes and directed edges connecting the nodes, each node represents a data table stored in the data warehouse or an indicator obtained after data processing is performed on the data table, and each directed edge connecting the nodes represents the data flow relationship between the data tables or indicators corresponding to the connected nodes.

[0054] As some examples, data warehouses such as Figure 1 In order to improve the processing efficiency of data in the data warehouse 120, a knowledge graph constructed for the data warehouse can be obtained to facilitate the subsequent use of the knowledge graph to quickly process data.

[0055] Among them, the knowledge graph is also called the metadata knowledge graph, which can contain multiple nodes and directed edges connecting multiple nodes. Any of the multiple nodes can represent a part of the data in the knowledge graph, such as a data table, an indicator, etc. The data table can be one data table, multiple data tables, etc. The indicator can be one indicator, an indicator table, multiple indicators, an indicator table where multiple indicators are located, etc. Two nodes with a data flow relationship among the multiple nodes can be connected by a directed edge. The directed edge between the two nodes represents the data flow relationship between the data corresponding to the two nodes (such as a data table or an indicator, etc.).

[0056] As some examples, data tables and indicators can be used as nodes of the knowledge graph, the properties of the data table itself can be used as attributes of the table node, the properties of the indicator itself can be used as attributes of the indicator node, the relationship between data tables / the relationship between data tables and indicators can be used as directed edges (from the upstream data table to the downstream data table / downstream indicator), and the running time and running time of the tasks involved in the table can be used as attributes of the directed edges.

[0057] As some examples, knowledge graphs such as Figure 3 As shown, Figure 3 A schematic diagram of a metadata knowledge graph according to some embodiments of the present disclosure is shown. Figure 3 Including node 310-1, node 310-2, node 310-3, node 310-4, node 310-5, etc., which can also be collectively or individually referred to as node 310. Two nodes are connected by a directed edge, which represents the data flow relationship between the two nodes. For example, the directed edge from node 310-1 to node 310-2 represents the data flow relationship from node 310-1 to node 310-2, and the directed edge from node 310-3 to node 310-4 represents the data flow relationship from node 310-3 to node 310-4.

[0058] Back to Figure 2 In some embodiments, the knowledge graph can be constructed using the metadata information of the data warehouse, that is, the metadata information in the data warehouse is determined, and multiple data tables and multiple indicators to be used to construct the knowledge graph are determined from the metadata information, and the knowledge graph is constructed based on the multiple data tables, multiple indicators, and the data flow relationship between the multiple data tables and the multiple indicators.

[0059] As some examples, the electronic device 110 can obtain metadata information, that is, extract metadata information from a data warehouse. Metadata information includes data tables, data fields, data types, data sources and / or data update frequencies, etc. The data table contains basic information of the data table, task information of the data table, storage information of the data table, etc. The basic information of the data table includes table name, table description, subject line, included fields and / or field types, etc. The task information of the data table includes task creation time, source table, downstream table, task running time, task start time and / or task end time, etc. The storage information of the data table includes storage capacity, number of files, cold data storage capacity, number of small files, access frequency in the short term (e.g., the past 7 days), access frequency in the long term (e.g., the past 30 days) and / or compression format, etc.

[0060] In some embodiments, the data tables in the data warehouse may have related processing task links, based on which the upstream and downstream processing links related to the data tables in the data warehouse and the data tables on which the processing links depend may be determined.

[0061] Furthermore, in order to facilitate subsequent data analysis, such as data observation, the data table can be processed into one or more indicators according to project requirements, and when it is determined that updated project requirements are generated, the data table can be processed into one or more updated indicators according to the updated project requirements.

[0062] Based on this, the indicator and related information of the indicator can be included in the metadata information. The data table is used as the starting point and the indicator as the end point. The indicator can be traced back to determine the task link that generates the indicator and the data table related to the indicator.

[0063] As some examples, the indicator-related information contained in the metadata may be the indicator name, the indicator Chinese name, the indicator dimension, the indicator caliber, the indicator table and / or the indicator corresponding application, etc.

[0064] In some embodiments, when it is determined that the data tables corresponding to the metadata information have different sources, the data tables in the metadata information may be cleaned, converted, integrated, etc. to ensure the consistency and accuracy of the data tables in the metadata information.

[0065] In some embodiments, in order to improve the analysis capability of the knowledge graph, the above metadata information may be stored in a graph database to improve data processing efficiency and data security.

[0066] In some embodiments, a graph computing model can be constructed to realize the calculation of the knowledge graph, that is, the information in the above-mentioned metadata knowledge graph can be included in the graph computing model, and the graph computing model obtains the final target path from the knowledge graph through calculation. The following is a detailed explanation using the calculation logic of a graph computing model as an example, but is not limited to this.

[0067] Back to Figure 2 At box 220, path processing is performed on the knowledge graph based on the predetermined analysis goal and the preset path requirements to determine the target path corresponding to the data warehouse, where the target path is associated with the preset analysis goal and meets the preset path requirements.

[0068] As some examples, the predetermined analysis target can be set by analyzing the project requirements of the data warehouse to determine the indicators, data flows, etc. that need more attention. For example, if the predetermined analysis target is a human cost target, reducing human cost in the field of human cost can determine the labor cost indicator that needs more attention, etc. For another example, if the predetermined analysis target is an operation management target, optimizing operation management in the field of operation management can determine the cost rate indicator that needs more attention, etc.

[0069] The preset path requirements include the requirements that need to be referred to in determining the target path, such as the length of the path, the importance of the node, the weight of the edge (i.e., directed edge), etc. The length of the path (also known as the path length) can be determined by the running time of the task in the data table. The longer the running time, the longer the path length; when there is relevant data task information on the node, the running start time and end time of the data task can be used as the sequence of the nodes. The importance of the node can be determined by the access frequency of the indicator and the data table. For example, the higher the access frequency in the long term or in the short term, the higher the importance of the node. According to the requirements of data warehouse governance, the storage capacity of the data table, the number of files, the storage capacity of cold data, the number of small files, the storage cost, the computing cost, etc., are weighted by the preset weights to indicate the degree of optimization of the node. The weight of an edge can be understood as the weight of the fields of the downstream node composed of different proportions of the fields of the upstream nodes. For example, among the 10 fields of the downstream node C, 20% of the fields come from the upstream node A, and 80% of the fields come from the upstream node B. Then the weight of the directed edge from node A to node C is 20%, and the weight of the directed edge from node B to node C is 80%, and so on.

[0070] In some embodiments, the electronic device 110 determines multiple initial paths associated with a predetermined analysis target from the knowledge graph; based on preset path requirements, determines a target path from the multiple initial paths.

[0071] As some examples, the traversal of the knowledge graph can be recorded to avoid multiple visits to the same node. Based on this, the electronic device 110 can perform path exploration from the knowledge graph through traversal algorithms such as depth-first algorithm and breadth-first algorithm, obtain access paths, and determine multiple initial paths based on the paths in the access paths.

[0072] Taking the depth-first algorithm as an example, in some embodiments, the electronic device 110 determines the object to be analyzed based on a predetermined analysis target, wherein the object to be analyzed is a data flow relationship between indicators or data units; and traverses the knowledge graph by taking the object to be analyzed as the starting point of traversal to obtain multiple initial paths.

[0073] As some examples, the object to be analyzed can be determined based on the predetermined analysis goal and the actual project. Based on the analysis of the object to be analyzed, it can be determined whether a certain path can meet the predetermined analysis goal. The object to be analyzed may be, for example, a data table, an indicator, or the data flow relationship between a data table and an indicator, etc.

[0074] Taking the indicator corresponding to a node of a directed graph as an example, starting from the node in the directed graph (such as the knowledge graph mentioned above, or a subgraph of the knowledge graph), visit the node, and traverse the directed graph depth-first from the neighboring points connected to the node through directed edges until all nodes in the directed graph that have paths connecting to the node are visited.

[0075] For example, Figure 4A As shown, Figure 4A A schematic diagram of a graph to be traversed according to some embodiments of the present disclosure is shown, using a depth-first algorithm to explore the following Figure 4A Node 410-1, node 410-2 ... node 410-8 in the traversal, with node 410-1 as the traversal starting point (that is, node 410-1 is the object to be analyzed, and the object to be analyzed is the traversal starting point), traversal such as Figure 4A The knowledge graph shown. Both adjacent nodes 410-2 and 410-3 have not been visited, so node 410-2 can be selected as a new starting point. After visiting node 410-2, find the adjacent nodes of node 410-2 that have not been visited. Adjacent to node 410-2 are nodes 410-1, 410-4 and 410-5, of which node 410-1 has been visited, so node 410-4 can be selected as a new starting point. Repeat the above search process, and continue to visit nodes 410-8 and 410-5 in turn. After visiting node 410-5, since the vertices adjacent to node 410-5 have been visited, the search returns to node 410-8, and visits another adjacent node 410-6 of node 410-8. Next, visit nodes 410-3 and 410-7 in turn.

[0076] Based on this, we can get Figure 4B The access sequence shown, 4B shows a schematic diagram of the graph traversal process according to some embodiments of the present disclosure. In Figure 4, the access sequence (i.e., the initial path) obtained by the above path exploration is node 410-1→node 410-2→node 410-4→node 410-8→node 410-5→node 410-6→node 410-3→node 410-7.

[0077] Furthermore, based on the initial path, the processing data source, processing caliber, operation information of upstream tasks, etc. of the node 410 - 1 can be obtained, that is, the data processing flow of the indicator corresponding to the node 410 - 1 is generated.

[0078] Back to Figure 2 In the case where there are multiple predetermined analysis targets, multiple starting nodes or starting directed edges in the knowledge graph can be determined to obtain multiple initial paths. Based on the preset path requirements, the target path can be determined from multiple initial paths.

[0079] In some embodiments, the preset path requirement can be used as a path screening condition to determine a target path from multiple initial paths. Taking the preset path requirement as the path length as an example, the electronic device 110 determines the initial data processing sub-flow corresponding to each initial path in the multiple initial paths; determines the path length corresponding to each initial path based on the process execution time of each initial data processing sub-flow; and screens the multiple initial paths based on the path length requirement and the path length corresponding to each initial path to obtain a target path, wherein the path length of the target path is greater than or equal to the path length of any initial path in the multiple initial paths.

[0080] As some examples, each initial path corresponds to an initial data processing sub-process for obtaining the object to be analyzed, including obtaining data tables, data links, data processing tasks, etc. on which the object to be analyzed depends.

[0081] In specific implementation, the process execution time corresponding to each initial data processing sub-process can be determined. Then, the sum of the process execution time of the data processing tasks in the initial data processing sub-process is determined as the path length of the initial path. Based on the path length of each initial path, a target path whose path length is greater than or equal to the length of other paths is determined as the target path.

[0082] As some examples, the process execution time corresponding to each initial data processing sub-process can be obtained by tracking the logs of each data table and indicator in the data warehouse, thereby obtaining the above-mentioned target path, which will not be repeated here.

[0083] As some examples, Figure 5 As shown, Figure 5 A schematic diagram showing path length calculation according to some embodiments of the present disclosure is shown. The electronic device may also establish a Figure 5 The edges shown represent an active network (i.e., an AOE network, AOE stands for Activity On Edge network), where vertices (also called nodes) represent events, directed edges represent activities (such as data processing tasks), and the weights on directed edges represent the duration of activities. In the AOE network, vertices without incoming edges are called starting points (or source points), and vertices without outgoing edges are called end points (or sink points). In the AOE network, only after the event represented by a vertex occurs can the activities starting from the vertex begin; only after all activities entering a vertex are completed can the event represented by the vertex occur.

[0084] Among them, in the AOE network, the path with the maximum path length (the sum of the duration of each activity on the path) from the starting point to the end point can be determined as the critical path, and the activities on the critical path can be determined as the critical activities. Each activity has a start time and an end time. To find the shortest output time that meets the predetermined analysis goals, the critical path can be found, and the key activities on the critical path can be improved to achieve the purpose of improving data efficiency. Furthermore, the efficiency of the data processing tasks of the key nodes on the critical path can be optimized to achieve the purpose of optimizing the path.

[0085] Furthermore, after the critical path is obtained, the critical path may be determined as the target path, or the critical path may be optimized and adjusted to obtain the target path, and so on.

[0086] As some examples, taking V(0) as the source point and V(n-1) as the sink point, the above critical path determination process is elaborated in detail.

[0087] Specifically, assuming that vertex V(0) is the source point and V(n-1) is the sink point, the occurrence time of event V(0) is time 0, and the longest path from V(0) to V(i) is called the earliest occurrence time of event V(i). Let e(i) represent the earliest start time of the activity, and e(i) determines the earliest start time of activity ai represented by all data links ending with Vi. Let l(i) represent the latest start time of the activity. The difference between the two, l(i)-e(i), means the time margin for completing activity ai. When l(i)=e(i), activity ai becomes a critical activity. Let Ve(j) represent the earliest start time of event V(j). Let Vl(j) represent the latest start time of event V(j).

[0088] Based on this, the following steps 520 to 528 are executed.

[0089] Step 520: Starting from the source point V(0), let Ve(0) = 0, and find Ve(j) = maxi{Ve(i) + |V(i), V(j)|} for the remaining vertices in topologically ordered sequence.<V(i),V(j)> ∈T.

[0090] Where: T is the set of all arcs with Vj as their head. If the number of vertices in the obtained topological ordered sequence is less than the number of vertices n in the network, it means that there is a loop in the network and the critical path cannot be found.

[0091] Step 522: Starting from the sink V(n-1), let Vl(n-1) = Ve(n-1), and find Vl(i) = minj{Vl(j) - |V(i), V(j)} for the remaining vertices according to the inverse topological ordered sequence.<V(i),V(j)> ∈S, where: S is the set of all arcs with Vj as their tail.

[0092] Step 524: Find the earliest start time e(i)=Ve(j) of each activity ai (1<=i<=n), where Vj is the starting point of ai.

[0093] Step 526: Find the latest starting time of each activity ai: l(i)=Vl(j)-|V(i),V(j)|, where Vi is the starting point of ai and Vj is the end point of ai.

[0094] Step 528: When it is determined that e(i)=l(i), activity ai can be determined to be a key activity.

[0095] Based on this, the target path can be determined.

[0096] Back to Figure 2 , at block 230 , a data processing flow associated with the data warehouse is updated based on the target path, wherein the data processing flow is configured to perform data processing on a data table stored in the data warehouse.

[0097] As some examples, when the target path is determined, the target path can be optimized to optimize the data processing flow. Alternatively, the target path is evaluated, and when the evaluation result shows that the target path needs to be optimized, the target path is optimized to update the data processing flow, and when the evaluation result shows that the task does not need to optimize the target path, the data processing flow is not optimized, and the data processing flow of the task data warehouse can meet the project requirements.

[0098] In some embodiments, the electronic device 110 performs path evaluation on the target path based on preset evaluation factors to obtain an evaluation result for the target path; adjusts the target path when it is determined that the evaluation result indicates that path adjustment needs to be performed; and updates the data processing flow associated with the data warehouse based on the adjusted target path.

[0099] As some examples, the preset evaluation factors include the running time of the data processing task on the target path, the computing cost, the storage cost, the number of small files and the number of zombie data, etc.

[0100] Taking the preset evaluation factor as the running time of the data processing task on the target path as an example, that is, corresponding to the path length of the above target path. Based on the running time of the data processing task on the target path, the target path can be evaluated, for example, the running time of the data processing task on the target path, the end time of the last data processing task on the target path, and so on.

[0101] The situations where the evaluation results indicate that path adjustment needs to be performed include but are not limited to the following situations: the running time of the data processing task on the target path of the evaluation result task is too long, the end time of the last data processing task on the target path exceeds the preset end time, etc.

[0102] When the evaluation result indicates that path adjustment needs to be performed, the target path may be adjusted, for example, the data flow direction of the target path may be adjusted, the node attribute information and directed edge attribute information of the target path may be adjusted, and the like.

[0103] In some embodiments, a target data processing sub-process corresponding to a target path can be determined, wherein the target data processing sub-process includes a target data processing task; when it is determined that the target data processing task satisfies a preset task adjustment condition, the target data processing task is adjusted based on the preset task adjustment condition; and based on the adjusted target data processing task, the target path is adjusted.

[0104] As some examples, adjusting the target path can be done by Figure 6 The path optimization process shown is implemented to adjust the target data processing tasks included in the target path.

[0105] Figure 6 A flow chart of a path optimization process according to some embodiments of the present disclosure is shown. Figure 6 The path optimization process shown optimizes the target path, which can reduce the running time and improve the running efficiency.

[0106] The following combination Figure 6 , the path optimization process is elaborated in detail.

[0107] At block 610, the parent task list is updated. The data processing tasks in the target path have dependencies, and the dependent data processing task can be understood as the parent task of the dependent data processing task. Based on this, the parent task of the data processing task in the target path can be determined.

[0108] In order to improve the analysis efficiency, the parent tasks that have been completed in the previous cycle can be removed from the parent tasks, so as to obtain an updated parent task list.

[0109] At block 620 , it is determined whether the parent task is valid. The electronic device 110 may determine whether the parent task of the data processing task is a valid task, that is, whether the parent task still needs to perform data processing.

[0110] If the parent task is a valid task, the target start time is calculated at block 630. The electronic device 110 calculates the end time of all parent tasks, and takes the parent task corresponding to the latest end time among the end times of the parent tasks as the end task. Based on this, the end task of the data processing task in the target path can be determined, that is, the task that ends latest among the upper parent tasks of the data processing task.

[0111] Furthermore, based on the end time of the data processing task, the task start time of the data processing task, that is, the target start time, can be determined. It can be understood that the data processing task can only start execution when it is determined that the end task of the data processing task has been completed.

[0112] At block 640 , the optimization strategy is executed. The electronic device 110 may compare the target start time of the data processing task with the actual start time of the data processing task in the target path to determine whether the scheduling time of the data processing task can be adjusted.

[0113] When it is determined that the target start time of the data processing task is earlier than the actual start time of the data processing task in the target path, the actual start time of the data processing task may be modified to complete the scheduling time adjustment of the data processing task.

[0114] In some embodiments, the electronic device 110 can adjust the scheduling time of the data processing task, which will cause the end time of the data processing task to change. Based on this, the end time of the downstream tasks of the data processing task can be re-estimated, thereby optimizing the task scheduling time of the data processing sub-process corresponding to the target path.

[0115] In some embodiments, the electronic device 110 can estimate the queuing time of the data processing task based on the current moment, and replace the execution engine of the data processing task with an execution engine that improves the data processing speed when the queuing time of the data processing task meets a preset time threshold. For example, an execution engine with better data processing quality can be replaced with an execution engine with better data processing speed, thereby reducing the execution time of the data processing task and reducing the queuing time of the data processing task.

[0116] In some embodiments, the electronic device 110 can check the running time delay, that is, check whether the data processing task has a start time delay due to rerunning or error reporting, and based on this, the task execution logic of the data processing task can be adjusted to reduce the probability of running time delay.

[0117] In some embodiments, the electronic device 110 may generate detailed information of the final blocked link, such as generating a final blocked link information table, and optimize the blocked links that affect the task end time (i.e., the end time of the last task of the target path) based on the blocked link information table.

[0118] Based on one or more of the above optimization methods, the data processing sub-flow corresponding to the target path is optimized.

[0119] Furthermore, at block 650, it is determined whether the target is achieved, that is, whether the target end time corresponding to the target path meets a preset end time threshold, or whether the data processing sub-process corresponding to the target path meets an early stopping condition.

[0120] When it is determined that the target path has not reached the target, at box 660, optimization continues, that is, based on one or more of the above-mentioned optimization methods, the data processing sub-process corresponding to the target path continues to be optimized. The selection of the optimization method can be determined based on the actual project.

[0121] In some embodiments, the optimization direction may be determined, that is, whether to look for the optimization direction upstream or downstream may be determined according to the end time of the current data processing task.

[0122] To ensure the normal operation of the electronic device 110, at block 670, it is determined whether the number of optimizations exceeds the limit. If so, the optimization is terminated; if not, the step optimization strategy is continued until the preset end condition is met (the target end time corresponding to the target path meets the preset end time threshold, or the data processing sub-process corresponding to the target path reaches the early stop condition, or the number of optimizations exceeds the limit, etc.).

[0123] In some embodiments, if it is determined that the target path meets the target, at block 680, an optimization report is generated to determine an updated target path.

[0124] Furthermore, based on the adjusted target path, the data processing flow is updated, that is, the data processing flow of the indicators generated by the data table is updated, such as adjusting the data processing tasks in the data processing flow, adjusting the scheduling of the data processing tasks in the data processing flow, and so on.

[0125] In some embodiments, the target data processing sub-process corresponding to the target path can be adjusted based on the adjusted target path to obtain the adjusted data processing sub-process; based on the adjusted data processing sub-process, the data processing process associated with the data warehouse is updated, thereby realizing the update of the entire data processing process.

[0126] In some embodiments, when the update of the data processing flow is completed, an updated data processing process can be executed based on the updated data processing flow.

[0127] Back to Figure 2 At block 240 , in response to detecting an update to the data table, data processing is performed on the updated data table according to an updated data processing flow.

[0128] As some examples, the data warehouse can be dynamically updated at all times, that is, at each moment, the data in the data warehouse will change dynamically. Based on this, when there is an update to the data table, in response to detecting the update to the data table, data processing can be performed on the updated data table according to the updated data processing flow.

[0129] In some embodiments, changes in key links may not occur frequently, so different update cycles of data processing flows may be designed according to application requirements of different analysis topics to set update times for different target paths.

[0130] As some examples, the data processing flow updated in each update cycle may be used to perform data processing on the data table of the next update cycle.

[0131] The following describes the beneficial effects that the data processing method of the present disclosure can bring compared with manual link search.

[0132] On the one hand, the data processing method of the present disclosure consumes less resources compared with manually searching for links.

[0133] In some embodiments, by adopting a method based on a graph computing model and a knowledge graph, utilizing the efficient indexing and query capabilities of the graph computing model from the knowledge graph, and exploring and identifying the correlations, causal relationships, or other potential associations between different indicators, in order to construct target paths under different predetermined analysis objectives, the efficiency of constructing data processing links in the data warehouse and the data query performance are improved, thereby improving R&D efficiency and reducing data computing costs.

[0134] As some examples, a logistics operation data network graph model containing 100 nodes and 500 edges is taken as an example of the above knowledge graph for simulation, and different types of tasks and links are set for the 100 nodes and 500 edges to simulate the diversity in actual operations.

[0135] refer to Figure 2 The process 200 and the above data processing method are used to identify the target path of the logistics operation data network. According to the task association relationship in the graph database, an optimization scheme (such as one or more of the above optimization methods) is applied to optimize the path, and finally a golden link (i.e., target path) is formed in the logistics operation data scenario, and the following performance indicators are used for comparison:

[0136] (1) Runtime efficiency: Through simulation experiments, the processing time of key links before and after optimization is recorded and compared to measure the improvement in operation efficiency.

[0137] (2) Number of link tasks: Analyze the number of tasks on the key link (i.e., target path) before and after optimization to evaluate the balance of link load and the efficiency of task flow.

[0138] (3) Storage cost: Evaluate the demand for storage resources of the data processing method and compare the storage costs before and after optimization to reflect the optimization of resource utilization.

[0139] The simulation results are as follows Figure 7 As shown, Figure 7 A schematic diagram of simulation experiment results according to some embodiments of the present disclosure is shown.

[0140] based on Figure 7 It can be determined that the data processing method disclosed in the present invention improves the running efficiency by about 20%, shortens the processing time of key links, reduces the number of link tasks by about 15%, and makes the task allocation more reasonable, reduces the storage cost by about 10%, and optimizes the resource utilization.

[0141] Based on this, the data processing method disclosed in the present invention has significantly improved the operating efficiency of the logistics operation data warehouse, effectively reduced the number of link tasks, and significantly reduced the storage cost.

[0142] On the other hand, the data processing method disclosed in the present invention can automatically perform data processing in a dynamically updated data warehouse scenario, improve the data processing capabilities and real-time performance of the data warehouse, reduce platform operating costs, and improve the accuracy and flexibility of decisions based on the data warehouse, thereby meeting ever-changing project needs.

[0143] In summary, the data processing method disclosed in the present invention improves the efficiency of analyzing the relationship between data in the data warehouse, increases the update speed of data in the data warehouse, reduces the operation and maintenance cost of the data warehouse, and automatically determines the target path based on predetermined analysis goals, thereby automatically updating the data processing process, significantly improving the flexibility, scalability and expansion speed of the data warehouse.

[0144] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 8 A schematic structural block diagram of an apparatus 800 for data processing according to some embodiments of the present disclosure is shown. The apparatus 800 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 800 may be implemented by hardware, software, firmware or any combination thereof.

[0145] like Figure 8 As shown, the device 800 includes a knowledge graph acquisition module 810, which is configured to obtain a knowledge graph constructed for a data warehouse, wherein the knowledge graph includes multiple nodes and directed edges connecting the nodes, each node represents a data table stored in the data warehouse or an indicator obtained after data processing is performed on the data table, and each directed edge connecting the nodes represents the data flow relationship between the data tables or indicators corresponding to the connected nodes; a target path determination module 820, which is configured to perform path processing on the knowledge graph based on a predetermined analysis target and a preset path requirement, and determine the target path corresponding to the data warehouse, wherein the target path is associated with the preset analysis target and meets the preset path requirement; a data processing flow update module 830, which is configured to update the data processing flow associated with the data warehouse based on the target path, wherein the data processing flow is configured to perform data processing on the data table stored in the data warehouse; and a data processing module 840, which is configured to perform data processing on the updated data table according to the updated data processing flow in response to detecting an update to the data table.

[0146] In some embodiments, the target path determination module 820 is further configured to: determine multiple initial paths associated with a predetermined analysis target from the knowledge graph; and determine a target path from the multiple initial paths based on preset path requirements.

[0147] In some embodiments, the target path determination module 820 is further configured to: determine the object to be analyzed based on a predetermined analysis target, wherein the object to be analyzed is a data flow relationship between indicators or data units; and traverse the knowledge graph by taking the object to be analyzed as the starting point of traversal to obtain multiple initial paths.

[0148] In some embodiments, the target path determination module 820 is further configured to: determine an initial data processing sub-process corresponding to each initial path among multiple initial paths; determine a path length corresponding to each initial path based on the process execution time of each initial data processing sub-process; based on the path length requirement and the path length corresponding to each initial path, screen multiple initial paths to obtain a target path, wherein the path length of the target path is greater than or equal to the path length of any one of the multiple initial paths.

[0149] In some embodiments, the data processing flow updating module 830 is further configured to:

[0150] Based on preset evaluation factors, a path evaluation is performed on a target path to obtain an evaluation result for the target path; when it is determined that the evaluation result indicates that a path adjustment needs to be performed, the target path is adjusted; and based on the adjusted target path, a data processing flow associated with the data warehouse is updated.

[0151] In some embodiments, the data processing flow update module 830 is further configured to: determine a target data processing sub-flow corresponding to a target path, wherein the target data processing sub-flow includes a target data processing task; when it is determined that the target data processing task satisfies a preset task adjustment condition, adjust the target data processing task based on the preset task adjustment condition; and adjust the target path based on the adjusted target data processing task.

[0152] In some embodiments, the data processing flow update module 830 is further configured to: based on the adjusted target path, adjust the target data processing sub-flow corresponding to the target path to obtain the adjusted data processing sub-flow; based on the adjusted data processing sub-flow, update the data processing flow associated with the data warehouse.

[0153] In some embodiments, the device 800 also includes a knowledge graph construction module, which is configured to: determine metadata information stored in a data warehouse; determine multiple data tables and multiple indicators to be used to construct a knowledge graph from the metadata information; and construct a knowledge graph based on multiple data tables and multiple indicators and data flow relationships between the multiple data tables and multiple indicators.

[0154] The units and / or modules included in the device 800 can be implemented in various ways, including software, hardware, firmware or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 800 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0155] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or a combination of electronic devices. Such an electronic device or a combination of electronic devices may include, for example, Figure 1 The electronic device 110 in.

[0156] Fig. 9 900 is a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. It should be understood that Fig. 9 The electronic device 900 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Fig. 9 The electronic device 900 shown can be used to implement Figure 1 Electronic device 110 or Figure 8 Device 800.

[0157] like Fig. 9 As shown, the electronic device 900 is in the form of a general electronic device. The components of the electronic device 900 may include, but are not limited to, one or more processors or processing units 910, a memory 920, a storage device 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. The processing unit 910 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 920. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 900.

[0158] The electronic device 900 typically includes a plurality of computer storage media. Such media can be any available media accessible to the electronic device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 920 can be a volatile memory (e.g., register, cache, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory) or some combination thereof. The storage device 930 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 900.

[0159] The electronic device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig. 9 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 920 may include a computer program product 925 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0160] The communication unit 940 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 900 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 900 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0161] The input device 950 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 960 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 900 may also communicate with one or more external devices (not shown) through the communication unit 940 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 900, or communicate with any device that allows the electronic device 900 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0162] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0163] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0164] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0165] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0166] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0167] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for data processing, comprising: Obtain a knowledge graph constructed for a data warehouse, wherein the knowledge graph comprises a plurality of nodes and directed edges connecting the nodes, each node represents a data table stored in the data warehouse or an indicator obtained after data processing is performed on the data table, and each directed edge connecting the nodes represents a data flow relationship between the data tables or indicators corresponding to the connected nodes; Based on a predetermined analysis target and a preset path requirement, path processing is performed on the knowledge graph to determine a target path corresponding to the data warehouse, wherein the target path is associated with the preset analysis target and satisfies the preset path requirement; updating a data processing flow associated with the data warehouse based on the target path, wherein the data processing flow is configured to perform data processing on a data table stored in the data warehouse; as well as In response to detecting an update to the data table, data processing is performed on the updated data table according to the updated data processing flow.

2. The method according to claim 1, wherein based on the predetermined analysis target and the preset path requirement, path processing is performed on the knowledge graph, and determining the target path comprises: Determining a plurality of initial paths associated with the predetermined analysis target from the knowledge graph; as well as Based on the preset path requirement, the target path is determined from the multiple initial paths.

3. The method according to claim 2, wherein determining the initial path from the knowledge graph comprises: Based on the predetermined analysis target, determining an object to be analyzed, wherein the object to be analyzed is a data flow relationship between the indicators or the data units; and The knowledge graph is traversed by taking the object to be analyzed as a traversal starting point to obtain the multiple initial paths.

4. The method according to claim 2, wherein the preset path requirement includes a path length requirement, and based on the preset path requirement, determining the target path from the plurality of initial paths comprises: Determine an initial data processing sub-flow corresponding to each initial path in the multiple initial paths; Determining the path length corresponding to each initial path based on the process execution time of each initial data processing sub-process; as well as Based on the path length requirement and the path length corresponding to each of the initial paths, the multiple initial paths are screened to obtain the target path, wherein the path length of the target path is greater than or equal to the path length of any one of the multiple initial paths.

5. The method according to claim 1, wherein updating the data processing flow based on the target path comprises: Based on preset evaluation factors, performing path evaluation on the target path to obtain an evaluation result for the target path; If it is determined that the evaluation result indicates that path adjustment needs to be performed, adjusting the target path; as well as Based on the adjusted target path, the data processing flow associated with the data warehouse is updated.

6. The method of claim 5, wherein adjusting the target path comprises: Determine a target data processing sub-process corresponding to the target path, wherein the target data processing sub-process includes a target data processing task; In a case where it is determined that the target data processing task satisfies a preset task adjustment condition, adjusting the target data processing task based on the preset task adjustment condition; and Based on the adjusted target data processing task, the target path is adjusted.

7. The method according to claim 5, wherein updating the data processing flow associated with the data warehouse based on the adjusted target path comprises: Based on the adjusted target path, adjusting the target data processing sub-flow corresponding to the target path to obtain an adjusted data processing sub-flow; as well as Based on the adjusted data processing sub-process, the data processing process associated with the data warehouse is updated.

8. The method according to claim 1, further comprising: Determining metadata information stored in the data warehouse; Determine, from the metadata information, a plurality of data tables and a plurality of indicators to be used for constructing the knowledge graph; as well as The knowledge graph is constructed based on the multiple data tables and the multiple indicators and the data flow relationship between the multiple data tables and the multiple indicators.

9. A device for data processing, comprising: A knowledge graph acquisition module is configured to obtain a knowledge graph constructed for a data warehouse, wherein the knowledge graph comprises a plurality of nodes and directed edges connecting the nodes, each node represents a data table stored in the data warehouse or an indicator obtained after data processing is performed on the data table, and each directed edge connecting the nodes represents a data flow relationship between the data tables or indicators corresponding to the connected nodes; A target path determination module is configured to perform path processing on the knowledge graph based on a predetermined analysis target and a preset path requirement, and determine a target path corresponding to the data warehouse, wherein the target path is associated with the preset analysis target and satisfies the preset path requirement; a data processing flow updating module, configured to update a data processing flow associated with the data warehouse based on the target path, wherein the data processing flow is configured to perform data processing on a data table stored in the data warehouse; as well as The data processing module is configured to, in response to detecting an update to the data table, perform data processing on the updated data table according to the updated data processing flow.

10. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 8.

Images