Data processing method, data processing equipment and computer readable storage medium
By acquiring the current operating parameters and the configuration information of the theoretical directed acyclic graph (DAG), the DAG is reconstructed and optimized, solving the problems of low task processing efficiency and low resource utilization of static DAG in complex environments, and realizing the improvement of the intelligence and dynamic adaptability of DAG.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JUHAOKAN TECH CO LTD
- Filing Date
- 2025-12-22
- Publication Date
- 2026-05-19
AI Technical Summary
Traditional task scheduling mechanisms based on static DAGs are difficult to adapt to complex and ever-changing data environments, resulting in low task processing efficiency and low resource utilization, and failing to meet the needs of efficient and flexible data processing.
By acquiring the current operating parameters and the configuration information of the theoretical directed acyclic graph, the directed acyclic graph is reconstructed. The reconstruction score is used to select a better reconstructed directed acyclic graph, and the task execution order is dynamically adjusted to improve the level of intelligence and adaptability.
It improves the intelligence and dynamic adaptability of DAG, optimizes task processing efficiency and resource utilization, and solves the problem of low efficiency of static DAG in complex environments.
Smart Images
Figure CN122064432A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the fields of big data processing and distributed computing technology, and in particular to a data processing method, a data processing device, and a computer-readable storage medium. Background Technology
[0002] In the field of big data processing, Directed Acyclic Graphs (DAGs) are widely used to describe the dependencies and execution order between tasks. Traditional task scheduling mechanisms based on static DAGs typically execute strictly according to user-preset structures. However, different users may set up significantly different DAG structures for the same business objective, and this experience-based manual configuration method lacks dynamic adaptability. In complex and ever-changing data environments, static DAGs struggle to adaptively adjust to actual data readiness, resource load changes, and task execution status, leading to problems such as low task processing efficiency, low resource utilization, and delayed business response. Especially in scenarios where data partitioning granularity is continuously refined, relying entirely on initially manually set DAGs can no longer meet the demands for efficient and flexible data processing.
[0003] Therefore, how to improve the intelligence level and dynamic adaptability of DAG has become an urgent technical problem to be solved. Summary of the Invention
[0004] To address the aforementioned technical problems, this disclosure provides a data processing method, a data processing apparatus, and a computer-readable storage medium.
[0005] In a first aspect, this disclosure provides a data processing device, comprising: a communicator configured to communicate with a client; and a controller configured to: when executing a task containing a theoretical directed acyclic graph (DAG), acquire current operating parameters and first configuration information of the theoretical DAG; wherein the first configuration information indicates at least one node contained in the theoretical DAG, tables contained in each node, and table partitions of each table; reconstruct the DAG based on the first configuration information to obtain at least one reconstructed DAG; for each reconstructed DAG, determine a reconstruction score of the reconstructed DAG based on the reconstructed DAG and the current operating parameters; filter each reconstructed DAG according to pre-configured structure configuration information and reconstruction score of the DAG to determine a filtered DAG; wherein the filtered DAG is different from the theoretical DAG, and the reconstruction score of the filtered DAG is greater than the reconstruction score of the theoretical DAG; and execute the task according to the filtered DAG to obtain the task result.
[0006] Secondly, this disclosure provides a data processing method, comprising: when executing a task containing a theoretical directed acyclic graph, obtaining current operating parameters and first configuration information of the theoretical directed acyclic graph; wherein the first configuration information indicates at least one node contained in the theoretical directed acyclic graph, tables contained in each node, and table partitions of each table; reconstructing the directed acyclic graph based on the first configuration information to obtain at least one reconstructed directed acyclic graph; for each reconstructed directed acyclic graph, determining a reconstruction score of the reconstructed directed acyclic graph based on the reconstructed directed acyclic graph and the current operating parameters; filtering each reconstructed directed acyclic graph according to the pre-configured structural configuration information and reconstruction score of the directed acyclic graph to determine a filtered directed acyclic graph; wherein the filtered directed acyclic graph is different from the theoretical directed acyclic graph, and the reconstruction score of the filtered directed acyclic graph is greater than the reconstruction score of the theoretical directed acyclic graph; and executing the task according to the filtered directed acyclic graph to obtain the task result.
[0007] Thirdly, this disclosure provides a computer-readable storage medium, comprising: storing a computer program on the computer-readable storage medium, the computer program being executed by a controller using a data processing method as provided in any of the second aspects.
[0008] Fourthly, this disclosure provides a computer program product that, when run on a computer, causes the computer to perform any of the data processing methods provided in the second aspect.
[0009] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on the first computer-readable storage medium. The first computer-readable storage medium may be encapsulated together with the controller of the data processing device, or it may be encapsulated separately from the controller of the data processing device; this disclosure does not impose any limitations on this.
[0010] The descriptions of the second, third, and fourth aspects in this disclosure can be referenced to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, and fourth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.
[0011] In this disclosure, the names of the aforementioned data processing devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this disclosure, they fall within the scope of this disclosure and its equivalents.
[0012] These or other aspects of this disclosure will become more readily apparent in the following description.
[0013] The technical solution provided in this disclosure has the following advantages compared with the prior art: The data processing device disclosed herein includes: a communicator configured to communicate with a client; and a controller configured to: when executing a task containing a theoretical directed acyclic graph (DAG), acquire current operating parameters and first configuration information of the theoretical DAG; reconstruct the DAG based on the first configuration information to obtain at least one reconstructed DAG; for each reconstructed DAG, determine a reconstruction score based on the reconstructed DAG and the current operating parameters; filter each reconstructed DAG according to pre-configured structure configuration information and reconstruction scores to determine selected DAGs; and execute the task according to the selected DAGs to obtain the task result. In this way, the data processing device can automatically optimize the user-configured theoretical DAG, thereby improving the intelligence level and dynamic adaptability of the DAG, solving the problem of how to improve the intelligence level and dynamic adaptability of the DAG. Attached Figure Description
[0014] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0015] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 One of the flowcharts illustrating the data processing method provided in the embodiments of this application; Figure 2 A schematic diagram of the server structure for the data processing method provided in the embodiments of this application; Figure 3 A second schematic flowchart illustrating the data processing method provided in this application embodiment; Figure 4 The third schematic flowchart of the data processing method provided in the embodiments of this application; Figure 5 The fourth flowchart illustrating the data processing method provided in the embodiments of this application; Figure 6 Fifth flowchart illustrating the data processing method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application; Figure 8 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation
[0017] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0018] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0019] The display device provided in this application can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is one specific embodiment of the display device of this application.
[0020] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0021] The atomic switching provided in this disclosure is used to ensure the consistency of data state and task state during reconstruction, provide an all-or-nothing DAG switching guarantee, and automatically roll back to the original DAG state when reconstruction fails.
[0022] The data processing device provided in this disclosure can be a server. When the server executes the data processing method provided in this disclosure, the server's processor may execute the data processing method provided in this disclosure.
[0023] In the following embodiments, the execution subject of the data processing method provided in the embodiments of this disclosure is the aforementioned server, which will be used as an example to illustrate the method of the embodiments of this application.
[0024] This application provides a data processing method, such as... Figure 1 As shown, the data processing method may include S11-S15.
[0025] S11. When executing a task containing a theoretical directed acyclic graph, obtain the current running parameters and the first configuration information of the theoretical directed acyclic graph; wherein, the first configuration information indicates at least one node contained in the theoretical directed acyclic graph, the tables contained in each node, and the table partitions of each table.
[0026] In some examples, current operating parameters include the usage of resources such as the Central Processing Unit (CPU), memory, disk I / O (input / output), and network bandwidth.
[0027] In some examples, the initial configuration information can be used to detect the generation and changes of table partitions in each table of the node in real time through file system event listening and metadata inspection mechanisms.
[0028] In some examples, the system architecture of the server executing the data processing method provided in the embodiments of this disclosure is as follows: Figure 2 As shown, it includes a state monitoring layer, a dynamic analysis layer, and a reconstruction engine layer. The state monitoring layer is connected to the data source, the dynamic analysis layer is connected to the historical state database, and the reconstruction engine layer is connected to the rule knowledge base (used to store structure configuration information) and the business strategy database (used to store the theoretical directed acyclic graph of the updated tasks).
[0029] The status monitoring layer includes: a data status monitor, a resource status monitor, and a task status monitor. The data status monitor is used to monitor the tables contained in each node of the theoretical directed acyclic graph and the generation and readiness status of table partitions of each table in real time through file system event listening and metadata inspection mechanisms. The resource status monitor is used to collect the usage of resources such as CPU, memory, disk I / O, and network bandwidth in real time. The task status monitor is used to collect the task status (completed or incomplete) of the theoretical directed acyclic graph and the status of upstream and downstream tasks in real time.
[0030] The dynamic analysis layer comprises a state analyzer, a constraint analyzer, and an optimization opportunity identifyer. The state analyzer acquires task states collected by the task state monitor. The constraint analyzer obtains the pre-configured structural configuration information of the directed acyclic graph (DAG). The optimization opportunity identifyer, based on the historical state database, task states, and the pre-configured DAG structural configuration information, combines conditional dependency resolution, probabilistic dependency prediction, and conflict dependency detection to identify optimization opportunities for dynamic DAG reconstruction. Examples include: dynamically calculating dependencies based on runtime state conditional expressions; predicting future dependency changes using probabilistic graphical models; identifying resource conflicts, data conflicts, and timing conflicts; and providing solutions. It also identifies subsets of independent tasks that can be executed in advance, discovers task reordering and resource reallocation opportunities based on resource states, and adjusts the task execution order according to the business context.
[0031] The task status monitor includes a risk assessor, a DAG generator, and a strategy selector. The DAG generator is used to reconstruct directed acyclic graphs (DAGs) based on the first configuration information, resulting in at least one reconstructed DAG. The risk assessor is used to determine the reconstruction score of each reconstructed DAG based on the reconstructed DAG and the current running parameters. Each reconstructed DAG is then filtered according to the pre-configured structure configuration information and reconstruction score to determine the selected DAGs. The strategy selector is used to execute the task according to the selected DAGs to obtain the task result.
[0032] S12. Based on the first configuration information, reconstruct the directed acyclic graph to obtain at least one reconstructed directed acyclic graph.
[0033] In some examples, when reconstructing a directed acyclic graph based on the first configuration information to obtain at least one reconstructed directed acyclic graph, the first configuration information can be input into a probabilistic model to predict dependencies, thereby obtaining at least one reconstructed directed acyclic graph. The training process of the probabilistic model includes: Obtain the first training sample data and the first labeling result of the first training sample data. The first training sample data includes: the actual configuration information of historical DAGs, and the first labeling result includes at least one reconstructed directed acyclic graph corresponding to each actual configuration information.
[0034] The first training sample data is input into the first neural network model for learning, and the first prediction result of the first neural network model on the first training sample data is obtained.
[0035] Based on the first prediction result and the first labeling result, the network parameters of the first neural network model are adjusted until the first neural network model converges to obtain the probability model.
[0036] In some examples, when reconstructing a directed acyclic graph (DAG) based on the first configuration information to obtain at least one reconstructed DAG, the nodes in the first configuration information can first be randomly sorted to obtain at least one first sorting result. Then, based on each sorting result, the tables and table partitions within each node are randomly sorted to obtain at least one second sorting result corresponding to each sorting result. The node dependencies in each of the at least one second sorting result are the same as the node dependencies in the original sorting result. Then, based on the second sorting results, the reconstructed DAG is determined to be the second sorting result. Each second sorting result corresponds to one reconstructed DAG.
[0037] For example, suppose the first configuration information includes three nodes: node A, node B, and node C. Node A includes table 1, which includes table partition 1 and table partition 2. Node B includes table 2, which includes table partition 3 and table partition 4. Node C includes tables 3 and 4, which include table partition 5 and table partition 6 and table partition 7. The dependencies between the nodes in the first configuration information are node A - node B and - node C. Table partition 3 requires data from table partition 1, and table partition 4 requires data from table partition 2.
[0038] First, nodes A, B, and C are randomly sorted to obtain at least one first sorting result, such as: node A - node B and - node C, node A - node C and - node B, node B - node A and - node C, node B - node C and - node A, node C - node A and - node B, node C - node B and - node A; then, the tables and table partitions in the first sorting result (node A - node B and - node C) are randomly sorted to obtain at least one sorting result corresponding to each sorting result. The second sorting result is, for example: table partition 1 - table partition 3 - table partition 5 - table partition 6, table partition 1 - table partition 3 - table partition 5 - table partition 7, table partition 1 - table partition 4 - table partition 5 - table partition 6, table partition 1 - table partition 4 - table partition 5 - table partition 7, table partition 2 - table partition 3 - table partition 5 - table partition 6, table partition 2 - table partition 3 - table partition 5 - table partition 7, table partition 2 - table partition 4 - table partition 5 - table partition 6, table partition 2 - table partition 4 - table partition 5 - table partition 7. Thus, at least one second sorting result is obtained corresponding to the first sorting result (node A - node B and - node C).
[0039] S13. For each reconstructed directed acyclic graph, determine the reconstruction score of the reconstructed directed acyclic graph based on the reconstructed directed acyclic graph and the current running parameters.
[0040] In some examples, when determining the reconstruction score of a directed acyclic graph (DAG) based on the DAG and the current running parameters, the actual total time spent executing the DAG can be determined based on the current running parameters; the reconstruction score of the DAG can then be determined as the actual total time spent. Alternatively, for each DAG, the second configuration information of the DAG and the current running parameters can be input into a weighted scoring model for scoring to obtain the reconstruction score of the DAG.
[0041] In some examples, when determining the reconstruction score of a reconstructed DAG based on the reconstructed DAG and current running parameters, the acyclicity of the DAG (topological sorting) can be checked to ensure node connectivity (no isolated nodes). Then, the execution order is determined based on the topological sort of the reconstructed DAG, and resource allocation simulations are performed (CPU / Memory, calculating task waiting time t1, total latency (task completion time t2), throughput (number of tasks per unit time), and resource utilization). Finally, the reconstruction score is determined based on task waiting time t1, total latency, throughput, and resource utilization.
[0042] S14. According to the pre-configured structural configuration information and reconstruction score of the directed acyclic graph, each reconstructed directed acyclic graph is screened to determine the screened directed acyclic graph; wherein, the screened directed acyclic graph is different from the theoretical directed acyclic graph, and the reconstruction score of the screened directed acyclic graph is greater than the reconstruction score of the theoretical directed acyclic graph.
[0043] In some examples, when a theoretical directed acyclic graph (DAG) needs to be completed by two or more reconstructed DAGs, the DAGs are selected to include two or more reconstructed DAGs that can complete the theoretical DAG.
[0044] In some examples, the structural configuration information specifies one or more of the following: acyclic property, node connectivity, and topological sorting. Acyclicity is defined as the requirement that there exists a unique path between all nodes in a DAG, and that there are no circular dependencies. Verification method: Use networkx.is_directed_acyclic_graph() to check for acyclicity.
[0045] Verify the executable order of nodes using topological sort (topological_sort()).
[0046] Check node connectivity (is_weakly_connected()) to ensure all nodes are reachable.
[0047] Example: Input: [('A', 'B'), ('B', 'C'), ('A', 'C')]; Output: True - The DAG structure is valid and acyclic, and the executability verification passed; Note: This DAG is acyclic, all nodes are connected, and the topology sorting is successful.
[0048] Node connectivity is defined as ensuring that all nodes are connected through edges, avoiding execution interruptions caused by isolated nodes. Verification method: Use is_weakly_connected() to check node connectivity.
[0049] A discrepancy between the number of nodes in the topological sort result and the number of nodes in the graph indicates the presence of isolated nodes.
[0050] The definition of topological sorting is: ensuring that tasks are executed in dependency order, and that an acyclic graph always has a unique topological sort result; verification method: A failure to sort a topology (throwing a NetworkXUnfeasible exception) indicates the presence of a cycle.
[0051] A discrepancy between the number of nodes in the topological sort result and the number of nodes in the graph indicates the presence of isolated nodes.
[0052] In some examples, each reconstructed directed acyclic graph (DAG) is filtered according to the pre-configured structure configuration information and reconstruction score of the DAG. When determining which DAGs to filter, each DAG can be filtered according to the pre-configured structure configuration information to identify the reconstructed DAGs that meet the structure configuration information; and then the reconstruction score of the reconstructed DAGs that meet the structure configuration information is used to determine which DAGs to filter.
[0053] In some examples, the higher the reconstruction score, the less resource consumption is required to reconstruct the directed acyclic graph (DAG) than the theoretical DAG, and the shorter the task time for reconstructing the DAG is compared to the theoretical DAG.
[0054] S15. Perform the task according to the filtered directed acyclic graph to obtain the task result.
[0055] In some examples, if a directed acyclic graph is reconstructed based on the first configuration information, and a reconstructed directed acyclic graph is obtained, and the reconstructed directed acyclic graph is the same as the theoretical directed acyclic graph, then the task result is obtained directly according to the central task of the theoretical directed acyclic graph.
[0056] In some examples, upon obtaining the filtered directed acyclic graph (DAG), an atomic switch is required. The execution server can then execute the task according to the filtered DAG and obtain the task result. Subsequently, in the next execution of the task, the theoretical DAG contained in the task will be the filtered DAG.
[0057] As described above, the data processing method provided in this embodiment, when executing a task involving a theoretical directed acyclic graph (DAG), acquires the current operating parameters and the first configuration information of the theoretical DAG; reconstructs the DAG based on the first configuration information to obtain at least one reconstructed DAG; for each reconstructed DAG, determines a reconstruction score based on the reconstructed DAG and the current operating parameters; filters each reconstructed DAG according to the pre-configured structure configuration information and reconstruction score of the DAG to determine the selected DAGs; and executes the task according to the selected DAGs to obtain the task result. In this way, the data processing device can automatically optimize the user-configured theoretical DAG, thereby improving the intelligence level and dynamic adaptability of the DAG.
[0058] In some feasible examples, combining Figure 1 ,like Figure 3 As shown, the above S12 can be implemented by the following S120.
[0059] S120. Predict the dependencies between nodes, tables, and table partitions in the first configuration information to obtain at least one reconstructed directed acyclic graph.
[0060] As described above, the data processing method provided in this embodiment, when executing a task involving a theoretical directed acyclic graph (DAG), obtains the current operating parameters and the first configuration information of the theoretical DAG; predicts the dependencies between nodes, tables, and table partitions in the first configuration information to obtain at least one reconstructed DAG; for each reconstructed DAG, a reconstruction score is determined based on the reconstructed DAG and the current operating parameters; each reconstructed DAG is filtered according to the pre-configured structure configuration information and reconstruction score of the DAG to determine the filtered DAG; and the task is executed according to the filtered DAG to obtain the task result. In this way, the data processing device can automatically optimize the user-configured theoretical DAG, thereby improving the intelligence level and dynamic adaptability of the DAG.
[0061] In some feasible examples, combining Figure 1 ,like Figure 4 As shown, the above S13 can be implemented by the following S130.
[0062] S130. For each reconstructed directed acyclic graph, based on the current running parameters, determine the actual total time spent reconstructing the directed acyclic graph; based on the actual total time spent, determine the reconstruction score of the reconstructed directed acyclic graph as the actual total time spent.
[0063] As described above, the data processing method provided in this embodiment, when executing a task involving a theoretical directed acyclic graph (DAG), obtains the current operating parameters and the first configuration information of the theoretical DAG; reconstructs the DAG based on the first configuration information to obtain at least one reconstructed DAG; for each reconstructed DAG, determines the actual total time spent reconstructing the DAG based on the current operating parameters; determines the reconstruction score of the reconstructed DAG as the actual total time spent based on the actual total time spent; filters each reconstructed DAG according to the pre-configured structure configuration information and reconstruction score of the DAG to determine the filtered DAG; and executes the task according to the filtered DAG to obtain the task result. In this way, the data processing device can automatically optimize the user-configured theoretical DAG, thereby improving the intelligence level and dynamic adaptability of the DAG.
[0064] In some feasible examples, combining Figure 1 ,like Figure 5 As shown, the above S13 can be implemented in the following S131.
[0065] S131. For each reconstructed directed acyclic graph, the second configuration information and current running parameters of the reconstructed directed acyclic graph are input into the weighted scoring model for scoring, and the reconstruction score of the reconstructed directed acyclic graph is obtained.
[0066] As described above, the data processing method provided in this embodiment, when executing a task involving a theoretical directed acyclic graph (DAG), obtains the current operating parameters and the first configuration information of the theoretical DAG; reconstructs the DAG based on the first configuration information to obtain at least one reconstructed DAG; for each reconstructed DAG, the second configuration information of the reconstructed DAG and the current operating parameters are input into a weighted scoring model for scoring to obtain a reconstruction score for the reconstructed DAG; according to the pre-configured structure configuration information and reconstruction score of the DAG, each reconstructed DAG is filtered to determine the filtered DAG; the task is executed according to the filtered DAG to obtain the task result. In this way, the data processing device can automatically optimize the user-configured theoretical DAG, thereby improving the intelligence level and dynamic adaptability of the DAG.
[0067] In some feasible examples, combining Figure 1 ,like Figure 6 As shown, the above S14 can be implemented by the following S140 and S141.
[0068] S140. According to the pre-configured structural configuration information of the directed acyclic graph, each reconstructed directed acyclic graph is filtered to determine the reconstructed directed acyclic graphs that meet the structural configuration information. S141. Filter the directed acyclic graphs (DAGs) that meet the structural configuration information according to their reconstruction scores to determine the selected DAGs.
[0069] In some examples, when filtering directed acyclic graphs (DAGs) based on their reconstruction scores that conform to the structural configuration information, the DAG corresponding to the highest reconstruction score can be used as the selected DAG.
[0070] As described above, the data processing method provided in this embodiment, when executing a task involving a theoretical directed acyclic graph (DAG), obtains the current operating parameters and the first configuration information of the theoretical DAG; reconstructs the DAG based on the first configuration information to obtain at least one reconstructed DAG; for each reconstructed DAG, a reconstruction score is determined based on the reconstructed DAG and the current operating parameters; each reconstructed DAG is filtered according to the pre-configured structural configuration information of the DAG to determine the reconstructed DAGs that conform to the structural configuration information; the reconstructed DAGs are further filtered according to their reconstruction scores to determine the selected DAGs; and the task is executed according to the selected DAGs to obtain the task result. In this way, the data processing device can automatically optimize the user-configured theoretical DAG, thereby improving the intelligence level and dynamic adaptability of the DAG.
[0071] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0072] This application embodiment can divide the data processing device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0073] like Figure 7 As shown in the diagram, an embodiment of this application provides a schematic diagram of a data processing device. It includes a communicator 101 and a controller 102.
[0074] When executing a task containing a theoretical directed acyclic graph, the current running parameters and the first configuration information of the theoretical directed acyclic graph are obtained; wherein, the first configuration information indicates at least one node contained in the theoretical directed acyclic graph, the tables contained in each node, and the table partitions of each table. Based on the first configuration information, a directed acyclic graph is reconstructed to obtain at least one reconstructed directed acyclic graph; For each reconstructed directed acyclic graph, the reconstruction score of the reconstructed directed acyclic graph is determined based on the reconstructed directed acyclic graph and the current running parameters. Based on the pre-configured structural configuration information and reconstruction score of the directed acyclic graph, each reconstructed directed acyclic graph is screened to determine the selected directed acyclic graph; among them, the selected directed acyclic graph is different from the theoretical directed acyclic graph, and the reconstruction score of the selected directed acyclic graph is greater than the reconstruction score of the theoretical directed acyclic graph. The task is executed by filtering the directed acyclic graph, and the task results are obtained.
[0075] In some implementable examples, the controller, when performing directed acyclic graph reconstruction based on the first configuration information to obtain at least one reconstructed directed acyclic graph, is further configured to: The dependencies between nodes, tables, and table partitions in the first configuration information are predicted to obtain at least one reconstructed directed acyclic graph.
[0076] In some implementable examples, the controller, when performing the refactoring score of each refactored directed acyclic graph (DAG) based on the refactored DAG and current running parameters, is further configured to: For each reconstructed directed acyclic graph, the actual total time spent reconstructing the directed acyclic graph is determined based on the current running parameters; based on the actual total time spent, the reconstruction score of the reconstructed directed acyclic graph is determined as the actual total time spent.
[0077] In some implementable examples, the controller, when performing the refactoring score of each refactored directed acyclic graph (DAG) based on the refactored DAG and current running parameters, is further configured to: For each reconstructed directed acyclic graph, the second configuration information and current running parameters of the reconstructed directed acyclic graph are input into the weighted scoring model for scoring, and the reconstruction score of the reconstructed directed acyclic graph is obtained.
[0078] In some implementable examples, the controller, when performing a screening of each reconstructed directed acyclic graph according to pre-configured directed acyclic graph structure configuration information and reconstruction scores, and determining the selected directed acyclic graph, is further configured as follows: Based on the pre-configured structural configuration information of the directed acyclic graph, each reconstructed directed acyclic graph is filtered to determine the reconstructed directed acyclic graphs that meet the structural configuration information. The directed acyclic graphs (DAGs) that meet the structural configuration information are screened based on their reconstruction scores to determine the selected DAGs.
[0079] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and their functions will not be repeated here.
[0080] Of course, the data processing device provided in this application embodiment includes, but is not limited to, the modules described above. For example, the data processing device may also include a memory 103. The memory 103 may be used to store the program code of the data processing device, and may also be used to store data generated by the data processing device during operation, such as data in write requests.
[0081] like Figure 8 As shown, this application embodiment also provides a chip system that can be applied to the data processing device in the foregoing embodiments. The chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 may be the processor in the aforementioned data processing device. The processor 1501 and the interface circuit 1502 are interconnected via a line. The processor 1501 can receive and execute computer instructions from the memory of the aforementioned data processing device through the interface circuit 1502. When the computer instructions are executed by the processor 1501, the data processing device can perform the various steps executed by the data processing device in the foregoing embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.
[0082] This application also provides a computer-readable storage medium for storing computer instructions for operating the aforementioned data processing device.
[0083] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing device, characterized in that, include: The communicator is configured to communicate with the client. The controller is configured as follows: When performing a task involving a theoretical directed acyclic graph, the current running parameters and the first configuration information of the theoretical directed acyclic graph are obtained; wherein, the first configuration information indicates at least one node contained in the theoretical directed acyclic graph, the tables contained in each node, and the table partitions of each table. Based on the first configuration information, a directed acyclic graph is reconstructed to obtain at least one reconstructed directed acyclic graph; For each of the reconstructed directed acyclic graphs, a reconstruction score for the reconstructed directed acyclic graph is determined based on the reconstructed directed acyclic graph and the current running parameters. According to the pre-configured structural configuration information of the directed acyclic graph and the reconstruction score, each of the reconstructed directed acyclic graphs is screened to determine the screened directed acyclic graphs; wherein, the screened directed acyclic graphs are different from the theoretical directed acyclic graphs, and the reconstruction score of the screened directed acyclic graphs is greater than the reconstruction score of the theoretical directed acyclic graphs; The task is executed according to the filtered directed acyclic graph to obtain the task result.
2. The data processing device according to claim 1, characterized in that, The controller, when performing directed acyclic graph reconstruction based on the first configuration information to obtain at least one reconstructed directed acyclic graph, is further configured as follows: The dependencies between nodes, tables, and table partitions in the first configuration information are predicted to obtain at least one reconstructed directed acyclic graph.
3. The data processing device according to claim 1, characterized in that, The controller, when performing the operation of determining the reconstruction score of each reconstructed directed acyclic graph based on the reconstructed directed acyclic graph and the current operating parameters, is further configured to: For each of the reconstructed directed acyclic graphs, the actual total time spent executing the reconstructed directed acyclic graph is determined based on the current running parameters; Based on the actual total time consumed, the reconstruction score of the reconstructed directed acyclic graph is determined as the actual total time consumed.
4. The data processing device according to claim 1, characterized in that, The controller, when performing the operation of determining the reconstruction score of each reconstructed directed acyclic graph based on the reconstructed directed acyclic graph and the current operating parameters, is further configured to: For each of the reconstructed directed acyclic graphs, the second configuration information of the reconstructed directed acyclic graph and the current running parameters are input into a weighted scoring model for scoring, thereby obtaining the reconstruction score of the reconstructed directed acyclic graph.
5. The data processing device according to claim 1, characterized in that, The controller, when performing the filtering of each reconstructed directed acyclic graph according to the pre-configured structure configuration information of the directed acyclic graph and the reconstruction score, and determining the selected directed acyclic graph, is further configured as follows: According to the pre-configured structural configuration information of the directed acyclic graph, each of the reconstructed directed acyclic graphs is filtered to determine the reconstructed directed acyclic graphs that conform to the structural configuration information. The directed acyclic graphs (DAGs) that are reconstructed according to the reconstruction scores that conform to the structural configuration information are selected to determine the selected DAGs.
6. A data processing method, characterized in that, include: When performing a task involving a theoretical directed acyclic graph, the current running parameters and the first configuration information of the theoretical directed acyclic graph are obtained; wherein, the first configuration information indicates at least one node contained in the theoretical directed acyclic graph, the tables contained in each node, and the table partitions of each table. Based on the first configuration information, a directed acyclic graph is reconstructed to obtain at least one reconstructed directed acyclic graph; For each of the reconstructed directed acyclic graphs, a reconstruction score for the reconstructed directed acyclic graph is determined based on the reconstructed directed acyclic graph and the current running parameters. According to the pre-configured structural configuration information of the directed acyclic graph and the reconstruction score, each of the reconstructed directed acyclic graphs is screened to determine the screened directed acyclic graphs; wherein, the screened directed acyclic graphs are different from the theoretical directed acyclic graphs, and the reconstruction score of the screened directed acyclic graphs is greater than the reconstruction score of the theoretical directed acyclic graphs; The task is executed according to the filtered directed acyclic graph to obtain the task result.
7. The data processing method according to claim 6, characterized in that, The process of reconstructing a directed acyclic graph based on the first configuration information to obtain at least one reconstructed directed acyclic graph includes: The dependencies between nodes, tables, and table partitions in the first configuration information are predicted to obtain at least one reconstructed directed acyclic graph.
8. The data processing method according to claim 6, characterized in that, For each of the reconstructed directed acyclic graphs (DAGs), based on the reconstructed DAG and the current operating parameters, a reconstruction score for the reconstructed DAG is determined, including: For each of the reconstructed directed acyclic graphs, the actual total time spent executing the reconstructed directed acyclic graph is determined based on the current running parameters; based on the actual total time spent, the reconstruction score of the reconstructed directed acyclic graph is determined as the actual total time spent.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data processing method as described in any one of claims 6-8.
10. A computer program product, characterized in that, When the computer program product is run on a computer, the computer enables the computer to implement the data processing method as described in any one of claims 6-8.