Data attribution method and device based on distributed computing architecture

Abstract

The invention discloses a data attribution method and device based on a distributed computing architecture, and the method comprises the steps: splitting a multi-dimensional attribution SQL according to preset time when the multi-dimensional attribution SQL is received, and obtaining a plurality of microtasks; for each micro-task, distributing the micro-task to an execution node; executing the microtask through the execution node to obtain an intermediate execution result; and combining all the intermediate execution results to obtain a data attribution result. The multi-dimensional attribution SQL is divided into a plurality of micro-tasks according to the preset time, the micro-tasks are distributed to the execution nodes, each micro-task independently processes data in a specific time period, the tasks are completed through the efficient calculation capacity of the execution nodes, and therefore the resource scheduling bottleneck in a traditional distributed calculation framework is avoided, and the resource scheduling efficiency is improved. Therefore, timely and accurate data analysis can be realized in a quickly changing service scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data analysis, and particularly relates to a data attribution method and device based on a distributed computing architecture. BACKGROUND

[0002] With the advancement of digital transformation, enterprises have increasingly high requirements for the real-time performance and processing capacity of data analysis. As a core tool for analyzing the reasons for fluctuations in business indicators, intelligent attribution systems help enterprises monitor and optimize marketing effectiveness, user behavior, etc. in real time.

[0003] Existing target data attribution methods mainly rely on traditional distributed architectures such as Hadoop and Spark. Although the Hadoop ecosystem can support batch processing of massive data, it is difficult to meet the demand for "minute-level" response of intelligent attribution due to its complex architecture, low resource utilization, and inability to provide real-time data analysis. Although the Spark framework improves performance through in-memory computing, its resource scheduling is still statically allocated, which makes it prone to resource waste or computing bottlenecks when facing attribution tasks with large fluctuations in computing volume. Therefore, the existing technology cannot efficiently support the high real-time performance, large-scale processing, and complex computing requirements of intelligent attribution systems, and cannot provide timely and accurate data analysis in rapidly changing business scenarios. SUMMARY

[0004] The present application provides a data attribution method and device based on a distributed computing architecture, aiming to provide timely and accurate data analysis in rapidly changing business scenarios.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] A data attribution method based on a distributed computing architecture, comprising:

[0007] When a multi-dimensional attribution SQL is received, the multi-dimensional attribution SQL is split according to a preset time to obtain a plurality of micro-tasks;

[0008] For each micro-task, the micro-task is allocated to an execution node;

[0009] The micro-task is executed by the execution node to obtain an intermediate execution result;

[0010] All intermediate execution results are merged to obtain a data attribution result.

[0011] Optionally, for each micro-task, the micro-task is allocated to an execution node, comprising:

[0012] acquiring real-time resource usage of each execution node; the real-time resource usage at least includes disk IO and CPU utilization;

[0013] analyzing all real-time resource usage to obtain an analysis result; the analysis result indicates idle state of execution node resource;

[0014] for each micro task, assigning the micro task to an execution node according to the analysis result.

[0015] Optionally, the executing the micro task by the execution node to obtain an intermediate execution result comprises:

[0016] checking whether the local disk in the execution node contains required attribution data;

[0017] if the local disk in the execution node contains the required attribution data, executing the micro task by the execution node according to the required attribution data to obtain an intermediate execution result;

[0018] if the local disk in the execution node does not contain the required attribution data, loading required converted attribution data from a preset service;

[0019] executing the micro task by the execution node using the converted attribution data to obtain an intermediate execution result.

[0020] Optionally, the method further comprises:

[0021] acquiring attribution data and business date of the attribution data;

[0022] converting data format of the attribution data to obtain converted attribution data;

[0023] partitioning the converted attribution data according to the business date of the attribution data to obtain a directory path to which the converted attribution data belongs;

[0024] storing the converted attribution data into a preset service according to the directory path.

[0025] Optionally, the loading required converted attribution data from a preset service comprises:

[0026] acquiring business date of the micro task;

[0027] filtering converted attribution data corresponding to the business date from the preset service.

[0028] A data attribution device based on a distributed computing architecture comprises:

[0029] a splitting unit, configured to split the multi-dimensional attribution SQL into a plurality of micro-tasks according to a preset time when the multi-dimensional attribution SQL is received;

[0030] an allocating unit, configured to allocate each of the micro-tasks to an execution node;

[0031] an executing unit, configured to execute the micro-tasks by the execution node to obtain intermediate execution results;

[0032] a merging unit, configured to merge all the intermediate execution results to obtain a data attribution result.

[0033] Optionally, the allocating unit is specifically configured to:

[0034] obtain real-time resource usage of each execution node; the real-time resource usage at least includes disk IO and CPU utilization;

[0035] analyze all the real-time resource usage to obtain an analysis result; the analysis result indicates an idle state of the resource of the execution node;

[0036] for each of the micro-tasks, allocate the micro-task to the execution node according to the analysis result.

[0037] Optionally, the executing unit includes:

[0038] a checking sub-unit, configured to check whether a local disk in the execution node contains required attribution data;

[0039] a first executing sub-unit, configured to execute the micro-task according to the required attribution data by the execution node to obtain an intermediate execution result if the local disk in the execution node contains the required attribution data;

[0040] a loading sub-unit, configured to load required converted attribution data from a preset service if the local disk in the execution node does not contain the required attribution data;

[0041] a second executing sub-unit, configured to execute the micro-task by the execution node using the converted attribution data to obtain an intermediate execution result.

[0042] Optionally, the method further includes:

[0043] an obtaining sub-unit, configured to obtain attribution data and a business date of the attribution data;

[0044] a converting sub-unit, configured to convert the attribution data in a data format to obtain converted attribution data;

[0045] a partition subunit, configured to partition the converted attribution data according to a business date of the attribution data, to obtain a directory path to which the converted attribution data belongs;

[0046] a storage subunit, configured to store the converted attribution data into a preset service according to the directory path.

[0047] Optionally, the loading subunit is specifically configured to:

[0048] obtain a business date of the micro task;

[0049] filter the converted attribution data corresponding to the business date from the preset service.

[0050] The technical scheme provided in the application comprises the following steps: when a multi-dimensional attribution SQL is received, the multi-dimensional attribution SQL is split according to a preset time, to obtain a plurality of micro tasks; for each micro task, the micro task is allocated to an execution node; the micro task is executed by the execution node, to obtain an intermediate execution result; all the intermediate execution results are merged, to obtain a data attribution result. By splitting the multi-dimensional attribution SQL into a plurality of micro tasks according to the preset time, and allocating the micro tasks to the execution nodes, each micro task processes data of a specific time period independently, and the high-efficiency computing capability of the execution node is utilized to complete the task, so that the resource scheduling bottleneck in the traditional distributed computing architecture is avoided, and timely and accurate data analysis can be realized in a rapidly changing business scenario. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0052] Figure 1 a flowchart of a data attribution method based on a distributed computing architecture provided by an embodiment of the present application;

[0053] Figure 2 a flowchart of an attribution data storage method provided by an embodiment of the present application;

[0054] Figure 3 a flowchart of an attribution process provided by an embodiment of the present application;

[0055] Figure 4 an architecture schematic diagram of a data attribution apparatus based on a distributed computing architecture provided by an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0057] In the present application, the term "comprising", "containing" or any other variant thereof is intended to cover the non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the sentence "including a" does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0058] As shown in Figure 1 A flowchart of a data attribution method based on a distributed computing architecture provided by an embodiment of the present application includes the following steps:

[0059] S101: When receiving a multi-dimensional attribution SQL, the multi-dimensional attribution SQL is split according to a preset time to obtain a plurality of micro-tasks.

[0060] The multi-dimensional attribution SQL refers to a standard SQL language that generates a complete multi-dimensional data set of all dimension combinations (equivalent to "Cartesian product of all possible groups") in one query. For example, the multi-dimensional attribution SQL is used to analyze the reasons for the order volume fluctuation in the past week. At this time, we need to calculate the JS divergence (Jensen-Shannon Divergence, a measure of the difference between probability distributions in information theory) of each week of all dimensions under the index in the past n weeks.

[0061] Optionally, the preset time can be 1 minute, 1 hour, 1 day, 1 week and 1 month.

[0062] It can be understood that according to the Union keyword in the multi-dimensional attribution SQL, the multi-dimensional attribution SQL is split into a plurality of independent micro-tasks according to the preset time range, and each micro-task is only responsible for pulling and calculating the data of its corresponding time period.

[0063] It should be noted that when the multi-dimensional attribution SQL is received, the multi-dimensional attribution SQL can be parsed by Ray Serve API to obtain a preset time, and then the multi-dimensional attribution SQL is split according to the preset time to obtain a plurality of micro-tasks.

[0064] S102: For each microtask, the microtask is assigned to an execution node.

[0065] Optionally, the execution node includes but is not limited to Worker.

[0066] Optionally, in another embodiment of the present application, the specific implementation of step S102 includes processes A1 to A3.

[0067] A1: Obtain the real-time resource usage of each execution node.

[0068] The real-time resource usage includes at least disk IO and CPU utilization.

[0069] It can be understood that the real-time resource usage reported by each execution node can be obtained through the Ray scheduler.

[0070] A2: Analyze all real-time resource usage to obtain an analysis result.

[0071] The analysis result indicates the idle state of the execution node resource.

[0072] Optionally, the Ray scheduler is used to analyze all real-time resource usage.

[0073] A3: For each microtask, the microtask is assigned to an execution node according to the analysis result.

[0074] It can be understood that when the execution node resource is idle, the microtask is assigned to the node, and when the execution node resource is busy, the microtask is not assigned to the node.

[0075] S103: Execute the microtask through the execution node to obtain an intermediate execution result.

[0076] The intermediate execution result indicates the result of executing the microtask by the execution node.

[0077] Optionally, in another embodiment of the present application, the specific implementation of step S103 includes processes B1 to B4.

[0078] B1: Check whether the local disk in the execution node contains the required attribution data.

[0079] If the local disk in the execution node contains the required attribution data, process B2 is executed, and if the local disk in the execution node does not contain the required attribution data, process B3 is executed.

[0080] It can be understood that checking whether the local disk in the execution node contains the required attribution data is to check whether the local disk in the execution node contains date data, each date data has corresponding attribution data, if there is date data, it indicates that the local disk in the execution node contains the required attribution data; if there is no date data, it indicates that the local disk in the execution node does not contain the required attribution data.

[0081] B2: obtaining an intermediate execution result by executing the microtask according to the required attribution data by the execution node.

[0082] Optionally, if the local disk of the execution node already stores the attribution data required by the task, the execution node directly starts the DuckDB instance. Subsequently, the DuckDB instance quickly completes the microtask allocated by the Ray scheduler by using the efficient vectorized execution engine thereof.

[0083] B3: loading the required converted attribution data from a preset service.

[0084] The preset service is a service supporting an Amazon object storage standard.

[0085] It can be understood that the execution node first obtains data from the preset service. The data is pre-converted attribution data and has been partitioned by date; each partition is an independent directory and is then completely loaded onto the local disk.

[0086] It should be noted that by loading the converted attribution data onto the local disk, it is avoided to transmit data through the network each time, thereby reducing the calculation time consumption.

[0087] Optionally, before process B3, the converted attribution data needs to be stored to the preset service first, and the subsequent application can directly load the ready data, thereby saving the calculation time consumption of real-time conversion, and thereby improving the data access speed. Therefore, an attribution data storage method is provided in the embodiments of the present application, as shown in the following table. Figure 2 The method comprises the following steps:

[0088] S201: obtaining attribution data and business dates of the attribution data.

[0089] It can be understood that the attribution data and the business dates of the attribution data are obtained from the task platform Airflow at a timing.

[0090] S202: converting the attribution data to obtain converted attribution data.

[0091] It can be understood that the attribution data is originally stored in the columnar storage database ClickHouse, and therefore it is necessary to convert the attribution data from the data format of ClickHouse to a data file in the Apache Parquet columnar storage format supported by Duckdb for high-performance computing, so as to obtain the converted attribution data.

[0092] S203: Partitioning the converted attribution data according to the business date of the attribution data to obtain a directory path to which the converted attribution data belongs.

[0093] It can be understood that the converted attribution data is partitioned according to the business date field, and the core is to distinguish data of different dates through the directory path, so as to make full use of the "pruning" function of Duckdb, directly locate the required data through the directory, and reduce a large amount of invalid disk IO.

[0094] On the other hand, RowGroup is a data organization unit in the Parquet format, representing the amount of data read by a single I / O operation, and its size directly affects the parallel efficiency and disk load of DuckDB. The configuration of RowGroup should follow a basic principle: when disk I / O is not a bottleneck, the number of RowGroup should be consistent with the number of CPU cores to fully utilize the parallel computing capability; when disk I / O becomes a system bottleneck, the number of RowGroup should be appropriately reduced to reduce the concurrent pressure of DuckDB and relieve I / O contention.

[0095] S204: Storing the converted attribution data into a preset service according to the directory path.

[0096] Optionally, in another embodiment of the present application, the specific implementation process of process B3 is: obtaining the business date of the micro task; and screening the converted attribution data corresponding to the business date from the preset service.

[0097] B4: Executing the micro task by using the converted attribution data through the execution node to obtain an intermediate execution result.

[0098] It should be noted that the specific implementation of process B4 can be correspondingly referred to process B2, which will not be described here.

[0099] S104: Merging all intermediate execution results to obtain a data attribution result.

[0100] The data attribution result includes a key dimension and a contribution value.

[0101] It can be understood that all intermediate execution results are first aggregated to the Driver node, and all intermediate execution results are merged through the Python logic in the Driver node to obtain the data attribution result.

[0102] Based on the foregoing, the embodiment of the application further constructs a distributed computing architecture based on Ray and DuckDB, and the core technical solution is as follows:

[0103] The architecture adopts a hierarchical design and is divided into an access layer, a computing layer, an infrastructure layer, and a storage layer, realizing complete separation of computing and storage. The responsibilities of each layer are clear and work in coordination, as follows:

[0104] The access layer serves as a unified service entrance and provides standardized REST API externally, and is responsible for receiving external task requests and performing unified scheduling and distribution.

[0105] The computing layer takes Ray cluster as the core and provides dynamic resource scheduling, distributed object storage, and task fault tolerance, and the like, supporting efficient execution of distributed computing tasks.

[0106] The infrastructure layer deploys KubeRay Operator based on Kubernetes, realizes automatic deployment and elastic scaling of Ray cluster, and integrates observability components to ensure stable operation of the system for 7x24 hours.

[0107] The storage layer adopts a hybrid strategy of "object storage + local storage". Raw data is stored in the preset service in Parquet format after ETL processing, and is reasonably partitioned according to business query mode; the local disk of the Worker node caches the preloaded Parquet data shards for sharing by multiple tasks on the same node, effectively reducing repeated I / O operations.

[0108] As can be seen, the technical solution has achieved remarkable results in practical application. Facing attribution analysis tasks of more than 900 million data, the time consumption of a single query is stably reduced from more than 40 seconds to less than 15 seconds, and the computing efficiency is improved by more than 160%, successfully realizing the core demand of "minute-level" response on the business side. Through a strict resource isolation mechanism, the attribution calculation task has been completely separated from the ClickHouse core cluster, completely eliminating the resource interference on key businesses such as real-time monitoring and report query. At the same time, the elastic architecture based on Ray supports second-level scaling, which can quickly expand the Worker node during business peak and timely release resources during the trough, greatly improving the hardware utilization rate and optimizing the cost structure. In addition, the architecture has good scalability, lays a solid foundation for future seamless integration of distributed machine learning capabilities, realizes the deep integration of attribution analysis and model training and real-time inference, and provides strong support for the continuous evolution of the intelligent attribution system.

[0109] To better illustrate the above, see Figure 3 An attribution flowchart is shown. When receiving an attribution request sent by a user, the attribution system obtains a multi-dimensional attribution SQL from the attribution request and sends it to a Ray cluster. The Ray cluster splits the multi-dimensional attribution SQL according to a preset time to obtain a plurality of microtasks. For each microtask, the microtask is assigned to an execution node. It is checked whether the local disk in the execution node contains the required attribution data. If the local disk in the execution node contains the required attribution data, the microtask is executed by the execution node according to the required attribution data to obtain an intermediate execution result. If the local disk in the execution node does not contain the required attribution data, the required converted attribution data is loaded from a preset service (i.e., S3 storage in the figure). The microtask is executed by the execution node using the converted attribution data to obtain an intermediate execution result. All intermediate execution results are merged to obtain a data attribution result.

[0110] As can be seen, through the above implementation, through the design architecture of "storage-computing separation" and "dynamic computing resource scheduling", the pain points of Hadoop ecology "complex architecture, low resource utilization (<50%), poor real-time performance" and Spark "static resource scheduling, heavy architecture, high integration of machine learning threshold" are solved. Through the way of complete resource isolation, the heavy computing task is separated from the ClickHouse core cluster, avoiding the occupation of other core business query services. At the same time, aiming at the pain points of ClickHouse computing storage coupling and high cost of single node expansion, the new architecture storage is independently expanded based on the preset service, without relying on the computing node. The computing layer can realize the expansion and contraction of nodes at the second level through Kubernetes (container orchestration system), which is more flexible. And it supports releasing nodes in the trough, further saving hardware resource cost.

[0111] In summary, by splitting the multi-dimensional attribution SQL into a plurality of microtasks according to a preset time, and assigning these microtasks to execution nodes, each microtask independently processes data of a specific time period, and the efficient computing power of the execution node is used to complete the task, thereby avoiding the resource scheduling bottleneck in the traditional distributed computing architecture, so that timely and accurate data analysis can be realized in rapidly changing business scenarios.

[0112] As Figure 4 shown, an architecture schematic diagram of a data attribution apparatus based on a distributed computing architecture provided by an embodiment of the present application, the data attribution apparatus comprising: a splitting unit 100, an assignment unit 200, an execution unit 300, and a merging unit 400.

[0113] The splitting unit 100 is configured to split the multi-dimensional attribution SQL into a plurality of micro-tasks according to a preset time when the multi-dimensional attribution SQL is received.

[0114] The distribution unit 200 is configured to distribute each of the micro-tasks to an execution node.

[0115] The distribution unit 200 is specifically configured to: acquire real-time resource usage of each execution node; the real-time resource usage at least includes disk IO and CPU utilization; analyze all real-time resource usage to obtain an analysis result; the analysis result indicates an idle state of the resource of the execution node; and distribute each of the micro-tasks to the execution node according to the analysis result.

[0116] The execution unit 300 is configured to execute the micro-tasks by the execution node to obtain intermediate execution results.

[0117] The execution unit 300 includes:

[0118] The checking sub-unit is configured to check whether the local disk in the execution node contains required attribution data.

[0119] The first execution sub-unit is configured to execute the micro-tasks by the execution node according to the required attribution data to obtain intermediate execution results, if the local disk in the execution node contains the required attribution data.

[0120] The loading sub-unit is configured to load required converted attribution data from a preset service, if the local disk in the execution node does not contain the required attribution data.

[0121] The loading sub-unit is specifically configured to: acquire a business date of the micro-task; and filter out converted attribution data corresponding to the business date from the preset service.

[0122] The second execution sub-unit is configured to execute the micro-tasks by the execution node using the converted attribution data to obtain intermediate execution results.

[0123] The merging unit 400 is configured to merge all the intermediate execution results to obtain data attribution results.

[0124] In summary, by splitting the multi-dimensional attribution SQL into a plurality of micro-tasks according to a preset time, and distributing the micro-tasks to execution nodes, each micro-task independently processes data of a specific time period, and the high-efficiency computing capability of the execution node is utilized to complete the task, thereby avoiding the resource scheduling bottleneck in the traditional distributed computing architecture, and thus enabling timely and accurate data analysis in rapidly changing business scenarios.

[0125] In combinationFigure 3 The data attribution apparatus shown in the content further comprises:

[0126] An acquisition subunit is configured to acquire attribution data and a business date of the attribution data.

[0127] A conversion subunit is configured to perform data format conversion on the attribution data to obtain converted attribution data.

[0128] A partition subunit is configured to partition the converted attribution data according to the business date of the attribution data to obtain a directory path to which the converted attribution data belongs.

[0129] A storage subunit is configured to store the converted attribution data into a preset service according to the directory path.

[0130] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, the system or system embodiments are described more simply because they are basically similar to the method embodiments. The relevant parts can be referred to the part of the method embodiment. The above-described system and system embodiments are only illustrative, and the units described as separate components can be or can not be physically separated, and the components shown as units can be or can not be physical units, that is, they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to the actual needs. Those skilled in the art can understand and implement without creative labor.

[0131] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0132] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data attribution method based on a distributed computing architecture, characterized in that, include: When a multidimensional attribution SQL is received, the multidimensional attribution SQL is split into multiple micro-tasks according to a preset time. For each of the micro-tasks, the micro-task is assigned to an execution node; The micro-tasks are executed through the execution nodes to obtain intermediate execution results; All intermediate execution results are merged to obtain the data attribution results.

2. The method according to claim 1, characterized in that, The step of assigning each microtask to an execution node includes: Obtain the real-time resource utilization rate of each execution node; the real-time resource utilization rate includes at least disk I / O and CPU utilization. The utilization rate of all real-time resources is analyzed to obtain the analysis results; the analysis results indicate the idle status of the execution node resources. For each microtask, the microtask is assigned to an execution node based on the analysis results.

3. The method according to claim 1, characterized in that, The process of executing the micro-task through the execution node to obtain intermediate execution results includes: Check whether the local disk in the execution node contains the required attribution data; If the local disk in the execution node contains the required attribution data, then the microtask is executed by the execution node based on the required attribution data to obtain intermediate execution results; If the local disk in the execution node does not contain the required attribution data, then the required transformed attribution data is loaded from the preset service; The micro-task is executed by the execution node using the transformed attribution data to obtain intermediate execution results.

4. The method according to claim 3, characterized in that, Also includes: Obtain attribution data and the business date of the attribution data; The attribution data is converted into a new data format to obtain the converted attribution data. The transformed attribution data is partitioned according to the business date of the attribution data to obtain the directory path to which the transformed attribution data belongs; According to the directory path, the converted attribution data is stored in the preset service.

5. The method according to claim 3, characterized in that, The loading of the required transformed attribution data from the preset service includes: Obtain the business date of the micro-task; The converted attribution data corresponding to the business date is filtered from the preset services.

6. A data attribution device based on a distributed computing architecture, characterized in that, include: The splitting unit is used to split the multidimensional attribution SQL according to a preset time when it receives the multidimensional attribution SQL, so as to obtain multiple micro-tasks; An allocation unit is used to allocate each microtask to an execution node. An execution unit is used to execute the micro-task through the execution node to obtain intermediate execution results; The merging unit is used to merge all intermediate execution results to obtain the data attribution result.

7. The apparatus according to claim 6, characterized in that, The allocation unit is specifically used for: Obtain the real-time resource utilization rate of each execution node; the real-time resource utilization rate includes at least disk I / O and CPU utilization. The analysis results are obtained by analyzing the utilization of all real-time resources. The analysis results indicate the idle status of the execution node resources; For each microtask, the microtask is assigned to an execution node based on the analysis results.

8. The apparatus according to claim 6, characterized in that, The execution unit includes: The inspection subunit is used to check whether the local disk in the execution node contains the required attribution data; The first execution subunit is configured to execute the microtask based on the required attribution data through the execution node if the local disk in the execution node contains the required attribution data, thereby obtaining an intermediate execution result; A loading subunit is used to load the required transformed attribution data from a preset service if the local disk in the execution node does not contain the required attribution data. The second execution subunit is used to execute the micro-task through the execution node using the transformed attribution data to obtain intermediate execution results.

9. The apparatus according to claim 8, characterized in that, Also includes: The acquisition sub-unit is used to acquire attribution data and the business date of the attribution data; The conversion subunit is used to convert the attribution data into a data format to obtain the converted attribution data. The partitioning subunit is used to partition the transformed attribution data according to the business date of the attribution data, and obtain the directory path to which the transformed attribution data belongs; The storage subunit is used to store the converted attribution data in a preset service according to the directory path.

10. The apparatus according to claim 8, characterized in that, The loading subunit is specifically used for: Obtain the business date of the micro-task; The converted attribution data corresponding to the business date is filtered from the preset services.

Images