A data partitioning computing method, device and program product

Abstract

The present application proposes a data divide-and-conquer computing method, device, and program product, which confirms the data volume of the target computing task corresponding to the computing request. Determine the splitting granularity of the target computing task according to the data volume. Split the target computing task into at least two sub-computing tasks according to the splitting granularity. And use the sub-computing tasks to separately obtain their respective sub-computing results to fully utilize the computing performance of the computing system, and then complete the reorganization of the sub-computing tasks through linked computing. While utilizing the computing performance, the accuracy of the calculation is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of data computing, and in particular, to a data divide-and-conquer computing method, device, and program product. Background Art

[0002] In the field of multi-dimensional time series data analysis, it is necessary to slice, dice, aggregate, and analyze data in multiple dimensions, such as time, geography, and product.

[0003] However, in the calculation process, taking the financial scenario as an example, when analyzing a large amount or even massive amount of historical positions and transaction details, it often involves data in the time dimension and different product and asset level dimensions. These data in different dimensions are often interdependent, which leads to the need to load a large amount of data into the computing system at the same time for multi-dimensional cube calculations.

[0004] At this time, if the computing system is a stand-alone system, loading a large amount of data may lead to stability risks such as Out of Memory (OOM); if the computing system is a distributed system, the calculation of the above data cannot be broken down in the time dimension and asset dimension, which makes it impossible to fully utilize the computing power of the distributed system to improve computing performance. Summary of the invention

[0005] The purpose of this application is to provide a data divide-and-conquer computing method, device and program product, which are used to improve the full and stable use of the computing performance of a computing device while ensuring the accurate completion of computing tasks.

[0006] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a data divide-and-conquer computing method, including:

[0008] Confirm the data volume of the target computing task corresponding to the computing request;

[0009] Determine the splitting granularity of the target computing task according to the data volume;

[0010] Splitting the target computing task into at least two sub-computing tasks according to the splitting granularity;

[0011] Calculate the sub-computation tasks respectively to obtain sub-computation results corresponding to the sub-computation tasks;

[0012] Calculating the final cumulative link values ​​of the sub-computation tasks respectively, wherein the final cumulative link values ​​represent the continuous relationship between different sub-computation results and are used for the breakpoint continuation of each of the sub-computation results after they are split;

[0013] A link calculation is performed according to the end-of-period cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

[0014] Optionally, respectively calculating the final cumulative link values ​​of the sub-computing tasks includes:

[0015] Calculating the inter-period factor of each of the sub-computing tasks, and determining the end-of-period link value corresponding to the sub-computing task according to the inter-period factor;

[0016] The end-of-period cumulative link value is obtained according to each end-of-period link value.

[0017] Optionally, the method is applied to a computing system, the computing system comprising a master node and at least two working nodes; the step of confirming the data volume of the target computing task corresponding to the computing request comprises:

[0018] The master node receives the computing request;

[0019] The master node confirms the data volume of the target computing task;

[0020] The step of determining the splitting granularity of the target computing task according to the data volume includes:

[0021] The master node determines the splitting granularity of the target computing task according to the data volume;

[0022] The step of splitting the target computing task into at least two sub-computing tasks according to the split granularity includes:

[0023] The master node splits the target computing task into at least two sub-computing tasks according to the split granularity;

[0024] The master node assigns each of the sub-computing tasks to the corresponding working nodes respectively;

[0025] The steps of respectively calculating the sub-computing tasks and obtaining sub-computing results corresponding to the sub-computing tasks include:

[0026] Each of the working nodes calculates the sub-computation results of the sub-computation tasks respectively through a multi-dimensional time series data accumulation algorithm;

[0027] The step of respectively calculating the final cumulative link values ​​of the sub-computing tasks comprises:

[0028] The master node obtains corresponding sub-computation results from each working node respectively;

[0029] The master node calculates the final cumulative link value of each of the sub-computing tasks;

[0030] The master node performs link calculation according to the final cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

[0031] Optionally, the step of confirming the data volume of the target computing task corresponding to the computing request includes:

[0032] Analyze the data section of the target computing task to obtain computing parameters; the computing parameters are used to characterize the data volume and time length of the target computing task;

[0033] The step of determining the splitting granularity of the target computing task according to the data volume includes:

[0034] Determine whether the target computing task needs to be split according to the computing parameters;

[0035] If necessary, determine the date split granularity based on the data volume and memory parameters;

[0036] The step of splitting the target computing task into at least two sub-computing tasks according to the split granularity includes:

[0037] The target computing task is split into at least two sub-computing tasks according to the date splitting granularity, and each of the sub-computing tasks has corresponding date sequence information.

[0038] Optionally, the step of respectively calculating the sub-computing tasks and obtaining sub-computing results corresponding to the sub-computing tasks includes:

[0039] Calculate the corresponding calculation indicators of the sub-computing tasks respectively through a multi-dimensional time series data accumulation algorithm;

[0040] Calculating the inter-period factor of the sub-computing task, and determining the end-of-period link value corresponding to the sub-computing task according to the inter-period factor;

[0041] The calculation index and the end-of-period link value are used as the sub-calculation result;

[0042] The step of performing link calculation according to the end-of-period cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request includes:

[0043] Calculate the end-of-period cumulative link value based on each of the end-of-period link values;

[0044] The calculation indexes of each of the sub-computing tasks are linked to obtain the cumulative calculation result according to the final cumulative link value.

[0045] In a second aspect, an embodiment of the present application provides a data divide-and-conquer computing device, including:

[0046] A splitting module is used to confirm the data volume of the target computing task corresponding to the computing request; determine the splitting granularity of the target computing task according to the data volume; and split the target computing task into at least two sub-computing tasks according to the splitting granularity;

[0047] A task calculation module, used to calculate the sub-computing tasks respectively and obtain the sub-computing results corresponding to the sub-computing tasks;

[0048] The link calculation module is used to calculate the final cumulative link values ​​of the sub-calculation tasks respectively, wherein the final cumulative link values ​​represent the continuous relationship between different sub-calculation results and are used to connect the breakpoints of each sub-calculation result after splitting; link calculation is performed based on the final cumulative link values ​​and the corresponding sub-calculation results to obtain the cumulative calculation result corresponding to the calculation request.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0050] A memory for storing one or more programs;

[0051] processor;

[0052] When the one or more programs are executed by the processor, the method as described in any one of the first aspects above is implemented.

[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a method as described in any one of the above-mentioned first aspects.

[0054] In a fifth aspect, an embodiment of the present application provides a program product, which, when executed by a processor, implements a method as described in any one of the above-mentioned first aspects.

[0055] Compared with the prior art, the data divide-and-conquer computing method, device, and program product provided in the embodiments of the present application confirm the data volume of the target computing task corresponding to the computing request. Determine the splitting granularity of the target computing task according to the data volume. Split the target computing task into at least two sub-computing tasks according to the splitting granularity. And use the sub-computing tasks to separately obtain their respective sub-computing results to fully utilize the computing performance of the computing system, and then complete the reorganization of the sub-computing tasks through linked computing. While utilizing the computing performance, the accuracy of the calculation is guaranteed.

[0056] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 This is a schematic diagram of the main calculation process and data dimensions of the portfolio attribution model indicators;

[0059] Figure 2 A schematic diagram of the architecture of a data divide-and-conquer computing method provided by an embodiment of the present invention;

[0060] Figure 3 A flowchart of a data divide-and-conquer computing method provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of the architecture of a computing system provided by an embodiment of the present invention;

[0062] Figure 5 A schematic diagram of a processing architecture of a distributed computing system provided by an embodiment of the present invention;

[0063] Figure 6 A schematic diagram of a processing architecture for single-machine deployment provided by an embodiment of the present invention;

[0064] Figure 7 A schematic diagram of a flow chart of another data divide-and-conquer computing method provided by an embodiment of the present invention;

[0065] Figure 8 A schematic diagram of a flow chart of another data divide-and-conquer computing method provided by an embodiment of the present invention;

[0066] Fig. 9 A schematic diagram of a core solution for divide-and-conquer computing provided by an embodiment of the present invention;

[0067] Fig.10 A schematic diagram of the structure of a data divide-and-conquer computing device provided by an embodiment of the present invention;

[0068] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0070] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0071] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0072] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0073] For multi-dimensional time series data analysis, this application will exemplify the business scenario of attribution analysis of investment portfolios. However, the scenarios to which the technical solution of this application can be applied are not limited to this scenario. Based on the examples described below in this application, those skilled in the art can combine the cumulative calculation of time series data and multi-dimensional cube calculation without any creative work, design appropriate divide-and-conquer calculation units for time series data, derive the corresponding divide-and-conquer link algorithm, and apply the following solution of this application.

[0074] In the prior art, when conducting company-level or department-level analysis on financial data, it is necessary to analyze a large amount or even massive amount of data holdings and transaction detail fact data, and it is necessary to calculate the asset dimension cubic data in real time based on the detail data.

[0075] Due to the continuity of time-accumulation dimension calculation and the interdependence of asset-level dimension calculation, a large amount of data needs to be loaded into memory at the same time for multi-dimensional cube calculation. Figure 1 This is the main calculation process of the portfolio attribution model indicator in the existing technology. Figure 1 From the calculation process on the left side (a), we can see that for a calculation system, it first loads the business details data that need to be calculated. Then, the portfolio inter-period factor α, as well as the asset level dimension weights and income items are calculated respectively; then the single-period asset level dimension attribution index is calculated. Then, the portfolio inter-period factor α is integrated to calculate the multi-period asset level dimension attribution index.

[0076] Among them, the portfolio inter-period factor α is calculated as follows:

[0077]

[0078] t is the date variable, R is the portfolio return rate, from this formula and Figure 1 As can be seen from the right side (b), within the calculation date interval, the inter-period factor α on day t is obtained by multiplying the portfolio returns of all dates before day t by one. The calculation of the time accumulation dimension is continuous, so the existing method cannot be disassembled in the time dimension. k is the date variable in the cumulative formula, which describes the individual state of each day in the time period. The difference from t is that t is the cumulative state at a specific time point. The formula is expanded and k is visualized as .

[0079] Continue to see Figure 1 , where C is the attributable contribution, and its calculation formula is as follows:

[0080]

[0081] j is the asset dimension variable (including details), w is the weight, and r is the attributed income item. From this formula and Figure 1 It can be seen from the right side (b) that the attribution contribution calculation of each dimensional layer in the formula (such as the investment portfolio P, stock dimension layer S,j, industry dimension layer I,j, individual security detail layer j, etc. in the right side (b)) depends on α at the investment portfolio layer, while R in the α calculation is calculated from the detail dimension data. The asset level dimensions are interdependent, so the existing methods cannot be disassembled at the asset dimension.

[0082] It should be noted that the investment portfolio described in this application generally refers to a complete asset portfolio. Figure 1 The asset hierarchy shown is used to describe the multi-level relationships that result from categorizing and grouping multiple assets according to specific dimensions.

[0083] Therefore, based on the dependency of data in the time accumulation dimension and the asset level dimension in the above calculation process, in the deployment of existing technology computing systems, for single-machine system deployment scenarios, a large amount of data is loaded into the memory in real time for calculation, and the investment management analysis system has stability risks such as memory overflow OOM; and for distributed system deployment scenarios, since the existing calculation methods cannot be disassembled in the time dimension and asset dimension, it is impossible to divide and conquer and use distributed computing to improve performance, and the investment management analysis system has performance bottlenecks.

[0084] In view of the above existing technologies facing the computational problems of dependencies in the time accumulation dimension and the hierarchical dimension. In order to solve the impact of the computational process on the computational performance, this application introduces a divide-and-conquer link computation mechanism. Specifically, Figure 2 A schematic diagram of the architecture of a data partitioning computing method provided by an embodiment of the present invention is shown in FIG. Figure 2 Continuing with the above portfolio attribution analysis example, we split the business detail data for a period of time into tasks, and keep the asset dimension calculation independent and complete within the split sub-calculation tasks, which can be reused Figure 1 The calculation process is shown in Figure 1. After each sub-calculation task is completed, the results of each period are linked and calculated to obtain the final calculation result.

[0085] Optionally, because portfolio attribution analysis generally uses attribution models (such as the Brinson attribution model, the Campisi attribution model, etc.), the present application can directly reuse the original attribution model calculation of the system to reduce the intrusion of the calculation process as much as possible and improve the reusability of the system. Custom annotations and aspect-oriented programming (AOP) technology can be used to implement surround processing of reused processes. In addition, in order to achieve task splitting, the map-reduce model (Map-Reduce) can also be used, in which the mapping stage is used to achieve splitting, and the reduction stage is used to complete the link calculation.

[0086] Furthermore, combined with Figure 2 The architecture of the data divide-and-conquer computing method shown in Figure 1 is as follows: Figure 3 A flowchart of a data partitioning computing method provided by an embodiment of the present invention is shown in FIG. Figure 3 , the method comprising:

[0087] Step 100: confirm the data volume of the target computing task corresponding to the computing request.

[0088] Step 101: Determine the splitting granularity of the target computing task according to the data volume.

[0089] Step 102: Split the target computing task into at least two sub-computing tasks according to the splitting granularity.

[0090] Step 103: Calculate the sub-computation tasks respectively and obtain the sub-computation results corresponding to the sub-computation tasks.

[0091] Step 104: Calculate the final cumulative link values ​​of the sub-computation tasks respectively.

[0092] The final cumulative link value represents the continuous relationship between different sub-calculation results and is used to connect the breakpoints of each sub-calculation result after it is split.

[0093] Step 105: Perform link calculation based on the final cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

[0094] The data divide-and-conquer computing method provided by the embodiment of the present invention confirms the data volume of the target computing task corresponding to the computing request. The splitting granularity of the target computing task is determined according to the data volume. The target computing task is split into at least two sub-computing tasks according to the splitting granularity. The sub-computing tasks are used to separately obtain their respective sub-computing results to fully utilize the computing performance of the computing system, and then the reorganization of the sub-computing tasks is completed through link computing. While utilizing the computing performance, the accuracy of the calculation is guaranteed.

[0095] In order to implement the above process, a possible implementation method of a computing system is provided below. Specifically, Figure 4 A schematic diagram of the architecture of a computing system provided by an embodiment of the present invention, see Figure 4 The computing system is a distributed system, including: master node 1, master node 2, worker node 1, worker node 2 and worker node 3.

[0096] Among them, the master node 1 and the working node 1, working node 2 and working node 3; the master node 2 and the working node 1, working node 2 and working node 3 can all constitute a distributed computing system that can independently complete the target computing task calculation.

[0097] Among them, the master node (Master) is used to receive computing requests, split the target computing tasks, and link calculations.

[0098] Worker nodes are used to calculate sub-computing tasks of a single period.

[0099] When a master node receives a computing request, it can configure a number of working nodes that matches the computing request based on the current computing performance and load conditions of the working nodes that it can schedule.

[0100] Of course, for the working node, under the premise that its computing performance meets the requirements, it can execute the sub-computing tasks split under different target computing tasks respectively; for example, for the working node 2, it can simultaneously execute the sub-computing task A1 assigned by the master node 1 and the sub-computing task B1 assigned by the master node 2.

[0101] Below Figure 4 Take the master node 1 and worker nodes 1, 2, and 3 as examples. Figure 3 Specifically, Figure 5 A schematic diagram of a processing architecture of a distributed computing system provided by an embodiment of the present invention, see Figure 5 :

[0102] First, the master node 1 receives a computing request; then the master node 1 confirms the data volume of the target computing task; and determines the splitting granularity of the target computing task according to the data volume.

[0103] Then, the master node 1 splits the target computing task into three sub-computing tasks according to the splitting granularity. The master node 1 allocates one sub-computing task to each of the worker nodes 1, 2 and 3.

[0104] Worker node 1, worker node 2, and worker node 3 respectively calculate the sub-computation results of the sub-computation tasks through a multi-dimensional time series data accumulation algorithm.

[0105] Furthermore, the master node 1 obtains the corresponding sub-computation results from each working node respectively; the master node 1 calculates the final cumulative link value of each sub-computation task; the master node 1 performs link calculation based on the final cumulative link value and the corresponding sub-computation result to obtain the cumulative calculation result corresponding to the calculation request.

[0106] It should be noted that each working node can undertake one or more sub-computing tasks based on its own computing power and memory resources, which is not limited here.

[0107] Optional, for Figure 5 The distributed computing system shown in the figure can adopt a parallel strategy for the configuration and calculation of sub-computing tasks, that is, the master node allocates each sub-computing task in parallel to the corresponding working node for the corresponding calculation. This implementation method can effectively improve the computing performance for resource-rich scenarios. It has been verified that its implementation method can improve the computing performance by 50% compared with the existing technology.

[0108] In another implementation, the above-mentioned distributed computing system may not be used. Instead, the above-mentioned processing is performed based on a single device. Specifically, Figure 6 A schematic diagram of a single-machine deployment processing architecture provided by an embodiment of the present invention, see Figure 6 :

[0109] For a single-machine deployed device, all subtask calculations are completed within a single device. When the amount of data is limited in the system deployment resources, the single-machine serial mode can be used to achieve a stable and reliable system in large-scale data calculation scenarios, avoiding insufficient memory and causing system unavailability. After testing and verification, for the same single-machine device, the existing calculation method will have memory overflow (OOM) in the test setting scenario, while the present application solution can stably complete the calculation.

[0110] The following is a possible implementation method for splitting the target computing tasks. Figure 3 On the basis of Figure 7 A flowchart of another data partitioning computing method provided by an embodiment of the present invention is shown in FIG. Figure 7 , step 100, specifically:

[0111] Step 100 - 1 , analyze the data slice of the target computing task to obtain computing parameters.

[0112] The calculation parameters are used to characterize the data volume and time length of the target calculation task. Specifically, using the AOP technology described above, a data aspect is inserted into the calculation process of the target calculation task to obtain parsed and calculated parameters.

[0113] Optionally, in order to reduce the impact on process intrusion, you can use custom annotations to encapsulate the data aspect obtained by AOP technology into a simple code identifier (e.g. @SplitCalculation) for direct use in multiple locations in the system. For example, you can use custom annotations to mark specific pointcuts in the original calculation process so that AOP technology can use these pointcuts to insert the above data aspect.

[0114] Step 101 is specifically as follows:

[0115] Step 101 - 1 : Determine whether the target computing task needs to be split according to the computing parameters.

[0116] If splitting is required, step 101 - 2 is executed; if not required, the target computing task is calculated without splitting.

[0117] Step 101-2: Determine the date splitting granularity according to the data volume and memory parameters.

[0118] The date splitting granularity may be divided based on the time granularity of the business scenario, such as splitting by year, splitting by month, or splitting by day.

[0119] In a possible implementation, the data volume can be: single-day data volume (Scale); memory parameters can be: metadata (minimum computing unit) memory usage (Meta), worker node heap memory size (Heap), etc. Then, the single-day data volume, metadata memory usage, and worker node heap memory size are used to calculate the appropriate date split granularity to ensure that the memory used to execute the sub-computing tasks is reliable and available. Specifically, the calculation formula for the date split granularity (Period) is as follows:

[0120]

[0121] Among them, 0.618 is the golden section constant (0.618 is the initial value in the execution process, which can be adjusted according to the actual scenario later and is not limited in this application), which is used to set the Heap threshold (to prevent the heap memory from being full).

[0122] It should be noted that: for the setting of Heap, in a possible implementation scenario, it is necessary to agree that the memory configuration of different working nodes is consistent or in the same resource range. If different working nodes are in the same cluster, the Heap threshold can be consistent; if different working nodes are in different clusters, then for this application solution, for a computing request, the sub-computing tasks can be assigned to the working nodes in the same cluster. For a single-machine deployment, the Heap value executes the memory configuration of the computing device.

[0123] Step 102 is specifically:

[0124] Step 102 - 1 , split the target computing task into at least two sub-computing tasks according to the date splitting granularity, and each sub-computing task has corresponding date sequence information.

[0125] Specifically, the date sequence is used to mark sub-computation tasks, and subsequent linked calculations can be performed based on the date sequence information.

[0126] The following is a possible implementation method for link calculation. Figure 3 On the basis of Figure 8 A flowchart of another data partitioning calculation method provided by an embodiment of the present invention is shown in FIG. Figure 8 , step 103, specifically:

[0127] Step 103 - 1 , respectively calculate the corresponding sub-computation results of the sub-computation tasks through the multi-dimensional time series data accumulation algorithm.

[0128] Step 103-2: Calculate the inter-period factor of the sub-computing task, and determine the end-of-period link value corresponding to the sub-computing task according to the inter-period factor.

[0129] Specifically, as explained in the previous example, when the multi-dimensional time series data accumulation algorithm is specifically an attribution model calculation, it can be seen from the α calculation formula above that due to the continuity of the cumulative multiplication, the above-mentioned end-of-period link value can be the end-of-period link multiplier. When each period is calculated independently, a breakpoint is generated at the beginning of the period relative to the overall time period. The sub-calculation task needs to calculate and return an end-of-period link multiplier for the next period in each period, which is used to connect the breakpoints after the multi-dimensional time series data is split. The calculation formula for the end-of-period link multiplier is:

[0130]

[0131] Where t is the time variable, Used to describe the single-day rate of return of each subtask at the end of the period after splitting. The formula is described in Chinese as: End-of-period link multiplier = End-of-period inter-period factor * (1 + end-of-period daily rate of return).

[0132] Continuing with the example of cumulative continuity involving attribution model calculation, optionally, since the end-of-period link multiplier acts on the sub-calculation results of the next period, each sub-calculation result must also return the corresponding date sequence information when returning it, which is used to sort and match the respective end-of-period link multipliers during link calculation. Under this setting, the end-of-period link multiplier acting on the first period is constant at 1.

[0133] Step 104 is specifically as follows:

[0134] Step 104-1, calculate the end-of-period cumulative link value based on each end-of-period link value.

[0135] Specifically, the final cumulative link multiplier (final cumulative link value) of each sub-computing task is calculated by multiplying the final link multipliers (final link values) of all sub-computing tasks before the current sub-computing task. The calculation formula is:

[0136]

[0137] Where acc represents the cumulative concept.

[0138] Step 104-2: perform link calculation on each sub-calculation result according to the final cumulative link value to obtain a cumulative calculation result.

[0139] Specifically, continuing to take the cumulative multiplication continuity involving attribution model calculation as an example, step 104-2 represents the chained calculation of multiple sub-computation tasks: the cumulative calculation results of multiple sub-computation tasks (that is, the attribution model results) are chained and calculated using the following formula:

[0140]

[0141] Among them, the attribution model calculation result = SUM (each sub-calculation result * the cumulative link multiplier at the end of the period), where IR is the sub-calculation result, p is the number of split sub-calculation tasks, cur represents the current sub-calculation task concept, and link represents the link concept.

[0142] The following provides a specific example of formula derivation for the above multi-period link calculation (taking a single sub-calculation task as an example).

[0143] In this calculation method, the cumulative inter-period factor is equivalent to the inter-period factor in the conventional calculation mode, and the calculation formula can be derived as follows:

[0144] (Formula 1)

[0145] The Chinese description of Formula 1 is: Cumulative inter-period factor = current inter-period factor * cumulative link multiplier at the end of the period.

[0146] The calculation formula for the cumulative result of each sub-computation task can be derived as follows:

[0147] (Formula 2)

[0148] The Chinese description of Formula 2 is: The cumulative calculation result of the current sub-calculation task = the sub-calculation result of the current sub-calculation task * the current inter-period factor.

[0149] Substitute formula 1 into the following formula and combine it with formula 2 to get:

[0150]

[0151] Specifically, for calculations involving attribution models, Fig. 9 A schematic diagram of the core solution of divide-and-conquer computing provided by an embodiment of the present invention is shown in FIG. Fig. 9 , which demonstrates the divide-and-conquer approach of this solution: the target computing task is split into multiple periods of sub-computing tasks, and each period of sub-computing tasks independently performs attribution calculations. At the same time, the end-of-period link multiplier of each period is used to calculate the end-of-period cumulative link multiplier, and finally the link calculations of multiple periods are completed. Therefore, in the process of "independent attribution calculation for each period", parallel deployment or serial deployment can be used to fully utilize computing performance or alleviate computing memory pressure.

[0152] Optionally, for the implementation of this solution, since the calculation process will generate intermediate indicator data such as "end-of-period link multiplier" and "date sequence information" that have no output requirements for the calculation request, the "end-of-period link multiplier" and "date sequence information" can be removed from the data structure before the attribution model calculation results are finally displayed.

[0153] The present application also provides a data partitioning computing device to implement each step in the above example to achieve the corresponding technical effect. Specifically, Fig.10 A schematic diagram of a data partitioning computing device provided by an embodiment of the present invention is shown in FIG. Fig.10 The device 20 includes: a splitting module 200, a task calculation module 201 and a link calculation module 202.

[0154] The splitting module 200 is used to confirm the data volume of the target computing task corresponding to the computing request; determine the splitting granularity of the target computing task according to the data volume; and split the target computing task into at least two sub-computing tasks according to the splitting granularity.

[0155] The task calculation module 201 is used to calculate the sub-computing tasks respectively and obtain the sub-computing results corresponding to the sub-computing tasks.

[0156] The link calculation module 202 is used to calculate the final cumulative link value of each sub-calculation task. The final cumulative link value represents the continuous relationship between different sub-calculation results and is used to connect the breakpoints of each sub-calculation result after it is split. Link calculation is performed based on the final cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

[0157] Specifically, the splitting module 200 can execute each step related to the splitting of the target computing task in the above example; the task computing module 201 can execute each step related to the sub-computing task calculation in the above example; and the link computing module 202 can execute each step related to the link calculation in the above example.

[0158] The embodiment of the invention further provides an electronic device, which can execute all the steps of the above examples of the embodiment of the invention to achieve the corresponding technical effects. Specifically, Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, see Fig.11 , the electronic device 50, comprises: a memory 501, a processor 500;

[0159] Memory 501, used to store one or more programs;

[0160] Processor 500;

[0161] When one or more programs are executed by the processor, when the electronic device 50 is used to execute the steps shown in the above-mentioned method examples, it can achieve each step and the corresponding technical effects.

[0162] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0164] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a program product, which is stored in a computer-readable storage medium and includes a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0165] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0166] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A data divide-and-conquer computing method, characterized in that: include: Confirm the data volume of the target computing task corresponding to the computing request; Determine the splitting granularity of the target computing task according to the data volume; Splitting the target computing task into at least two sub-computing tasks according to the splitting granularity; Calculate the sub-computation tasks respectively to obtain sub-computation results corresponding to the sub-computation tasks; Calculating the inter-period factor of each of the sub-computing tasks, and determining the end-of-period link value corresponding to the sub-computing task according to the inter-period factor; the end-of-period link value is used for the breakpoint continuation after the multi-dimensional time series data is split; Obtaining a period-end cumulative link value according to each of the period-end link values, wherein the period-end cumulative link value represents a continuous relationship between different sub-calculation results and is used for breakpoint continuation after each of the sub-calculation results is split; A link calculation is performed according to the end-of-period cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

2. The method according to claim 1, characterized in that The method is applied to a computing system, wherein the computing system includes a master node and at least two working nodes; the step of confirming the data volume of the target computing task corresponding to the computing request includes: The master node receives the computing request; The master node confirms the data volume of the target computing task; The step of determining the splitting granularity of the target computing task according to the data volume includes: The master node determines the splitting granularity of the target computing task according to the data volume; The step of splitting the target computing task into at least two sub-computing tasks according to the split granularity includes: The master node splits the target computing task into at least two sub-computing tasks according to the split granularity; The master node assigns each of the sub-computing tasks to the corresponding working nodes respectively; The steps of respectively calculating the sub-computing tasks and obtaining sub-computing results corresponding to the sub-computing tasks include: Each of the working nodes calculates the sub-computation results of the sub-computation tasks respectively through a multi-dimensional time series data accumulation algorithm; The step of respectively calculating the final cumulative link values ​​of the sub-computing tasks comprises: The master node obtains corresponding sub-computation results from each working node respectively; The master node calculates the final cumulative link value of each of the sub-computing tasks; The master node performs link calculation according to the final cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

3. The method according to claim 1, characterized in that The step of confirming the data volume of the target computing task corresponding to the computing request includes: Analyze the data section of the target computing task to obtain computing parameters; the computing parameters are used to characterize the data volume and time length of the target computing task; The step of determining the splitting granularity of the target computing task according to the data volume includes: Determine whether the target computing task needs to be split according to the computing parameters; If necessary, determine the date split granularity based on the data volume and memory parameters; The step of splitting the target computing task into at least two sub-computing tasks according to the split granularity includes: The target computing task is split into at least two sub-computing tasks according to the date splitting granularity, and each of the sub-computing tasks has corresponding date sequence information.

4. The method according to claim 1, characterized in that: The step of respectively calculating the sub-computing tasks and obtaining sub-computing results corresponding to the sub-computing tasks includes: Calculating the corresponding sub-computation results of the sub-computation tasks respectively through a multi-dimensional time series data accumulation algorithm; Calculating the inter-period factor of the sub-computing task, and determining the end-of-period link value corresponding to the sub-computing task according to the inter-period factor; The step of performing link calculation according to the end-of-period cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request includes: Calculate the end-of-period cumulative link value based on each of the end-of-period link values; Link calculation is performed on each of the sub-calculation results according to the final cumulative link value to obtain the cumulative calculation result.

5. A data divide-and-conquer computing device, characterized in that: include: The splitting module is used to confirm the data volume of the target computing task corresponding to the computing request; Determine the splitting granularity of the target computing task according to the data volume; Splitting the target computing task into at least two sub-computing tasks according to the splitting granularity; A task calculation module, used to calculate the sub-computing tasks respectively and obtain the sub-computing results corresponding to the sub-computing tasks; The link calculation module is used to calculate the final cumulative link value based on each final link value; According to the final cumulative link value, link calculation is performed on the calculation indicators of each of the sub-calculation tasks to obtain a cumulative calculation result, wherein the final link value is used for the breakpoint continuation after the multi-dimensional time series data is split; the final cumulative link value represents the continuous relationship between different sub-calculation results, and is used for the breakpoint continuation after the sub-calculation results are split; A link calculation is performed according to the end-of-period cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

6. The device according to claim 5, characterized in that The task calculation module is specifically used to calculate the corresponding calculation indicators of the sub-calculation task through a multi-dimensional time series data accumulation algorithm; calculate the inter-period factor of the sub-calculation task, and determine the end-of-period link value corresponding to the sub-calculation task according to the inter-period factor; and use the calculation indicator and the end-of-period link value as the sub-calculation result.

7. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 4 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

9. A program product, characterized in that When the program product is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images