Data divide-and-conquer calculation method and device and program product

Abstract

The invention provides a data divide-and-conquer calculation method and device and a program product. The data size of a target calculation task corresponding to a calculation request is confirmed. And determining the splitting granularity of the target calculation task according to the data volume. And splitting the target calculation task into at least two sub-calculation tasks according to the splitting granularity. And the sub-calculation tasks are utilized to separately obtain respective sub-calculation results, so that the calculation performance of the calculation system is fully utilized, and then the recombination of the sub-calculation tasks is completed through link calculation. And the calculation accuracy is ensured while the calculation performance is utilized.

Description

Technical Field

[0001] The present application relates to the field of data computing, and more specifically, to a data divide-and-conquer computing method, device, and program product. Background Art

[0002] In the field of multidimensional time series data analysis, it is necessary to slice, dice, aggregate, and analyze data in multiple dimensions, such as time, geography, and product.

[0003] However, during the calculation process, taking the financial scenario as an example, when analyzing large amounts of, or even massive amounts of, historical holdings and transaction details data, it often involves data in the time dimension and different product and asset level dimensions. These data in different dimensions are often interdependent, resulting in the need to load a large amount of data into the computing system at the same time for multi-dimensional cube calculations.

[0004] At this time, if the computing system is a stand-alone system, loading a large amount of data may lead to stability risks such as out of memory (OOM). If the computing system is a distributed system, the calculation of the above data cannot be broken down in the time and asset dimensions, resulting in the inability to fully utilize the computing power of the distributed system to improve computing performance. Summary of the Invention

[0005] The purpose of this application is to provide a data divide-and-conquer computing method, device and program product, which are used to improve the full and stable use of the computing performance of computing equipment while ensuring the accurate completion of computing tasks.

[0006] In order to achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows: In a first aspect, an embodiment of the present application provides a data divide-and-conquer computing method, comprising: Confirm the data volume of the target computing task corresponding to the computing request; Determine the splitting granularity of the target computing task according to the data volume; Splitting the target computing task into at least two sub-computing tasks according to the splitting granularity; Calculate the sub-computation tasks respectively to obtain sub-computation results corresponding to the sub-computation tasks; Calculating the final cumulative link value of each of the sub-computation tasks, wherein the final cumulative link value represents the continuity relationship between different sub-computation results and is used to reconnect the sub-computation results after they are split; A link calculation is performed based on the end-of-period cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

[0007] Optionally, respectively calculating the final cumulative link values ​​of the sub-computing tasks includes: Calculating the inter-period factor of each of the sub-computing tasks, and determining the end-of-period link value corresponding to the sub-computing task according to the inter-period factor; The end-of-period cumulative link value is obtained according to each end-of-period link value.

[0008] Optionally, the method is applied to a computing system comprising a master node and at least two working nodes; the step of confirming the data volume of the target computing task corresponding to the computing request comprises: The master node receives the computing request; The master node confirms the data volume of the target computing task; The step of determining the splitting granularity of the target computing task according to the data volume includes: The master node determines the splitting granularity of the target computing task according to the data volume; The step of splitting the target computing task into at least two sub-computing tasks according to the split granularity includes: The master node splits the target computing task into at least two sub-computing tasks according to the split granularity; The master node assigns each of the sub-computing tasks to the corresponding working nodes; The steps of respectively calculating the sub-computation tasks and obtaining sub-computation results corresponding to the sub-computation tasks include: Each of the working nodes calculates the sub-computation results of the sub-computation tasks respectively through a multi-dimensional time series data accumulation algorithm; The step of respectively calculating the final cumulative link value of each of the sub-computing tasks includes: The master node obtains the corresponding sub-computation results from each working node respectively; The master node calculates the final cumulative link value of each sub-computing task; The master node performs link calculation based on the end-of-period cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

[0009] Optionally, the step of confirming the data volume of the target computing task corresponding to the computing request includes: Analyze the data section of the target computing task to obtain computing parameters; the computing parameters are used to characterize the data volume and time length of the target computing task; The step of determining the splitting granularity of the target computing task according to the data volume includes: Determine whether the target computing task needs to be split according to the computing parameters; If necessary, determine the date split granularity based on the data volume and memory parameters; The step of splitting the target computing task into at least two sub-computing tasks according to the split granularity includes: The target computing task is split into at least two sub-computing tasks according to the date splitting granularity, and each of the sub-computing tasks has corresponding date sequence information.

[0010] Optionally, the step of respectively calculating the sub-computing tasks and obtaining sub-computing results corresponding to the sub-computing tasks includes: Calculate the corresponding calculation indicators of the sub-computing tasks respectively through a multi-dimensional time series data accumulation algorithm; Calculating the inter-period factor of the sub-computing task, and determining the end-of-period link value corresponding to the sub-computing task according to the inter-period factor; Using the calculation index and the end-of-period link value as the sub-calculation result; The step of performing link calculation based on the end-of-period cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request includes: Calculate the end-of-period cumulative link value based on each of the end-of-period link values; The cumulative calculation result is obtained by performing link calculation on the calculation indicators of each of the sub-computing tasks according to the final cumulative link value.

[0011] In a second aspect, an embodiment of the present application provides a data divide-and-conquer computing device, comprising: A splitting module is configured to confirm the data volume of the target computing task corresponding to the computing request; determine the splitting granularity of the target computing task according to the data volume; and split the target computing task into at least two sub-computing tasks according to the splitting granularity; A task calculation module, configured to calculate the sub-computing tasks respectively and obtain sub-computing results corresponding to the sub-computing tasks; The link calculation module is used to calculate the end-of-period cumulative link values ​​of the sub-computation tasks respectively, wherein the end-of-period cumulative link values ​​represent the continuous relationship between different sub-computation results and are used to reconnect the breakpoints of each sub-computation result after splitting; link calculation is performed based on the end-of-period cumulative link values ​​and the corresponding sub-computation results to obtain the cumulative calculation result corresponding to the calculation request.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method as described in any one of the above-mentioned first aspects is implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, it implements the method as described in any one of the above-mentioned first aspects.

[0014] In a fifth aspect, an embodiment of the present application provides a program product, which, when executed by a processor, implements the method as described in any one of the above-mentioned first aspects.

[0015] Compared to the prior art, the data divide-and-conquer computing method, apparatus, and program product provided in the embodiments of the present application confirm the data volume of the target computing task corresponding to the computing request. Based on the data volume, the target computing task is split into at least two sub-computing tasks according to the splitting granularity. The sub-computing tasks are then used to obtain their respective sub-computing results, thereby fully utilizing the computing performance of the computing system. The sub-computing tasks are then reorganized through chained computing. While utilizing computing performance, the accuracy of the calculation is ensured.

[0016] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a diagram of the main calculation process and data dimensions of the portfolio attribution model indicators; Figure 2 A schematic diagram of the architecture of a data divide-and-conquer computing method provided by an embodiment of the present invention; Figure 3 A flowchart of a data divide-and-conquer computing method provided by an embodiment of the present invention; Figure 4 A schematic diagram of the architecture of a computing system provided by an embodiment of the present invention; Figure 5 A schematic diagram of a processing architecture of a distributed computing system provided by an embodiment of the present invention; Figure 6 A schematic diagram of a processing architecture for stand-alone deployment provided by an embodiment of the present invention; Figure 7 A schematic diagram of a flow chart of another data divide-and-conquer computing method provided by an embodiment of the present invention; Figure 8 A schematic diagram of a flow chart of another data divide-and-conquer computing method provided by an embodiment of the present invention; Figure 9 A schematic diagram of the core divide-and-conquer computing solution provided by an embodiment of the present invention; Figure 10 A schematic diagram of the structure of a data divide-and-conquer computing device provided by an embodiment of the present invention; Figure 11 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0021] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0022] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0023] For multi-dimensional time series data analysis, this application will illustrate the business scenario of portfolio attribution analysis. However, the scenarios to which the technical solution of this application can be applied are not limited to this scenario. Based on the examples described below in this application, those skilled in the art can, without any creative work, combine the cumulative calculation of time series data with multi-dimensional cube calculation, design appropriate divide-and-conquer calculation units for time series data, derive the corresponding divide-and-conquer link algorithm, and apply the following solution of this application.

[0024] In existing technologies, when conducting company-level and department-level analysis of financial data, it is necessary to analyze large amounts of data holdings and transaction details, and to calculate asset dimension cube data in real time based on the detailed data.

[0025] Due to the continuity of time-accumulation dimension calculation and the interdependence of asset-level dimension calculation, a large amount of data needs to be loaded into memory at the same time for multi-dimensional cube calculation. Figure 1 This is the main calculation process of the portfolio attribution model indicators in the existing technology. Figure 1 The calculation process on the left side (a) shows that for a calculation system, it first loads the detailed business data to be calculated. Then, it calculates the portfolio inter-period factor α, as well as the asset-level dimension weights and return items. It then calculates the single-period asset-level dimension attribution indicators. Finally, it combines the portfolio inter-period factor α to calculate the multi-period asset-level dimension attribution indicators.

[0026] Among them, the portfolio inter-period factor α is calculated as follows:

[0027] t is the date variable, R is the portfolio return rate, from this formula and Figure 1 As can be seen from the right part (b), within the calculation date interval, the inter-period factor α on day t is obtained by multiplying the portfolio returns of all dates before day t by one. The calculation of the time accumulation dimension is continuous, so the existing method cannot be disassembled in the time dimension. k is the date variable in the cumulative formula, which describes the individual state of each day in the time period. The difference from t is that t is the cumulative state at a specific time point. The formula is expanded and k is visualized as .

[0028] Continue to see Figure 1 , where C is the attributable contribution, which is calculated as follows:

[0029] j is the asset dimension variable (including details), w is the weight, and r is the attributed income item. Figure 1As can be seen from part (b) on the right, the attribution contribution calculation of each dimensional layer in the formula (such as the portfolio P, stock dimension layer S,j, industry dimension layer I,j, individual bond detail layer j, etc. in (b) on the right) depends on α at the portfolio layer, while R in the α calculation is calculated from the detail dimension data. The asset-level dimensions are interdependent, so the existing method cannot be disassembled at the asset dimension.

[0030] It should be noted that the investment portfolio described in this application generally refers to a complete asset portfolio. Figure 1 The asset hierarchy shown is used to describe the multi-layered relationships that arise when multiple assets are categorized and grouped according to specific dimensions.

[0031] Therefore, based on the data dependencies in the time accumulation dimension and the asset hierarchy dimension during the above calculation process, in existing computing system deployment scenarios, for single-machine system deployment scenarios, large amounts of data are loaded into memory in real time for calculation, and investment management and analysis systems face stability risks such as memory overflow and out-of-memory (OOM). Furthermore, for distributed system deployment scenarios, because existing computing methods cannot be separated in the time and asset dimensions, it is impossible to use distributed computing to improve performance, resulting in performance bottlenecks in investment management and analysis systems.

[0032] In view of the above existing technologies facing the computational problems of dependencies in the time accumulation dimension and the hierarchical dimension. In order to solve the impact of the computational process on the computational performance, this application introduces a divide-and-conquer link computation mechanism. Specifically, Figure 2 A schematic diagram of the architecture of a data divide-and-conquer computing method provided by an embodiment of the present invention is shown in FIG. Figure 2 Continuing with the above portfolio attribution analysis example, we split the business details data for a certain period of time into tasks, and keep the asset dimension calculation independent and complete within the split sub-calculation tasks, which can be reused. Figure 1 The calculation process is shown in the figure. After each sub-calculation task is completed, the results of each period are linked and calculated to obtain the final calculation result.

[0033] Optionally, because portfolio attribution analysis generally uses attribution models (such as the Brinson and Campisi models), this application can directly reuse the system's existing attribution model calculations. To minimize computational intrusion and improve system reusability, custom annotations and aspect-oriented programming (AOP) techniques can be used to implement wraparound processing for the reused process. Furthermore, to achieve task decomposition, a map-reduce model can be used, with the map phase used to implement decomposition and the reduce phase used to complete the linked computation.

[0034] Furthermore, combined with Figure 2 The architecture of the data divide-and-conquer computing method shown, Figure 3 A flow chart of a data divide-and-conquer computing method provided by an embodiment of the present invention is shown in FIG. Figure 3 , the method comprising: Step 100: Confirm the data volume of the target computing task corresponding to the computing request.

[0035] Step 101: Determine the splitting granularity of the target computing task according to the data volume.

[0036] Step 102: Split the target computing task into at least two sub-computing tasks according to the splitting granularity.

[0037] Step 103: Calculate the sub-computation tasks respectively to obtain the sub-computation results corresponding to the sub-computation tasks.

[0038] Step 104: Calculate the final cumulative link value of each sub-computation task.

[0039] The cumulative link value at the end of the period represents the continuous relationship between different sub-calculation results and is used to connect the breakpoints of each sub-calculation result after it is split.

[0040] Step 105: Perform link calculation based on the final cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

[0041] The data divide-and-conquer computing method provided by the embodiments of the present invention determines the data volume of the target computing task corresponding to the computing request. Based on the data volume, the target computing task is split into at least two sub-computing tasks according to the splitting granularity. Separate sub-computing results are obtained from the sub-computing tasks to fully utilize the computing performance of the computing system. The sub-computing tasks are then reassembled through chained computing. This maximizes computing performance while ensuring computational accuracy.

[0042] In order to implement the above process, a possible implementation of a computing system is provided below. Specifically, Figure 4 A schematic diagram of the architecture of a computing system provided by an embodiment of the present invention is shown in FIG. Figure 4 The computing system is a distributed system, including: master node 1, master node 2, worker node 1, worker node 2 and worker node 3.

[0043] Among them, the master node 1 and the working node 1, working node 2 and working node 3; the master node 2 and the working node 1, working node 2 and working node 3 can all form a distributed computing system that can independently complete the target computing task calculation.

[0044] Among them, the master node (Master) is used to receive computing requests, split the target computing tasks, and link calculations.

[0045] Worker nodes are used to calculate sub-computing tasks of a single period.

[0046] When a master node receives a computing request, it can configure a number of worker nodes that matches the computing request based on the current computing performance and load conditions of the worker nodes it can schedule.

[0047] Of course, for the working node, under the premise that its computing performance meets the requirements, it can execute the sub-computing tasks split under different target computing tasks respectively; for example, for the working node 2, it can simultaneously execute the sub-computing task A1 assigned by the master node 1 and the sub-computing task B1 assigned by the master node 2.

[0048] Below Figure 4 For example, master node 1 and worker nodes 1, 2, and 3, Figure 3 Specifically, Figure 5 A schematic diagram of a distributed computing system processing architecture provided by an embodiment of the present invention, see Figure 5 : First, the master node 1 receives a computing request; then the master node 1 confirms the data volume of the target computing task; and determines the splitting granularity of the target computing task based on the data volume.

[0049] Then, the master node 1 splits the target computing task into three sub-computing tasks according to the splitting granularity. The master node 1 assigns one sub-computing task to each of the worker nodes 1, 2, and 3.

[0050] Worker node 1, worker node 2, and worker node 3 respectively calculate the sub-computation results of the sub-computation tasks through a multi-dimensional time series data accumulation algorithm.

[0051] Furthermore, the master node 1 obtains the corresponding sub-calculation results from each working node respectively; the master node 1 calculates the final cumulative link value of each sub-calculation task; the master node 1 performs link calculation based on the final cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

[0052] It should be noted that each working node can undertake one or more sub-computing tasks based on its own computing power and memory resources, which is not limited here.

[0053] Optional, for Figure 5The distributed computing system shown in this example employs a parallel strategy for configuring and computing subtasks. This strategy involves the master node assigning each subtask in parallel to the corresponding worker nodes for computation. This approach can effectively improve computing performance in resource-rich scenarios, and has been shown to improve computing performance by 50% compared to existing technologies.

[0054] In another implementation, the above-mentioned distributed computing system may not be used. Instead, the above-mentioned processing may be performed based on a single device. Specifically, Figure 6 A schematic diagram of a single-machine deployment processing architecture provided by an embodiment of the present invention is shown in FIG. Figure 6 : For a single-machine deployment, all subtask calculations are completed within the single machine. When data volumes are limited and system deployment resources are limited, the single-machine serial mode ensures system stability and reliability for large-scale data calculations, preventing system unavailability due to insufficient memory. Testing has verified that existing calculation methods on the same single machine will cause out-of-memory (OOM) in the test scenario, while this proposed solution can stably complete the calculation.

[0055] The following provides a possible implementation method for splitting the target computing tasks. Figure 3 On the basis of Figure 7 A flow chart of another data divide-and-conquer computing method provided by an embodiment of the present invention is shown in FIG. Figure 7 , step 100, specifically: Step 100 - 1 : Analyze the data slice of the target computing task to obtain computing parameters.

[0056] The calculation parameters are used to characterize the data volume and time length of the target calculation task. Specifically, using the AOP technology described above, a data aspect is inserted into the calculation process of the target calculation task to obtain parsed and calculated parameters.

[0057] Optionally, to reduce the impact of process intrusion, you can use custom annotations to encapsulate the data aspects obtained by AOP technology into a simple code identifier (for example, @SplitCalculation), which can be directly used in multiple locations in the system. For example, you can use custom annotations to mark specific pointcuts in the original calculation process so that AOP technology can use these pointcuts to insert the above data aspects.

[0058] Step 101 is specifically as follows: Step 101 - 1 : Determine whether the target computing task needs to be split according to the computing parameters.

[0059] If splitting is required, step 101 - 2 is executed; if not, the target computing task is calculated without splitting.

[0060] Step 101-2: Determine the date splitting granularity based on the data volume and memory parameters.

[0061] The date splitting granularity can be divided based on the time granularity of the business scenario, such as splitting by year, splitting by month, or splitting by day.

[0062] In one possible implementation, the data volume can be: daily data volume (Scale); memory parameters can be: metadata (minimum computing unit) memory usage (Meta), worker node heap memory size (Heap), etc. The daily data volume, metadata memory usage, and worker node heap memory size are then used to calculate the appropriate date split granularity to ensure reliable and available memory for executing sub-computing tasks. Specifically, the calculation formula for the date split granularity (Period) is as follows:

[0063] Among them, 0.618 is the golden section constant (0.618 is the initial value in the execution process and can be adjusted according to the actual scenario. This application does not limit it). It is used to set the heap threshold (to prevent the heap memory from being full).

[0064] It should be noted that: for the Heap setting, in one possible implementation scenario, it is necessary to agree that the memory configuration of different working nodes is consistent or in the same resource range. If different working nodes are in the same cluster, the Heap threshold can be consistent; if different working nodes are in different clusters, then for this application solution, for a computing request, the sub-computing tasks can be assigned to working nodes in the same cluster. For stand-alone deployment, the Heap value executes the memory configuration of the computing device.

[0065] Step 102 is specifically as follows: Step 102 - 1 : Split the target computing task into at least two sub-computing tasks according to the date splitting granularity, and each sub-computing task has corresponding date sequence information.

[0066] Specifically, the date sequence is used to mark sub-computation tasks, and subsequent linked calculations can be performed based on the date sequence information.

[0067] The following provides a possible implementation method for link calculation. Figure 3 On the basis of Figure 8 This is a flow chart of another data divide-and-conquer calculation method provided by an embodiment of the present invention, continuing with the above portfolio attribution analysis example, see Figure 8, step 103, specifically: Step 103 - 1 : Calculate the corresponding sub-computation results of the sub-computation tasks respectively through the multi-dimensional time series data accumulation algorithm.

[0068] Step 103 - 2 : Calculate the inter-period factor of the sub-computing task, and determine the end-of-period link value corresponding to the sub-computing task based on the inter-period factor.

[0069] Specifically, as explained in the previous example, when the multidimensional time series data accumulation algorithm is specifically an attribution model calculation, as can be seen from the α calculation formula above, due to the continuity of the cumulative multiplication, the above-mentioned end-of-period link value can be used as the end-of-period link multiplier. When each period is calculated independently, a breakpoint is generated at the beginning of the period relative to the overall time period. The sub-calculation task must calculate and return an end-of-period link multiplier for each period that is applied to the next period, used to connect the breakpoints after the multidimensional time series data is split. The calculation formula for this end-of-period link multiplier is:

[0070] Where t is the time variable, Used to describe the single-day rate of return of each subtask at the end of the period after splitting. The formula is described in Chinese as: End-of-period link multiplier = End-of-period inter-period factor * (1 + End-of-period daily rate of return).

[0071] Continuing with the example of cumulative multiplication continuity involving attribution model calculations, since the end-of-period link multiplier applies to the sub-calculation results of the next period, each sub-calculation result must also return corresponding date sequence information when returning it. This is used to sort and match the respective end-of-period link multipliers during the link calculation. In this setting, the end-of-period link multiplier applied to the first period is always 1.

[0072] Step 104 is specifically as follows: Step 104 - 1 : Calculate the cumulative link value at the end of each period based on the link value at the end of each period.

[0073] Specifically, the final cumulative link multiplier (final cumulative link value) of each sub-computation task is calculated by multiplying the final link multipliers (final link values) of all sub-computation tasks before the current sub-computation task. The calculation formula is:

[0074] Where acc represents the cumulative concept.

[0075] Step 104 - 2 : Perform link calculation on each sub-calculation result according to the final cumulative link value to obtain a cumulative calculation result.

[0076] Specifically, continuing with the example of cumulative multiplication continuity involving attribution model calculation, step 104-2 represents the chained calculation of multiple sub-computation tasks: the cumulative calculation results of multiple sub-computation tasks (i.e., attribution model results) are chained and calculated using the following formula:

[0077] The attribution model calculation result = SUM (each sub-calculation result * cumulative link multiplier at the end of the period), where IR is the sub-calculation result, p is the number of split sub-calculation tasks, cur represents the current sub-calculation task concept, and link represents the link concept.

[0078] The following provides a specific example of formula derivation for the above multi-period link calculation (taking a single sub-calculation task as an example).

[0079] In this calculation method, the cumulative inter-period factor is equivalent to the inter-period factor in the conventional calculation model, and the calculation formula can be derived as follows: (Formula 1) The Chinese description of Formula 1 is: Cumulative inter-period factor = current inter-period factor * cumulative link multiplier at the end of the period.

[0080] The calculation formula for the cumulative result of each sub-computation task can be derived as follows: (Formula 2) The Chinese description of Formula 2 is: The cumulative calculation result of the current sub-calculation task = the sub-calculation result of the current sub-calculation task * the current inter-period factor.

[0081] Substitute Formula 1 into the following formula and deduce and combine it through Formula 2, so we have:

[0082] Specifically, for calculations involving attribution models, Figure 9 Schematic diagram of the core solution of divide-and-conquer computing provided by the embodiment of the present invention, see Figure 9 This demonstrates the divide-and-conquer approach of this solution: the target computation task is broken down into multiple sub-computation tasks, each of which independently performs attribution calculations. Simultaneously, the end-of-period link multiplier is used to calculate the cumulative end-of-period link multiplier, ultimately completing link calculations across multiple periods. This "independent attribution calculation for each period" allows for parallel or serial deployment to fully utilize computing performance and alleviate computing memory pressure.

[0083] Optionally, for the implementation of this solution, since the calculation process will generate intermediate indicator data such as "end-of-period link multiplier" and "date sequence information" that have no output requirements for the calculation request, the "end-of-period link multiplier" and "date sequence information" can be removed from the data structure before the attribution model calculation results are finally displayed.

[0084] This application also provides a data partitioning computing device to implement each step in the above example to achieve the corresponding technical effect. Specifically, Figure 10 A schematic diagram of a data partitioning computing device provided by an embodiment of the present invention is shown in FIG. Figure 10 The device 20 includes: a splitting module 200, a task calculation module 201 and a link calculation module 202.

[0085] The splitting module 200 is used to confirm the data volume of the target computing task corresponding to the computing request; determine the splitting granularity of the target computing task according to the data volume; and split the target computing task into at least two sub-computing tasks according to the splitting granularity.

[0086] The task calculation module 201 is used to calculate the sub-computing tasks respectively and obtain the sub-computing results corresponding to the sub-computing tasks.

[0087] Link calculation module 202 is configured to calculate the final cumulative link value for each sub-computation task. This final cumulative link value represents the continuity between different sub-computation results and is used to reconnect the sub-computation results after they have been split. Link calculation is performed based on the final cumulative link value and the corresponding sub-computation results to obtain the cumulative calculation result corresponding to the calculation request.

[0088] Specifically, the splitting module 200 can execute each step related to the splitting of the target computing task in the above example; the task computing module 201 can execute each step related to the calculation of the sub-computing task in the above example; and the link computing module 202 can execute each step related to the link calculation in the above example.

[0089] The embodiment of the invention further provides an electronic device that can execute all the steps of the above examples of the embodiment of the invention to achieve the corresponding technical effects. Specifically, Figure 11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, see Figure 11 , the electronic device 50 , comprises: a memory 501 , a processor 500 ; Memory 501, used to store one or more programs; Processor 500; When one or more programs are executed by the processor, when the electronic device 50 is used to perform the steps shown in the above-mentioned method examples, it can achieve each step and corresponding technical effects.

[0090] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0091] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0092] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a program product. The program product is stored in a computer-readable storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0093] The above are merely preferred embodiments of the present application and are not intended to limit the present application. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

[0094] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A data divide-and-conquer computing method, characterized in that: include: Confirm the data volume of the target computing task corresponding to the computing request; Determine the splitting granularity of the target computing task according to the data volume; Splitting the target computing task into at least two sub-computing tasks according to the splitting granularity; Calculate the sub-computation tasks respectively to obtain sub-computation results corresponding to the sub-computation tasks; Calculating the final cumulative link values ​​of the sub-computation tasks respectively, wherein the final cumulative link values ​​represent the continuous relationship between different sub-computation results and are used for the breakpoint continuation of each of the sub-computation results after they are split; A link calculation is performed according to the end-of-period cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

2. The method according to claim 1, characterized in that The respectively calculating the final cumulative link values ​​of the sub-computing tasks includes: Calculating the inter-period factor of each of the sub-computing tasks, and determining the end-of-period link value corresponding to the sub-computing task according to the inter-period factor; The end-of-period cumulative link value is obtained according to each end-of-period link value.

3. The method according to claim 1, characterized in that The method is applied to a computing system, the computing system comprising a master node and at least two working nodes; The step of confirming the data volume of the target computing task corresponding to the computing request includes: The master node receives the computing request; The master node confirms the data volume of the target computing task; The step of determining the splitting granularity of the target computing task according to the data volume includes: The master node determines the splitting granularity of the target computing task according to the data volume; The step of splitting the target computing task into at least two sub-computing tasks according to the split granularity includes: The master node splits the target computing task into at least two sub-computing tasks according to the split granularity; The master node assigns each of the sub-computing tasks to the corresponding working nodes respectively; The steps of respectively calculating the sub-computing tasks and obtaining sub-computing results corresponding to the sub-computing tasks include: Each of the working nodes calculates the sub-computation results of the sub-computation tasks respectively through a multi-dimensional time series data accumulation algorithm; The step of respectively calculating the final cumulative link values ​​of the sub-computing tasks comprises: The master node obtains corresponding sub-computation results from each working node respectively; The master node calculates the final cumulative link value of each of the sub-computing tasks; The master node performs link calculation according to the final cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

4. The method according to claim 1, characterized in that The step of confirming the data volume of the target computing task corresponding to the computing request includes: Analyze the data section of the target computing task to obtain computing parameters; the computing parameters are used to characterize the data volume and time length of the target computing task; The step of determining the splitting granularity of the target computing task according to the data volume includes: Determine whether the target computing task needs to be split according to the computing parameters; If necessary, determine the date split granularity based on the data volume and memory parameters; The step of splitting the target computing task into at least two sub-computing tasks according to the split granularity includes: The target computing task is split into at least two sub-computing tasks according to the date splitting granularity, and each of the sub-computing tasks has corresponding date sequence information.

5. The method according to claim 1, characterized in that The step of respectively calculating the sub-computing tasks and obtaining sub-computing results corresponding to the sub-computing tasks includes: Calculating the corresponding sub-computation results of the sub-computation tasks respectively through a multi-dimensional time series data accumulation algorithm; Calculating the inter-period factor of the sub-computing task, and determining the end-of-period link value corresponding to the sub-computing task according to the inter-period factor; The step of performing link calculation according to the end-of-period cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request includes: Calculate the end-of-period cumulative link value based on each of the end-of-period link values; Link calculation is performed on each of the sub-calculation results according to the final cumulative link value to obtain the cumulative calculation result.

6. A data divide-and-conquer computing device, characterized in that: include: The splitting module is used to confirm the data volume of the target computing task corresponding to the computing request; Determine the splitting granularity of the target computing task according to the data volume; Splitting the target computing task into at least two sub-computing tasks according to the splitting granularity; A task calculation module, used to calculate the sub-computing tasks respectively and obtain the sub-computing results corresponding to the sub-computing tasks; A link calculation module, used to calculate the final cumulative link values ​​of the sub-calculation tasks respectively, wherein the final cumulative link values ​​represent the continuous relationship between different sub-calculation results and are used to connect the breakpoints of each of the sub-calculation results after they are split; A link calculation is performed according to the end-of-period cumulative link value and the corresponding sub-calculation result to obtain the cumulative calculation result corresponding to the calculation request.

7. The device according to claim 6, characterized in that The task calculation module is specifically used to calculate the corresponding calculation index of the sub-calculation task through the multi-dimensional time series data accumulation algorithm; calculate the inter-period factor of the sub-calculation task, and determine the end-of-period link value corresponding to the sub-calculation task according to the inter-period factor; and use the calculation index and the end-of-period link value as the sub-calculation result; The link calculation module is specifically used to calculate the final cumulative link value according to each final link value; and to perform link calculation on the calculation indicators of each sub-computing task according to the final cumulative link value to obtain the cumulative calculation result.

8. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

10. A program product, characterized in that When the program product is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images