Calculation method and device based on gradient elevator algorithm, equipment and medium

Abstract

The invention relates to the technical field of AI calculation and distributed iterative calculation, and discloses a calculation method, device and equipment based on a gradient elevator algorithm and a medium, the method comprises the following steps: adopting the gradient elevator algorithm to control a Tracker cluster to perform task state management on a plurality of calculation nodes, and sending a task state instruction; when each computing node receives a task state instruction, controlling the computing nodes to perform distributed iterative computation based on the task state instruction by adopting a gradient elevator algorithm to obtain an intermediate result, and storing the intermediate result in the form of a plurality of copies; and controlling the Tracker cluster to carry out reduce calculation on the intermediate result copies of the plurality of calculation nodes to obtain a calculation result, and carrying out storage management on the calculation result. According to the method, it is guaranteed that recovery can be carried out at the minimum calculation cost when calculation is abnormal every time, a large amount of recovery calculation caused by faults is not needed, and stable operation of offline calculation of a recommendation system is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of AI computing and distributed iterative computing technology, and in particular to a computing method, device, equipment and medium based on a gradient boosting algorithm. Background Art

[0002] In the field of search recommendation systems, GBM (gradient boosting machine) is widely used in feature engineering and large-scale recall sorting scenarios. Such scenarios are characterized by large data scale and huge feature scale, so a distributed architecture is generally used to apply the algorithm for training and inference calculations. At present, the mainstream distributed architecture in the industry is the Allreduce architecture (a communication operation commonly used in parallel computing and distributed computing, mainly used to aggregate data between multiple computing nodes (such as GPUs or CPUs)). Each node first completes the computing task of the local data shard, and then aggregates the data. Finally, the aggregated global state will reach a consistent state between each computing node. However, in the Allreduce architecture calculation, some calculation failures due to node failures or network failures cannot be avoided.

[0003] The Allreduce cluster itself does not have the ability to quickly detect and recover from faults. When a node is unavailable, all calculations will be blocked. Even if the fault is recovered and the node is restarted, the calculation will be interrupted and it will need to start from the last model checkpoint. However, in the scenario of a large-scale recommendation system, each checkpoint calculation may involve hundreds or thousands of iterations. Because the calculation is interrupted locally, all iterations need to be recalculated. The extra time spent is likely to cause the offline calculation to time out, and the online data cannot be updated in time, thus affecting online revenue.

[0004] The current Allreduce-based GBM implementation lacks consideration for cluster stability. When the data scale is very large, the number of computing nodes will also increase accordingly, and the probability of node failure or network failure will also increase. How to design a system that can detect and recover from errors more quickly is of great significance to the offline computing of the entire recommendation system. Summary of the invention

[0005] In view of this, the present invention provides a computing method, device, equipment and medium based on a gradient boosting algorithm to solve the problem that a distributed computing cluster cannot recover quickly from a fault, resulting in a long fault blocking computing time.

[0006] In a first aspect, the present invention provides a calculation method based on a gradient boosting algorithm, which is applied to a distributed computing system, wherein the distributed computing system includes a distributed computing cluster and a Tracker cluster, and the distributed computing cluster includes multiple computing nodes; the calculation method includes:

[0007] The gradient boosting algorithm is used to control the Tracker cluster to manage the task status of multiple computing nodes and send task status instructions;

[0008] When each computing node receives a task status instruction, the gradient boosting algorithm is used to control the computing node to perform distributed iterative calculations based on the task status instruction to obtain intermediate results, and store the intermediate results in the form of multiple copies. The storage distribution of multiple copies is recorded in the Tracker cluster;

[0009] Control the Tracker cluster to perform reduce calculations on the intermediate result copies of multiple computing nodes to obtain the calculation results, and store and manage the calculation results.

[0010] The calculation method based on the gradient boosting algorithm provided by the present invention introduces a Tracker cluster to manage the entire distributed computing system, including task status management, intermediate result storage copy routing, and in order to avoid repeated calculation of the intermediate results of past iterations lost due to node failure, a reliable copy storage and compression recovery strategy is proposed to ensure that the cluster will not have time-consuming repeated backtracking calculations, and to ensure that each calculation exception can be restored at the minimum computing cost, without a large number of recovery calculations due to failures, ensuring the stable operation of offline calculations of the recommendation system, reducing the risk caused by node failures, and making the computing cluster larger, improving computing power, and having stronger fault tolerance and faster fault detection and recovery capabilities in large-scale distributed cluster scenarios. In the large-scale feature data recall and sorting scenario, a large number of repeated calculations caused by failures are avoided, and offline computing tasks are avoided from timeouts due to failures, affecting the timeliness of important business online data, and solving the problem that the distributed computing cluster cannot recover quickly from failures, resulting in long failure blocking calculation time.

[0011] In an optional implementation, the Tracker cluster includes multiple Tracker components, each of which performs a heartbeat session with a computing node; and the computing method further includes:

[0012] A distributed consistency algorithm is used to maintain strong consistency in the heartbeat sessions between multiple Tracker components and computing nodes.

[0013] The computing method based on the gradient boosting algorithm provided by the present invention adopts a distributed consistency algorithm to maintain the strong consistency of the heartbeat sessions between multiple Tracker components and computing nodes, ensuring that multiple Tracker components in the Tracker cluster in the distributed system form a strong consistency system, so that time recovery when restoring failed computing nodes, recovery from the task stage, and reading of intermediate result copies can proceed smoothly.

[0014] In an optional implementation, a gradient boosting algorithm is used to control the Tracker cluster to manage the task status of multiple computing nodes, and sending a task status instruction includes:

[0015] The states of multiple computing nodes are abstracted into a finite state machine model, and the gradient boosting algorithm is used to control the Tracker component to register the computing nodes and then send distributed iterative computing task instructions.

[0016] In an optional implementation, when each computing node receives a task status instruction, using a gradient boosting algorithm to control the computing node to perform distributed iterative calculations based on the task status instruction to obtain an intermediate result includes:

[0017] When each computing node receives a task status instruction, the gradient boosting algorithm is used to control the computing node to perform distributed iterative calculations in the scatter phase and the gather phase based on the task status instruction to obtain intermediate results;

[0018] When one of the computing nodes fails, the Tracker component corresponding to the failed computing node is used to replace the failed computing node with a non-faulty computing node as a new node, and a distributed iterative computing task instruction is sent after the new node is registered;

[0019] Based on the distributed iterative computing task instructions, a new node is used to load the data shards and intermediate result copies required for the calculation. The intermediate result copies are intermediate result copies calculated by the faulty node or intermediate result copies calculated by other non-faulty nodes.

[0020] When the loading is complete, the new node is controlled to enter the normal distributed iterative computing state.

[0021] The calculation method based on the gradient boosting machine algorithm provided by the present invention, when each computing node receives a task state instruction, uses the gradient boosting machine algorithm to control the computing node to perform distributed iterative calculations in the scatter stage and the gather stage based on the task state instruction, and obtains an intermediate result, so that the Tracker component can better manage the task state of each computing node, so that the task execution of the computing node in the distributed computing cluster is carried out in an orderly manner, which is conducive to the Tracker component to maintain the health status of the computing node in real time. The gradient boosting machine algorithm (GBM algorithm, gradient boosting machine) is used to control the Tracker cluster to quickly recover normal calculations from the distributed computing cluster failure, reduce the blocking time of offline computing tasks, and ensure that each calculation abnormality can be recovered at the minimum computing cost, without a large number of recovery calculations caused by the failure, ensuring the stable operation of the offline calculation of the recommendation system. The Tracker component performs health monitoring, task progress control, and task automatic recovery management on the entire distributed computing cluster. The state machine model of the node can effectively control any fault state of the cluster to avoid faults. By caching the calculation result copy mechanism, a large number of recovery calculations are also avoided.

[0022] In an optional implementation, using a new node to load data shards and intermediate result copies required for calculation based on a distributed iterative computing task instruction includes:

[0023] When the new node is started, it directly takes over all intermediate result copies of the corresponding load of the failed node, and selects the nearest computing node to complete the replication of the intermediate result copies;

[0024] Alternatively, a higher weight is given to the new node than other computing nodes, and the new node is directly selected according to the load weight strategy corresponding to the failed node to migrate the intermediate result copy and complete the intermediate result copy calculation.

[0025] The computing method based on the gradient boosting algorithm provided by the present invention directly takes over all the intermediate result copies of the load corresponding to the failed node when the new node is started, and selects the nearest computing node to complete the replication of the intermediate result copies; or, assigns the new node a higher weight than other computing nodes, directly selects the new node according to the load weight strategy corresponding to the failed node to migrate the intermediate result copies, completes the calculation of the intermediate result copies, and reduces the time for computing node recovery as much as possible, overcomes the defect that when the computing node scale is large, the copying of multiple copies consumes a lot of time overhead, and can ensure load balancing in the distributed computing cluster.

[0026] In an optional implementation, controlling the Tracker cluster to perform reduce calculations on the intermediate result copies of each of the multiple computing nodes to obtain calculation results, and storing and managing the calculation results includes:

[0027] Select computing nodes according to the computing node selection principle, and use the intermediate result copy of each computing node as data input to perform reduce calculation to obtain the calculation result; the computing node selection principle is to select computing nodes with local storage, computing nodes with storage allowed by the same rack load, and computing nodes with different rack loads in turn;

[0028] or,

[0029] When any of the selected computing nodes fails, reduce calculations are first performed on the computing nodes that have not failed to obtain the first reduce calculation result. After the failed node is replaced by a new node and restored, reduce calculations are then performed on the first reduce calculation result and the intermediate result copy of the new node to obtain the calculation result.

[0030] The time sliding window method is used to filter out the intermediate result copies participating in the reduce calculation in each computing node for storage, and the memory space occupied by the intermediate result copies not participating in the reduce calculation in each computing node is reclaimed.

[0031] The computing method based on the gradient boosting algorithm provided by the present invention and the partially blocked distributed computing model efficiently utilize the computing resources of the distributed computing cluster, that is, the part that can obtain the result is allowed to perform reduce calculation first, and the normal computing nodes are allowed to perform reduce calculation while waiting for the faulty node to recover. After the faulty node is recovered and the result output is completed, only the pre-calculated reduce result and the new output result need to be reduced again, which can reduce the impact of the fault on the calculation, make the fault recovery time and the reduce calculation calculated in parallel, save the calculation time, and improve the computing power.

[0032] In an optional implementation, the intermediate result copy includes a partial histogram calculated by each computing node in each iteration; and the calculation method further includes:

[0033] Obtain the spatial complexity of the partial histogram calculated by each computing node in each iteration of the calculation result, where the spatial complexity includes the number of features and the number of bins of the partial histogram;

[0034] The gradient boosting machine algorithm is used to enumerate all the segmentation points in each partial histogram, and a preset number of features before the maximum segmentation benefit are selected to form a global histogram, that is, the spatial complexity of the global histogram is compressed to the spatial complexity of the number of features and the number of bins including the preset number of features before the maximum segmentation benefit.

[0035] The calculation method based on the gradient boosting machine algorithm provided by the present invention adopts the gradient boosting machine algorithm to optimize the space complexity of the intermediate result copy, thereby ensuring that the distributed computing cluster does not have time-consuming repeated backtracking calculations, thereby improving the computing power.

[0036] In a second aspect, the present invention provides a computing device based on a gradient boosting algorithm, which is applied to a distributed computing system, wherein the distributed computing system includes a distributed computing cluster and a Tracker cluster, and the distributed computing cluster includes multiple computing nodes; the computing device includes:

[0037] The task status management module is used to use the gradient boosting algorithm to control the Tracker cluster to manage the task status of multiple computing nodes and send task status instructions;

[0038] The intermediate result replica calculation module is used to control the computing node to perform distributed iterative calculation based on the task status instruction using the gradient boosting algorithm when each computing node receives the task status instruction, and store the intermediate result in the form of multiple replicas. The storage distribution of multiple replicas is recorded in the Tracker cluster.

[0039] The storage management module is used to control the Tracker cluster to perform reduce calculations on the intermediate result copies of multiple computing nodes to obtain calculation results, and to store and manage the calculation results.

[0040] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the calculation method based on the gradient boosting machine algorithm of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the calculation method based on the gradient boosting algorithm of the above-mentioned first aspect or any corresponding embodiment thereof.

[0042] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the calculation method based on the gradient boosting algorithm of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 is a flow chart of a calculation method based on a gradient boosting algorithm according to an embodiment of the present invention;

[0045] Figure 2 is a flow chart of another calculation method based on the gradient boosting algorithm according to an embodiment of the present invention;

[0046] Figure 3 is a flow chart of another calculation method based on the gradient boosting algorithm according to an embodiment of the present invention;

[0047] Figure 4 is a schematic diagram of the structure of a distributed computing system according to an embodiment of the present invention;

[0048] Figure 5 is a schematic diagram of the operation of a Tracker component according to an embodiment of the present invention;

[0049] Figure 6 is a schematic diagram of the state flow of computing node tasks according to an embodiment of the present invention;

[0050] Figure 7 is a schematic diagram of waiting for a computing node failure to recover according to an embodiment of the present invention;

[0051] Figure 8 is a schematic diagram of local calculation result compression according to an embodiment of the present invention;

[0052] Fig. 9 is a flow chart of another calculation method based on the gradient boosting algorithm according to an embodiment of the present invention;

[0053] Fig.10 is a structural block diagram of a computing device based on a gradient boosting algorithm according to an embodiment of the present invention;

[0054] Fig.11 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0056] GBM (gradient boosting machine) is a decision tree algorithm that learns to select features and split points that maximize the benefit of the objective function by fitting data, and continuously generates new leaf nodes. After multiple iterations and multiple rounds of fitting, the machine learning algorithm fits the data to the maximum extent. Distributed computing clusters, also known as Allreduce clusters, are a distributed computing architecture widely used in large-scale machine learning platforms, divided into two stages: scatter and gather. The former aggregates the partial results of each node's shard data calculation through reduce calculations, while the latter keeps all nodes in a globally consistent state. Each node in the cluster calculates partial results of its own data shard, and forms a computing cluster through a ring or tree network topology. The global state is finally aggregated through the node's reduce calculation, and the global consistency state is finally consistent in all nodes. In the existing implementation of the GBM algorithm based on Allreduce, there is a lack of consideration for cluster stability. When the data scale is very large, the number of computing nodes will also increase accordingly, and the probability of node failure or network failure will also increase. How to quickly recover from a failure and prevent the failure from blocking the calculation for too long is of great significance to the search recommendation system of near-line computing.

[0057] The embodiment of the present invention provides a calculation method based on the gradient boosting algorithm, which manages the entire distributed computing cluster through the Tracker component, so that each abnormality in the calculation can be recovered at the minimum computing cost, without the need for a large amount of recovery calculations due to failures. It ensures the stable operation of the offline calculation of the recommendation system, reduces the risk of node failures, and can also make the computing cluster larger and improve the computing power.

[0058] According to an embodiment of the present invention, a computing method embodiment based on a gradient boosting machine algorithm is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0059] In this embodiment, a calculation method based on a gradient boosting algorithm is provided, which can be used in a distributed computing system, such as Figure 4 As shown, the distributed computing system includes a distributed computing cluster (hereinafter referred to as Allreduce cluster) and a Tracker cluster. The distributed computing cluster includes multiple computing nodes (computing nodes are referred to as Workers). Figure 1 is a flow chart of a calculation method based on a gradient boosting algorithm according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0060] Step S101, using the gradient boosting algorithm to control the Tracker cluster to manage the task status of multiple computing nodes and send task status instructions.

[0061] Specifically, the gradient boosting machine algorithm is referred to as the GBM algorithm. Figure 4 As shown in the figure, the Tracker cluster includes multiple Tracker components. The Tracker component manages the entire Allreduce cluster, including task status management, intermediate result storage replica routing, distributed computing cluster topology status, and corresponding node health status maintenance. The Tracker component is a component that guarantees strong consistency. Figure 4 Here, Worker represents a computing node. In this embodiment, multiple computing nodes are connected using a decision tree topology.

[0062] The GBM algorithm is used to control the Tracker component in the Tracker cluster to manage the task status of multiple computing nodes and send task status instructions to the computing nodes.

[0063] Step S102, when each computing node receives a task status instruction, a gradient boosting algorithm is used to control the computing node to perform distributed iterative calculations based on the task status instruction to obtain intermediate results, and the intermediate results are stored in the form of multiple copies, and the storage distribution of the multiple copies is recorded in the Tracker cluster.

[0064] Specifically, when each computing node receives a task status instruction, the gradient boosting algorithm is used to control the computing node to perform distributed iterative calculations based on the task status instruction to obtain intermediate results. Each Worker in the distributed computing cluster will maintain a heartbeat session with the Tracker component. The Tracker component will maintain the health status of the node in real time. Once the Worker is unavailable during the calculation process, the Tracker component will automatically complete the node switch between the old and new nodes. The new node only needs to register with the Tracker component to start distributed computing. Workers can automatically complete fault migration by being deployed on a containerized base such as k8s (short for Kubernetes, Kubernetes is an open source container cluster management system for automatically deploying, scaling, and managing containerized applications).

[0065] After the new node is started, there is no need to recalculate the completed calculations. With the solution of pre-storing intermediate results, after each partial calculation (a distributed iterative calculation) is completed, the Worker will store the intermediate results of the calculation into n copies. The distribution of the n copies is recorded in the Tracker component, and the node selection is performed according to the rack-aware mechanism (in the distributed storage system, by identifying the rack location of each node, the data storage and access strategy is optimized to adapt to the network topology). When a new node is started, if the current computing task is blocked on this node, it will request the Tracker component to obtain the latest copy of the intermediate result and directly read the latest intermediate result without re-iteration. This step will greatly speed up the recovery of offline calculations from failures.

[0066] Step S103, controlling the Tracker cluster to perform reduce calculations on the intermediate result copies of multiple computing nodes to obtain calculation results, and performing storage management on the calculation results.

[0067] Specifically, the working diagram of the Tracker component is as follows: Figure 5As shown in the figure, statesFSM represents the finite state machine, TRaftgroup represents the consistency algorithm, and replicasstore represents the storage of intermediate result replicas. The intermediate results include the global histogram of this round of Allreduce cluster distributed computing. partialGBDT (Partial Gradient Boosting Decision Tree) is an improved GBDT (Gradient Boosting Decision Tree) algorithm, which aims to solve the computing and storage problems that traditional GBDT algorithms may encounter when processing large-scale data. This is a state that is consistent across the entire cluster and will be referenced by subsequent iterative computing. In the scenario of trillion-scale features, this state is very large, so the storage of intermediate result replicas also supports division into fixed-size blocks as basic units.

[0068] In order to reduce the impact of failures on computing, this embodiment proposes a partially blocked Allreduce computing mode. Figure 7 As shown, when computing node h fails, the traditional Allreduce must wait until all partial results are ready before starting to run the reduce calculation (reduce calculation is used to accumulate all elements in an array into a single value through specified operations). Once a failure occurs in this mode, the reduce calculation will be blocked all the time. Reduce calculation in large-scale data scenarios is very time-consuming. Reduce calculation can actually meet the law of combination. The partial blocking calculation of the present invention allows the part that can obtain the result to perform the reduce calculation first, so there is no need to wait for the normal node to perform the reduce calculation. After the failed node completes the result output, it only needs to reduce the pre-calculated reduce result and the new output result again.

[0069] In the calculation of the GBM algorithm, Allreduce generally constructs a histogram of the gradient of the data sample and calculates the maximum split point benefit. The data sharding calculation results of each Worker node and even the size of the partial result data after reduce are relatively stable, so the copy storage overhead is acceptable for offline tasks. When an Allreduce process is completed, the GBM algorithm generates a new leaf node. In order to prevent the intermediate result copies generated by the past iteration from occupying too much space in the cluster, the present invention proposes to retain the intermediate result copies of the past T iterations in a time sliding window manner, and the remaining expired intermediate result copies will be recycled.

[0070] The calculation method based on the gradient boosting algorithm provided in this embodiment introduces a Tracker cluster to manage the entire distributed computing system, including task status management, intermediate result storage copy routing, and in order to avoid repeated calculations of intermediate results of past iterations lost due to node failures, a reliable copy storage and compression recovery strategy is proposed to ensure that the cluster will not have time-consuming repeated backtracking calculations, and to ensure that each calculation exception can be restored at the minimum computing cost, without a large number of recovery calculations due to failures, ensuring the stable operation of offline calculations of the recommendation system, reducing the risk of node failures, and making the computing cluster larger, improving computing power, and having stronger fault tolerance and faster fault detection and recovery capabilities in large-scale distributed cluster scenarios. In large-scale feature data recall and sorting scenarios, a large number of repeated calculations caused by failures are avoided, and offline computing tasks are avoided due to failures. Timeouts affect the timeliness of important business online data, and the problem that the distributed computing cluster cannot recover quickly from failures, resulting in long failure blocking calculation time, is solved.

[0071] In this embodiment, a calculation method based on a gradient boosting algorithm is provided, which can be used in a distributed computing system, such as Figure 4 As shown, the distributed computing system includes a distributed computing cluster, a Tracker cluster, and a monitor. The distributed computing cluster includes multiple computing nodes, the Tracker cluster includes multiple Tracker components, and the Tracker cluster and the monitor. Figure 2 is a flow chart of a calculation method based on a gradient boosting algorithm according to an embodiment of the present invention, such as Figure 2 As shown, the process includes the following steps:

[0072] Step S201, using a distributed consistency algorithm to maintain strong consistency of heartbeat sessions between multiple Tracker components and computing nodes.

[0073] Specifically, Figure 4 As shown, the Tracker component manages the task status of each computing node of the Allreduce cluster in the system. How the Allreduce cluster recovers, from which task to recover, and where to read the intermediate result copy all depend on this component, so the Tracker component must be a strong consistency system. The present invention adopts the Raft algorithm (distributed consistency algorithm) to ensure the strong consistency of the Tracker component.

[0074] Step S202, using the gradient boosting algorithm to control the Tracker cluster to manage the task status of multiple computing nodes and send task status instructions.

[0075] Specifically, the above step S202 includes:

[0076] Step a. Abstract the states of multiple computing nodes into a finite state machine model, use the gradient boosting algorithm to control the Tracker component to register the computing nodes and then send distributed iterative computing task instructions.

[0077] The states of multiple computing nodes can be abstracted into a FSM model (Finite State Machine). The gradient boosting algorithm is used to control the Tracker component to register the computing nodes and then send distributed iterative computing task instructions.

[0078] Step S203, when each computing node receives a task status instruction, the gradient boosting algorithm is used to control the computing node to perform distributed iterative calculations based on the task status instruction to obtain intermediate results, and the intermediate results are stored in the form of multiple copies, and the storage distribution of the multiple copies is recorded in the Tracker cluster.

[0079] Specifically, the Tracker component monitors the health status of all Workers, so it can also expose the corresponding monitoring API to integrate the monitoring system, so as to achieve monitoring and alarm, and improve the observability of the distributed system. The above step S203 includes:

[0080] Step S2031, when each computing node receives a task status instruction, a gradient boosting algorithm is used to control the computing node to perform distributed iterative calculations in the scatter phase and the gather phase based on the task status instruction to obtain an intermediate result.

[0081] When each computing node is normal, when each computing node receives the task status instruction, it performs distributed iterative calculations in the scatter phase and the gather phase according to the task status instruction, obtains the intermediate results, and stores the intermediate results in the form of multiple copies. The storage distribution of multiple copies is recorded in the Tracker cluster.

[0082] Step S2032, abstract the states of multiple computing nodes into a finite state machine model. When one of the computing nodes fails, the Tracker component corresponding to the failed computing node is used to replace the failed computing node as a new node, and the new node is registered and then a distributed iterative computing task instruction is sent.

[0083] Specifically, Figure 6As shown, the states of multiple computing nodes can be abstracted into a FSM model (Finite State Machine, finite state machine model). This embodiment is based on the Allreduce model, which is a tree structure (the simplest case is a binary tree). Whether a computing node is blocked depends on whether a subtree node in the tree structure is blocked, or whether a subtree is down and a computing node switch is in progress. Of course, the premise of entering the Pending state (the Pending state refers to a task or operation that is waiting to be processed or has not yet been completed) is that the computing node has completed the local partial histogram calculation and the reduce calculation of the histogram output by the available subtree. Figure 6 In the table, Running indicates the running state of the computing node, Pending indicates the waiting state of the computing node, Resuming indicates the resuming state of the computing node, subTree.state == PendingII indicates the condition for the subtree node to enter the Pending state, subTree.state == ResumingII indicates the condition for the subtree node to enter the Resuming state, subTree.state == dead indicates that the subtree node is in a blocked state, subTree.state == Running indicates the condition for the subtree node to enter the Running state, replicaLoaded&&subTree.state == Running indicates the condition for the intermediate result replica to be copied and loaded to enter the Running state, and autoRecovery indicates automatic recovery.

[0084] When a new node replaces a faulty node and goes online, it needs to load a copy of some intermediate results of the calculation. This copy of the intermediate result may be a copy of the intermediate result calculated by the faulty node, or it may be a copy of some intermediate results generated by other non-faulty nodes. After loading the data shards and intermediate calculation result copies required for this distributed iterative calculation, the random node can enter the normal operation state and the calculation task can continue. For the reverse gather calculation phase, the calculation node state flow changes from relying on the subtree node to relying on the parent node.

[0085] Step S2033, based on the distributed iterative computing task instruction, a new node is used to load the data shards and intermediate result copies required for the calculation, where the intermediate result copies are intermediate result copies calculated by the faulty node or intermediate result copies calculated by other non-faulty nodes.

[0086] Each node stores multiple copies of the intermediate results of the gather calculation, so once a failover occurs, the copies maintained by the original failed node need to be redistributed. The Tracker will select a suitable new host for the copies of the failed node after the failure occurs and ensure load balancing.

[0087] In an optional implementation, step S2033 includes:

[0088] Step a1: When the new node is started, it directly takes over all intermediate result copies of the corresponding load of the failed node, and selects the nearest computing node to complete the replication of the intermediate result copies.

[0089] Specifically, after the new node is started, the new node will take over all copies of the corresponding load of the old computing node and select the nearest node to complete the replication of the intermediate result copies.

[0090] or,

[0091] Step a2, assign the new node a higher weight than other computing nodes, directly select the new node according to the load weight strategy corresponding to the failed node to migrate the intermediate result copy and complete the intermediate result copy calculation.

[0092] Specifically, there is no need to worry about new nodes. You can directly select computing nodes according to the load weight strategy to migrate intermediate result copies. However, you can give newly started computing nodes a higher weight to ensure that after the new computing nodes are started and registered, they are more likely to be loaded with the intermediate result copies of the original load.

[0093] The two strategies recorded in step a1 and step a2 above are provided in a configurable manner. If the distributed computing cluster is deployed on a container orchestration base with second-level fault migration, the mode of step a1 can be enabled. This mode minimizes the metadata that needs to be modified by the Tracker component, which saves the cost of maintaining consistency for the Raft cluster (a Raft cluster is a distributed system managed using the Raft algorithm).

[0094] If the failover cannot be performed quickly, or even if manual operation and maintenance is required, it is recommended to use step a2 to minimize the node recovery time. After all, when the node scale is large, copying multiple replicas also takes a lot of time.

[0095] Step S2034, when the loading is completed, the new node is controlled to enter the normal distributed iterative computing state.

[0096] Specifically, when the computing node completes the gather calculation, it means that the intermediate result will be used as data input for subsequent iterations. Therefore, the computing node will send an RPC request (Remote Procedure Call, a mechanism that allows a program to call a procedure or function in another address space) to the Tracker component to obtain the target set that the replica needs to replicate. The Tracker will select the configured number of replica nodes based on the locally maintained topological results and the load of the node replica and return them to the Worker. After the Worker obtains the intermediate result replica set, it starts transmission and replication.

[0097] When a node fails and is replaced by a new node, the Tracker's replica routing strategy will allocate the node where the available replica is located according to the topological distance priority principle, and the new Worker will copy the intermediate result replica and start computing directly.

[0098] Each Worker calculates the histogram as a pure function. For consistent intermediate results and data sharding, the final calculation result must be certain.

[0099] In the present invention, the failure of a node will only block part of the calculation. Figure 7 As shown in the figure, after node h goes down, only the branch marked in red will be blocked, but the green branch can still perform reduce calculation. This is to simplify the binary tree scenario. If it is a multi-branch tree scenario, the efficiency will be improved even more efficiently.

[0100] Step S204: Control the Tracker cluster to perform reduce calculations on the intermediate result copies of multiple computing nodes to obtain calculation results, and store and manage the calculation results. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0101] The calculation method based on the gradient boosting machine algorithm provided in this embodiment adopts a distributed consistency algorithm to maintain the strong consistency of the heartbeat session between multiple Tracker components and computing nodes, ensuring that multiple Tracker components in the Tracker cluster in the distributed system form a strong consistency system, so that the time recovery when restoring the failed computing node, the recovery from the task stage, and the reading of the intermediate result copy can proceed smoothly. The gradient boosting machine algorithm (GBM algorithm, gradient boosting machine) is used to control the Tracker cluster to quickly recover normal calculations from the distributed computing cluster failure, reduce the blocking time of offline computing tasks, and ensure that each calculation exception can be recovered at the minimum computing cost, without a large number of recovery calculations caused by failures, ensuring the stable operation of offline calculations of the recommendation system. The Tracker component monitors the health of the entire distributed computing cluster, controls the progress of tasks, and manages automatic task recovery. The state machine model of the node can effectively control any fault state of the cluster to avoid faults. A large number of recovery calculations are also avoided by caching the calculation result copy mechanism.

[0102] In this embodiment, a calculation method based on a gradient boosting algorithm is provided, which can be used in a distributed computing system, such as Figure 4 As shown, the distributed computing system includes a distributed computing cluster and a Tracker cluster. The distributed computing cluster includes multiple computing nodes. Figure 3 is a flow chart of a calculation method based on a gradient boosting algorithm according to an embodiment of the present invention, such as Figure 3 As shown, the process includes the following steps:

[0103] Step S301: Use the gradient boosting algorithm to control the Tracker cluster to manage the task status of multiple computing nodes and send task status instructions. Figure 2 Step S202 of the illustrated embodiment will not be described in detail here.

[0104] Step S302: When each computing node receives the task status instruction, it uses the gradient boosting algorithm to control the computing node to perform distributed iterative calculations based on the task status instruction to obtain intermediate results, and stores the intermediate results in multiple copies. The storage distribution of the multiple copies is recorded in the Tracker cluster. For details, please refer to Figure 2 Step S203 of the illustrated embodiment will not be described in detail here.

[0105] Step S303, controlling the Tracker cluster to perform reduce calculations on the intermediate result copies of multiple computing nodes to obtain calculation results, and performing storage management on the calculation results.

[0106] Specifically, the above step S303 includes:

[0107] Step S3031, select a computing node according to the computing node selection principle, and use the intermediate result copy of each computing node as data input to perform reduce calculation to obtain the calculation result; the computing node selection principle is to select computing nodes with local storage, computing nodes with storage allowed by the same rack load, and computing nodes with storage allowed by different rack loads in turn.

[0108] like Figure 7 As shown in the figure, according to the computing node selection principle, computing nodes f and h that have the same rack load and upper computing node ① are selected for reduce calculation, and computing nodes 0, ①, and ② are selected for reduce calculation, and the overall calculation result is output by node 0. Figure 7 In the example, rackaware represents node selection, repliceStore represents replication operation, and ckptstorage outputs and stores.

[0109] Step S3032: When any of the selected computing nodes fails, reduce calculation is first performed on the computing nodes that have not failed to obtain the first reduce calculation result. After the failed node is replaced by a new node and recovered, the first reduce calculation result and the intermediate result copy of the new node are then reduce calculated to obtain the calculation result.

[0110] Specifically, Figure 7 As shown in the figure, when node h goes down, the branches of node h, node ① and node 0 are blocked. While waiting for node h to recover, node f and node ① are first reduced to obtain the first reduce calculation result. After node h recovers or is replaced by a new node, the first reduce calculation result is reduced with the intermediate result copy of node h or the new node to obtain the calculation result of node ①, node f and node h. Then, the calculation result of node ①, node f and node h is reduced with node ② until node 0 outputs the calculation result.

[0111] Step S3033, using a time sliding window method to filter out the intermediate result copies participating in the reduce calculation in each computing node for storage, and reclaim the memory space occupied by the intermediate result copies not participating in the reduce calculation in each computing node.

[0112] When an Allreduce process is completed, the GBM algorithm generates a new leaf node. In order to prevent the intermediate result copies generated by past iterations from occupying too much cluster space, the present invention proposes to retain the intermediate result copies of the past T iterations in a time sliding window manner, and the remaining expired copies will have their space reclaimed.

[0113] Since the intermediate results of the past T iterations are retained, there is no need to configure too many copies of the results of each iteration. The space overhead of the overall intermediate result copies is O(T*N), where N is the number of copies configured for each iteration. When all copies of iteration t are lost due to a failure, the calculation can still be performed from iteration t-1. According to the pure function property of the calculation, the result of iteration t can be calculated from the past calculation results. The possibility of recalculating all iterations due to the loss of copies caused by the failure of the computing node is almost zero.

[0114] Step S304, obtaining the spatial complexity of the partial histogram of each computing node in each iteration of the calculation result, the spatial complexity including the number of features and the number of bins of the partial histogram; using the gradient boosting algorithm to enumerate all the segmentation points in each partial histogram, and selecting a preset number of features before the maximum segmentation benefit to form a global histogram, that is, compressing the spatial complexity of the global histogram into the spatial complexity including the preset number of features and the number of bins before the maximum segmentation benefit.

[0115] Specifically, Figure 8 As shown in the figure, in the Worker calculation, the space complexity of each partial histogram is O(F*K), where F is the number of features and K is the number of bins (binary number) of the histogram. K is a constant that can be configured by the algorithm and can be adjusted in scale, but F depends on the feature dimension of the data, which is a very large number in large-scale recall sorting. If multiple copies of intermediate results are to be stored, the local calculation results must be compressed.

[0116] In addition to the conventional compression algorithm for file transmission, this embodiment adopts a calculation method optimized based on the characteristics of GBM.

[0117] The purpose of the GBM algorithm is to enumerate all the segmentation points of the gradient histogram composed of all the data, and select the position with the largest segmentation benefit as the result. However, each worker node only calculates part of the data shards. If all the data shards are independent and identically distributed, the topJ features with the largest candidate segmentation point benefits can be pre-sorted in the partial histogram of each node. In the final reduce stage, only the histogram of these J features is reduced. The scale of intermediate result storage is optimized from O(F*K) to O(J*K), and the space complexity also depends entirely on the configuration parameters. The larger the value of J, the higher the accuracy of the algorithm calculation, but the greater the storage overhead of the machine, and the calculation time will also increase. It can be configured according to different business scenarios. The schematic diagram of local calculation result compression is shown in the figure. Figure 8 As shown, slicedHistagram represents a partial histogram, Allreduce represents the completion of all reduce calculations, and globalstate represents the global histogram.

[0118] The calculation method based on the gradient boosting algorithm provided in this embodiment is a partially blocked distributed computing model, which efficiently utilizes the computing resources of the distributed computing cluster, that is, the part that can obtain the result is allowed to perform the reduce calculation first, and the normal computing nodes are allowed to perform the reduce calculation while waiting for the faulty node to recover. After the faulty node recovers and completes the result output, it is only necessary to reduce the pre-calculated reduce result and the new output result again, which can reduce the impact of the fault on the calculation, make the fault recovery time and the reduce calculation parallel, save the calculation time, and improve the computing power. The gradient boosting algorithm is used to optimize the spatial complexity of the intermediate result copy, ensure that the distributed computing cluster will not have time-consuming repeated backtracking calculations, and improve the computing power.

[0119] As one or more specific application embodiments of the present invention, Fig. 9 The calculation based on the gradient boosting algorithm provided by the present invention is further described in detail, specifically including:

[0120] The method of the present invention can be used in distributed computing systems, such as Figure 4 As shown, the distributed computing system includes a distributed computing cluster and a Tracker cluster. The distributed computing cluster includes multiple computing nodes, and the Tracker cluster includes multiple Tracker components, such as Fig. 9 As shown:

[0121] Step S1: The Tracker cluster decides which nodes need to wait for recovery and which nodes can continue computing without being blocked. The Tracker component manages the task status of each computing node in the Allreduce cluster in the system. How the Allreduce cluster recovers, from which task, and where to read the intermediate result replica all depend on this component. The Tracker component is a strongly consistent system.

[0122] The states of multiple computing nodes can be abstracted into a FSM model (Finite State Machine). The gradient boosting algorithm is used to control the Tracker component to register the computing nodes and then send distributed iterative computing task instructions.

[0123] When each computing node is normal, when each computing node receives the task status instruction, it performs distributed iterative calculations in the scatter phase and the gather phase according to the task status instruction, obtains the intermediate results, and stores the intermediate results in the form of multiple copies. The storage distribution of multiple copies is recorded in the Tracker cluster.

[0124] This embodiment is based on the Allreduce model, which is a tree structure (the simplest case is a binary tree). Whether a computing node is blocked depends on whether a subtree node in the tree structure is blocked, or whether a subtree is down and a computing node switch is in progress. Of course, the premise of entering the pending state (the pending state refers to a task or operation that is waiting to be processed or has not yet been completed) is that the computing node has completed the local histogram calculation and the reduce calculation of the histogram output by the available subtree.

[0125] Step S2: After the new node comes online, it needs to load the intermediate calculation results that need to be calculated according to the optimal strategy and complete the replica block migration work.

[0126] When a new node replaces a faulty node and comes online, it needs to load a copy of some intermediate results of the calculation. This copy of the intermediate result may be a copy of the intermediate result calculated by the faulty node, or it may be a copy of some intermediate results generated by other non-faulty nodes.

[0127] Step S3: The task can be automatically replied immediately after the data is ready, and the Tracker cluster randomly puts the blocked computing node into the running state.

[0128] After the data shards and intermediate calculation result copies required for this distributed iterative calculation are loaded, the random node can enter the normal operation state and the calculation task can continue. For the reverse gather calculation phase, the calculation node state flow changes from relying on the subtree node to relying on the parent node.

[0129] Step S4: After each Allreduce iteration is completed, the partial results of this iteration are stored in a multi-copy cluster in the dimension of blocks. The metadata is managed by the Tracker cluster.

[0130] The calculation method based on the gradient boosting algorithm provided in this embodiment ensures that each calculation exception can be recovered at the lowest computing cost, without the need for a large amount of recovery calculations due to failures. Ensuring the stable operation of offline calculations of the recommendation system reduces the risk of node failures and can also make the computing cluster larger and improve computing power. The present invention introduces a new Tracker component to manage the entire cluster, including managing the progress of tasks, to ensure that tasks can be quickly restored after the node is back online. In order to avoid repeated calculations of intermediate results of past iterations that are lost due to node failures, a reliable copy storage and compression recovery strategy is proposed to ensure that the cluster does not perform time-consuming repeated backtracking calculations. In order to avoid failures blocking the entire calculation process, the present invention designs a partial blocking mechanism to more efficiently utilize the computing resources of the cluster.

[0131] In the present embodiment, a computing device based on a gradient boosting algorithm is also provided, which is used to implement the above-mentioned embodiments and preferred implementations, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0132] This embodiment provides a computing device based on a gradient boosting algorithm, such as Fig.10 As shown, it is applied to a distributed computing system, the distributed computing system includes a distributed computing cluster and a Tracker cluster, the distributed computing cluster includes multiple computing nodes; the computing device includes:

[0133] The task status management module 1001 is used to use the gradient boosting algorithm to control the Tracker cluster to perform task status management on multiple computing nodes and send task status instructions.

[0134] The intermediate result copy calculation module 1002 is used to control the computing node to perform distributed iterative calculation based on the task status instruction to obtain the intermediate result when each computing node receives the task status instruction, and store the intermediate result in the form of multiple copies. The storage distribution of the multiple copies is recorded in the Tracker cluster.

[0135] The storage management module 1003 is used to control the Tracker cluster to perform reduce calculations on the intermediate result copies of multiple computing nodes to obtain calculation results, and to store and manage the calculation results.

[0136] In some optional implementations, the task status management module 1001 includes:

[0137] The states of multiple computing nodes are abstracted into a finite state machine model, and the gradient boosting algorithm is used to control the Tracker component to register the computing nodes and then send distributed iterative computing task instructions.

[0138] In some optional implementations, the intermediate result replica calculation module 1002 includes:

[0139] The intermediate result calculation unit is used to control the computing node to perform distributed iterative calculations in the scatter phase and the gather phase based on the task status instruction using the gradient boosting algorithm when each computing node receives the task status instruction to obtain the intermediate result.

[0140] The new node registration unit is used to use the Tracker component corresponding to the faulty computing node to replace the faulty computing node as a new node when one of the computing nodes fails, and send distributed iterative computing task instructions after registering the new node.

[0141] The intermediate result copy migration unit is used to adopt a new node to load the data shards and intermediate result copies required for calculation based on the distributed iterative computing task instructions. The intermediate result copies are intermediate result copies calculated by the faulty node or intermediate result copies calculated by other non-faulty nodes.

[0142] The new node distributed iterative computing unit is used to control the new node to enter the normal distributed iterative computing state after the loading is completed.

[0143] In some optional implementations, the intermediate result copy migration unit includes:

[0144] The intermediate result copy direct replication subunit is used to directly take over all intermediate result copies of the corresponding load of the failed node when the new node is started, and select the nearest computing node to complete the intermediate result copy replication.

[0145] The weight assignment subunit is used to assign a higher weight to the new node than other computing nodes, and directly select the new node to migrate the intermediate result copy according to the load weight strategy corresponding to the failed node to complete the intermediate result copy calculation.

[0146] In some optional implementations, the storage management module 1003 includes:

[0147] The node selection unit is used to select computing nodes according to the computing node selection principle, and use the intermediate result copy of each computing node as data input to perform reduce calculation to obtain the calculation result; the computing node selection principle is to select computing nodes with local storage, computing nodes with storage allowed by the same rack load, and computing nodes with different rack loads in turn.

[0148] The reduce computing unit is used to perform reduce calculation on the computing nodes that have not failed to obtain the first reduce calculation result when any computing node among the selected computing nodes fails, and then perform reduce calculation on the first reduce calculation result and the intermediate result copy of the new node to obtain the calculation result after the failed node is replaced and restored by a new node.

[0149] The storage unit is used to filter out the intermediate result copies participating in the reduce calculation in each computing node by using a time sliding window method for storage, and reclaim the memory space occupied by the intermediate result copies not participating in the reduce calculation in each computing node.

[0150] In some optional embodiments, the computing device based on the gradient boosting algorithm further includes:

[0151] The consistency module is used to maintain the strong consistency of heartbeat sessions between multiple Tracker components and computing nodes using a distributed consistency algorithm.

[0152] The compression module is used to obtain the spatial complexity of the partial histogram calculated by each computing node in each iteration of the calculation result, and the spatial complexity includes the number of features and the number of bins of the global histogram; the gradient boosting algorithm is used to enumerate all the segmentation points in each partial histogram, and the features of a preset number before the maximum segmentation benefit are selected to form a global histogram, that is, the spatial complexity of the global histogram is compressed to include the spatial complexity of the number of features and the number of bins of the preset number before the maximum segmentation benefit.

[0153] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0154] The computing device based on the gradient boosting algorithm in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0155] The embodiment of the present invention also provides a computer device having the above Fig.10A computing device based on the gradient boosting machine algorithm is shown.

[0156] See also Fig.11 , Fig.11 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.11 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.11 A processor 10 is taken as an example.

[0157] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0158] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0159] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0160] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0161] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0162] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0163] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0164] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0165] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A calculation method based on a gradient boosting algorithm, characterized in that: Applied in a distributed computing system, the distributed computing system includes a distributed computing cluster and a Tracker cluster, the distributed computing cluster includes multiple computing nodes; the method includes: Using a gradient boosting algorithm to control the Tracker cluster to perform task status management on the multiple computing nodes and send task status instructions; When each computing node receives a task status instruction, a gradient boosting algorithm is used to control the computing node to perform distributed iterative calculation based on the task status instruction to obtain an intermediate result, and the intermediate result is stored in the form of multiple copies, and the storage distribution of the multiple copies is recorded in the Tracker cluster; The Tracker cluster is controlled to perform reduce calculations on the intermediate result copies of multiple computing nodes to obtain calculation results, and the calculation results are stored and managed.

2. The method according to claim 1, characterized in that The Tracker cluster includes a plurality of Tracker components, each of which performs a heartbeat session with a computing node; the method further includes: A distributed consistency algorithm is used to maintain strong consistency in the heartbeat sessions between multiple Tracker components and computing nodes.

3. The method according to claim 2, characterized in that Using the gradient boosting algorithm to control the Tracker cluster to manage the task status of the multiple computing nodes, and sending the task status instruction includes: The states of the multiple computing nodes are abstracted into a finite state machine model, and a gradient boosting algorithm is used to control the Tracker component to register the computing nodes and then send distributed iterative computing task instructions.

4. The method according to claim 2, characterized in that: When each computing node receives a task status instruction, using a gradient boosting algorithm to control the computing node to perform distributed iterative calculations based on the task status instruction to obtain an intermediate result includes: When each computing node receives a task status instruction, a gradient boosting algorithm is used to control the computing node to perform distributed iterative calculations in the scatter phase and the gather phase based on the task status instruction to obtain an intermediate result; When one of the computing nodes fails, the Tracker component corresponding to the failed computing node is used to replace the failed computing node with a non-faulty computing node as a new node, and a distributed iterative computing task instruction is sent after the new node is registered; Based on the distributed iterative computing task instruction, a new node is used to load the data shards and intermediate result copies required for the calculation, where the intermediate result copies are intermediate result copies calculated by the faulty node or intermediate result copies calculated by other non-faulty nodes; When the loading is complete, the new node is controlled to enter the normal distributed iterative computing state.

5. The method according to claim 4, characterized in that The method of using a new node to load the data shards and intermediate result copies required for the calculation based on the distributed iterative computing task instruction includes: When the new node is started, it directly takes over all intermediate result copies of the corresponding load of the failed node, and selects the nearest computing node to complete the replication of the intermediate result copies; Alternatively, a higher weight is given to the new node than other computing nodes, and the new node is directly selected according to the load weight strategy corresponding to the failed node to migrate the intermediate result copy and complete the intermediate result copy calculation.

6. The method according to claim 2, characterized in that Controlling the Tracker cluster to perform reduce calculations on the intermediate result copies of each of the multiple computing nodes to obtain calculation results, and storing and managing the calculation results includes: Select computing nodes according to the computing node selection principle, and use the intermediate result copy of each computing node as data input to perform reduce calculation to obtain the calculation result; the computing node selection principle is to select computing nodes with local storage, computing nodes with storage allowed by the same rack load, and computing nodes with storage allowed by different rack loads in turn; or, When any of the selected computing nodes fails, reduce calculations are first performed on the computing nodes that have not failed to obtain the first reduce calculation result. After the failed node is replaced by a new node and restored, reduce calculations are then performed on the first reduce calculation result and the intermediate result copy of the new node to obtain the calculation result. The time sliding window method is used to filter out the intermediate result copies participating in the reduce calculation in each computing node for storage, and the memory space occupied by the intermediate result copies not participating in the reduce calculation in each computing node is reclaimed.

7. The method according to claim 1, characterized in that The intermediate result copy includes a partial histogram calculated by each computing node at each iteration; the method further includes: Obtain the spatial complexity of the partial histogram calculated by each computing node in each iteration of the calculation result, where the spatial complexity includes the number of features and the number of bins of the partial histogram; The gradient boosting machine algorithm is used to enumerate all the segmentation points in each partial histogram, and a preset number of features before the maximum segmentation benefit are selected to form a global histogram, that is, the spatial complexity of the global histogram is compressed to the spatial complexity of the number of features and the number of bins including the preset number of features before the maximum segmentation benefit.

8. A computing device based on a gradient boosting algorithm, characterized in that: Applied in a distributed computing system, the distributed computing system includes a distributed computing cluster and a Tracker cluster, the distributed computing cluster includes a plurality of computing nodes; the device includes: A task status management module, used to control the Tracker cluster to perform task status management on the multiple computing nodes using a gradient boosting algorithm and send task status instructions; An intermediate result replica calculation module is used for, when each computing node receives a task status instruction, using a gradient boosting algorithm to control the computing node to perform distributed iterative calculation based on the task status instruction to obtain an intermediate result, and storing the intermediate result in the form of multiple replicas, and the storage distribution of the multiple replicas is recorded in the Tracker cluster; The storage management module is used to control the Tracker cluster to perform reduce calculations on the intermediate result copies of multiple computing nodes to obtain calculation results, and to store and manage the calculation results.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the calculation method based on the gradient boosting algorithm according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the calculation method based on the gradient boosting machine algorithm according to any one of claims 1 to 7.

Images