Construction Method, Device and Electronic Equipment of Multi-Class Gradient Boosting Tree

Abstract

The present application provides a method and device for constructing a multi-class gradient boosting tree, including: a collaborating device obtains the gradients corresponding to each training sample in the training sample sets of each participating device; wherein, the training sample sets of each participating device constitute a total sample set; sort the training samples in the total sample set according to the gradient magnitude to obtain a sample sequence; determine the sample quantiles of the sample sequence, and determine multiple key samples in the total sample set based on the sample quantiles; determine the key samples belonging to each participating device among the multiple key samples, and determine the number of target non-key samples corresponding to each participating device; send the key samples and the number of target non-key samples to the corresponding participating devices, so that each participating device constructs a multi-class gradient boosting tree in cooperation with the collaborating device based on the key samples and the number of target non-key samples. In this way, the model is constructed through horizontal gradient sorting sampling, thereby reducing the overhead in multi-class tasks.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular, to a method, apparatus, electronic device, storage medium, and computer program product for constructing a multi-class gradient boosting tree. Background Art

[0002] In the field of federated learning in artificial intelligence, a model is jointly trained by a data provider and data users, and then each data user makes corresponding predictions based on its own model. Under the current general trend of gradually strengthening data privacy protection in all walks of life, federated learning, a technology that can collaborate on multi-party data to establish machine learning while protecting data privacy, has become one of the keys to cooperation among enterprises / industries. Nowadays, federated tree models: multi-class gradient boosting trees, random forests, and classification multi-class gradient boosting trees have been widely used as common and powerful algorithms in vertical federated scenarios, especially in the fields of finance and risk control where classification tasks are most frequently used. However, when constructing a horizontal federated multi-class gradient boosting tree, for horizontal federated scenarios with a large amount of data and a large number of labels, it is difficult to complete the construction of the horizontal federated multi-class gradient boosting tree within an appropriate time or at a low machine cost. The huge computational / communication overhead in the construction process of the horizontal federated multi-class gradient boosting tree brings many difficulties to the construction of the horizontal federated multi-class gradient boosting tree. Summary of the Invention

[0003] Embodiments of the present application provide a method, apparatus, electronic device, computer-readable storage medium, and computer program product for constructing a multi-class gradient boosting tree, which can perform model construction through horizontal gradient sorting sampling, thereby reducing the overhead in multi-class tasks.

[0004] The technical solution of the embodiments of the present application is implemented as follows:

[0005] Embodiments of the present application provide a method for constructing a multi-class gradient boosting tree,

[0006] Based on a federated learning system, the federated learning system includes a collaborating device and at least two participating devices, and the method includes:

[0007] The collaborating device obtains the gradients corresponding to each training sample in the training sample sets of each participating device; wherein, the training sample sets of each participating device constitute a total sample set;

[0008] Sort the training samples in the total sample set according to the gradient magnitude to obtain a sample sequence;

[0009] Determine the sample quantiles of the sample sequence, and determine multiple key samples in the total sample set based on the sample quantiles;

[0010] Determine the key samples in the multiple key samples that belong to each of the participating party devices, and determine the number of target non-key samples corresponding to each of the participating party devices;

[0011] Send the key samples and the number of target non-key samples to the corresponding participating party devices, so that each of the participating party devices collaborates with the cooperating party device to construct the multi-class gradient boosting tree based on the key samples and the number of target non-key samples.

[0012] An embodiment of the present application provides a device for constructing a multi-class gradient boosting tree, including:

[0013] An acquisition module, configured to acquire, for a cooperating party device, the gradient corresponding to each training sample in the training sample set of each participating party device; wherein, the training sample sets of each of the participating party devices constitute a total sample set;

[0014] A sorting module, configured to sort the training samples in the total sample set according to the gradient magnitude to obtain a sample sequence;

[0015] A first determination module, configured to determine the sample quantile of the sample sequence, and determine multiple key samples in the total sample set based on the sample quantile;

[0016] A second determination module, configured to determine the key samples in the multiple key samples that belong to each of the participating party devices, and determine the number of target non-key samples corresponding to each of the participating party devices;

[0017] A construction module, configured to send the key samples and the number of target non-key samples to the corresponding participating party devices, so that each of the participating party devices collaborates with the cooperating party device to construct the multi-class gradient boosting tree based on the key samples and the number of target non-key samples.

[0018] In the above solution, the first determination module is further configured to obtain a preset key sample ratio and the total number of samples in the total sample set; multiply the key sample ratio by the total number of samples to determine the number of key samples in the total sample set; determine the sample quantile of the sample sequence based on the number of key samples; select the number of training samples from the sample sequence based on the sample quantile as the multiple key samples.

[0019] In the above solution, the second determination module is further configured to obtain the first identifier corresponding to each training sample in the training sample set of each participating party device, and the second identifier corresponding to each key sample in the total sample set; match the first identifier with the second identifier, and determine the key samples in the multiple key samples that belong to each of the participating party devices based on the matching result.

[0020] In the above solution, the second determination module is further configured to obtain a preset target non-critical sample ratio and the total number of samples in the total sample set; subtract the number of critical samples from the total number of samples to determine the number of non-critical samples; multiply the target non-critical sample ratio by the number of non-critical samples to determine the total number of target non-critical samples; and determine the number of target non-critical samples corresponding to each of the participating party devices based on the total number of target non-critical samples.

[0021] In the above solution, the second determination module is further configured to screen out the multiple critical samples from the sample sequence to obtain a non-critical sample sequence; obtain the attribution relationship between the target non-critical samples and each of the participating party devices; and determine the number of target non-critical samples corresponding to each of the participating party devices based on the total number of target non-critical samples, the attribution relationship, and the non-critical sample sequence.

[0022] In the above solution, the second determination module is further configured to obtain the minimum value of the number of target non-critical samples corresponding to each of the participating party devices; and determine the number of target non-critical samples corresponding to each of the participating party devices based on the minimum value, the total number of training samples of each of the participating party devices, and the number of critical samples of each of the participating party devices.

[0023] In the above solution, the second determination module is further configured to obtain the importance degree of each of the participating party devices, and determine the target non-critical sample ratio corresponding to each of the participating party devices based on the importance degree; and determine the number of target non-critical samples corresponding to each of the participating party devices based on the target non-critical sample ratio corresponding to each of the participating party devices and the total number of target non-critical samples.

[0024] In the above solution, the construction module is further configured to receive the feature splitting gain histograms sent by each of the participating party devices, where the feature splitting gain histograms are obtained by each of the participating party devices determining a target training sample set based on the number of critical samples and target non-critical samples and traversing the splitting points of the multi-class gradient boosting tree based on the target training sample set; determine the gain value of the splitting point of the multi-class gradient boosting tree based on the feature splitting gain histograms, and select the splitting point with the largest gain value as the global splitting point; and construct the multi-class gradient boosting tree based on the global splitting point.

[0025] In the above solution, the obtaining module is further configured to enable a collaborating device to obtain multi-dimensional vectors sent by each participating device and used to represent the gradients corresponding to each training sample, where the number of dimensions of the multi-dimensional vector corresponds to the number of label categories of the multi-class gradient boosting tree, and the multi-dimensional vector is determined by each participating device based on the labels carried by each training sample in the training sample set and the corresponding predicted values.

[0026] In the above solution, the sorting module is further configured to add the gradients corresponding to each dimension in the multi-dimensional vector to obtain the sum of the gradients corresponding to each training sample in the training sample set; sort the training samples in the total sample set according to the magnitude of the sum of the gradients to obtain a sample sequence.

[0027] An embodiment of the present application provides an electronic device, including:

[0028] A memory for storing executable instructions;

[0029] A processor, when executing the executable instructions stored in the memory, implements the method for constructing a multi-class gradient boosting tree provided by the embodiment of the present application.

[0030] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to implement the method for constructing a multi-class gradient boosting tree provided by the embodiment of the present application when executed.

[0031] An embodiment of the present application provides a computer program product including a computer program that implements the method for constructing a multi-class gradient boosting tree provided by the embodiment of the present application when executed by a processor.

[0032] The embodiment of the present application has the following beneficial effects:

[0033] When constructing a multi-class gradient boosting tree, the collaborating device sorts the gradients of each training sample according to the magnitude of the gradients of the training samples provided by each participating device to obtain a sample sequence, and then divides the sample sequence based on the sample quantiles of the sample sequence to determine the number of key samples and target non-key samples in each participating device and sends them to the corresponding participating devices, so that each participating device collaborates with the collaborating device to construct a multi-class gradient boosting tree based on the number of key samples and target non-key samples. In this way, by performing horizontal gradient sorting sampling for model construction, the efficiency and overhead problems of horizontal multi-class modeling are effectively solved. Description of the Drawings

[0034] Figure 1 is a schematic diagram of an implementation scenario of a method for constructing a multi-class gradient boosting tree based on a federated learning system provided by an embodiment of the present application;

[0035] Figure 2 is a schematic structural diagram of the electronic device provided by an embodiment of the present application;

[0036] Figure 3 is a schematic flowchart of a method for constructing a multi-class gradient boosting tree provided by an embodiment of the present application;

[0037] Figure 4 is an optional schematic flowchart of the sample sorting process of the method for constructing a multi-class gradient boosting tree provided by an embodiment of the present application;

[0038] Figure 5 is an optional schematic flowchart of determining the number of target non-critical samples corresponding to each participating party device provided by an embodiment of the present application;

[0039] Figure 6 is an optional splitting point splitting schematic diagram of the splitting point provided by an embodiment of the present application;

[0040] Figure 7 is an optional schematic flowchart of the method for constructing a multi-class gradient boosting tree provided by an embodiment of the present application;

[0041] Figure 8 is a schematic structural diagram of the apparatus for constructing a multi-class gradient boosting tree provided by an embodiment of the present application. Detailed implementation manners

[0042] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be construed as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0043] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0044] In the following description, the terms "first / second / third" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms used herein are for the purpose of describing embodiments of this application only and are not intended to limit this application.

[0046] Before further elaborating on the embodiments of this application, the nouns and terms involved in the embodiments of this application are described. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0047] 1) A decision tree is a machine learning method. Specifically, it is a tree structure where each internal node represents a judgment on an attribute, each branch represents the output of a judgment result, and finally each leaf node represents a classification result.

[0048] 2) Federated machine learning (also known as federated learning, collaborative learning, or consortium learning) is a machine learning framework that can effectively help multiple institutions conduct data usage and machine learning model building while meeting the requirements of user privacy protection, data security, and government regulations. As a distributed machine learning paradigm, federated learning can effectively solve the data silo problem, enabling participants to jointly build models without sharing data, technically breaking data silos and achieving collaborative intelligence.

[0049] Federated Learning (also known as Federated Machine Learning) can be divided into three categories: Horizontal Federated Learning, Vertical Federated Learning, and Federated Transfer Learning.

[0050] Among them, Horizontal Federated Learning is also known as Feature-Aligned Federated Learning, that is, the data features of the participants in Horizontal Federated Learning are aligned, and it is applicable to the situation where the data features of the participants overlap more, while the sample identities (IDs) overlap less. Vertical Federated Learning is also known as Sample-Aligned Federated Learning, that is, the training samples of the participants in Vertical Federated Learning are aligned, and it is applicable to the situation where the training sample IDs of the participants overlap more, while the data features overlap less.

[0051] 3) DH key exchange. The two communicating parties interact information that can be made public through a public channel to generate a shared digital password for secure communication on the public channel, and this shared digital password can be used as the key for a symmetric cipher.

[0052] 4) Homomorphic Encryption (HE) is a symmetric encryption algorithm proposed by Craig Gentry of IBM in 2009. The purpose of homomorphic encryption is to find an encryption algorithm that can perform addition and multiplication operations on ciphertexts, such that the result obtained by performing a certain operation on the encrypted ciphertext is exactly equal to the ciphertext obtained by encrypting the plaintext after performing the expected operation on the plaintext before encryption. Homomorphic encryption effectively ensures that the data processing party can directly process the ciphertext of the data without being able to obtain the plaintext information of the data it processes. This feature of homomorphic encryption enables the security of users' data and privacy to be correspondingly guaranteed. Therefore, homomorphic encryption is applied to many real-world scenarios to ensure data security.

[0053] If an encryption function satisfies both additive homomorphy and multiplicative homomorphy, it is called fully homomorphic encryption. Using this encryption function, various encrypted operations (addition, subtraction, multiplication, division, polynomial evaluation, exponentiation, logarithm, trigonometric functions, etc.) can be completed.

[0054] The inventors found that in the currently widely used horizontal tree model, horizontal federated learning is one of the scenarios of federated learning, which is mostly used in model modeling between parent / child institutions or different departments of the same enterprise. Under horizontal federated learning, different institutions have different user samples, but the features of all samples are the same. When building a horizontal gradient boosting tree model, the classification algorithm is most commonly used. In the existing solutions, the binary classification algorithm can already meet the production needs of most cases. However, in the federated scenario, multi-class modeling has a very large overhead.

[0055] The existing horizontal federated gradient boosting trees adopt a multi-classification modeling strategy consistent with xgboost / lightgbm. That is, during training, a binary classification tree is trained separately for the first-order and second-order derivatives of each class. However, when the number of classes in the data increases, such as 10 classes or 20 classes, then 10 or 20 trees will be trained in each generation. As a result, the computational / communication overhead will become 10 or 20 times the original. In each generation, if each tree is trained serially, there will be a large time overhead. If they are trained in parallel, then the memory / communication requirements may become 10 / 20 times the original, posing high requirements for the machine performance of each participating party. For a very large dataset, this will be a heavy burden on the machine. When the dataset is large and the number of labels is large, the old solution will be difficult to complete the gradient boosting tree modeling within a reasonable time or at a low machine cost, which brings many difficulties to the tuning and iteration of the model.

[0056] Based on this, the embodiments of the present application provide a method, device, electronic device, computer-readable storage medium, and computer program product for constructing a multi-classification gradient boosting tree. It can build a model through horizontal gradient sorting sampling, complete training with only a small part of the original data with a small loss in effect, and at the same time, instead of specifically constructing a multi-classification gradient boosting tree for each label category, it uses a multi-output multi-classification gradient boosting tree to perform multi-classification tasks, thereby reducing the overhead in multi-classification tasks.

[0057] Based on the above explanations of the nouns and terms involved in the embodiments of the present application, the following describes the implementation scenario of the method for constructing a multi-classification gradient boosting tree based on a federated learning system provided by the embodiments of the present application. Refer to Figure 1 , Figure 1 FIG. is a schematic diagram of the implementation scenario of the method for constructing a multi-classification gradient boosting tree based on a federated learning system provided by the embodiments of the present application. To support an exemplary application, the collaborating device 400 is connected to the participating devices 200-1, 200-2, ……, 200-n through the network 300. Among them, the participating devices 200-1, 200-2, ……, 200-n can be institutions holding the features and corresponding labels of the training samples. For example, they can be hospitals, banks, shopping malls, or supermarkets. The collaborating device 400 can be an institution with credibility. The participating devices 200-1, 200-2, ……, 200-n and the collaborating device 400 assist each other in federated learning so that the participating devices 200-1, 200-2, ……, 200-n can obtain a multi-classification gradient boosting tree. The network 300 can be a wide area network or a local area network, or a combination of the two, and uses wireless or wired links to achieve data transmission.

[0058] The participant device (including participant devices 200-1, 200-2, ..., 200-n) is configured to send the gradient corresponding to each training sample in the training sample set to the collaboration device, wherein the training sample set of each participant device constitutes the total sample set;

[0059] The collaborating party (including the collaborating party device 400) is configured to obtain the gradient corresponding to each training sample in the training sample set of each participant device; sort the training samples in the total sample set according to the gradient size to obtain a sample sequence; determine the sample quantile of the sample sequence, and determine multiple key samples in the total sample set based on the sample quantile; determine the key samples belonging to each participant device among the multiple key samples, and determine the number of target non-key samples corresponding to each participant device; and send the number of key samples and target non-key samples to the corresponding participant device.

[0060] The participant devices (including participant devices 200-1, 200-2, ..., 200-n) are also used to collaborate with the collaborating device to build a multi-classification gradient boosting tree based on the number of key samples and target non-key samples.

[0061] In actual applications, the collaboration party device 400 and the participant devices 200-1, 200-2, ..., 200-n can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The collaboration party device 400 and the participant devices 200-1, 200-2, ..., 200-n can also be smart phones, tablets, laptops, desktop computers, smart speakers, smart watches, etc., but are not limited to these. The collaboration party device 400 and the participant devices 200-1, 200-2, ..., 200-n can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions on this.

[0062] The hardware structure of the electronic device for implementing the method for constructing a multi-classification gradient boosting tree provided in the embodiment of the present application is described in detail below. The electronic device includes but is not limited to a server or a terminal. Figure 2 , Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application, Figure 2The electronic device 200 shown includes: at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. Each component in the electronic device 200 is coupled together through a bus system 240. It can be understood that the bus system 240 is used to realize the connection and communication between these components. In addition to including a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 2 all kinds of buses are labeled as the bus system 240.

[0063] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0064] The user interface 230 includes one or more output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons, and controls.

[0065] The memory 250 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 optionally includes one or more storage devices that are physically located away from the processor 210.

[0066] The memory 250 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. The non-volatile memory can be a read-only memory (ROM, Read Only Memory), and the volatile memory can be a random access memory (RAM, Random Access Memory). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0067] In some embodiments, the memory 250 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, which are illustrated below.

[0068] An operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0069] A network communication module 252 for reaching other computing devices via one or more (wired or wireless) network interfaces 220. Exemplary network interfaces 220 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB), etc.;

[0070] An input processing module 253 for detecting and translating one or more user inputs or interactions from one of one or more input devices 232.

[0071] In some embodiments, the multi-class gradient boosting tree construction device provided by the embodiments of the present application can be implemented in software. Figure 2 Shown in the memory 250 is the multi-class gradient boosting tree construction device 254, which can be software in the form of a program and a plugin, etc., including the following software modules: an acquisition module 2541, a sorting module 2542, a first determination module 2543, a second determination module 2544, and a construction module 2545. These modules are logical, so they can be combined arbitrarily or further split according to the functions to be implemented. The functions of each module will be described below.

[0072] In other embodiments, the multi-class gradient boosting tree construction device provided by the embodiments of the present application can be implemented in a combination of software and hardware. As an example, the multi-class gradient boosting tree construction device provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the multi-class gradient boosting tree construction method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can employ one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), or other electronic components.

[0073] Based on the above description of the implementation scenarios of the multi-class gradient boosting tree construction method and the electronic device in the embodiments of the present application, the multi-class gradient boosting tree construction method provided by the embodiments of the present application will be described below. Refer to Figure 3 , Figure 3 which is a schematic flowchart of the multi-class gradient boosting tree construction method provided by the embodiments of the present application. The multi-class gradient boosting tree construction method provided by the embodiments of the present application includes:

[0074] Step 101, the collaborating device obtains the gradients corresponding to each training sample in the training sample sets of each participating device; wherein, the training sample sets of each participating device constitute the total sample set.

[0075] In actual implementation, the training samples are the local users held by each participating device respectively. Each participating device stores the features and corresponding labels of the local users, and the gradients corresponding to each training sample are obtained based on the difference between the predicted values obtained by each participating device according to the features of the local training samples and the corresponding labels.

[0076] Exemplarily, when the multi-class gradient boosting tree is applied to risk control management or risk prediction, the label of the training sample can be a multi-class label indicating the user's credit level, or a multi-class label indicating the user's loyalty level, or a multi-class label for evaluating the user's quality level. Here, each participating device is a platform for credit assessment of users, such as a financial loan platform (such as a bank). The second participating party can be a platform holding the user's feature data, such as various types of consumption platforms (such as shopping malls, supermarkets, online shopping platforms, game platforms, mobile application stores, etc.), or Internet companies, etc. Among them, the user's feature data can include but is not limited to the user's basic information, business data, behavior data, and device information, etc. The basic information includes but is not limited to the user's name, gender, age, education level, occupation, mobile phone number, ID number, address, income data (such as annual salary, monthly salary), etc. The business data includes data in multiple business scenarios. For example, when the business scenario is a loan business, the business data includes loan data and repayment data, etc. The loan data includes the loan amount, loan time, and loan institution (such as xx bank), etc. The repayment data includes the repayment amount, repayment time, repayment institution, and overdue repayment data (such as the number of overdue times and the number of overdue days), etc. For example, when the business scenario is a consumption scenario, the business data includes the consumption purpose (such as educational consumption, shopping consumption, medical consumption, living consumption, etc.), consumption amount, consumption time, consumption institution (such as shopping malls, supermarkets, convenience stores, online shopping stores, etc.), consumption method (such as credit card consumption or debit card consumption, etc.), and the address where the user is located during consumption, etc. The behavior data can be, for example, the payment institutions commonly used by the user (such as WeChat Pay, bank card payment, etc.), the payment methods commonly used by the user (such as QR code payment method, card swiping payment method), and the operation frequency (such as the frequency of payment operations implemented within a certain period of time), etc. The device information includes the model of the user's commonly used device, the address where the device is located when the user's behavior is implemented, etc.

[0077] In some embodiments, for the cooperating device to obtain the gradients corresponding to the training samples in the training sample sets of the participating devices, specifically, it obtains the multi-dimensional vectors sent by the participating devices for characterizing the gradients corresponding to the training samples. The number of dimensions of the multi-dimensional vector corresponds to the number of label categories of the multi-class gradient boosting tree, and the multi-dimensional vector is determined by each participating device based on the labels carried by the training samples in the training sample set and the corresponding predicted values.

[0078] It should be noted that, before using the training samples for model training, each participating device in the embodiments of the present application also performs feature binning on the data of the training samples. For example, the feature of age is binned into less than 30 years old, 30 - 50 years old, greater than 50 years old, etc. The interpretability of the model obtained through feature binning training is stronger.

[0079] In actual implementation, after obtaining the labels corresponding to the feature-binned data and the predicted values of the samples; each participating device constructs two multi-dimensional vectors of the gradients corresponding to the samples based on the labels carried by the training samples in the training sample set and the corresponding predicted values. Here, the process of constructing the two multi-dimensional vectors of the gradients corresponding to the samples is to compare the predicted values with the corresponding labels to obtain a comparison result; based on the comparison result, determine the first multi-dimensional vector and the second multi-dimensional vector of the training sample. Among them, the first multi-dimensional vector includes the first-order gradients of each label category of the training sample, and the second multi-dimensional vector includes the second-order gradients of each label category of the training sample.

[0080] In some embodiments, the predicted value of the first-generation tree is directly assigned as 0. Then, when constructing each generation of multi-class gradient boosting trees, for each sample, the predicted values and labels of the previous t - 1 generations of trees are used to calculate the gradients, and two d-dimensional vectors will be obtained. The first d-dimensional vector g corresponds to the first-order gradients of each label category, and the second d-dimensional vector h corresponds to the second-order gradients of each label category. Exemplarily, there are a total of t generations of trees to be constructed, where t is a positive integer greater than or equal to 1. Among them, the height of each multi-class gradient boosting tree is the same. Each participating device counts the number of labels and calculates the first-order gradient g and the second-order gradient h corresponding to each label based on the local labels and the predicted values of the previous t - 1 generations of trees; based on the calculated first-order gradients and second-order gradients, determine the first d-dimensional vector g[g1, g2, ……, g d and the second d-dimensional vector h[h1, h2, ……, h d .

[0081] In actual implementation, after determining two multi-dimensional vectors, the predicted value of the current node can be determined. In some embodiments, we use the subscript i to represent all samples in a certain leaf node. Then the predicted value of this node is w*, where the predicted value of each dimension is calculated according to the following formula (1) based on two d-dimensional vectors. After obtaining the predicted values of d dimensions, compare the predicted values of d dimensions and select the largest predicted value as the current predicted value.

[0082]

[0083] As an example, here we take the multi-class gradient boosting tree for three-class classification as an example. That is, d here is 3. Then the first 3-dimensional vector g is [g1, g2, g3], and the second 3-dimensional vector h is [h1, h2, h3]. The predicted value of each dimension is respectively:

[0084]

[0085] Then compare and in size, and select the largest one as the predicted value of the current node.

[0086] Step 102: Sort the training samples in the total sample set according to the gradient magnitude to obtain a sample sequence.

[0087] In some embodiments, after the cooperating device obtains the multi-dimensional vectors sent by each participating device to represent the gradients corresponding to each training sample, it sorts the training samples in the total sample set according to the obtained multi-dimensional vectors. For the sorting process to obtain the sample sequence, refer to Figure 4 , Figure 4 which is an optional flowchart of the sample sorting process of the multi-class gradient boosting tree construction method provided by the embodiments of the present application. Based on Figure 3 , step 102 can also be implemented in the following manner:

[0088] Step 1021: Add the gradients corresponding to each dimension in the multi-dimensional vector to obtain the sum of the gradients corresponding to each training sample in the training sample set.

[0089] In actual implementation, after the cooperating device obtains the two multi-dimensional vectors corresponding to the gradients of the training samples in the total sample set, that is, the first-order gradient g [g1, g2,..., g d and the second-order gradient h [h1, h2,..., h d , based on the first-order gradient g [g1, g2,..., g d , add the gradients corresponding to each dimension in the multi-dimensional vector to obtain the sum of the gradients corresponding to each training sample in the training sample set, that is, g_sum. Specifically, g_sum is equal to g1, g2,..., and gd The sum.

[0090] Continuing with the above example, taking the three-class multi-class gradient boosting tree as an example, the first 3D vector g is [g1, g2, g3], and the second 3D vector h is [h1, h2, h3]. Here, g_sum is equal to the sum of g1, g2, and g3.

[0091] It should be noted that when the gradients corresponding to the training samples in the total sample set are represented by multi-dimensional vectors, the process of adding the gradients corresponding to each dimension in the multi-dimensional vector to obtain the sum of the gradients corresponding to each training sample in the training sample set can also be implemented by each participating device. Specifically, based on the labels and corresponding predicted values carried by each training sample in the training sample set, each participating device constructs two multi-dimensional vectors of the first-order gradient g [g1, g2,..., g d and the second-order gradient h [h1, h2,..., h d . After that, directly add the gradients corresponding to each dimension in the first-order gradient g [g1, g2,..., g d to obtain the sum of the gradients corresponding to each training sample in the training sample set, that is, g_sum. Then send the obtained g_sum to the collaborative device for the collaborative device to sort the training samples in the total sample set based on the magnitude of the sum of the gradients, that is, g_sum, to obtain a sample sequence.

[0092] Step 1022: Sort the training samples in the total sample set according to the magnitude of the sum of the gradients to obtain a sample sequence.

[0093] In actual implementation, after the collaborative device determines the sum of the gradients corresponding to each training sample in the training sample set, it sorts the training samples in the total sample set in ascending order according to the magnitude of the sum of the gradients to obtain a sample sequence.

[0094] It should be noted that the smaller the sum of the gradients, the smaller the difference between the predicted value obtained based on the sample corresponding to the sum of the gradients and the corresponding label, and the better the training effect of the model.

[0095] Step 103: Determine the sample quantiles of the sample sequence and determine multiple key samples in the total sample set based on the sample quantiles.

[0096] In actual implementation, first obtain the preset key sample ratio and the total number of samples in the total sample set; multiply the key sample ratio by the total number of samples to determine the number of key samples in the total sample set; based on the number of key samples, determine the sample quantile of the sample sequence. Here, the key sample ratio is a preset and changeable ratio value, the total number of samples in the total sample set is the sum of the training samples provided by each participating party's device, and the sample quantile is used to divide the sample sequence according to the preset key sample ratio to select the key samples calculated based on the key sample ratio and the total number of samples.

[0097] As an example, the preset key sample ratio is 20%, and the total number of samples in the total sample set is 10 million. The collaborating party's device sorts the ten million samples according to the gradient g, and then takes out 2 million samples with the top 20% gradient rankings.

[0098] Step 104: Determine the key samples among the multiple key samples that belong to each participating party's device, and determine the number of target non-key samples corresponding to each participating party's device.

[0099] In actual implementation, the process of determining the key samples among the multiple key samples that belong to each participating party's device is specifically as follows: obtain the first identifier corresponding to each training sample in the training sample set of each participating party's device, and the second identifier corresponding to each key sample in the total sample set; match the first identifier with the second identifier, and based on the matching result, determine the key samples among the multiple key samples that belong to each participating party's device.

[0100] It should be noted that in order to facilitate each participating party's device to determine which of the training samples it provides belong to the key samples, each participating party's device will identify each training sample in its own training sample set, that is, the first identifier corresponding to each training sample in the training sample set of each participating party's device. After the collaborating party's device determines the multiple key samples in the total sample set, based on the second identifier corresponding to each key sample in the total sample set, it matches the first identifier with the second identifier, so as to determine the key samples among the multiple key samples that belong to each participating party's device based on the matching result.

[0101] In actual implementation, for the process of determining the number of target non-key samples corresponding to each participating party's device, see Figure 5 , Figure 5 which is an optional process schematic diagram provided by the embodiments of the present application for determining the number of target non-key samples corresponding to each participating party's device. Based on Figure 3 ,the process of determining the number of target non-key samples corresponding to each participating party's device can be implemented in the following manner:

[0102] Step 1041: Obtain the preset target non-key sample ratio and the total number of samples in the total sample set.

[0103] In actual implementation, while obtaining the preset key sample ratio and the total number of samples in the total sample set, the collaborating device also obtains the preset target non-key sample ratio.

[0104] Continuing with the above example, if the preset key sample ratio is 20% and the total number of samples in the total sample set is 10 million, then the preset target non-key sample ratio can be 10%.

[0105] Step 1042: Subtract the number of key samples from the total number of samples to determine the number of non-key samples.

[0106] In actual implementation, after determining the total number of samples and the number of key samples, subtract the number of key samples from the total number of samples to obtain the number of non-key samples.

[0107] Continuing with the above example, if the total number of samples in the total sample set is 10 million and the number of key samples is 2 million, then the number of non-key samples is 8 million.

[0108] Step 1043: Multiply the target non-key sample ratio by the number of non-key samples to determine the total number of target non-key samples.

[0109] In actual implementation, multiply the preset target non-key sample ratio by the obtained number of non-key samples to get the total number of target non-key samples.

[0110] Continuing with the above example, if the preset target non-key sample ratio is 10% and the number of non-key samples is 8 million, then the total number of target non-key samples is 8 million * 10% = 800,000.

[0111] Step 1044: Based on the total number of target non-key samples, determine the number of target non-key samples corresponding to each participating device.

[0112] In actual implementation, after determining the total number of target non-key samples, the collaborating device distributes the target non-key samples to each participating device, that is, it is necessary to determine the number of target non-key samples corresponding to each participating device.

[0113] Next, the process of determining the number of target non-key samples corresponding to each participating device will be described.

[0114] In some embodiments, the process of determining the number of target non-critical samples corresponding to each participating party device is specifically as follows: First, a plurality of critical samples are screened out from the sample sequence to obtain a non-critical sample sequence. Then, the attribution relationship between the target non-critical samples and each participating party device is obtained. Next, based on the total number of target non-critical samples, the attribution relationship, and the non-critical sample sequence, the number of target non-critical samples corresponding to each participating party device is determined.

[0115] In actual implementation, after determining the non-critical sample sequence, based on the number of target non-critical samples, the corresponding number of target non-critical samples are sequentially selected from the first position in the non-critical sample sequence. Then, the first identifier corresponding to each training sample in the training sample set of each participating party device and the third identifier corresponding to each target non-critical sample are obtained. The first identifier and the third identifier are matched, and based on the matching result, the target non-critical samples belonging to each participating party device among the target non-critical samples are determined.

[0116] It should be noted that, for the convenience of determining which of the training samples provided by itself belong to the target non-critical samples, each participating party device will identify each training sample in its own training sample set, that is, the first identifier corresponding to each training sample in the training sample set of each participating party device. After the cooperating party device determines a plurality of target non-critical samples in the sample total set, based on the third identifier corresponding to each target non-critical sample in the sample total set, the first identifier and the third identifier are matched, so as to determine the target non-critical samples belonging to each participating party device among the plurality of target non-critical samples based on the matching result.

[0117] In some embodiments, the process of determining the number of target non-critical samples corresponding to each participating party device can also be as follows: First, obtain the minimum value of the number of target non-critical samples corresponding to each participating party device. Then, based on the minimum value, the total amount of training samples of each participating party device, and the number of critical samples of each participating party device, determine the number of target non-critical samples corresponding to each participating party device.

[0118] In actual implementation, first, determine the priorities of the devices of each participating party. Here, the priority refers to the degree of influence of the device of the participating party on model training. The device of the participating party with a higher priority has a greater impact on model training, and the device of the participating party with a lower priority has a smaller impact on model training. After determining the priorities of the devices of each participating party, obtain the minimum value of the number of target non-critical samples corresponding to each participating party device. Then, based on the total number of training samples of each participating party device and the number of critical samples of each participating party device, determine the target non-critical samples corresponding to each participating party device. Specifically, based on the priorities of each participating party from high to low, first subtract the number of critical samples of each participating party device from the total number of training samples corresponding to the device of the participating party with the highest priority to determine the maximum number of target non-critical samples corresponding to this participating party device. Then, randomly select target non-critical samples corresponding to the maximum number of non-critical samples from the target non-critical samples. Here, in actual implementation, the device of the participating party with the highest priority can also randomly determine the number of corresponding target non-critical samples within the range of the corresponding minimum value and the maximum number of non-critical samples, so as to select target non-critical samples based on the corresponding number of target non-critical samples. In this way, after the device of the participating party with the highest priority selects the corresponding number of target non-critical samples, determine the number of remaining target non-critical samples for the devices of the participating parties with lower priorities to perform the same process of selecting target non-critical samples as above.

[0119] It should be noted that when the total number of samples in the sample total set is very large, it is also possible to set the maximum value of the number of target non-critical samples corresponding to each participating party device, and then randomly select target non-critical samples corresponding to the maximum value from the target non-critical samples; or randomly determine the number of corresponding target non-critical samples within the range of the corresponding minimum value and the maximum value, so as to select target non-critical samples based on the corresponding number of target non-critical samples.

[0120] In some embodiments, the process of determining the number of target non-critical samples corresponding to each participating party device can also be specifically as follows: First, obtain the importance degree of each participating party device, and determine the proportion of target non-critical samples corresponding to each participating party device based on the importance degree. Then, based on the proportion of target non-critical samples corresponding to each participating party device and the total number of target non-critical samples, determine the number of target non-critical samples corresponding to each participating party device.

[0121] In actual implementation, the importance of each participating party's device is determined by the number of key samples held by each participating party's device. The participating party's device with a larger number of key samples has a higher importance, and the participating party's device with a smaller number of key samples has a lower importance. By determining the proportion of the number of key samples held by each participating party's device among multiple key samples, the proportion of the target non-key samples corresponding to each participating party's device among the target non-key samples is determined. Then, the proportion of the target non-key samples corresponding to each participating party's device is multiplied by the total number of target non-key samples, so as to determine the number of target non-key samples corresponding to each participating party's device.

[0122] It should be noted that when the collaborative device randomly distributes the number of target non-key samples, it is necessary to ensure that the number of target non-key samples received by each participating party's device plus the number of its own key samples does not exceed the number of the local training sample set. At the same time, after determining the total number of target non-key samples, the collaborative device will randomly distribute the total number of target non-key samples to each participating party's device. The application embodiment does not limit the method of randomly distributing the number of target non-key samples to each participating party's device.

[0123] Step 105: Send the number of key samples and target non-key samples to the corresponding participating party's device, so that each participating party's device constructs a multi-class gradient boosting tree in cooperation with the collaborative device based on the number of key samples and target non-key samples.

[0124] In actual implementation, after determining the number of key samples and target non-key samples, the number of key samples and target non-key samples is sent to the corresponding participating party's device. At the same time, the participating party's device will also obtain the second identifier corresponding to each key sample, so as to determine the key samples in the provided training samples based on the second identifier. After determining the key samples in the provided training samples, the key samples are screened out from the training sample set to obtain the remaining training sample set. Then, based on the number of target non-key samples, the corresponding number of training samples is randomly selected from the remaining training sample set. Finally, based on the selected training samples and the determined key samples, a target training sample set is formed. It should be noted that the training samples in the target training sample set are samples with labels.

[0125] In actual implementation, the training samples in the target training sample set are input into the multi-class gradient boosting tree to obtain the predicted values of the corresponding training samples. Based on the predicted values and the corresponding labels, two multi-dimensional vectors used to represent the gradients corresponding to the training samples are determined. Here, the process of obtaining the two multi-dimensional vectors refers to Step 101.

[0126] In actual implementation, after each participating device obtains two multi-dimensional vectors used to represent the gradients corresponding to the training samples, based on the labels of the training samples and the multi-dimensional gradient vectors, it traverses the split points of the multi-class gradient boosting tree to determine the feature split gain histograms corresponding to each participating device. It should be noted that each bucket in each histogram contains two multi-dimensional vectors g_bin[sum of g1, sum of g2, ……, sum of g d sum], h_bin[sum of h1, sum of h2, ……, h d sum]. Here, the data of each dimension in g_bin and h_bin are the sum of the gradients of the corresponding dimensions in the multi-dimensional gradient vectors corresponding to all training samples, that is, the sum of g1 is the result of adding all the g1 of all training samples, the sum of g2 is the result of adding all the g2 of all training samples, and the sum of g d sum is the result of adding all the g d of all training samples, the sum of h1 is the result of adding all the h1 of all training samples, the sum of h2 is the result of adding all the h2 of all training samples, and h d sum is the result of adding all the h d of all training samples. It should be noted that here, because the features are the same among the participating devices, the dimensions of the calculated feature split gain histograms are also the same.

[0127] In actual implementation, after each participating device obtains the feature split gain histogram, it performs privacy protection on the feature split gain histogram to obtain an encrypted feature split gain histogram, and sends the encrypted feature split gain histogram to the collaborative device.

[0128] In some embodiments, after obtaining the feature split gain histogram, each participating device performs homomorphic encryption on the feature split gain histogram and sends the homomorphically encrypted feature split gain histogram to the collaborative device. In other embodiments, each participating device uses the DH key exchange technology, that is, uses the same random seed to generate random numbers with the same dimension as the feature split gain histogram, and assigns positive and negative signs to the random numbers through numbering. The feature split gain histogram is added to the generated random numbers to mask the original data. In this way, the feature split gain histogram is encrypted to obtain an encrypted feature split gain histogram, and the encrypted feature split gain histogram is sent to the collaborative device.

[0129] In actual implementation, after the collaborating device receives the encrypted feature split gain histograms sent by each participating device, for different encryption methods, it performs corresponding processing on the encrypted feature split gain histograms. Specifically, when the collaborating device receives the feature split gain histograms that have been homomorphically encrypted and sent by each participating device, it directly decrypts the encrypted feature split gain histograms, and then adds up the decrypted feature split gain histograms to obtain the aggregated feature split gain histogram; when the collaborating device receives the feature split gain histograms that have undergone the DH key exchange technology and sent by each participating device, it adds up the feature split gain histograms to cancel out the random numbers to obtain the aggregated feature split gain histogram.

[0130] In actual implementation, after the collaborating device obtains the aggregated feature split gain histogram, it determines the gain value of the split point of the multi-class gradient boosting tree based on the aggregated feature split gain histogram, and selects the split point with the largest gain value as the global split point. Here, the process of determining the gain value of the split point of the multi-class gradient boosting tree and selecting the split point with the largest gain value as the global split point can be based on a preset gain formula to determine the split point with the largest gain value.

[0131] The preset gain formula here is:

[0132]

[0133]

[0134] Here, in formula (2), and are to traverse the current split point, split the node according to a certain feature value, divide it into two to obtain a left son and a right son, and then calculate the gains of the left son and the right son based on formula (3). The g j and h j in formula (3) are the first-order gradient and the second-order gradient of the dimension corresponding to the largest predicted value selected in step 101. Specifically, substitute the data corresponding to each dimension in g_bin and h_bin into formula (1) in step 101, and by comparing the magnitudes, select the first-order gradient and the second-order gradient of the dimension corresponding to the largest predicted value.

[0135] In some embodiments, after determining the global split point, the collaborating device synchronizes the global split point to each participating device. Specifically, the process of each participating device constructing a multi-class gradient boosting tree based on the global split point may be that each participating device obtains the eigenvalue and feature encoding of the global split point sent by the collaborating device; determines the features of the training samples corresponding to the feature encoding, and determines the feature splitting method in combination with the features and eigenvalues; and constructs a multi-class gradient boosting tree based on the feature splitting method.

[0136] Exemplarily, referring to Figure 6 , Figure 6 is an optional split point splitting schematic diagram of the split point provided by the embodiments of the present application. Here, the sample data falling on the split point includes the corresponding sample data of samples 2, 5, 7, 8, and 10. The feature encoding corresponding to the global split point is 3, and the eigenvalue = 0.4. Here, after determining the global split point, the collaborating device sends the global split point in the form of a number to each participating device, and each participating device obtains the feature encoding and eigenvalue corresponding to the global split point based on the number, so as to determine the feature splitting method in combination with the features corresponding to the feature encoding and the eigenvalue, and thus construct a multi-class gradient boosting tree based on the feature splitting method. As Figure 6 shown, the left child node corresponding to the split point after splitting corresponds to samples 2, 5, and 10, and the right child node corresponds to samples 7 and 8.

[0137] In actual implementation, if the multi-class gradient boosting tree where the split point is located does not reach the stop splitting condition, continue to split until the stop splitting condition is reached, and then complete the construction of this generation of multi-class gradient boosting tree. Here, update the current predicted value using the output of the previous generation of tree, and each participating device continues to calculate the loss function to determine whether the loss function converges. If it does not converge, continue to construct a new multi-class gradient boosting tree until the loss function converges, or determine whether the number of multi-class gradient boosting trees reaches the preset number. If not, continue to construct a new multi-class gradient boosting tree until the preset number is reached. After the loss function converges or the number of multi-class gradient boosting trees reaches the preset number, complete the construction of the multi-class gradient boosting tree.

[0138] It should be noted that the stop splitting condition may be that the depth of the multi-class gradient boosting tree reaches the preset depth, the gain value of the multi-class gradient boosting tree reaches the preset gain value, or the number of samples does not meet the condition for continuing to split.

[0139] Thus, based on the predicted values and corresponding labels of the training samples, the gradient information of multiple categories of the training samples is represented in the form of a multi-dimensional vector, so as to process the multi-dimensional vector representing the gradient information, and determine the corresponding split point data based on the processing result, thereby determining the global split point of the multi-class gradient boosting tree, and then constructing the multi-class gradient boosting tree. Thus, by using a multi-class gradient boosting tree that can output multiple results to perform the multi-classification task, the overhead in the multi-classification task is effectively reduced, and the training efficiency is improved.

[0140] Applying the above embodiments of the present application, when constructing the multi-class gradient boosting tree, the collaborating device sorts the gradients of the training samples provided by each participating device according to the magnitude of the gradients of the training samples to obtain a sample sequence, and then divides the sample sequence based on the sample quantiles of the sample sequence to determine the number of key samples and target non-key samples in each participating device and sends them to the corresponding participating devices, so that each participating device collaborates with the collaborating device to construct the multi-class gradient boosting tree based on the number of key samples and target non-key samples. Thus, by performing model construction through horizontal gradient sorting sampling, the efficiency and overhead problems of horizontal multi-classification modeling are effectively solved.

[0141] Next, taking the application scenario of the multi-class gradient boosting tree as an example, the construction method of the multi-class gradient boosting tree provided by the embodiments of the present application will be introduced. Figure 7 is an optional flowchart of the construction method of the multi-class gradient boosting tree provided by the embodiments of the present application. Refer to Figure 7 , the construction method of the multi-class gradient boosting tree provided by the embodiments of the present application is jointly implemented by a collaborating device (Server) and at least two participating devices (Client). Both the collaborating device and the participating device can be a server or a terminal. Refer to Figure 7 , the construction method of the multi-class gradient boosting tree provided by the embodiments of the present application includes:

[0142] Step 201, each participating device respectively completes local initialization, bins the local features, and starts to jointly construct the first-generation multi-class gradient boosting tree.

[0143] Here, a total of t generations of trees need to be constructed, where t is a positive integer greater than or equal to 1. Among them, the height of each multi-class gradient boosting tree is the same.

[0144] In actual implementation, the process of binning the features of the training samples can be, for example, binning the feature of age into less than 30 years old, 30 - 50 years old, greater than 50 years old, etc. In this way, the interpretability of the model obtained through feature binning training is stronger.

[0145] Step 202: Obtain two multi-dimensional vectors g and h based on local labels and the predicted values of the previous t-1 generations of trees.

[0146] In actual implementation, each participating device calculates the first multi-dimensional vector g [g1, g2,..., g d and the second multi-dimensional vector h [h1, h2,..., h d for each sample using the residuals of the previous t-1 generations of trees according to local labels and samples, where the first multi-dimensional vector includes the first-order gradients of each label category of the training samples, and the second multi-dimensional vector includes the second-order gradients of each label category of the training samples.

[0147] In actual implementation, after determining the two multi-dimensional vectors, the predicted value of the current node can be determined. As an example, taking the three-class multi-class gradient boosting tree as an example, the first 3D vector g is [g1, g2, g3], and the second 3D vector h is [h1, h2, h{3]. The predicted values of each dimension are respectively:

[0148]

[0149] Then compare and to select the largest as the predicted value of the current node.

[0150] Step 203: Add the gradients corresponding to each dimension in the multi-dimensional vector g to obtain the sum of the gradients corresponding to each training sample in the training sample set.

[0151] In actual implementation, after each participating device obtains the two multi-dimensional vectors corresponding to the gradients of the training samples in the total sample set, that is, the first-order gradient g [g1, g2,..., g d and the second-order gradient h [h1, h2,..., h d , based on the first-order gradient g [g1, g2,..., g d , add the gradients corresponding to each dimension in the first-order gradient g to obtain the sum of the gradients corresponding to each training sample in the training sample set, that is, g_sum. Specifically, g_sum is equal to the sum of g1, g2,..., and g d .

[0152] Step 204: Send the sum of the gradients corresponding to each training sample in the obtained training sample set to the collaborative device.

[0153] Step 205: The collaborative device sorts the training samples in the total sample set according to the magnitude of the sum of the gradients to obtain a sample sequence.

[0154] In actual implementation, after determining the sum of gradients corresponding to each training sample in the training sample set, the training samples in the total sample set are sorted based on the magnitude of the sum of gradients in ascending order to obtain a sample sequence.

[0155] Step 206: Determine the sample quantiles of the sample sequence and determine multiple key samples in the total sample set based on the sample quantiles.

[0156] In actual implementation, first obtain the preset key sample ratio and the total number of samples in the total sample set; multiply the key sample ratio by the total number of samples to determine the number of key samples in the total sample set; determine the sample quantiles of the sample sequence based on the number of key samples. Here, the number of samples in the total sample set is the sum of the training samples provided by each participating party's device, and the sample quantiles are used to divide the sample sequence according to the preset key sample ratio to select the key samples calculated based on the key sample ratio and the total number of samples.

[0157] As an example, the preset key sample ratio is 20%, and the total number of samples in the total sample set is 10 million. The collaborating party's device sorts 10 million samples according to the gradient g, and then selects the top 2 million samples with the top 20% gradient rankings.

[0158] Step 207: Determine the key samples belonging to each participating party's device among the multiple key samples and determine the number of target non-key samples corresponding to each participating party's device.

[0159] In actual implementation, after determining the multiple key samples in the total sample set, obtain the first identifier corresponding to each training sample in the training sample set of each participating party's device and the second identifier corresponding to each key sample in the total sample set; match the first identifier with the second identifier and determine the key samples belonging to each participating party's device based on the matching result.

[0160] In actual implementation, after determining the key samples that belong to each participating party's device among multiple key samples, the number of target non-key samples corresponding to each participating party's device is determined. Specifically, the preset target non-key sample ratio and the total number of samples in the sample set are obtained, and then the total number of samples is subtracted from the number of key samples to determine the number of non-key samples. Then, the target non-key sample ratio is multiplied by the number of non-key samples to determine the total number of target non-key samples. Finally, based on the total number of target non-key samples, the number of target non-key samples corresponding to each participating party's device is determined. Here, it should be noted that after determining the total number of target non-key samples, the collaborative device will randomly allocate the total number of target non-key samples to each participating party's device. Here, the method of randomly allocating the total number of target non-key samples to each participating party's device is not limited in the embodiments of the present application.

[0161] Continuing with the above example, if the preset key sample ratio is 20% and the total number of samples in the sample set is 10 million, then the preset target non-key sample ratio can be 10%. Given that the total number of samples in the sample set is 10 million and the number of key samples is 2 million, the number of non-key samples is 8 million. Thus, the total number of target non-key samples is 8 million * 10% = 800,000. After determining the total number of target non-key samples, the collaborative device randomly allocates the total number of target non-key samples to each participating party's device.

[0162] It should be noted that when the collaborative device randomly allocates the total number of target non-key samples, it is necessary to ensure that the number of target non-key samples received by each participating party's device plus the number of its own key samples does not exceed the number of the local training sample set.

[0163] Step 208, the collaborative device sends the determined key samples of each participating party's device and the number of target non-key samples corresponding to each participating party's device to the corresponding participating party's device.

[0164] In actual implementation, after the collaborative party determines the number of key samples and target non-key samples, it sends the number of key samples and target non-key samples to the corresponding participating party's device. At the same time, the collaborative device also sends the second identifier corresponding to each key sample so that each participating party's device can determine the key samples in the provided training samples based on the second identifier.

[0165] Step 209, each participating party's device constructs a new training data set based on the number of key samples and target non-key samples.

[0166] In actual implementation, each collaborating device determines the key samples in the provided training samples based on the second identifier and the key samples sent by the collaborating party, then screens out the key samples from the training sample set to obtain the remaining training sample set, and then randomly selects the corresponding number of training samples from the remaining training sample set based on the number of target non-key samples. Finally, a new training data set is constructed based on the selected training samples and the determined key samples.

[0167] Step 210, each participating device initializes the root node based on the new training data set. All samples fall into the root node, are added to the node list, and the node list is synchronized to the collaborating device.

[0168] Step 211, determine the feature split gain histogram in each node based on the new training data set.

[0169] In actual implementation, the training samples in the target training sample set are input into the multi-class gradient boosting tree to obtain the predicted values of the corresponding training samples. Based on the predicted values and the corresponding labels, two multi-dimensional vectors used to represent the gradients corresponding to the training samples are determined. Here, the process of obtaining the two multi-dimensional vectors refers to Step 202.

[0170] In actual implementation, after each participating device obtains the two multi-dimensional vectors used to represent the gradients corresponding to the training samples, it traverses the split points of the multi-class gradient boosting tree based on the labels of the training samples and the multi-dimensional gradient vectors to determine the feature split gain histogram corresponding to each participating device. It should be noted that each bucket in each histogram contains two multi-dimensional vectors g_bin[sum of g1, sum of g2,..., sum of g d sum], h_bin[sum of h1, sum of h2,..., sum of h d sum]. Here, the data in each dimension of g_bin and h_bin is the sum of the gradients in the corresponding dimension of all the multi-dimensional gradient vectors corresponding to the training samples, that is, the sum of g1 is the result of adding all the g1 of the training samples, the sum of g2 is the result of adding all the g2 of the training samples, the sum of g d sum is the result of adding all the g d of the training samples, the sum of h1 is the result of adding all the h1 of the training samples, the sum of h2 is the result of adding all the h2 of the training samples, and the sum of h d sum is the result of adding all the h d of the training samples. It should be noted that here, because the features of each participating device are the same, the dimensions of the calculated feature split gain histograms are also the same.

[0171] Step 212, send the encrypted feature split gain histogram to the collaborating device.

[0172] In actual implementation, after obtaining the feature split gain histogram, each participating device uses the DH key exchange technology, that is, uses the same random seed to generate random numbers with the same dimension as the feature split gain histogram, and assigns positive and negative signs to the random numbers through numbering. The feature split gain histogram is added to the generated random numbers to mask the original data. In this way, the feature split gain histogram is encrypted to obtain an encrypted feature split gain histogram, and the encrypted feature split gain histogram is sent to the collaborative device.

[0173] Step 213: The collaborative device decrypts each received encrypted feature split gain histogram and aggregates the decrypted feature split gain histograms to obtain an aggregated feature split gain histogram.

[0174] In actual implementation, after receiving the feature split gain histograms encrypted by the DH key exchange technology sent by each participating device, the collaborative device adds the feature split gain histograms to cancel out the random numbers and obtains an aggregated feature split gain histogram.

[0175] Step 214: Take out nodes from the node list without replacement, and calculate the global split point at the current split node based on the aggregated feature split gain histogram.

[0176] In actual implementation, after obtaining the aggregated feature split gain histogram, the collaborative device traverses the current split point based on the aggregated feature split gain histogram, splits the node according to a certain feature value, and divides it into two to obtain a left son and a right son. The gain calculation formula for splitting according to this feature value is:

[0177]

[0178]

[0179] Here, g j and h j in formula (5) are the first-order gradient and second-order gradient of the dimension corresponding to the largest predicted value selected in step 202.

[0180] Continuing with the above example, taking the three-class multi-class gradient boosting tree as an example, substitute the data corresponding to each dimension in g_bin[sum of g1, sum of g2, sum of g3] and h_bin[sum of h1, sum of h2, sum of h3] into formula (1) in step 101, and by comparing the sizes, select the first-order gradient and second-order gradient of the dimension corresponding to the largest predicted value.

[0181] In actual implementation, the collaborative device determines the global split point by comparing the gains calculated by splitting according to different feature values.

[0182] Step 215: Send the global splitting points of each node in the current node list to each participating device.

[0183] In actual implementation, after determining the global splitting point, the collaborating device synchronizes the global splitting point to each participating device. Specifically, the process for each participating device to construct a multi-class gradient boosting tree based on the global splitting point can be that each participating device obtains the eigenvalue and feature encoding of the global splitting point sent by the collaborating device; determines the features of the training samples corresponding to the feature encoding, and combines the features and eigenvalues to determine the feature splitting method; and constructs a multi-class gradient boosting tree based on the feature splitting method.

[0184] Step 216: Determine whether the stopping splitting condition is met. If it is met, return to Step 210; if not, execute Step 217.

[0185] It should be noted that the splitting condition here is that the maximum tree height or gain value has not been reached, and the number of samples meets the requirement for continued splitting.

[0186] In actual implementation, if the splitting condition is met, split the current node based on the eigenvalue and feature encoding, split the current node, remove the node from the node list, add a new layer of nodes to the node list, and return to Step 214 to repeat until the current tree is constructed. If not, stop building the multi-class gradient boosting tree and execute Step 217.

[0187] Step 217: Update the current predicted value using the output of the previous generation tree.

[0188] Step 218: Detect whether the stopping condition is met. If not, the collaborating device and each participating device continue to jointly construct the multi-class gradient boosting tree and return to Step 202. If it is met, execute Step 219.

[0189] Here, the stopping condition includes whether the loss function of the multi-class gradient boosting tree converges or whether the maximum number of trees is reached.

[0190] Step 219: Complete the construction of the multi-class gradient boosting tree.

[0191] Applying the above embodiments of the present application, when constructing a multi-class gradient boosting tree, the collaborating device sorts the gradients of the training samples provided by each participating device according to the magnitudes of the gradients of the training samples to obtain a sample sequence, and then divides the sample sequence based on the sample quantiles of the sample sequence to determine the number of key samples and target non-key samples in each participating device and sends them to the corresponding participating devices, so that each participating device collaborates with the collaborating device to construct a multi-class gradient boosting tree based on the number of key samples and target non-key samples. In this way, by performing model construction through horizontal gradient sorting sampling, the efficiency and cost issues of horizontal multi-class modeling are effectively solved.

[0192] The following continues to describe the multi-class gradient boosting tree construction device 254 provided by the embodiments of the present application. Refer to Figure 8 , Figure 8 which is a schematic structural diagram of the multi-class gradient boosting tree construction device 254 provided by the embodiments of the present application. The multi-class gradient boosting tree construction device 254 provided by the embodiments of the present application includes:

[0193] An acquisition module 2541, configured to enable the collaborating device to acquire the gradients corresponding to each training sample in the training sample sets of each participating device; wherein, the training sample sets of each of the participating devices constitute a total sample set;

[0194] A sorting module 2542, configured to sort the training samples in the total sample set according to the magnitudes of the gradients to obtain a sample sequence;

[0195] A first determination module 2543, configured to determine the sample quantiles of the sample sequence and determine a plurality of key samples in the total sample set based on the sample quantiles;

[0196] A second determination module 2544, configured to determine the key samples belonging to each of the participating devices among the plurality of key samples and determine the number of target non-key samples corresponding to each of the participating devices;

[0197] A construction module 2545, configured to send the key samples and the number of target non-key samples to the corresponding participating devices, so that each of the participating devices collaborates with the collaborating device to construct the multi-class gradient boosting tree based on the key samples and the number of target non-key samples.

[0198] In some embodiments, the first determination module 2543 is further configured to obtain a preset critical sample ratio and the total number of samples in the total sample set; multiply the critical sample ratio by the total number of samples to determine the number of critical samples in the total sample set; determine the sample quantile of the sample sequence based on the number of critical samples; and select the number of training samples from the sample sequence based on the sample quantile as the multiple critical samples.

[0199] In some embodiments, the second determination module 2544 is further configured to obtain the first identifier corresponding to each training sample in the training sample set of each participating party device and the second identifier corresponding to each critical sample in the total sample set; match the first identifier with the second identifier, and determine the critical samples belonging to each participating party device among the multiple critical samples based on the matching result.

[0200] In some embodiments, the second determination module 2544 is further configured to obtain a preset target non-critical sample ratio and the total number of samples in the total sample set; subtract the number of critical samples from the total number of samples to determine the number of non-critical samples; multiply the target non-critical sample ratio by the number of non-critical samples to determine the total number of target non-critical samples; and determine the number of target non-critical samples corresponding to each participating party device based on the total number of target non-critical samples.

[0201] In some embodiments, the second determination module 2544 is further configured to screen out the multiple critical samples from the sample sequence to obtain a non-critical sample sequence; obtain the attribution relationship between the target non-critical samples and each participating party device; and determine the number of target non-critical samples corresponding to each participating party device based on the total number of target non-critical samples, the attribution relationship, and the non-critical sample sequence.

[0202] In some embodiments, the second determination module 2544 is further configured to obtain the minimum value of the number of target non-critical samples corresponding to each participating party device; and determine the number of target non-critical samples corresponding to each participating party device based on the minimum value, the total number of training samples of each participating party device, and the number of critical samples of each participating party device.

[0203] In some embodiments, the second determination module 2544 is further configured to obtain the importance level of each participating party device, and determine the target non-critical sample ratio corresponding to each participating party device based on the importance level; and determine the number of target non-critical samples corresponding to each participating party device based on the target non-critical sample ratio corresponding to each participating party device and the total number of target non-critical samples.

[0204] In some embodiments, the building block 2545 is further configured to receive the feature split gain histograms sent by each of the participating party devices. The feature split gain histograms are obtained by each of the participating party devices determining a target training sample set based on the number of the key samples and the target non-key samples, and traversing split points of the multi-class gradient boosting tree based on the target training sample set; determining gain values of the split points of the multi-class gradient boosting tree based on the feature split gain histograms, and selecting the split point with the largest gain value as the global split point; and constructing the multi-class gradient boosting tree based on the global split point.

[0205] In some embodiments, the obtaining module 2541 is further configured to obtain, for a cooperating party device, multi-dimensional vectors sent by each of the participating party devices for characterizing gradients corresponding to respective training samples, where the number of dimensions of the multi-dimensional vectors corresponds to the number of label categories of the multi-class gradient boosting tree, and the multi-dimensional vectors are determined by each of the participating party devices based on labels carried by respective training samples in the training sample set and corresponding predicted values.

[0206] In some embodiments, the sorting module 2542 is further configured to add gradients corresponding to each dimension in the multi-dimensional vectors to obtain the sum of gradients corresponding to respective training samples in the training sample set; and sort the training samples in the total sample set according to the magnitudes of the sums of gradients to obtain a sample sequence.

[0207] Applying the foregoing embodiments of the present application, when constructing a multi-class gradient boosting tree, a cooperating party device sorts gradients of respective training samples provided by each of the participating party devices according to the magnitudes of the gradients to obtain a sample sequence, and then divides the sample sequence based on sample quantiles of the sample sequence to determine the number of key samples and target non-key samples in each of the participating party devices and sends same to the corresponding participating party devices, so that each of the participating party devices collaborates with the cooperating party device to construct a multi-class gradient boosting tree based on the number of key samples and target non-key samples. In this way, model construction is performed through horizontal gradient sorting sampling, thereby effectively solving the problems of efficiency and overhead in horizontal multi-class modeling.

[0208] An embodiment of the present application further provides an electronic device, where the electronic device includes:

[0209] a memory for storing executable instructions;

[0210] a processor, configured to implement the method for constructing a multi-class gradient boosting tree provided in the embodiments of the present application when executing the executable instructions stored in the memory.

[0211] An embodiment of the present application provides a computer program product, including a computer program which, when executed by a processor, implements the method for constructing a multi-class gradient boosting tree provided by the embodiment of the present application.

[0212] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, where the executable instructions, when executed by a processor, cause the processor to execute the method for constructing a multi-class gradient boosting tree provided by the embodiment of the present application.

[0213] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0214] In some embodiments, the executable instructions may be in the form of a program, software, software module, script, or code, and may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0215] As an example, the executable instructions may or may not correspond to a file in the file system, may be stored as part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program being discussed, or stored in multiple cooperating files (for example, files storing one or more modules, subroutines, or code portions).

[0216] As an example, the executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network.

[0217] In summary, through the embodiment of the present application, model construction can be performed through horizontal gradient sorting sampling, thereby reducing the overhead in multi-classification tasks.

[0218] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A construction method of a multi-class gradient boosting tree, characterized in that, Based on a federated learning system, the federated learning system includes a collaborating device and at least two participating devices, and the method includes: The collaborating device obtains the gradients corresponding to each training sample in the training sample sets of the participating devices; wherein, the training sample sets of the participating devices constitute a total sample set; Sort the training samples in the total sample set according to the gradient magnitudes to obtain a sample sequence; Determine the sample quantiles of the sample sequence, and determine multiple key samples in the total sample set based on the sample quantiles; Determine the key samples in the multiple key samples that belong to each participating device, and determine the number of target non-key samples corresponding to each participating device; Send the key samples and the number of target non-key samples to the corresponding participating devices, so that each participating device collaborates with the collaborating device to construct the multi-class gradient boosting tree based on the key samples and the number of target non-key samples; Each participating device determines a target training sample set based on the key samples and the number of target non-key samples, and traverses the split points of the multi-class gradient boosting tree based on the target training sample set to obtain a feature split gain histogram; Each participating device encrypts the feature split gain histogram and sends the encrypted feature split gain histogram to the collaborating device; The collaborating device receives the encrypted feature split gain histograms sent by the participating devices and decrypts or aggregates the encrypted feature split gain histograms to obtain an aggregated feature split gain histogram; Based on the aggregated feature split gain histogram, determine the gain value of the split point of the multi-class gradient boosting tree, and select the split point with the largest gain value as the global split point; Based on the global split point, construct the multi-class gradient boosting tree.

2. The method according to claim 1, characterized in that, The determining the sample quantiles of the sample sequence and determining multiple key samples in the total sample set based on the sample quantiles includes: Obtain a preset key sample ratio and the total number of samples in the total sample set; Multiply the key sample ratio by the total number of samples to determine the number of key samples in the total sample set; Based on the number of key samples, determine the sample quantiles of the sample sequence; Based on the sample quantiles, select the number of training samples from the sample sequence as the multiple key samples.

3. The method according to claim 1, wherein The determining the key samples in the multiple key samples that belong to each participating device includes: Obtain the first identifiers corresponding to each training sample in the training sample sets of the participating devices, and the second identifiers corresponding to each key sample in the total sample set; Match the first identifiers with the second identifiers, and determine the key samples in the multiple key samples that belong to each participating device based on the matching results.

4. The method according to claim 1, wherein The determining the number of target non-key samples corresponding to each participating device includes: Obtain a preset target non-key sample ratio and the total number of samples in the total sample set; Subtract the total amount of the samples from the number of the key samples to determine the number of non - key samples; Multiply the target non - key sample ratio by the number of the non - key samples to determine the total number of the target non - key samples; Based on the total number of the target non - key samples, determine the number of the target non - key samples corresponding to each of the participating party devices.

5. The method according to claim 4, wherein The determining, based on the total number of the target non - key samples, the number of the target non - key samples corresponding to each of the participating party devices includes: Screen out the multiple key samples from the sample sequence to obtain a non - key sample sequence; Obtain the attribution relationship between the target non - key samples and each of the participating party devices; Based on the total number of the target non - key samples, the attribution relationship, and the non - key sample sequence, determine the number of the target non - key samples corresponding to each of the participating party devices.

6. The method according to claim 4, wherein The determining, based on the total number of the target non - key samples, the number of the target non - key samples corresponding to each of the participating party devices includes: Obtain the minimum value of the number of the target non - key samples corresponding to each of the participating party devices; According to the minimum value, the total amount of the training samples of each of the participating party devices, and the number of the key samples of each of the participating party devices, determine the number of the target non - key samples corresponding to each of the participating party devices.

7. The method according to claim 4, characterized in that, The determining, based on the total number of the target non - key samples, the number of the target non - key samples corresponding to each of the participating party devices includes: Obtain the importance degree of each of the participating party devices, and based on the importance degree, determine the ratio of the target non - key samples corresponding to each of the participating party devices; Based on the ratio of the target non - key samples corresponding to each of the participating party devices and the total number of the target non - key samples, determine the number of the target non - key samples corresponding to each of the participating party devices.

8. The method according to claim 1, wherein The collaborative party device obtains the gradients corresponding to each training sample in the training sample sets of each participating party device, including: The collaborative party device obtains the multi - dimensional vectors sent by each participating party device for representing the gradients corresponding to each training sample, where the number of dimensions of the multi - dimensional vector corresponds to the number of label categories of the multi - classification gradient boosting tree, and the multi - dimensional vector is determined by each participating party device based on the labels carried by each training sample in the training sample set and the corresponding predicted values.

9. The method according to claim 8, wherein The sorting, according to the gradient magnitudes, the training samples in the total sample set to obtain a sample sequence includes: Add the gradients corresponding to each dimension in the multi - dimensional vector to obtain the sum of the gradients corresponding to each training sample in the training sample set; Sort the training samples in the total sample set according to the magnitudes of the gradient sums to obtain a sample sequence.

10. An apparatus for constructing a multi-class gradient boosting tree, characterized in that, The device includes: An acquisition module, configured to enable the collaborative party device to obtain the gradients corresponding to each training sample in the training sample sets of each participating party device; wherein, the training sample sets of each participating party device constitute a total sample set; A sorting module, configured to sort the training samples in the total sample set according to the gradient magnitudes to obtain a sample sequence; A first determination module, configured to determine the sample quantiles of the sample sequence and determine multiple key samples in the sample population based on the sample quantiles; A second determination module, configured to determine the key samples belonging to each of the participating device and determine the number of target non-key samples corresponding to each of the participating devices; A construction module, configured to send the key samples and the number of target non-key samples to the corresponding participating devices, so that each of the participating devices collaborates with the collaborating device to construct the multi-class gradient boosting tree based on the key samples and the number of target non-key samples. Specifically, the construction module is configured to enable each of the participating devices to determine a target training sample set based on the key samples and the number of target non-key samples, and traverse the split points of the multi-class gradient boosting tree based on the target training sample set to obtain a feature split gain histogram; each of the participating devices encrypts the feature split gain histogram and sends the encrypted feature split gain histogram to the collaborating device; The collaborating device receives the encrypted feature split gain histograms sent by each of the participating devices, decrypts or aggregates the encrypted feature split gain histograms to obtain an aggregated feature split gain histogram; determines the gain value of the split point of the multi-class gradient boosting tree based on the aggregated feature split gain histogram, and selects the split point with the largest gain value as the global split point; constructs the multi-class gradient boosting tree based on the global split point.

11. An electronic device, characterized in that, The electronic device includes: A memory, configured to store executable instructions; A processor, configured to implement the method according to any one of claims 1 to 9 when executing the executable instructions stored in the memory.

12. A computer-readable storage medium, characterized in that, Stored with executable instructions, configured to implement the method according to any one of claims 1 to 9 when being executed by a processor.

13. A computer program product, comprising a computer program, characterized in that, The computer program, when being executed by a processor, implements the method according to any one of claims 1 to 9.

Images