Neural rule network continuous input feature binarization method and system capable of end-to-end optimization and application

Abstract

The invention discloses a neural rule network continuous input feature binarization method capable of end-to-end optimization, and the method comprises the following steps: obtaining interval centers of different intervals, and representing the interval centers in an increment manner; obtaining interval boundaries, setting boundaries on two sides according to the maximum value and the minimum value of a given feature, and setting an internal boundary by taking an interval center as a reference; a binary vector of the continuous numerical value features is obtained, the probability that a certain specific feature value of the given continuous feature belongs to each interval is calculated, the interval with the maximum probability is selected as the interval to which the specific feature value belongs, and then binary vector representation is obtained; selecting a clustering algorithm and setting a clustering loss function, so that the clustering loss function assists in optimizing interval division; and the STE method is adopted to directly take the gradient of the binary vector as the gradient of probability distribution, and end-to-end optimization of the interval division process is realized. The invention further discloses a system for implementing the method and application of the method or the system.

Description

Technical Field

[0001] The present invention belongs to the field of computer science and technology, and specifically relates to a method, system and application for binarizing continuous input features of a neural rule network that can be optimized end-to-end. Background Art

[0002] In recent years, neural rule networks have become a hot topic because they can use logical rules to equivalent fully connected neural networks. Unlike traditional deep learning models known as "black boxes", neural rule models can not only achieve high accuracy, but also be interpretable, and the model is more transparent inside, which makes it easier for humans to understand its learning nature. Neural rule models usually need to learn logical rules composed of Boolean variables. Therefore, it is particularly important to convert the input features of the model into binary representations. Discrete features can be directly converted through one-hot encoding, while continuous features cannot be directly discretized due to their continuous characteristics. They need to be discretized into binary representations by first dividing them into different numerical range intervals and then checking which intervals the feature values ​​meet.

[0003] However, the traditional method of dividing intervals is usually to divide directly based on experience, for example, to divide a number of equal-width intervals between the minimum and maximum eigenvalues ​​of a feature, and then fix the interval boundaries. This method of division has the following problems: 1) the interval division is not optimal due to the fixed interval boundaries; 2) the interval boundaries cannot be directly optimized according to the final training objectives. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a method for binarizing continuous input features of a neural rule network that can be optimized end-to-end. The core of the method is to set the position of the interval center in an incremental manner, and determine the interval boundary according to the interval center, so that the model can optimize the interval division by optimizing the increment. And the clustering loss is introduced to further optimize the interval division to ensure that the interval division is more in line with the distribution of the characteristic values. The present invention proposes a method for interval division that combines a neural rule network to optimize the interval center increment in an end-to-end manner, automatically binarizes the continuous input features of the neural rule model, takes into account the adaptability of the interval division and the final training goal, and can effectively divide reasonable intervals according to different training goals and different continuous features. The present invention enables the model to automatically learn interval division, effectively solving the defects in the prior art.

[0005] The method for binarizing continuous input features of a neural rule network that can be optimized end-to-end proposed by the present invention comprises the following steps:

[0006] Step 1: Get the interval center of different intervals and express the interval center in an incremental manner. In order to ensure the correctness of the final logic rule learning, the center obtained in this step is a pseudo center and is not at the center of the interval;

[0007] Step 2: Get the interval boundary, set the boundaries on both sides according to the maximum and minimum values ​​of the given feature, set the internal boundary based on the center of the interval, and define the interval boundary as the middle position between the two interval centers;

[0008] Step 3: Obtain a binary vector representation of the continuous numerical feature. For a specific eigenvalue of a given continuous numerical feature, calculate the probability that the specific eigenvalue belongs to each interval, and select the interval with the maximum probability as the interval to which the specific eigenvalue belongs, thereby obtaining a binary vector representation;

[0009] Step 4: Combined with clustering optimization interval division, different intervals can be regarded as the distribution of eigenvalues. Therefore, the center of the interval is regarded as the clustering center, and the clustering algorithm is selected and the clustering loss function is set. That is, the greater the probability that the eigenvalue belongs to a certain interval, the smaller the distance to the interval is, so that the clustering loss function assists in optimizing the division of intervals;

[0010] Step 5: Design an interval optimization method so that the interval partitioning process can achieve end-to-end optimization. Inputting the probability distribution of the relationship between the eigenvalue and the interval into the threshold function can obtain a binary vector, but the threshold function is not differentiable, so the straight-through estimator (STE) method is used to directly use the gradient of the binary vector as the gradient of the probability distribution in an approximate way to solve the end-to-end optimization problem.

[0011] When obtaining the interval center, assume that the interval center is obtained by ordinary partitioning. The number of intervals is K, and the leftmost interval boundary is b. min , the width of the kth interval is Then the kth interval is (b min +δ1+…+δ k-1 ,b min +δ1+…+δ k ], the center of the kth interval is b min +δ1+…+δ k-1 +δ k / 2. However, this division method has the following problem: given interval a = [1,2] and interval b = (2,6], for the eigenvalue 2.5, its distance from the center of interval a, 1.5, is smaller than its distance from the center of interval b, 4. Since the probability of an eigenvalue belonging to an interval is calculated based on its distance to the center of an interval, the probability that the eigenvalue 2.5 belongs to interval a is greater, but it is obvious that the eigenvalue 2.5 is within the range of interval b. In order to avoid the above problems, this method uses an incremental method to directly represent the position of the interval center, and then sets the interval boundary to the middle position of the two interval centers. Therefore, the interval center here is a pseudo-center, the purpose is to ensure the correctness of the final logical rule;

[0012] In the first step, obtaining the interval center includes the following steps:

[0013] a1. Denote the number of intervals as K, and the center of the k-th interval as

[0014] a2. Introduce an increment where Δ k represents the increment between the center of the k-th interval and the center of the (k - 1)-th interval, and the increment is the interval between the centers;

[0015] a3. Denote the left boundary value of the leftmost interval as b min , and the increment between the center of the first interval and the leftmost boundary is Δ1. Then the center of the first interval can be expressed as c1 = b min + Δ1. The increment between the center of the second interval and the center of the first interval is Δ2. Then the center of the second interval can be expressed as c2 = c1 + Δ2. Substituting the expression of c1, we get c2 = b min + Δ1 + Δ2;

[0016] a4. For the center of the k-th interval, where 1 ≤ k ≤ K, its position can be expressed as c k = b min + Δ1 + … + Δ k-1 + Δ k ;

[0017] In the second step, obtaining the interval boundaries includes the following steps:

[0018] b1. Divide according to the position. The whole includes three types of interval boundaries: the leftmost boundary, the rightmost boundary, and the boundaries between internal intervals;

[0019] b2. For the leftmost boundary, set it as the minimum eigenvalue that the continuous numerical feature can take, ensuring that there is no eigenvalue on the left side of the leftmost boundary;

[0020] b3. For the rightmost boundary, set it as the maximum eigenvalue that the continuous numerical feature can take, ensuring that there is no eigenvalue on the right side of the rightmost boundary;

[0021] b4. For the boundaries between internal intervals, taking the k-th interval as an example, where 1 < k < K, its left boundary is located at the middle position between the center of the (k - 1)-th interval and the center of the k-th interval. Since the increment of the center of the k-th interval relative to the center of the (k - 1)-th interval is Δ k , then the increment of the left boundary of the k-th interval relative to the center of the (k - 1)-th interval is Therefore, the left boundary of the k-th interval is expressed as b min + … + Δ k-1 + Δk / 2; the right boundary of the kth interval is located in the middle of the center of the kth interval and the center of the k+1th interval. Since the increment of the center of the k+1th interval relative to the center of the kth interval is Δ k+1 , then the increment of the right boundary of the kth interval relative to the center of the kth interval is Therefore, the right boundary of the kth interval is represented by b min +…+Δ k +Δ k+1 / 2. In summary, the kth interval is expressed as (b min +…+Δ k-1 +Δ k / 2, b min +…+Δ k +Δ k+1 / 2].

[0022] In step 3, calculating the probability that the characteristic value belongs to each interval includes the following steps:

[0023] c1. Let a specific eigenvalue of a continuous numerical feature be x, and calculate the distance from the eigenvalue x to the center of the kth interval using the distance calculation function D(·). Distance:

[0024] D(x,c k )=(xc k ) 2

[0025] c2. Calculate the probability that the eigenvalue x belongs to the kth interval based on the distance from the center of all intervals:

[0026]

[0027] Among them, τ1 is a hyperparameter used to scale the distance, thereby controlling the smoothness of the probability distribution. The larger τ1 is, the gap between different distances is magnified, and the distribution is sharper; the smaller τ1 is, the gap between different distances is reduced, and the distribution is smoother. k' and k both represent the index of the interval center, k' appears in the denominator, and k appears in the numerator. For each interval center k in the numerator, the denominator must be calculated K times.

[0028] In step 3, the process of obtaining a continuous eigenvalue binary vector includes the following steps:

[0029] c3. Let the continuous eigenvalue be x. Based on the probability distribution p of all intervals, select the interval with the maximum probability as the interval to which the eigenvalue x belongs.

[0030] c4. Let the binary vector be b, which consists of two values ​​0 and 1, and has a dimension of K. If the kth interval is the interval to which the eigenvalue x belongs, that is, the eigenvalue x is within the boundary range of the kth interval, then in the binary vector b, b k The value of is 1, and the values ​​of other positions are 0:

[0031]

[0032] The arg max operation is used to find the independent variable or subscript corresponding to the maximum value in the input. In this method, the input is a probability distribution p, and the probability that the eigenvalue belongs to the k'th interval is the largest, that is, p k' has the maximum value, then arg max(p) returns the index k'.

[0033] In step 4, the process of combining clustering optimization interval division includes the following steps:

[0034] d1. Set the cluster center. Clustering can reflect the distribution of all eigenvalues. Different intervals are regarded as different clusters, and the interval center is regarded as the cluster center, that is,

[0035] d2. Set the clustering objective function The clustering objective function should be the increment of the interval center Can be derived to optimize the interval partition and clustering objective function for:

[0036]

[0037] in, represents the probability that the eigenvalue x belongs to the kth interval, D(x,c k ) represents the eigenvalue x to the cluster center c k When optimizing the clustering loss function, if the probability that the eigenvalue x belongs to the kth interval is large, the loss function will reduce the distance from the eigenvalue x to the cluster center c. k distance.

[0038] In the step 5, the design interval optimization method includes the following steps:

[0039] e1. The interval division needs to be optimized in combination with the neural rule network. For the classification loss of the entire network, it is necessary to increase the loss and the interval Generate a back-propagation calculation path between them to achieve end-to-end optimization;

[0040] e2. Since the threshold function that converts the probability distribution p into a binary vector b involves the arg max operation, which is not differentiable, the straight-through estimator (STE) method is used to solve the gradient propagation problem in an approximate way:

[0041] b=p+Stop-grad(bp)

[0042] Stop-grad makes the forward propagation go through b and the backward propagation go through p. p represents the probability distribution of the eigenvalue belonging to each interval, and b represents the vector obtained after the probability distribution p is binarized.

[0043] The straight-through estimator (STE) method refers to a method of approximating the gradient, which directly uses the gradient of the output of the threshold function as the gradient of the input of the threshold function. By bypassing the threshold function, the problem that the gradient inside the threshold function cannot be back-propagated is solved. In this method, the probability distribution p is converted into a binary vector b through the threshold function. During back-propagation, the gradient of the binary vector b is first calculated, and then the gradient of b is directly passed to the probability distribution p. The threshold function in this method is as follows:

[0044]

[0045] The present invention also proposes a binarization system for implementing the above binarization method, the binarization system comprising: a differentiable binarization component, a differentiable clustering component, and a neural rule model component;

[0046] The differentiable binarization component is used to represent the interval center and the interval boundary in an incremental manner, calculate the probability distribution of the continuous numerical feature belonging to the interval corresponding to the continuous feature, convert the continuous numerical feature into a binary vector through a threshold function, and use a straight-through estimator to solve the gradient conduction problem between the probability distribution and the binary vector, so as to achieve optimizable interval division;

[0047] The derivable clustering component is used to regard the interval center as the cluster center, design the clustering loss function, directly optimize the interval center increment, and assist in interval division;

[0048] The neural rule model component is used to input the discrete binary vector of the processed continuous features and finally output an interpretable rule.

[0049] The present invention also proposes the application of the above binarization method or binarization system in binarization of continuous input features.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) A differentiable binarization method is used to binarize continuous features, and the interval partitioning automatic optimization of the neural rule model is realized in an end-to-end manner;

[0052] (2) Use clustering loss and specific task loss to jointly optimize the interval center, so that the learned interval is closer to the original eigenvalue distribution;

[0053] (3) It effectively solves the defects of artificially dividing intervals in the problem of binarizing continuous input features of neural rule models, such as the need to fix interval boundaries and the inability to optimize. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 It is a flow chart of the method for binarizing continuous input features of an end-to-end optimized neural rule model according to the present invention.

[0056] Figure 2 It is a framework diagram of the invention that can end-to-end optimize interval partitioning combined with neural rule network output rules.

[0057] Figure 3 It is a detailed structural diagram of the present invention.

[0058] Figure 4 This is the interval division of the continuous numerical feature age of the Adult data set by AutoInt of the present invention and other methods.

[0059] Figure 5 This is a performance comparison between the AutoInt of the present invention and other methods on multiple data sets, where the evaluation indicator is F1-score. DETAILED DESCRIPTION

[0060] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0061] The present invention provides a method for binarizing continuous features of a learnable neural rule model, the flow chart of which is as follows: Figure 1 As shown, the method comprises the following steps:

[0062] Step 1: Get the interval center of different intervals and express the interval center in an incremental manner. In order to ensure the correctness of the final logic rule learning, the center obtained in this step is a pseudo center and is not at the center of the interval;

[0063] Step 2: Get the interval boundary, set the boundaries on both sides according to the maximum and minimum values ​​of the given feature, set the internal boundary based on the center of the interval, and define the interval boundary as the middle position between the two interval centers;

[0064] Step 3: Obtain a binary vector representation of the continuous numerical feature. For a specific feature value of a given continuous numerical feature, calculate the probability that it belongs to each interval, and select the interval with the maximum probability as the interval to which it belongs, thereby obtaining a binary vector representation;

[0065] Step 4: Combined with clustering optimization interval division, different intervals can be regarded as the distribution of eigenvalues. Therefore, the center of the interval is regarded as the clustering center, and the clustering algorithm is selected and the clustering loss function is set. That is, the greater the probability that the eigenvalue belongs to a certain interval, the smaller the distance to the interval is, so that the clustering loss function assists in optimizing the division of intervals;

[0066] Step 5: Design an interval optimization method so that the interval partitioning process can achieve end-to-end optimization. Inputting the probability distribution of the relationship between the eigenvalue and the interval into the threshold function can obtain a binary vector, but the threshold function is not differentiable, so the straight-through estimator (STE) method is used to directly use the gradient of the binary vector as the gradient of the probability distribution in an approximate way to solve the end-to-end optimization problem.

[0067] More specifically, first, use the Module module in Pytorch to obtain the centers of all intervals, including the following steps:

[0068] a1. Set the number of intervals to K. Each interval contains an interval center. There are K interval centers in total, recorded as In practice, the value of K varies with the data set, and the possible values ​​are {5, 10, 15, 20, 30, 50};

[0069] a2. Introduce learnable parameters, namely increments, denoted as Where Δ k It represents the increment between the center of the kth interval and the center of the k-1th interval, and the increment is the interval between the centers of the intervals;

[0070] a3.Remember the left boundary value b of the leftmost interval min , the increment between the center of the first interval and the leftmost boundary is Δ1, then the center of the first interval can be expressed as c1=b min+Δ1. The increment between the center of the second interval and the center of the first interval is Δ2, then the center of the second interval can be expressed as c2 = c1 + Δ2, and further as c2 = b min +Δ1 + Δ2; and so on;

[0071] a4. For the center of the k-th interval, where 1 ≤ j ≤ K, its position can be expressed as c k = b min +Δ1 + … + Δ k-1 +Δ k ;

[0072] Using the Module module in Pytorch, based on the obtained interval centers, set the interval boundaries, including the following steps:

[0073] b1. Divide according to the position. The overall contains three types of interval boundaries, the leftmost boundary, the rightmost boundary, and the boundaries between internal intervals;

[0074] b2. For the leftmost boundary, set it to the minimum eigenvalue that the continuous numerical feature can take, ensuring that there is no eigenvalue to the left of the leftmost boundary;

[0075] b3. For the rightmost boundary, set it to the maximum eigenvalue that the continuous numerical feature can take, ensuring that there is no eigenvalue to the right of the rightmost boundary;

[0076] b4. For the boundaries between internal intervals, taking the k-th interval as an example, where 1 < k < K, its left boundary is located at the middle position between the center of the (k - 1)-th interval and the center of the k-th interval. Since the increment of the center of the k-th interval relative to the center of the (k - 1)-th interval is Δ k , then the distance between the left boundary of the k-th interval and the center of the (k - 1)-th interval is Therefore, the left boundary of the k-th interval is expressed as b min +… + Δ k-1 +Δ k / 2; The right boundary of the k-th interval is located at the middle position between the center of the k-th interval and the center of the (k + 1)-th interval. Since the increment of the center of the (k + 1)-th interval relative to the center of the k-th interval is Δ k+1 , then the increment of the right boundary of the k-th interval relative to the center of the k-th interval is Therefore, the right boundary of the k-th interval is expressed as b min +… + Δ k +Δ k+1 / 2;

[0077] Next, calculate the probability that the eigenvalue belongs to each interval corresponding to the continuous feature by the distance between the eigenvalue of each continuous feature of the sample and the interval center, including the following steps:

[0078] c1. Let the eigenvalue of a continuous numerical feature be x, and calculate the distance from the eigenvalue x to the center of the interval using the distance calculation function D(·). distance;

[0079] D(x,c k )=(xc k ) 2 ;

[0080] c2. Calculate the probability that the eigenvalue x belongs to the kth interval based on the distance from the center of all intervals:

[0081]

[0082] Where τ1 is a hyperparameter used to scale the distance, thereby controlling the smoothness of the probability distribution. In practice, the value of τ1 is set to 1000.

[0083] Based on the probability distribution p of the eigenvalues ​​belonging to different intervals, obtaining its binary vector b includes the following steps:

[0084] c3. Based on the probability distribution p of the eigenvalue x belonging to different intervals, select the interval with the maximum probability as the interval to which the eigenvalue x belongs;

[0085] c4. The binary vector b consists of two values, 0 and 1, and has a dimension of K. If the kth interval is the interval to which the eigenvalue x belongs, that is, the eigenvalue x is within the boundary range of the kth interval, then in the binary vector b, b k The value of is 1, and the rest are 0:

[0086]

[0087] Combined with the use of clustering algorithms, the clustering process is applied to the interval center to further optimize the interval division, including the following steps:

[0088] d1. Get the cluster center. Clustering can reflect the distribution of the overall value. Different intervals are regarded as different clusters, and the interval center represents the cluster center, that is,

[0089] d2. Set the clustering objective function. The clustering objective function should be the increment of the interval center The present invention uses the soft K-means algorithm (the present invention can also use other clustering algorithms that are incrementally differentiable for the center of the interval), and the clustering objective function is as follows:

[0090]

[0091] in, represents the probability that the eigenvalue x belongs to the kth interval, D(x,c k ) represents the eigenvalue x to the cluster center c k When optimizing the clustering loss function, if the probability that the eigenvalue x belongs to the kth interval is large, the loss function will reduce the distance from the eigenvalue x to the cluster center c. k distance.

[0092] Using the autograd mechanism provided by pytorch (the autograd mechanism automatically calculates the gradient according to the input and the computational graph, and stores the gradient in the .grad attribute of the tensor), an interval optimization method is designed, including the following steps:

[0093] e1. Select the neural rule model;

[0094] e2. Select a data set and process the continuous features of each sample into discrete binary vectors using this method;

[0095] e3. Input the binary vector of continuous features into the neural rule model to obtain the classification loss, which is expressed as

[0096] e4. Since this method converts the probability distribution of the interval corresponding to the continuous feature into a binary vector, the threshold function used involves the non-differentiable argmax operation, resulting in It is not possible to directly optimize the interval center increment Therefore, the straight-through estimator (STE) method is used to solve the gradient propagation problem in an approximate way:

[0097] b=p+Stop-grad(bp)

[0098] Stop-grad makes the forward propagation go through b and the backward propagation go through p. p represents the probability distribution of the eigenvalue belonging to each interval, and b represents the vector obtained after the probability distribution p is binarized.

[0099] The argmax operation is used to find the independent variable or subscript corresponding to the maximum value in the input. In this method, the input is a probability distribution p, and the probability that the eigenvalue belongs to the k'th interval is the largest, that is, p k' has the maximum value, then arg max(p) returns the index k'.

[0100] The straight-through estimator (STE) method refers to a method of approximating the gradient, which directly uses the gradient of the output of the threshold function as the gradient of the input of the threshold function. By bypassing the threshold function, the problem that the gradient inside the threshold function cannot be back-propagated is solved. In this method, the probability distribution p is converted into a binary vector b through the threshold function. During back-propagation, the gradient of the binary vector b is first calculated, and then the gradient of b is directly passed to the probability distribution p. The threshold function in this method is as follows:

[0101]

[0102] e5. Considering the effects of clustering loss and classification loss, construct a mixed loss:

[0103]

[0104] in is the classification loss, is the clustering loss, Represents the mixed loss of the model. The hyperparameter λ is used to adjust the contribution of clustering loss to the optimization process. When using the adult dataset, the value of λ is set to 0.1.

[0105] The present invention can optimize the interval division end-to-end and combine the framework diagram of the neural rule network output rule, such as Figure 3 As shown, the entire framework includes the following components:

[0106] f1. Differentiable binary component: represents the interval center and interval boundary in an incremental manner, calculates the probability distribution of the continuous numerical feature belonging to the interval corresponding to the continuous feature, converts it into a binary vector through a threshold function, and uses a straight-through estimator to solve the gradient conduction problem between the probability distribution and the binary vector, thus achieving optimizable interval partitioning;

[0107] f2. Derivable clustering component: The center of the interval is regarded as the cluster center, and the clustering loss function is designed to reduce the distance from the continuous numerical feature to the interval with large probability. This loss function can directly optimize the interval center increment and assist in interval division;

[0108] f3. Neural rule model component: You can choose any neural rule model that requires the input to be discrete, use the discrete binary vector of continuous features processed by this method as part of the input, and finally output an interpretable rule.

[0109] In the present invention, Figure 3As shown, the differentiable binarization and differentiable clustering are the main innovations of the present invention. The differentiable binarization part represents the interval center and interval boundary in the form of increments, and obtains the probability distribution of the interval by calculating the distance from the continuous input feature to the interval corresponding to the continuous feature, and then converts it into a binary vector, and uses the straight-through estimator (STE) to solve the gradient propagation problem between the probability distribution and the binary vector. The differentiable clustering part designs a clustering loss function, which can directly optimize the interval center increment, thereby assisting in optimizing the interval division. The overall loss function consists of two parts, namely the classification loss and the clustering loss.

[0110] Figure 2 A comparison chart between this method and other methods is given. Figure 2 The method in the lower left box represents a fixed interval partition and therefore cannot be optimized by backpropagation. Figure 2 The method in the lower right corner box is the method proposed by the present invention. This method solves the gradient conduction problem and can therefore be optimized through back propagation. The vectors binarized by different interval division methods are input into the neural rule model to output interpretable rules.

[0111] In the step of designing the interval optimization method in the present invention, it is necessary to input the processed features into the neural rule model to calculate the loss. In addition to continuous numerical features, the features here generally also use discrete features, such as gender, job category and other features in the adult data set. For continuous numerical features, the method in the present invention can be used to convert them into discrete binary vectors; for discrete features (category features), the OneHotEncoder method provided by scikit-learn can be directly used to convert them into one-hot vectors (OneHotEncoder is used to convert each element of the classification feature into a directly calculable value, that is, digitize the feature value). The processed continuous numerical features and discrete vectors of discrete features are spliced ​​together as the input of the neural rule model.

[0112] The parameters mentioned in the present invention are determined based on experimental results, that is, different parameter combinations are tested, a set of parameters with better evaluation indicators on the validation set is selected, and the results are evaluated on the test set. In actual testing, the above parameters can be appropriately adjusted according to needs to achieve the purpose of the present invention.

[0113] The data sets that can be used in the present invention include but are not limited to adult data sets, bank data sets, credit data sets, redwineQ data sets, whitewineQ data sets, wine data sets, etc.

[0114] Example

[0115] The Adult dataset is a commonly used dataset for classification tasks. Based on a person's features, it predicts whether his annual income exceeds 50k$. It contains 6 continuous numerical features and 8 discrete features. Next, this method is used to binarize the age features in the Adult dataset, and the binarization results are input into the neural rule model RRL for classification. The specific steps are as follows:

[0116] First, check the eigenvalue distribution of the age feature and initialize the interval center, including the following steps:

[0117] a1. Set the number of intervals K to 13, that is, for a given specific age value, a 13-dimensional one-hot vector will be obtained after binarization;

[0118] a2. Set the smallest age in the age feature as the leftmost boundary of the interval, i.e. b min =20; set the largest age in the age feature as the rightmost boundary of the interval, i.e. b max =75;

[0119] a3. The center of the kth interval is The center of the interval is a specific age value within the age interval;

[0120] a4. Introduce increments Where Δ1 represents the center of the first interval relative to the left boundary b min , Δ2 represents the distance between the center of the second interval and the center of the first interval. Therefore, the center of the kth interval c k The increment is expressed as c k =b min +Δ1+…+Δ k-1 +Δ k ;

[0121] Next, based on the obtained interval center, set the interval boundary, including the following steps:

[0122] b1. Leftmost boundary b min Set to 20, the youngest age that appears in the age feature;

[0123] b2. rightmost boundary b max Set to 75, the largest age that appears in the age feature;

[0124] b3. For internal boundaries That is, the boundary between the center of the k+1th interval and the center of the kth interval is located in the middle of the two interval centers and is represented by b k =b min +…+Δ k +Δ k+1 / 2;

[0125] Based on the obtained interval center, calculate the probability that the given age feature value x belongs to each interval:

[0126] c1. Use the distance function D(x,c k )=(xc k ) 2 Calculate the distance from age x to the center of K = 13 intervals;

[0127] c2. According to the distance from age x to the center of each interval, calculate the probability that it belongs to each interval. The probability that age x belongs to the kth interval is:

[0128]

[0129] Among them, τ1 is a hyperparameter used to scale the distance, thereby controlling the smoothness of the probability distribution. The larger τ1 is, the gap between different distances is magnified, and the distribution is sharper; the smaller τ1 is, the gap between different distances is reduced, and the distribution is smoother. Both k' and k are used to index the center of the interval.

[0130] According to the probability that age x belongs to each interval, its binary vector representation is obtained, including the following steps:

[0131] d1. Based on the probability distribution p of the age feature x belonging to the interval, select the interval with the maximum probability as the interval to which the age feature x belongs;

[0132] d2. Let the binary vector be b, which consists of two values ​​0 and 1, and has a dimension of 13, which is the number of intervals. If the probability that the age feature x belongs to the kth interval is the largest, then the binary vector b k =1; if the probability that the age feature x belongs to the kth interval is not the largest, then b in the binary vector b k =0:

[0133]

[0134] where argmax returns p k' The interval subscript k' has the maximum probability in .

[0135] The clustering algorithm is further introduced to assist in optimizing the division of intervals by optimizing the clustering loss function. The specific process is as follows:

[0136] e1. Take the interval center as the cluster center, that is,

[0137] e2. Combined with the Soft K-means clustering algorithm, the interval division is optimized by optimizing the increment between interval centers, and the clustering loss function It is expressed as:

[0138]

[0139] Where x represents age, c k In the K-means clustering algorithm, each sample is assigned an N-dimensional one-hot vector to indicate the category to which it belongs, where N is the number of categories. In the SoftK-means algorithm, each sample is assigned an N-dimensional probability vector to indicate the probability of belonging to each category.

[0140] So far, the eigenvalue x of the age feature has been processed into a binary vector b by this method. Next, the binary vector b is input into the neural rule model RRL for training. RRL is a model for classification tasks. Its input is a discrete vector and its output is a probability distribution. After the training is completed, logical rules can be extracted from the model. The specific process is as follows:

[0141] f1. Convert the eight discrete features of each sample in the Adult dataset into a one-hot vector using one-hot encoding, expressed as [v' i,1 ;…;v' i,M' ], where i represents the i-th sample, v' i,1 The one-hot vector representing the first discrete feature of the i-th sample;

[0142] f2. Convert the six continuous features of each sample in the Adult dataset into binary vectors using the binarization method mentioned above, expressed as [b” i,1 ;…;b” i,M” ], where i represents the i samples, b" i,1 A binary vector representing the first continuous feature of the i-th sample;

[0143] f3. Concatenate the one-hot vectors of the 8 discrete features and the binary vectors of the 6 continuous features of the i-th sample as the input of the neural rule model RRL:

[0144]

[0145] in is the prediction result obtained by the RRL model for the input features of the i-th sample. Its dimension is the number of classifications, which indicates the probability that RRL predicts that the i-th sample belongs to each category.

[0146] In the present invention, Figure 4This is a visualization of the interval division of the age feature of the adult dataset by different interval division methods. There are four interval division methods, namely this method AutoInt, its variant AutoInt without clustering loss, KInt+, and FreInt+. The interval division of AutoInt is roughly distributed between the interval division of AutoInt without clustering loss and KInt+. This is because AutoInt without clustering loss only uses classification loss for optimization, KInt+ only uses clustering, and AutoInt combines classification loss and clustering loss for optimization.

[0147] Figure 5 is a performance comparison of different interval partitioning methods. Among them, AutoInt represents this method, RanInt, WidInt, FreInt, KInt, and EntInt are comparison methods, and RanInt+ is a variant of RanInt, which means that overlapping intervals are not allowed, while RanInt allows interval overlap, and the same is true for variants of other methods. The experiment uses RRL as the neural rule model, and combines different interval partitioning methods with RRL to test the performance of RRL on different data sets. The higher the RRL experimental result, the better the interval partitioning method. In addition, XGBoost is used as a complete classification task model for comparison with RRL. Figure 5 In the figure, bold indicates the highest experimental result, and underline indicates the second highest result. It can be seen that RRL combined with this method has achieved the highest results on most data sets, and is second only to the XGBoost model on the whitewineQ data set, proving the excellence of this method.

[0148] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A method for binarizing continuous input features of a neural rule network that can be optimized end-to-end, characterized in that: The method comprises the following steps: Step 1: Obtain the interval centers of different intervals and represent the interval centers in an incremental manner; Step 2: Get the interval boundary, set the boundaries on both sides according to the maximum and minimum values ​​of the given feature, set the internal boundary based on the center of the interval, and define the interval boundary as the middle position between the two interval centers; Step 3: Obtain a binary vector of a continuous numerical feature. For a specific eigenvalue of a given continuous feature, calculate the probability that the specific eigenvalue belongs to each interval, and select the interval with the maximum probability as the interval to which the specific eigenvalue belongs, thereby obtaining a binary vector representation. Step 4: Select a clustering algorithm and set a clustering loss function so that the clustering loss function assists in optimizing the division of the interval; Step 5: Use the straight-through estimator (STE) method to directly use the gradient of the binary vector as the gradient of the probability distribution to achieve end-to-end optimization of the interval partitioning process.

2. The binarization method according to claim 1, characterized in that: In the step 1, the number of intervals is K, and obtaining the interval center includes the following steps: a1. Let the center of the kth interval be a2. Introduce increments where Δ k represents the increment between the center of the kth interval and the center of the k-1th interval, and the increment is the interval between the interval centers; a3.Remember the left boundary value b of the leftmost interval min , the increment between the center of the first interval and the leftmost boundary is Δ1, then the center of the first interval is expressed as c1=b min +Δ1; the increment between the center of the second interval and the center of the first interval is Δ2, then the center of the second interval is expressed as c2=c1+Δ2, substituting the expression of c1 into c2=b min +Δ1+Δ2; and so on; a4. For the kth interval center, where 1≤k≤K, its position can be expressed as c k =b min +Δ1+…+Δ k-1 +Δ k .

3. The binarization method according to claim 1, characterized in that: In the step 2, the process of obtaining the interval boundary includes the following steps: b1. Classify the boundaries according to their positions. There are three types of interval boundaries: the leftmost boundary, the rightmost boundary, and the boundaries between internal intervals. b2. For the leftmost boundary, set it to the minimum eigenvalue that the continuous feature can take, ensuring that no eigenvalue is located to the left of the leftmost boundary; b3. For the rightmost boundary, set it to the maximum eigenvalue that the continuous feature can take, ensuring that no eigenvalue is located to the right of the rightmost boundary; b4. For the boundaries between intervals, for the k-th interval, where 1 < k < K, its left boundary is located at the midpoint between the center of the (k - 1)-th interval and the center of the k-th interval; since the increment of the center of the k-th interval relative to the center of the (k - 1)-th interval is Δ k , the distance between the left boundary of the k-th interval and the center of the (k - 1)-th interval is Therefore, the left boundary of the k-th interval is expressed as b min +…+Δ k-1 +Δ k / 2; similarly, the right boundary of the k-th interval is located at the midpoint between the center of the k-th interval and the center of the (k + 1)-th interval. Since the increment of the center of the (k + 1)-th interval relative to the center of the k-th interval is Δ k+1 , the increment of the right boundary of the k-th interval relative to the center of the k-th interval is Therefore, the right boundary of the k-th interval is expressed as b min +…+Δ k +Δ k+1 / 2; the k-th interval is expressed as (b min +…+Δ k -1+Δ k / 2, b min +…+Δ k +Δ k+1 / 2).

4. The binarization method according to claim 1, characterized in that: In the step 3, calculating the probability that the characteristic value belongs to each interval includes the following steps: c1. Let a specific eigenvalue of a continuous numerical feature be x, and calculate the distance function D(·) from the eigenvalue x to the center of the kth interval The distance is denoted as D(x, c k ), D(x,c k )=(x-c k ) 2 ; c2. Calculate the probability that the eigenvalue x belongs to the kth interval based on the distance from the eigenvalue x to the center of the interval: k∈{1,…,K}, Among them, τ1 is a hyperparameter used to scale the distance and control the smoothness of the probability distribution. The larger τ1 is, the larger the gap between different distances is, and the sharper the distribution is. The smaller τ1 is, the smaller the gap between different distances is, and the smoother the distribution is. k′ and k both represent the index of the interval center, k′ appears in the denominator, and k appears in the numerator. For each interval center k in the numerator, the denominator must be calculated K times.

5. The binarization method according to claim 1, characterized in that: In the step 3, the process of obtaining the eigenvalue binary vector includes the following steps: c3. Let the continuous eigenvalue be x. Based on the probability distribution p of the eigenvalue x belonging to all intervals, select the interval corresponding to the maximum probability as the interval to which the eigenvalue x belongs; c4. Let the binary vector be b, which consists of two values ​​0 and 1, and has a dimension of K; if the kth interval is the interval to which the eigenvalue x belongs, then in the binary vector b, b k The value of is 1, and the values ​​of other positions are 0:

6. The binarization method according to claim 1, characterized in that: In step 4, the process of optimizing the division of intervals using a clustering algorithm includes the following steps: d1. Set the cluster center, regard different intervals as different clusters, and regard the interval center as the cluster center, that is, d2. Set the clustering objective function The increment of clustering objective function to the center of interval Can be derived to optimize interval parameters; The clustering objective function is recorded as: in, represents the probability that the eigenvalue x belongs to the kth interval, D(x, c k ) represents the eigenvalue x to the cluster center c k distance.

7. The binarization method according to claim 1, characterized in that: In the step 5, performing end-to-end optimization on the interval division includes the following steps: e1. Combine the neural rule network to optimize the interval division, set For the classification loss of the entire network, it is necessary to increase the loss and the interval Generate a back-propagation calculation path between them to achieve end-to-end optimization; e2. The straight-through estimator STE method is used to solve the gradient propagation problem in an approximate way: b=p+Stop-grad(bp), Among them, Stop-grad makes the forward calculation through b, and the back propagation is performed through p; p represents the probability distribution of the eigenvalue belonging to each interval, and b represents the vector obtained after the probability distribution p is binarized.

8. A binarization system for implementing the binarization method according to any one of claims 1 to 7, characterized in that: The binarization system includes: a differentiable binarization component, a differentiable clustering component, and a neural rule model component; The differentiable binarization component is used to represent the interval center and the interval boundary in an incremental manner, calculate the probability distribution of the continuous numerical feature belonging to the interval corresponding to the continuous feature, convert the continuous numerical feature into a binary vector through a threshold function, and use a straight-through estimator to solve the gradient conduction problem between the probability distribution and the binary vector, so as to achieve optimizable interval division; The derivable clustering component is used to regard the interval center as the cluster center, design the clustering loss function, directly optimize the interval center increment, and assist in interval division; The neural rule model component is used to input the discrete binary vector of the processed continuous features and finally output an interpretable rule.

9. Application of the binarization method according to any one of claims 1 to 7, or the binarization system according to claim 8 in binarization of continuous input features.

Images