Fast sampling method for softmax probability distribution based on inner product

Abstract

The present invention discloses a fast sampling method for inner product-based softmax probability distribution, comprising: dividing each item vector q into m segments on average, that is, #imgabs0# clustering each segment vector of all items into K cluster centers using a clustering method #imgabs1# assuming that the codeword index corresponding to the mth segment vector of the ith item is i m , then the corresponding codeword is #imgabs2#, so the i-th item vector is mapped to #imgabs3# Let m = 2, and add a residual vector #imgabs4# The item vector is represented by #imgabs5# Divide the context vector into 2 segments, represented by #imgabs6# Divide all items into K 2 Sets, assuming that k1 is a codeword index in the first codebook, k2 is a codeword index in the second codebook, and the corresponding item sets are marked as #imgabs7# Based on z and k1, k2, define the probability of item i being sampled #imgabs8# Based on z and k1, define the probability of codeword index k2 being sampled #imgabs9# Based on z, define the probability of codeword index k1 being sampled #imgabs10# The original softmax probability distribution is decomposed into: #imgabs11# #imgabs12#

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular, to a fast sampling method for inner product-based softmax probability distribution, which can be widely used in training machine learning models to quickly sample a small-scale but high-quality set of items from a large-scale set of items to obtain a negative sample. Background Art

[0002] In the field of machine learning, there are a large number of models that require large-scale positive and negative samples during training. These samples are usually represented by a triple (z, i, l), which means that when in context z (such as user attributes, environment, query statement, etc.), item i is a positive sample (l = 1) or a negative sample (l = 0). In actual scenarios (such as implicit feedback recommendation, question-answer search, etc.), user feedback (such as click behavior) can usually be used as positive samples, while negative samples are not available. Under such data, in order to obtain negative samples, one way is to sample a set of items from all uninteracted items and use them as negative samples. However, different sampling results will greatly affect the training effect and convergence speed of the model. Therefore, how to effectively perform negative sample sampling is an important research issue in the field of machine learning.

[0003] Around this research problem, researchers have proposed a variety of methods. Among them, based on the inner product between the context and the item, a probability distribution is calculated through a softmax operation and then sampled from it, which is a proven effective negative sampling method. However, when faced with a large-scale collection of items, the efficiency of this method is extremely low. Therefore, it is urgent to provide an efficient sampling method to solve the above technical problems. Summary of the invention

[0004] The purpose of the present invention is to provide a fast sampling method for inner product-based softmax probability distribution, the time complexity of the fast sampling method is significantly reduced, and the error between the fast sampling method and the original probability distribution has an upper bound and greatly improves the sampling efficiency.

[0005] In order to achieve the above object, the present invention provides a fast sampling method for inner product-based softmax probability distribution, the fast sampling method comprising:

[0006] Assume that the context is represented by a D-dimensional vector z and the set of items is represented by Indicates that the i-th item is represented by a D-dimensional vector q i Indicates that the softmax probability distribution based on the inner product can be expressed as And the time complexity of calculating this distribution is O(ND);

[0007] Step 1: Divide each item vector q into m segments evenly, that is in, Indicates a connection operation;

[0008] Cluster each segment vector of all items into K cluster centers using clustering methods Among them, m is the segment subscript;

[0009] Let the codeword index corresponding to the mth segment vector of the i-th item be i m , then the corresponding codeword is Thus, the i-th item vector is mapped to

[0010] Let m = 2, and add a residual vector Used to make up the difference between the original vector and the codeword;

[0011] At this time, the item vector is expressed as The context vector is evenly divided into 2 segments, expressed as

[0012] Step 2: Divide all items into K 2 Sets, assuming k1 is a codeword index in the first codebook, k2 is a codeword index in the second codebook, the corresponding item set is marked as Based on the context vector z and codeword indices k1 and k2, the probability of item i being sampled is defined as follows:

[0013]

[0014]

[0015] Step 3: Based on step 2, the probability of codeword index k2 being sampled is defined as follows based on the context vector z and codeword index k1:

[0016]

[0017]

[0018] Step 4: Based on step 3 and context vector z, define the probability of codeword index k1 being sampled as follows:

[0019]

[0020] Step 5. Based on the probability distribution defined in steps 2-4, prove that the original softmax probability distribution can be decomposed into the following form:

[0021]

[0022] Preferably, in step 5, the time complexity of sampling is O(ND).

[0023] Preferably, the probability distribution in step 2 is Replace it with a uniform distribution: in, Represents the number of items in the item set, and keeps the and P in step 3 z 2 unchanged; at this time, the form of the approximate sampling distribution is as follows:

[0024]

[0025] Preferably, the probability distribution in step 2 is Replaced with a popularity distribution: Among them, pop(i) represents an indicator to measure the popularity of the item, and keep the and in step 3 unchanged; at this time, the form of the approximate sampling distribution is as follows:

[0026]

[0027] Preferably, the clustering method used in step 1 is K-means.

[0028] According to the above technical solution, the present invention assumes that the context is represented by a D-dimensional vector z, and the item set is represented by Indicates that the i-th item is represented by a D-dimensional vector q i Indicates that the softmax probability distribution based on the inner product can be expressed as And the time complexity of calculating the distribution is O(ND). Based on this, the present invention provides an approximate method for fast sampling from the probability distribution P(i|z), and the approximate probability distribution forms are respectively and The time complexity of sampling is O(K 2 +KD), usually K<<N, so the time complexity of sampling is much less than O(ND), thus achieving the purpose of fast sampling. In addition, it can be proved that there is an upper bound on the KL divergence between the approximate probability distribution and the original probability distribution, thus ensuring that the sampled samples are of high quality, and the upper bound is as follows:

[0029] KL(P uni (i|z)||P(i|z))≤2T||z||

[0030]

[0031] in, ||·|| represents the magnitude of a vector.

[0032] In summary, the time complexity of this approximation method is much less than O(ND), and the error with the original probability distribution has an upper bound. By decomposing the softmax probability distribution and approximating some of the probabilities, the sampling efficiency is greatly improved within a certain error.

[0033] Other features and advantages of the present invention will be described in detail in the following detailed description. DETAILED DESCRIPTION

[0034] The specific embodiments of the present invention are described in detail below. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0035] Assume that the context is represented by a D-dimensional vector z and the set of items is represented by Indicates that the i-th item is represented by a D-dimensional vector q i Indicates that the softmax probability distribution based on the inner product can be expressed as And the time complexity of calculating the distribution is O(ND). The purpose of the present invention is to provide an approximate method for fast sampling from a probability distribution P(i|z), and the fast sampling method includes:

[0036] Step 1: Divide each item vector q into m segments, using {q 1 ,q 2 , ..., q m}, then the original vector can be expressed as in Represents a connection operation. Extracting the first segment vector of all items can obtain a set of vectors, represented as Then use clustering methods (such as K-means) to cluster these N vectors into K classes and generate K cluster centers, expressed as Among them, C 1 It is also called a codebook, and each cluster center vector is also called a codeword. After clustering, the first segment vector of each item corresponds to a unique codeword in the codebook. Therefore, assuming that the codeword index corresponding to the first segment vector of the i-th item is i1, then the corresponding codeword is Similarly, the second segment vectors of all items are extracted and clustered to obtain the second codebook and codeword, that is, The mth segment vectors of all items are extracted and clustered to obtain the mth codebook and codeword, that is, Since each vector in the object corresponds to a unique codebook and codeword, assuming that the m codeword indexes corresponding to the m vectors are {i1, i2, ..., i m}, then the i-th item vector can be mapped to In the present invention, for the sake of simplicity, m=2 is used for description. In order to make up for the difference between the original vector and the codeword, a residual vector Finally, the item vector can be expressed as In addition, the context vector also needs to be divided into two segments, expressed as

[0037] Step 2: Since each vector in an item corresponds to a unique codebook and codeword, and each codebook has K codewords, all items will be divided into K 2 Assume that k1 is a codeword index in the first codebook, k2 is a codeword index in the second codebook, and the corresponding item set is marked as Based on the context vector z and codeword indices k1 and k2, the probability of item i being sampled is defined as follows:

[0038]

[0039]

[0040] Step 3: Based on step 2, the probability of codeword index k2 being sampled is defined as follows based on the context vector z and codeword index k1:

[0041]

[0042]

[0043] Step 4: Based on step 3 and context vector z, define the probability of codeword index k1 being sampled as follows:

[0044]

[0045] Step 5. Based on the probability distribution defined in steps 2-4, prove that the original softmax probability distribution can be decomposed into the following form:

[0046]

[0047] At this point, sampling an item from the probability distribution P can be broken down into three steps: First, from the probability distribution Sample the codeword index k1 of the first codebook; secondly, based on k1, from the probability distribution Sample the codeword index of the second codebook in ; Finally, based on k1, k2, from the probability distribution Sample an item i from .

[0048] In addition, in another embodiment, the probability distribution in step 2 can be Replace it with a uniform distribution: in, Represents the number of items in the item set, and keeps the and in step 3 unchanged; at this time, the form of the approximate sampling distribution is as follows:

[0049]

[0050] In addition, in another embodiment, the probability distribution in step 2 can be Replaced with a popularity distribution: Among them, pop(i) represents an indicator to measure the popularity of the item (such as the frequency of the item in the data set, etc.), and keep the and in step 3 unchanged; at this time, the form of the approximate sampling distribution is as follows:

[0051]

[0052] Through the above technical solution, the present invention provides an approximate method for fast sampling from the probability distribution P(i|z), and the approximate probability distribution forms are respectively and The time complexity of sampling is O(K 2 +KD), usually K<<N, so the time complexity of sampling is much less than O(ND), thus achieving the purpose of fast sampling. In addition, it can be proved that there is an upper bound on the KL divergence between the approximate probability distribution and the original probability distribution, thus ensuring that the sampled samples are of high quality, and the upper bound is as follows:

[0053] KL(P uni (i|z)||P(i|z))≤2T||z||

[0054]

[0055] in, ||·|| represents the magnitude of a vector.

[0056] In summary, the time complexity of this approximation method is much less than O(ND), and the error with the original probability distribution has an upper bound. By decomposing the softmax probability distribution and approximating some of the probabilities, the sampling efficiency is greatly improved within a certain error.

[0057] The preferred embodiments of the present invention are described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, a variety of simple modifications can be made to the technical solution of the present invention, and these simple modifications all belong to the protection scope of the present invention.

[0058] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0059] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A fast sampling method for inner product-based softmax probability distribution, applied to question-answering search, where the sample is represented by a triple (z, i, l), where z is the attribute, environment and query statement, item i is a positive sample (l = 1) or a negative sample (l = 0), and the user's click behavior is used as a positive sample, characterized in that: The rapid sampling method comprises: Assume that the context is represented by a D-dimensional vector z and the set of items is represented by Represented by, where the i-th item is represented by a D-dimensional vector q i Indicates that the softmax probability distribution based on the inner product can be expressed as And the time complexity of calculating this distribution is O(ND); Step 1: Divide each item vector q into m segments evenly, that is in, Indicates a connection operation; Cluster each segment vector of all items into K cluster centers using clustering methods Among them, m is the segment subscript; Let the codeword index corresponding to the mth segment vector of the i-th item be i m , then the corresponding codeword is Thus, the i-th item vector is mapped to Let m = 2, and add a residual vector Used to make up the difference between the original vector and the codeword; At this time, the item vector is expressed as The context vector is evenly divided into 2 segments, expressed as Step 2: Divide all items into K 2 Sets, assuming k1 is a codeword index in the first codebook, k2 is a codeword index in the second codebook, the corresponding item set is marked as Based on the context vector z and codeword indices k1 and k2, the probability of item i being sampled is defined as follows: Step 3: Based on step 2, the probability of codeword index k2 being sampled is defined as follows based on the context vector z and codeword index k1: Step 4: Based on step 3 and context vector z, define the probability of codeword index k1 being sampled as follows: Step 5. Based on the probability distribution defined in steps 2-4, prove that the original softmax probability distribution can be decomposed into the following form: The probability distribution in step 2 Replace it with a uniform distribution: in, Represents the number of items in the item set, and keeps the and in step 3 unchanged; at this time, the form of the approximate sampling distribution is as follows: The probability distribution in step 2 Replaced with a popularity distribution: Among them, pop(i) represents an indicator to measure the popularity of the item, and keep the and in step 3 unchanged; at this time, the form of the approximate sampling distribution is as follows:

2. The rapid sampling method according to claim 1, characterized in that: In step 5, the time complexity of sampling is O(ND).

3. The rapid sampling method according to claim 1, characterized in that: The clustering method used in step 1 is K-means.

Images