Community response prediction method based on improved field level feature interaction algorithm

Abstract

Based on the field-aware level feature interaction algorithm, the knowledge community response prediction algorithm defines the response prediction problem as a potential response user recommendation problem, and each neuron in the feature representation of each attribute is regarded as a non-independent structure. The transpose combined with the dense transform transformation makes each neuron retain the full neuron information of the entire feature representation to some extent. The improved CTR (Click-Through-Rate) algorithm based on feature interaction is used to enhance the expression ability of the field-aware level feature interaction information by using two different Hadamard product operation results through splicing operation. The last dimension of the final output is segmented according to the fixed dimension of the encoder, and the field-attention is used to adjust the weight of each attribute in each fold data to the final prediction result, so as to realize the prediction of the potential response user of the new question.

Description

Technical Field

[0001] This invention relates to the field of intelligent recommendation algorithm technology, and in particular to a method for predicting the response of potential users in online communities based on predicted CTR. Background Technology

[0002] Knowledge-sharing communities are one of the main ways people seek answers to questions online, where users exchange knowledge by asking and answering questions. Compared to search engines based on short-text analysis, questions and answers in online communities are often long texts, typically containing background information and the question itself, which short-text analysis struggles to capture. In today's knowledge-sharing communities, with the continuous growth of users, the number of new questions asked per unit of time is also constantly increasing. On the one hand, potential answerers of new questions need to spend a significant amount of time and effort searching for questions related to their expertise. On the other hand, question posters need to spend hours or even days waiting for their questions to be answered by other users.

[0003] Finding an effective response recommendation mechanism is the main approach to solving this problem. Current solutions mainly fall into three categories: The first is to build a directed graph of question-and-answer relationships based on users in the community, and then use an expert recommendation based on a graph ranking algorithm. For example, articles...

[0004] The Topical PageRank (TPR) algorithm proposed in "Automatic Keyphrase Extraction via Topic Decomposition" performs random walks across different topic graphs to obtain the importance of each word. Given a topic distribution, it calculates the semantic word ranking score for each user and selects the top-ranked users as prediction results. The second approach utilizes rich textual feature data from online communities, employing topic optimization for semantic analysis. The paper "Ranks of Restaurant Reviews Based on LDA Model" proposes the RankingLDA algorithm, which uses Linear Discriminant Analysis (LDA) to perform topic analysis on newly proposed questions and previously answered questions, then selects users with high topic similarity as prediction results. The LDA algorithm forms a three-layer Bayesian network comprising the overall question, question words, and question topic, using a bag-of-words method to transform textual information into numerical information for modeling. The third approach combines graph ranking algorithms with text analysis algorithms. For example, the TopicSensitive PageRank (TSPR) algorithm ranks the final results based on a combination of topic relevance score, question quality score, and TSPR score. The TSPR score is obtained by querying the entire dataset for the topic, significantly improving the topic relevance of the returned results. While these studies have made some progress, the following problems remain: 1. The cold start problem: new users struggle to obtain sufficient directed graph associations, leading to inaccurate predictions of new users in the community by graph ranking algorithms. 2. The poor generalization ability of topic-based text analysis models makes it difficult to provide effective response predictions for new topics in the community. 3. Text features and user features are not effectively utilized. Online communities typically contain a large number of category tags and basic user information, all of which can be used to characterize text and users. However, graph ranking and text analysis algorithms do not utilize this data. Summary of the Invention

[0005] To overcome the shortcomings of the existing methods, the present invention aims to provide a knowledge community response prediction algorithm based on a field-aware level feature interaction algorithm. The response prediction problem is defined as a potential response user recommendation problem. It makes full use of the text features and user features in the knowledge community and adopts an improved CTR (Click-Through-Rate) algorithm based on feature interaction to predict potential response users for new questions.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] The knowledge community response prediction method based on the Field-Aware level feature interaction algorithm includes the following steps:

[0008] (1) Set the parameters included in a single data entry, including continuous numerical parameters and discrete multi-valued parameters; numerical parameters include user age, daily active time (hours), weekly active time (days), monthly active time (days), number of questions posted, number of questions answered, number of accepted answers, number of comments on answers, and number of likes on answers; multi-valued parameters include user ID, user gender, user location, article ID, article tags, article title word segmentation, article content word segmentation, whether the user follows the tag, whether the user has read the question, whether the user likes the question, whether the user saves the question, whether the user views the answer to the question, whether the user likes the answer to the question, and whether the user answers the question.

[0009] (2) Processing numerical parameters: If z>2, z is transformed into log(z), where z represents numerical data. Because the large variance of numerical data will make it difficult for the neural network to converge when training, thus reducing the prediction accuracy; then an encoder is used to encode each numerical parameter into a fixed dimension D.

[0010] The encoder is defined as:

[0011] x' = Encoder(x) (1)

[0012] The original input x is a one-row, one-column matrix, represented as x∈R 1*1 The encoder transforms the matrix into a matrix of one row and D columns, represented as x'∈R. 1*D The encoder itself is also a matrix with one row and D columns, also represented as Encoder∈R 1*D ;

[0013] (3) Processing discrete parameters: The discrete spaces of discrete parameters are different. The values ​​in each discrete space are marked with uniqueness, and then different encoders are used to encode the values ​​in the corresponding discrete spaces into a fixed dimension D.

[0014] The encoder is defined as follows:

[0015] x' i =Encoder i (x i (2)

[0016] Original input x i Represented by one-hot encoding, denoted as x i ∈R 1*m, where m is the size of the discrete space for each discrete parameter; similarly, it is transformed into a matrix of one row and D columns by the encoder, represented as x' i ∈R 1*D Unlike the encoder in step (2), different discrete spaces require different dimensional encoders, denoted as Encoder. i ∈R m*D ;

[0017] (4) Field-aware level feature representation of the raw data obtained in steps (2) and (3). When 1≤i≤m are concatenated together, the formula becomes:

[0018]

[0019] Where m represents the total number of numerical and discrete parameters in each data point; X 1 In the text, X represents the feature data, and the superscript 1 indicates the order of the feature data. In the field of CTR prediction algorithms, steps (2), (3), and (4) undertake the work of the encoding layer, EmbeddingLayer.

[0020] (5) Represent the original features obtained in step (4) as X 0 Perform a dense transform, then combine it with the original feature representation X. 0 Perform the Hadamard product; the original data is represented by formula (3), and the transformed data is represented by formula (4):

[0021]

[0022]

[0023] The Field-aware level feature representation obtained in step (4) In this context, each dimension is considered independent of the others, while the feature data... After the dense transform, each unit aggregates all units from the original feature representation, and the weights of each unit are controlled by the trainable weight matrix in the dense transform, as follows:

[0024] W dense =[w1,…,w D ] T (6)

[0025] Among them, w i ∈R 1*D ,1≤i≤D; then the feature representation X 1Perform the Hadamard product operation with the transformed result obtained from formula (4), that is:

[0026]

[0027] (6) Transpose the original feature representation obtained in step (4), then perform a dense transform, and finally combine it with the original feature representation X. 1 Perform the Hadamard product; similarly, the original data is represented by formula (3), and the transformed data is represented by formula (8);

[0028]

[0029] Unlike formula (4), each unit in formula (8) does not contain all the unit information of the current attribute, but rather contains the information of all the units corresponding to the attributes. The activation function δ is equivalent to formula (5), and the dense transform is equivalent to formula (6). Then, the result obtained from formula (8) is multiplied by the original feature representation X0, i.e.:

[0030]

[0031] (7) Obtain the results from steps (5) and (6). and These are combined to form a new second-order feature representation:

[0032]

[0033] in, 1 << i << m; This indicates that the last dimension is used for splicing; similarly, in the field of CTR prediction algorithms, steps (5), (6), and (7) serve as the interaction layer.

[0034] (8) Step (7) uses two different forms of field-aware level feature representations to distinguish it from existing methods; on this basis, in order to simulate higher-order interactions, the present invention sets up three interaction layers l, 1 << l << 3, and repeats steps (5), (6), and (7) three times; each interaction layer uses a new feature representation. With the feature representation X of the input l Perform the Hadamard product operation instead of the original feature representation X. 1 The innovation of this invention lies in performing the Hadamard product operation; that is, the difference between formula (10) and formula (11):

[0035] X l+1 =X l ·X 1 (11)

[0036] When l = 3, the result obtained by formula (11) can best fit the original feature X. 1 The third-order interaction, the result obtained by formula (10) can fit the original feature X at most. 1 The 16th-order interaction; In this invention, the number of interaction layers is set to 3, and the maximum number of feature interactions that can be fitted is 16th-order.

[0037] (9) Collect all the original features and the outputs corresponding to the three interaction layers, and concatenate them according to the last dimension to obtain the final output:

[0038]

[0039] (10) The result obtained in step (9) is segmented according to the fixed encoding dimension D of the encoder, with the segmentation dimension being the last dimension; then, the field-attention score is calculated for each of the K fold data to adjust the contribution of the result obtained from each interaction layer to the final prediction result; among them, the feature representations corresponding to the m attributes of each fold data need to be calculated to obtain a corresponding score, defined as formula (13):

[0040] a i =x i ·w i ,1<<i<<K (13)

[0041] Where, x i ∈R m *D,w i ∈R D*1 ,a i ∈R m*1 Then, each attribute in each fold of data is multiplied by its corresponding weighting factor:

[0042] x' i,j =a i,j *x i,j ,1<<i<<K,1<<j<<m (14)

[0043] Then, the adjusted data is reassembled according to the last dimension, resulting in the following output:

[0044]

[0045] (11) Expand the output obtained in step (10) into one dimension, represented as h0; then input it into a three-layer neural network MLP for forward propagation calculation;

[0046] h j =δ(w j ·hj-1 ),1<<j<<3 (16)

[0047] Wherein, the activation function δ is Equation (5);

[0048] (12) Input the result obtained in step (11) into the final prediction layer to obtain the final prediction result y':

[0049] y'=σ(W pre ·h3) (17)

[0050]

[0051] To ensure the range of values ​​for the output results, the Sigmoid activation function is used to guarantee that the output result is 0 ≤ y' ≤ 1. When the predicted result y' is closer to 1, it means that the user represented by the parameter user Id in the original input is more likely to answer the question represented by the parameter article Id. Conversely, when the predicted result y' is closer to 0, it means that the user represented by the parameter user Id in the original input is less likely to answer the question represented by the parameter article Id.

[0052] Compared with the prior art, the advantages of the present invention are:

[0053] (1) In this invention, each neuron in the feature representation of each attribute is regarded as a non-independent structure. The transpose combined with dense transform is used so that each neuron retains all neuronal information of the entire feature representation to a certain extent.

[0054] (2) By splicing, two different Hadamard product operation results are used to enhance the expressive power of field-aware feature interaction information, making the results more robust.

[0055] (3) Using same-order interaction makes the order of the interaction result increase exponentially. Compared with interacting with the original feature representation, this method can fit higher-order feature interaction without increasing the amount of computation.

[0056] (4) The last dimension of the final output is divided according to the fixed dimension of the encoder, and field-attention is used to adjust the weight of each attribute in each fold of data to the final prediction result.

[0057] (5) By studying the interaction direction of features in the field of CTR prediction, a theoretical decision-making foundation is laid for better fitting of real interaction methods, reducing the complexity and time cost of implementation, and improving the prediction accuracy of potential answer users. Attached Figure Description

[0058] Figure 1This is a diagram of the overall architecture of the prediction algorithm.

[0059] Figure 2 This is a schematic diagram of the interaction layer structure.

[0060] Figure 3 This is a schematic diagram of the field-attention layer structure. Detailed Implementation

[0061] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0062] The knowledge community response prediction method based on the Field-Aware level feature interaction algorithm includes the following steps:

[0063] (1) Set the parameters included in a single data entry, including continuous numerical parameters and discrete multi-valued parameters. Numerical parameters include user age, daily active time (hours), weekly active time (days), monthly active time (days), number of questions posted, number of questions answered, number of accepted answers, number of comments on answers, and number of likes on answers. Multi-valued parameters include user ID, user gender, user location, article ID, article tags, article title word segmentation, article content word segmentation, whether the user follows the tag, whether the user has read the entire question, whether the user likes the question, whether the user has saved the question, whether the user has viewed the question answer, whether the user likes the question answer, and whether the user has answered the question.

[0064] The parameter settings, constraints, and independent variable constraint boundary conditions are shown in Table 1. Discrete multi-valued parameters are displayed as numerical data after being uniquely determined and encoded.

[0065] Table 1. Parameter settings, constraint boundaries, and constraint conditions for a single data entry.

[0066]

[0067]

[0068] (2) Processing numerical parameters: If z > 2, z is transformed into log(z). Here, z represents numerical data, because the large variance of numerical data will make it difficult for the neural network to converge during training, reducing the prediction accuracy. Then, an encoder is used to encode each numerical parameter into a fixed dimension of 16.

[0069] The encoder can be defined as:

[0070]

[0071] The original input x is a one-row, one-column matrix, represented as x∈R 1*1The encoder transforms it into a 1x16 matrix, represented as x'∈R. 1*16 The encoder itself is also a 1x16 matrix, also represented as Encoder∈R 1*16 .

[0072] (3) Processing discrete parameters: The discrete spaces of discrete parameters are different. The values ​​in each discrete space are marked with uniqueness, and then different encoders are used to encode the values ​​in the corresponding discrete spaces into a fixed dimension 16.

[0073] The encoder can be defined as:

[0074] x' i =Encoder i (x i (20)

[0075] Original input x i Represented by one-hot encoding, denoted as x i ∈R 1*m , where m is the size of the discrete space for each discrete parameter. Similarly, after being encoded, it is transformed into a 1x16 matrix, represented as x'. i ∈R 1*16 Unlike the encoder in step (2), different discrete spaces require different dimensional encoders, denoted as Encoder. i ∈R m*16 .

[0076] (4) As attached Figure 1 As shown, the field-aware feature representations of the original data obtained in steps (2) and (3) are concatenated together as the input of this invention, and the formula is as follows:

[0077]

[0078] Where 23 represents the total number of numerical and discrete parameters in each data point; X 0 The superscript 0 indicates the order of the original feature, and X represents the feature data.

[0079] (5) Represent the original features obtained in step (4) as X 1 Perform a dense transform, and then perform a Hadamard product with the original feature representation. The original data is represented by Equation (21), and the transformed data is represented by Equation (22).

[0080]

[0081]

[0082] In the field-aware feature representation obtained in step (4), each dimension is treated as independent of each other. After the original feature representation undergoes dense transform, each unit aggregates all units from the original feature representation, and the weights of each unit are controlled by the trainable weight matrix in the dense transform, as follows:

[0083] W dense =[w1,…,w 16 ] T (twenty four)

[0084] Among them, w i ∈R 1*16 ,1≤i≤16. Then, the original feature representation X0 and the transformed result obtained from formula (22) are subjected to the Hadamard product operation, that is:

[0085]

[0086] (6) The original feature representation obtained in step (4) is first transposed, then subjected to dense transform, and then Hadamard product with the original feature representation. Similarly, the original data is represented by formula (21), and the transformed data is represented by formula (26).

[0087]

[0088] Unlike formula (22), each unit in formula (26) does not contain all unit information for the current attribute, but rather information for all units corresponding to all attributes. The activation function δ is equivalent to formula (23), and dense transform is equivalent to formula (24). Then, the result obtained from formula (26) is compared with the original feature representation X. 1 Perform the Hadamard product operation, that is:

[0089]

[0090] (7) Obtain the results from steps (5) and (6). and A new feature representation is formed by piecing together the last dimension:

[0091]

[0092] in, 1 << i << m. Steps (5), (6), and (7) are implemented as detailed in the appendix. Figure 2 The interaction layer structure is shown below.

[0093] (8) Step (7) uses two forms to express the field-aware level feature representation, which is different from the existing model. On this basis, in order to simulate higher-order interactions, the innovation of this invention is to use the new feature representation to perform a Hadamard product operation with the input feature representation instead of performing a Hadamard product operation with the original feature representation X0. That is, the difference between formula (28) and formula (29).

[0094] X l+1 =X l ·X 1 (29)

[0095] When l = 3, the result obtained by formula (29) can fit the highest order 3 interaction of the original feature X0, and the result obtained by formula (28) can fit the highest order 16 interaction of the original feature X0. In this invention, the number of interaction layers is set to 3, and the highest order 16 feature interaction can be fitted.

[0096] (9) To find the interaction method with the best prediction effect, both high-order feature interactions and low-order feature interactions need to participate in the prediction. Collect all the outputs of the original features and the three interaction layers, and concatenate them according to the last dimension to obtain the final output:

[0097]

[0098] (10) The result obtained in step (9) is segmented according to the fixed encoding dimension 16 of the encoder, with the segmentation dimension being the last dimension. Then, the field-attention score is calculated for each of the 10 folds of data to adjust the contribution of the result obtained from each interaction layer to the final prediction result. Among them, the feature representations corresponding to the m attributes of each fold of data need to be calculated to obtain a corresponding score, which can be defined as formula (31):

[0099] a i =x i ·w i ,1<<i<<10 (31)

[0100] Where, x i ∈R m*16 ,w i ∈R 16*1 ,a i ∈R m*1 Then, each attribute in each fold of data is multiplied by its corresponding weighting coefficient.

[0101] x' i,j =a i,j *x i,j ,1<<i<<10,1<<j<<23 (32)

[0102] Then, the adjusted data is reassembled according to the last dimension. Specific implementation details are attached. Figure 3 As shown. The final output is as follows:

[0103]

[0104] (11) Expand the output obtained in step (10) into one dimension, denoted as h0. Then input it into a three-layer neural network MLP for forward propagation calculation, where the number of hidden neurons in each layer is set to 300.

[0105] h j =δ(w j ·h j-1 ),1<<j<<3 (34)

[0106] The activation function δ is given by formula (24), which ultimately represents each data point as a 300-dimensional feature vector.

[0107] (12) Input the result obtained in step (11) into the final prediction layer to obtain the final prediction result y'.

[0108] y'=σ(W pre ·h3) (35)

[0109]

[0110] To ensure the range of values ​​for the output, the Sigmoid activation function is used to guarantee that the output result is 0 ≤ y' ≤ 1. When the predicted result y' is closer to 1, it means that the user represented by the parameter user Id in the original input is more likely to answer the question represented by the parameter article Id. Conversely, when the predicted result y' is closer to 0, it means that the user represented by the parameter user Id in the original input is less likely to answer the question represented by the parameter article Id.

[0111] The error between the predicted result y' and the true value y is calculated using the log loss function, and this error is used as one of the indicators to measure the performance of the algorithm; loss=-(ylog(y')+(1-y)log(1-y')); another indicator is the prediction result obtained in step (12).

Claims

1. A knowledge community response prediction method based on a field-aware level feature interaction algorithm, characterized in that, Includes the following steps: (1) Set the parameters included in a single data entry, including continuous numerical parameters and discrete multi-valued parameters; (2) Processing numerical parameters: If Then Transform into ,in, The data is represented as numerical data; then an encoder is used to encode each numerical parameter into a fixed dimension D. The encoder is defined as: (1) The original input x is a matrix with one row and one column, represented as follows: The encoder converts it into a matrix of one row and D columns, represented as follows: The encoder itself is also a matrix with one row and D columns, and is also represented as... ; (3) Processing discrete parameters: The discrete spaces of discrete parameters are different. The values ​​in each discrete space are marked with uniqueness, and then different encoders are used to encode the values ​​in the corresponding discrete spaces into a fixed dimension D. The encoder is defined as follows: (2) raw input Represented by one-hot encoding, as , where m is the size of the discrete space for each discrete parameter; similarly, it is transformed into a matrix of one row and D columns by the encoder, represented as... Unlike the encoder in step (2), different discrete spaces require different dimensional encoders, denoted as... ; (4) Field-aware feature representation of the raw data obtained in steps (2) and (3). When pieced together, the formula becomes: (3) Where m represents the total number of numerical and discrete parameters in each data point; In the text, X represents the feature data, and the superscript 1 indicates the order of the feature data. In the field of CTR prediction algorithms, steps (2), (3), and (4) undertake the work of the encoding layer, EmbeddingLayer. (5) Represent the original features obtained in step (4). Perform a dense transformation, then combine it with the original feature representation. Perform the Hadamard product; the original data is represented by formula (3), and the transformed data is represented by formula (4): (4) (5) The Field-aware level feature representation obtained in step (4) In this context, each dimension is considered independent of the others, while the feature data... After the dense transform, each unit aggregates all units from the original feature representation, and the weights of each unit are controlled by the trainable weight matrix in the dense transform, as follows: (6) in, Then the feature representation Perform the Hadamard product operation with the transformed result obtained from formula (4), that is: (7) (6) Transpose the original feature representation obtained in step (4), then perform dense transformation, and then combine it with the original feature representation. Perform the Hadamard product; similarly, the original data is represented by formula (3), and the transformed data is represented by formula (8). (8) Unlike formula (4), each cell in formula (8) does not contain all cell information for the current attribute, but rather information for all cells corresponding to all attributes. Activation function Equivalent to formula (5), dense transform is equivalent to formula (6), and then formula (8) is used to obtain the result and the original feature representation. Perform the Hadamard product operation, that is: (9) (7) Obtain the results from steps (5) and (6). and These are combined to form a new second-order feature representation: (10) in, ; This indicates that the last dimension is used for splicing; similarly, in the field of CTR prediction algorithms, steps (5), (6), and (7) serve as the interaction layer. (8) Step (7) uses two different forms of Field-aware level feature representation to distinguish it from existing methods; on this basis, in order to simulate higher-order interactions, this invention sets up three interaction layers. Repeat steps (5), (6), and (7) three times; use a new feature representation in each interaction layer. Feature representation of the input Perform the Hadamard product operation instead of the original feature representation. The innovation of this invention lies in performing the Hadamard product operation; that is, the difference between formula (10) and formula (11): (11) When l=3, the result obtained by formula (11) best fits the original features. The third-order interaction, and the result obtained by formula (10) best fits the original features. The 16th-order interaction; In this invention, the number of interaction layers is set to 3, and the highest fitted feature interaction is 16th-order. (9) Collect all the original features and the outputs corresponding to the three interaction layers, and concatenate them according to the last dimension to obtain the final output: (12) (10) Segment the result obtained in step (9) according to the fixed encoding dimension D of the encoder, with the segmentation dimension being the last dimension; then calculate the field-attention score for each of the K folds of data to adjust the contribution of the result obtained from each interaction layer to the final prediction result; where, for each fold of data, the feature representation corresponding to the m attributes needs to be calculated to have a corresponding score, defined as formula (13): (13) in, Then, each attribute in each fold of data is multiplied by its corresponding weighting factor: (14) Then, the adjusted data is reassembled according to the last dimension, resulting in the following output: (15) (11) Expand the output obtained in step (10) into one dimension, as follows: Then it is input into a three-layer neural network MLP for forward propagation calculation; (16) Among them, activation function Formula (5); (12) Input the result obtained in step (11) into the final prediction layer to obtain the final prediction result. Prediction results Specifically: (17) (18) To ensure the range of values ​​for the output result, the Sigmoid activation function is used to guarantee the output result. When the prediction result A value closer to 1 indicates that the user represented by the parameter UserId in the original input is more likely to answer the question represented by the parameter ArticleId; conversely, a value closer to 1 indicates that the predicted result is less likely to answer the question. A value closer to 0 indicates that the user represented by the parameter UserId in the original input is less likely to answer the question represented by the parameter ArticleId. In step (1), the numerical parameters include user age, daily active time (hours), weekly active time (days), monthly active time (days), number of questions posted, number of questions answered, number of accepted answers, number of comments on answers, and number of likes on answers; the multi-value parameters include user ID, user gender, user location, article ID, article tags, article title word segmentation, article content word segmentation, whether the user follows the tag, whether the user has read the question, whether the user likes the question, whether the user has saved the question, whether the user has viewed the answer to the question, whether the user likes the answer to the question, and whether the user has answered the question.

Images