Content recommendation based on embedding vector summaries

Abstract

This disclosure presents methods, apparatus, and computer program products for content recommendation based on embedding vector summarization. Text input can be obtained, including a base input and a corresponding context input, the base input including at least candidate content items. An embedding vector sequence corresponding to the base input and an embedding vector sequence corresponding to the context input can be generated. Pooled embedding vectors can be generated by performing a pooling operation on the embedding vector sequence corresponding to the base input. A representative embedding vector sequence corresponding to the context input can be obtained by performing a summarization operation on the embedding vector sequence corresponding to the context input using the pooled embedding vectors. A text input representation of the text input can be generated based at least on the embedding vector sequence corresponding to the base input and the representative embedding vector sequence corresponding to the context input. The click probability of a candidate content item being clicked can be predicted based on the text input representation.

Description

Background Technology

[0001] With the development of network technology and the growth of online information, recommender systems are playing an increasingly important role in many online services. Different recommender systems exist based on the content being recommended, such as news recommender systems, video recommender systems, book recommender systems, and product recommender systems. These systems can typically perform personalized recommendations for target users. In this paper, the target user refers to the user to whom content recommendations are performed. For example, recommender systems often capture the target user's interests and predict content that the user might be interested in based on those interests, then recommend it to the user. Summary of the Invention

[0002] This invention is provided to introduce a set of concepts, which will be further described in the following detailed description. This invention is not intended to identify key or essential features of the protected subject matter, nor is it intended to limit the scope of the protected subject matter.

[0003] Embodiments of this disclosure provide methods, apparatus, and computer program products for content recommendation based on embedding vector summarization. Text input can be obtained, comprising a base input and a context input corresponding to the base input, the base input including at least candidate content items. An embedding vector sequence corresponding to the base input and an embedding vector sequence corresponding to the context input can be generated. A pooled embedding vector corresponding to the base input can be generated by performing a pooling operation on the embedding vector sequence corresponding to the base input. A representative embedding vector sequence corresponding to the context input can be obtained by performing a summarization operation on the embedding vector sequence corresponding to the context input using the pooled embedding vectors. A text input representation of the text input can be generated, at least based on the embedding vector sequence corresponding to the base input and the representative embedding vector sequence corresponding to the context input. The click probability of the candidate content item being clicked can be predicted based on the text input representation.

[0004] It should be noted that one or more of the above aspects include the features described in detail below and specifically pointed out in the claims. The following description and drawings set forth certain illustrative features of the one or more aspects in detail. These features merely indicate various ways in which the principles of each aspect can be employed, and this disclosure is intended to include all such aspects and their equivalents. Attached Figure Description

[0005] The following description will take into account several aspects disclosed, which are provided to illustrate rather than limit the aspects disclosed.

[0006] Figure 1An exemplary process for content recommendation based on embedded vector summaries, according to embodiments of this disclosure, is illustrated.

[0007] Figure 2 An exemplary process for performing a summary operation on an embedded vector sequence corresponding to a context input, according to an embodiment of the present disclosure, is shown.

[0008] Figure 3 An exemplary process for predicting the click probability of a candidate content item being clicked using a click probability prediction model, according to embodiments of the present disclosure, is shown.

[0009] Figure 4 Another exemplary process is shown according to embodiments of the present disclosure for predicting the click probability of a candidate content item being clicked using a click probability prediction model.

[0010] Figure 5 An exemplary process for the first stage training of a click probability prediction model according to an embodiment of this disclosure is shown.

[0011] Figure 6 This is a flowchart of an exemplary method for content recommendation based on embedded vector summaries, according to embodiments of this disclosure.

[0012] Figure 7 An exemplary apparatus for content recommendation based on embedded vector summaries, according to embodiments of this disclosure, is shown.

[0013] Figure 8 An exemplary apparatus for content recommendation based on embedded vector summaries, according to embodiments of this disclosure, is shown. Detailed Implementation

[0014] This disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that this discussion of embodiments is merely intended to enable those skilled in the art to better understand and thus implement the embodiments of this disclosure, and is not intended to teach any limitation on the scope of this disclosure.

[0015] A set of click probabilities can be obtained by predicting the click probability of a target user clicking on each candidate content item in a set of candidate content items using a neural network model based on a transformer layer structure. In this paper, a content item refers to an individual item with specific content. For example, a news article, a movie, a video, or a product can be considered a content item. The content items to be recommended to the target user can be determined by ranking this set of click probabilities. Neural network models based on transformer layer structures can include, for example, Bidirectional Encoder Representations from Transformers (BERT) models, Generative Pre-trained Transformer (GPT) models, Robustly Optimized BERT Approach (RoBERTa) models, Decoding-enhanced BERT with Disentangled Attention (DeBERTa) models, etc. Text input for content recommendation can be provided to the transformer layer-based model. The text input includes candidate content items. The transformer layer-based model can generate a text input representation of the text input by performing self-attention computation based on a self-attention mechanism. Furthermore, models based on transformer layer structures can predict the click probability of candidate content items based on the generated text input representation. Since these models typically rely on complex deep networks with a massive number of parameters, they can generate accurate text input representations and thus obtain accurate click probability predictions. However, using such complex models for online prediction is very time-consuming, leading to high prediction latency, especially when the input is long text containing a large number of words.

[0016] Embodiments of this disclosure propose content recommendation based on embedding summarization. Here, embedding vectors refer to a set of information generated from raw data in a form favorable for processing by a neural network model. Text input, including basic input and corresponding contextual input, can be provided to a neural network model based on a transformer layer structure according to embodiments of this disclosure. The basic input may include only candidate content items or both candidate content items and a query. Candidate content items may be content items from a set of candidate content items that can be recommended to a target user. The query may be a search query from the target user. The contextual input may include the target user's historical interactions, historical search results for the query, etc. The neural network model can predict the click probability of a candidate content item being clicked based on the received text input. Here, the neural network model used to predict the click probability is referred to as a click probability prediction model. The click probability prediction model can generate an embedding vector sequence corresponding to the basic input and an embedding vector sequence corresponding to the contextual input. Pooling operations can be performed on the embedding vector sequence corresponding to the basic input to generate pooled embedding vectors corresponding to the basic input. These pooled embedding vectors can be used to perform a summarization operation on the embedding vector sequence corresponding to the contextual input to obtain a representative embedding vector sequence corresponding to the contextual input. The number of embedding vectors in the representative embedding vector sequence corresponding to the context input can be significantly smaller than the number of embedding vectors in the corresponding context input sequence. The embedding vector sequence corresponding to the base input and the representative embedding vector sequence corresponding to the context input can be provided to a subsequent set of transformer layers for further processing to generate a text input representation of the text input. The generated text input representation can be used to predict the click probability of a candidate content item being clicked. In the click probability prediction model, the number of transformers included in each transformer layer corresponds to the number of embedding vectors provided to that transformer layer. With a reduced number of embedding vectors to be processed, the number of transformers can also be reduced accordingly. Furthermore, the computational cost of the self-attention calculations required can also be reduced accordingly. Therefore, the latency in generating the text input representation and the latency in predicting the click probability can be reduced.

[0017] Pooled embedding vectors can be used to perform summarization operations on the sequence of embedding vectors corresponding to the context input, thereby reducing the number of embedding vectors provided to subsequent transformer layers. Pooled embedding vectors can be generated by performing a pooling operation on the sequence of embedding vectors corresponding to the underlying input, which includes candidate content items and / or queries. Accordingly, the pooled embedding vectors can represent the meaning of the candidate content items and / or queries. Using such pooled embedding vectors to perform summarization operations allows selection of embedding vectors relevant to the candidate content items and / or queries from the sequence of embedding vectors corresponding to the context input. Since the goal of click probability prediction models is to predict the click probability of candidate content items being clicked, selecting embedding vectors relevant to the candidate content items is beneficial. For example, the context input may include the target user's historical interactions. Historical interactions can reflect the target user's interests. Performing summarization operations using candidate content items allows determining whether the target user is interested in the candidate content items based on their historical interactions and can guide the target user's interests. Queries can reflect the target user's intent. Considering the query when selecting embedding vectors can help select embedding vectors relevant to the target user's intent. The above method can reduce the number of embedding vectors while retaining the key embedding vectors that are important for click probability prediction. This helps maintain the accuracy of the prediction results.

[0018] Before deploying a click probability prediction model for online click probability prediction, it can be trained. During training, the model's parameters or weights can be progressively optimized as the training loss is minimized, allowing the model to learn key embedding vectors important for click probability prediction. A representative sequence of embedding vectors corresponding to the context input is selected from the original embedding vector sequence by calculating similarity with candidate content items and / or the query. The selected representative sequence of embedding vectors corresponding to the context input is dynamically changed during training and can be progressively optimized. Self-attention operations can be performed on the selected representative embedding vector sequence through subsequent transformer layers to further ensure learning effectiveness. The model with the minimum training loss can be used as the final model and deployed for online click probability prediction tasks.

[0019] Traditional neural network models based on transformer layers typically include an embedding layer and a set of transformer layers. The click probability prediction model according to embodiments of this disclosure additionally includes a pooling layer and a summarizing layer. The pooling layer performs pooling operations on the sequence of embedding vectors corresponding to the basic input. The summarizing layer performs summarizing operations on the sequence of embedding vectors corresponding to the context input. Through the processing of the pooling and summarizing layers, the number of embedding vectors provided to subsequent transformer layers can be reduced. The pooling and summarizing layers can be located between the embedding layer and the bottommost transformer layer, or between two sets of transformer layers.

[0020] When the pooling and summarizing layers are located between the embedding layer and the bottommost transformer layer, the number of embedding vectors provided to the bottommost transformer layer can be reduced. Therefore, the number of transformers included in each transformer layer can be reduced. This can significantly reduce prediction latency. In this model architecture, pooling and summarizing operations are performed based on the sequence of embedding vectors generated by the embedding layer. To ensure that the embedding layer can generate a high-quality sequence of embedding vectors so that key embedding vectors important for click probability prediction can be selected during the summarizing operation, embodiments of this disclosure propose multi-stage training for the click probability prediction model, such as first-stage training and second-stage training. The first-stage training does not involve pooling and summarizing operations; it aims to optimize the embedding layer so that it has reasonable parameters or weights to generate a high-quality sequence of embedding vectors. In the first-stage training, the number of transformers included in each transformer layer can be consistent with the number of original embedding vectors without summarizing operations. The second-stage training can be performed based on the first-stage training. The embedding layer in the click probability prediction model can be the embedding layer obtained through the first-stage training. In the second-stage training, pooling and summarizing operations can be performed. Therefore, the number of embedding vectors provided to the set of transformer layers above the embedding layer can be reduced. In the second stage of training, the number of transformers included in each transformer layer can be consistent with the number of embedding vectors after the summarization operation. The click probability prediction model obtained after the second stage of training can be deployed to perform click probability prediction tasks online.

[0021] When pooling and summarizing layers are located between two sets of transformer layers, such as above one or more transformer layers, the number of embedding vectors provided only to the upper set of transformer layers can be reduced. Therefore, the number of transformers included in the upper set of transformer layers can be reduced. This can reduce prediction latency to some extent. The lower the pooling and summarizing layers are in the model, the greater the reduction in prediction latency. Furthermore, in this model architecture, pooling and summarizing operations are performed based on the embedding vector sequences generated by the transformer layers. These embedding vector sequences are generated through a self-attention mechanism and therefore have high quality. Therefore, in this case, single-stage training is also possible. For example, the second stage of training can be performed directly without performing the first stage of training described above. This can significantly save training costs.

[0022] The positions of the pooling and summarizing layers in the click probability prediction model can be determined based on the actual application requirements, including real-time prediction requirements and training cost requirements. If real-time prediction requirements are high, the pooling and summarizing layers can be placed between the embedding layer and the bottommost transformer layer. If training cost requirements are high, the pooling and summarizing layers can be placed between two sets of transformer layers.

[0023] Figure 1 An exemplary process 100 for content recommendation based on embedded vector summaries, according to an embodiment of this disclosure, is illustrated. Process 100 can predict the click probability of a candidate content item being clicked by a target user. The predicted click probability can be used to determine whether to recommend the candidate content item to the target user.

[0024] At point 102, text input is available. Text input includes basic input. Basic input may consist solely of candidate content items. Candidate content items can be content items from a set of candidate content items that can be recommended to the target user. Candidate content items can include, for example, news, videos, movies, books, music, web pages, product information, etc. Alternatively, in addition to candidate content items, basic input may also include queries. That is, basic input can include both candidate content items and queries. Queries can be search queries from the target user, such as search queries entered by the target user in a search engine, news software, video applications, etc. Additionally, text input includes contextual input corresponding to the basic input. Contextual input can include the target user's historical interactions, such as long-term historical interactions and real-time historical interactions. Long-term historical interactions can include, for example, queries entered by the target user over a relatively long period of time, such as the past 3 months, and titles of content items clicked. Real-time historical interactions can include, for example, queries entered by the target user over a relatively short period of time, such as the past 4 hours, and titles of content items clicked. The user's historical interactions are collected and used with the user's knowledge and permission. When the basic input includes a query, the contextual input can also include historical search results for that query. Historical search results can be used to interpret and expand the query.

[0025] At position 104, an embedding vector sequence corresponding to the basic input and an embedding vector sequence corresponding to the context input can be generated. Through tokenization, a basic input token sequence corresponding to the basic input and a context input token sequence corresponding to the context input can be obtained. In this paper, a token refers to the basic linguistic unit that constitutes text in different languages. The embedding vector sequence corresponding to the basic input can include the embedding vector corresponding to each token in the basic input token sequence. The embedding vector sequence corresponding to the context input can include the embedding vector corresponding to each token in the context input token sequence. This will be discussed later in conjunction with... Figure 3 and Figure 4 This illustrates an exemplary process for generating an embedding vector sequence corresponding to the basic input and an embedding vector sequence corresponding to the context input.

[0026] At position 106, pooled embedding vectors corresponding to the base input can be generated by performing a pooling operation on the sequence of embedding vectors corresponding to the base input. The pooling operation can be any known pooling operation, such as mean pooling, max pooling, attention pooling, etc.

[0027] At position 108, a representative embedding vector sequence corresponding to the context input can be obtained by performing a summarization operation on the pooled embedding vector pair corresponding to the context input. The number of embedding vectors in the representative embedding vector sequence corresponding to the context input can be significantly smaller than the number of embedding vectors in the embedding vector sequence corresponding to the context input. Preferably, the representative embedding vector sequence corresponding to the context input may include embedding vectors related to the basic input selected from the embedding vector sequence corresponding to the context input. Figure 2 An exemplary procedure 200 for performing a summary operation on an embedded vector sequence corresponding to a context input, according to an embodiment of the present disclosure, is shown.

[0028] The sequence of embedding vectors corresponding to the context input includes multiple context input embedding vectors. At position 202, the similarity between the pooled embedding vector corresponding to the base input and each of the multiple context input embedding vectors can be calculated to obtain multiple similarities. The similarity between the pooled embedding vector and the context input embedding vector can be calculated in several ways.

[0029] In one implementation, the similarity between the pooled embedding vector and the context input embedding vector can be calculated by computed an attention score. The pooled embedding vector corresponding to the base input can be labeled as... The first element in the embedding vector sequence corresponding to the context input can be... The embedding vectors are labeled as ,in , This is the number of embedding vectors included in the embedding vector sequence corresponding to the context input. The process of calculating the attention score can be illustrated by the following formula: in, These are the coefficients used to perform the normalization operation.

[0030] In another implementation, the similarity between the pooled embedding vector and the context input embedding vector can be calculated by measuring the cosine distance between them. The process of calculating the cosine distance can be illustrated by the following formula:

[0031] It should be understood that the above calculation of the attention score or cosine distance between the pooled embedding vector and the context input embedding vector is only an example for calculating the similarity between the two. Other methods can also be used to calculate the similarity between the pooled embedding vector and the context input embedding vector.

[0032] At position 204, multiple context input embedding vectors can be sorted based on the obtained similarity scores to obtain sorted context input embedding vectors. For example, the multiple context input embedding vectors can be sorted in descending order of similarity.

[0033] At position 206, a predetermined number of context input embedding vectors can be selected from the top-ranked multiple sorted context input embedding vectors.

[0034] At position 208, the selected context input embedding vectors can be combined into a representative embedding vector sequence corresponding to the context input.

[0035] Return to reference Figure 1 After obtaining the representative embedding vector sequence corresponding to the context input, at point 110, a text input representation of the text input can be generated based at least on the embedding vector sequence corresponding to the base input and the representative embedding vector sequence corresponding to the context input. This will be discussed later in conjunction with... Figure 3 and Figure 4 This illustrates an exemplary process for generating a text input representation.

[0036] At point 112, the click probability of a candidate content item being clicked can be predicted based on the text input representation. For example, when the context input includes the target user's historical interactions, the click probability of a candidate content item being clicked by the target user can be predicted.

[0037] For each content item in a set of candidate content items that can be recommended to the target user, process 100 can be executed to obtain a set of click probabilities. Based on the obtained set of click probabilities, the content item to be recommended to the target user can be determined.

[0038] It should be understood that the above text, in combination with... Figures 1 to 2 The described process for content recommendation based on embedding vector summaries is merely exemplary. Depending on the specific application requirements, the steps in the process can be replaced or modified in any way, and the process may include more or fewer steps. Furthermore, the specific order or hierarchy of the steps in process 100 is merely exemplary, and the process for content recommendation based on embedding vector summaries can be performed in an order different from that described.

[0039] The process 100 can be performed using a click probability prediction model according to embodiments of the present disclosure, thereby predicting the click probability of a candidate content item being clicked. Figure 3An exemplary process 300 according to an embodiment of the present disclosure is shown for predicting the click probability of a candidate content item being clicked using a click probability prediction model. In process 300, the click probability prediction model 320 includes an embedding layer 330, a pooling layer 340, a summarizing layer 350, a set of converter layers 360, a prediction layer 380, etc.

[0040] First, text input 302 can be obtained. Text input 302 includes basic input 304. Basic input 304 may include only candidate content items 306 or both candidate content items 306 and query 308. Candidate content items 306 may be content items from a set of candidate content items that can be recommended to the target user. Candidate content items may include, for example, news, videos, movies, books, music, web pages, product information, etc. Query 308 may be a search query from the target user, such as a search query entered by the target user in a search engine, news software, video application, etc. In addition, text input 302 also includes context input 310 corresponding to basic input 304. Context input 310 may include the target user's historical interactions, such as long-term historical interactions and real-time historical interactions. The user's historical interactions are collected and used with the user's knowledge and permission. If basic input 304 includes query 308, context input 310 may also include historical search results for query 308. Historical search results can be used to explain and expand query 308. The text length of the context input 310 is usually much longer than the text length of the base input 304.

[0041] Subsequently, an embedding vector sequence corresponding to the basic input 304 and an embedding vector sequence corresponding to the context input 310 can be generated. A basic input word block sequence corresponding to the basic input 304 and a context input word block sequence corresponding to the context input 310 can be obtained. Further, based on at least one of the token embedding sequence, segment embedding sequence, and position embedding sequence corresponding to the basic input word block sequence and the context input word block sequence, an embedding vector sequence corresponding to the basic input word block sequence and the context input word block sequence can be generated to obtain the embedding vector sequence corresponding to the basic input word block sequence and the context input word block sequence.

[0042] For example, word segmentation can be used to obtain candidate content item block sequence 314 corresponding to candidate content item 306, query block sequence 316 corresponding to query 308, and context input block sequence 318 corresponding to context input 310. Additionally, predefined codes can be added before the block sequences, such as the classification predefined code 312 "[CLS]". Predefined codes can also be inserted into the block sequences, such as the separator predefined code "[SEP]", to separate blocks corresponding to different text segments.

[0043] The classification predefined encoding 312, candidate content item chunk sequence 314, query chunk sequence 316, and context input chunk sequence 318 can be provided to the embedding layer 330 in the click probability prediction model 320. The structure and function of the embedding layer 330 can be similar to the structure and function of the embedding layer in known transformer-based models such as BERT. Compared to the classification predefined encoding 312, candidate content item chunk sequence 314, and query chunk sequence 316, the context input chunk sequence 318 typically contains a much larger number of chunks. That is, the vast majority of the input to the embedding layer 330 comes from the context input chunk sequence 318.

[0044] Embedding vectors corresponding to the predefined classification code 312 can be generated through embedding layer 330. For example, an initial classification embedding vector 332 corresponding to the predefined classification code 312 can be generated through embedding layer 330 based on at least one of the word chunk embedding vector, fragment embedding vector, and positional embedding vector corresponding to the predefined classification code 312. Similarly, an initial candidate content item embedding vector sequence 334 corresponding to candidate content item 306 can be generated through embedding layer 330 based on at least one of the word chunk embedding vector sequence, fragment embedding vector sequence, and positional embedding vector sequence corresponding to candidate content item word chunk sequence 314. An initial query embedding vector sequence 336 corresponding to query 308 can be generated through embedding layer 330 based on at least one of the word chunk embedding vector sequence, fragment embedding vector sequence, and positional embedding vector sequence corresponding to query word chunk sequence 316. An initial context input embedding vector sequence 338 corresponding to context input 310 can be generated through embedding layer 330 based on at least one of the word chunk embedding vector sequence, fragment embedding vector sequence, and positional embedding vector sequence corresponding to context input word chunk sequence 318.

[0045] The embedding vector sequence corresponding to the basic input 304, such as the initial candidate content item embedding vector sequence 334 and the initial query embedding vector sequence 336, can be provided to the pooling layer 340. Pooling operations can be performed on the initial candidate content item embedding vector sequence 334 and the initial query embedding vector sequence 336 by the pooling layer 340 to generate a pooled embedding vector 342 corresponding to the basic input 304. The pooling operation can be any known pooling operation, such as average pooling, max pooling, attention pooling, etc.

[0046] Next, a summarization operation can be performed on the initial context input embedding vector sequence 338 using the summarization layer 350 and pooling embedding vector 342 to obtain an initial representative context input embedding vector sequence 352 corresponding to the context input 310. The number of embedding vectors in the initial representative context input embedding vector sequence 352 can be significantly smaller than the number of embedding vectors in the initial context input embedding vector sequence 338. Preferably, the initial representative context input embedding vector sequence 352 may include embedding vectors selected from the initial context input embedding vector sequence 338 that are related to the base input 304. This can be achieved by combining... Figure 2 The described process 200 performs a summarization operation on the initial context input embedding vector sequence 338. For example, the similarity between the pooled embedding vector 342 and each of the multiple context input embedding vectors included in the initial context input embedding vector sequence 338 can be calculated first to obtain multiple similarities. Subsequently, these multiple context input embedding vectors can be sorted based on the obtained multiple similarities. Next, a predetermined number of context input embedding vectors with the top ranking can be selected from the sorted multiple context input embedding vectors. Then, the selected context input embedding vectors can be combined into an initial representative context input embedding vector sequence 352. Preferably, in order to separate the embedding vectors corresponding to different text segments from each other, the embedding vectors corresponding to the predefined code "[SEP]" of the delimiter can be retained.

[0047] After obtaining the initial representative context input embedding vector sequence 352, a self-attention mechanism can be used to generate classification embedding vectors, a basic input embedding vector sequence, and a representative context input embedding vector sequence based on the embedding vector corresponding to the classification predefined encoding 312, the embedding vector sequence corresponding to the basic input 304, and the representative embedding vector sequence corresponding to the context input 310. For example, at a set of transformer layers 306, a self-attention mechanism can be used to generate classification embedding vector 362, candidate content item embedding vector sequence 364, query embedding vector sequence 366, and representative context input embedding vector sequence 368 based on the initial classification embedding vector 332, the initial candidate content item embedding vector sequence 334, the initial query embedding vector sequence 336, and the initial representative context input embedding vector sequence 352. A set of transformer layers 360 includes, for example... Each transformer layer has a structure and function similar to those of transformer layers in known transformer-based models such as BERT. Subsequently, the classification embedding vector 362 can be used as the text input representation 370 of the text input 302.

[0048] Finally, the click probability 382 of the candidate content item 306 being clicked can be predicted by the prediction layer 380 based on the text input representation 370.

[0049] In process 300, after generating the initial candidate content item embedding vector sequence 334, the initial query embedding vector sequence 336, and the initial context input embedding vector sequence 338 through the embedding layer 330 in the click probability prediction model 320, a pooling operation is performed on the initial candidate content item embedding vector sequence 334 and the initial query embedding vector sequence 336 through the pooling layer 340 to generate a pooled embedding vector 342. Next, a summarizing operation is performed on the initial context input embedding vector sequence 338 through the summarizing layer 350 using the pooled embedding vector 342 to obtain the initial representative context input embedding vector sequence 352. The number of embedding vectors in the initial representative context input embedding vector sequence 352 can be significantly smaller than the number of embedding vectors in the initial context input embedding vector sequence 338. Therefore, by performing pooling and summarizing operations above the embedding layer 330, the number of embedding vectors provided to the set of converter layers 360 above the embedding layer 330 can be reduced. The number of converters included in each converter layer corresponds to the number of embedding vectors provided to that converter layer. With a reduction in the number of embedding vectors to be processed, the number of converters can also be reduced accordingly. Furthermore, the computational cost of the self-attention calculations required can be reduced accordingly. Therefore, the latency in generating the text input representation 370 and predicting the click probability 382 can be decreased.

[0050] In process 300, pooling and summarizing operations are performed between the embedding layer and the bottommost converter layer. Embodiments of this disclosure also propose performing pooling and summarizing operations between two sets of converter layers. Figure 4 Another exemplary process 400 for predicting the click probability of a candidate content item being clicked using a click probability prediction model according to an embodiment of the present disclosure is shown. In process 400, the click probability prediction model 420 includes an embedding layer 430, a lower set of converter layers 440, a pooling layer 450, a summarizing layer 460, an upper set of converter layers 470, a prediction layer 490, etc. The lower set of converter layers 440 includes, for example... A set of converter layers 470 on the upper part includes, for example, a converter layer. A converter layer, in which ,and .

[0051] First, text input 402 can be obtained. Text input 402 includes basic input 404. Basic input 404 may include only candidate content item 406 or both candidate content item 406 and query 408. Additionally, text input 402 also includes context input 410 corresponding to basic input 404. Text input 402, basic input 404, candidate content item 406, query 408, and context input 410 can respectively correspond to… Figure 3 The input consists of text input 302, basic input 304, candidate content item 306, query 308, and context input 310. The text length of context input 410 is usually much longer than the text length of basic input 404.

[0052] Subsequently, an embedding vector sequence corresponding to the basic input 404 and an embedding vector sequence corresponding to the context input 410 can be generated. A basic input chunk sequence corresponding to the basic input 404 and a context input chunk sequence corresponding to the context input 410 can be obtained. Further, an initial basic input embedding vector sequence and an initial context input embedding vector sequence can be generated based on at least one of the chunk embedding vector sequence, fragment embedding vector sequence, and position embedding vector sequence corresponding to the basic input chunk sequence and the context input chunk sequence.

[0053] For example, word segmentation can be used to obtain candidate content item block sequence 414 corresponding to candidate content item 406, query block sequence 416 corresponding to query 408, and context input block sequence 418 corresponding to context input 410. Additionally, predefined codes, such as the category predefined code 412 "[CLS]", can be added before the block sequence. Predefined codes, such as the separator predefined code "[SEP]", can also be inserted into the block sequence to separate blocks corresponding to different text segments.

[0054] The classification predefined encoding 412, candidate content item chunk sequence 414, query chunk sequence 416, and context input chunk sequence 418 can be provided to the embedding layer 430 in the click probability prediction model 420. The structure and function of the embedding layer 430 can be similar to the structure and function of the embedding layer in known transformer-based models such as BERT. Compared to the classification predefined encoding 412, candidate content item chunk sequence 414, and query chunk sequence 416, the context input chunk sequence 418 typically contains a much larger number of chunks. That is, the vast majority of the input to the embedding layer 430 comes from the context input chunk sequence 418.

[0055] Embedding vectors corresponding to the predefined classification code 412 can be generated through embedding layer 430. For example, an initial classification embedding vector 432 corresponding to the predefined classification code 412 can be generated through embedding layer 430 based on at least one of the word chunk embedding vector, fragment embedding vector, and position embedding vector corresponding to the predefined classification code 412. Similarly, an initial candidate content item embedding vector sequence 434 corresponding to candidate content item 406 can be generated through embedding layer 430 based on at least one of the word chunk embedding vector sequence, fragment embedding vector sequence, and position embedding vector sequence corresponding to candidate content item word chunk sequence 414. An initial query embedding vector sequence 436 corresponding to query 408 can be generated through embedding layer 430 based on at least one of the word chunk embedding vector sequence, fragment embedding vector sequence, and position embedding vector sequence corresponding to query word chunk sequence 416. An initial context input embedding vector sequence 438 corresponding to context input 410 can be generated through embedding layer 430 based on at least one of the word chunk embedding vector sequence, fragment embedding vector sequence, and position embedding vector sequence corresponding to context input word chunk sequence 418.

[0056] Subsequently, a self-attention mechanism can be used to generate an embedding vector sequence corresponding to the basic input 404 and an embedding vector sequence corresponding to the context input 410, based on the initial basic input embedding vector sequence and the initial context input embedding vector sequence. For example, the initial classification embedding vector 432, the initial candidate content item embedding vector sequence 434, the initial query embedding vector sequence 436, and the initial context input embedding vector sequence 438 can be provided to a set of lower transformer layers 440. The structure and function of each transformer layer can be similar to the structure and function of transformer layers in known transformer-layer-based models such as the BERT model. At transformer layer 440, an intermediate classification embedding vector 442, an intermediate candidate content item embedding vector sequence 444, an intermediate query embedding vector sequence 446, and an intermediate context input embedding vector sequence 448 can be generated based on the initial classification embedding vector 432, the initial candidate content item embedding vector sequence 434, the initial query embedding vector sequence 436, and the initial context input embedding vector sequence 438, using a self-attention mechanism.

[0057] The embedding vector sequence corresponding to the basic input 404, such as the intermediate candidate content item embedding vector sequence 444 and the intermediate query embedding vector sequence 446, can be provided to the pooling layer 450. The pooling layer 450 can perform a pooling operation on the intermediate candidate content item embedding vector sequence 444 and the intermediate query embedding vector sequence 446 to generate the pooled embedding vector 452 corresponding to the basic input 404. The pooling operation can be any known pooling operation, such as average pooling, max pooling, attention pooling, etc.

[0058] Next, a summarization operation can be performed on the intermediate context input embedding vector sequence 448 using the summarization layer 460 and pooling embedding vector 452 to obtain an initial representative context input embedding vector sequence 462 corresponding to the context input 410. The number of embedding vectors in the initial representative context input embedding vector sequence 462 can be significantly smaller than the number of embedding vectors in the intermediate context input embedding vector sequence 448. Preferably, the initial representative context input embedding vector sequence 462 may include embedding vectors selected from the intermediate context input embedding vector sequence 448 that are related to the base input 404. This can be achieved by combining... Figure 2The described process 200 performs a summarization operation on the intermediate context input embedding vector sequence 448. For example, the similarity between the pooled embedding vector 452 and each of the multiple context input embedding vectors included in the intermediate context input embedding vector sequence 448 can be calculated first to obtain multiple similarities. Subsequently, these multiple context input embedding vectors can be sorted based on the obtained multiple similarities. Next, a predetermined number of context input embedding vectors with the top ranking can be selected from the sorted multiple context input embedding vectors. Then, the selected context input embedding vectors can be combined into an initial representative context input embedding vector sequence 462. Preferably, in order to separate the embedding vectors corresponding to different text segments from each other, the embedding vectors corresponding to the predefined code "[SEP]" of the delimiter can be retained.

[0059] After obtaining the initial representative context input embedding vector sequence 462, a self-attention mechanism can be used to generate classification embedding vectors, basic input embedding vector sequences, and representative context input embedding vector sequences based on the embedding vectors corresponding to the classification predefined encoding 412, the embedding vector sequence corresponding to the basic input 404, and the representative embedding vector sequence corresponding to the context input 410. For example, at a set of transformer layers 470 at the top, a self-attention mechanism can be used to generate classification embedding vectors 472, candidate content item embedding vector sequences 474, query embedding vector sequences 476, and representative context input embedding vector sequences 478 based on intermediate classification embedding vectors 442, intermediate candidate content item embedding vector sequences 444, intermediate query embedding vector sequences 446, and intermediate context input embedding vector sequences 448. Subsequently, the classification embedding vector 472 can be used as the text input representation 480 of the text input 402.

[0060] Finally, the click probability 492 of the candidate content item 406 being clicked can be predicted by the prediction layer 490 based on the text input representation 480.

[0061] In process 400, after generating intermediate candidate content item embedding vector sequences 444, intermediate query embedding vector sequences 446, and intermediate context input embedding vector sequences 448 through a lower set of converter layers 440 in the click probability prediction model 420, a pooling embedding vector 452 is generated by performing a pooling operation on the intermediate candidate content item embedding vector sequences 444 and intermediate query embedding vector sequences 446 through a pooling layer 450. Next, a summarizing operation is performed on the intermediate context input embedding vector sequence 448 using the pooled embedding vector 452 through a summarizing layer 460 to obtain an initial representative context input embedding vector sequence 462. The number of embedding vectors in the initial representative context input embedding vector sequence 462 can be significantly smaller than the number of embedding vectors in the intermediate context input embedding vector sequence 448. Therefore, by performing pooling and summarizing operations above the lower set of converter layers 440, the number of embedding vectors provided to the upper set of converter layers 470 can be reduced. The number of converters included in each converter layer corresponds to the number of embedding vectors provided to that converter layer. With a reduced number of embedding vectors to be processed, the number of transformers can also be reduced accordingly. Furthermore, the computational cost of the self-attention calculations required can also be reduced. Therefore, the latency for generating the text input representation 480 and predicting the click probability 492 can be decreased.

[0062] In processes 300 and 400, pooled embedding vectors can be used to perform a summarization operation on the sequence of embedding vectors corresponding to the context input, thereby reducing the number of embedding vectors provided to subsequent transformer layers. Pooled embedding vectors can be generated by performing a pooling operation on the sequence of embedding vectors corresponding to the underlying input, which includes candidate content items and / or queries. Accordingly, the pooled embedding vectors can represent the meaning of the candidate content items and / or queries. Using such pooled embedding vectors to perform a summarization operation allows for the selection of embedding vectors relevant to the candidate content items and / or queries from the sequence of embedding vectors corresponding to the context input. Since the goal of the click probability prediction model is to predict the click probability of a candidate content item being clicked, selecting embedding vectors relevant to the candidate content items is beneficial. For example, the context input may include the target user's historical interactions. Historical interactions can reflect the target user's interests. Performing a summarization operation using candidate content items allows for the determination of the target user's interest in the candidate content items based on their historical interactions, and can guide the target user's interests. Queries can reflect the target user's intent. Considering the query when selecting embedding vectors can help select embedding vectors relevant to the target user's intent. The above method can reduce the number of embedding vectors while retaining the key embedding vectors that are important for click probability prediction. This helps maintain the accuracy of the prediction results.

[0063] It should be understood that the above text, in combination with... Figure 3 and Figure 4 The described process for predicting the click probability of a candidate content item being clicked using a click probability prediction model is merely exemplary. Depending on the specific application requirements, the steps in the process for predicting the click probability can be replaced or modified in any way, and the process may include more or fewer steps. For example, although in processes 300 and 400 the basic input includes both candidate content items and a query, in some embodiments, the basic input may include only candidate content items. In this case, only the sequence of embedding vectors corresponding to the candidate content items can be generated. Accordingly, pooled embedding vectors can be obtained simply by performing a pooling operation on the sequence of embedding vectors corresponding to the candidate content items. Furthermore, the specific order or hierarchy of the steps in processes 300 and 400 is merely exemplary, and the process for predicting the click probability may be performed in an order different from the described order. Moreover, Figure 3 and Figure 4 The click probability prediction models 320 and 420 shown are merely examples of click probability prediction models. Depending on the specific application requirements, click probability prediction models can have any other structure and can include more or fewer layers.

[0064] Before deploying a click probability prediction model for online click probability prediction, it can be trained. During training, the model's parameters or weights can be progressively optimized as the training loss is minimized, allowing the model to learn key embedding vectors important for click probability prediction. A representative sequence of embedding vectors corresponding to the context input is selected from the original embedding vector sequence by calculating similarity with candidate content items and / or the query. The selected representative sequence of embedding vectors corresponding to the context input is dynamically changed during training and can be progressively optimized. Self-attention operations can be performed on the selected representative embedding vector sequence through subsequent transformer layers to further ensure learning effectiveness. The model with the minimum training loss can be used as the final model and deployed for online click probability prediction tasks.

[0065] Click probability prediction models can be trained using a multi-stage training approach. For example, training a click probability prediction model can include a first-stage training and a second-stage training. Figure 5An exemplary process 500 for the first stage training of a click probability prediction model according to an embodiment of this disclosure is illustrated. In the first stage training, the architecture of the click probability prediction model may differ from that of an online-deployed click probability prediction model. As an example, click probability prediction models 320 and 420 may be online-deployed click probability prediction models. In process 500, click probability prediction model 520 includes an embedding layer 530, a set of transformer layers 540, a prediction layer 560, etc. Compared to click probability prediction models 320 and 420, click probability prediction model 520 does not include a pooling layer and a summarizing layer.

[0066] First, text input 502 can be obtained. Text input 502 includes basic input 504. Basic input 504 may include only candidate content item 506 or both candidate content item 506 and query 508. Additionally, text input 502 also includes context input 510 corresponding to basic input 504. Text input 502, basic input 504, candidate content item 506, query 508, and context input 510 can respectively correspond to… Figure 3 The input consists of text input 302, basic input 304, candidate content item 306, query 308, and context input 310. The text length of context input 510 is usually much longer than the text length of basic input 504.

[0067] Through word segmentation, we can obtain candidate content item block sequence 514 corresponding to candidate content item 506, query block sequence 516 corresponding to query 508, and context input block sequence 518 corresponding to context input 510. Additionally, predefined codes, such as the category predefined code 512 "[CLS]", can be added before the block sequences. Predefined codes, such as the separator predefined code "[SEP]", can also be inserted into the block sequences to separate blocks corresponding to different text segments.

[0068] The classification predefined code 512, candidate content item chunk sequence 514, query chunk sequence 516, and context input chunk sequence 518 can be provided to the embedding layer 530 in the click probability prediction model 520. The structure and function of the embedding layer 530 can be similar to the structure and function of the embedding layer in known transformer-based models such as BERT. An initial classification embedding vector 532 corresponding to the classification predefined code 512 can be generated by the embedding layer 530 based on at least one of the chunk embedding vector, fragment embedding vector, and position embedding vector corresponding to the classification predefined code 512. Similarly, an initial candidate content item embedding vector sequence 534 corresponding to the candidate content item 506 can be generated by the embedding layer 530 based on at least one of the chunk embedding vector sequence, fragment embedding vector sequence, and position embedding vector sequence corresponding to the candidate content item chunk sequence 514. An initial query embedding vector sequence 536 corresponding to query 508 can be generated through embedding layer 530 based on at least one of the word chunk embedding vector sequence, fragment embedding vector sequence, and position embedding vector sequence corresponding to query word chunk sequence 516. An initial context input embedding vector sequence 538 corresponding to context input 510 can be generated through embedding layer 530 based on at least one of the word chunk embedding vector sequence, fragment embedding vector sequence, and position embedding vector sequence corresponding to context input word chunk sequence 518.

[0069] The first phase of training does not involve pooling or summarizing operations. Accordingly, the text input representation can be generated at least based on the embedding vector sequence corresponding to the base input 504 and the embedding vector sequence corresponding to the context input 510. The initial classification embedding vector 532, the initial candidate content item embedding vector sequence 534, the initial query embedding vector sequence 536, and the initial context input embedding vector sequence 538 can be provided to a set of transformer layers 540. The set of transformer layers 540 includes, for example... There are several transformer layers. The structure and function of each transformer layer can be similar to those of transformer layers in known transformer-based models such as BERT. At a set of transformer layers 540, a self-attention mechanism can be used to generate a classification embedding vector 542, a candidate content item embedding vector sequence 544, a query embedding vector sequence 546, and a representative context input embedding vector sequence 548 based on an initial classification embedding vector 532, an initial candidate content item embedding vector sequence 534, an initial query embedding vector sequence 536, and an initial context input embedding vector sequence 538. Subsequently, the classification embedding vector 542 can be used as the text input representation 550 of the text input 502.

[0070] Finally, the click probability 562 of the candidate content item 506 being clicked can be predicted by the prediction layer 560 based on the text input representation 550.

[0071] The first stage of training, as shown in process 500, does not involve pooling or summarizing operations. Its purpose is to optimize the embedding layer 530 so that it has reasonable parameters or weights, thereby generating a high-quality sequence of embedding vectors. In this first stage of training, the number of transformers included in the transformer layer can be the same as the number of the original embedding vectors without summarizing operations.

[0072] After performing the first stage of training on the click probability prediction model, a second stage of training can be performed. The second stage of training builds upon the first stage. In the second stage, the architecture of the click probability prediction model can be similar to that of an online-deployed click probability prediction model. The embedding layers in the click probability prediction model can be the embedding layers obtained through the first stage of training. Pooling and summarizing operations can be performed in the second stage. Because the embedding layers obtained through the first stage of training have reasonable parameters or weights, high-quality embedding vector sequences can be generated. Accordingly, performing pooling and summarizing operations based on such embedding vector sequences allows for the selection of key, representative embedding vector sequences important for click probability prediction from the embedding vector sequences corresponding to the context input. Therefore, the number of embedding vectors provided to subsequent transformer layers can be reduced in the second stage of training. The number of transformers included in each transformer layer corresponds to the number of embedding vectors provided to that transformer layer. With a reduced number of embedding vectors to be processed, the number of transformers can also be reduced accordingly. The number of transformers included in a transformer layer can be consistent with the number of embedding vectors after the summarizing operation.

[0073] Therefore, the number of converters included in the converter layer of the click probability prediction model can differ between the first and second training phases. Assuming the converter layer can include a first number of converters in the first training phase and a second number of converters in the second training phase, the second number can be less than the first number. The click probability prediction model obtained after the second training phase can be deployed for online click probability prediction tasks.

[0074] Compared to traditional neural network models based on converter layer structures, the click probability prediction model according to embodiments of this disclosure, for example... Figure 3 and Figure 4Click probability prediction models 320 and 420, as shown, additionally include pooling layers and summarizing layers. Pooling layers perform pooling operations on the sequence of embedding vectors corresponding to the basic input. Summarizing layers perform summarizing operations on the sequence of embedding vectors corresponding to the context input. The processing by pooling and summarizing layers reduces the number of embedding vectors provided to subsequent transformer layers. Pooling and summarizing layers can be located between the embedding layer and the bottommost transformer layer, or between two sets of transformer layers. In click probability prediction model 320, the pooling and summarizing layers are located between the embedding layer and the bottommost transformer layer. The number of embedding vectors provided to the bottommost transformer layer can be reduced. Therefore, the number of transformers included in each transformer layer can be reduced. This can significantly reduce prediction latency. In click probability prediction model 420, the pooling and summarizing layers are located between two sets of transformer layers, for example, above one or more transformer layers. Only the number of embedding vectors provided to the upper set of transformer layers can be reduced. Therefore, only the number of transformers included in the upper set of transformer layers can be reduced. This can reduce prediction latency to some extent. The lower the pooling and summarizing layers are in the model, the greater the reduction in prediction latency. Furthermore, in the click probability prediction model 420, the pooling and summarizing operations are performed based on the embedding vector sequence generated by the transformer layer. This embedding vector sequence is generated through a self-attention mechanism, thus possessing high quality. Therefore, for the click probability prediction model 420, single-stage training is feasible. For example, the first stage of training described above can be skipped, and the second stage of training can be performed directly. This can significantly save training costs. The position of the pooling and summarizing layers in the click probability prediction model can be determined based on the actual application requirements, considering both real-time prediction requirements and training cost requirements. If real-time prediction requirements are high, the pooling and summarizing layers can be placed between the embedding layers and the bottom-most transformer layer; conversely, if training cost requirements are high, the pooling and summarizing layers can be placed between two sets of transformer layers.

[0075] Figure 6 This is a flowchart of an exemplary method 600 for content recommendation based on embedded vector summaries, according to embodiments of this disclosure.

[0076] At point 610, text input can be obtained, which includes basic input and contextual input corresponding to the basic input, the basic input including at least candidate content items.

[0077] At position 620, an embedding vector sequence corresponding to the basic input and an embedding vector sequence corresponding to the context input can be generated.

[0078] At 630, a pooled embedding vector corresponding to the basic input can be generated by performing a pooling operation on the sequence of embedding vectors corresponding to the basic input.

[0079] At 640, a representative sequence of embedding vectors corresponding to the context input can be obtained by performing a summary operation on the sequence of embedding vectors corresponding to the context input using the pooled embedding vectors.

[0080] At 650, the text input representation of the text input can be generated based at least on the embedding vector sequence corresponding to the basic input and the representative embedding vector sequence corresponding to the context input.

[0081] At position 660, the click probability of the candidate content item being clicked can be predicted based on the text input representation.

[0082] In one implementation, generating the embedding vector sequence corresponding to the basic input and the embedding vector sequence corresponding to the context input may include: obtaining a basic input chunk sequence corresponding to the basic input and a context input chunk sequence corresponding to the context input; and generating the embedding vector sequence corresponding to the basic input and the embedding vector sequence corresponding to the context input based on at least one of the chunk embedding vector sequence, the fragment embedding vector sequence, and the position embedding vector sequence corresponding to the basic input chunk sequence and the context input chunk sequence.

[0083] In one implementation, generating the embedding vector sequence corresponding to the basic input and the embedding vector sequence corresponding to the context input may include: obtaining a basic input chunk sequence corresponding to the basic input and a context input chunk sequence corresponding to the context input; generating an initial basic input embedding vector sequence and an initial context input embedding vector sequence based on at least one of a chunk embedding vector sequence, a fragment embedding vector sequence, and a position embedding vector sequence corresponding to the basic input chunk sequence and the context input chunk sequence; and generating the embedding vector sequence corresponding to the basic input and the embedding vector sequence corresponding to the context input based on the initial basic input embedding vector sequence and the initial context input embedding vector sequence through a self-attention mechanism.

[0084] In one implementation, the number of embedding vectors in the representative embedding vector sequence corresponding to the context input may be less than the number of embedding vectors in the embedding vector sequence corresponding to the context input.

[0085] In one implementation, the representative embedding vector sequence corresponding to the context input may include embedding vectors selected from the embedding vector sequence corresponding to the context input that are associated with the base input.

[0086] In one implementation, the sequence of embedding vectors corresponding to the context input may include multiple context input embedding vectors. The summarizing operation may include: calculating the similarity between the pooled embedding vector and each of the multiple context input embedding vectors to obtain multiple similarities; sorting the multiple context input embedding vectors based on the multiple similarities to obtain a sorted sequence of context input embedding vectors; selecting a predetermined number of top-ranking context input embedding vectors from the sorted sequence; and combining the selected context input embedding vectors into a representative embedding vector sequence corresponding to the context input.

[0087] In one implementation, generating the text input representation may include: generating an embedding vector corresponding to a predefined classification code; generating a classification embedding vector, a basic input embedding vector sequence, and a representative context input embedding vector sequence based on the embedding vector corresponding to the predefined classification code, the embedding vector sequence corresponding to the basic input, and the representative embedding vector sequence corresponding to the context input through a self-attention mechanism; and using the classification embedding vector as the text input representation.

[0088] In one implementation, the basic input may also include a query.

[0089] The context input may include historical search results for the query.

[0090] In one implementation, the contextual input may include the target user's historical interactions. The predicted click probability may include predicting the probability that the candidate content item will be clicked by the target user.

[0091] In one implementation, the click probability can be predicted using a click probability prediction model. The representative embedding vector sequence corresponding to the context input can be progressively optimized during the training of the click probability prediction model.

[0092] In one implementation, the click probability can be predicted using a click probability prediction model. Training the click probability prediction model can include a first-stage training and a second-stage training. The number of converters included in the converter layer of the click probability prediction model can differ between the first-stage training and the second-stage training.

[0093] In the first training phase, the converter layer in the click probability prediction model may include a first number of converters. In the second training phase, the converter layer in the click probability prediction model may include a second number of converters. The second number may be less than the first number.

[0094] In the first stage of training, the text input representation may be generated based at least on the embedding vector sequence corresponding to the base input and the embedding vector sequence corresponding to the context input.

[0095] In one implementation, the candidate content items may include at least one of news, videos, movies, books, music, web pages, and product information.

[0096] It should be understood that method 600 may also include any steps / processes for content recommendation based on embedded vector summaries according to embodiments of the present disclosure as described above.

[0097] Figure 7 An exemplary apparatus 700 for content recommendation based on embedded vector summaries is shown according to an embodiment of this disclosure.

[0098] The apparatus 700 may include: a text input acquisition module 710, for acquiring text input, the text input including a base input and a context input corresponding to the base input, the base input including at least candidate content items; an embedding vector sequence generation module 720, for generating an embedding vector sequence corresponding to the base input and an embedding vector sequence corresponding to the context input; a pooling embedding vector generation module 730, for generating a pooled embedding vector corresponding to the base input by performing a pooling operation on the embedding vector sequence corresponding to the base input; a representative embedding vector sequence acquisition module 740, for obtaining a representative embedding vector sequence corresponding to the context input by performing a summarization operation on the embedding vector sequence corresponding to the context input using the pooled embedding vector; a text input representation generation module 750, for generating a text input representation of the text input based at least on the embedding vector sequence corresponding to the base input and the representative embedding vector sequence corresponding to the context input; and a click probability prediction module 760, for predicting the click probability of the candidate content item being clicked based on the text input representation. Furthermore, the apparatus 700 may also include any other modules configured for content recommendation based on embedded vector summaries, according to embodiments of the present disclosure as described above.

[0099] Figure 8 An exemplary apparatus 800 for content recommendation based on embedded vector summaries is shown according to an embodiment of this disclosure.

[0100] The apparatus 800 may include: a processor 810; and a memory 820 storing computer-executable instructions. When executed, the computer-executable instructions cause the processor 810 to: obtain text input, the text input including a base input and a context input corresponding to the base input, the base input including at least candidate content items; generate an embedding vector sequence corresponding to the base input and an embedding vector sequence corresponding to the context input; generate a pooled embedding vector corresponding to the base input by performing a pooling operation on the embedding vector sequence corresponding to the base input; obtain a representative embedding vector sequence corresponding to the context input by performing a summarization operation on the embedding vector sequence corresponding to the context input using the pooled embedding vectors; generate a text input representation of the text input based at least on the embedding vector sequence corresponding to the base input and the representative embedding vector sequence corresponding to the context input; and predict the click probability of the candidate content item being clicked based on the text input representation.

[0101] In one implementation, the representative embedding vector sequence corresponding to the context input may include embedding vectors selected from the embedding vector sequence corresponding to the context input that are associated with the base input.

[0102] In one implementation, the basic input may also include a query.

[0103] In one implementation, the click probability can be predicted using a click probability prediction model. Training the click probability prediction model can include a first-stage training and a second-stage training. The number of converters included in the converter layer of the click probability prediction model can differ between the first-stage training and the second-stage training.

[0104] It should be understood that the processor 810 may also perform any other steps / processes of the method for content recommendation based on embedded vector summaries according to the embodiments of this disclosure as described above.

[0105] Embodiments of this disclosure propose a computer program product for content recommendation based on embedding vector summarization, comprising a computer program executed by a processor to: obtain text input, the text input including a base input and a context input corresponding to the base input, the base input including at least candidate content items; generate an embedding vector sequence corresponding to the base input and an embedding vector sequence corresponding to the context input; generate a pooled embedding vector corresponding to the base input by performing a pooling operation on the embedding vector sequence corresponding to the base input; obtain a representative embedding vector sequence corresponding to the context input by performing a summarization operation on the embedding vector sequence corresponding to the context input using the pooled embedding vector; generate a text input representation of the text input based at least on the embedding vector sequence corresponding to the base input and the representative embedding vector sequence corresponding to the context input; and predict a click probability of the candidate content item being clicked based on the text input representation. Furthermore, the computer program may also be executed to perform any other steps / processes for implementing the method for content recommendation based on embedding vector summarization according to embodiments of this disclosure as described above.

[0106] Embodiments of this disclosure may be embodied in a non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions that, when executed, cause one or more processors to perform any operation of the method for content recommendation based on embedded vector summaries according to embodiments of this disclosure as described above.

[0107] It should be understood that all operations in the methods described above are merely exemplary, and this disclosure is not limited to any operation in the methods or the order of such operations, but should cover all other equivalent variations under the same or similar concept. Furthermore, unless otherwise specified or clearly indicated from the context as relating to the singular form, the articles “a” and “an” as used in this specification and the appended claims should generally be interpreted as meaning “one” or “one or more”.

[0108] It should also be understood that all modules in the apparatus described above can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. Furthermore, any of these modules can be further functionally divided into sub-modules or combined together.

[0109] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software, or any combination thereof. Whether these processors are implemented as hardware or software will depend on the specific application and the overall design constraints imposed on the system. As an example, the processors, any portions of processors, or any combination of processors given in this disclosure can be implemented using microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic units, discrete hardware circuits, and other suitable processing components configured to perform the various functions described in this disclosure. The functionality of the processors, any portions of processors, or any combination of processors given in this disclosure can be implemented using software executed by a microprocessor, microcontroller, DSP, or other suitable platform.

[0110] Software should be broadly considered to refer to instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, procedures, functions, etc. Software may reside on a computer-readable medium. Computer-readable media may include, for example, memory, which may be, for example, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical disks, smart cards, flash memory devices, random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, or removable disks. Although memory is shown as separate from the processor in several aspects set forth in this disclosure, memory may also reside within the processor, such as in caches or registers.

[0111] The above description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of the elements of the various aspects described herein, known or likely to be known by one of ordinary skill in the art, are expressly incorporated herein and covered by the claims.

Claims

1. A method for content recommendation based on embedding vector summarization, comprising: Obtain text input, which includes basic input and context input corresponding to the basic input, wherein the basic input includes at least candidate content items; Generate an embedding vector sequence corresponding to the basic input and an embedding vector sequence corresponding to the context input; A pooled embedding vector corresponding to the basic input is generated by performing a pooling operation on the sequence of embedding vectors corresponding to the basic input. A representative embedding vector sequence corresponding to the context input is obtained by performing a summarization operation on the embedding vector sequence corresponding to the context input using the pooled embedding vector; A text input representation of the text input is generated based at least on the embedding vector sequence corresponding to the basic input and the representative embedding vector sequence corresponding to the context input; as well as Based on the text input representation, predict the click probability of the candidate content item being clicked. The click probability is predicted by a click probability prediction model. The training of the click probability prediction model includes a first stage training and a second stage training. In the first stage training, the number of converters in each converter layer of the click probability prediction model is consistent with the number of original embedding vectors. In the second stage training, the number of converters in each converter layer of the click probability prediction model is consistent with the number of embedding vectors after the summarization operation is performed.

2. The method according to claim 1, wherein, The generation of the embedding vector sequence corresponding to the basic input and the embedding vector sequence corresponding to the context input includes: Obtain the basic input word block sequence corresponding to the basic input and the context input word block sequence corresponding to the context input; and Based on at least one of the word block embedding vector sequence, fragment embedding vector sequence, and position embedding vector sequence corresponding to the basic input word block sequence and the context input word block sequence, generate an embedding vector sequence corresponding to the basic input and an embedding vector sequence corresponding to the context input.

3. The method according to claim 1, wherein, The generation of the embedding vector sequence corresponding to the basic input and the embedding vector sequence corresponding to the context input includes: Obtain the basic input word block sequence corresponding to the basic input and the context input word block sequence corresponding to the context input; Based on at least one of the word chunk embedding vector sequence, fragment embedding vector sequence, and position embedding vector sequence corresponding to the basic input word chunk sequence and the context input word chunk sequence, an initial basic input embedding vector sequence and an initial context input embedding vector sequence are generated; and By employing a self-attention mechanism, an embedding vector sequence corresponding to the basic input and an embedding vector sequence corresponding to the context input are generated based on the initial basic input embedding vector sequence and the initial context input embedding vector sequence.

4. The method according to claim 1, wherein, The number of embedding vectors in the representative embedding vector sequence corresponding to the context input is less than the number of embedding vectors in the embedding vector sequence corresponding to the context input.

5. The method according to claim 1, wherein, The representative embedding vector sequence corresponding to the context input includes embedding vectors selected from the embedding vector sequence corresponding to the context input that are related to the basic input.

6. The method according to claim 1, wherein, The sequence of embedding vectors corresponding to the context input includes multiple context input embedding vectors, and the summarization operation includes: Calculate the similarity between the pooled embedding vector and each of the plurality of context input embedding vectors to obtain multiple similarities; Based on the multiple similarities, the multiple context input embedding vectors are sorted to obtain multiple sorted context input embedding vectors; Select a predetermined number of top-ranked context input embedding vectors from a plurality of sorted context input embedding vectors; and The selected context input embedding vectors are combined into a representative embedding vector sequence corresponding to the context input.

7. The method according to claim 1, wherein, The generated text input representation includes: Generate embedding vectors corresponding to the predefined classification codes; Using a self-attention mechanism, based on the embedding vector corresponding to the predefined classification encoding, the embedding vector sequence corresponding to the basic input, and the representative embedding vector sequence corresponding to the context input, a classification embedding vector, a basic input embedding vector sequence, and a representative context input embedding vector sequence are generated; and The classification embedding vector is used as the text input representation.

8. The method according to claim 1, wherein, The basic input also includes queries.

9. The method according to claim 8, wherein, The context input includes historical search results for the query.

10. The method according to claim 1, wherein, The contextual input includes the target user's historical interactions, and the predicted click probability includes: Predict the click probability of the candidate content item being clicked by the target user.

11. The method according to claim 1, wherein, The click probability is predicted by a click probability prediction model, and the representative embedding vector sequence corresponding to the context input is progressively optimized during the training of the click probability prediction model.

12. The method according to claim 1, wherein, In the first stage of training, the converter layer in the click probability prediction model includes a first number of converters, and in the second stage of training, the converter layer in the click probability prediction model includes a second number of converters, and the second number is less than the first number.

13. The method according to claim 1, wherein, In the first phase of training, the text input representation is generated based at least on the embedding vector sequence corresponding to the base input and the embedding vector sequence corresponding to the context input.

14. The method according to claim 1, wherein, The candidate content items include at least one of the following: news, videos, movies, books, music, web pages, and product information.

15. An apparatus for content recommendation based on embedded vector summarization, comprising: processor; as well as A memory storing computer-executable instructions, which, when executed, cause the processor to: Obtain text input, which includes basic input and contextual input corresponding to the basic input. The basic input includes at least candidate content items. Generate an embedding vector sequence corresponding to the basic input and an embedding vector sequence corresponding to the context input. Pooled embedding vectors corresponding to the basic input are generated by performing a pooling operation on the sequence of embedding vectors corresponding to the basic input. A representative sequence of embedding vectors corresponding to the context input is obtained by performing a summarization operation on the pooled embedding vectors. At least based on the embedding vector sequence corresponding to the basic input and the representative embedding vector sequence corresponding to the context input, a text input representation of the text input is generated, and Based on the text input representation, predict the click probability of the candidate content item being clicked. The click probability is predicted by a click probability prediction model. The training of the click probability prediction model includes a first stage training and a second stage training. In the first stage training, the number of converters in each converter layer of the click probability prediction model is consistent with the number of original embedding vectors. In the second stage training, the number of converters in each converter layer of the click probability prediction model is consistent with the number of embedding vectors after the summarization operation is performed.

16. The apparatus according to claim 15, wherein, The representative embedding vector sequence corresponding to the context input includes embedding vectors selected from the embedding vector sequence corresponding to the context input that are related to the basic input.

17. The apparatus according to claim 15, wherein, The basic input also includes queries.

18. A computer program product for content recommendation based on embedded vector summaries, comprising a computer program executed by a processor for: Obtain text input, which includes basic input and context input corresponding to the basic input, wherein the basic input includes at least candidate content items; Generate an embedding vector sequence corresponding to the basic input and an embedding vector sequence corresponding to the context input; A pooled embedding vector corresponding to the basic input is generated by performing a pooling operation on the sequence of embedding vectors corresponding to the basic input. A representative embedding vector sequence corresponding to the context input is obtained by performing a summarization operation on the embedding vector sequence corresponding to the context input using the pooled embedding vector; A text input representation of the text input is generated based at least on the embedding vector sequence corresponding to the basic input and the representative embedding vector sequence corresponding to the context input; as well as Based on the text input representation, predict the click probability of the candidate content item being clicked. The click probability is predicted by a click probability prediction model. The training of the click probability prediction model includes a first stage training and a second stage training. In the first stage training, the number of converters in each converter layer of the click probability prediction model is consistent with the number of original embedding vectors. In the second stage training, the number of converters in each converter layer of the click probability prediction model is consistent with the number of embedding vectors after the summarization operation is performed.

Images