Method and apparatus for content recommendation, and device, medium and program product

Abstract

According to the embodiments of the present disclosure, provided are a method and apparatus for content recommendation, and a device, a medium and a program product. The method comprises: using a plurality of content filtering strategies to determine a plurality of groups of candidate recommended content from a content library, wherein the plurality of content filtering strategies are respectively based on different content ranking criteria; using a trained machine learning model to determine a recommendation score of each piece of candidate recommended content in the plurality of groups of candidate recommended content with respect to a target user on the basis of user reference information of the target user and content reference information of the plurality of groups of candidate recommended content; and on the basis of the recommendation score corresponding to each piece of candidate recommended content in the plurality of groups of candidate recommended content, determining a group of target recommended content from among the plurality of groups of candidate recommended content, so as to provide the group of target recommended content to the target user. Thus, the accuracy and efficiency of content recommendation can be improved.

Description

Methods, apparatus, devices, media, and program products for content recommendation Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for content recommendation. Background Technology

[0002] The internet provides access to a wide variety of resources. For example, it allows access to various applications, products, audio and video content, and more. Furthermore, content delivery and service promotion via the internet have become a widely used new form of information dissemination. Content recommendation systems support displaying recommended content or services to users, enabling them to browse and obtain relevant services as needed. How to provide users with more predictable and higher-quality recommended content is a question that has been continuously researched in recommendation scenarios.

[0003] Summary of the Invention

[0004] In a first aspect of this disclosure, a method for content recommendation is provided. The method includes: determining multiple sets of candidate recommended content from a content library using multiple content filtering strategies, each based on different content ranking criteria; determining a recommendation score for each candidate recommended content relative to the target user using a trained machine learning model, based on user reference information of the target user and content reference information of the multiple sets of candidate recommended content; and determining a set of target recommended content from the multiple sets of candidate recommended content based on the recommendation scores corresponding to each candidate recommended content, for provision to the target user.

[0005] In a second aspect of this disclosure, an apparatus for content recommendation is provided. The apparatus includes: a candidate content determination module configured to determine multiple sets of candidate recommended content from a content library using multiple content filtering strategies, each content filtering strategy being based on different content ranking criteria; a recommendation score determination module configured to determine a recommendation score for each candidate recommended content relative to the target user using a trained machine learning model, based on user reference information of a target user and content reference information of the multiple sets of candidate recommended content; and a target content determination module configured to determine a set of target recommended content from the multiple sets of candidate recommended content based on the recommendation scores corresponding to each candidate recommended content in the multiple sets of candidate recommended content, for provision to the target user.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores a computer program that, when executed by a processor, implements the method of the first aspect.

[0008] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0009] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 2 illustrates a schematic diagram of an architecture for content recommendation according to some embodiments of the present disclosure;

[0013] Figure 3 illustrates an example of a filtering unit utilizing a first content filtering strategy according to some embodiments of the present disclosure;

[0014] Figure 4 illustrates an example of a filtering unit utilizing a second content filtering strategy according to some embodiments of the present disclosure;

[0015] Figure 5 illustrates an example of a filtering unit utilizing a third content filtering strategy according to some embodiments of the present disclosure;

[0016] Figure 6 shows an example of training data for a user ranking model according to some embodiments of the present disclosure;

[0017] Figure 7 illustrates an example of using a trained machine learning model to determine the recommendation score of each candidate recommendation content relative to the target user, according to some embodiments of the present disclosure;

[0018] Figure 8 shows a flowchart of a method for content recommendation according to some embodiments of the present disclosure;

[0019] Figure 9 shows a block diagram of an apparatus for content recommendation according to some embodiments of the present disclosure; and

[0020] Figure 10 shows a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. Detailed Implementation

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0025] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0026] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0028] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0029] A neural network is a machine learning network based on deep learning. A neural network can process inputs and provide corresponding outputs. It typically consists of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the inputs to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the layer above.

[0030] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as an input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0031] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. One or more content providers may use a recommendation management system 150 to manage content to be delivered on a content delivery platform 110. One or more client devices 130-1, 130-2, 130-3, etc. (collectively or individually referred to as client device 130 for ease of discussion) are associated with the content delivery platform 110 and may access various types of content provided on the content delivery platform 110, for example, based on corresponding users 132-1, 132-2, 132-3, etc. (collectively or individually referred to as user 132 for ease of discussion). As an example, the content delivery platform 110 may be an application, a website, a webpage, or other accessible platform. Client devices 130 may have applications installed for accessing the content delivery platform 110, or may access the content delivery platform 110 in a suitable manner.

[0032] The content delivery platform 110 can be configured to deliver one or more specific recommended content related to one or more objects to a user group based on corresponding strategies (e.g., provided or presented on client device 130). The recommended content to be delivered may include one or more recommended content 122-1, 122-2, ... 122-M from the content library 120 (also referred to as content database, database, recommended content library, etc.) (for ease of discussion, they are collectively referred to or individually as recommended content 122, and recommended content can be simply referred to as recommended content).

[0033] Examples of recommendable objects may include, but are not limited to: applications, physical goods / services, virtual goods / services, digital content / physical content, etc. In this article, "recommended content" refers to content associated with the recommended object, which can be presented to the corresponding user group to achieve the purpose of recommending the object. Recommended content is sometimes also called object-related material content, and examples can include advertisements, including videos, images, graphic works, plain text content, etc. Recommended content can include content uploaded by the requesting party from a data source specified by the requesting party, or it can be user-generated content (UGC) (note that the use of user-generated content is authorized by the user), etc.

[0034] In this document, a user group may include one or more user members, such as user 132. User members can be any potential consumer of the service, such as users, groups, organizations, entities, etc. In some embodiments, the content delivery platform 110 may distribute corresponding recommended content 122 to user 130 based on requests from various recommendation requesters 152-1, 152-2, 152-3, etc. (collectively or individually referred to as "recommendation requester" 152).

[0035] In some embodiments, the service provider may also provide certain cost expenditures to the content delivery platform 110 based on the presentation of the recommended content and subsequent conversions. The conversion results of the recommended content may include viewing, clicking, downloading, paying for, adding to the shopping cart, etc., and the specific conversion behavior is related to the recommended object and the service provider.

[0036] In environment 100, client device 130 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, client device 130 may also support any type of user-facing interface (such as "wearable" circuitry).

[0037] In environment 100, the content delivery platform 110 and / or recommendation management system 150 may be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and so on. Although shown separately, one or more of the content delivery platform 110 and / or recommendation management system 150 may be combined.

[0038] It should be understood that the components and arrangements shown in Figure 1 are merely examples, and a computing system suitable for implementing the exemplary embodiments described in this disclosure may include one or more different components, other components, and / or different arrangements. The number of elements shown in Figure 1 is also merely an example, and there may actually be more or fewer elements.

[0039] Traditionally, a set of candidate recommendations is retrieved from a content library, coarsely ranked, and then a set of target recommendations to be presented to the user is determined based on the ranking results. However, when the number of recommendations in the content library is small, traditional methods of simple retrieval, coarse ranking, and content selection result in poor performance. Traditionally, cold-start algorithms such as multiple deployments, collaborative filtering, data modeling, and expert strategies can also be used to achieve content recommendation when the number of recommendations in the content library is small. However, multiple deployments place high demands on system capabilities; the small number of recommendations in the content library leads to limited recall results from collaborative filtering, failing to guarantee diversity; data modeling requires a large amount of data, and a small number of recommendations results in poor data modeling quality; and expert strategies rely on human experience, incurring high labor costs.

[0040] In view of this, according to embodiments of the present disclosure, an improved content recommendation scheme is provided. According to this scheme, multiple content filtering strategies are used to determine multiple sets of candidate recommended content from a content library, each content filtering strategy being based on different content ranking criteria. Using a trained machine learning model, based on user reference information of the target user and content reference information of the multiple sets of candidate recommended content, a recommendation score is determined relative to the target user for each candidate recommended content in the multiple sets of candidate recommended content. Based on the recommendation scores corresponding to each candidate recommended content in the multiple sets of candidate recommended content, a set of target recommended content is determined from the multiple sets of candidate recommended content for provision to the target user.

[0041] Therefore, before performing content sorting and selection for a specific user, various content sorting mechanisms can be used to select from multiple recommended content in the content library according to corresponding content filtering strategies. Then, a model is used to select candidate recommended content for the specific user. This not only improves the accuracy and efficiency of content recommendation, and is not limited by the number and type of recommended content included in the content library, but also improves the efficiency of performing content sorting and filtering for a specific user.

[0042] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0043] Figure 2 illustrates a schematic diagram of an architecture 200 for content recommendation according to some embodiments of the present disclosure. For ease of description, the architecture 200 is described using an example implemented at a content delivery platform 110. The architecture 200 will be described with reference to the environment 100 of Figure 1. The architecture 200 involves at least a plurality of filtering units (e.g., filtering unit 222, filtering unit 224, and filtering unit 226) and a machine learning model 250. It will be understood that although only three filtering units are shown in the figure, any number of filtering units may actually be included.

[0044] Multiple filtering units can utilize multiple content filtering strategies to determine multiple sets of candidate recommended content from the content library 120. In some embodiments, to improve the accuracy of determining multiple sets of candidate recommended content, the architecture 200 may also involve a filtering unit 210. The filtering unit 210 can filter the content library 120 based on a predetermined filtering strategy. The predetermined filtering strategy may, for example, instruct the filtering out of recommended content from a specific data source, the filtering out of recommended content containing specific content, the filtering out of recommended content created by a specific creator, and so on. For example, if the filtering strategy instructs the filtering out of recommended content containing food-related content, then the filtering unit 210 can use this filtering strategy to filter out recommended content containing food-related content from the content library 120, and the filtered content library 120 will not include food-related recommended content. After completing the initial filtering, the multiple filtering units can determine multiple sets of candidate recommended content from the filtered content library 120 based on the corresponding content filtering strategy.

[0045] Different filtering units can utilize different content filtering strategies. For example, filtering unit 222 can use a first content filtering strategy (e.g., a numerical filtering strategy, also known as a numerical recall strategy) to determine a first group of candidate recommended content 232 from the content library 120. Filtering unit 224 can use a second content filtering strategy (e.g., a high-explosive filtering strategy, also known as a high-explosive recall strategy) to determine a second group of candidate recommended content 234 from the content library 120. Filtering unit 226 can use a third content filtering strategy (e.g., a similarity filtering strategy, also known as a similarity recall strategy) to determine a third group of candidate recommended content 236 from the content library 120.

[0046] Different content filtering strategies can be based on different content ranking criteria. For each filtering unit, it can rank the recommended content in the content library 120 according to the content ranking criteria corresponding to the content filtering strategy, and select a set of candidate recommended content from the content library 120 based on the ranking results. It can be understood that different filtering units can rank the recommended content in the content library 120 based on different content ranking criteria. For example, each filtering unit can determine a predetermined number of top-ranked recommended content as a set of candidate recommended content determined by that filtering unit based on the corresponding ranking results. By selecting (also called recalling) a portion of the recommended content that is determined to be of higher quality under the corresponding criteria from the content library according to different content ranking criteria, it is ensured that recommended content that is more beneficial to be provided to each user is selected.

[0047] After multiple filtering units identify multiple sets of candidate recommended content, the content delivery platform 110 can utilize a trained machine learning model 250 (sometimes also called a recall and ranking model) to determine the recommendation score 260 of each candidate recommended content relative to the target user (e.g., any one or more users from user 132). The machine learning model 250 can be deployed on the content delivery platform 110 or on other devices. The machine learning model 250 can be based on any suitable model architecture, including but not limited to Transformer models, convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep neural networks (DNNs), etc. In some embodiments, the machine learning model 250 can be based on a language model. Language models, by learning from a large corpus, are capable of natural language processing (including but not limited to semantic analysis capabilities, question answering capabilities, etc.). The machine learning model 250 can also be based on other suitable models. It should be noted that the machine learning model 250 can include one or more machine learning models, and this disclosure is not limited thereto.

[0048] In some embodiments, to reduce the workload of the model and improve its efficiency, the architecture 200 may also include a merging unit 240. The merging unit 240 can be deployed among multiple filtering units and the machine learning model 250. After multiple sets of candidate recommendation content are determined, they can be directly provided to the merging unit 240. The merging unit 240 can perform operations such as merging and deduplication on the multiple sets of candidate recommendation content to obtain a merged candidate content set. The merging unit 240 can provide the merged candidate content set to the machine learning model 250 to determine the recommendation score 260 of each candidate recommendation content relative to the target user.

[0049] Regarding the specific method of determining the recommendation score 260 using the machine learning model 250, in some embodiments, the content delivery platform 110 can obtain user reference information of the target user and content reference information of multiple sets of candidate recommended content. User reference information may include user behavior data over a past period, interaction data with recommended content or other types of content, various available user attribute information, and other reference information deemed suitable for determining appropriate recommended content. Content reference information of the recommended content may include the creator, source, and specific content (e.g., text, visual data, audio data, etc.) of the candidate recommended content, user interaction data on the candidate recommended content, and other reference information deemed suitable for determining whether it is suitable to be recommended to the user. Note that embodiments of this disclosure do not limit the specific type and form of the user reference information and content reference information to be used by the machine learning model 250. Depending on the specific application scenario and requirements, the information that can be selected for reference is very diverse, and different machine learning models can be configured to reference various types of information.

[0050] It is understandable that if architecture 200 involves merging unit 240, content delivery platform 110 can obtain user reference information of the target user and content reference information of each candidate recommended content in the merged candidate content set. For ease of description, the following example uses multiple sets of candidate recommended content for illustration.

[0051] Content delivery platform 110 can, for example, generate prompt word input for machine learning model 250 based on user reference information of the target user and content reference information of multiple sets of candidate recommended content. Content delivery platform 110 can provide this prompt word input to machine learning model 250 to instruct machine learning model 250 to determine the recommendation score 260 of each candidate recommended content relative to the target user based on the user reference information of the target user and the content reference information of multiple sets of candidate recommended content.

[0052] In some embodiments, a recommendation score can be defined as the value of one or more recommendation metrics of interest in content recommendation. For example, a recommendation score can be based on the probability that a target user performs a specific conversion action (e.g., click, favorite, comment, purchase, etc.) on the recommended content after it has been provided to that user. A higher probability of this conversion action is determined to result in a higher recommendation score. Alternatively or additionally, the recommendation score can also be based on metrics of interest to the content delivery platform, such as the target user's retention time on the content delivery platform after the recommended content has been provided, the number of new users the content recommendation platform can acquire, and so on. The embodiments of this disclosure do not impose specific limitations on the specific method of measuring the recommendation score. In general, a recommendation score can be used to measure the importance or effectiveness of recommended content for a particular user.

[0053] The content delivery platform 110 can then determine a set of target recommended content 270 from the multiple sets of candidate recommended content based on the recommendation scores 260 corresponding to each candidate recommended content, for delivery to the target user. The content delivery platform 110 can use any appropriate method to determine the set of target recommended content 270.

[0054] In some embodiments, the content delivery platform 110 can obtain a threshold score and compare the recommendation score 260 corresponding to each candidate recommendation content in multiple groups of candidate recommendation content with the threshold score. The content delivery platform 110 can determine a group of recommendation content whose corresponding recommendation score 260 reaches the threshold score as a group of target recommendation content 270.

[0055] Alternatively or additionally, in some embodiments, the content delivery platform 110 may also sort all candidate recommended content in multiple groups of candidate recommended content in descending order based on the recommendation score 260 corresponding to each candidate recommended content (that is, the higher the recommendation score 260 of the candidate recommended content ranked higher). It can be understood that multiple groups of candidate recommended content correspond to one ranking result. For example, the content delivery platform 110 may determine a predetermined number of recommended content items that rank higher in the ranking result as a group of target recommended content 270.

[0056] It is important to note that a set of target recommended content 270 is a collection of recommended content determined from all candidate recommended content included in multiple sets of candidate recommended content. For example, it may include 3 sets of candidate recommended content, each set of candidate recommended content may include 100 recommended content items, and the final set of target recommended content may include 50 recommended content items, of which 20 are from the first set of candidate recommended content items, 15 are from the second set of candidate recommended content items, and 15 are from the third set of candidate recommended content items.

[0057] It is understandable that if the architecture 200 involves a merging unit 240, after the machine learning model 250 determines the recommendation scores 260 of each of the merged candidate content sets output by the merging unit 240, the content delivery platform 110 can directly determine a set of target recommended content 270 from the merged candidate content set based on the recommendation scores 260. The number of recommended content included in a set of target recommended content 270 can be less than or equal to the number of recommended content included in the merged candidate content set.

[0058] Regarding the specific methods for determining multiple sets of candidate recommended content from the content library 120 using multiple content filtering strategies, in some embodiments, for the filtering unit 222 using the first content filtering strategy, the filtering unit 222 can determine at least one feature related to a predetermined recommendation metric. The predetermined recommendation metric is a core business indicator used to measure the effectiveness of recommendations, including but not limited to user retention time on the platform, average daily new users, cost required to acquire new users, conversion rate of specific conversion behaviors, etc. At least one feature related to the predetermined recommendation metric may include, for example, features positively correlated with the core business metric, including but not limited to the number of user likes, the number of favorites, and the number of author followers.

[0059] For each of the at least one features, the filtering unit 222 can sort the recommended content in the content library 120 based on the feature values ​​of each recommended content in the content library 120 for that feature (e.g., in descending order). Each recommended content has corresponding feature values ​​for multiple features. For each recommended content, the filtering unit 222 can directly obtain the feature value of that recommended content for that feature. The filtering unit 222 can select a first group of candidate recommended content from the content library 120 based on the sorting results for at least one feature. For example, the filtering unit 222 can determine a predetermined number of candidate contents that rank highly (e.g., the top 100 candidate contents in the sorting) as the first group of candidate recommended content 232. Alternatively, the filtering unit 222 can also determine a number of candidate contents whose corresponding feature values ​​reach a threshold as the first group of candidate recommended content 232.

[0060] Figure 3 illustrates an example of a filtering unit 222 utilizing a first content filtering strategy according to some embodiments of the present disclosure. The filtering unit 222 may include, for example, a merging unit 320, a sorting unit 330, and a selection unit 340. The filtering unit 222 may obtain feature data 310 corresponding to each recommended content in the content library 120. For each recommended content, the feature data 310 may indicate the feature value of that recommended content on various features.

[0061] In some embodiments, a portion of the recommended content in the content library 120 is provided to users in advance (it can be understood that this user can include multiple users, that is, this user can refer to a user group). The filtering unit 222 can obtain the recommendation effect data 301 of this portion of recommended content. The recommendation effect data 301 may indicate, for example, the platform on which the recommended content is delivered, the user feedback on the recommended content after it is delivered (e.g., number of likes, average viewing time, number of favorites, number of shares, number of comments, comment content, etc.), the conversion data corresponding to the recommended content (e.g., how many users are converted based on the recommended content), and so on.

[0062] For each piece of recommended content provided to the user (also referred to as each piece of recommended content delivered), the filtering unit 222 can determine the tag data 302 corresponding to the recommended content based on the recommendation effect data 301. The tag data 302 corresponding to each piece of recommended content can indicate the recommendation effect of the recommended content. Recommended content that has been delivered and has corresponding tag data 302 can be referred to as labeled recommended content. For each labeled recommended content, the merging unit 320 can merge the feature data 310 and the corresponding tag data 302. The merged result (i.e., tag data 302 + feature data 310) will be provided to the sorting unit 330 along with the pre-acquired predetermined recommendation index 325.

[0063] In some embodiments, the filtering unit 222 can also determine at least one feature that has a significant impact on the recommendation effect (i.e., at least one feature related to the predetermined recommendation indicator) based on the merging result corresponding to each labeled recommended content and the predetermined recommendation indicator 325. For example, if there are a total of 10 features, and 3 of them have a significant impact on the recommendation effect of the labeled recommended content (i.e., changes in the feature values ​​of these 3 features will more significantly affect the recommendation indicator value corresponding to the recommended content), these 3 features can be determined as features related to the predetermined recommendation indicator. The filtering unit 222 can, for example, provide only the merging result of the feature data and label data of each labeled recommended content on this at least one feature to the sorting unit 330.

[0064] In some embodiments, if at least one feature related to a predetermined recommendation metric includes multiple features, the sorting unit 330 can divide (332) the 120 recommended contents in the content library into multiple content sub-libraries corresponding to the multiple division dimensions according to multiple predetermined division dimensions. For example, the 120 recommended contents in the content library can be divided into multiple content sub-libraries corresponding to the multiple division dimensions according to multiple division dimensions such as source, content, and creator, with each content sub-library corresponding to one division dimension. The reason for dividing the content library by dimension is that the recommended content that can be recommended under different dimensions is different, and the factors affecting the recommendation effect of the recommended content corresponding to different dimensions are also different.

[0065] For a given partition dimension (which can be any of the multiple partition dimensions), the sorting unit 330 can determine the index value of the labeled recommended content that is partitioned into the given partition dimension and is related to that feature under a predetermined recommendation index for each of the multiple features. Specifically, the sorting unit 330 can determine multiple feature values ​​of the multiple labeled recommended content on the multiple features. For each feature, the sorting unit 330 can sort the multiple labeled recommended content according to the feature values ​​of the multiple labeled recommended content on that feature (334), and this sorting can be, for example, descending order. The sorting unit 330 can determine the labeled recommended content related to that feature based on the sorting result. For example, the sorting unit 330 can determine the top predetermined number of labeled recommended content in the sorting result as the labeled recommended content related to that feature.

[0066] For each feature, the sorting unit 330 can determine the index value of the labeled recommended content related to that feature under a predetermined recommendation metric based on the feature value of that feature. Taking a total of 1000 labeled recommended content items divided into partitioning dimension A as an example, for feature A among multiple features, the sorting unit 330 can determine the feature value of each of the 1000 labeled recommended content items on feature A, and sort the 1000 labeled recommended content items in descending order based on their respective feature values. For example, the sorting unit 330 can determine the 100 labeled recommended content items that rank in the top 100 according to the sorting result as the 100 labeled recommended content items related to feature A for partitioning dimension A. The sorting unit 330 can determine the average feature value of these 100 labeled recommended content items under feature A, and determine this average value as the index value of the labeled recommended content items divided into a given partitioning dimension and related to feature A under a predetermined recommendation metric (also referred to as the index value determined for feature A under a given partitioning dimension).

[0067] The sorting unit 330 can determine, under a given partitioning dimension, the index values ​​of the labeled recommended content related to multiple features and classified to the given partitioning dimension under a predetermined recommendation index (which can be referred to as multiple index values ​​determined for multiple features respectively). For each of the multiple features, the sorting unit 330 can also determine the difference between the index value determined for that feature and the reference index value (which may be, for example, the overall index value corresponding to the core business index) (which may be, for example, the difference between the index value and the reference index value). The sorting unit 330 can determine (336) the differences between each of the multiple index values ​​and the reference index value in a similar manner.

[0068] The sorting unit 330 can, for example, select a reference feature for a given partitioning dimension from multiple features based on the differences between the corresponding index values ​​(or the differences between the corresponding features). The sorting unit 330 can, for example, determine the feature with the largest (or smallest, which can be determined according to settings) difference as the reference feature (i.e., the optimal feature) for the given partitioning dimension. For example, if multiple features include feature A, feature B, and feature C, the sorting unit 330 can determine the differences between the index values ​​corresponding to each of these three features and the reference index value under a given partitioning dimension (i.e., determine the differences corresponding to each of these three features). If the index difference corresponding to feature A is greater than the index difference corresponding to feature B and greater than the index difference corresponding to feature C, then the sorting unit 330 can determine feature A as the reference feature for the given partitioning dimension. This reference feature is considered the preferred feature under a specific partitioning dimension. This preferred feature can significantly influence the recommendation index value of the recommended content under a given partitioning dimension. Of course, for a given partitioning index, more than one reference feature can be selected.

[0069] The sorting unit 330 can determine the reference features corresponding to each of the multiple partitioning dimensions in a similar manner. It is understood that different partitioning dimensions can have their own reference features. For each of the multiple partitioning dimensions, the sorting unit 330 can sort the recommended content in the content sub-library corresponding to that partitioning dimension (i.e., the multiple recommended contents partitioned to that partitioning dimension, including recommended contents that do not have corresponding tag data, i.e., have not been delivered) based on the feature values ​​(or weighted feature values ​​of multiple reference features) of the recommended content on the corresponding reference features, thus obtaining the sorting result for the multiple partitioning dimensions. The sorting here can also be, for example, a descending order.

[0070] Therefore, we can first determine the reference features corresponding to each partitioning dimension based on a small amount of labeled recommended content with tag data. For each partitioning dimension, we sort all recommended content in content library 120 according to the feature values ​​of the reference features (e.g., descending order) to obtain the ranking results for each partitioning dimension.

[0071] For each partitioning dimension, selection unit 340 can determine a subset of candidate recommended content corresponding to that partitioning dimension based on the ranking results. For example, selection unit 340 can determine a predetermined number of recommended contents from the top of the ranking results as a subset of candidate recommended content corresponding to that partitioning dimension. Selection unit 340 can then determine a first set of candidate recommended content 232 based on multiple subsets of candidate recommended content corresponding to multiple partitioning dimensions. For example, for each partitioning dimension, selection unit 340 can determine the top 10,000 recommended contents from the ranking results as a subset of candidate recommended content corresponding to that partitioning dimension. If there are three partitioning dimensions, selection unit 340 can determine three subsets of candidate recommended content, each containing 10,000 recommended contents. Selection unit 340 can merge these three subsets of candidate recommended content and remove duplicates from them to obtain the first set of candidate recommended content 232.

[0072] In some embodiments, for a filtering unit 224 utilizing a second content filtering strategy, the filtering unit 224 may determine a set of features associated with a reference user. A "reference user" (i.e., a high-value user) can be determined in various appropriate ways. In some embodiments, after providing recommended content to a user, the user's metric values ​​on a relevant metric are determined. For example, if the focus is on user retention time within the application, retention time can be used as a metric. If it is determined that the retention time of one or more users meets expectations after being provided with recommended content (e.g., retention time exceeds a threshold), then this or these users can be identified as reference users. Further, features associated with the reference user can be selected from multiple features based on the feature values ​​of the recommended content already provided to this or these reference users on various features. Features associated with the reference user are considered to be features that can contribute to the relevant metric (e.g., retention time).

[0073] The filtering unit 224 can sort the recommended content in the content library 120 based on the feature values ​​of each recommended content in a set of features, and select a second group of candidate recommended content from the content library 120 based on the sorting result. Similarly, the sorting here can be in descending order, and the filtering unit 224 can determine a predetermined number of recommended content items that rank highly as the second group of candidate recommended content.

[0074] Figure 4 illustrates an example of a filtering unit 224 utilizing a second content filtering strategy according to some embodiments of the present disclosure. The filtering unit 224 may include, for example, a sorting unit 430 and a selection unit 440. The filtering unit 224 may obtain relevant information about a reference user 410, such as indicating a set of features associated with the reference user 410. The filtering unit 224 may also obtain feature data 420 corresponding to each recommended content in the content library 120. For each recommended content, the feature data 420 may indicate the feature value of the recommended content on each feature. Alternatively or additionally, the set of features associated with the reference user 410 may also be determined by the filtering unit 224 based on the feature data 420.

[0075] The sorting unit 430 can divide (432) the 120 recommended contents in the content library into multiple content sub-libraries corresponding to the multiple division dimensions according to multiple predetermined division dimensions (the recommended contents included in each content sub-library are also the recommended contents divided into a given division dimension). For example, the 120 recommended contents in the content library can be divided into multiple content sub-libraries corresponding to the multiple division dimensions according to multiple division dimensions such as source, content, and creator, with each content sub-library corresponding to one division dimension. In some embodiments, the filtering unit 224 can also determine multiple reference users corresponding to the multiple division dimensions and determine multiple sets of features associated with the multiple reference users for each division dimension. That is, different division dimensions can correspond to different reference users, and different reference users can be associated with different features.

[0076] For a given partition dimension among multiple partition dimensions, the sorting unit 430 can sort the recommended content in the content sub-library corresponding to the given partition dimension based on the feature values ​​of the content sub-library on the corresponding set of features (434). For each partition dimension, the recommended content in the content sub-library can be sorted in descending order based on the feature values ​​of each recommended content in the corresponding content sub-library on the set of features. Specifically, if the set of features determined by the filtering unit 224 for a given partition dimension may include multiple features, the sorting unit 430 can determine the weights corresponding to the multiple features respectively. For each recommended content in the content sub-library of the given partition dimension, the feature value of the recommended content on the set of features can be determined based on the feature values ​​of the recommended content on the multiple features and the weights corresponding to the multiple features respectively.

[0077] Taking a set of features including features A, B, and C as an example, the weights of these three features can be represented as a, b, and c, respectively. If the feature values ​​of recommended content A on these three features are x, y, and z, then the feature value of recommended content A on this set of features is ax + by + cz. The sorting unit 430 can similarly determine the feature value of each recommended content in the content sub-library corresponding to a given partitioning dimension on its corresponding set of features, and determine the feature values ​​of the content sub-libraries corresponding to each partitioning dimension on their respective set of features.

[0078] Similarly, for each partitioning dimension, selection unit 440 can determine a subset of candidate recommended content corresponding to that partitioning dimension based on the ranking results. For example, selection unit 440 can determine a predetermined number of recommended contents that rank highly in the ranking results as a subset of candidate recommended content corresponding to that partitioning dimension. Selection unit 440 can then determine a second set of candidate recommended content 234 based on multiple subsets of candidate recommended content corresponding to multiple partitioning dimensions.

[0079] In some embodiments, for the filtering unit 226 utilizing the third content filtering strategy, the filtering unit 226 can determine multiple reference recommended content (also referred to as seed recommended content). As mentioned above, some recommended content in the content library 120 may be provided to the user in advance. The filtering unit can determine some recommended content with higher recommendation performance based on this part of the recommended content and determine these recommended content as reference recommended content. The filtering unit 226 can also obtain some reference recommended content manually entered by relevant professionals. That is, the reference recommended content can be determined by the filtering unit 226 itself or manually entered.

[0080] For each of the multiple reference recommended content items, the filtering unit 226 can sort the recommended content items in the content library 120 based on the similarity between each recommended content item in the content library 120 and that reference recommended content item (e.g., sorting in descending order based on similarity). The filtering unit 226 can then select a third group of candidate recommended content items from the content library 120 based on the sorting results for the multiple reference recommended content items. Similarly, the sorting here can be in descending order, and the filtering unit 226 can determine a predetermined number of the top-ranked recommended content items as the third group of candidate recommended content items.

[0081] Figure 5 illustrates an example of a filtering unit 226 utilizing a third content filtering strategy according to some embodiments of the present disclosure. The filtering unit 226 may include, for example, a sorting unit 550 and a first selection unit 560. The filtering unit 226 can determine the vectors (e.g., multimodal vector 512 and recommendation system vector 514) corresponding to each recommended content in the content library 120. The filtering unit 226 can also acquire feature data 516 of the recommended content. For example, the filtering unit 226 can determine the multimodal vector 512 of the recommended content based on its specific content (e.g., the specific content it includes, such as images, videos, text, audio, etc.). For example, the filtering unit 226 can determine the recommendation system vector 514 of the recommended content based on its specific content and the user behavior corresponding to the recommended content. The filtering unit 226 can store the vectors in corresponding vector libraries. It is understood that different types of vectors are stored in different vector libraries. For example, the multimodal vector 512 is stored in vector library 522 (e.g., a database, a data table, etc.), and the recommendation system vector 514 is stored in vector library 524 (e.g., a search engine).

[0082] In some embodiments, the filtering unit 226 may also determine the tag data 535 of the recommended content that has already been delivered, based on the recommendation effect data 530 of the recommended content that has already been delivered in the content library 120. For example, the filtering unit 226 may determine a set of reference recommended content 540 based on the tag data 535 of the recommended content that has already been delivered. Alternatively or additionally, the filtering unit 226 may also obtain a set of reference recommended content 540 that has been manually entered. The filtering unit 226 may provide all the determined or obtained reference recommended content 540 to the sorting unit 550.

[0083] Regarding the specific method for determining the similarity between each recommended content and the reference recommended content 540, in some embodiments, the sorting unit 550 may determine a reference vector for each reference recommended content 540, and obtain the vector of each recommended content from a vector library. The sorting unit 550 may determine (552) the similarity between the reference vector and the vector, and define this similarity as the similarity between the reference recommended content and the recommended content. The sorting unit 550 may determine the similarity between vectors in any suitable manner; for example, the sorting unit 550 may determine the similarity between vectors by calculating the cosine similarity between vectors. It is understood that the filtering unit 226 may also determine the similarity between the reference recommended content and the recommended content in any other suitable manner, and this disclosure does not limit the specific method.

[0084] For example, referring to FIG5, the sorting unit 550 can obtain the multimodal vector 512 corresponding to each recommended content from the vector library 522. The sorting unit 550 can, for example, determine the reference multimodal vector of each reference recommended content 540 based on its specific content. For each reference recommended content 540, the sorting unit 550 can determine the similarity between the reference multimodal vector of that reference recommended content 540 and the multimodal vector 512 of each recommended content. The sorting unit 550 can sort the multiple recommended content based on the similarity corresponding to each recommended content (554), for example, in descending order.

[0085] The selection unit 560 can determine multiple recommended contents that meet the similarity requirements based on the ranking results (e.g., a predetermined number of recommended contents at the top of the ranking results, or multiple recommended contents with similarity greater than a threshold), and determine a set of candidate recommended contents 570 based on these multiple recommended contents (that is, a set of candidate recommended contents 570 may include these multiple recommended contents).

[0086] Alternatively or additionally, the sorting unit 550 can also directly read a set of recommendation system vectors 514 from the vector library 524 that have a similarity to the reference recommendation content 540 that reaches a threshold (or a predetermined number of similarities). The sorting unit 550 can determine the set of recommendation content corresponding to this set of recommendation system vectors 514 as a set of recommendation content that has a similarity to the reference recommendation content 540 that reaches the threshold (or a predetermined number of similarities). A set of candidate recommendation content 570 may also include this set of recommendation content.

[0087] In some embodiments, a set of candidate recommended content 570 can be directly determined as the third set of candidate recommended content 236. Alternatively or additionally, in some embodiments, the filtering unit 226 may further include a second selection unit 580. The second selection unit 580 may combine the first content filtering strategy to perform secondary filtering on the set of candidate recommended content 570. In this case, referring to FIG3, the set of candidate recommended content 570 can be regarded as the content library 120 in FIG3. The set of candidate recommended content 570 includes previously delivered recommended content. The second selection unit 580 can determine the feature data and tag data corresponding to the set of candidate recommended content 570 from the feature data 516 and the tag data 535. The feature data corresponding to the set of candidate recommended content 570 may correspond to the feature data 310 in FIG3, and the tag data corresponding to the set of candidate recommended content 570 may correspond to the tag data 302 in FIG3. The second selection unit 580 can select a set of candidate recommendation content 590 from a set of candidate recommendation content 570 based on the feature data and tag data corresponding to a set of candidate recommendation content 570 and the predetermined recommendation indicators. This set of candidate recommendation content 590 is also the third set of candidate recommendation content 236.

[0088] Regarding the training method of the machine learning model, the machine learning model 250 can be trained on the device corresponding to the content delivery platform 110, or it can be trained on other devices and then provided to the content delivery platform 110. For ease of description, the electronic device used to train the machine learning model 250 can be referred to as the model training system. The model training system can use a training sample set to train the machine learning model. The training sample set contains reference information for multiple sample recommendation content-sample user pairs, as well as multiple labels corresponding to multiple sample recommendation contents. Each label indicates the labeled recommendation score of the corresponding sample recommendation content relative to the corresponding sample user. The labeled recommendation score can be, for example, a value in the range [0,1]. The closer the labeled recommendation score is to 1, the more likely the corresponding sample recommendation content should be recommended to the sample user.

[0089] The model training system can provide content reference information of the sample recommended content and user reference information of the sample users to an untrained machine learning model 250. The model output of the machine learning model 250 can indicate the estimated recommendation score of the sample recommended content relative to the sample users. The model training system can determine the difference between the estimated recommendation score and the labeled recommendation score for each sample recommended content. The training objective can be, for example, to make the difference between the estimated recommendation score and the labeled recommendation score for multiple sample recommended content less than a threshold (e.g., 0).

[0090] The training sample set can be directly obtained by the model training system or automatically generated by the model training system in an appropriate manner. In some embodiments, the model training system can obtain user behavior features, content features, and recommendation effects corresponding to multiple recommended contents that have been delivered (the multiple delivered recommended contents can be regarded as multiple sample recommended contents). Figure 6 shows an example 600 of training data for a user ranking model according to some embodiments of this disclosure. As shown in Figure 6, the model training system can obtain user behavior features after each sample recommended content is delivered and content features of each sample recommended content (these two features can be collectively referred to as features). The user behavior features of each sample recommended content can be obtained periodically.

[0091] As an example, the model training system can obtain the sum of user behavior features over a period of time (e.g., 7 days) after the sample recommended content is recommended to the user (i.e., the cumulative 7-day user behavior features in Figure 6), the sum of user behavior features over a longer period (e.g., 14 days) after the sample recommended content is recommended to the user (i.e., the cumulative 14-day user behavior features in Figure 6), the sum of user behavior features over an even longer period (e.g., 21 days) after the sample recommended content is recommended to the user (i.e., the cumulative 21-day user behavior features in the figure), and so on. The model training system can also obtain the cumulative index value of the sample recommended content for a given recommendation metric after it is recommended to a specific user (e.g., the cumulative index value after the sample recommended content is recommended). This cumulative index value reflects the recommendation effect of the sample recommended content; the higher the cumulative index value, the better the recommendation effect. The collected cumulative index values ​​for the sample recommended content and the user can be called labels. If the cumulative index value indicated by the label is higher, it means that the specific recommended content has a higher probability of being recommended to the corresponding user. Note that the specific method of feature data collection here is only an example and does not imply any limitations. In practical applications, other feature data can be set as needed.

[0092] Therefore, the model training system can acquire the feature data and tags of each sample recommendation content-sample user. The feature data of each sample recommendation-sample user can be regarded as content reference information of the sample recommendation content and user reference information of the sample user. The model training system can construct a sample set based on the acquired feature data and tags of multiple sample recommendation content-sample users. For example, the model training system can directly regard this sample set as the training sample set and use it to train the machine learning model 250. Alternatively or additionally, in some embodiments, the model training system can also divide the sample set into three non-overlapping parts, one part of which is determined as the training sample set, one part of which is determined as the validation sample set, and one part of which is determined as the test sample set. The model training system can sequentially train the machine learning model 250 with the training sample set, validate the machine learning model 250 with the validation sample set, and test the machine learning model 250 with the test sample set.

[0093] After training, validation, and testing, the trained machine learning model 250 can be used to determine the recommendation score of each candidate recommendation content relative to the target user based on the target user's user reference information and the content reference information of multiple sets of candidate recommendation content.

[0094] Content delivery platform 110 can acquire a trained machine learning model 250. The specific method of determining the recommendation score of each candidate content item relative to the target user using the trained machine learning model 250 is described below with reference to FIG7. FIG7 illustrates an example 700 of determining the recommendation score of each candidate content item relative to the target user using the trained machine learning model 250 according to some embodiments of the present disclosure. In example 700, the machine learning model 250 includes multiple machine learning models (which may be simply referred to as models). In some embodiments, multiple sets of candidate content items obtained using multiple content filtering strategies, or the merged set of candidate content items output by the merging unit 240, can be referred to as the candidate content set 710.

[0095] In some embodiments, the machine learning model 250 includes multiple first machine learning models 720 (which may include, for example, four first machine learning models, model 1 to model 4; this is merely an example, and in practice, any number of first machine learning models may be included). For each candidate recommended content in the candidate content set 710, the content delivery platform 110 can use the multiple first machine learning models 720 to determine the first intermediate recommendation score for each candidate recommended content, thereby obtaining multiple first intermediate recommendation scores 730 for each candidate recommended content (for example, the four first intermediate recommendation scores, score 1 to score 4, output by the four first machine learning models in the figure).

[0096] The content delivery platform 110 can determine the recommendation score of each candidate recommended content based on multiple first intermediate recommendation scores 730. In some embodiments, for each candidate recommended content, the content delivery platform 110 can determine a target score (e.g., score 5740 shown in the figure) corresponding to the multiple first intermediate recommendation scores 730. The target score can be the average score, maximum score, minimum score, etc., of the multiple first intermediate scores 730, and this disclosure does not limit it. For example, the content delivery platform 110 can determine this target score as the recommendation score corresponding to the candidate recommended content. The content delivery platform 110 can then determine the recommendation score of each candidate recommended content in the candidate content set 710 in this way.

[0097] In some embodiments, machine learning model 250 may further include a second machine learning model (the second machine learning model may also include at least one machine learning model; in the figure, only model 5750 is used as an example of the second machine learning model). Content delivery platform 110 may also utilize the second machine learning model to determine a second intermediate recommendation score (e.g., score 6760 shown in the figure) for each candidate recommendation content based on multiple first intermediate recommendation scores (or target scores of multiple first intermediate scores) and each candidate recommendation content in multiple sets of candidate recommendation content. Content delivery platform 110 may, for example, directly determine the second intermediate recommendation score of each candidate recommendation content as the recommendation score of each candidate recommendation content.

[0098] Alternatively or additionally, in some embodiments, the content delivery platform 110 may further include a calculation unit 770. The calculation unit 770 can determine the average score (e.g., score 7780 in the figure) of multiple first intermediate recommendation scores and second intermediate scores for each candidate recommended content. The content delivery platform 110 may, for example, determine the average score of each candidate recommended content as the recommendation score corresponding to each candidate recommended content.

[0099] In summary, according to the embodiments of this disclosure, multiple content filtering strategies can be used to select from multiple recommended content in the content library before sorting, and then only the selected candidate recommended content can be processed by the model. This can improve the accuracy and efficiency of content recommendation, is not limited by the number and type of recommended content included in the content library, and also reduces the efficiency of recommended content processing.

[0100] Figure 8 illustrates a flowchart of a method 800 for content recommendation according to some embodiments of the present disclosure. Method 800 may be implemented at a content delivery platform 110. Method 800 will be described with reference to the environment 100 of Figure 1.

[0101] In box 810, the content delivery platform 110 uses multiple content filtering strategies to determine multiple sets of candidate recommended content from the content library. Each of the multiple content filtering strategies is based on a different content ranking criterion.

[0102] In box 820, the content delivery platform 110 uses a trained machine learning model to determine the recommendation score of each candidate recommendation content relative to the target user, based on the user reference information of the target user and the content reference information of multiple sets of candidate recommendation content.

[0103] In box 830, the content delivery platform 110 determines a set of target recommended content from the multiple sets of candidate recommended content based on the recommendation scores corresponding to each candidate recommended content, in order to provide it to the target user.

[0104] In some embodiments, determining multiple sets of candidate recommended content from a content library using multiple content filtering strategies includes: for each of the multiple content filtering strategies, sorting the recommended content in the content library based on the content sorting criteria corresponding to that content filtering strategy; and selecting a set of candidate recommended content from the content library based on the sorting results.

[0105] In some embodiments, determining multiple sets of candidate recommended content from a content library using multiple content filtering strategies includes: for a first content filtering strategy among the multiple content filtering strategies, determining at least one feature related to a predetermined recommendation metric; for each of the at least one feature, ranking the recommended content in the content library based on the feature value of each recommended content in the content library for that feature; and selecting a first set of candidate recommended content from the content library based on the ranking result for the at least one feature.

[0106] In some embodiments, at least one feature includes multiple features. In some embodiments, sorting the recommended content in the content library based on the feature values ​​of the recommended content in the content library on each feature includes: dividing the recommended content in the content library into multiple content sub-libraries corresponding to the multiple division dimensions according to multiple division dimensions; for a given division dimension among the multiple division dimensions, for each of the multiple features, determining the index value of the labeled recommended content that is divided into the given division dimension and is related to the feature under a predetermined recommendation index, and selecting a reference feature for the given division dimension from the multiple features based on the difference between the index value determined for the multiple features and the reference index value; and for each division dimension among the multiple division dimensions, sorting the recommended content in the content sub-library corresponding to the division dimension based on the feature values ​​of the recommended content in the content sub-library corresponding to the division dimension on the corresponding reference feature, to obtain a sorting result for the multiple division dimensions.

[0107] In some embodiments, determining multiple sets of candidate recommended content from a content library using multiple content filtering strategies includes: for a second content filtering strategy among the multiple content filtering strategies, determining a set of features associated with a reference user; ranking the recommended content in the content library based on the feature values ​​of each recommended content in the content library on the set of features; and selecting a second set of candidate recommended content from the content library based on the ranking result.

[0108] In some embodiments, determining at least one feature associated with a reference user includes: determining multiple reference users corresponding to multiple partitioning dimensions of recommended content in the content library; and determining multiple sets of features associated with the multiple reference users; and wherein sorting the recommended content in the content library based on the feature values ​​of each recommended content in the content library on each feature of a set of features includes: dividing the recommended content in the content library into multiple content sub-libraries corresponding to multiple partitioning dimensions according to multiple partitioning dimensions; and for a given partitioning dimension among the multiple partitioning dimensions, sorting the recommended content in the content sub-library corresponding to the given partitioning dimension based on the feature values ​​of the content sub-library corresponding to the given partitioning dimension on the corresponding set of features.

[0109] In some embodiments, determining multiple sets of candidate recommended content from a content library using multiple content filtering strategies includes: determining multiple reference recommended content for a third content filtering strategy among the multiple content filtering strategies; for each reference recommended content, ranking the recommended content in the content library based on the similarity between each recommended content in the content library and the reference recommended content; and selecting a third set of candidate recommended content from the content library based on the ranking result of the multiple reference recommended content.

[0110] In some embodiments, the machine learning model includes multiple first machine learning models, and determining the recommendation score of each candidate recommendation content relative to the target user in multiple sets of candidate recommendation content includes: for each candidate recommendation content in multiple sets of candidate recommendation content, using multiple first machine learning models to determine the first intermediate recommendation score of each candidate recommendation content, thereby obtaining multiple first intermediate recommendation scores for each candidate recommendation content; and determining the recommendation score of each candidate recommendation content based on the multiple first intermediate recommendation scores.

[0111] In some embodiments, the machine learning model further includes a second machine learning model, and wherein determining the recommendation score of each candidate recommendation content based on a plurality of first intermediate recommendation scores includes: using the second machine learning model to determine a second intermediate recommendation score for each candidate recommendation content based on a plurality of first intermediate recommendation scores and each candidate recommendation content in a plurality of sets of candidate recommendation content; and determining the recommendation score of each candidate recommendation content based at least on the second intermediate recommendation scores.

[0112] In some embodiments, determining the recommendation score of each candidate content based at least on the second intermediate recommendation score includes: for each candidate content, determining the recommendation score of each candidate content based on the second intermediate recommendation score and multiple first intermediate recommendation scores.

[0113] Embodiments of this disclosure also provide corresponding apparatus for implementing the methods or processes described above. FIG9 shows an exemplary structural block diagram of an apparatus 900 for content recommendation according to some embodiments of this disclosure. The apparatus 900 may be implemented as or included in the content delivery platform 110. The various modules / components in the apparatus 900 may be implemented by hardware, software, firmware, or any combination thereof.

[0114] As shown in Figure 9, the device 900 includes a candidate content determination module 910, configured to determine multiple sets of candidate recommended content from a content library using multiple content filtering strategies, each based on different content ranking criteria. The device 900 also includes a recommendation score determination module 920, configured to use a trained machine learning model to determine the recommendation score of each candidate recommended content relative to the target user, based on the target user's user reference information and the content reference information of the multiple sets of candidate recommended content. The device 900 further includes a target content determination module 930, configured to determine a set of target recommended content from the multiple sets of candidate recommended content based on the recommendation scores corresponding to each candidate recommended content, for provision to the target user.

[0115] In some embodiments, the candidate content determination module 910 is further configured to: for each of the multiple content filtering strategies, sort the recommended content in the content library based on the content sorting criteria corresponding to the content filtering strategy; and select a set of candidate recommended content in the content library based on the sorting results.

[0116] In some embodiments, the candidate content determination module 910 is further configured to: for a first content filtering strategy among a plurality of content filtering strategies, determine at least one feature related to a predetermined recommendation metric; for each of the at least one feature, sort the recommended content in the content library based on the feature value of each recommended content in the content library on that feature; and select a first group of candidate recommended content from the content library based on the sorting result for the at least one feature.

[0117] In some embodiments, at least one feature includes multiple features, and the candidate content determination module 910 is further configured to: divide the recommended content in the content library into multiple content sub-libraries corresponding to the multiple division dimensions according to multiple division dimensions; for a given division dimension among the multiple division dimensions, for each of the multiple features, determine the index value of the labeled recommended content that is divided into the given division dimension and is related to the feature under a predetermined recommendation index, and select a reference feature for the given division dimension from the multiple features based on the difference between the index value determined for the multiple features and the reference index value; and for each of the multiple division dimensions, sort the recommended content in the content sub-library corresponding to the division dimension based on the feature value of the recommended content in the content sub-library corresponding to the division dimension on the corresponding reference feature, to obtain a sorting result for the multiple division dimensions.

[0118] In some embodiments, the candidate content determination module 910 is further configured to: for a second content filtering strategy among multiple content filtering strategies, determine a set of features associated with a reference user; sort the recommended content in the content library based on the feature values ​​of each recommended content in the content library on the set of features; and select a second set of candidate recommended content from the content library based on the sorting results.

[0119] In some embodiments, the candidate content determination module 910 is further configured to: determine multiple reference users corresponding to multiple partitioning dimensions of recommended content in the content library; and determine multiple sets of features associated with the multiple reference users; and wherein sorting the recommended content in the content library based on the feature values ​​of each recommended content in the content library on each feature of a set of features includes: dividing the recommended content in the content library into multiple content sub-libraries corresponding to the multiple partitioning dimensions according to the multiple partitioning dimensions; and for a given partitioning dimension among the multiple partitioning dimensions, sorting the recommended content in the content sub-library corresponding to the given partitioning dimension based on the feature values ​​of the content sub-library corresponding to the given partitioning dimension on the corresponding set of features.

[0120] In some embodiments, the candidate content determination module 910 is further configured to: determine a plurality of reference recommended content for a third content filtering strategy among a plurality of content filtering strategies; for each of the plurality of reference recommended content, sort the recommended content in the content library based on the similarity between each recommended content in the content library and the reference recommended content; and select a third group of candidate recommended content from the content library based on the sorting result for the plurality of reference recommended content.

[0121] In some embodiments, the machine learning model includes multiple first machine learning models, and the recommendation score determination module 920 is further configured to: for each candidate recommendation content in multiple sets of candidate recommendation content, use multiple first machine learning models to determine the first intermediate recommendation score of each candidate recommendation content, thereby obtaining multiple first intermediate recommendation scores of each candidate recommendation content; and determine the recommendation score of each candidate recommendation content based on the multiple first intermediate recommendation scores.

[0122] In some embodiments, the machine learning model further includes a second machine learning model, and the recommendation score determination module 920 is further configured to: use the second machine learning model to determine a second intermediate recommendation score for each candidate recommendation content based on a plurality of first intermediate recommendation scores and each candidate recommendation content in a plurality of candidate recommendation content groups; and determine a recommendation score for each candidate recommendation content based at least on the second intermediate recommendation scores.

[0123] In some embodiments, the recommendation score determination module 920 is further configured to: for each candidate recommendation content, determine the recommendation score of each candidate recommendation content based on the second intermediate recommendation score and multiple first intermediate recommendation scores.

[0124] The units and / or modules included in device 900 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 900 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0125] It should be understood that one or more steps in the above methods can be performed by appropriate electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices may, for example, include the content delivery platform 110 in Figure 1.

[0126] Figure 10 illustrates a block diagram of an electronic device 1000 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 1000 shown in Figure 10 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 1000 shown in Figure 10 can be used to implement the content delivery platform 110 of Figure 1 and / or the apparatus 900 of Figure 9.

[0127] As shown in Figure 10, the electronic device 1000 is in the form of a general-purpose electronic device. Components of the electronic device 1000 may include, but are not limited to, one or more processors or processing units 1010, memory 1020, storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. The processing unit 1010 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1020. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 1000.

[0128] Electronic device 1000 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1020 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 1030 can be removable or non-removable media and may include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 1000.

[0129] Electronic device 1000 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 10, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 1020 may include computer program product 1025 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0130] The communication unit 1040 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 1000 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 1000 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0131] Input device 1050 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1060 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 1000 can also communicate with one or more external devices (not shown) via communication unit 1040 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 1000, or with any device that enables electronic device 1000 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0132] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0133] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0134] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0135] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0137] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for content recommendation, comprising: Multiple content filtering strategies are used to determine multiple sets of candidate recommended content from the content library, and the multiple content filtering strategies are based on different content ranking criteria. Using a trained machine learning model, based on the user reference information of the target user and the content reference information of the multiple sets of candidate recommendation content, the recommendation score of each candidate recommendation content relative to the target user is determined. as well as Based on the recommendation scores corresponding to each candidate recommendation content in the multiple sets of candidate recommendation content, a set of target recommendation content is determined from the multiple sets of candidate recommendation content for use in providing to the target user.

2. The method according to claim 1, wherein determining multiple sets of candidate recommended content from the content library using multiple content filtering strategies includes: For each of the multiple content filtering strategies... Based on the content sorting criteria corresponding to this content filtering strategy, the recommended content in the content library is sorted. as well as Based on the ranking results, a set of candidate recommended content is selected from the content library.

3. The method according to claim 1, wherein determining multiple sets of candidate recommended content from the content library using multiple content filtering strategies includes: For the first content filtering strategy among the multiple content filtering strategies... Identify at least one feature that is associated with the predetermined recommendation metric; For each of the at least one feature, the recommended content in the content library is sorted based on the feature value of each recommended content in the content library for that feature; as well as Based on the ranking results for the at least one feature, a first group of candidate recommended content is selected from the content library.

4. The method according to claim 3, wherein the at least one feature comprises multiple features, and sorting the recommended content in the content library based on the feature value of the recommended content in the content library on each feature comprises: The recommended content in the content library is divided into multiple content sub-libraries corresponding to the multiple division dimensions, respectively, according to multiple division dimensions. For a given partition dimension among the multiple partition dimensions For each of the plurality of features, determine the index value of the labeled recommended content that is classified into the given classification dimension and is associated with that feature under the predetermined recommendation index, and Based on the difference between the indicator values ​​determined for the multiple features and the reference indicator values, a reference feature for the given partitioning dimension is selected from the multiple features. as well as For each of the multiple partitioning dimensions, based on the feature values ​​of the recommended content in the content sub-library corresponding to that partitioning dimension on the corresponding reference features, the recommended content in the content sub-library corresponding to that partitioning dimension is sorted to obtain the sorting result for the multiple partitioning dimensions.

5. The method according to claim 1, wherein determining multiple sets of candidate recommended content from the content library using multiple content filtering strategies includes: For the second content filtering strategy among the multiple content filtering strategies... Identify a set of characteristics associated with the reference user; Based on the feature values ​​of each recommended content in the content library on the set of features, the recommended content in the content library is sorted. as well as Based on the ranking results, a second group of candidate recommended content is selected from the content library.

6. The method of claim 5, wherein determining at least one feature associated with the reference user comprises: For the multiple dimensions of recommended content in the content library, Identify the multiple reference users corresponding to the multiple partitioning dimensions; as well as Determine multiple sets of features that are associated with the plurality of reference users respectively; Furthermore, the ranking of recommended content in the content library based on the feature values ​​of each recommended content item on each feature of the set of features includes: The recommended content in the content library is divided into categories based on multiple dimensions. The description of multiple content sub-libraries corresponding to multiple division dimensions; as well as For a given partitioning dimension among the multiple partitioning dimensions, the recommended content in the content sub-library corresponding to the given partitioning dimension is sorted based on the feature values ​​of the content sub-library corresponding to the given partitioning dimension on the corresponding set of features.

7. The method according to claim 1, wherein determining multiple sets of candidate recommended content from the content library using multiple content filtering strategies includes: For the third content filtering strategy among the multiple content filtering strategies... Identify multiple reference recommendations; For each of the plurality of reference recommended content, the recommended content in the content library is sorted based on the similarity between each recommended content in the content library and the reference recommended content; as well as Based on the ranking results of the multiple reference recommended contents, a third group of candidate recommended contents is selected from the content library.

8. The method of claim 1, wherein the machine learning model comprises a plurality of first machine learning models, and wherein determining the recommendation score of each candidate recommendation content relative to the target user in the plurality of candidate recommendation content groups includes: For each candidate recommendation content in the multiple sets of candidate recommendation content The first intermediate recommendation scores of each candidate recommendation content are determined by using the multiple first machine learning models, thereby obtaining multiple first intermediate recommendation scores for each candidate recommendation content. The recommendation score for each candidate recommendation is determined based on the multiple first intermediate recommendation scores.

9. The method of claim 8, wherein the machine learning model further comprises a second machine learning model, and wherein determining the recommendation score of each candidate recommendation content based on the plurality of first intermediate recommendation scores includes: Using the second machine learning model, based on the multiple first intermediate recommendation scores and each candidate recommendation content in the multiple sets of candidate recommendation content, the second intermediate recommendation score of each candidate recommendation content is determined; as well as Based at least on the second intermediate recommendation score, the recommended content for each candidate is determined. Recommended score.

10. The method of claim 9, wherein determining the recommendation score for each candidate recommendation content based at least on the second intermediate recommendation score includes: For each candidate recommended content, the recommendation score for each candidate recommended content is determined based on the second intermediate recommendation score and the plurality of first intermediate recommendation scores.

11. An apparatus for content recommendation, comprising: The candidate content determination module is configured to determine multiple sets of candidate recommended content from the content library using multiple content filtering strategies, wherein the multiple content filtering strategies are based on different content ranking criteria. The recommendation score determination module is configured to use a trained machine learning model to determine the recommendation score of each candidate recommendation content relative to the target user, based on the user reference information of the target user and the content reference information of the multiple sets of candidate recommendation content. as well as The target content determination module is configured to determine a set of target recommended content from the multiple sets of candidate recommended content based on the recommendation scores corresponding to each candidate recommended content in the multiple sets of candidate recommended content, for use in providing to the target user.

12. An electronic device, comprising: At least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

14. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 10.

Images