Method and apparatus for content recommendation, device, medium, and program product

Abstract

Embodiments of the present disclosure provide a method and apparatus for content recommendation, a device, a medium, and a program product. The method comprises: acquiring historical conversion parameter values corresponding to a group of historical recommended contents related to a plurality of content elements, wherein each historical recommended content comprises at least one content element among the plurality of content elements; on the basis of the historical conversion parameter values corresponding to the group of historical recommended contents, determining respective contribution scores of the plurality of content elements during conversion; at least on the basis of the respective contribution scores of the plurality of content elements, determining importance scores corresponding to a plurality of candidate contents to be recommended, wherein each candidate content to be recommended comprises at least one content element among the plurality of content elements; and on the basis of the importance scores corresponding to the plurality of candidate contents to be recommended, selecting at least one content to be recommended from among the plurality of candidate contents to be recommended, so as to provide, to a first user group, the at least one content to be recommended.

Description

Method, device, equipment, medium and program product for content recommendation TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and in particular, to a method, device, electronic device, computer-readable storage medium and computer program product for content recommendation. BACKGROUND

[0002] The Internet provides access to a wide variety of resources. For example, various applications, goods, audio and video content, etc. can be accessed through the Internet. In addition, content delivery and service promotion through the Internet has become a new form of information dissemination and is widely used. An advertising system supports displaying recommended content to users in different advertising display opportunities, enabling users to browse, obtain corresponding services, etc. according to needs, and achieving conversion of the recommended content. How to provide users with more expected and higher quality recommended content is a problem that has been studied in the recommendation scenario.

[0003] SUMMARY

[0004] In a first aspect of the present disclosure, a method for content recommendation is provided. The method comprises: obtaining a set of historical conversion parameter values corresponding to a plurality of historical recommended contents, each historical recommended content comprising at least one content element in the plurality of content elements; determining a contribution score of each of the plurality of content elements in conversion based on the set of historical conversion parameter values corresponding to the plurality of historical recommended contents; determining an importance score corresponding to a plurality of candidate recommended contents based on at least the contribution score of each of the plurality of content elements, each candidate recommended content comprising at least one content element in the plurality of content elements; and selecting at least one recommended content from the plurality of candidate recommended contents based on the importance score corresponding to the plurality of candidate recommended contents, for providing to a first user group.

[0005] In a second aspect of the disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the device to perform: obtaining historical conversion parameter values corresponding to a set of historical recommended contents related to a plurality of content elements, each historical recommended content including at least one content element of the plurality of content elements; determining a contribution score of each of the plurality of content elements in conversion based on the historical conversion parameter values corresponding to the set of historical recommended contents; determining an importance score corresponding to a plurality of candidate recommended contents based on at least the contribution score of each of the plurality of content elements, each candidate recommended content including at least one content element of the plurality of content elements; and selecting at least one recommended content from the plurality of candidate recommended contents based on the importance score corresponding to the plurality of candidate recommended contents for providing to a first user group.

[0006] In a third aspect of the disclosure, an apparatus for content recommendation is provided. The apparatus includes: a conversion obtaining module configured to obtain historical conversion parameter values corresponding to a set of historical recommended contents related to a plurality of content elements, each historical recommended content including at least one content element of the plurality of content elements; a contribution determining module configured to determine a contribution score of each of the plurality of content elements in conversion based on the historical conversion parameter values corresponding to the set of historical recommended contents; an importance determining module configured to determine an importance score corresponding to a plurality of candidate recommended contents based on at least the contribution score of each of the plurality of content elements, each candidate recommended content including at least one content element of the plurality of content elements; and a content selecting module configured to select at least one recommended content from the plurality of candidate recommended contents based on the importance score corresponding to the plurality of candidate recommended contents for providing to a first user group.

[0007] In a fourth aspect of the disclosure, a computer readable storage medium is provided. The medium has stored thereon a computer program which, when executed by a processor, implements the method of the first aspect.

[0008] In a fifth aspect of the disclosure, a computer program product is provided. The computer program product includes computer executable instructions that, when executed by a processor, implement the method according to the first aspect of the disclosure.

[0009] It should be understood that the contents described in this section are not intended to limit the key features or important features of the embodiments of the disclosure, nor are they used to limit the scope of the disclosure. Other features of the disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0011] FIG. 1 illustrates a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] FIG. 2 illustrates a flowchart of a process of recommendation content generation according to some embodiments of the present disclosure;

[0013] FIG. 3 illustrates a flowchart of a process for content recommendation according to some embodiments of the present disclosure;

[0014] FIG. 4 illustrates a block diagram of an apparatus for content recommendation according to some embodiments of the present disclosure; and

[0015] FIG. 5 illustrates a block diagram of an electronic device in which one or more embodiments of the present disclosure can be implemented. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are illustrated in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted as being limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It will be appreciated that the drawings of the present disclosure and the embodiments thereof are only for illustrative purposes and should not be construed as limiting the scope of protection of the present disclosure.

[0017] In the description of embodiments of the present disclosure, the term "comprising" and its conjugations should be understood to encompass the meanings of "including but not limited to", i.e., "comprising but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions can also be included below.

[0018] It can be understood that the data involved in the technical solutions of the present disclosure (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws and regulations and relevant provisions.

[0019] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the scope of use, the scenario of use, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0020] For example, in response to receiving an active request of a user, a prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed by the user will require obtaining and using personal information of the user, so that the user can autonomously select whether to provide the personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium, etc. performing the operation of the technical solution of the present disclosure according to the prompt information.

[0021] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending a prompt information to the user may, for example, be a pop-up window manner, and the prompt information may be presented in the pop-up window in the form of text. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0022] It can be understood that the above notification and obtaining user authorization process is only illustrative and does not limit the implementation manner of the present disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0023] As used herein, the term "model" can learn an association between respective inputs and outputs from training data, such that a corresponding output can be generated for a given input after training is completed. The generation of a model can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. A neural network model is one example of a model based on deep learning. In this document, a "model" can also be referred to as a "machine learning model", a "learning model", a "machine learning network", or a "learning network", which are used interchangeably herein.

[0024] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, which generally includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications generally include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence, such that the output of a previous layer is provided as input to a subsequent layer, with the input layer receiving the input to the neural network and the output of the output layer as the final output of the neural network. Each layer of a neural network includes one or more nodes (also referred to as processing nodes or neurons), each of which processes input from the previous layer.

[0025] Generally, machine learning can include three stages, namely a training stage, a testing stage, and an application stage (also referred to as an inference stage). In the training stage, a given model can be trained using a large amount of training data, iteratively updating parameter values until the model is able to obtain consistent inferences from the training data that satisfy an expected objective. Through training, the model can be considered to have learned an association (also referred to as a mapping) from input to output from the training data. The parameter values of the trained model are determined. In the testing stage, test inputs are applied to the trained model to determine whether the model is able to provide correct outputs, thereby determining the performance of the model. In the application stage, the model can be used to process actual inputs based on the trained parameter values to determine corresponding outputs.

[0026] FIG. 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. One or more client devices 130-1, 130-2, 130-3, etc. (collectively or individually referred to as client devices 130 for ease of discussion) are associated with a content placement platform 110 and can access various types of content provided on the content placement platform 110, e.g., based on respective users 132-1, 132-2, 132-3, etc. (collectively or individually referred to as users 132 for ease of discussion). As an example, the content placement platform 110 can be an application, a website, a web page, and other accessible platforms. The client devices 130 can be installed with an application for accessing the content placement platform 110 or can access the content placement platform 110 in a suitable manner.

[0027] The content placement platform 110 can be configured to place one or more specific recommended content (e.g., provided or presented at the client devices 130) related to one or more objects to a user group based on respective recommendation policies. The recommended content to be placed can include, for example, one or more recommended content 122-1, 122-2,... 122-M (collectively or individually referred to as recommended content 122 for ease of discussion) in a content database 120. The recommended content 122 is also sometimes referred to as “material” or recommended material.

[0028] In this document, examples of objects that can be recommended can include applications, physical goods / services, virtual goods / services, digital content / physical content, etc. In this document, “recommended content” refers to content that is presented for the purpose of recommending a corresponding object. Examples of recommended content can include advertisements. In this document, a user group can include one or more user members, e.g., users 132. A user member can be any potential object recipient or consumer of a service, e.g., a user, a group, an organization, an entity, etc. In some embodiments, the content placement platform 110 can provide corresponding recommended content 122 to a user group based on a request by a recommendation requester. In the context of advertisement placement, the service provider is also sometimes referred to as an advertiser.

[0029] The recommended content 122 served by the content serving platform 110 can be generated by the content generation system 140. The content generation system 140 can be configured to select content elements from the content elements 155-1, 155-2,... 155-M (collectively or individually referred to as content elements 155) stored in the element / template database 150 for generating the recommended content 122. Each recommended content 122 can include one or more content elements 155. The content elements 155 can include images, videos, texts, music, other audio, and the like multimedia content. By organizing these content elements, more complex recommended content can be formed for providing to users. The content elements 155 are sometimes also referred to as "materials" or "material elements" or "material units" for generating the material.

[0030] The sources of the content elements 155 can be diverse, which can include uploaded by the recommendation requester, from data sources specified by the recommendation requester, or can be user generated content (UGC) (of course, the use of the content is authorized by the user), and the like.

[0031] In some embodiments, the content generation system 140 can also access the templates 156-1, 156-2,... 156-K (collectively or individually referred to as templates 156) in the element / template database 150. The content generation system 140 can select the templates 156 for generating the recommended content 122. Each template can define a content layout or an organization / editing manner of the content of the recommended content, and can define placeholders, each of which can be used to fill in a content element 155. By utilizing the templates, the content generation system 140 can more efficiently generate diverse recommended content 122.

[0032] In the environment 100, the client device 130 can be any type of mobile terminal, fixed terminal, or portable terminal including a mobile handset, a tablet computer, a laptop computer, a notebook computer, a netbook computer, a smart device, a media player, a navigation device, a personal navigation device, a personal digital assistant (PDA), a smartphone, a tablet computer, a media player, a navigation device, a television receiver, a radio broadcast receiver, an e-book device, a game device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. In some embodiments, the client device 130 can also be capable of supporting any type of interface to a user (such as "wearable" circuitry, etc.).

[0033] In the environment 100, the content delivery platform 110, the recommendation conversion component 140, and / or the recommendation management system 150 may, for example, be various types of computing systems / servers capable of providing computing capabilities, including but not limited to mainframes, edge computing nodes, computing devices in a cloud environment, and the like. Although shown separately, one or more of the content delivery platform 110, the recommendation conversion component 140, and / or the recommendation management system 150 can be combined.

[0034] It should be appreciated that the components and arrangements in the environment shown in FIG. 1 are merely examples, and a computing system suitable for implementing the example embodiments described in this disclosure can include one or more different components, other components, and / or different arrangements.

[0035] In content recommendation, in order to improve the efficiency and effect of recommendation, it is desirable to be able to provide high-quality recommended content (material) for delivery.

[0036] Generally, when generating material for recommendation, content elements can be screened according to relevant indicators for synthesizing recommended content. For example, high-quality videos can be screened according to relevant video indicators such as video or picture play counts, like counts, collection counts, etc., and can be used as production materials for recommended content. When generating recommended content, different videos or pictures can be combined in a certain spatial or temporal manner with the help of video synthesis capabilities. In addition, audio synthesis capabilities can also be used to batch generate diversified recommended content for delivery.

[0037] In the traditional recommended content generation process, the recall logic of content elements, the synthesis logic of recommended content, and the delivery logic of the finally generated recommended content are all independent of each other and do not depend on each other. The strategy of each stage depends on manual configuration. In this way, when analyzing the conversion effect of the recommended content after delivery, it is difficult to attribute to the specific strategy in the previous process. This will affect the mining and iteration of higher quality recommended content. Generally, the number of initial content elements is large, and the combination of content elements is more. How to determine the highest quality recommended content from a large number of combination ways is a very worthy of attention and improvement problem.

[0038] In embodiments of the present disclosure, an improved content recommendation scheme is proposed. For a plurality of content elements, first, based on historical recommended content generated by the content items, a conversion of the historical recommended content is analyzed to obtain a historical conversion parameter value corresponding to the historical recommended content. Based on the historical conversion parameter value corresponding to the historical recommended content, a contribution score of each of the plurality of content elements in conversion is determined. Based on at least the contribution score of each of the plurality of content elements, an importance score corresponding to a plurality of candidate recommended content is determined, each of the candidate recommended content including at least one of the plurality of content elements. Based on the importance score corresponding to the plurality of candidate recommended content, at least one recommended content is selected from the plurality of candidate recommended content for providing to a first user group.

[0039] According to the scheme, the granularity of each content element is attributed from the historical conversion parameter value of the recommended content. By measuring the contribution of each content element in the historical conversion, the value of the candidate recommended content, i.e., the importance score, can be estimated. In this way, high-quality recommended content can be screened out at the content production stage for subsequent delivery. This can improve the conversion efficiency of subsequent content delivery, so that users can receive high-quality recommended content.

[0040] Some example embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0041] FIG. 2 shows a flowchart of a process 200 of recommended content generation according to some embodiments of the present disclosure. For ease of discussion, it will be described in conjunction with FIG. 1. The process 200 can be implemented at the content generation system 140.

[0042] The content generation system 140 can determine to generate one or more recommended content 122 for providing to a user group in response to a content generation task scheduling 202. The content generation task scheduling 202 can be periodic, or triggered in response to a configured condition. Embodiments of the present disclosure do not limit this.

[0043] At the stage of element and template recall 210, the content generation system 140 can select a plurality of content elements 155 for content generation according to a recall strategy. In some embodiments, the plurality of content elements 155 can be content elements related to a specific object. The recall strategy can specify an identifier of the object. In some embodiments, the recall strategy can further include preconfigured rules including content requirements specified by the recommendation requester, a play volume requirement, a like volume requirement, a comment volume requirement, whether having a specific tag, etc.

[0044] In some embodiments, for some types of recommended content, or depending on the specific content generation strategy, the content generation system 140 can also select one or more templates 156 for content generation. The screened content elements 155 and templates 156 can go through a stage of content pre-processing 220 to pre-process various types of content elements. The pre-processing operations can be, for example, size transformation, filter beautification, and the like, of the content elements.

[0045] Generally, the number of screened content elements 155 and templates 156 is still very large, if not infinite, under the requirement of meeting the demand. If these content elements 155 are embedded into specific templates 156 in a way of permutation and combination for image or video generation, a large number of recommended contents will be generated, some of which can not be of high quality and can not be able to obtain user conversion. Therefore, it is desirable for the content generation system 140 to generate higher quality recommended contents in the content generation stage.

[0046] In embodiments of the present disclosure, the content generation system 140 performs a stage of content importance evaluation 230. For the content importance evaluation 230, the content generation system 140 analyzes historical conversion data 232 of recommended contents (also referred to as historical recommended contents) that have been launched in the past period of time to determine a historical conversion parameter value of each historical recommended content item. The historical recommended contents of interest here refer to recommended contents that include one or more content elements 155 screened in the stage of element and template recall 210. In some embodiments, the historical recommended contents also include the screened templates 156 in the case of considering using templates 156.

[0047] The historical conversion data 232 can be conversion data collected after the historical recommended contents are provided to a user group for a period of time. In the initial stage, the launch of such recommended contents is referred to as the cold start stage of the recommended contents. At this time, there can be no historical conversion data to refer to.

[0048] The conversion parameter is used to measure the conversion efficiency brought by the recommended content after being provided to a plurality of users or being launched for a period of time, including the probability of a user performing a conversion behavior, or the index value (or revenue value) corresponding to the user performing the conversion behavior. When the historical recommended content is provided to a user group, the conversion parameter (or conversion index) of interest can be determined according to the specific application scenario and the object type corresponding to the launched recommended content. For example, if the object involved in the recommended content is an application, and the conversion behavior of interest is the installation of the application, then the conversion parameter can include the installation per creative (IPC) and / or the installation per mile (IPM) of the recommended content. For other types of objects, the conversion behavior or conversion parameter of interest can be different. For example, for online content, the conversion behavior of interest is an interaction behavior such as a like, a collection, a comment, etc., and the conversion parameter of interest can be the number of likes, the number of collections, or the number of comments per thousand exposures of the recommended content; for goods, the conversion behavior of interest can be behaviors such as visiting a product detail page, adding to a shopping cart, and purchasing, and the conversion parameter of interest can be the number of visits, the number of additions to a shopping cart, or the number of purchases per thousand exposures of the recommended content, etc.

[0049] In the cold start phase, in order to collect conversion data, a plurality of content elements 155 can be randomly combined (and a template 156 is also randomly selected) to generate a set of recommended content for launching to a certain user group, and the conversion data of these recommended content is collected to obtain the corresponding historical conversion parameter value. These recommended content are used as historical recommended content in the subsequent, and their conversion parameter values are used as historical conversion parameter values for improving the subsequent recommended content generation process. In some embodiments, after the recommended content generated according to the above embodiments is launched, the conversion data of these recommended content can continue to be collected and stored as historical conversion data 232 to continue to support the generation process of more subsequent recommended content.

[0050] After obtaining the historical conversion parameter values corresponding to the set of historical recommended content related to the plurality of content elements 155, the contribution score of each of the plurality of content elements 155 in conversion can be determined based on the historical conversion parameter values corresponding to the historical recommended content. In this way, although only the overall conversion parameter value of the recommended content can be collected after the recommended content is generated and launched, the contribution of the unit content element contained in the recommended content can be measured based on the overall conversion parameter. The contribution score of each content element 155 refers to the conversion data corresponding to the part of the recommended content that includes the content element in the recommended content that has been launched previously.

[0051] In this way, the importance score of each candidate recommendation can be determined based on the contribution score of each content element 155. Each candidate recommendation to be evaluated includes at least one content element 155. In some embodiments, the contribution score of each template 156 in conversion can also be determined based on the historical conversion parameter values of historical recommendation content that include the template 156, similarly taking into account the template 156. In this way, the importance score of each candidate recommendation can be determined based on the contribution score of each content element 155 and the contribution score of each template 156. Each candidate recommendation also includes a template 156 for organizing the content elements 155.

[0052] In some embodiments, in determining the contribution score of a content element 155, the content generation system 140 can determine the historical conversion parameter values of each historical recommendation that includes the content element 155 from the set of historical conversion parameter values of each historical recommendation. The contribution score of the content element 155 can be determined based on the number of historical recommendations that include the content element 155 and the historical conversion parameter values of each historical recommendation.

[0053] Similarly, in determining the contribution score of a template 156, the content generation system 140 can determine the historical conversion parameter values of each historical recommendation that includes the template 156 from the set of historical conversion parameter values of each historical recommendation. The contribution score of the template 156 can be determined based on the number of historical recommendations that include the template 156 and the historical conversion parameter values of each historical recommendation.

[0054] In determining the importance score of each candidate recommendation, the content generation system 140 can determine the number of content elements 155 to be included in each candidate recommendation. In some embodiments, the number of content elements 155 to be included in each candidate recommendation is related to the selected template, e.g., depending on the placeholders defined in the template. In some embodiments, if not dependent on a template, the number of content elements 155 to be included in each candidate recommendation can be randomly defined.

[0055] In this way, when calculating the importance score of each candidate recommended content, the importance score of the candidate recommended content can be determined by weighted summation, weighted average, etc. based on the contribution score of the included content elements 155 and the contribution score of the template 156. In some embodiments, the weight of the content elements 155 and the weight of the template 156 can be configured separately, for example, can be configured as different weights. For example, the weight of the content elements 155 can be greater than the weight of the template 156. In some embodiments, the weights of different types of content elements 155 can also be different. The specific values of the weights can be configured according to actual application, and the scope of embodiments of the present disclosure is not limited in this regard.

[0056] Suppose the screened content elements are x1, x2, …, x n , the template is y1, y2, … y m , and each template includes k placeholders (so it can include k content elements). Note that here, for the sake of convenience of discussion, it is assumed that each template includes the same number of placeholders, but in actual application, the number of placeholders included in different templates can be different. Suppose that after the aforementioned content elements and templates are arranged and combined to obtain recommended contents and used for delivery, the historical conversion parameter of each recommended content is represented as V. At this time, the contribution scores of the content elements and the template can be determined by the following, and the importance score of the candidate recommended content can be estimated:

[0057] The contribution score of the content element 155 = (V1+V2+…Vn1) / n1, the number of historical recommended contents generated using this content element.

[0058] The contribution score of the template 156 = (V1+V2+…Vn2) / n2, the number of historical recommended contents generated using this template.

[0059] Where w1 and w1 indicate the weight of the content element and the weight of the template respectively, (xi+xj…+xk) represents the k content elements included in the candidate recommended content (note that different candidate recommended contents include different content elements).

[0060] It should be understood that the above gives only an example of a way to calculate the importance score of the candidate recommended content, and other calculation methods can also be configured in practice.

[0061] The importance score corresponding to the candidate recommended content can measure the value of the recommended content, and this importance is determined by the contribution degree of the content elements to be included in the candidate recommended content verified in the historical recommendation process.

[0062] After the importance scores are estimated, in the content combination strategy determination 240 stage, the content generation system 140 can determine which combination manner to select to generate the candidate recommendation content based on the importance scores of the plurality of candidate recommendation contents determined. That is, in the content importance evaluation, it is not necessary to complete the synthesis of the candidate recommendation content, but to evaluate the importance that different combination manners of the content elements 155 and the templates 156 can correspond to. Then, depending on the determined importance scores, the content generation system 240 can select the candidate recommendation content with the highest importance score or the importance score satisfying a certain number of expected ones to generate.

[0063] In the recommendation content generation 250 stage, the content generation system 240 can generate at least one recommendation content with higher importance based on the determined content combination strategy. In the recommendation content delivery 260 stage, the content generation system 240 provides the generated at least one recommendation content to the user group.

[0064] As mentioned earlier, in the cold start stage, the conversion data can be obtained by random combination manner, and after the recommendation content is generated based on the importance score, the conversion data of the recommendation content can still be collected, and this part of the conversion data can be used to update the contribution scores of the content elements and the templates, and the advantage strategy accumulation can be realized. In this way, the subsequent evaluation of the importance of the recommendation content can be more accurate.

[0065] In some embodiments, after the important recommendation content is selected based on the importance score for delivery, if the conversion parameter value of the recommendation content is high (for example, the conversion parameter value exceeds the conversion threshold), then the label or other information of the content element 155 or the template 156 included in the at least one recommendation content can be determined. These labels or other related information can be used to guide and optimize the recall strategy in the recall stage, so as to select other content elements / templates similar to the content elements / templates in the high-quality recommendation content in the recall stage for generating recommendation content.

[0066] In some embodiments, since the amount of recommendation content finally put into use is limited, and the combination strategy is unlimited, in order to maximize the activity of the strategy, in addition to selecting the recommendation content for delivery based on the importance score, another recommendation content can also be generated by randomly combining a plurality of content elements 155. Within a predetermined time period, the additional recommendation content and the recommendation content selected based on the importance score are provided to the user group together. That is, adaptive elimination can be realized, and a certain proportion of recommendation content production uses a better strategy, while the rest of the recommendation content production still maintains a random combination manner, which can ensure the flexibility of the strategy, and in the future, new advantage combination strategies can be determined to replace the old and decaying combination strategies.

[0067] FIG. 3 illustrates a flowchart of a process 300 for content recommendation, according to some embodiments of the present disclosure. The process 300 can be implemented, for example, in the content generation system 140 of FIG. 1.

[0068] At block 310, the content generation system 140 obtains historical conversion parameter values corresponding to a set of historical recommended contents related to a plurality of content elements, each historical recommended content including at least one content element of the plurality of content elements.

[0069] At block 320, the content generation system 140 determines a contribution score of each of the plurality of content elements in conversion based on the historical conversion parameter values corresponding to the set of historical recommended contents.

[0070] At block 330, the content generation system 140 determines an importance score corresponding to a plurality of candidate recommended contents based on the contribution scores of the plurality of content elements, each candidate recommended content including at least one content element of the plurality of content elements.

[0071] At block 340, the content generation system 140 selects at least one recommended content from the plurality of candidate recommended contents based on the importance scores corresponding to the plurality of candidate recommended contents for providing to the first user group.

[0072] In some embodiments, determining the contribution score of each of the plurality of content elements in conversion includes, for each content element of the plurality of content elements, determining historical conversion parameter values corresponding to respective historical recommended contents including the content element from the historical conversion parameter values corresponding to the set of historical recommended contents; and determining the contribution score of the content element based on a number of the respective historical recommended contents including the content element and the historical conversion parameter values corresponding to the respective historical recommended contents.

[0073] In some embodiments, each historical recommended content further includes a template selected from a plurality of templates for organizing the at least one content element, wherein determining the importance score corresponding to the plurality of candidate recommended contents includes determining a contribution score of each of the plurality of templates in conversion based on the historical conversion parameter values corresponding to the set of historical recommended contents; and determining the importance score corresponding to the plurality of candidate recommended contents based on the contribution scores of the plurality of content elements and the contribution scores of the plurality of templates, each recommended content further including the template for organizing the at least one content element.

[0074] In some embodiments, determining the contribution score of each of the plurality of templates in conversion includes, for each template of the plurality of templates, determining historical conversion parameter values corresponding to respective historical recommended contents including the template from the historical conversion parameter values corresponding to the set of historical recommended contents; and determining the contribution score of the template based on a number of the respective historical recommended contents including the template and the historical conversion parameter values corresponding to the respective historical recommended contents.

[0075] In some embodiments, the process 300 further includes: generating at least one additional recommended content by randomly combining the plurality of content elements; and providing the at least one additional recommended content together with the selected at least one recommended content to the first user group within a predetermined time period.

[0076] In some embodiments, the acquiring the historical conversion parameter values corresponding to the set of historical recommended contents related to the plurality of content elements includes: generating the set of historical recommended contents by randomly combining the plurality of content elements; and collecting the historical conversion parameter values corresponding to the set of historical recommended contents after providing the set of historical recommended contents to a second user group.

[0077] In some embodiments, the process 300 further includes: collecting a conversion parameter value corresponding to the at least one recommended content after providing the at least one recommended content to the first user group; determining at least one label of a content element included in the at least one recommended content in response to the conversion parameter value corresponding to the at least one recommended content exceeding a conversion threshold; and selecting a content element from the set of content elements for generating a recommended content based at least on the at least one label.

[0078] FIG. 4 illustrates a schematic structural block diagram of an apparatus 400 for content recommendation, according to some embodiments of the present disclosure. The apparatus 400 can be implemented as or included in the content generation system 140 of FIG. 1. Various modules / components in the apparatus 400 can be implemented by hardware, software, firmware, or any combination thereof.

[0079] As shown, the apparatus 400 includes a conversion acquisition module 410 configured to acquire historical conversion parameter values corresponding to a set of historical recommended contents related to a plurality of content elements, each historical recommended content including at least one content element of the plurality of content elements. The apparatus 400 further includes a contribution determination module 420 configured to determine a contribution score of each of the plurality of content elements in conversion based on the historical conversion parameter values corresponding to the set of historical recommended contents. The apparatus 400 includes an importance determination module 430 configured to determine an importance score corresponding to a plurality of candidate recommended contents based at least on the contribution score of each of the plurality of content elements, each candidate recommended content including at least one content element of the plurality of content elements. The apparatus 400 further includes a content selection module 440 configured to select at least one recommended content from the plurality of candidate recommended contents based on the importance score corresponding to the plurality of candidate recommended contents for providing to a first user group.

[0080] In some embodiments, the contribution determination module 420 is further configured to, for each content element in the plurality of content elements, determine, from the historical conversion parameter values respectively corresponding to the set of historical recommendation contents, a historical conversion parameter value corresponding to each historical recommendation content including the content element; and determine, based on the number of each historical recommendation content including the content element and the historical conversion parameter value corresponding to each historical recommendation content, the contribution score of the content element.

[0081] In some embodiments, each historical recommendation content further includes a template selected from a plurality of templates for organizing the at least one content element. The importance determination module 430 is further configured to determine, based on the historical conversion parameter values respectively corresponding to the set of historical recommendation contents, a contribution score of each of the plurality of templates in conversion; and determine, based on the contribution score of each of the plurality of content elements and the contribution score of each of the plurality of templates, the importance score of each of the plurality of candidate recommendation contents, each of the recommendation contents further including a template for organizing the at least one content element.

[0082] In some embodiments, the importance determination module 430 is further configured to, for each template in the plurality of templates, determine, from the historical conversion parameter values respectively corresponding to the set of historical recommendation contents, a historical conversion parameter value corresponding to each historical recommendation content including the template; and determine, based on the number of each historical recommendation content including the template and the historical conversion parameter value corresponding to each historical recommendation content, the contribution score of the template.

[0083] In some embodiments, the apparatus 400 further includes a random combination module configured to generate at least one additional recommendation content by randomly combining the plurality of content elements; and a combination providing module configured to provide, to the first user group, the at least one additional recommendation content together with the selected at least one recommendation content within a predetermined time period.

[0084] In some embodiments, the conversion acquisition module 410 is further configured to generate the set of historical recommendation contents by randomly combining the plurality of content elements; and acquire the historical conversion parameter values respectively corresponding to the set of historical recommendation contents after providing the set of historical recommendation contents to a second user group.

[0085] In some embodiments, the apparatus 400 further includes a conversion acquisition module configured to acquire, after providing the selected at least one recommendation content to the first user group, a conversion parameter value corresponding to the at least one recommendation content; a label determination module configured to determine, in response to the conversion parameter value corresponding to the at least one recommendation content exceeding a conversion threshold, at least one label of a content element included in the at least one recommendation content; and an element selection module configured to select, based at least on the at least one label, a content element from the set of content elements for generating a recommendation content.

[0086] FIG. 5 illustrates a block diagram of an electronic device 500 in which one or more embodiments of the disclosure can be implemented. It should be understood that the electronic device 500 illustrated in FIG. 5 is merely an example and should not be construed to limit the functionality and scope of the embodiments described herein. The electronic device 500 illustrated in FIG. 5 can be used to implement the content generation system 140. The electronic device 500 can include or be implemented as the apparatus 400 of FIG. 4.

[0087] As illustrated in FIG. 5, the electronic device 500 is in the form of a general- purpose computing device. Components of the electronic device 500 can include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 can be a real or virtual processor and capable of executing various processing in accordance with programs stored in the memory 520. In a multi-processing system, multiple processing units execute computer-executable instructions in parallel to improve the processing power of the electronic device 500.

[0088] The electronic device 500 typically includes a plurality of computer storage media. Such media can be any available media that is located either internally or externally to the electronic device 500, including, but not limited to, volatile and non-volatile media, removable and non-removable media. The memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 can be a removable or non-removable media, and can include machine-readable media, such as a flash drive, a magnetic disk, or any other media that can be used to store information and / or data (e.g., training data for training) and that can be accessed by the electronic device 500.

[0089] The electronic device 500 can further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 5, a disk drive and a disk drive interface can be provided for reading from or writing to a removable, non- volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive and disk interface can be provided for reading from or writing to a removable, non-volatile optical disk (e.g., a "CD-ROM" or "DVD"). In these instances, each drive can be connected to the bus (not shown) by one or more data media interfaces. The memory 520 can include a computer program product 525 having one or more program modules configured to carry out the various methods or actions of the various embodiments of the present disclosure.

[0090] The communication unit 540 enables communication through the communication medium with other electronic devices. Additionally, the functionality of the components of the electronic device 500 can be implemented in a single computing cluster or a plurality of computer machines that are capable of communicating over a communication connection. As such, the electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0091] The input device 550 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 560 can be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 can also communicate with one or more external devices (not shown), such as a storage device, a display device, etc., through the communication unit 540, as necessary, with one or more devices that enable a user to interact with the electronic device 500, or with any device (e.g., a network card, a modem, etc.) that enables the electronic device 500 to communicate with one or more other electronic devices. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0092] According to an example implementation of the present disclosure, there is provided a computer-readable storage medium having computer-executable instructions stored thereon, where the computer-executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, there is also provided a computer program product tangibly stored on a non-transitory computer-readable medium and including computer-executable instructions, where the computer-executable instructions are executed by a processor to implement the method described above.

[0093] Various aspects of the disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, and computer program products implemented in accordance with the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0094] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including a processor executable program of instructions implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0095] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0096] The flow diagrams and block diagrams in the attached figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various implementations of the present disclosure. In this regard, each block in the flow diagrams and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logic functions (s). It should also be noted that in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or acts or combinations of

[0097] implementations have been described above, the description is intended to be illustrative, and not restrictive, and is not intended to exclude other implementations from the scope of the implementations disclosed herein. Many modifications and variations to the implementations described herein are possible and are within the scope of the implementations described herein. The order or sequence of any process or method can be different from that described. Furthermore, any features or combinations of features in the present disclosure are meant to be exemplary and are not meant to suggest that the features are exhaustive or that the combinations are the only ones that can be made. A variety of alternatives and equivalents will be apparent to those of ordinary skill in the art. The terminology used by the inventor in this disclosure has been chosen for reasons of readability and inclusivity, to make a disclosure suitable for use by others skilled in the art.

Claims

1. A method of content recommendation, comprising: obtaining historical conversion parameter values corresponding to a set of historical recommendation contents associated with a plurality of content elements, each historical recommendation content comprising at least one content element of the plurality of content elements; determining a contribution score of each of the plurality of content elements in conversion based on the historical conversion parameter values corresponding to the set of historical recommendation contents; determining an importance score corresponding to a plurality of candidate recommendation contents based on at least the contribution score of each of the plurality of content elements, each candidate recommendation content comprising at least one content element of the plurality of content elements; and selecting at least one recommendation content from the plurality of candidate recommendation contents based on the importance score corresponding to the plurality of candidate recommendation contents for providing to a first user group. 2.The method of claim 1, wherein determining the contribution score of each of the plurality of content elements in conversion comprises: for each content element of the plurality of content elements, determining historical conversion parameter values corresponding to each historical recommendation content comprising the content element from the historical conversion parameter values corresponding to the set of historical recommendation contents; and determining the contribution score of the content element based on the number of each historical recommendation content comprising the content element and the historical conversion parameter values corresponding to each historical recommendation content. 3.The method of claim 1, wherein each historical recommendation content further comprises a template selected from a plurality of templates for organizing the at least one content element, and wherein determining the importance score corresponding to the plurality of candidate recommendation contents comprises: determining a contribution score of each of the plurality of templates in conversion based on the historical conversion parameter values corresponding to the set of historical recommendation contents; and determining the importance score corresponding to the plurality of candidate recommendation contents based on the contribution score of each of the plurality of content elements and the contribution score of each of the plurality of templates, each recommendation content further comprising a template for organizing the at least one content element. 4.The method of claim 3, wherein determining the contribution score of each of the plurality of templates in conversion comprises: for each template of the plurality of templates, determining historical conversion parameter values corresponding to each historical recommendation content comprising the template from the historical conversion parameter values corresponding to the set of historical recommendation contents; and determining the contribution score of the template based on the number of each historical recommendation content comprising the template and the historical conversion parameter values corresponding to each historical recommendation content. 5.The method of claim 1, further comprising: generating at least one additional recommendation content by randomly combining the plurality of content elements; and providing the at least one additional recommendation content together with the selected at least one recommendation content to the first user group within a predetermined time period. 6.The method of claim 1, wherein obtaining the historical conversion parameter values corresponding to the set of historical recommendation contents associated with the plurality of content elements comprises: generating the set of historical recommendation contents by randomly combining the plurality of content elements; and collecting the historical conversion parameter values corresponding to the set of historical recommendation contents after providing the set of historical recommendation contents to a second user group. ​ ​ ​ 7. The method of claim 1, further comprising: collecting a conversion parameter value corresponding to the at least one recommended content after providing the at least one recommended content to the first group of users; in response to the conversion parameter value corresponding to the at least one recommended content exceeding a conversion threshold, determining at least one tag of a content element included in the at least one recommended content; and based at least on the at least one tag, selecting a content element from a set of content elements for generating a recommended content.

8. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions which, when executed by the at least one processing unit, cause the electronic device to perform operations comprising: obtaining historical conversion parameter values corresponding to a set of historical recommended contents related to a plurality of content elements, each historical recommended content including at least one content element of the plurality of content elements; based on the historical conversion parameter values corresponding to the set of historical recommended contents, determining a contribution score of each of the plurality of content elements in conversion; based at least on the contribution score of each of the plurality of content elements, determining an importance score corresponding to a plurality of candidate recommended contents, each candidate recommended content including at least one content element of the plurality of content elements; and based on the importance score corresponding to the plurality of candidate recommended contents, selecting at least one recommended content from the plurality of candidate recommended contents for providing to a first group of users.

9. The electronic device of claim 8, wherein determining the contribution score of each of the plurality of content elements in conversion comprises: for each content element of the plurality of content elements, from the historical conversion parameter values corresponding to the set of historical recommended contents, determining historical conversion parameter values corresponding to historical recommended contents including the content element; based on the number of historical recommended contents including the content element and the historical conversion parameter values corresponding to the historical recommended contents, determining the contribution score of the content element.

10. The electronic device of claim 8, wherein each historical recommended content further includes a template selected from a plurality of templates for organizing the at least one content element, and wherein determining the importance score corresponding to the plurality of candidate recommended contents comprises: based on the historical conversion parameter values corresponding to the set of historical recommended contents, determining a contribution score of each of the plurality of templates in conversion; and based on the contribution score of each of the plurality of content elements and the contribution score of each of the plurality of templates, determining the importance score corresponding to the plurality of candidate recommended contents, each candidate recommended content further including a template for organizing the at least one content element.

11. The electronic device of claim 10, wherein determining the contribution score of each of the plurality of templates in conversion comprises: for each template of the plurality of templates, from the historical conversion parameter values corresponding to the set of historical recommended contents, determining historical conversion parameter values corresponding to historical recommended contents including the template; and based on the number of historical recommended contents including the template and the historical conversion parameter values corresponding to the historical recommended contents, determining the contribution score of the template. ​ ​ ​ Based on the number of the historical recommendation contents including the template and the historical conversion parameter values corresponding to the historical recommendation contents, a contribution score of the template is determined. 12.The electronic device of claim 8, further comprising: generating at least one additional recommendation content by randomly combining the plurality of content elements; providing the at least one additional recommendation content together with the selected at least one recommendation content to the first user group within a predetermined time period. 13.The electronic device of claim 8, wherein the obtaining the historical conversion parameter values corresponding to the set of historical recommendation contents related to the plurality of content elements comprises: generating the set of historical recommendation contents by randomly combining the plurality of content elements; and collecting the historical conversion parameter values corresponding to the set of historical recommendation contents after providing the set of historical recommendation contents to a second user group. 14.The electronic device of claim 8, wherein the actions further comprise: collecting conversion parameter values corresponding to the selected at least one recommendation content after providing the selected at least one recommendation content to the first user group; determining at least one label of content elements included in the at least one recommendation content in response to the conversion parameter values corresponding to the at least one recommendation content exceeding a conversion threshold; and selecting content elements from a set of content elements for generating recommendation contents based on at least the at least one label. 15.An apparatus for content recommendation, comprising: a conversion obtaining module configured to obtain historical conversion parameter values corresponding to a set of historical recommendation contents related to a plurality of content elements, each historical recommendation content including at least one content element of the plurality of content elements; a contribution determining module configured to determine contribution scores of the plurality of content elements respectively in conversion based on the historical conversion parameter values corresponding to the set of historical recommendation contents; an importance determining module configured to determine importance scores corresponding to a plurality of candidate recommendation contents based on at least the contribution scores of the plurality of content elements respectively, each candidate recommendation content including at least one content element of the plurality of content elements; and a content selecting module configured to select at least one recommendation content from the plurality of candidate recommendation contents based on the importance scores corresponding to the plurality of candidate recommendation contents for providing to a first user group. 16.A computer-readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the method according to any one of claims 1 to 7. 17.A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 7. ​ ​ ​ ​

Images