Method and device for determining click rate of content, electronic equipment and medium

Abstract

The embodiment of the invention relates to a method and device for determining the click rate of content, electronic equipment and a medium. The method includes determining a first click rate of the content based on content characteristics of the content and user characteristics. The method also includes determining a confidence interval for the first click rate. The method further includes determining a second click rate of the category based on the category to which the content belongs and the user characteristics. The method further includes determining a click rate of the content based on the confidence interval and the second click rate. In this way, the content click rate can be corrected based on the confidence interval, the problem of estimation deviation of the content click rate caused by too little sample data is avoided, and therefore the accuracy of the content click rate and the accuracy of the final click rate are improved. In addition, according to the embodiment of the invention, the final click rate is calculated in combination with the category click rate of the category, so that exploration only aiming at the content is avoided, the range of the recommended content is prevented from being smaller and smaller, and the accuracy of the final click rate is further improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of computers, and more particularly, to a method, apparatus, electronic device, and medium for determining a click rate of content. Background Art

[0002] With the enrichment of online content, online content recommendation has become an important way to filter and provide content. Through online content recommendation, specific content can be sent to users to help them explore content needs and avoid users from filtering through massive amounts of online content, thereby improving the efficiency and experience of users browsing online content.

[0003] For online content recommendation, predicting the click-through rate of content is an important means to improve the recommendation effect. The click-through rate refers to the probability that a specific content is clicked by a user. Through the prediction results of the click-through rate, we can know the matching degree between the content and the user and optimize the recommendation process. Therefore, how to improve the accuracy of the click-through rate is one of the core issues of online content recommendation. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, apparatus, electronic device, and medium for determining a click rate of content.

[0005] In a first aspect of the present disclosure, a method for determining a click-through rate of content is provided. The method includes determining a first click-through rate of the content based on content features and user features of the content. The method also includes determining a confidence interval of the first click-through rate. The method also includes determining a second click-through rate of a category based on a category to which the content belongs and user features. In addition, the method also includes determining the click-through rate of the content based on the confidence interval and the second click-through rate.

[0006] In a second aspect of the present disclosure, a device for determining a click rate of content is provided. The device includes a first click rate determination module, which is configured to determine a first click rate of content based on content features and user features of the content. The device also includes a confidence interval determination module, which is configured to determine a confidence interval of the first click rate. The device also includes a second click rate determination module, which is configured to determine a second click rate of a category based on the category to which the content belongs and user features. In addition, the device also includes a click rate determination module, which is configured to determine the click rate of the content based on the confidence interval and the second click rate.

[0007] In a third aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory coupled to the processor, wherein the memory has instructions stored therein, and when the instructions are executed by the processor, the electronic device executes the method according to the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to the first aspect.

[0009] The purpose of this Summary is to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram illustrating an example environment in which some embodiments of the present disclosure may be implemented;

[0012] Figure 2 A flowchart showing a method for determining a click-through rate of content according to some embodiments of the present disclosure is shown;

[0013] Figure 3 A schematic diagram showing a process of determining a score of content according to some embodiments of the present disclosure;

[0014] Figure 4 A schematic diagram showing a process of determining an upper bound and a lower bound of a confidence interval according to some embodiments of the present disclosure;

[0015] Figure 5 A schematic diagram of a coordinate axis for displaying a confidence interval according to some embodiments of the present disclosure is shown;

[0016] Figure 6 A schematic diagram showing a process of determining recommended content and sample content in some embodiments of the present disclosure;

[0017] Figure 7 A block diagram showing an apparatus for determining a click rate of content according to some embodiments of the present disclosure; and

[0018] Figure 8 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0019] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, when receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0027] Click-through rate prediction is a modeling method used by recommendation systems when recommending content to users. It builds a recommendation algorithm model and predicts the probability of a specific content being clicked by a user (i.e., click-through rate) based on the recommendation algorithm model. Traditionally, the prediction of click-through rate relies on historical data. Generally speaking, the larger the amount of historical data, the more accurate the prediction of click-through rate. However, for users with low activity, users with a lot of long-tail content, and new users, there is less historical data to rely on, which leads to inaccurate prediction of click-through rate. If the estimated value of the click-through rate is too low, it will not be conducive to the exploration of new content, resulting in a narrower range of content recommended to users. If the estimated value of the click-through rate is too high, it will lead to a large number of recommendations of new content, among which lower-quality content will cause a decline in click-through rate, affecting the user experience.

[0028] In the embodiment of the present disclosure, the probability of the content being clicked by the user, i.e., the content click-through rate (which may be referred to as the first click-through rate), is determined based on the content features and user features of the content, and the corresponding confidence interval is determined based on the content click-through rate. Then, based on the category of the content and the user features, the probability of clicking on the content of this type, i.e., the type click-through rate (which may be referred to as the second click-through rate), is determined. Further, based on the confidence interval and the type click-through rate, the final click-through rate for the content is determined. In this way, the content click-through rate can be corrected through the confidence interval to avoid the problem of estimated deviation of the content click-through rate caused by too little sample data, thereby improving the accuracy of the content click-through rate and the final click-through rate. In addition, in the process of determining the final click-through rate, the category click-through rate (as prior knowledge) for predicting the category is also combined, thereby avoiding exploration only for the content and preventing the scope of the recommended content from becoming smaller and smaller, thereby further improving the accuracy of the final click-through rate.

[0029] Figure 1 A schematic diagram of an example environment 100 is shown in which some embodiments of the present disclosure may be implemented. Figure 1 The example environment includes a server 102 and an electronic device 118. The server 102 can be configured as a computing system, a single server, a distributed server, or a cloud-based server, etc. The electronic device 118 can be configured as a user terminal, a mobile device, a computer, etc.

[0030] Continue to refer Figure 1, the server 102 determines the category feature 104 and the content feature 108 for the content, and the user feature 106 for the user. Then, based on the category feature 104 and the user feature 106, the category click rate 110 (which may be referred to as the second click rate) of the content of the category being clicked by the user is determined. At the same time, based on the content feature 108 and the user feature 106, the content click rate 112 (which may be referred to as the first click rate) of the content being clicked by the user is determined. Then, the confidence interval 114 of the content click rate 112 is calculated, and the final click rate 116 is calculated based on the confidence interval 114 and the category click rate 110.

[0031] Continue to refer Figure 1 After the server 102 determines the final click rate 116, if the click rate 116 satisfies the condition, the corresponding content 120 is sent to the user's electronic device 118, and the content 120 can be displayed on the screen of the electronic device 118. Then, the click result 122 of the content 120 is determined, and the click result 122 is sent to the server 102. In some embodiments, the click result 122 includes the result that the content 120 is exposed and clicked (in this case, the content 120 can be used as a positive sample content) and the result that the content 120 is exposed but not clicked (in this case, the content 120 can be used as a negative sample content).

[0032] In this way, the content click rate 112 can be corrected through the confidence interval 114 to avoid the estimation deviation of the content click rate 112 caused by too small sample data, thereby improving the accuracy of the content click rate 112 and the final click rate 116. In addition, in the process of determining the final click rate 116, the category click rate 110 predicted for the category is also combined, thereby avoiding the exploration of only the content and preventing the range of recommended content from becoming smaller and smaller, thereby further improving the accuracy of the final click rate 116.

[0033] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure may also be applied to other environments with different structures and / or functions.

[0034] The following will combine Figures 2 to 8 The process according to the embodiment of the present disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or the actions shown may be omitted, and the scope of the present disclosure is not limited in this respect.

[0035] Figure 2Flowchart of method 200 for determining click-through rate of content in some embodiments of the present disclosure is shown. In some embodiments, Figure 1 The server 102 in the method 200 may serve as the execution subject of the method 200 .

[0036] In block 202, a first click rate of the content is determined based on content features and user features of the content. Figure 1 In the example environment 100 shown, the server 102 determines the probability of the corresponding content being clicked by the user, that is, the content click rate 112 (which may be referred to as the first click rate) based on the content feature 108 and the user feature 106. In some embodiments, the content includes but is not limited to articles, pictures, videos, etc., and the content feature 108 includes but is not limited to the title, logo, summary, scene, time period and other features of the content.

[0037] At block 204, a confidence interval for the first click rate is determined. Figure 1 In the example environment 100 shown, after obtaining the content click rate 112, the corresponding confidence interval 114 is determined according to the content click rate 112. In some embodiments, the confidence interval 114 is determined based on the confidence level and the number of sample contents. For example, if the confidence level is 95%, it means that there is a 95% probability that the actual value of the content click rate 112 will fall into the confidence interval 114. It can be understood that the confidence interval 114 reflects the correctable range of the content click rate 112. The fewer the number of sample contents, the lower the authenticity of the content click rate 112, and the larger the correctable range, i.e., the confidence interval 114. On the contrary, the more the number of sample contents, the higher the authenticity of the content click rate 112, and the larger the correctable range, i.e., the confidence interval 114. In some embodiments, the confidence interval 114 is a Wilson confidence interval.

[0038] In block 206, a second click rate for the category is determined based on the category to which the content belongs and the user characteristics. Figure 1 In the example environment 100 shown, the server 102 determines the probability of the content in the category being clicked by the user, that is, the category click rate 110 (which may be referred to as the second click rate) based on the category feature 104 and the user feature 106. In some embodiments, the category to which the content belongs is determined, and then multiple contents in the category are obtained and common features in the multiple contents are extracted as the category feature 104.

[0039] At block 208, based on the confidence interval and the second click rate, a click rate for the content is determined. Figure 1In the example environment 100 shown, after the server 102 determines the category click rate 110 and the confidence interval 114, it determines the final click rate 116 of the content clicked by the user based on the category click rate 110 and the confidence interval 114. In some embodiments, if the server 102 has a high exploration intention (which may be called recommendation intention) for the content, the upper limit of the confidence interval 114 may be selected, and the click rate 116 may be determined based on the upper limit value and the category click rate 104. It can be understood that by selecting the upper limit value of the confidence interval 114, the proportion of the content click rate 112 in the final click rate 116 can be increased, thereby increasing the degree of influence of the content on the final click rate 116.

[0040] In the embodiments of the present disclosure, the content click rate can be corrected through the confidence interval to avoid the problem of content click rate estimation deviation caused by too small sample data, thereby improving the accuracy of the content click rate and the final click rate. In addition, the category click rate predicted by the category is also combined to avoid exploring only the content and prevent the range of recommended content from becoming smaller and smaller, thereby further improving the accuracy of the final click rate.

[0041] In some embodiments, the UCB (upper confidence bound) algorithm can be used to solve the problem of the increasingly narrow range of recommended content. The UCB algorithm balances the exploration of new content and the use of historical content by measuring the uncertainty of the estimated benefit. The specific calculation formula is as follows:

[0042]

[0043] in, is the average revenue of historical records, N is the total number of exposures, and n is the number of exposures of the current content. In formula (1), the square root term is the exploration term for new content. The more times the new content is exposed, the lower the exploration value. It can be understood that when N increases and n remains unchanged, the proportion of new content exposure decreases, the UCB value increases, and the revenue of the next exposure of new content increases. When N remains unchanged and n increases, the proportion of new content exposure increases, the UCB value decreases, and the revenue of the next exposure of new content decreases. It can be understood that although the UCB algorithm can prevent the scope of recommended content from becoming smaller and smaller, since the exploration term is relatively simple and relies on a large number of samples, the UCB algorithm is not suitable for scenarios with a small number of samples, and is not suitable for users with low activity, users with a lot of long-tail content, and new users, etc., who have less historical data to use.

[0044] Figure 3 FIG. 3 is a schematic diagram showing a process 300 of determining a score of content according to some embodiments of the present disclosure. Figure 3In some embodiments, the process 300 involves a content prediction model (which may be referred to as a first network model) and a category prediction model (which may be referred to as a second network model), the content prediction model includes a content network layer 320, a content network layer 322, and a content network layer 324, and the category prediction model includes a category network layer 312, a category network layer 314, and a category network layer 316. It can be understood that Figure 3 The content prediction model and the category prediction model shown in both exemplarily adopt a three-layer network structure, which does not impose any restrictions on the first network model and the second network model in the present disclosure, and the network parameters of the first network model and the second network model do not need to be consistent.

[0045] Continue to refer Figure 3 , content features include but are not limited to content identification features 308, content attribute features 310, and context features 304. In some embodiments, content identification features 308, content attribute features 310, context features 304, and category features 302 are all recorded in the form of feature embedding. In some embodiments, content attribute features 310 include but are not limited to the title, text, summary, etc. of the content, and context features 304 include but are not limited to the scene (e.g., sports, cars) and time period of the content. The content prediction model processes the context features 304, user features 306, content identification features 308, and content attribute features 310 through the content network layer 320, the content network layer 322, and the content network layer 324, thereby obtaining a content prediction model score 326 (which may be referred to as the first click rate).

[0046] The category prediction model processes the content features 302, context features 304, and user features 306 through the category network layer 312, the category network layer 314, and the category network layer 316, thereby obtaining a category prediction model score 318 (which may be referred to as a second click-through rate). In some implementations, the category to which the content belongs is determined, and then multiple contents under the category are obtained (for example, if the content is a car video, the corresponding category is "car", and multiple videos under the "car" category can be obtained at this time), and then common features are extracted from the multiple contents as the category features 302, and the category prediction model score 318 is further obtained by combining the content features 302, the context features 304, and the user features 306.

[0047] After obtaining the content prediction model score 326, the corresponding Wilson confidence interval can be determined for the content prediction model score 326, thereby obtaining the Wilson confidence interval upper bound 328. In some embodiments, the number of positive sample content and negative sample content input into the content prediction model during the training process can be determined, and then the upper and lower bounds of the corresponding confidence interval are calculated according to the predetermined confidence level, the content prediction model score 326, and the number of positive sample content and negative sample content. The specific calculation formula is as follows:

[0048]

[0049] Wherein, S is the upper and lower bounds of the confidence interval ("+" and "-" correspond to the upper and lower bounds, respectively), p is the content prediction model score 326 (i.e., the first click rate), n is the number of content samples (which can be obtained by statistics during the training of the content prediction model), and z is the parameter corresponding to the confidence level. In some embodiments, the confidence level is 95%, and the corresponding parameter z is 1.96.

[0050] When the exploration intention (which may be referred to as the recommendation intention) for the current content reaches a certain level (which may be referred to as the recommendation intention meets the conditions, and at this time there is an intention to recommend the current content), the upper limit of the confidence interval (which may be referred to as the upper limit value) may be obtained. It can be understood that the upper limit of the confidence interval may be used as a correction value for the content prediction model score 326. In this way, when there is an exploration intention for the content, the content prediction model score 326 may be improved, thereby increasing the proportion of the content prediction model score 326 in the final score 330.

[0051] After obtaining the category prediction model score 318 and the upper bound of the Wilson confidence interval 328, the category prediction model score 318 is recorded as S 1 , the upper bound of the Wilson confidence interval is 328 for S 2 , determine the hyperparameters α and β (which can be called at least one parameter for the upper limit value) for the Wilson confidence interval upper bound 328, where the hyperparameters α and β are used to adjust the influence of the Wilson confidence interval upper bound 328 on the final score 330 (which can be called the click-through rate of the content). When the hyperparameters α and β meet the conditions, the Wilson confidence interval upper bound 328 is adjusted by the hyperparameters α and β, and then the sum of the adjusted Wilson confidence interval upper bound and the category prediction model score 318 is used as the final score 330 (denoted as ctr), and the calculation formula is as follows:

[0052]

[0053] In this way, the content prediction model score 326 can be corrected through the Wilson confidence interval upper bound 328 to avoid the estimation bias of the content prediction model score 326 caused by too little sample data, thereby improving the accuracy of the content prediction model score 326 and the final score 330. In addition, in the process of determining the final score 330, the category prediction model score 318 (as prior knowledge) is also combined, thereby avoiding exploration only for content and preventing the range of recommended content from becoming smaller and smaller, thereby further improving the accuracy of the final score 330.

[0054] In some embodiments, reference Figure 3 , the positive sample content and the negative sample content are used to train the category prediction model (including the category network layer 312, the category network layer 314 and the category network layer 316) and the content prediction model (including the content network layer 320, the content network layer 322 and the content network layer 324), the positive sample content includes the content that has been exposed and clicked, and the negative sample content includes all the exposed content. Alternatively or additionally, the negative sample content only includes the content that has been exposed and not clicked by the user.

[0055] In some embodiments, the square root term (i.e., the exploration term for the content) in formula (1) can be replaced by the upper bound of the Wilson confidence interval in formula (2), by Figure 3 The category prediction model in the formula (1) calculates the average return of the historical records (i.e., the category prediction model score 318), through Figure 3 The content prediction model in Calculate the exploration term in formula (1) (i.e., the upper bound of the Wilson confidence interval in formula (2)).

[0056] In this way, the UCB algorithm can be combined with the Wilson confidence interval correction to predict the click-through rate of content, so that the click-through rate of the category to which the content belongs can be predicted through the category prediction model, reflecting the user's prior knowledge of the click-through rate of the content, and the click-through rate of the content can be predicted through the content prediction model. The sum of the two click-through rates is used as the final click-through rate, so that the final click-through rate can take into account both content exploration and experience utilization. At the same time, the score of the content prediction model selects the upper bound of the Wilson confidence interval to solve the problem of deviation when the number of samples is low, and when the click-through rates of different content are the same, the less the content is exposed, the wider its Wilson confidence interval, the higher the upper bound of the Wilson confidence interval, the greater the impact on the final click-through rate, and it is beneficial to the start-up of the content.

[0057] Figure 4A schematic diagram of a process 400 for determining the upper and lower bounds of a confidence interval in some embodiments of the present disclosure is shown. In some embodiments, user features 402 and content features 404 are obtained, and the user features 402 and content features 404 are input into a content prediction model 406, thereby obtaining the probability that the content is clicked by the user, that is, the content click-through rate 410 (which may be referred to as the first click-through rate). Then, the corresponding confidence interval 414 is calculated in combination with the sample number 408 of the positive sample content and the negative sample content and the pre-set confidence level 412, thereby obtaining the upper and lower bounds of the confidence interval 414. In this way, the problem of deviation in the content click-through rate 410 due to too few samples 408 can be avoided.

[0058] Continue to refer Figure 4 , after obtaining the upper and lower bounds of the confidence interval 414, determine whether there is a content exploration intention 416 (which can be called a recommendation intention) for the content. If there is a content exploration intention 416 for the content, the content click-through rate 410 can be corrected to the upper bound 418 of the confidence interval. If there is no content exploration intention 416 for the content, the content click-through rate 410 can be corrected to the lower bound 420 of the confidence interval. It can be understood that when the upper bound 418 of the confidence interval is selected, the influence of the content click-through rate 410 on the final click-through rate can be increased, that is, the proportion of content exploration is increased, which can help the content to start. When the lower bound 420 of the confidence interval is selected, the influence of the content click-through rate 410 on the final click-through rate can be reduced, that is, the proportion of content exploration is reduced. At this time, the content start-up can be suppressed, thereby increasing the influence of category analysis on the results.

[0059] Figure 5 A schematic diagram of a coordinate axis 500 for displaying a confidence interval in some embodiments of the present disclosure is shown. In some embodiments, the first click rate is P. When the number of samples is high, the credibility of the first click rate P is high and the correction range is small. When the number of samples is low, the credibility of the first click rate P is low and the correction range is large. Therefore, the confidence interval is a correction to the credibility of the first click rate P, so as to compensate for the problem that the first click rate P is inaccurate due to the low number of samples.

[0060] In some embodiments, when the sample size of the content is low, the confidence interval ① of the first click rate P is [A1, B1] (setting the confidence level to 95%, the actual probability of the first click rate P has a 95% probability of falling within [A1, B1]). When the sample size of the content is high, the confidence interval ② of the first click rate P is [A2, B2]. It can be understood that the range of the confidence interval ② is smaller than the range of the confidence interval ①, that is, the higher the sample size, the smaller the correction range of the first click rate P.

[0061] Figure 6A schematic diagram of a process 600 for determining recommended content and sample content in some embodiments of the present disclosure is shown. In some embodiments, for multiple candidate contents 602, multiple corresponding click rates 604 can be determined respectively through the scheme of the present disclosure. Then, click rate sorting 606 is performed to sort the multiple click rates 604 in descending order, and select the click rate with the highest value (which can be called the click rate that meets the sorting condition). Further, the corresponding candidate content is determined as recommended content 608 according to the click rate with the highest value. In this way, the content that best matches the user can be quickly determined, thereby improving the accuracy of the recommendation.

[0062] Continue to refer Figure 6 , after the server determines the recommended content 608, it sends the recommended content 608 to the user device, and the user device receives the recommended content 608 and displays it on the screen (exposes it). Then, the user device determines whether the recommended content is clicked 610. If the recommended content 608 is clicked by the user, a message that the recommended content 608 can be used as a positive sample content 612 is fed back to the server. If the recommended content 608 is not clicked by the user, a message that the recommended content 608 can be used as a negative sample content 614 is fed back to the server. The positive sample content 612 and the negative sample content 614 can be used for model training when the click-through rate is determined later. In some embodiments, the negative sample content 614 can include both the recommended content that has been clicked by the user and the recommended content that has not been clicked by the user. In this way, training samples can be generated in the recommendation process, thereby continuously optimizing the recommendation process, thereby improving the accuracy of the click-through rate.

[0063] Figure 7 FIG. 7 is a block diagram of an apparatus 700 for determining a click rate of content according to some embodiments of the present disclosure. Figure 7 The device 700 includes a first click rate determination module 702, which is configured to determine a first click rate of the content based on the content characteristics and user characteristics of the content. The device 700 also includes a confidence interval determination module 704, which is configured to determine a confidence interval of the first click rate. The device 700 also includes a second click rate determination module 706, which is configured to determine a second click rate of the category based on the category to which the content belongs and the user characteristics. In addition, the device 700 also includes a click rate determination module 708, which is configured to determine the click rate of the content based on the confidence interval and the second click rate.

[0064] Figure 8 800 is a block diagram of an electronic device 800 according to some embodiments of the present disclosure. The device 800 may be a device or apparatus described in the embodiments of the present disclosure. Figure 8As shown, the device 800 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 802, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 804 or loaded from a storage unit 816 to a random access memory (RAM) 806. In the RAM 806, various programs and data required for the operation of the device 800 can also be stored. The CPU / GPU 802, the ROM 804, and the RAM 806 are connected to each other via a bus 808. An input / output (I / O) interface 810 is also connected to the bus 808. Although not shown in FIG. Figure 8 As shown in FIG. 8 , device 800 may also include a co-processor.

[0065] A number of components in the device 800 are connected to the I / O interface 810, including: an input unit 812, such as a keyboard, a mouse, etc.; an output unit 814, such as various types of displays, speakers, etc.; a storage unit 816, such as a disk, an optical disk, etc.; and a communication unit 818, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 818 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0066] The various methods or processes described above may be performed by the CPU / GPU 802. For example, in some embodiments, the methods may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 816. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 804 and / or the communication unit 818. When the computer program is loaded into the RAM 806 and executed by the CPU / GPU 802, one or more steps or actions in the methods or processes described above may be performed.

[0067] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0068] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0069] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0070] The computer program instructions for performing the disclosed operation may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, programming languages ​​including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of a computer-readable program instruction to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0071] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0072] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0073] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0074] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0075] Some example implementations of the present disclosure are listed below.

[0076] Example 1. A method for determining a click-through rate of content, comprising:

[0077] Determining a first click rate of the content based on content characteristics and user characteristics of the content;

[0078] determining a confidence interval of the first click rate;

[0079] Determining a second click rate of the category based on the category to which the content belongs and the user characteristics; and

[0080] A click rate of the content is determined based on the confidence interval and the second click rate.

[0081] Example 2. The method according to claim 1, wherein the content feature comprises at least one of a content identification feature, a content attribute feature, and a context feature, and based on the content feature and user feature of the content, determining the first click rate of the content comprises:

[0082] Acquire at least one of the content identification feature, the content attribute feature, and the context feature; and

[0083] The first click rate of the content is determined by a first network model based on the content identification feature, the content attribute feature, at least one of the context feature and the user feature.

[0084] Example 3. The method according to claim 1 or 2, wherein based on the category to which the content belongs and the user characteristics, determining the second click rate of the category comprises:

[0085] Acquire multiple contents under the category to which the contents belong;

[0086] Based on the plurality of contents, determining a category feature for the category; and

[0087] The second click rate of the category is determined by a second network model based on the category feature, the user feature, and the context feature.

[0088] Example 4. A method according to any one of claims 1-3, wherein the contextual feature includes at least one of a scene and a time period of the content.

[0089] Example 5. The method according to any one of claims 1 to 4, wherein training the first network model and the second network model comprises:

[0090] Acquire a plurality of positive sample contents and a plurality of negative sample contents, wherein the positive sample contents include contents that have been exposed and clicked, and the negative sample contents include contents that have been exposed; and

[0091] The first network model and the second network model are trained based on the multiple positive sample contents and the multiple negative sample contents.

[0092] Example 6. The method of any one of claims 1-5, wherein determining a confidence interval of the first click rate comprises:

[0093] Determine the number of the plurality of positive sample contents and the number of the plurality of negative sample contents based on the first network model;

[0094] Obtaining a confidence level corresponding to the content; and

[0095] The confidence interval is determined based on the first click rate, the number of the plurality of positive sample contents and the plurality of negative sample contents, and the confidence level.

[0096] Example 7. The method of any one of claims 1-6, wherein determining the click rate of the content based on the confidence interval and the second click rate comprises:

[0097] Determine the recommendation intent for said content;

[0098] In response to the recommendation intention satisfying a condition, obtaining an upper limit value of the confidence interval; and

[0099] The click rate of the content is determined based on the upper limit value and the second click rate.

[0100] Example 8. The method according to any one of claims 1-7, wherein determining the click rate of the content based on the upper limit value and the second click rate comprises:

[0101] Determine at least one parameter for the upper limit value, wherein the at least one parameter is used to determine the influence degree of the upper limit value on the click rate;

[0102] In response to the at least one parameter satisfying a condition, adjusting the upper limit value based on the at least one parameter; and

[0103] The click rate of the content is determined based on the adjusted upper limit value and the second click rate.

[0104] Example 9. A method according to any one of claims 1-8, wherein the confidence interval is a Wilson confidence interval.

[0105] Example 10. The method according to any one of claims 1-9, further comprising:

[0106] Determine the ranking results of multiple click-through rates of multiple contents;

[0107] Based on the ranking results of the plurality of click rates, determining a click rate that satisfies a ranking condition among the plurality of click rates; and

[0108] Content corresponding to a click rate satisfying the ranking condition is determined from among the plurality of contents.

[0109] Example 11. An apparatus for determining a click-through rate of content, comprising:

[0110] A first click rate determination module, configured to determine a first click rate of the content based on content features and user features of the content;

[0111] A confidence interval determination module, configured to determine a confidence interval of the first click rate;

[0112] A second click rate determination module is configured to determine a second click rate of the category based on the category to which the content belongs and the user characteristics; and

[0113] The click rate determination module is configured to determine the click rate of the content based on the confidence interval and the second click rate.

[0114] Example 12. The apparatus according to claim 11, wherein the content feature comprises at least one of a content identification feature, a content attribute feature, and a context feature, and the first click rate determination module is further configured to:

[0115] Acquire at least one of the content identification feature, the content attribute feature, and the context feature; and

[0116] The first click rate of the content is determined by a first network model based on the content identification feature, the content attribute feature, at least one of the context feature and the user feature.

[0117] Example 13. The apparatus according to claim 11 or 12, wherein the second click rate determination module is further configured to:

[0118] Acquire multiple contents under the category to which the contents belong;

[0119] Based on the plurality of contents, determining a category feature for the category; and

[0120] The second click rate of the category is determined by a second network model based on the category feature, the user feature, and the context feature.

[0121] Example 14. An apparatus according to any one of claims 11-13, wherein the contextual feature includes at least one of a scene and a time period of the content.

[0122] Example 15. The apparatus according to any one of claims 11-14, further comprising:

[0123] A sample content acquisition module is configured to acquire a plurality of positive sample contents and a plurality of negative sample contents, wherein the positive sample contents include exposed and clicked contents, and the negative sample contents include exposed contents; and

[0124] A training module is configured to train the first network model and the second network model based on the multiple positive sample contents and the multiple negative sample contents.

[0125] Example 16. The apparatus according to any one of claims 11-15, wherein the confidence interval determination module is further configured to:

[0126] Determine the number of the plurality of positive sample contents and the number of the plurality of negative sample contents based on the first network model;

[0127] Obtaining a confidence level corresponding to the content; and

[0128] The confidence interval is determined based on the first click rate, the number of the plurality of positive sample contents and the plurality of negative sample contents, and the confidence level.

[0129] Example 17. The apparatus according to any one of claims 11-16, wherein the click rate determination module is further configured to:

[0130] Determine the recommendation intent for said content;

[0131] In response to the recommendation intention satisfying a condition, obtaining an upper limit value of the confidence interval; and

[0132] The click rate of the content is determined based on the upper limit value and the second click rate.

[0133] Example 18. The apparatus according to any one of claims 11-17, wherein the click rate determination module is further configured to:

[0134] Determine at least one parameter for the upper limit value, wherein the at least one parameter is used to determine the influence degree of the upper limit value on the click rate;

[0135] In response to the at least one parameter satisfying a condition, adjusting the upper limit value based on the at least one parameter; and

[0136] The click rate of the content is determined based on the adjusted upper limit value and the second click rate.

[0137] Example 19. An apparatus according to any one of claims 11-18, wherein the confidence interval is a Wilson confidence interval.

[0138] Example 20. The apparatus according to any one of claims 11-19, further comprising:

[0139] A ranking result determination module is configured to determine ranking results of multiple click rates of multiple contents;

[0140] The click rate determination module is further configured to determine a click rate that satisfies a sorting condition among the multiple click rates based on the sorting results of the multiple click rates; and

[0141] The content determination module is configured to determine, from the plurality of contents, the content corresponding to the click rate that satisfies the sorting condition.

[0142] Example 21. An electronic device comprising:

[0143] Processor; and

[0144] A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device performs actions, the actions comprising:

[0145] Determining a first click rate of the content based on content characteristics and user characteristics of the content;

[0146] determining a confidence interval of the first click rate;

[0147] Determining a second click rate of the category based on the category to which the content belongs and the user characteristics; and

[0148] A click rate of the content is determined based on the confidence interval and the second click rate.

[0149] Example 22. The electronic device of claim 21, wherein the content feature comprises at least one of a content identification feature, a content attribute feature, and a context feature, and based on the content feature and the user feature of the content, determining the first click rate of the content comprises:

[0150] Acquire at least one of the content identification feature, the content attribute feature, and the context feature; and

[0151] The first click rate of the content is determined by a first network model based on the content identification feature, the content attribute feature, at least one of the context feature and the user feature.

[0152] Example 23. The electronic device of claim 21 or 22, wherein based on the category to which the content belongs and the user characteristics, determining the second click rate of the category comprises:

[0153] Acquire multiple contents under the category to which the contents belong;

[0154] Based on the plurality of contents, determining a category feature for the category; and

[0155] The second click rate of the category is determined by a second network model based on the category feature, the user feature, and the context feature.

[0156] Example 24. An electronic device according to any one of claims 21-23, wherein the contextual feature includes at least one of a scene and a time period of the content.

[0157] Example 25. The electronic device of any one of claims 21-24, wherein training the first network model and the second network model comprises:

[0158] Acquire a plurality of positive sample contents and a plurality of negative sample contents, wherein the positive sample contents include contents that have been exposed and clicked, and the negative sample contents include contents that have been exposed; and

[0159] The first network model and the second network model are trained based on the multiple positive sample contents and the multiple negative sample contents.

[0160] Example 26. The electronic device of any one of claims 21-25, wherein determining a confidence interval for the first click rate comprises:

[0161] Determine the number of the plurality of positive sample contents and the number of the plurality of negative sample contents based on the first network model;

[0162] Obtaining a confidence level corresponding to the content; and

[0163] The confidence interval is determined based on the first click rate, the number of the plurality of positive sample contents and the plurality of negative sample contents, and the confidence level.

[0164] Example 27. The electronic device of any one of claims 21-26, wherein determining the click rate of the content based on the confidence interval and the second click rate comprises:

[0165] Determine the recommendation intent for said content;

[0166] In response to the recommendation intention satisfying a condition, obtaining an upper limit value of the confidence interval; and

[0167] The click rate of the content is determined based on the upper limit value and the second click rate.

[0168] Example 28. The electronic device of any one of claims 21-27, wherein determining the click rate of the content based on the upper limit value and the second click rate comprises:

[0169] Determine at least one parameter for the upper limit value, wherein the at least one parameter is used to determine the influence degree of the upper limit value on the click rate;

[0170] In response to the at least one parameter satisfying a condition, adjusting the upper limit value based on the at least one parameter; and

[0171] The click rate of the content is determined based on the adjusted upper limit value and the second click rate.

[0172] Example 29. An electronic device according to any of claims 21-28, wherein the confidence interval is a Wilson confidence interval.

[0173] Example 30. The electronic device of any one of claims 21-29, wherein the actions further include:

[0174] Determine the ranking results of multiple click-through rates of multiple contents;

[0175] Based on the ranking results of the plurality of click rates, determining a click rate that satisfies a ranking condition among the plurality of click rates; and

[0176] Content corresponding to a click rate satisfying the ranking condition is determined from among the plurality of contents.

[0177] Example 31. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement a method according to any one of Examples 1 to 10.

[0178] Example 32. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform a method according to any one of Examples 1 to 10.

[0179] Although the disclosure has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for determining a click-through rate of content, comprising: Determining a first click rate of the content based on content characteristics and user characteristics of the content; determining a confidence interval of the first click rate; Determining a second click rate of the category based on the category to which the content belongs and the user characteristics; as well as A click rate of the content is determined based on the confidence interval and the second click rate.

2. The method according to claim 1, wherein the content feature comprises at least one of a content identification feature, a content attribute feature, and a context feature, and based on the content feature and user feature of the content, determining the first click rate of the content comprises: Acquire at least one of the content identification feature, the content attribute feature, and the context feature; as well as The first click rate of the content is determined by a first network model based on the content identification feature, the content attribute feature, at least one of the context feature and the user feature.

3. The method according to claim 2, wherein based on the category to which the content belongs and the user characteristics, determining the second click rate of the category comprises: Acquire multiple contents under the category to which the contents belong; Based on the plurality of contents, determining a category feature for the category; as well as The second click rate of the category is determined by a second network model based on the category feature, the user feature, and the context feature. The method according to claim 2 or 3, wherein the context feature comprises at least one of a scene and a time period of the content.

5. The method according to claim 3, wherein training the first network model and the second network model comprises: Acquire multiple positive sample contents and multiple negative sample contents, wherein the positive sample contents include the contents that have been exposed and clicked, and the negative sample contents include the contents that have been exposed; as well as The first network model and the second network model are trained based on the multiple positive sample contents and the multiple negative sample contents.

6. The method according to claim 5, wherein determining the confidence interval of the first click rate comprises: Determine the number of the plurality of positive sample contents and the number of the plurality of negative sample contents based on the first network model; Obtaining a confidence level corresponding to the content; as well as The confidence interval is determined based on the first click rate, the number of the plurality of positive sample contents and the plurality of negative sample contents, and the confidence level.

7. The method according to claim 1, wherein determining the click rate of the content based on the confidence interval and the second click rate comprises: Determine the recommendation intent for said content; In response to the recommendation intention satisfying a condition, obtaining an upper limit value of the confidence interval; as well as The click rate of the content is determined based on the upper limit value and the second click rate.

8. The method according to claim 7, wherein determining the click rate of the content based on the upper limit value and the second click rate comprises: Determine at least one parameter for the upper limit value, wherein the at least one parameter is used to determine the influence degree of the upper limit value on the click rate; In response to the at least one parameter satisfying a condition, adjusting the upper limit value based on the at least one parameter; as well as The click rate of the content is determined based on the adjusted upper limit value and the second click rate.

9. The method of claim 1, wherein the confidence interval is a Wilson confidence interval.

10. The method according to claim 1, further comprising: Determine the ranking results of multiple click-through rates of multiple contents; Based on the ranking results of the multiple click rates, determining a click rate that meets a ranking condition among the multiple click rates; as well as Content corresponding to a click rate satisfying the ranking condition is determined from among the plurality of contents.

11. A device for determining a click rate of content, comprising: A first click rate determination module, configured to determine a first click rate of the content based on content features and user features of the content; A confidence interval determination module, configured to determine a confidence interval of the first click rate; A second click rate determination module is configured to determine a second click rate of the category based on the category to which the content belongs and the user characteristics; as well as The click rate determination module is configured to determine the click rate of the content based on the confidence interval and the second click rate.

12. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 10.

Images