Promotion information recall method and device, computer device and storage medium

Abstract

The application relates to a promotion information recall method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining candidate promotion information output by at least two recall branches in response to a target request; determining an expected total amount of recalled promotion information based on the current quality score of each candidate promotion information; for each candidate promotion information, determining the quality score threshold corresponding to the recall branch of the candidate promotion information based on the current consumption data of the candidate promotion information and the historical quality score when the candidate promotion information was recalled by the corresponding recall branch; screening target promotion information in the corresponding recall branch according to the current quality score of the candidate promotion information and the quality score threshold; and when the total amount of the target promotion information screened from the at least two recall branches meets the expected total amount, the screened target promotion information is taken as a recall result in response to the target request. The method can improve the recall effect of promotion information.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, and storage medium for recalling promotional information. Background Technology

[0002] With the rapid development of computer and internet technologies, promotional information is ubiquitous. From the perspective of the target audience, how to provide suitable users for the promotional information to achieve its promotional goals; from the perspective of the promotional platform, how to increase the exposure value of the promotional information; and from the user's perspective, how to push appropriate promotional information to enhance the user experience, these are all issues that information promotion needs to comprehensively consider.

[0003] Information promotion generally consists of two stages: recall and ranking. Recall refers to filtering a candidate set of promotional information that users may be interested in from a massive collection of promotional information. Ranking refers to further sorting this candidate set according to its relevance to the user and pushing the most suitable candidate to the user.

[0004] It is evident that the purpose of recall is to reduce the number of candidates participating in the ranking, facilitating subsequent accurate ranking. Currently, the common approach to recall is to identify as many highly relevant candidates as possible from a massive set of promotional information, considering multiple dimensions. However, due to the large number of recalled results, this method requires random truncation before ranking, resulting in the discarding of some high-quality promotional information while retaining some lower-quality information, leading to poor recall effectiveness. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer equipment, and storage medium for recalling promotional information that can improve the recall effect of promotional information, in response to the above-mentioned technical problems.

[0006] A method for recalling promotional information, the method comprising:

[0007] Obtain candidate promotion information output by responding to the target request through at least two recall branches;

[0008] Based on the quality score of each candidate promotional message, the expected total number of promotional messages to be recalled is determined;

[0009] Determine the quality score threshold for each candidate promotion information in the corresponding recall branch, and dynamically adjust the quality score threshold based on the consumption data of the candidate promotion information;

[0010] Based on the quality score of the candidate promotion information and the quality score threshold corresponding to the candidate promotion information in the corresponding recall branch, the target promotion information is filtered out in the corresponding recall branch;

[0011] When the total amount of target promotion information filtered from the at least two recall branches meets the expected total amount, the filtered target promotion information is used as recall information in response to the target request.

[0012] A device for recalling promotional information, the device comprising:

[0013] The candidate promotion information acquisition module is used to acquire candidate promotion information output by responding to the target request through at least two recall branches.

[0014] The recall total quantity determination module is used to determine the expected total quantity of recall promotion information based on the current quality score of each candidate promotion information;

[0015] The quality score threshold determination module is used to determine the quality score threshold of each candidate promotion information in the corresponding recall branch based on the current consumption data of the candidate promotion information and the historical quality score when it was recalled by the corresponding recall branch.

[0016] The target promotion information filtering module is used to filter out target promotion information in the corresponding recall branch according to the current quality score of the candidate promotion information and the quality score threshold of the candidate promotion information in the corresponding recall branch;

[0017] The recall response module is used to use the selected target promotion information as the recall result in response to the target request when the total amount of target promotion information filtered from the at least two recall branches meets the expected total amount.

[0018] In one embodiment, the candidate promotion information acquisition module is further configured to extract the user identifier carried in the target request; acquire the user's historical behavior data corresponding to the user identifier; and determine the promotion information matching the user's historical behavior data from the promotion information set through a pre-trained neural network model, as the candidate promotion information output by the neural network-based recall branch.

[0019] In one embodiment, the candidate promotion information acquisition module is further configured to extract the user identifier carried in the target request; acquire the user tag data corresponding to the user identifier; match the user tag data with the promotion information tag data according to a preset matching strategy; and obtain the candidate promotion information output by the recall branch based on the matching strategy from the promotion information set according to the matching result.

[0020] In one embodiment, the recall total determination module is further configured to: determine the current quality score for each candidate promotional information; obtain the historical average quality score corresponding to the historical recall promotional information of each recall branch; obtain the total demand for promotional information during the sorting stage; and determine the range of the expected total amount of promotional information to be recalled from the candidate promotional information based on the current quality score of each candidate promotional information, the historical average quality score, the total demand, and the preset margin control ratio.

[0021] In one embodiment, the apparatus further includes: a quality score calculation module, configured to predict the click-through rate of the candidate promotional information to obtain the current estimated click-through rate; predict the conversion rate of the candidate promotional information to obtain the current estimated conversion rate; obtain the target bid corresponding to the candidate promotional information; and calculate the current quality score of the candidate promotional information based on the target bid corresponding to the candidate promotional information, the current estimated click-through rate, and the current estimated conversion rate.

[0022] In one embodiment, the apparatus further includes: a quality score threshold index table calculation module, used to obtain the historical quality score of each candidate promotion information when it was recalled by the corresponding recall branch; sort the historical quality scores and divide them into preset equal parts; wherein each equal part corresponds to a quantile; take the maximum value of the historical quality score in each equal part as the quantile quality score corresponding to each quantile; and obtain the quality score threshold index table of the candidate promotion information in the corresponding recall branch according to each quantile and the corresponding quantile quality score.

[0023] In one embodiment, the quality score threshold determination module is further configured to obtain the current consumption data of the candidate promotion information, the consumption data including the total consumption of the candidate promotion information on the current day, the consumption generated by the candidate promotion information on the corresponding recall branch on the current day, the total effect deviation of the candidate promotion information on the current day, and the effect deviation of the candidate promotion information on the corresponding recall branch on the current day; obtain the percentile corresponding to the quality score threshold of the previously updated promotion information; adjust the percentile corresponding to the quality score threshold of the previously updated promotion information according to the current consumption data of the candidate promotion information, and use the percentile quality score corresponding to the adjusted percentile in the quality score threshold index table corresponding to the corresponding recall branch as the quality score threshold of the candidate promotion information in the current update corresponding to the corresponding recall branch; wherein, the initial quality score threshold of the promotion information in the corresponding recall branch is the percentile quality score corresponding to the preset target percentile in the quality score threshold index table.

[0024] In one embodiment, the quality score threshold determination module is further configured to determine an adjustment coefficient and an adjustment step size for adjusting the percentile; determine the current value of the adjustment coefficient based on the current consumption data of the candidate promotion information; when the adjustment coefficient is non-zero, increase or decrease the percentile corresponding to the quality score threshold of the previously updated promotion information by the adjustment step size; when the adjustment coefficient is zero, use the quality score threshold of the previously updated promotion information as the current quality score threshold of the candidate promotion information.

[0025] In one example, the device further includes a recall total control module, configured to iteratively execute the following steps when the total amount of target promotion information filtered from the at least two recall branches does not meet the expected total amount: adjusting the quality score threshold of each candidate promotion information in the corresponding recall branch, and then filtering target promotion information from the corresponding recall branch according to the current quality score of the candidate promotion information and the adjusted quality score threshold of the candidate promotion information in the corresponding recall branch, until the total amount of target promotion information filtered from the at least two recall branches meets the expected total amount, at which point the iteration stops.

[0026] In one example, the total recall control module is further configured to: determine the percentile corresponding to the quality score threshold obtained after the previous adjustment for each candidate promotion information; adjust the percentile corresponding to the quality score threshold to obtain the percentile for the current time; query the quality score threshold index table corresponding to the candidate promotion information in the corresponding recall branch, and use the quality score of the percentile corresponding to the current time as the quality score threshold of the candidate promotion information in the corresponding recall branch; for each recall branch, randomly select from the output candidate promotion information according to the target ratio; for each randomly selected candidate promotion information in the recall branch, retain the candidate promotion information with a quality score greater than the quality score threshold for the current time, and obtain the target promotion information selected from the corresponding recall branch.

[0027] In one example, the device further includes a sorting module and a push module. The sorting module is used to perform a sorting operation on the target promotion information in the recall result; determine the information to be pushed from the target promotion information based on the sorting result; and the push module is used to send a target response carrying the information to be pushed to the terminal that triggered the target request.

[0028] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0029] Obtain candidate promotion information output by responding to the target request through at least two recall branches;

[0030] Based on the current quality score of each candidate promotional message, determine the expected total number of promotional messages to be recalled;

[0031] For each candidate promotion information, a quality score threshold corresponding to the corresponding recall branch is determined based on the current consumption data of the candidate promotion information and the historical quality score when it was recalled by the corresponding recall branch.

[0032] Based on the current quality score of the candidate promotion information and the quality score threshold corresponding to the candidate promotion information in the corresponding recall branch, the target promotion information is filtered out in the corresponding recall branch;

[0033] When the total amount of target promotion information filtered from the at least two recall branches meets the expected total amount, the filtered target promotion information is used as the recall result in response to the target request.

[0034] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0035] Obtain candidate promotion information output by responding to the target request through at least two recall branches;

[0036] Based on the current quality score of each candidate promotional message, determine the expected total number of promotional messages to be recalled;

[0037] For each candidate promotion information, a quality score threshold corresponding to the corresponding recall branch is determined based on the current consumption data of the candidate promotion information and the historical quality score when it was recalled by the corresponding recall branch.

[0038] Based on the current quality score of the candidate promotion information and the quality score threshold corresponding to the candidate promotion information in the corresponding recall branch, the target promotion information is filtered out in the corresponding recall branch;

[0039] When the total amount of target promotion information filtered from the at least two recall branches meets the expected total amount, the filtered target promotion information is used as the recall result in response to the target request.

[0040] A computer program includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the steps of the above-described promotional information recall method.

[0041] A computer program product includes a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of the above-mentioned promotional information recall method.

[0042] The aforementioned method, apparatus, computer equipment, and storage medium for recalling promotional information, when faced with candidate promotional information output from multiple recall branches, on the one hand, from the perspective of individual promotional information, utilize the current quality score of each candidate promotional information to control the expected total amount of promotional information to be recalled. This allows for more exposure opportunities for high-quality promotional information when the overall quality level of candidate promotional information output from each recall branch is relatively high. On the other hand, a corresponding quality score threshold is determined for each candidate promotional information on the corresponding recall branch. The retention of a candidate promotional information is determined based on the relationship between its quality score and the quality score threshold. Since the quality score threshold corresponding to each candidate promotional information is dynamically adjusted based on its current consumption data and historical quality score, this effectively controls the total amount of promotional information ultimately recalled while also providing more exposure opportunities for high-quality promotional information in the recall branches. Subsequently, when the total amount of target promotional information selected from multiple recall branches meets the aforementioned expected total amount, the selected target promotional information is used as the recall result of the recall phase. It can not only flexibly adjust the number of promotional messages selected from each recall branch, but also achieve the goal of recalling high-quality promotional messages as much as possible, thereby improving the recall effect of promotional messages. Attached Figure Description

[0043] Figure 1 This is a diagram illustrating the application environment of a promotional information recall method in one embodiment.

[0044] Figure 2 This is a schematic diagram illustrating the integration of candidate promotion information from multiple recall branches in related technologies.

[0045] Figure 3 This is a schematic diagram of the overall recall process in the relevant technology.

[0046] Figure 4 This is a flowchart illustrating a promotional information recall method in one embodiment;

[0047] Figure 5 This is a flowchart illustrating the process of determining the expected total number of recalled promotional information based on the quality score of each candidate promotional information in one embodiment.

[0048] Figure 6 This is a schematic diagram of a promotional information recall method in one embodiment;

[0049] Figure 7 This is a flowchart illustrating the process of determining the quality score threshold corresponding to the relevant recall branch for candidate promotion information in one embodiment.

[0050] Figure 8 This is a flowchart illustrating the process of adjusting the percentile corresponding to the quality score threshold of promotional information based on the current consumption data of candidate promotional information in one embodiment.

[0051] Figure 9 This is a schematic diagram of the framework of a promotional information recall method in one embodiment;

[0052] Figure 10 This is a flowchart illustrating a promotional information recall method in a specific embodiment.

[0053] Figure 11 This is a structural block diagram of a promotional information recall device in one embodiment;

[0054] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] The promotional information recall method provided in this application embodiment may involve artificial intelligence technology. For example, artificial intelligence technology can be used to establish a recall branch based on a neural network model, or artificial intelligence technology can be used to establish a recall branch based on a matching strategy.

[0057] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0058] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0059] The promotional information recall method provided in this application embodiment can also involve cloud technology. For example, the data processing and data computing involved in the embodiment can be implemented based on cloud computing. Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, the resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go.

[0060] Before providing a detailed description of the embodiments of this application, the terms and concepts involved in the embodiments of this application will be explained below for the purpose of describing the embodiments of this application, rather than for the purpose of limiting this application.

[0061] Original targeting: The targeting conditions or combinations of targeting conditions set independently by the advertiser for their promotional information. For example, the age of the target audience for the promotional information: under 55 years old.

[0062] Unbreakable targeting: Unignorable targeting refers to targeting conditions that must be met by the advertiser of the promotional information, regardless of the circumstances.

[0063] Intelligent targeting, also known as intelligent recall, aims to solve the problems of traditional targeting (demographic attributes, device attributes, geographic location, behavioral interests, etc.) which rely on a lot of prior knowledge from humans, are difficult to manually optimize, and have high trial and error costs. It can help advertisers automatically target the most suitable audience and improve the efficiency of advertising delivery.

[0064] ANN: Artificial Neural Network, refers to a complex network structure formed by a large number of interconnected processing units. It is an abstraction, simplification and simulation of the human brain's organizational structure and operating mechanism.

[0065] ANN Recall Branch: A method for recalling promotional information based on a neural network model. It outputs a batch of candidate promotional information that matches a user's request from a massive amount of promotional information by establishing a neural network model.

[0066] TAG Recall Branch: A method for recalling promotional information based on a matching strategy. It tags promotional information and users by configuring and mining tags, and then outputs a batch of candidate promotional information that matches a user request by matching the tag information.

[0067] Concentration problem: also known as the head problem, it is mainly reflected in the fact that different users tend to recall the same small set of promotional information, resulting in other promotional information not having an exposure opportunity.

[0068] The promotional information recall method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 can initiate a current request to server 104 based on the user's exposure behavior. After receiving the target request, server 104 obtains the candidate promotion information output in response to the target request through at least two recall branches; determines the expected total amount of recall promotion information based on the current quality score of each candidate promotion information; for each candidate promotion information, determines the quality score threshold corresponding to the corresponding recall branch based on the current consumption data of the candidate promotion information and the historical quality score when the candidate promotion information was recalled by the corresponding recall branch; filters target promotion information in the corresponding recall branch according to the current quality score of the candidate promotion information and the quality score threshold corresponding to the candidate promotion information in the corresponding recall branch; when the total amount of target promotion information filtered from at least two recall branches meets the expected total amount, the filtered target promotion information is used as the recall result in response to the target request.

[0069] Optionally, as the recall result of the recall phase, the server 104 can further perform a sorting operation on the target promotion information in the recall result; determine the information to be pushed from the target promotion information based on the sorting result; and send a target response carrying the information to be pushed to the terminal that triggered the target request. After receiving the target response, the terminal 102 extracts the promotion information therein and displays the promotion information in the user interface.

[0070] The terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, smart TV, etc., but is not limited to these. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal 102 and the server 104 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0071] In related technologies, when faced with candidate promotion information output from multiple recall branches, the usual approach is to directly merge the candidate promotion information output from these multiple recall branches. However, due to the large number of merged information, the merging result is randomly truncated according to the target number of promotion information to be recalled, and this truncated result is used as the recall result in the recall stage.

[0072] like Figure 2The diagram shown illustrates the integration of candidate promotion information from multiple recall branches in related technologies. (Refer to...) Figure 2 P1 is the recall result output by the ANN recall branch, P2 is the recall result output by the TAG recall branch, and A and B are the original targeting conditions set by the advertiser to which the promotional information belongs. Figure 2 The summation shown is equivalent to taking the union of the results of P1 and P2, and then intersecting it with the original orientations A and B.

[0073] like Figure 3 The diagram shown illustrates the overall recall process in related technologies. (Refer to...) Figure 3 The candidate promotion information output by recall branch A, recall branch B, recall branch C, and recall branch D are directly merged to obtain a total set of candidate promotion information with a total amount of num. Then, the candidate promotion information set is randomly truncated according to the cut rate cut_rate to obtain the recall results output in the recall stage with a total amount of num_cut.

[0074] Figure 2 and Figure 3 The fusion method shown appears comprehensive at first glance, recalling candidate promotional information from various angles and performing a union fusion of the candidate promotional information without loss. However, this process has a significant problem: the number of recalled candidate promotional information is too large. Therefore, before entering the ranking stage, the recalled candidate promotional information needs to be truncated to a certain extent. For example, if a total of 4000 ads are recalled, only 2000 need to enter the next round of ranking. This requires randomly selecting 2000 ads from the 4000 ads. This truncating process lacks a ranking basis, leading to some high-quality ads being discarded after truncating, while some low-quality ads are retained, resulting in poor ad recall performance.

[0075] The promotion information recall method provided in this application proposes a novel recall result fusion strategy when faced with candidate promotion information output from multiple recall branches. This strategy selectively adopts the recall results from each recall branch, retaining high-quality promotion information and eliminating low-quality information. During the recall phase, the expected total amount of recalled promotion information is controlled, ensuring that higher-quality promotion information is recalled while some relatively lower-quality information is eliminated at this stage. This targeted approach allows more high-quality promotion information to enter the recall results. Consequently, during subsequent truncation, the probability of high-quality promotion information being eliminated while low-quality information is retained is lower, thus better ensuring the effectiveness of promotion information recall and improving the overall performance of the promotion information recommendation system.

[0076] On the one hand, from the perspective of individual promotional information, the expected total amount of recalled promotional information is controlled by using the current quality score of each candidate promotional information. This allows for more exposure opportunities for high-quality promotional information when the overall quality level of candidate promotional information output from each recall branch is relatively high. On the other hand, a corresponding quality score threshold is determined for each candidate promotional information on the relevant recall branch. The retention of a candidate promotional information is determined based on the relationship between its quality score and this threshold. Since the quality score threshold for each candidate promotional information is dynamically adjusted based on its current consumption data and historical quality score, this effectively controls the total amount of promotional information ultimately recalled while also providing more exposure opportunities for high-quality promotional information in the recall branches. Subsequently, when the total amount of target promotional information selected from multiple recall branches meets the aforementioned expected total amount, the selected target promotional information is used as the recall result in the recall phase. This approach not only allows for flexible adjustment of the number of promotional information selected from each recall branch but also aims to maximize the recall of high-quality promotional information, thereby improving the effectiveness of promotional information recall.

[0077] In one embodiment, such as Figure 4 As shown, a method for recalling promotional information is provided, which can be applied to... Figure 1 The following steps are used as an example of computer equipment (terminal 102 or server 104):

[0078] Step 402: Obtain candidate promotion information output by responding to the target request through at least two recall branches.

[0079] Promotional information refers to the online medium through which the promoted object is disseminated. For example, promotional information can be advertising content. Promotional information can be text, images, keywords, or videos, or combinations thereof. Computer devices can use text descriptions, image displays, or video presentations to showcase the promoted object to users. The promoted object can be, for example, an application, product, or service provided by the advertiser.

[0080] Candidate promotion information is a collection of promotional information that matches a specific target request. The output candidate promotion information may differ based on target requests triggered by different user identifiers, as well as target requests triggered by the same user identifier at different times.

[0081] Recall pathways are models used to retrieve candidate promotional information from a massive amount of promotional information. Different recall pathways employ different modeling methods. Recall pathways can generally be divided into two main categories: ANN recall pathways and TAG recall pathways. ANN recall pathways are based on neural network-based recall models. They output a batch of candidate promotional information that matches a specific target request from a massive amount of promotional information by building a neural network model. TAG recall pathways are based on matching strategies. They label promotional information and users by configuring targeted tags and mining tags, and then output a batch of candidate promotional information that matches a specific target request through tag matching.

[0082] In one embodiment, after detecting a user's exposure behavior, the terminal can trigger a target request carrying the user's user identifier and send the target request to the server. Upon receiving the target request, the server can retrieve candidate promotion information corresponding to the user identifier through at least two different recall branches. Optionally, in addition to the user identifier, the terminal can also generate a corresponding target request based on data such as the user's current location information and the user interaction interface where the user's exposure behavior occurred.

[0083] User exposure behaviors can include actions such as logging in to download an application (App) or opening the App; user exposure behaviors can also include browsing the social circles of a social application; opening web pages of a portal website; entering search keywords in a search engine, and so on.

[0084] A user identifier is a string of characters used to uniquely identify a user. For example, it could be a user's mobile phone number or user account. In one embodiment, the user identifier can also be a terminal identifier corresponding to any terminal. For example, for a mobile terminal, the user identifier could be the mobile terminal's IMEI (International Mobile Equipment Identity).

[0085] In one embodiment, there are at least two recall branches, which may include one ANN recall branch and multiple TAG recall branches. Each of the multiple TAG recall branches corresponds to a different matching strategy. For example, TAG recall branch 1 is implemented based on a behavioral interest-based matching strategy. The computer device uses this branch to identify the user's intentions and needs based on their behavior, including but not limited to the webpage content the user browsed, the advertisements they clicked, and the categories of installed apps, and determines the content the user is interested in, outputting candidate promotional information matching the user's interests. TAG recall branch 2 is implemented based on an app behavior-based matching strategy. The computer device uses this branch to output corresponding candidate promotional information based on the apps the user has installed, the categories of the installed apps, and the user's behavior on those apps. TAG Recall Path 3, based on a custom audience matching strategy, categorizes users, such as common audience packages including e-commerce audience packages, student audience packages, beauty audience packages, etc. Advertisers can select a certain audience for targeted advertising. Computer devices use this TAG Recall Path 3 to find and output candidate promotional information that matches the current user's audience package.

[0086] In one embodiment, at least two recall branches include neural network-based recall branches, and the method further includes: extracting user identifiers carried in the target request; obtaining user historical behavior data corresponding to the user identifiers; and determining promotional information matching the user historical behavior data from the promotional information set through a pre-trained neural network model, as candidate promotional information output by the neural network-based recall branch.

[0087] The pre-trained neural network model can be a multi-dimensional, end-to-end model capable of representing users as user feature vectors and promotional information as promotional information feature vectors. The computer device can use this pre-trained neural network model to obtain user feature vectors (userembeddings) based on input user historical behavior data. The computer device can also obtain the promotional information feature vectors (ad embeddings) corresponding to each promotional information in the set of promotional information output by the pre-trained neural network model, calculate the distance between the user embedding and each ad embedding, filter them according to a similarity threshold, and output candidate promotional information.

[0088] When using the trained model for online prediction, the similarity threshold can be adjusted based on various promotional effectiveness evaluation data, thereby controlling the delivery of promotional information to users. This similarity threshold can be adjusted in real time; for example, it can be adjusted promptly based on the consumption and cost of promotional information to control whether or not promotional information is delivered to the targeted audience.

[0089] During the training phase of this model, predefined training objectives can include CTR (Click-Through-Rate), CVR (Conversion Rate), CTCVR (Probability of Click-Through-Rate), and so on. During training, the model is guided to learn and predict these metrics for each sample, enabling it to learn the ability to extract promotional information and user feature vectors. This neural network model can be based on a dual-tower structure (DSSM, DeepStructured Semantic Models), also known as a deep semantic matching model. The DSSM model uses two relatively independent complex networks to construct user-related embeddings and promotional information-related embeddings, hence the name dual-tower model. Furthermore, to alleviate the problem of concentrated promotional information recall during training, the computer device can introduce contextual information in the user-related network, using context-aware user embeddings instead of traditional user embeddings. This introduces richer user and promotional features, optimizing the model.

[0090] In one embodiment, at least two recall branches include recall branches based on a matching strategy, and the method further includes: extracting the user identifier carried in the target request; obtaining user tag data corresponding to the user identifier; matching the user tag data with promotional information tag data according to a preset matching strategy; and obtaining candidate promotional information output by the recall branch based on the matching strategy from the promotional information set according to the matching result.

[0091] The essence of the recall branch based on matching strategy is to select targeting tags from the promoters of the simulated promotional information. Each promotional information has a promotional information profile, and each user has a user profile. The recall branch based on matching strategy matches each user with suitable promotional information by formulating a matching strategy between the two.

[0092] There can be multiple recall branches based on matching strategies. Each recall branch uses a different matching strategy, and each different matching strategy corresponds to a recall branch. In this way, by formulating matching strategies from different perspectives, the richness of the recall promotion information can be increased as much as possible.

[0093] For each recall branch based on a matching strategy, the computer needs to obtain user tag data corresponding to the user identifier in the target request and the matching strategy. This data could be data representing user behavioral interests, user app behavior, or the user's demographic group. The computer then matches the obtained user tag data with the promotional information tag data according to the matching strategy for that recall branch, filtering candidate promotional information based on the matching results. For example, if the obtained user behavioral interests indicate that the user strongly likes games, and a certain promotional information is for advertising game products, then the matching result will indicate a high match between the user and the promotional information, and that promotional information will be used as a candidate promotional information output by that recall branch. It can be understood that recall branches based on matching strategies can solve the problems of concentration in promotional information recall and the cold start problem for new promotional information.

[0094] Step 404: Determine the expected total number of recalled promotional information based on the current quality score of each candidate promotional information.

[0095] The quality score is used to quantify the quality of promotional information. It can be understood that the quality score of each promotional information is determined based on its promotional effect after it is delivered to users. Since delivery is a continuous process, the quality score of each promotional information is constantly changing. The computer device can re-determine the current quality score for each candidate promotional information output by the above at least two recall branches after each target request is obtained.

[0096] The expected total (target_num) is the expected value of the total amount of promotional information recalled after fusing the candidate promotional information of each recall branch through the advertising recall method provided in this application embodiment. The expected total is the result of flexible adjustment, appropriately controlling the total amount after fusion of each branch to be around a certain level, neither too much nor too little. This can avoid the problem of too much promotional information recalled after fusion, which would lead to too much promotional information needing to be directly truncated during the sorting stage, causing high-quality promotional information to be mistakenly damaged, and can also avoid the problem of insufficient diversity of recalled promotional information.

[0097] Specifically, the computer device determines the expected total number of recalled promotional messages based on the current quality score of the candidate promotional messages output from each recall branch. This is to control the total number of recalled promotional messages by utilizing the quality score of each promotional message itself. The current quality score of each candidate promotional message reflects the overall quality of the candidate promotional messages output from each recall branch. If the quality of the candidate promotional messages in each branch is currently very good, more will be recalled accordingly, meaning the expected total number will be larger. This allows for the recall of high-quality promotional messages, increasing and balancing the exposure opportunities of high-quality promotional messages. Conversely, if the quality of the candidate promotional messages in each branch is currently average, fewer will be recalled accordingly, meaning the expected total number will be smaller.

[0098] It should be noted that, compared to the approach of treating the candidate promotional information output by the recall branch as a similar group, using the average quality of the candidate promotional information output by the same recall branch as the quality score of that recall branch, and recalling more promotional information from recall branches with high quality scores—a method that focuses on the recall branch—this embodiment focuses on the individual promotional information, flexibly adjusting the number of promotional information recalled in each recall branch to maximize the selection of high-quality promotional information. It utilizes the characteristics of the promotional information itself to control the total amount of recalled promotional information. This avoids two problems with the above methods: 1) the overly strict assumption of similarity among promotional information within the same recall branch, as differences between promotional information within the same recall branch are obvious; 2) the ineffective utilization of the characteristics of the promotional information itself, because controlling the number of recalled promotional information using the quality score of the recall branch cannot effectively reflect the characteristics of each promotional information.

[0099] To control the probability of a promotional message winning from a certain recall route based on its historical performance, and to flexibly adjust the number of promotional messages winning from each recall route in order to maximize the success of high-quality promotional messages, it is necessary to consider how to determine the expected total number of promotional messages to be recalled from the candidate promotional messages output from each recall route.

[0100] Optionally, the expected total amount of promotional information recalled from each recall branch is positively correlated with the amount of promotional information needed in subsequent processes. Therefore, the more promotional information needed in subsequent processes, the higher the expected total amount of promotional information to be recalled from each recall branch. Optionally, the expected total amount of promotional information recalled from each recall branch is related to the remaining proportion of recalled promotional information. Maintaining a certain remaining proportion can preserve the diversity of promotional information recall and improve the recall effect. This remaining proportion can be controlled within a certain range, thereby keeping the expected total amount within a certain range. Optionally, the expected total amount of promotional information recalled from each recall branch is related to the overall quality level of the candidate promotional information output by each recall branch. If the quality of the candidate promotional information in each recall branch is good, more will be recalled accordingly, so that as many high-quality promotional information as possible can be recalled.

[0101] In one embodiment, the computer device can adjust the expected total amount of recall information output during the recall stage based on various factors such as the amount of promotional information required in the subsequent sorting stage, the current quality score of each candidate promotional information output by each recall branch, and the historical quality score corresponding to the candidate promotional information in each recall branch.

[0102] like Figure 5 The diagram shown illustrates the steps in one embodiment for determining the expected total number of recalled promotional messages based on the quality score of each candidate promotional message. (Refer to...) Figure 5 ,include:

[0103] Step 502: For each candidate promotional message, determine the current quality score;

[0104] Step 504: Obtain the average historical quality score corresponding to the historical recall promotion information of each recall branch;

[0105] Step 506: Obtain the total demand for promotional information during the ranking stage;

[0106] Step 508: Based on the current quality score, historical average quality score, total demand, and preset margin control ratio of each candidate promotional information, determine the range of the expected total amount of promotional information to be recalled from the candidate promotional information.

[0107] The historical quality score average corresponding to the historical recall promotional information of each recall branch can be the average quality score of each recall branch when the recalled promotional information was recalled on that day. In one embodiment, the computer device can obtain the promotional information recalled by each recall branch each time it responds to different target requests on that day, record the quality score when the promotional information was recalled, and calculate the average quality score of these recalled promotional information as the historical quality score average.

[0108] In one embodiment, the expected total amount of promotional information recalled from each recall branch can be calculated using the following formula:

[0109]

[0110] Where num represents the total timeframe for recalling promotional information across all recall branches in at least two recall branches. num_cut represents the total demand, such as the total demand in the subsequent ranking phase. M represents the total number of candidate promotional messages output by each recall branch in response to the target request. j This represents the current quality score of the j-th candidate promotional message among M candidate promotional messages. `history_average_quality` represents the average historical quality score of the historical promotional messages recalled through each recall route. `per` represents the preset margin control ratio, which can be set to 10% to 30%. After other parameters are determined, the expected total number of promotional messages recalled from the candidate promotional messages can be calculated based on the minimum and maximum values ​​of `per`. If the final number of target promotional messages recalled from each recall route falls within this range, it is considered reasonable and acceptable.

[0111] For example, there are four recall branches: ANN recall branch, TAG recall branch 1, TAG recall branch 2, and TAG recall branch 3. The number of candidate promotion information output by them are N1, N2, N3, and N4, respectively, and the sum is M. The computer device calculates the average quality score of the current promotion information of the M promotion information. Furthermore, at the time the target request was received on that day, the quality scores of the promotional information recalled by the ANN recall branch were, in order, ann_q1, ann_q2, ann_q3…; the quality scores of the promotional information recalled by the TAG recall branch 1 were, in order, tag1_q1, tag1_q2, tag1_q3…; the quality scores of the promotional information recalled by the TAG recall branch 2 were, in order, tag2_q1, tag2_q2, tag2_q3…; and the quality scores of the promotional information recalled by the TAG recall branch 3 were, in order, tag3_q1, tag3_q2, tag3_q3…. The computer device calculated the historical average quality score of these historically recalled promotional information as a reference for the quality level of the historically recalled promotional information.

[0112] In the above formula, the computer device uses the ratio of the average current quality score of M promotional information to the average historical quality score as a measure of the overall quality level of the candidate promotional information output by each recall branch. The larger the ratio, the more outstanding the quality of this batch of candidate promotional information is compared with the historical recall. In this case, the expected total amount of recall for this batch of candidate promotional information can be appropriately increased to give more high-quality promotional information more exposure opportunities. Conversely, the smaller the ratio, the more average the quality of this batch of candidate promotional information is compared with the historical recall. In this case, the expected total amount of recall needs to be appropriately reduced.

[0113] To control the probability of a promotional message winning from a certain recall route based on its historical performance, and to flexibly adjust the number of promotional messages winning from each recall route in order to maximize the chances of high-quality promotional messages winning, it is also necessary to consider how to determine the current quality score of the candidate promotional messages output from each recall route.

[0114] In one embodiment, the method further includes: predicting the click-through rate of candidate promotional information to obtain the current estimated click-through rate; predicting the conversion rate of candidate promotional information to obtain the current estimated conversion rate; obtaining the target bid corresponding to the candidate promotional information; and calculating the current quality score of candidate promotional information based on the target bid corresponding to the candidate promotional information, the current estimated click-through rate, and the current estimated conversion rate.

[0115] Click-through rate (CTR) prediction is the process of predicting whether a user will click on a promotional message after it has been exposed to them. The predicted CTR represents the probability that the promotional message will be clicked by a user. It can be understood that the higher the predicted CTR, the greater the match between the promotional message and the user. Therefore, it can be used as one of the factors that affect the current quality score of the promotional message.

[0116] Conversion rate prediction forecasts whether a user will click on an advertisement that has been exposed to them, leading to a conversion. The relationship between click-through rate (CTR) and conversion rate can be understood as follows: the probability that a user sees or clicks on an advertisement is the CTR; the probability that a user will take a corresponding conversion action based on that advertisement is the conversion rate. For example, after an app advertisement is pushed to a user, the probability that the user clicks on the app advertisement is the CTR; the probability that the user downloads the app and performs actions such as activation, registration, or payment within the app is the conversion rate. In short, even if a candidate advertisement has a low CTR, once a user clicks on it, the probability of the user making a purchase (i.e., taking a conversion action) may be very high.

[0117] In one embodiment, the current quality score of the candidate promotion information can be calculated using the following formula:

[0118] quality=target_cpa*pctr*pcvr;

[0119] Here, quality represents the current quality score of the candidate ad message, target_cpa represents the advertiser's target bid for its candidate ad message, such as the cost per thousand clicks, pctr represents the current estimated click-through rate of the candidate ad message, and pcvr represents the current estimated conversion rate of the candidate ad message.

[0120] In one embodiment, the computer device can use a pre-trained click-through rate (CTR) prediction model to predict the probability of the promotional information being exposed to the user and clicked by the user, based on user profile data corresponding to the user initiating the target request, promotional information profile data corresponding to the promotional information, and context information when the current request was initiated. This yields the current estimated CTR of the promotional information. In this manner, for each candidate information output by each recall branch, the computer device performs CTR prediction to obtain the corresponding estimated CTR.

[0121] This pre-trained click-through rate (CTR) prediction model can be a machine learning model. The computer uses pre-processed business feature data and trains the machine learning model with initial weight parameters under a predefined CTR target to obtain the pre-trained machine learning model. The machine learning model can be implemented based on feedforward neural networks, convolutional neural networks, or recurrent neural networks.

[0122] Similarly, computer devices can use pre-trained conversion rate prediction models to predict the probability of a user clicking on a promotional message and converting it, based on user profile data corresponding to the user who initiated the target request, promotional message profile data corresponding to the promotional message, and context information when the current request was initiated. This yields the current estimated conversion rate of the promotional message.

[0123] Step 406: For each candidate promotion information, determine the quality score threshold corresponding to the corresponding recall branch based on the current consumption data of the candidate promotion information and the historical quality score when it was recalled by the corresponding recall branch.

[0124] Step 408: Based on the current quality score of the candidate promotion information and the quality score threshold corresponding to the candidate promotion information in the corresponding recall branch, filter out the target promotion information in the corresponding recall branch.

[0125] In order to control the probability of a promotional message winning in a certain recall branch based on the historical performance of an individual promotional message, and thus flexibly adjust the number of promotional messages winning in each recall branch, so as to maximize the success of high-quality promotional messages, it is also necessary to consider how to control whether a candidate promotional message wins in a certain recall branch.

[0126] In this embodiment, for a candidate promotional information output by a certain recall branch, a quality score threshold is set for that candidate promotional information on that recall branch. If the current quality score of the candidate promotional information is greater than the quality score threshold of the candidate promotional information on that recall branch, then the promotional information is retained and enters the final recall queue, that is, the candidate promotional information wins from that recall branch; otherwise, the candidate promotional information is filtered out. In this way, promotional information that is better than others but has a poor historical performance compared to itself can be filtered out from each recall branch, giving more promotional information more exposure opportunities.

[0127] Specifically, the quality score threshold for each candidate promotional message on the corresponding recall branch is adjusted based on both the current consumption data and historical quality score of the candidate promotional message, thereby controlling the final number of promotional messages recalled. The current consumption data and historical quality score of the candidate promotional message can reflect the exposure effect of the promotional message to a certain extent.

[0128] Cost data refers to the cost incurred by the exposure of promotional information. It represents the promotional fee that the advertiser needs to pay to the advertising platform, and is usually related to the real-time bid placed by the advertiser for the promotional information. The current cost data of candidate promotional information can be the cost data generated by the candidate promotional information up to the time when the target request is received.

[0129] For a given candidate ad message, the historical quality score of the candidate ad message when it was recalled by the corresponding recall branch can be the historical quality score of the candidate ad message when it was recalled by the corresponding recall branch within the past month. For example, randomly sampling 500,000 user requests in the past month, the candidate ad message ad001 was exposed a total of 30,000 times: recalled 5,000 times by the ANN recall branch, 8,000 times by the TAG recall branch 1, 6,000 times by the TAG recall branch 2, and 11,000 times by the TAG recall branch 3. For this target request, only the ANN recall branch and TAG recall branch 2 output the candidate promotion information ad001. For the ANN recall branch, the computer device determines the quality score threshold of the candidate promotion information ad001 in the ANN recall branch based on the current consumption data and the historical quality score during the above 5000 recalls. For the TAG recall branch 2, the computer device determines the quality score threshold of the candidate promotion information ad001 in the TAG recall branch 2 based on the current consumption data and the historical quality score during the above 6000 recalls.

[0130] It is understandable that even within the same recall branch, the current consumption data of candidate promotional information is dynamically changing, and its corresponding quality score threshold in the same recall branch will also dynamically change. The quality score of candidate promotional information also dynamically changes with the promotional effect after it is delivered to users. By comparing the current quality score of candidate promotional information determined based on the current target request with the quality score threshold of candidate promotional information in the corresponding recall branch determined based on the current target request, it is possible to dynamically control whether the candidate promotional information can win in the corresponding recall branch each time it responds to the target request, and to flexibly adjust the number of high-quality promotional information winning in each recall branch.

[0131] Step 410: When the total amount of target promotion information filtered from at least two recall branches meets the expected total amount, the filtered target promotion information is used as the recall result in response to the target request.

[0132] Specifically, the computer device filters out target promotional information from each recall branch according to the aforementioned steps, and then counts the total amount of target promotional information filtered out from each recall branch. If the total amount matches the expected total amount, the filtered target promotional information is used as the recall result in response to the target request.

[0133] like Figure 6 The diagram shown is a schematic representation of a promotional information recall method in one embodiment. (Refer to...) Figure 6For the candidate promotion information output by the ANN recall branch, TAG recall branch 1, TAG recall branch 2, and TAG recall branch 3, the expected total amount of recall promotion information, target_num, is determined based on the current quality score of each candidate promotion information. Then, the merging process of this application is performed: according to the historical quality score and current consumption data of each candidate promotion information in the corresponding recall branch, the quality score threshold, quality_threshold, is determined in the corresponding recall branch. Then, target promotion information is filtered from each recall branch, and the total amount of filtered target promotion information is merge_num.

[0134] The aforementioned method for recalling promotional information, when faced with candidate promotional information output from multiple recall channels, takes two approaches. First, from the perspective of individual promotional information, it uses the current quality score of each candidate promotional information to control the expected total amount of promotional information to be recalled. This allows for more exposure opportunities for high-quality promotional information when the overall quality level of candidate promotional information output from each recall channel is relatively high. Second, it determines a corresponding quality score threshold for each candidate promotional information on its respective recall channel. The retention of a candidate promotional information is determined based on its quality score and this threshold. Since the quality score threshold for each candidate promotional information is dynamically adjusted based on its current consumption data and historical quality score, it effectively controls the total amount of promotional information recalled while also providing more exposure opportunities for high-quality promotional information in the recall channels. Then, when the total amount of target promotional information selected from multiple recall channels meets the aforementioned expected total, the selected target promotional information is used as the recall result for the recall phase. This method not only flexibly adjusts the number of promotional information selected from each recall channel but also aims to maximize the recall of high-quality promotional information, thereby improving the promotional information recall effect.

[0135] like Figure 7 The diagram shown illustrates a flowchart of the steps for determining the quality score threshold corresponding to the relevant recall branch for candidate promotion information in one embodiment, including:

[0136] Step 702: For each candidate promotion information, obtain the historical quality score when it was recalled by the corresponding recall branch;

[0137] Step 704: Sort the historical quality scores and divide them into preset equal parts; where each equal part corresponds to a quantile.

[0138] Step 706: Take the maximum historical quality score in each segment as the quantile quality score corresponding to each quantile.

[0139] Step 708: Based on each quantile and its corresponding quantile quality score, obtain the quality score threshold index table of the candidate promotion information in the corresponding recall branch.

[0140] For example, taking the candidate promotion information ad001 mentioned above, the computer device determines the quality score threshold for candidate promotion information ad001 in the ANN recall branch based on the current consumption data and the historical quality scores from the aforementioned 5000 recalls. The computer device can sort the historical quality scores of candidate promotion information ad001 from the 5000 recalls in ascending order, divide them into 100 equal parts, with 50 data points in each part. The quality score corresponding to the quantile of these 100 parts is the quantile quality score, which can be the maximum quality score in each part. The quality score threshold index table is shown in Table 1 below:

[0141] Table 1

[0142] quantile quality score quantiles 0.02 1 0.035 2 … … 0.52 100

[0143] In the above quality score threshold index table, the quality score of the first quantile is the maximum quality score of the first part after being divided into 100 equal parts. The second quantile, the third quantile, ..., the 100th quantile are all calculated in the same way. In this way, the quality score threshold index table corresponding to the candidate promotion information ad001 in the ANN recall branch is obtained.

[0144] Of course, the computer device can also divide the historical quality score into 50 equal parts, 200 equal parts, etc., as needed, and this application embodiment does not limit this. The computer device can also use the median or average value of each part as the quantile quality score corresponding to each part, and the computer device can also divide the historical quality score into several parts after sorting them in descending order, and this application embodiment does not limit this.

[0145] Similarly, for TAG recall branch 2, the computer device determines the quality score threshold for the candidate promotion information ad001 in TAG recall branch 2 based on the current consumption data and the historical quality score during the aforementioned 6000 recalls. In other words, if the same candidate promotion information has been recalled by multiple different recall branches, the computer device can calculate multiple different quality score threshold index tables corresponding to that same candidate promotion information; different candidate promotion information belonging to the same recall branch will have different corresponding quality score threshold index tables due to their different historical quality scores.

[0146] Optionally, to balance the response speed of the computer device with the accurate representation of the historical quality level of candidate promotional information, the computer device can pre-generate a quality score threshold index table for each promotional information corresponding to each recall branch. This way, when responding to a current request, it avoids the inefficiency of real-time calculation each time; the computer device can directly query the pre-generated quality score threshold index table. Furthermore, the computer device can update the quality score threshold index table for each recall branch corresponding to the promotional information at regular intervals (e.g., daily) based on the historical quality scores when the resampled promotional information was exposed.

[0147] For each candidate promotional information in the corresponding recall branch's quality score threshold index table, the computer device can use the quality score corresponding to the Nth quantile as the initial quality score threshold for the candidate promotional information. For example, the Nth quantile could be the 40th quantile, the 50th quantile, or the 60th quantile, etc., and this application embodiment does not limit this. It is understood that different candidate promotional information belonging to the same recall branch will have different quality score threshold index tables, and even if the quality score corresponding to the Nth quantile is used as the initial quality score threshold, the value may differ.

[0148] Based on this initial quality score threshold, the computer device iterates through the candidate promotion information according to the current consumption data:

[0149] threshold t =threshold t-1 +d*step;

[0150] quality_threshold t =f(threshold) t );

[0151] Among them, threshold t-1 This indicates the quantile corresponding to the quality score threshold of the previously updated promotional information; threshold t This represents the adjusted quantiles obtained by adjusting the quantiles corresponding to the quality score threshold of the previously updated promotional information. `step` represents the adjustment step size, for example, 3. `d` represents the adjustment coefficient, which has three values: `d` = 1, indicating an increase in quantiles and thus an increase in the quality score threshold; `d` = 0, indicating that the quantiles remain unchanged and the quality score threshold remains unchanged; `d` = -1, indicating a decrease in quantiles and thus a decrease in the quality score threshold. `f(threshold)` t The function represents the operation of querying the above quality score threshold index table, based on the quantile threshold. t The corresponding quantile quality score was found.t This indicates the quality score threshold corresponding to the current recall branch for the updated candidate promotion information.

[0152] In other words, the computer device can update and iterate the quality score threshold of the candidate promotion information in the manner described above. Within a time period during which the quality score threshold index table remains unchanged (e.g., 1 day), the computer device can iterate and update the quality score threshold of the candidate promotion information once every preset time, for example, once every hour.

[0153] In one embodiment, step 208 includes: obtaining the current consumption data of the candidate promotion information; obtaining the quantile corresponding to the quality score threshold of the previously updated promotion information; adjusting the quantile corresponding to the quality score threshold of the previously updated promotion information based on the current consumption data of the candidate promotion information, and using the adjusted quantile quality score in the quality score threshold index table corresponding to the corresponding recall branch as the quality score threshold of the candidate promotion information in the current update in the corresponding recall branch; wherein, the initial quality score threshold of the promotion information in the corresponding recall branch is the quantile quality score corresponding to the preset target quantile in the quality score threshold index table.

[0154] The consumption data includes the total consumption of candidate promotion information on the same day, the consumption generated by candidate promotion information on the corresponding recall branch on the same day (branch consumption), the total effect deviation of candidate promotion information on the same day, and the effect deviation of candidate promotion information on the corresponding recall branch on the same day.

[0155] For example, starting from 0:00, the computer device samples the historical quality scores of promotional information recalled over the past three months. This includes the historical quality scores of each promotional message recalled through various recall channels out of 500,000 user requests. The computer device generates a quality score threshold index table for each of these channels. The quality score threshold corresponding to the 40th percentile in this index table is the initial threshold. Based on the consumption data from 0:00 to the present, the computer device adjusts the threshold every hour from the 40th percentile. For example, the first adjustment moves the percentile to the 43rd percentile, and the resulting quality score threshold is the threshold corresponding to the 43rd percentile. The second adjustment adds another percentile, moving to the 46th percentile, and the resulting quality score threshold is the threshold corresponding to the 46th percentile. This process iterates.

[0156] The following explains how to adjust the percentile based on consumption data to adjust the quality score threshold.

[0157] like Figure 8 The diagram illustrates a process in one embodiment where the quantile corresponding to the quality score threshold of the previously updated promotional information is adjusted based on the current consumption data of the candidate promotional information. (Refer to...) Figure 8,include:

[0158] Step 802: Determine the adjustment coefficient and adjustment step size used to adjust the quantiles;

[0159] Step 804: Determine the current value of the adjustment coefficient based on the current consumption data of the candidate promotion information;

[0160] Step 806: When the adjustment coefficient is non-zero, the quantile corresponding to the quality score threshold of the previously updated promotion information is increased or decreased by adjusting the step size.

[0161] Step 808: When the adjustment coefficient is zero, the quality score threshold of the previously updated promotion information is used as the quality score threshold of the candidate promotion information for the current time.

[0162] As can be seen from the formula above, the adjustment of the quality score threshold is mainly determined by the value of the adjustment coefficient, which determines whether to increase, maintain, or decrease the quality score threshold.

[0163] The value of the adjustment coefficient d can be determined as shown in Table 2 below:

[0164] Table 2

[0165]

[0166]

[0167] As mentioned earlier, the expenditure of a candidate ad spend refers to the advertising cost that advertisers need to pay for the candidate ad spend. Sufficient expenditure means that the expenditure is greater than the target bid for the candidate ad spend, indicating that the candidate ad spend has sufficient exposure opportunities and good exposure results. To give other high-quality ads in the recall channel more exposure opportunities, the quality score threshold of the candidate ad spend in the corresponding recall channel can be appropriately increased, thereby raising the threshold for the candidate ad spend to win in the corresponding recall channel. Conversely, if the expenditure does not meet the condition of sufficient expenditure, it means that the candidate ad spend is insufficient and its exposure opportunities are not sufficient. To give it more exposure opportunities, the quality score threshold of the candidate ad spend in the corresponding recall channel can be appropriately decreased, thereby increasing the probability of the candidate ad spend winning in the corresponding recall channel and increasing exposure opportunities.

[0168] For example, sufficient overall consumption means that the overall consumption is greater than or equal to 6 * the target bid corresponding to the candidate promotion information; otherwise, it is considered insufficient overall consumption. Sufficient branch consumption means that the branch consumption is greater than or equal to 3 * the target bid corresponding to the candidate promotion information; otherwise, it is considered insufficient branch consumption. It is understood that the thresholds in Table 2 above can be adjusted as needed, and this application embodiment does not impose any restrictions on this. Of course, regarding the way consumption data affects the adjustment coefficient d, other methods different from those in Table 2 above can be used, and this application embodiment does not impose any restrictions on this.

[0169] To control the probability of a promotional message winning from a certain recall channel based on its historical performance, and to flexibly adjust the number of promotional messages winning from each recall channel in order to maximize the success of high-quality promotional messages, it is also necessary to consider how to adjust the total number of target promotional messages selected from each recall channel according to the previous steps if it does not meet the expected total.

[0170] In one embodiment, the above method further includes:

[0171] When the total number of target promotion information selected from at least two recall branches does not meet the expected total number, the process iteratively performs the step of adjusting the quality score threshold of each candidate promotion information in the corresponding recall branch, and then selecting target promotion information from the corresponding recall branch according to the current quality score of the candidate promotion information and the adjusted quality score threshold of the candidate promotion information in the corresponding recall branch, until the total number of target promotion information selected from at least two recall branches meets the expected total number, at which point the iteration stops.

[0172] After the above steps, the computer device has screened out target promotional information from each recall branch. However, the total amount of these target promotional information may not meet the expected total amount. Therefore, when the total amount does not meet the expected total amount, the computer device adjusts the quality score threshold of each candidate promotional information in the corresponding recall branch so that the total amount of the final recalled target promotional information meets the expected total amount.

[0173] In one embodiment, when the total amount of target promotion information filtered from at least two recall branches is greater than the expected total amount, the quantiles used to determine the quality score threshold of the candidate promotion information are increased by a preset step size to increase the quality score threshold of the candidate promotion information in the corresponding recall branch. Based on the current quality score of the candidate promotion information and the increased quality score threshold of the candidate promotion information in the corresponding recall branch, target promotion information is filtered from the corresponding recall branch. If the total amount is still greater than the expected total amount, the above steps are repeated based on the previously increased quantiles until the total amount of target promotion information filtered meets the expected total amount.

[0174] In one embodiment, when the total amount of target promotion information selected from at least two recall branches is less than the expected total amount, the quantiles used to determine the quality score threshold of the candidate promotion information are reduced by a preset step size to reduce the quality score threshold of the candidate promotion information in the corresponding recall branch. Based on the current quality score of the candidate promotion information and the reduced quality score threshold of the candidate promotion information in the corresponding recall branch, target promotion information is selected from the corresponding recall branch. If the total amount still reduces the expected total amount, the above steps are repeated based on the quantiles after the previous reduction until the total amount of target promotion information selected meets the expected total amount.

[0175] In one embodiment, after adjusting the quality score threshold corresponding to each candidate promotion information in the corresponding recall branch, target promotion information is selected from the corresponding recall branch based on the current quality score of the candidate promotion information and the adjusted quality score threshold of the candidate promotion information in the corresponding recall branch. This includes: for each candidate promotion information, determining the percentile corresponding to the quality score threshold obtained after the previous adjustment; adjusting the percentile corresponding to the quality score threshold to obtain the current percentile; querying the quality score threshold index table corresponding to the candidate promotion information in the corresponding recall branch, and using the quality score of the percentile corresponding to the current percentile as the current quality score threshold of the candidate promotion information in the corresponding recall branch; for each recall branch, randomly selecting from the output candidate promotion information according to the target ratio; for each randomly selected candidate promotion information in the recall branch, retaining the candidate promotion information with a quality score greater than the current quality score threshold, and obtaining the target promotion information selected from the corresponding recall branch.

[0176] For example, follow these steps:

[0177] 1) If the total amount of target promotion information selected from the corresponding recall branch is greater than the expected total amount, then let a = 1; if the total amount is less than the expected total amount, then let a = -1; initialize i = 1.

[0178] 2) Adjust the quantiles according to the following formula and query the quality score threshold according to the adjusted quantiles;

[0179] threshold iter =threshold iter-1 +a*i;

[0180] quality_threshold iter =f(threshold) iter );

[0181] 3) Initialize j to 1. Randomly select j% of candidate promotional information from each branch. Following step 2, re-select target promotional information from each recall branch according to the adjusted quality score threshold. If the total number of selected promotional information matches the expected total, end the process. If the total number of selected promotional information does not match the expected total, set j = j + 1, which increases the number of randomly selected candidate promotional information, and then re-select target promotional information according to the adjusted quality score threshold. Repeat step 3 until j = 100.

[0182] 4) If the total amount of promotional information after repeating step 3 already meets the expected total amount, then end the process; otherwise, i = i + 1, that is, after increasing the adjustment range of the quality score threshold, repeat steps 2 and 3 until i is 3.

[0183] In one embodiment, the method further includes: performing a sorting operation on the target promotion information in the recall results; determining the information to be pushed from the target promotion information based on the sorting results; and sending a target response carrying the information to be pushed to the terminal that triggered the target request.

[0184] Specifically, for the target promotion information in the final recall results, the computer device can use a sorting algorithm or sorting model to score and sort the information to obtain the sorting results. Based on the final sorting results, the information to be pushed is determined from the target promotion information. After generating the target response based on the information to be pushed, it is sent to the user terminal that initiated the target request, and the information to be pushed is displayed to the user.

[0185] like Figure 9 The diagram shown is a framework schematic of a promotional information recall method in one embodiment. (Refer to...) Figure 9 For each candidate promotional message output by each recall branch, the current quality score of each candidate promotional message is calculated. The quality score is used to determine the expected total number of recall results and whether the promotional message can win from the branch. Based on the historical quality scores of each candidate promotional message that has been recalled by each recall branch, a quality score threshold index table is calculated for each recall branch. Then, combined with the current consumption data of each candidate promotional message, the corresponding quality score threshold is looked up from the quality score threshold index table to determine whether the candidate promotional message can win from the recall branch, thereby achieving truncation of a large number of candidate promotional messages. When the total number of winning candidate promotional messages does not meet the expected total number, candidate promotional messages are randomly selected from the recall branch and the quality score threshold is adjusted to make the total number meet the expected total number.

[0186] like Figure 10 The diagram shown is a flowchart illustrating a promotional information recall method in a specific embodiment. (Refer to...) Figure 10 The method includes the following steps:

[0187] Step 1002: Obtain candidate promotion information output by responding to the target request through at least two recall branches;

[0188] Step 1004: For each candidate promotional information, calculate the current quality score of the candidate promotional information based on the target bid, the current estimated click-through rate, and the current estimated conversion rate.

[0189] Step 1006: Obtain the average historical quality score corresponding to the historical recall promotion information of each recall branch;

[0190] Step 1008: Obtain the total demand for promotional information during the ranking stage;

[0191] Step 1010: Based on the current quality score, historical average quality score, total demand, and preset margin control ratio of each candidate promotion information, determine the range of the expected total amount of promotion information to be recalled from the candidate promotion information.

[0192] Step 1012: For each candidate promotion information, obtain the historical quality score when it was recalled by the corresponding recall branch;

[0193] Step 1014: After sorting the historical quality scores, divide them into preset equal parts; each part corresponds to a quantile, and the maximum value of the historical quality scores in each part is taken as the quantile quality score corresponding to each quantile; based on each quantile and the corresponding quantile quality score, obtain the quality score threshold index table of the candidate promotion information in the corresponding recall branch.

[0194] Step 1016: Use the quality score of the target quantile corresponding to the preset quantile in the quality score threshold index table as the initial quality score threshold for the corresponding recall branch in the promotion information.

[0195] Step 1018: Determine the adjustment coefficient and adjustment step size used to adjust the quantiles. For each candidate promotion information, determine the current value of the adjustment coefficient based on the current consumption data of the candidate promotion information.

[0196] Step 1020: Obtain the quantile corresponding to the quality score threshold of the previously updated candidate promotion information, wherein the quality score threshold of the promotion information is updated when the output of the recalled branch is used as candidate promotion information.

[0197] Step 1022: When the adjustment coefficient is non-zero, the quantile corresponding to the quality score threshold of the previously updated promotion information is increased or decreased by adjusting the step size. The quality score of the quantile corresponding to the adjusted quantile in the quality score threshold index table of the corresponding recall branch is used as the quality score threshold of the candidate promotion information in the corresponding recall branch in the current update.

[0198] Step 1024: When the adjustment coefficient is zero, the quality score threshold of the previously updated promotion information is used as the quality score threshold of the candidate promotion information for the current time.

[0199] Step 1026: Filter out candidate promotion information in the corresponding recall branch whose current quality score is greater than the quality score threshold;

[0200] Step 1028: Determine whether the total amount of the filtered target promotion information falls within the range of the expected total amount. If yes, proceed to step 1042; otherwise, proceed to step 1030.

[0201] Step 1030: If the total amount of selected target promotion information is greater than the expected total amount, then for each candidate promotion information in the recall branch, increase the quantile corresponding to the quality score threshold obtained after the previous adjustment to obtain the quantile for the current time. If the total amount of selected target promotion information is less than the expected total amount, then for each candidate promotion information in the recall branch, decrease the quantile corresponding to the quality score threshold obtained after the previous adjustment to obtain the quantile for the current time. Query the quality score threshold index table corresponding to the candidate promotion information in the corresponding recall branch, and use the quantile quality score corresponding to the current quantile as the quality score threshold of the candidate promotion information in the corresponding recall branch for the current time.

[0202] Step 1032, let j = 1;

[0203] Step 1034: For each recall branch, randomly select from the output candidate promotion information at a ratio of j%.

[0204] Step 1036: For each recall branch randomly selected candidate promotion information, retain the candidate promotion information with a quality score greater than the current quality score threshold, and obtain the target promotion information selected from the corresponding recall branch.

[0205] Step 1038: Determine whether the total amount of the selected target promotion information falls within the range of the expected total amount; if not, proceed to step 1040; if yes, proceed to step 1042.

[0206] Step 1040: Determine if j is 100. If not, set j = j + 1 and return to step 1034; if yes, return to step 1030.

[0207] Step 1042: Use the selected target promotion information as the recall result in response to the target request;

[0208] Step 1044: Perform a sorting operation on the target promotion information in the recall results;

[0209] Step 1046: Determine the information to be pushed from the target promotion information based on the sorting results;

[0210] Step 1048: Send a target response carrying the information to be pushed to the terminal that triggered the target request.

[0211] The aforementioned method for recalling promotional information, when faced with candidate promotional information output from multiple recall channels, takes two approaches. First, from the perspective of individual promotional information, it uses the current quality score of each candidate promotional information to control the expected total amount of promotional information to be recalled. This allows for more exposure opportunities for high-quality promotional information when the overall quality level of candidate promotional information output from each recall channel is relatively high. Second, it determines a corresponding quality score threshold for each candidate promotional information on its respective recall channel. The retention of a candidate promotional information is determined based on its quality score and this threshold. Since the quality score threshold for each candidate promotional information is dynamically adjusted based on its current consumption data and historical quality score, it effectively controls the total amount of promotional information recalled while also providing more exposure opportunities for high-quality promotional information in the recall channels. Then, when the total amount of target promotional information selected from multiple recall channels meets the aforementioned expected total, the selected target promotional information is used as the recall result for the recall phase. This method not only flexibly adjusts the number of promotional information selected from each recall channel but also aims to maximize the recall of high-quality promotional information, thereby improving the promotional information recall effect.

[0212] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple steps or stages, which are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0213] In one embodiment, such as Figure 11 As shown, a promotional information recall device 1100 is provided. This device can be a software module, a hardware module, or a combination of both integrated into a computer device. Specifically, the device includes: a candidate promotional information acquisition module 1102, a total recall quantity determination module 1104, a quality score threshold determination module 1106, a target promotional information filtering module 1108, and a recall response module 1110, wherein:

[0214] The candidate promotion information acquisition module 1102 is used to acquire candidate promotion information output by responding to the target request through at least two recall branches;

[0215] The total recall quantity determination module 1104 is used to determine the expected total quantity of recall promotion information based on the current quality score of each candidate promotion information.

[0216] The quality score threshold determination module 1106 is used to determine the quality score threshold of each candidate promotion information in the corresponding recall branch based on the current consumption data of the candidate promotion information and the historical quality score when it was recalled by the corresponding recall branch.

[0217] The target promotion information filtering module 1108 is used to filter target promotion information from the corresponding recall branch according to the current quality score of the candidate promotion information and the quality score threshold of the candidate promotion information in the corresponding recall branch.

[0218] The recall response module 1110 is used to use the selected target promotion information as the recall result in response to the target request when the total amount of target promotion information filtered from at least two recall branches meets the expected total amount.

[0219] In one embodiment, the candidate promotion information acquisition module 1102 is further configured to extract the user identifier carried in the target request; acquire the user's historical behavior data corresponding to the user identifier; and determine the promotion information that matches the user's historical behavior data from the promotion information set through a pre-trained neural network model, as candidate promotion information output by the recall branch based on the neural network.

[0220] In one embodiment, the candidate promotion information acquisition module 1102 is further configured to extract the user identifier carried in the target request; acquire the user tag data corresponding to the user identifier; match the user tag data with the promotion information tag data according to a preset matching strategy; and obtain the candidate promotion information output by the recall branch based on the matching strategy from the promotion information set according to the matching result.

[0221] In one embodiment, the recall total determination module 1104 is further configured to: determine the current quality score for each candidate promotional information; obtain the historical average quality score corresponding to the historical recall promotional information of each recall branch; obtain the total demand for promotional information during the sorting stage; and determine the range of the expected total amount of promotional information to be recalled from the candidate promotional information based on the current quality score, historical average quality score, total demand, and preset margin control ratio of each candidate promotional information.

[0222] In one embodiment, the promotion information recall device 1100 further includes: a quality score calculation module, used to predict the click-through rate of candidate promotion information to obtain the current estimated click-through rate; predict the conversion rate of candidate promotion information to obtain the current estimated conversion rate; obtain the target bid corresponding to the candidate promotion information; and calculate the current quality score of the candidate promotion information based on the target bid corresponding to the candidate promotion information, the current estimated click-through rate, and the current estimated conversion rate.

[0223] In one embodiment, the promotion information recall device 1100 further includes: a quality score threshold index table calculation module, used to obtain the historical quality score when each candidate promotion information was recalled by the corresponding recall branch; sort the historical quality scores and divide them into preset equal parts; wherein each equal part corresponds to a quantile; take the maximum value of the historical quality score in each equal part as the quantile quality score corresponding to each quantile; and obtain the quality score threshold index table of the candidate promotion information in the corresponding recall branch according to each quantile and the corresponding quantile quality score.

[0224] In one embodiment, the quality score threshold determination module 1106 is further configured to obtain the current consumption data of the candidate promotion information, including the total consumption of the candidate promotion information on the current day, the consumption generated by the candidate promotion information on the corresponding recall branch on the current day, the total effect deviation of the candidate promotion information on the current day, and the effect deviation of the candidate promotion information on the corresponding recall branch on the current day; obtain the quantile corresponding to the quality score threshold of the previously updated promotion information; adjust the quantile corresponding to the quality score threshold of the previously updated promotion information according to the current consumption data of the candidate promotion information, and use the quantile quality score corresponding to the adjusted quantile in the quality score threshold index table corresponding to the corresponding recall branch as the quality score threshold of the candidate promotion information in the current update in the corresponding recall branch; wherein, the initial quality score threshold of the promotion information in the corresponding recall branch is the quantile quality score corresponding to the preset target quantile in the quality score threshold index table.

[0225] In one embodiment, the quality score threshold determination module 1106 is further configured to determine the adjustment coefficient and adjustment step size for adjusting the quantiles; determine the current value of the adjustment coefficient based on the current consumption data of the candidate promotion information; when the adjustment coefficient is non-zero, increase or decrease the quantile corresponding to the quality score threshold of the previously updated promotion information by adjusting the adjustment step size; when the adjustment coefficient is zero, use the quality score threshold of the previously updated promotion information as the current quality score threshold of the candidate promotion information.

[0226] In one example, the promotional information recall device 1100 further includes a recall total quantity control module, which is used to iteratively execute the steps of adjusting the quality score threshold of each candidate promotional information in the corresponding recall branch and then selecting target promotional information from the corresponding recall branch according to the current quality score of the candidate promotional information and the adjusted quality score threshold of the candidate promotional information in the corresponding recall branch, until the total amount of target promotional information selected from at least two recall branches meets the expected total amount, and then stop iterating.

[0227] In one example, the recall total control module is also used to determine the percentile corresponding to the quality score threshold obtained after the previous adjustment for each candidate promotion information; adjust the percentile corresponding to the quality score threshold to obtain the percentile for the current time; query the quality score threshold index table corresponding to the candidate promotion information in the corresponding recall branch, and use the quality score of the percentile corresponding to the current time as the quality score threshold of the candidate promotion information in the corresponding recall branch; for each recall branch, randomly select from the output candidate promotion information according to the target ratio; for the candidate promotion information randomly selected from each recall branch, retain the candidate promotion information with a quality score greater than the quality score threshold for the current time, and obtain the target promotion information selected from the corresponding recall branch.

[0228] In one instance, the promotional information recall device 1100 further includes: a sorting module and a push module. The sorting module is used to perform a sorting operation on the target promotional information in the recall results; and to determine the information to be pushed from the target promotional information based on the sorting results. The push module is used to send a target response carrying the information to be pushed to the terminal that triggered the target request.

[0229] The aforementioned promotional information recall device 1100, when faced with candidate promotional information output from multiple recall branches, on the one hand, controls the expected total amount of recalled promotional information based on the current quality score of each candidate promotional information from the perspective of individual promotional information. This allows for more exposure opportunities for high-quality promotional information when the overall quality level of candidate promotional information output from each recall branch is relatively high. On the other hand, it determines a corresponding quality score threshold for each candidate promotional information on the relevant recall branch, and decides whether to retain the candidate promotional information based on the relationship between the candidate promotional information's quality score and the quality score threshold. Since the quality score threshold corresponding to each candidate promotional information is dynamically adjusted based on the current consumption data and historical quality score of the candidate promotional information, it can effectively control the total amount of promotional information ultimately recalled while also giving more exposure opportunities to more high-quality promotional information in the recall branches. Subsequently, when the total amount of target promotional information selected from multiple recall branches meets the aforementioned expected total amount, the selected target promotional information is used as the recall result of the recall stage. This not only allows for flexible adjustment of the number of promotional information selected from each recall branch but also achieves the goal of recalling as many high-quality promotional information as possible, thereby improving the promotional information recall effect.

[0230] Specific limitations regarding the promotional information recall device 1100 can be found in the limitations of the promotional information recall method described above, and will not be repeated here. Each module in the aforementioned promotional information recall device 1100 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0231] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores historical quality scores of promotional information. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a method for recalling promotional information.

[0232] Those skilled in the art will understand that Figure 12The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0233] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0234] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0235] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0236] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0237] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0238] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for recalling promotional information, characterized in that, The method includes: Obtain candidate promotion information output by responding to the target request through at least two recall branches; For each candidate promotional information, determine the current quality score and calculate the average of the current quality scores of the output candidate promotional information. Calculate the average historical quality score corresponding to the historical recall promotional information of each recall branch. The average historical quality score is the average quality score of each recall branch when the recalled promotional information was recalled on the same day. Based on the ratio of the average current quality score of the output candidate promotion information to the average historical quality score, and the total demand for promotion information during the sorting stage, the expected total amount of recall promotion information is determined. For each candidate promotion information, obtain the quality score threshold index table corresponding to the candidate promotion information in the corresponding recall branch. The quality score threshold index table is generated by sorting the historical quality scores when the candidate promotion information was recalled by the corresponding recall branch and dividing it into preset equal parts. Each equal part corresponds to a quantile. The quantile quality score corresponding to each quantile is determined based on the multiple historical quality scores of each equal part. For each candidate promotional information, the percentile corresponding to the previously updated quality score threshold is obtained. Based on the current consumption data of the candidate promotional information, the percentile corresponding to the previously updated quality score threshold is adjusted. The quality score of the percentile corresponding to the adjusted percentile is retrieved from the quality score threshold index table of the corresponding recall branch. This quality score is the quality score threshold of the candidate promotional information updated in the corresponding recall branch. The consumption data is the cost incurred by the promotional information being exposed. The current consumption data of the candidate promotional information is the consumption data generated by the candidate promotional information up to the time when the target request is received. Based on the current quality score of the candidate promotion information and the updated quality score threshold of the candidate promotion information in the corresponding recall branch, the target promotion information is filtered out in the corresponding recall branch. When the total amount of target promotion information filtered from the at least two recall branches meets the expected total amount, the filtered target promotion information is used as the recall result in response to the target request.

2. The method according to claim 1, characterized in that, The at least two recall branches include neural network-based recall branches, and the method further includes: Extract the user identifier carried in the target request; Obtain the user's historical behavior data corresponding to the user identifier; Using a pre-trained neural network model, promotional information matching the user's historical behavior data is determined from the promotional information set, and used as candidate promotional information output by the neural network-based recall branch.

3. The method according to claim 1, characterized in that, The at least two recall branches include recall branches based on a matching strategy, and the method further includes: Extract the user identifier carried in the target request; Obtain the user tag data corresponding to the user identifier; According to a preset matching strategy, the user tag data is matched with the promotional information tag data; Based on the matching results, the candidate promotion information output by the recall branch based on the matching strategy is obtained from the promotion information set.

4. The method according to claim 1, characterized in that, The expected total amount of promotional information recalled from each recall branch is related to the remaining balance control ratio of recalled promotional information. The determination of the expected total amount of recalled promotional information, based on the ratio of the average current quality score of the output candidate promotional information to the average historical quality score, and the total demand for promotional information during the sorting stage, includes: Obtain the preset margin control ratio; Based on the ratio of the average current quality score of the output candidate promotion information to the average historical quality score, the total demand for promotion information during the sorting stage, and the preset margin control ratio, the expected total amount of promotion information to be recalled from the candidate promotion information is determined within a certain range.

5. The method according to claim 1, characterized in that, The method further includes: The click-through rate (CTR) of the candidate promotional information is predicted to obtain the current estimated CTR. The conversion rate of the candidate promotion information is predicted to obtain the current estimated conversion rate; Obtain the target bid corresponding to the candidate promotion information; The current quality score of the candidate promotion information is calculated based on the target bid corresponding to the candidate promotion information, the current estimated click-through rate, and the current estimated conversion rate.

6. The method according to claim 1, characterized in that, The steps for generating the quality score threshold index table corresponding to the relevant recall branch for the candidate promotion information include: For each candidate promotion information, obtain the historical quality score when it was recalled by the corresponding recall branch; The historical quality scores are sorted and divided into preset equal parts; each part corresponds to a quantile. The maximum historical quality score in each segment is taken as the quality score of each quantile. Based on each quantile and its corresponding quantile quality score, obtain the quality score threshold index table for the candidate promotion information in the corresponding recall branch.

7. The method according to claim 6, characterized in that, The consumption data includes the total consumption of the candidate promotion information on the same day, the consumption generated by the candidate promotion information on the corresponding recall branch on the same day, the total effect deviation of the candidate promotion information on the same day, and the effect deviation of the candidate promotion information on the corresponding recall branch on the same day. The initial quality score threshold corresponding to the candidate promotion information on the corresponding recall branch is the quality score of the target percentile corresponding to the preset quality score threshold index table. The total effect deviation on the same day = (total consumption on the same day / total transaction amount) - 1, and the effect deviation on the corresponding recall branch on the same day = (consumption generated on the corresponding recall branch on the same day / total transaction amount of the branch) - 1.

8. The method according to claim 7, characterized in that, The step of adjusting the quantile corresponding to the previously updated quality score threshold based on the current consumption data of the candidate promotion information includes: Determine the adjustment coefficient and adjustment step size used to adjust the quantiles; Based on the current consumption data of the candidate promotion information, determine the current value of the adjustment coefficient; When the adjustment coefficient is non-zero, the quantile corresponding to the previously updated quality score threshold is increased or decreased by the adjustment step size. The method further includes: When the adjustment coefficient is zero, the previously updated quality score threshold is used as the quality score threshold for the candidate promotion information in the corresponding recall branch in the current update.

9. The method according to claim 1, characterized in that, The method further includes: When the total amount of target promotion information filtered from the at least two recall branches does not meet the expected total amount, the process iteratively performs the step of adjusting the quality score threshold of each candidate promotion information in the corresponding recall branch, and then filtering target promotion information from the corresponding recall branch according to the current quality score of the candidate promotion information and the adjusted quality score threshold of the candidate promotion information in the corresponding recall branch, until the total amount of target promotion information filtered from the at least two recall branches meets the expected total amount, at which point the iteration stops.

10. The method according to claim 9, characterized in that, After adjusting the quality score threshold for each candidate promotional information in the corresponding recall branch, the target promotional information is selected from the corresponding recall branch based on the current quality score of the candidate promotional information and the adjusted quality score threshold of the candidate promotional information in the corresponding recall branch, including: For each candidate promotion information, determine the quantile corresponding to the quality score threshold obtained after the previous adjustment; The quantiles corresponding to the quality score threshold are adjusted to obtain the quantiles for the current time. Query the quality score threshold index table corresponding to the candidate promotion information in the corresponding recall branch, and take the quality score of the quantile corresponding to the current quantile as the quality score threshold of the candidate promotion information in the corresponding recall branch. For each recall branch, randomly select from the output candidate promotion information according to the target proportion; For each recall branch, candidate promotion information randomly selected is retained, and the candidate promotion information with a quality score greater than the current quality score threshold is obtained to obtain the target promotion information selected from the corresponding recall branch.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: Perform a sorting operation on the target promotion information in the recall results; Based on the sorting results, determine the information to be pushed from the target promotion information; Send a target response carrying the information to be pushed to the terminal that triggered the target request.

12. A device for recalling promotional information, characterized in that, The device includes: The candidate promotion information acquisition module is used to acquire candidate promotion information output by responding to the target request through at least two recall branches. The total recall volume determination module is used to determine the current quality score for each candidate promotion information, calculate the average current quality score of the output candidate promotion information, and statistically analyze the historical average quality score corresponding to the historical recall promotion information of each recall branch. The historical average quality score is the average quality score of each recall branch when the recalled promotion information was recalled on the same day. Based on the ratio of the average current quality score of the output candidate promotion information to the historical average quality score, and the total demand for promotion information in the sorting stage, the expected total amount of recall promotion information is determined. The quality score threshold determination module is used to, for each candidate promotional information, obtain a quality score threshold index table corresponding to the corresponding recall branch. This index table is generated by sorting the historical quality scores of the candidate promotional information when it was recalled by the corresponding recall branch and dividing it into preset equal parts, with each part corresponding to a quantile. The quantile quality score for each quantile is determined based on multiple historical quality scores for each part. For each candidate promotional information, the module obtains the quantile corresponding to the previously updated quality score threshold. Based on the current consumption data of the candidate promotional information, it adjusts the quantile corresponding to the previously updated quality score threshold. The module then queries the quality score index table corresponding to the adjusted quantile to obtain the quantile quality score corresponding to the adjusted quantile in the current updated quality score threshold of the corresponding recall branch. The consumption data is the cost incurred by the promotional information being exposed, and the current consumption data of the candidate promotional information is the consumption data generated by the candidate promotional information up to the point when the target request is received. The target promotion information filtering module is used to filter out target promotion information in the corresponding recall branch according to the current quality score of the candidate promotion information and the quality score threshold updated in the corresponding recall branch. The recall response module is used to use the selected target promotion information as the recall result in response to the target request when the total amount of target promotion information filtered from the at least two recall branches meets the expected total amount.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images