Model training method, advertisement material prediction method and device

Abstract

The invention provides a model training method and device and an advertisement material prediction method and device, and the model training method comprises the steps: determining a training sample set based on an advertisement material set with a call end record; determining an advertisement material basic feature, an advertisement material playing feature, a user basic feature, a user historical behavior feature and a cross feature between the advertisement material and the user corresponding to the training sample; inputting an advertisement material basic feature, an advertisement material playing feature, a user basic feature, a user historical behavior feature and a cross feature between the advertisement material and the user corresponding to the training sample into an initial advertisement material scoring model for prediction processing to obtain a first prediction score corresponding to the advertisement material in the training sample; and according to the first prediction score corresponding to the advertisement material in the training sample and the positive and negative sample labels of the training sample, performing parameter adjustment on the advertisement material scoring model until a convergence condition is reached, and obtaining a trained advertisement material scoring model.

Description

Technical Field

[0001] The present application relates to the field of computer technologies, and in particular, to a method for training a model, a method and device for predicting advertising materials. Background Art

[0002] Weibo, as a social platform with a large user base, provides various types of advertisements, including splash screen advertisements and in-feed advertisements. The advertising placement platform provides rich audience targeting functions, and can perform precise targeting based on multiple dimensions such as the age, gender, region, interests, etc. of users.

[0003] However, the current sampling process performs negative sampling at the granularity of call-end users, with a fixed sampling ratio for each user, and the channels and specifications cannot be distinguished, resulting in a large number of invalid samples, and further leading to the inability to precisely place high-quality advertising materials. Summary of the Invention

[0004] The present application provides a method for training a model, a method and device for predicting advertising materials, so as to achieve precise placement of high-quality advertising materials.

[0005] The technical solution of the present application is implemented as follows:

[0006] The present application provides a method for training a model. The method includes: determining a training sample set based on an advertising material set with call-end records; the call-end records are used to record the call-end behavior of users clicking on the specified page of the advertiser by being exposed to the advertising materials on the advertising placement platform; the training sample set includes multiple training samples corresponding to each advertising material, where the positive samples in the multiple training samples are the matching pairs of the advertising materials and the corresponding clicking users, and the negative samples in the multiple training samples are the matching pairs of the advertising materials and non-clicking users. A clicking user refers to a user who has a call-end behavior towards the advertising material, and a non-clicking user refers to a user who does not have a call-end behavior towards the advertising material and has a call-end behavior towards other advertising materials in the same advertising position; determining the basic features of the advertising material corresponding to the training sample, the advertising material playback features, the basic features of the user, the user's historical behavior features, and the cross features between the advertising material and the user; inputting the basic features of the advertising material corresponding to the training sample, the advertising material playback features, the basic features of the user, the user's historical behavior features, and the cross features between the advertising material and the user into an initial advertising material scoring model for prediction processing to obtain a first prediction score corresponding to the advertising material in the training sample; the first prediction score is used to characterize the degree of interest of the user in the advertising material in the training sample; adjusting the parameters of the advertising material scoring model according to the first prediction score corresponding to the advertising material in the training sample and the positive and negative sample labels of the training sample until the convergence condition is reached, and obtaining a trained advertising material scoring model.

[0007] The present application provides an advertising material prediction method, which includes: obtaining the advertising material basic features and advertising material playback features of the target advertising material to be predicted, the user basic features and user historical behavior features of multiple active users corresponding to the advertising position where the target advertising material is located, and the cross features between the target advertising material and each active user; wherein, the multiple active users corresponding to the advertising position refer to the set of users who have a call end behavior for any advertising material at the advertising position within a preset time period; inputting the advertising material basic features and advertising material playback features of the target advertising material, the user basic features and user historical behavior features of multiple active users, and the cross features into the trained advertising material scoring model for prediction processing to obtain multiple second prediction scores corresponding to the target advertising material; the advertising material scoring model is trained by the above model training method, and the second prediction score is used to represent the degree of interest of the active user in the target advertising material; based on the multiple second prediction scores corresponding to the target advertising material, determining the prediction total score of the target advertising material corresponding to multiple active users, so as to sort the target advertising material based on the prediction total score.

[0008] The present application provides a training device for a model. The above device includes: a sampling module, configured to determine a training sample set based on an advertising material set with call end records; the call end record is used to record the call end behavior of a user clicking into the page designated by the advertiser through the advertising material exposed on the advertising placement platform; the training sample set includes multiple training samples corresponding to each advertising material, wherein the positive samples in the multiple training samples are the matching pairs of the advertising material and the corresponding clicking users, and the negative samples in the multiple training samples are the matching pairs of the advertising material and non-clicking users. The clicking user refers to a user who has a call end behavior for the advertising material, and the non-clicking user refers to a user who does not have a call end behavior for the advertising material and has a call end behavior for other advertising materials at the same advertising position; a data processing module, configured to determine the advertising material basic features, advertising material playback features, user basic features, user historical behavior features, and cross features between the advertising material and the user corresponding to the training sample; a scoring training module, configured to input the advertising material basic features, advertising material playback features, user basic features, user historical behavior features, and cross features between the advertising material and the user corresponding to the training sample into an initial advertising material scoring model for prediction processing to obtain a first prediction score corresponding to the advertising material in the training sample; the first prediction score is used to represent the degree of interest of the user in the advertising material in the training sample; a training module, configured to adjust the parameters of the advertising material scoring model according to the first prediction score corresponding to the advertising material in the training sample and the positive and negative sample labels of the training sample until the convergence condition is reached, so as to obtain the trained advertising material scoring model.

[0009] An embodiment of the present application provides an advertising material prediction device, which includes: a data acquisition module, configured to obtain the advertising material basic features and advertising material playback features of the target advertising material to be predicted, the user basic features and user historical behavior features of multiple active users corresponding to the advertising position where the target advertising material is located, and the cross features between the target advertising material and each active user; wherein, the multiple active users corresponding to the advertising position refer to the set of users who have a call end behavior for any advertising material at the advertising position within a preset time period; a scoring module, configured to input the advertising material basic features and advertising material playback features of the target advertising material, the user basic features and user historical behavior features of multiple active users, and the cross features into the trained advertising material scoring model for prediction processing, to obtain multiple second prediction scores corresponding to the target advertising material; the advertising material scoring model is trained by the above model training method, and the second prediction score is used to characterize the degree of interest of the active user in the target advertising material; a sorting module, configured to determine the prediction total score of the target advertising material corresponding to multiple active users based on the multiple second prediction scores corresponding to the target advertising material, so as to sort the target advertising material based on the prediction total score.

[0010] The present application provides an electronic device, including: a memory, configured to store executable instructions; a processor, configured to implement the method provided by the present application when executing the executable instructions stored in the memory.

[0011] The present application provides a computer storage medium, storing executable instructions, which are used to implement the method provided by the present application when the executable instructions are executed by a processor.

[0012] The present application provides a computer program product, including a computer program or instruction, which is used to implement the method provided by the present application when the computer program or instruction is executed by a processor.

[0013] The present application has the following beneficial effects:

[0014] In the present application, according to the exposure log and call end log of the advertising delivery platform and the preset overall sampling ratio, positive samples and negative samples are sampled, realizing dynamic sampling based on the exposure and click situations of advertising materials in the direction of material granularity, optimizing the sampling method of negative samples, and improving the quality of positive samples and negative samples. The advertising material scoring model trained based on this can achieve precise delivery of high-quality materials.

[0015] Furthermore, during the sampling process, adding the advertising position as a sampling dimension can be closer to the scenario of programmatic customer acquisition; and a better-quality database can be constructed from the perspective of material granularity.

[0016] Furthermore, in combination with the actual business scenario of programmatic customer acquisition, playback features that can effectively evaluate video quality are added. By using the existing call-side data, through the combination of interest tags, tag scores, and duration coefficients, and calculating the matching scores of each group of advertising materials, user historical behavior features that can reflect users' interest preferences are constructed. The introduction of cross features increases the overall feature dimension of the advertising material scoring model and improves the fitting ability of the advertising material scoring model for complex data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic structural diagram of a recommendation system provided by an embodiment of the present application;

[0018] Figure 2 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0019] Figure 3 is an optional flowchart of a method for training a model provided by an embodiment of the present application;

[0020] Figure 4 is an optional flowchart of a method for training a model provided by an embodiment of the present application;

[0021] Figure 5 is a schematic structural diagram of an advertising material scoring model provided by an embodiment of the present application;

[0022] Figure 6 is an optional flowchart of a method for predicting advertising materials provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be construed as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0024] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0025] If descriptions such as "first / second" appear in the application documents, the following description shall be added. In the following description, the terms "first / second / third" only distinguish similar objects and do not represent a specific order for the objects. Understandably, "first / second / third" can be interchanged in a specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0027] The off-site channel advertising placement of Weibo is mainly manual, and the customer acquisition efficiency is relatively low. To improve the customer acquisition efficiency of channel users, a large-scale high-quality material library can be constructed, a machine learning placement model can be introduced for multiple channels, the advertising materials can be scored and sorted, and the efficiency and effect of user acquisition and activation can be improved. Compared with conventional manual purchases, it can greatly improve the efficiency, scale, and placement strategy of advertising purchases.

[0028] Programmatic customer acquisition is a technical means for off-site placement of Weibo. It mines high-quality materials in the Weibo scenario, generates creative advertising materials, and conducts the placement and management of advertisements through a real-time bidding placement platform (demand-side platform, DSP) provided by the demand side. In Weibo programmatic customer acquisition, the advertising placement mode is the cost per click (CPC) mode, that is, the bid price per single click of the advertiser × the predicted click-through rate (pCTR) of this traffic for the advertisement. The greater the product of the bid price per single click and the predicted click-through rate, the higher the probability that this advertising material will be exposed to users through competition. Therefore, when the bid price per single click is the same, there are mainly two methods to improve the platform's predicted pCTR: 1. Upgrade the advertising creativity and design excellent advertising creativity so that the content of the material is presented to users in a more excellent way; 2. Optimize the selection of programmatic customer acquisition advertising materials, and select truly high-quality materials from the massive advertising materials every day through a sorting algorithm and provide them to the advertising placement platform to obtain a higher exposure opportunity. The upgrade of advertising creativity is often limited by the actual advertising placement platform. Therefore, optimizing the selection of advertising materials from the advertiser's perspective is the fundamental way to increase the exposure of advertising placements. The current programmatic customer acquisition sorting system performs random negative sampling at the granularity of call-end users, constructs a training data set by combining the known basic characteristics of the materials and user characteristics to train the model, then matches the daily selected advertising materials with the specified user package, makes a prediction score through the model, votes and scores each advertising material by setting a threshold, and selects the top N advertising materials as high-quality materials according to the scoring results.

[0029] However, in the scenario of programmatic customer acquisition, advertisers cannot obtain real material exposure data. The difficulty in data recovery leads to the lack of negative samples for model training. Moreover, the disadvantage of negative sampling at the granularity of call-end users is that the sampling ratio of each user is fixed, and channels and specifications cannot be distinguished, resulting in many invalid samples.

[0030] In addition, at present, 99% of the placements of programmatic advertisements are video materials. In addition to the basic features of video materials, the playback data of videos can also be used as an important feature to reflect the quality relationship of advertising materials. However, with a single programmatic scenario and limited data, reasonable video playback data cannot be statistically obtained. In the material scoring stage, by voting through a threshold, a scoring method where a value greater than the threshold is 1 and otherwise is 0 can distinguish different materials. However, it has high requirements for threshold setting and insufficient discrimination, and it is easy to have the situation where some materials with relatively average quality have the same score, resulting in the inability to accurately place high-quality advertising materials.

[0031] In view of the above problems, the embodiments of the present application provide a training method for a model to achieve accurate placement of high-quality advertising materials.

[0032] The following describes the exemplary applications of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of user terminals such as laptop computers, tablet computers, desktop computers, mobile devices (such as mobile phones, wearable smart watches, dedicated messaging devices), etc., or can also be implemented as a server. Below, the exemplary application when the electronic device is implemented as a server will be described.

[0033] See Figure 1 , Figure 1 is a schematic diagram of the architecture of the recommendation system provided by the embodiments of the present application. The electronic devices (exemplarily shown as electronic device 40-1 and electronic device 40-2) are connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0034] In some possible implementation manners, user A can publish material information A through terminal 40-1, and user B can publish material information B through terminal 40-2. The material information A and the material information B are uploaded to the server 200 through the network 300. The server 200 can store the material information A and the material information B in the database 500. In order to achieve accurate material push according to the interests of users, the server 200 can perform sorting processing and push the material with the highest matching degree with the interests of the user to the user. The material pushed to the user can be displayed on the graphical interfaces (exemplarily shown as graphical interface 41-1 and graphical interface 41-2) of the electronic device 400.

[0035] In some embodiments, the server 200 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms. The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected through wired or wireless communication means, which are not limited in the embodiments of the present invention.

[0036] See Figure 2 , Figure 2 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Figure 2 The electronic device 400 shown may be the above terminal and / or server 200. The electronic device 400 includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. Each component in the electronic device 400 is coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 2 all kinds of buses are labeled as the bus system 440.

[0037] The processor 410 may be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0038] The user interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as keyboards, mice, microphones, touch screen displays, cameras, and other input buttons and controls.

[0039] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memories, hard disk drives, optical disc drives, etc. The memory 450 optionally includes one or more storage devices that are physically located away from the processor 410.

[0040] The memory 450 includes volatile memory or non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be a read only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0041] In some embodiments, the memory 450 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, which will be exemplarily described below.

[0042] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0043] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB), etc.;

[0044] The presentation module 453 is used to enable the presentation of information (such as a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with the user interface 430 (such as a display screen, a speaker, etc.);

[0045] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one of one or more input devices 432.

[0046] In some embodiments, the training device of the model and the advertising material prediction device provided in the embodiments of the present application may be implemented in software. Figure 2 Shown are the training device 455 of the model and the advertising material prediction device 456 stored in the memory 450, which may be software in the form of programs and plugins, etc. The training device 455 of the model includes the following software modules: a sampling module 4551, a data processing module 4552, a scoring training module 4553, and a training module 4554; the advertising material prediction device 456 includes the following software modules: a data collection module 4561, a scoring module 4562, and a sorting module 4563.

[0047] These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0048] In some other embodiments, the training device and the advertisement material prediction device of the model provided by the embodiments of the present application can be implemented in a hardware manner. As an example, the device provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the model training method and the advertisement material prediction method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0049] In some embodiments, the recommendation system provided by the embodiments of the present application can be a material recommendation system. Hereinafter, the model training method provided by the embodiments of the present application will be described by taking the material recommendation system as an example.

[0050] Next, the model training method provided by the embodiments of the present application will be described in combination with the exemplary applications and implementations of the electronic devices provided by the embodiments of the present application.

[0051] It should be noted that, for the convenience of description, the model training device can be abbreviated as the training device.

[0052] See Figure 3 , Figure 3 is an optional flowchart of the model training method provided by the embodiments of the present application. Next, the model training method will be described in combination with the steps shown in Figure 3 to illustrate the model training method.

[0053] It should be noted that the advertisement material is an off-site advertisement for the material recommendation system placed on other application programs of the terminal. The advertisement material can be the material of the material recommendation system. That is, the advertisement material is generated based on the material released by the material recommendation system, and the advertisement material is placed in the corresponding advertisement position. The user can click on the advertisement material in the advertisement position to enter the in-site page of the application program corresponding to the material recommendation system. The application program corresponding to the material recommendation system can be Weibo.

[0054] S301, determine a training sample set based on the advertisement material set with call-end records.

[0055] In some embodiments, the call-end record is used to record the call-end behavior of a user who clicks through an advertisement material exposed on an advertisement placement platform to enter a page designated by an advertiser; the training sample set includes multiple training samples corresponding to each advertisement material. Among them, the positive samples in the multiple training samples are the matching pairs of the advertisement material and the corresponding clicking user, and the negative samples in the multiple training samples are the matching pairs of the advertisement material and the non-clicking user. A clicking user refers to a user who has call-end behavior towards the advertisement material, and a non-clicking user refers to a user who has no call-end behavior towards the advertisement material and has call-end behavior towards other advertisement materials in the same advertisement position.

[0056] In some embodiments, the exposure log of the advertisement placement platform records the exposure situations of multiple advertisement materials corresponding to each advertisement position. For example, advertisement material a placed in advertisement position A has been exposed 100 times in the past 24 hours, and advertisement material b placed in advertisement position A has been exposed 150 times in the past 24 hours. The call-end log records the call-end behavior (i.e., click behavior) of users towards the advertisement materials. For example, advertisement material a located in advertisement position A has been clicked 50 times in the past 24 hours, and advertisement material b located in advertisement position A has been clicked 65 times in the past 24 hours.

[0057] In some embodiments, an advertisement material that is exposed and clicked by a user can be used as a positive sample. For example, (s i , u 1 ) indicates that user u 1 clicks advertisement material s i , and (s i , u 1 ) is a positive sample corresponding to advertisement material s i . An advertisement material that is exposed but not clicked by a user can be used as a negative sample. For example, (s i , u m+1 ) indicates that user u m+1 does not click advertisement material s i , and (s i , u m+1 ) is a negative sample corresponding to advertisement material s i . It should be noted that in the case of an advertisement material being exposed multiple times, it can be clicked by some users and not clicked by other users. Therefore, the same advertisement material can appear as both a positive sample and a negative sample.

[0058] In some possible embodiments, the above S301 may include: determining, according to the call-end logs of the advertising platform, an advertising material set having call-end records within a preset first time period, and positive samples corresponding to each advertising material in the advertising material set; wherein, the call-end logs include all call-end records in the advertising platform, and the number of positive samples corresponding to an advertising material is equal to the number of users with call-end behavior for the advertising material; determining, according to the exposure logs, call-end logs of the advertising platform and a preset overall sampling ratio, negative samples corresponding to each advertising material in the advertising material set; wherein, the exposure logs include exposure records of each advertising material in the advertising platform, the overall sampling ratio is the ratio between positive samples and negative samples for an advertising position, and the number of negative samples corresponding to an advertising material is determined based on the exposure volume, call-end volume of the advertising material, and the overall sampling ratio.

[0059] It can be understood that the number of users who click on each advertising material can be collected from the call-end logs of the advertising platform. Based on the number of users who click on each advertising material and the users who click on the advertising material, a positive sample set corresponding to each advertising material can be generated. The number of positive samples in the positive sample set is the number of users who click on each advertising material. In order to avoid the generation of invalid negative samples, in the embodiments of the present application, negative samples are artificially constructed. It is necessary to first determine the number of negative samples corresponding to each advertising material, and then determine the negative sample set.

[0060] In some embodiments, the collection date of the advertising material set is preset. For example, according to the call-end logs updated the previous day, an advertising material set having call-end records is determined.

[0061] In some embodiments, the call-end logs are used to record the click behavior of users on sample advertising materials; the overall sampling ratio is the ratio between positive samples and negative samples of all sample advertising materials for an advertising position.

[0062] In one example, the positive sample set corresponding to advertising material s i can be expressed as {(s i , u 1 ), (s i , u 2 ), …, (s i , u m )}, that is, users u 1 , users u 2 , …, users u m respectively had click behaviors on advertising material s i .

[0063] In one example, the negative sample set corresponding to advertising material s i can be expressed as That is, user u m+1, user u m+2 , …, user did not perform a click behavior on the advertising material s i .

[0064] In some possible implementation manners, determining, according to the exposure log, the call end log of the advertising platform, and a preset overall sampling ratio, a negative sample corresponding to each advertising material in the advertising material set may include: for any advertising material in the advertising material set, obtaining the exposure amount of the advertising material and the total exposure amount of all advertising materials in the advertising position to which the advertising material belongs from the exposure log; obtaining the call end amount of the advertising material and the total call end amount of all advertising materials in the advertising position to which the advertising material belongs based on the call end record; the call end amount is used to represent the number of users who perform a call end behavior on the advertising material; calculating a negative sample coefficient corresponding to the advertising material according to the overall sampling ratio, the total exposure amount of all advertising materials, and the total call end amount of all advertising materials; multiplying the difference between the exposure amount of the advertising material and the call end amount of the advertising material by the negative sample coefficient to obtain the negative sample quantity corresponding to the advertising material; and selecting, according to the negative sample quantity corresponding to the advertising material, non-click users of the corresponding quantity of advertising materials from the negative sample candidate set of the advertising position to which the advertising material belongs to form the negative sample corresponding to the advertising material; wherein, the negative sample candidate set is a set of users who perform a call end behavior on any advertising material in the same advertising position.

[0065] It can be understood that the sum of the exposure amounts of each advertising material located in the same advertising position is the total exposure amount of all advertising materials. Similarly, the sum of the call end amounts of each advertising material is the total call end amount of all advertising materials. The overall sampling ratio is preset, and the overall sampling ratio is the ratio of the number of positive samples to the number of negative samples.

[0066] In some embodiments, the expression (1) of the negative sample coefficient is as follows:

[0067]

[0068] In formula (1), r is the negative sample coefficient, R is the overall sampling ratio, C is the total call end amount of all advertising materials, and E is the total exposure amount of all advertising materials. It can be seen from formula (1) that the negative sample coefficient corresponding to each advertising material in the advertising position (i.e., the same advertising position) is the same.

[0069] In some embodiments, the expression (2) for calculating the negative sample quantity corresponding to each advertising material is as follows:

[0070] N m = r × (e i − c i ) (2)

[0071] In formula (2), r is the negative sample coefficient, N m is the number of negative samples of the advertising material s i , e i is the exposure volume of the advertising material s i , c i is the call end volume of the advertising material s i .

[0072] In some embodiments, for the same advertising material, the positive-negative sample ratio can be expressed as m∶r×(e i -c i ). The positive-negative sample ratio is inversely proportional to e i -c i of the advertising material, that is, the larger e i -c i , the smaller the positive-negative sample ratio; the smaller e i -c i , the larger the positive-negative sample ratio. And e i -c i reflects the exposure effect of the advertising material. The smaller the difference, the larger the proportion of clicks on the advertising material, and the better the performance of the advertising material. The higher the sample importance of the advertising material with good performance, the higher the information value. Increasing the proportion of important samples will be beneficial to the training of the model (i.e., the advertising material scoring model).

[0073] In some embodiments, after determining the number of negative samples corresponding to the advertising material, a corresponding number of non-click users can be selected from the non-click users of the advertising material, and the matching pairs of these non-click users and the advertising material are the negative sample set.

[0074] In some embodiments, the advertising material s i is exposed to m+r i ×(e i -c i ) users, and the total user set is expressed as Among them, the user set corresponding to the positive samples is expressed as {u 1 , u 2 , …, um}. After filtering out the user set corresponding to the positive samples in the total user set, r i ×(e i -c i ) users are randomly selected from the remaining users as the user set corresponding to the negative samples, and the user set corresponding to the negative samples can be expressed as

[0075] In some embodiments, according to the above calculation method, after determining the number of negative samples in the negative sample set corresponding to each advertising material, the above sampling method can be verified, and the expression of the derivation process of the verification formula is as follows in formula (3):

[0076]

[0077] In formula (3), the advertisement materials in the same advertisement position are represented as {s 1 , s 2 , …, s n}, e i is the exposure volume of the advertisement material s i , and c i is the call end volume of the advertisement material s i . Formula (3) verifies that the above sampling scheme can achieve dynamic sampling based on e i -c i while ensuring that the overall sampling ratio is controllable.

[0078] In the embodiments of the present application, the negative sampling method of randomly selecting unclicked exposed advertisement materials for users from the user perspective is optimized to a sampling method of dynamically sampling unclicked exposed users for advertisement materials from the advertisement material perspective according to the exposure and click situation. The random sampling range changes from all advertisement materials to the active users corresponding to the advertisement position, increasing the distinction of the advertisement position and making it closer to the scenario of programmatic customer acquisition. And starting from the perspective of advertisement materials, it is closer to the original intention of the algorithm for constructing a high-quality material library. Moreover, the sampling with the same ratio for each user changes to dynamic sampling according to the exposure and click situation of the materials. The sampling quantity for the advertisement materials with better performance and higher value is increased, and the sampling quantity for the advertisement materials with poorer performance is decreased. Without changing the total sampling quantity, the overall quality of the positive samples and negative samples is increased.

[0079] S302. Determine the advertisement material basic features, advertisement material playback features, user basic features, user historical behavior features, and cross features between the advertisement material and the user corresponding to the training samples.

[0080] It can be understood that according to the database of the advertisement placement platform and the positive sample set and negative sample set corresponding to each advertisement material, the user basic features and user historical behavior features corresponding to the user can be determined, the advertisement material basic features and advertisement material playback features corresponding to each advertisement material, and the cross features between each user and each advertisement material.

[0081] In some embodiments, the training device can pair multiple users and multiple advertisement materials one by one to obtain the cross features between each user and each advertisement material. For example, the users include user u 1 and user u 2 , and the advertisement materials include s 1 and s 2 ; the users and advertisement materials can form 4 groups of matching pairs, which are (u 1 , s 1 ), (u1 , s 2 ), (u 2 , s 1 ), and (u 2 , s 2 ).

[0082] In some possible embodiments, the above S302 may include: the training device obtains the user's basic user characteristics in the user feature library of the material recommendation system for the training samples; in the material library of the material recommendation system, obtains the basic advertisement material characteristics of the advertisement materials in the training samples; in the call-end log of the advertisement placement platform, obtains the user's historical user behavior characteristics in the training samples; in the whole-site video playback database of the material recommendation system, obtains the advertisement material playback characteristics of the advertisement materials in the training samples; and determines the cross characteristics between the user and the advertisement materials in the training samples according to the user information of the users in the training samples and the material information of the advertisement materials.

[0083] In some embodiments, the user feature library of the material recommendation system stores the basic user characteristics of the users. The basic user characteristics are used to characterize the user's personal portrait; for example, the basic user characteristics may represent information in dimensions such as the user's age, the frequency of logging in to the material recommendation system, the first interest tag, the permanent city, the mobile phone brand, the model, etc., and may also include information in other dimensions, which is not specifically limited in the embodiments of the present application.

[0084] In some embodiments, the material library of the material recommendation system stores the basic advertisement material characteristics of each advertisement material. The basic advertisement material characteristics of the advertisement materials are used to characterize the material information of the advertisement materials; for example, the material characteristics may represent information in dimensions such as the material label, the second interest tag, the advertisement position to which the material belongs, the number of pictures, the video duration, the static characteristics of the material blogger, the number of days of material exposure, etc., and may also include information in other dimensions, which is not specifically limited in the embodiments of the present application.

[0085] In some possible embodiments, obtaining the user's historical behavior characteristics of the user in the training sample from the call-end log of the advertising placement platform may include: in the call-end log, obtaining multiple historical advertising materials clicked by the user in the training sample within a preset second time period; based on the call-end log, determining the duration coefficient of the multiple historical advertising materials; the duration coefficient is used to represent the difference between the month when the historical advertising material was released and the current month; obtaining the interest label and label score corresponding to each historical advertising material; dividing the multiple historical advertising materials into multiple groups of historical advertising materials according to the interest label; each group of historical advertising materials corresponds to an interest label; according to the label score and duration coefficient corresponding to each historical advertising material, calculating the matching score of the user for each group of historical advertising materials; sorting the multiple matching scores, and using the groups of historical advertising materials whose matching scores meet the preset threshold as the user's historical behavior characteristics.

[0086] It can be understood that the call-end data in the call-end log can be statistically analyzed in a sliding window manner to obtain the historical behavior characteristics of each user. The size of the window (i.e., the second time period) can be preset, such as one year.

[0087] In some embodiments, the training device obtains the release time of the historical advertising material in the call-end log, subtracts the current month from the month of the release time, and the obtained difference is the duration coefficient of the historical advertising material. For example, if it is December now and the call-end moment of the i-th historical advertising material occurred in October, then the duration coefficient t i = 2.

[0088] In some embodiments, each historical advertising material has a corresponding interest label (i.e., the second interest label), and the second interest label is used to represent the interest field to which the advertising material belongs. Among them, the information related to the interest label of the historical advertising material is stored in the database of the material recommendation system.

[0089] In an example, the second interest label of historical advertising material A is "entertainment star", the second interest label of historical advertising material B is "variety show", and the second interest label of historical advertising material C is "animal".

[0090] In some embodiments, each historical advertising material corresponds to a label score, and the label score is stored in the database of the material recommendation system. When determining the second interest label corresponding to each historical advertising material, the label score corresponding to each historical advertising material can be obtained.

[0091] In some embodiments, after determining the second interest tags of each historical advertisement material, the historical advertisement materials belonging to the same second interest tag can be grouped together. That is, the multiple historical advertisement materials are grouped into multiple groups of historical advertisement materials, each group includes at least one historical advertisement material, and each group of historical advertisement materials corresponds to an interest tag.

[0092] In one example, the second interest tags of historical advertisement material A and historical advertisement material D are both "entertainment stars", the second interest tags of historical advertisement material B and historical advertisement material E are both "variety shows", and the second interest tags of historical advertisement material C and historical advertisement material F are both "animals"; then, historical advertisement material A and historical advertisement material D can be classified into one group, and this group of historical advertisement materials belongs to the same type of interest tag; historical advertisement material B and historical advertisement material E can be classified into one group, and this group of historical advertisement materials belongs to the same type of interest tag; historical advertisement material C and historical advertisement material F can be classified into one group, and this group of historical advertisement materials belongs to the same type of interest tag.

[0093] In some embodiments, according to the tag score and duration coefficient corresponding to each historical advertisement material, the expression (4) for calculating the matching score of each group of historical advertisement materials by the user is as follows:

[0094]

[0095] In formula (4), the number of historical advertisement materials is n, S a represents the matching score of this group of historical advertisement materials with the interest tag a, t i represents the duration coefficient of the i-th historical advertisement material, f i represents the tag score of the i-th historical advertisement material with respect to the interest tag a.

[0096] In one example, taking the interest tags, tag scores, and duration coefficients shown in Table 1 as an example, the matching scores corresponding to the 3 groups of historical advertisement materials are calculated according to expression (4).

[0097] Table 1

[0098]

[0099] In some embodiments, taking Table 1 as an example, after calculating the matching scores of the 3 groups of historical advertisement materials, the 3 groups of historical advertisement materials can be sorted from largest to smallest. Select the historical advertisement materials corresponding to the top 2 rankings as the user's historical behavior characteristics; then, referring to Table 1, the user's historical behavior characteristics are advertisement materials a and b belonging to "entertainment stars"; and advertisement materials c, d, and e belonging to "variety shows".

[0100] In some possible embodiments, the playback features may include one or any combination of the following: actual playback volume, actual playback rate, average playback rate, x% completion rate, and completion rate. Among them, the actual playback volume is used to represent the number of times an advertising material is played; the actual playback rate is used to represent the ratio between the number of times an advertising material is played and the exposure volume of the advertising material; the average playback rate is used to represent the average value of the actual playback rates of all advertising materials in the same advertising position; the x% completion rate is used to represent the ratio between the number of times an advertising material is played when the ratio of the duration of the user playing the advertising material to the complete duration of the advertising material is greater than x% and the actual playback volume; the completion rate represents the ratio between the number of times a user completely plays an advertising material and the actual playback volume.

[0101] In some possible embodiments, in the whole-site video playback database of the material recommendation system, obtaining the playback features corresponding to the advertising materials in the training samples may include: the training device obtaining the actual playback volume and exposure volume corresponding to the advertising materials in the training samples in the whole-site video playback database; calculating the actual playback rate and average playback rate based on the actual playback volume and exposure volume corresponding to the advertising materials; obtaining the playback duration corresponding to the advertising materials in the whole-site video playback database; in the case where the advertising material is completely played, obtaining the completion rate of the advertising material by calculating the ratio of the number of times the advertising material is completely played to its corresponding actual playback volume; in the case where the advertising material is not completely played, obtaining the x% completion rate of the advertising material by calculating the ratio of the number of times the ratio of the playback duration of the advertising material is greater than x% to its corresponding actual playback volume.

[0102] It can be understood that in the programmatic customer acquisition scenario, 99% of the advertising materials are video materials. Therefore, in the embodiments of the present application, playback features that can better reflect the quality of advertising materials are introduced. Among them, the x% completion rate may specifically include: 10% completion rate, 30% completion rate, 50% completion rate, and 80% completion rate.

[0103] In some embodiments, the statistics of the video playback volume start from when the video starts to be played. However, such a statistical logic may have situations such as users accidentally clicking, accidentally touching, or quickly exiting, resulting in the statistical result often being higher than the actual video playback volume. The actual playback volume described in the embodiments of the present application sets a certain threshold, such as starting to count the playback quantity of the video only when the playback duration exceeds 5% of the total duration of the video. Such a statistical method is closer to the actual video playback situation.

[0104] In some embodiments, the actual playback rate is the ratio between the actual playback volume of an advertising material and the exposure volume of the advertising material; the average playback rate is the average value of the actual playback rates of all advertising materials in the same advertising position.

[0105] It is understandable that, in addition to the statistical features in terms of the playback volume, the features of the playback rate can also largely reflect the quality of the advertising materials to a great extent. In the case of a low playback volume, the advertising materials with a high playback rate are often high-quality advertising materials.

[0106] In some embodiments, the whole-site video playback database stores a data table recording the whole-site video playback data. In the programmatic scenario, the requirement for the timeliness of advertising materials is relatively high. Therefore, the playback data of the previous week can be retrieved in the form of a sliding window for data statistics.

[0107] In some embodiments, the playback duration corresponding to each advertising material is obtained from the whole-site video playback database; when the advertising material is played completely (i.e., the ratio of the playback duration of the advertising material to the complete duration of the advertising material is 1), the completion rate of the advertising material is calculated.

[0108] In some embodiments, when the advertising material is not played completely, by calculating the ratio of the number of times that the playback duration ratio of each advertising material is greater than x% to its corresponding actual playback volume, the x% completion rate of each advertising material is obtained.

[0109] In an example, taking the 50% completion rate as an example, calculate the advertising material s i The expression (5) of the 50% completion rate of is as follows:

[0110]

[0111] In formula (5), W 50％ represents the 50% completion rate of the advertising material s i , br represents the actual playback duration of the advertising material s i , t represents the video duration of the advertising material si, (b r / t)>50% means that the ratio of the actual playback duration of the advertising material si to the video duration is greater than 50%, (b r / t)>5% means that the ratio of the actual playback duration of the advertising material si to the video duration is greater than 5%, and Num() represents the quantity.

[0112] It should be noted that the calculation methods of other x% completion rates are the same as those of the above 50% completion rate, and the embodiments of the present application do not make specific limitations in this regard.

[0113] In some possible embodiments, the cross feature may include a type cross feature. Determining the cross feature between the user in the training sample and the advertisement material according to the user information of the user and the material information of the advertisement material in the training sample may include: determining a first label corresponding to the user and a second label corresponding to the advertisement material according to the user information of the user and the material information of the advertisement material in the training sample; when the first label and the second label are the same, determining the type cross feature between the user corresponding to the first label and the advertisement material corresponding to the second label as the first type cross feature; when the first label and the second label are different, determining the type cross feature between the user corresponding to the first label and the advertisement material corresponding to the second label as the second type cross feature.

[0114] It can be understood that according to the user information of the user, multiple first labels corresponding to the user can be determined. For example, the province where user A is located is "Shaanxi" and the city is "Xi'an", and the first labels corresponding to user A may include: "Shaanxi" and "Xi'an". According to the material information of the advertisement material, multiple second labels corresponding to the advertisement material can be determined. For example, the province where the blogger who publishes advertisement material A is located is "Sichuan" and the city is "Chengdu", and the second labels corresponding to advertisement material A may include: "Sichuan" and "Chengdu".

[0115] In some embodiments, for the province information, the first label of user A is "Shaanxi" and the second label of advertisement material A is "Sichuan". Since the first label of user A and the second label of advertisement material A are the same, the cross feature regarding the province information between user A and advertisement material A is determined as the first type cross feature, and the first type cross feature can be assigned a value of 1.

[0116] In some embodiments, for the province information, the first label of user A is "Shaanxi" and the second label of advertisement material B is "Shaanxi". Since the first label of user A and the second label of advertisement material B are the same, the cross feature regarding the province information between user A and advertisement material B is determined as the second type cross feature, and the second type cross feature can be assigned a value of 0.

[0117] In some possible embodiments, the cross features may further include a first numerical cross feature and a second numerical cross feature; the first label includes a plurality of first interest labels, and the second interest label includes a plurality of second interest labels. Determining the cross features between the user in the training sample and the advertisement material according to the user information of the user in the training sample and the material information of the advertisement material may include: determining a plurality of first interest labels corresponding to the user in the training sample, and determining a plurality of second interest labels corresponding to the advertisement material in the training sample; performing a matching process on the plurality of first interest labels of the user in the training sample and the plurality of second interest labels of the advertisement material, and determining the number of the mutually corresponding first interest labels and second interest labels as the first numerical cross feature between the user and the advertisement material; in the case that there is a first numerical cross feature between the user and the advertisement material, obtaining a preset interest label weight; one first interest label corresponds to one interest label weight; based on the first numerical cross feature and the interest label weight, determining the second numerical cross feature between the user and the advertisement material.

[0118] It can be understood that the first interest label is used to characterize the user's interest field. For example, the interest fields of user A are "entertainment stars" and "variety shows", and "entertainment stars" and "variety shows" are two first interest labels of user A. The second interest label is used to characterize the interest field of the advertisement material. For example, the interest fields of advertisement material A are "finance" and "funds", and "finance" and "funds" are two second interest labels of advertisement material A.

[0119] In some embodiments, in the case that the user and the advertisement material belong to the same interest field, the first interest label and the second interest label correspond to each other (that is, the first interest label and the second interest label are the same). For example, the interest fields of user B and advertisement material C are both "pets", then the first interest label corresponding to user B and the second interest label corresponding to advertisement material C correspond to each other.

[0120] In some embodiments, there may be multiple identical interest fields between the user's interest field and the advertisement material's interest field. For example, the interest fields of user C include "variety shows", "entertainment stars" and "TV dramas". The interest fields of advertisement material D include "entertainment stars" and "TV dramas". Then, there are two mutually corresponding interest labels between user C and advertisement material D, namely "entertainment stars" and "TV dramas", and the first numerical cross feature between user C and advertisement material D is 2.

[0121] In some embodiments, each user corresponds to a plurality of first interest labels, and each has an interest label weight corresponding to it one by one. The interest label weight can be obtained from the database of the material recommendation system.

[0122] In some embodiments, when the first interest tag hits (i.e., is the same as) the second interest tag, calculating the average matching score between the first interest tag and the second interest tag (i.e., the value of the second numerical cross - feature) can represent the degree of interest of the user in the advertisement material.

[0123] In some embodiments, based on the first numerical cross - feature and the interest tag weight, the expression of the second numerical cross - feature between the user and the advertisement material is determined as shown in formula (6) below:

[0124]

[0125] In formula (6), Avg score represents the average matching score between user u j and advertisement material s i . User u j corresponds to n first interest tags, w is the interest tag weight of the first interest tag corresponding to user u j , and user u j corresponds to n interest tag weights.

[0126] In some embodiments, taking the cross - features including 3 type - cross types and 2 numerical - cross types as an example, they are summarized in Table 2.

[0127] Table 2

[0128] Feature Name Type of Cross Feature Whether the user's province matches the material blogger's province Category Cross Feature Whether the user's city matches the material blogger's city Category Cross Feature Whether the first interest tag matches the second interest tag Category Cross Feature Number of hits between the first interest tag and the second interest tag First Numerical Cross Feature Mean matching score between the first interest tag and the second interest tag Second Numerical Cross Feature

[0129] In Table 2, the material blogger is the abbreviation of the blogger who publishes the advertisement material.

[0130] S303. The training device inputs the advertisement material basic features, advertisement material playback features, user basic features, user historical behavior features, and the cross - features between the advertisement material and the user corresponding to the training samples into the initial advertisement material scoring model for prediction processing, and obtains the first prediction score corresponding to the advertisement material in the training samples. The first prediction score is used to represent the degree of interest of the user in the advertisement material in the training samples.

[0131] It can be understood that after obtaining the user basic features and user historical behavior features corresponding to each user, and the advertisement material basic features, advertisement material playback features, and cross - features between the advertisement material and the user corresponding to each advertisement material, the above - mentioned features are input into the advertisement material scoring model for prediction scoring processing. The higher the first prediction score output by the advertisement material scoring model, the higher the degree of interest of the user in the advertisement material.

[0132] In some embodiments, during the process of training the advertisement material scoring model, user basic features, user historical behavior features, advertisement material basic features, advertisement material playback features, and cross features between the advertisement material and the user can be input into the advertisement material scoring model in the form of vectors for training.

[0133] In some embodiments, Figure 4 is an optional flowchart of the training method of the model provided by the embodiments of the present application. Refer to Figure 4 As shown, taking the advertisement material scoring model as an FM (factorization machine) model as an example for illustration.

[0134] S401, the training device collects positive and negative samples based on the exposure log and the call end log.

[0135] S402, the training device generates user basic features, user historical behavior features, advertisement material basic features, advertisement material playback features, and cross features based on the positive sample set and the negative sample set.

[0136] S403, input the user basic features, user historical behavior features, advertisement material basic features, advertisement material playback features, and cross features into the FM model for training.

[0137] In some embodiments, Figure 5 is a schematic structural diagram of the advertisement material scoring model provided by the embodiments of the present application. Refer to Figure 5 As shown, combine the user basic features and user historical behavior features corresponding to each user into user features and input them into the advertisement material scoring model, and combine the advertisement material basic features and advertisement material playback features corresponding to each advertisement material into material features and input them into the advertisement material scoring model. The cross features between the advertisement material and the user are input into the embedding layer 51 in the advertisement material scoring model for feature extraction processing. The embedding layer 51 inputs the processed data into the feature cross layer 52 for feature cross processing. After the feature cross layer 52 finishes the feature cross processing, it inputs the data into the sigmoid layer 53, and the sigmoid layer 53 is used to perform prediction scoring processing on the data and output the first prediction score.

[0138] In some embodiments, the expression (7) of the processing process of the sigmoid layer in the advertisement material scoring model for the input data is as follows:

[0139]

[0140] In formula (7), y j represents the first prediction score of user u i for advertisement material s i , n represents the number of features input into the sigmoid layer, and xi represents the eigenvalue corresponding to the i-th feature; x j represents the eigenvalue corresponding to the j-th feature, w 0 represents a preset bias value; w i represents a preset feature x i corresponding weight, w ij represents a preset feature x i and the feature x j two-dimensional cross weight of.

[0141] S304. The training device adjusts the parameters of the advertisement material scoring model according to the first prediction score corresponding to the advertisement material in the training sample and the positive and negative sample labels of the training sample until the convergence condition is reached, and obtains the trained advertisement material scoring model.

[0142] It can be understood that after obtaining the first prediction score corresponding to each advertisement material, obtain the true click behavior of the user who has a true click behavior on the advertisement material, and based on the comparison result of the first prediction score and the true click behavior, adjust the parameters of the advertisement material scoring model until the trained advertisement material scoring model is obtained.

[0143] An embodiment of the present application provides an advertisement material prediction method, Figure 6 is an optional flowchart of the advertisement material prediction method provided by the embodiment of the present application. See Figure 6 As shown, the above method may include:

[0144] S601. Obtain the advertisement material basic features and advertisement material playback features of the target advertisement material to be predicted, the user basic features and user historical behavior features of multiple active users corresponding to the advertisement position to which the target advertisement material belongs, and the cross features between the target advertisement material and each active user.

[0145] Among them, the multiple active users corresponding to the advertisement position refer to the set of users who have a calling behavior on any advertisement material of the advertisement position within a preset time period.

[0146] In some embodiments, the sample active users are the set of users who have clicked on the advertisement materials of the advertisement position within a preset time period. For example, the sample active users {u 1 , u 2 , …, u m} represent m users who have clicked on the advertisement materials of advertisement position A in the past 4 days.

[0147] S602. Input the basic characteristics and playing characteristics of the target advertising material, the basic characteristics and historical behavior characteristics of multiple active users, and the cross characteristics into the trained advertising material scoring model for prediction processing to obtain multiple second prediction scores corresponding to the target advertising material. Among them, the advertising material scoring model is trained by the above model training method, and the second prediction score is used to represent the degree of interest of active users in the target advertising material.

[0148] S603. Based on the multiple second prediction scores corresponding to the target advertising material, determine the total prediction score of the target advertising material corresponding to multiple active users, and sort the target advertising material based on the total prediction score.

[0149] It can be understood that after the advertising material scoring model is trained, it will be put into online use. During the online use process, after obtaining multiple second prediction scores corresponding to the target advertising material, the target advertising material can be sorted.

[0150] In some possible implementation manners, the above S603 may include: performing a summation calculation on the multiple second prediction scores corresponding to the target advertising material to obtain the total prediction score of the target advertising material corresponding to multiple active users; determining the difference between the total prediction score of the target advertising material and the minimum value of the multiple second prediction scores as the first difference; determining the difference between the maximum value and the minimum value of the multiple second prediction scores as the second difference; determining the ratio of the first difference to the second difference as the normalized prediction score of the target advertising material; sorting the target advertising material based on the normalized prediction score of the target advertising material.

[0151] It can be understood that the multiple second prediction scores and the total prediction score corresponding to the target advertising material can be normalized to obtain the normalized prediction score corresponding to each target advertising material.

[0152] In some embodiments, taking the number of sample active users as n as an example, the target advertising material s i corresponds to n second prediction scores, and the set of second prediction scores of the target advertising material s i can be expressed as {y 1 , y 2 , …, y n}. The expression (8) for calculating the total prediction score of the target advertising material s i is as follows:

[0153]

[0154] In formula (8), Y i represents the target advertising material s iThe total predicted score, and yj is the first predicted score of the j-th sample active user for the target advertisement material s i

[0155] In some embodiments, the expression (9) for calculating the standardized predicted score corresponding to each target advertisement material is as follows:

[0156] Y ave =(Y i -Y min ) / (Y max -Y min ) (9)

[0157] In formula (9), Ye represents the standardized predicted score corresponding to the target advertisement material s av , Y represents the total predicted score of the target advertisement material s i , Y represents the minimum value among multiple second predicted scores corresponding to the target advertisement material s i , and Y represents the maximum value among multiple second predicted scores corresponding to the target advertisement material s i . min i max i

[0158] In some embodiments, after determining the standardized predicted score corresponding to each target advertisement material, the target advertisement materials are sorted based on the standardized predicted score, and the target advertisement materials can be used as high-quality advertisement materials and placed on the advertisement positions in the next time period.

[0159] In the embodiments of the present application, after the advertisement material scoring model outputs the second predicted score, the second predicted scores corresponding to each target advertisement material are aggregated and used as the total predicted score of the target advertisement material to participate in the ranking, which can facilitate the distinction of the quality of the advertisement materials.

[0160] In the embodiments of the present application, according to the exposure log and call-end log of the advertisement placement platform and the preset overall sampling ratio, positive samples and negative samples are sampled, realizing dynamic sampling based on the exposure and click conditions of the advertisement materials in the direction of material granularity, optimizing the sampling method of negative samples, and improving the quality of positive samples and negative samples. The advertisement material scoring model trained based on this can achieve precise placement of high-quality materials.

[0161] Furthermore, in the sampling process, adding the advertisement position as a sampling dimension can be closer to the scenario of programmatic customer acquisition; and from the perspective of material granularity, a better database can be constructed.

[0162] ​​​​​Furthermore, in combination with the actual business scenario of programmatic customer acquisition, playback features that can effectively evaluate video quality are added. Using the existing call-end data, by combining interest tags, tag scores, and duration coefficients, a matching score for each group of advertising materials is calculated, and user historical behavior features that can reflect users' interest preferences are constructed. The introduction of cross features increases the overall feature dimension of the advertising material scoring model and improves the fitting ability of the advertising material scoring model for complex data.

[0163] Next, the exemplary structure of the training device 455 of the model provided in the embodiments of the present application implemented as a software module will be further described. In some embodiments, as Figure 2 shown, the software module in the training device 455 of the advertising material scoring model stored in the memory 450 may include: a sampling module 4551, configured to determine a training sample set based on an advertising material set with call-end records; the call-end records are used to record the call-end behavior of users who click on the specified page of the advertiser by being exposed to the advertising materials on the advertising platform; the training sample set includes multiple training samples corresponding to each advertising material, where the positive samples in the multiple training samples are the matching pairs of the advertising material and the corresponding clicking users, and the negative samples in the multiple training samples are the matching pairs of the advertising material and non-clicking users. A clicking user refers to a user who has a call-end behavior for the advertising material, and a non-clicking user refers to a user who does not have a call-end behavior for the advertising material and has a call-end behavior for other advertising materials in the same advertising position; a data processing module 4552, configured to determine the advertising material basic features, advertising material playback features, user basic features, user historical behavior features, and cross features between the advertising material and the user corresponding to the training samples; a scoring training module 4553, configured to input the advertising material basic features, advertising material playback features, user basic features, user historical behavior features, and cross features between the advertising material and the user corresponding to the training samples into an initial advertising material scoring model for prediction processing to obtain a first prediction score corresponding to the advertising material in the training samples; the first prediction score is used to characterize the degree of interest of the user in the advertising material in the training samples; a training module 4554, configured to adjust the parameters of the advertising material scoring model according to the first prediction score corresponding to the advertising material in the training samples and the positive and negative sample labels of the training samples until the convergence condition is reached, and obtain a trained advertising material scoring model.

[0164] In some possible embodiments, the sampling module 4551 is configured to determine, according to the call-end logs of the advertising platform, an advertising material set having call-end records within a preset first time period, and positive samples corresponding to each advertising material in the advertising material set; wherein the call-end logs include all call-end records in the advertising platform, and the number of positive samples corresponding to an advertising material is equal to the number of users having call-end behavior for the advertising material; to determine negative samples corresponding to each advertising material in the advertising material set according to the exposure logs, call-end logs of the advertising platform, and a preset overall sampling ratio; wherein the exposure logs include exposure records of each advertising material in the advertising platform, the overall sampling ratio is the ratio between positive samples and negative samples set for the advertising position, and the number of negative samples corresponding to an advertising material is determined based on the exposure volume, call-end volume, and overall sampling ratio corresponding to the advertising material.

[0165] In some possible embodiments, the sampling module 4551 is configured to determine, according to the call-end logs of the advertising platform, an advertising material set having call-end records within a preset first time period, and positive samples corresponding to each advertising material in the advertising material set; wherein the call-end logs include all call-end records in the advertising platform, and the number of positive samples corresponding to an advertising material is equal to the number of users having call-end behavior for the advertising material; to determine negative samples corresponding to each advertising material in the advertising material set according to the exposure logs, call-end logs of the advertising platform, and a preset overall sampling ratio; wherein the exposure logs include exposure records of each advertising material in the advertising platform, the overall sampling ratio is the ratio between positive samples and negative samples set for the advertising position, and the number of negative samples corresponding to an advertising material is determined based on the exposure volume, call-end volume, and overall sampling ratio corresponding to the advertising material.

[0166] In some possible embodiments, the data processing module 4552 is configured to obtain the user basic features of the users in the training samples in the user feature library of the material recommendation system; to obtain the advertising material basic features of the advertising materials in the training samples in the material library of the material recommendation system; to obtain the user historical behavior features of the users in the training samples in the call-end logs of the advertising platform; to obtain the advertising material playback features of the advertising materials in the training samples in the full-site video playback database of the material recommendation system; to determine the cross features between the users and the advertising materials in the training samples according to the user information of the users and the material information of the advertising materials in the training samples.

[0167] In some possible implementation manners, the data processing module 4552 is configured to obtain, from the call-end log, a plurality of historical advertisement materials clicked by a user in a preset second time period in the training sample; determine a duration coefficient of the plurality of historical advertisement materials based on the call-end log; the duration coefficient is used to represent the difference between the month when the historical advertisement material was released and the current month; obtain an interest tag and a tag score corresponding to each historical advertisement material; divide the plurality of historical advertisement materials into multiple groups of historical advertisement materials according to the interest tags; each group of historical advertisement materials corresponds to one interest tag; calculate a matching score of the user for each group of historical advertisement materials according to the tag score and the duration coefficient corresponding to each historical advertisement material; sort the multiple matching scores, and use the multiple groups of historical advertisement materials whose matching scores meet a preset threshold as the user's historical behavior characteristics.

[0168] In some possible implementation manners, the playback features include one or any combination of the following: actual playback volume, actual playback rate, average playback rate, x% completion rate, and completion rate; wherein, the actual playback volume is used to represent the number of times an advertisement material is played; the actual playback rate is used to represent the ratio between the number of times an advertisement material is played and the exposure volume of the advertisement material; the average playback rate is used to represent the average value of the actual playback rates of all advertisement materials in the same advertisement position; the x% completion rate is used to represent the ratio between the number of times an advertisement material is played when the ratio of the duration of the advertisement material played by a user to the complete duration of the advertisement material is greater than x% and the actual playback volume; the completion rate represents the ratio between the number of times a user plays an advertisement material completely and the actual playback volume; the data processing module 4552 is configured to obtain the actual playback volume and the exposure volume corresponding to the advertisement material in the training sample from the whole-site video playback database; calculate the actual playback rate and the average playback rate based on the actual playback volume and the exposure volume corresponding to the advertisement material; obtain the playback duration corresponding to the advertisement material in the whole-site video playback database; in the case where the advertisement material is played completely, obtain the completion rate of the advertisement material by calculating the ratio between the number of times the advertisement material is played completely and its corresponding actual playback volume; in the case where the advertisement material is not played completely, obtain the x% completion rate of the advertisement material by calculating the ratio between the number of times the ratio of the playback duration of the advertisement material is greater than x% and its corresponding actual playback volume.

[0169] In some possible implementation manners, the cross features include type cross features; the data processing module 4552 is configured to determine a first tag corresponding to a user and a second tag corresponding to an advertisement material according to the user information of the user in the training sample and the material information of the advertisement material; in the case where the first tag and the second tag are the same, determine the type cross feature between the user corresponding to the first tag and the advertisement material corresponding to the second tag as the first type cross feature; in the case where the first tag and the second tag are different, determine the type cross feature between the user corresponding to the first tag and the advertisement material corresponding to the second tag as the second type cross feature.

[0170] In some possible embodiments, the cross feature further includes a first numerical cross feature and a second numerical cross feature; the first label includes a plurality of first interest labels, and the second interest label includes a plurality of second interest labels; the data processing module 4552 is configured to determine a plurality of first interest labels corresponding to the user in the training sample, and determine a plurality of second interest labels corresponding to the advertising material in the training sample; perform a matching process on the plurality of first interest labels of the user and the plurality of second interest labels of the advertising material, and determine the number of mutually matching first interest labels and second interest labels as the first numerical cross feature between the user and the advertising material; in the case where there is a first numerical cross feature between the user and the advertising material, obtain a preset interest label weight; one first interest label corresponds to one interest label weight; based on the first numerical cross feature and the interest label weight, determine the second numerical cross feature between the user and the advertising material.

[0171] Next, the exemplary structure in which the advertising material prediction device 456 provided in the embodiment of the present application is implemented as a software module will be continued. In some embodiments, as Figure 2 shown, the software module in the advertising material prediction device 456 stored in the memory 450 may include: a data acquisition module 4561, configured to obtain the advertising material basic features and advertising material playback features of the target advertising material to be predicted, the user basic features and user historical behavior features of a plurality of active users corresponding to the advertising position to which the target advertising material belongs, and the cross features between the target advertising material and each active user; wherein, the plurality of active users corresponding to the advertising position refers to the set of users who have a calling behavior for any advertising material at the advertising position within a preset time period; a scoring module 4562, configured to input the advertising material basic features and advertising material playback features of the target advertising material, the user basic features and user historical behavior features of a plurality of active users, and the cross features into the trained advertising material scoring model for prediction processing, and obtain a plurality of second prediction scores corresponding to the target advertising material; the advertising material scoring model is trained by the above model training method, and the second prediction score is used to characterize the degree of interest of the active user in the target advertising material; a sorting module 4563, configured to determine the prediction total score of the target advertising material corresponding to a plurality of active users based on the plurality of second prediction scores corresponding to the target advertising material, so as to sort the target advertising material based on the prediction total score.

[0172] In some possible embodiments, the sorting module 4563 is configured to perform a summation calculation on multiple second prediction scores corresponding to the target advertisement material to obtain a total prediction score of the target advertisement material corresponding to multiple active users; determine a first difference as the difference between the total prediction score of the target advertisement material and the minimum value of the multiple second prediction scores; determine a second difference as the difference between the maximum value and the minimum value of the multiple second prediction scores; determine the ratio of the first difference to the second difference as the normalized prediction score of the target advertisement material; and sort the target advertisement material based on the normalized prediction score of the target advertisement material.

[0173] The embodiments of the present application provide a computer program product or a computer program, which includes computer instructions stored in a computer storage medium. The processor of the computer device reads the computer instructions from the computer storage medium, and the processor executes the computer instructions, so that the computer device executes the model training method and the advertisement material prediction method described above in the embodiments of the present application.

[0174] The embodiments of the present application provide a computer storage medium storing executable instructions, where the executable instructions, when executed by a processor, will cause the processor to execute the model training method and the advertisement material prediction method provided in the embodiments of the present application, for example Figure 3 the model training method shown and Figure 6 the advertisement material prediction method shown.

[0175] In some embodiments, the computer storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0176] In some embodiments, the executable instructions may be in the form of a program, software, software module, script, or code, and may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0177] As an example, the executable instructions may or may not correspond to a file in a file system, and may be stored as part of a file that holds other programs or data. For example, they may be stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program being discussed, or stored in multiple cooperating files (such as files that store one or more modules, subroutines, or code portions).

[0178] As an example, the executable instructions may be deployed to execute on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed at multiple sites and interconnected by a communication network.

[0179] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the present application are all included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: include: Determine a training sample set based on a collection of advertising materials with call-end records; The call-end record is used to record the call-end behavior of users who click on the advertising material exposed on the advertising delivery platform to enter the advertiser's designated page; the training sample set includes multiple training samples corresponding to each advertising material, wherein the positive samples in the multiple training samples are matching pairs of the advertising material and the corresponding click users, and the negative samples in the multiple training samples are matching pairs of the advertising material and non-click users, the click users refer to users who have call-end behavior to the advertising material, and the non-click users refer to users who have no call-end behavior to the advertising material, but have call-end behavior to other advertising materials in the same advertising position; Determining the basic features of the advertisement material, the advertisement material playback features, the basic features of the user, the historical behavior features of the user, and the cross-features between the advertisement material and the user corresponding to the training sample; Input the advertisement material basic features, advertisement material playback features, user basic features, user historical behavior features, and cross-features between advertisement material and user corresponding to the training sample into the initial advertisement material scoring model for prediction processing, and obtain a first prediction score corresponding to the advertisement material in the training sample; the first prediction score is used to represent the interest degree of the user in the training sample in the advertisement material; According to the first prediction scores corresponding to the advertising materials in the training samples and the positive and negative sample labels of the training samples, the parameters of the advertising material scoring model are adjusted until a convergence condition is reached, thereby obtaining a trained advertising material scoring model.

2. The method according to claim 1, characterized in that The step of determining a training sample set based on a set of advertisement materials having call-end records includes: Determine, according to the call end log of the advertising delivery platform, a set of advertising materials having call end records within a preset first time period, and positive samples corresponding to each advertising material in the set of advertising materials; wherein the call end log includes all call end records in the advertising delivery platform, and the number of positive samples corresponding to the advertising materials is equal to the number of users who have call end behaviors for the advertising materials; According to the exposure log, call end log and preset overall sampling ratio of the advertising delivery platform, the negative sample corresponding to each advertising material in the advertising material set is determined; wherein the exposure log includes the exposure record of each advertising material in the advertising delivery platform, the overall sampling ratio is the ratio between the positive samples and the negative samples set for the advertising position, and the number of negative samples corresponding to the advertising material is determined based on the exposure amount, call end amount and the overall sampling ratio corresponding to the advertising material.

3. The method according to claim 2, characterized in that The determining, according to the exposure log, the call end log and the preset overall sampling ratio of the advertising delivery platform, the negative sample corresponding to each advertising material in the advertising material set includes: For any advertising creative in the advertising creative set, obtaining, in the exposure log, the exposure amount of the advertising creative and the total exposure amount of all advertising creatives in the ad slot to which the advertising creative belongs; Based on the call end record, the call end volume of the advertising material and the total call end volume of all advertising materials in the advertising position to which the advertising material belongs are obtained; the call end volume is used to represent the number of users who have call end behavior to the advertising material; Calculate the negative sample coefficient corresponding to the advertising material according to the overall sampling ratio, the total exposure of all the advertising materials, and the total call volume of all the advertising materials; Multiplying the difference between the exposure amount of the advertising material and the call amount of the advertising material by the negative sample coefficient to obtain the number of negative samples corresponding to the advertising material; According to the number of negative samples corresponding to the advertising material, a corresponding number of non-clicking users of the advertising material are selected from the negative sample candidate set of the advertising space to which the advertising material belongs, to form negative samples corresponding to the advertising material; wherein the negative sample candidate set is a set of users who have call-end behavior for any advertising material in the same advertising space.

4. The method according to claim 1, characterized in that: The step of determining the basic features of the advertisement material, the advertisement material playback features, the basic features of the user, the historical behavior features of the user, and the cross-features between the advertisement material and the user corresponding to the training sample includes: In a user feature library of a material recommendation system, basic user features of users in the training sample are obtained; In a material library of a material recommendation system, basic advertising material features of the advertising materials in the training samples are obtained; Obtaining historical user behavior characteristics of users in the training sample from the call end log of the advertising delivery platform; Acquiring, from the site-wide video playback database of the material recommendation system, advertising material playback features of the advertising materials in the training samples; According to the user information of the users in the training samples and the material information of the advertising materials, the cross-features between the users in the training samples and the advertising materials are determined.

5. The method according to claim 4, characterized in that The acquiring of the user historical behavior characteristics of the users in the training sample from the call end log of the advertisement delivery platform includes: In the call end log, a plurality of historical advertisement materials clicked by the user in the training sample within a preset second time period are obtained; Determine the duration coefficients of the plurality of historical advertising materials based on the call end log; the duration coefficients are used to represent the difference between the month in which the historical advertising materials were released and the current month; Get the interest tags and tag scores corresponding to each historical advertising creative; Dividing the plurality of historical advertising materials into a plurality of groups of historical advertising materials according to the interest tags; each group of historical advertising materials corresponds to one interest tag; Calculating the matching score of the user for each group of historical advertising materials according to the label score and duration coefficient corresponding to each historical advertising material; The multiple matching scores are sorted, and the multiple groups of historical advertising materials whose matching scores meet a preset threshold are used as the user historical behavior features of the user.

6. The method according to claim 4, characterized in that The playback characteristics include one of the following or any combination thereof: real playback volume, real playback rate, average playback rate, x% completion rate and completion rate; wherein, the real playback volume is used to represent the number of times the advertising material is played; the real playback rate is used to represent the ratio between the number of times the advertising material is played and the exposure of the advertising material; the average playback rate is used to represent the average value of the real playback rates of all advertising materials in the same advertising position; the x% completion rate is used to represent the ratio between the number of times the advertising material is played by the user and the real playback volume when the ratio of the duration of the advertising material played by the user to the complete duration of the advertising material is greater than x%; the completion rate represents the ratio between the number of times the advertising material is played completely by the user and the real playback volume; The acquiring, in the whole-station video playback database of the material recommendation system, the advertising material playback features of the advertising material in the training sample comprises: In the whole-station video playback database, obtaining the actual playback volume and exposure volume corresponding to the advertising material in the training sample; Calculating the actual playback rate and the average playback rate based on the actual playback volume and exposure volume corresponding to the advertising material; Obtaining the playback duration corresponding to the advertisement material from the whole-site video playback database; In the case where the advertisement material is played completely, the completion rate of the advertisement material is obtained by calculating the ratio of the number of times the advertisement material is played completely to the corresponding actual playback volume; In the case that the advertising material is not completely played, the x% completion rate of the advertising material is obtained by calculating the ratio of the number of times the playing time of the advertising material accounts for more than x% and the corresponding actual playing volume.

7. The method according to claim 4, characterized in that The intersection feature includes a type intersection feature; The determining, based on the user information of the user in the training sample and the material information of the advertising material, cross-features between the user in the training sample and the advertising material includes: Determine, according to the user information of the user and the material information of the advertising material in the training sample, a first label corresponding to the user and a second label corresponding to the advertising material; In a case where the first tag and the second tag are the same, determining that a type intersection feature between a user corresponding to the first tag and an advertisement material corresponding to the second tag is a first type intersection feature; In the case that the first tag and the second tag are different, it is determined that the type intersection feature between the user corresponding to the first tag and the advertising material corresponding to the second tag is a second type intersection feature.

8. The method according to claim 7, characterized in that The intersection feature further includes a first numerical intersection feature and a second numerical intersection feature; the first tag includes a plurality of first interest tags, and the second interest tag includes a plurality of second interest tags; The determining, based on the user information of the user in the training sample and the material information of the advertising material, cross-features between the user in the training sample and the advertising material includes: Determine a plurality of first interest tags corresponding to the users in the training sample, and determine a plurality of second interest tags corresponding to the advertising materials in the training sample; Performing matching processing on the plurality of first interest tags of the user and the plurality of second interest tags of the advertising material, and determining the number of the mutually matching first interest tags and second interest tags as a first numerical cross feature between the user and the advertising material; In the case where there is a first numerical cross feature between the user and the advertising material, obtaining a preset interest tag weight; one first interest tag corresponds to one interest tag weight; Based on the first numerical intersection feature and the interest tag weight, a second numerical intersection feature between the user and the advertising material is determined.

9. A method for predicting advertising material, characterized in that: include: Obtaining the basic features of the target advertising material to be predicted and the advertising material playback features, the basic features of the multiple active users corresponding to the advertising slot to which the target advertising material belongs and the user historical behavior features, and the cross-features between the target advertising material and each active user; wherein the multiple active users corresponding to the advertising slot refer to a set of users who have a call-end behavior to any advertising material of the advertising slot within a preset time period; The advertising material basic features and advertising material playback features of the target advertising material, the user basic features and user historical behavior features of the multiple active users, and the cross-features are input into the trained advertising material scoring model for prediction processing to obtain multiple second prediction scores corresponding to the target advertising material; the advertising material scoring model is trained by the method according to any one of claims 1 to 8, and the second prediction score is used to represent the degree of interest of the active user in the target advertising material; Based on a plurality of second prediction scores corresponding to the target advertising creatives, predicted total scores of the target advertising creatives corresponding to the plurality of active users are determined, so as to rank the target advertising creatives based on the predicted total scores.

10. The method according to claim 9, characterized in that The determining, based on a plurality of second prediction scores corresponding to the target advertising creatives, predicted total scores corresponding to the plurality of active users of the target advertising creatives, so as to sort the target advertising creatives based on the predicted total scores, comprises: Summing up a plurality of second prediction scores corresponding to the target advertising material to obtain a total prediction score of the target advertising material corresponding to the plurality of active users; Determine a difference between the predicted total score of the target advertising material and a minimum value of the plurality of second predicted scores as a first difference; Determine a difference between a maximum value of the plurality of second prediction scores and a minimum value of the plurality of second prediction scores as a second difference; determining a ratio of the first difference to the second difference as a normalized prediction score of the target advertising creative; The target advertising creatives are ranked based on the normalized prediction scores of the target advertising creatives.

11. An advertising material prediction device, characterized in that: include: A data collection module is used to obtain the basic features of the target advertising material to be predicted and the advertising material playback features, the basic user features and user historical behavior features of multiple active users corresponding to the advertising slot to which the target advertising material belongs, and the cross-features between the target advertising material and each active user; wherein the multiple active users corresponding to the advertising slot refer to the set of users who have a call-end behavior to any advertising material of the advertising slot within a preset time period; A scoring module, configured to perform prediction processing on the advertising material basic features and advertising material playback features of the target advertising material, the user basic features and user historical behavior features of the multiple active users, and the cross-features input into the trained advertising material scoring model, so as to obtain multiple second prediction scores corresponding to the target advertising material; the advertising material scoring model is trained by the method according to any one of claims 1 to 8, and the second prediction scores are used to represent the degree of interest of the active users in the target advertising material; The ranking module is configured to determine, based on a plurality of second prediction scores corresponding to the target advertising creatives, predicted total scores of the target advertising creatives corresponding to the plurality of active users, so as to rank the target advertising creatives based on the predicted total scores.

Images