Advertisement text automatic generation system, advertisement text automatic generation method, and advertisement text automatic generation program

Abstract

To provide an advertising copy automatic generation system, an advertising copy automatic generation method, and an advertising copy automatic generation program capable of estimating a less-biased potential advertising effect to generate an advertising copy without heuristic reward design in generating an advertising copy.SOLUTION: An advertising copy automatic generation system includes: information acquisition means which acquires information necessary for a summary of an advertising copy; language model adjustment means which adjusts a language model learned in advance with a corpus prepared in advance, with the information acquired by the information acquisition means; and advertising copy generation means which performs learning in consideration of a less-biased advertising effect predicted and estimated from past distributed advertisement and distribution history thereof, using the language model adjusted by the language model adjustment means as an initial value, so as to maximize a reward to be used for the learning.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a technology for automatically generating advertising texts by artificial intelligence.

Background Art

[0002] In recent years, with the increase in Internet advertising due to the spread of the Internet, technologies for estimating advertising effects and automatically generating advertising texts have been developed.

[0003] For example, Patent Document 1 discloses an invention of an advertising text automatic creation system having a text generation means for generating texts of a plurality of advertisements, an advertising effect estimation means for estimating the advertising effects of the advertising texts, and a means for regenerating a new advertising text from the texts of the plurality of advertisements based on the advertising effects.

[0004] In addition, Non-Patent Document 1 discloses an advertising text generation model that uses the click-through rate (CTR) and the presence or absence of false words (hereinafter referred to as "prohibited words") leading to a high CTR as rewards.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Documents

[0006]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] Listing ads displayed together with search results for a user's search keyword (KW) account for a large portion of the search engine's revenue. Ads related to the user's interests can significantly increase the click-through rate (CTR). Also, in order to generate (create) an attractive ad text, since information on the landing page (LP), which is the web page displayed when the search keyword and the ad text are clicked, is an important reference, it is common to obtain the searched keyword (KW) and information on its landing page (LP) to generate a text ad.

[0008] Since the search engine displays ad texts related to the search KW, ad texts that are not relevant to the search KW will lead to a decrease in the number of display times. Even if an ad is generated according to the user's interests, if it deviates from the content of the LP, the conversion rate (the percentage of users who have taken a specific action set for each LP, such as purchasing a product or requesting materials) may decrease. Therefore, in order to generate an ad text that is attractive to users, it is necessary to consider both the search KW and the LP.

[0009] However, for multiple search KWs, it is time-consuming to generate text ads that conform to the content of the LP and, moreover, to generate attractive ad texts that can appeal to the interests of users with in-depth knowledge of the domain and products.

[0010] In addition, the combination of KW and LP increases the number of text ads per LP. Therefore, it is difficult to generate ad texts corresponding to all effective combinations of KW and LP due to human resource constraints.

[0011] In view of such circumstances, technologies for automating the generation of ad texts as described above have been proposed. In those ad text automatic generation technologies, it is common to use templates that automatically insert specific search KWs into the ad texts. However, the ad texts generated by the template-based approach lack flexibility and still require manual modification.

[0012] Therefore, in recent years, technologies have been proposed to generate search-linked ad texts that are highly CTR and natural as if written by humans from the LP text using reinforcement learning with the CTR included in the reward. In such ad text generation using reinforcement learning, reward design and the base model are important. However, in reward design, if the model learns to maximize the CTR blindly, the possibility increases that the model will include information different from the LP in the ad text. In addition, since the CTR is assigned to the ad text rather than to each output word, it is unclear which word contributed to the CTR, and it is difficult to learn which word to output to achieve a high CTR. Additionally, since the CTR itself also depends on the posting rank, it is necessary to remove such bias. This is because ad texts with relatively large budgets tend to have higher posting ranks, and there is a risk of imitating such ad texts from the publishers.

[0013] It is also possible to use an index such as a quality score as the reward. However, since this is assigned to the combination of the set of ad texts and the search keywords, it is unclear which ad text has a high quality score, and it is difficult to make the reward at the sentence level.

[0014] On the one hand, for the base model, a language model pre-trained on a large corpus is required. Since it is difficult to obtain a publicly available Japanese generation model suitable for generating advertising texts, it is necessary to prepare one independently.

[0015] As described above, in the field of automatic advertising text generation technology, when generating an advertising text, a model that generates an advertising text based on the content of the LP while ensuring the relevance to the search KW is required. Furthermore, for generating more attractive advertising texts, a devised reward design and reinforcement learning based on it are important.

[0016] However, the advertising text automatic creation system described in Patent Document 1 above does not separately predict the advertising effect due to factors other than the advertising text, such as the attributes of the advertiser, the advertising budget amount, and the distribution settings, and the advertising effect due to the advertising text. Therefore, it is impossible to distinguish between the improvement of the advertising effect of the newly generated advertising text, that is, the bias-reduced potential advertising effect, and the bias of the predicted value of the advertising effect caused by factors other than the advertising text, such as the attributes of the advertiser inferred from the advertising text, the advertising budget amount, and the distribution settings, and there may be cases where false expressions are used during generation.

[0017] In addition, since the advertising text generation model described in Non-Patent Document 1 above lists prohibited words visually and gives a negative reward when they are included in the output, there is a risk of including errors caused by heuristic methods, and although a solution close to the correct answer can be obtained to a certain level, it does not always lead to the correct answer. As a result, when all prohibited words cannot be listed, problems such as outputting a brand name with a high CTR as an advertising text of another company may occur.

[0018] The present invention has been made in view of the above circumstances, and an object thereof is to provide an automatic advertising text generation system, an automatic advertising text generation method, and an automatic advertising text generation program that can estimate a bias-reduced potential advertising effect and generate an advertising text without heuristic reward design in advertising text generation.

Means for Solving the Problems

[0019] In order to achieve the above object, an advertisement text automatic generation system according to a first aspect includes: an information acquisition unit that acquires information necessary for summarizing an advertisement text; a language model adjustment unit that adjusts a language model pre-trained with a corpus using the information acquired by the information acquisition unit; and an advertisement text generation unit that performs learning considering a bias-relaxed advertisement effect predicted and estimated from past distributed advertisements and their distribution histories, with the language model adjusted by the language model adjustment unit as an initial value, and generates an advertisement text so that the reward used for the learning becomes maximum. The advertisement text automatic generation system is provided with these units.

[0020] According to the advertisement text automatic generation system according to the first aspect, by adding a simple output layer to a model that has learned language characteristics with a corpus and adjusting it, that is, by performing fine-tuning (transfer learning), a high-performance advertisement text automatic generation model can be obtained.

[0021] Then, using the language model generated by this fine-tuning as an initial value, learning considering a bias-relaxed advertisement effect is performed. Generally, indices for estimating advertisement effects such as CTR vary depending on factors such as budget and placement position, resulting in bias. In order to perform learning considering the advertisement effect with this bias relaxed, it is possible to estimate the potential advertisement effect without heuristic reward design and automatically generate an advertisement text.

[0022] In an advertisement text automatic generation system according to a second aspect, in the advertisement text automatic generation system according to the first aspect, the information acquired by the information acquisition unit is a keyword (KW) and a landing page (LP).

[0023] According to the advertisement text automatic generation system according to the second aspect, since fine-tuning is performed to generate an advertisement text from KW and LP using a pre-trained model, it is possible to automatically generate a more attractive advertisement text without heuristic reward design.

[0024] In the advertisement text automatic generation system according to the third aspect, in the advertisement text automatic generation system according to the first aspect, the bias-relaxed advertisement effect is the bias-relaxed click-through rate (CTR).

[0025] According to the advertisement text automatic generation system according to the third aspect, the bias-relaxed CTR is used as an index of the advertisement effect. Since a click represents the user's actions on the web and also leads to actions such as transition to the LP or purchase on the page (conversion), it is possible to estimate the advertisement effect using the CTR as an objective and useful index for advertisement effect estimation and automatically generate the advertisement text.

[0026] In the advertisement text automatic generation system according to the fourth aspect, in the advertisement text automatic generation system according to the first aspect, the bias-relaxed advertisement effect is the advertisement effect of a potential advertisement text obtained by excluding the advertisement effect of elements other than the advertisement text.

[0027] In the advertisement text automatic generation system according to the fourth aspect, by excluding the advertisement effect of elements other than the advertisement text from the advertisement effect, the advertisement effect of the potential advertisement text is estimated. As a result, it is possible to estimate the bias-relaxed potential advertisement effect and automatically generate the advertisement text without heuristic reward setting.

[0028] In the advertisement text automatic generation system according to the fifth aspect, in the advertisement text automatic generation system according to the third aspect, the bias-relaxed CTR is obtained by subtracting the logit of the CTR predicted from features independent of the advertisement text from the logit of the actual CTR and predicting it from the advertisement text.

[0029] According to the advertisement text automatic generation system according to the fifth aspect, the difference between the logit of the actual CTR and the logit of the CTR predicted from the feature quantities independent of the advertisement text is used as the target variable. In order to give rewards excluding factors that affect the CTR regardless of the content of the advertisement text such as budget, distribution settings, and placement positions, advertisement text generation can be performed without heuristic reward design so that the advertisement effect of potential advertisement texts with higher advertisement effects, excluding the advertisement effects of factors other than the advertisement text such as the attributes of the advertisement publisher, the budget amount of the advertisement, and the distribution settings, can be achieved.

[0030] In the advertisement text automatic generation system according to the sixth aspect, in the advertisement text automatic generation system according to the fifth aspect, the CTR predicted from the feature quantities independent of the advertisement text is estimated by treating at least one of the budget and the quality score that determine the advertisement placement rank as a proxy variable for the unobtainable placement rank.

[0031] According to the advertisement text automatic generation system according to the sixth aspect, at least one of the budget and the quality score that determine the advertisement placement rank is used as a proxy variable for the placement rank, and the CTR predicted from the feature quantities independent of the advertisement text is calculated.

[0032] Generally, although the placement rank affects the CTR, the method of treating the placement position as explicit input data can only be performed from the perspective of the advertisement platform side. Users of the advertisement platform cannot even obtain information such as the average placement rank and can only obtain information such as the ratio displayed at the top of the page.

[0033] And since the budget and the quality score that determine the advertisement placement rank are associated with feature quantities independent of the advertisement text such as campaigns and advertisement groups, by treating the budget and the quality score as proxy variables for the unobtainable placement rank, the CTR predicted from the feature quantities independent of the advertisement text can be calculated. As a result, advertisement text generation can be performed without heuristic reward design so that the advertisement effect of potential advertisement texts with higher advertisement effects, excluding the advertisement effects of factors other than the advertisement text such as the attributes of the advertisement publisher, the budget amount of the advertisement, and the distribution settings, can be achieved.

[0034] In the advertisement text automatic generation system according to the seventh aspect, in the advertisement text automatic generation system according to the fifth aspect, the bias-relaxed CTR has its bias relaxed by two-stage learning, and in the second stage of the two-stage learning, the CTR is predicted using the objective variable based on the difference in logits, and only the predicted value of the CTR is used as the reward.

[0035] According to the advertisement text automatic generation system according to the seventh aspect, the bias is relaxed by two-stage learning. The predicted value at the second stage when predicting the CTR in two stages (hereinafter referred to as "relative CTR") can predict the ranking of the CTR in the same setting with a higher correlation coefficient than the value of the CTR when predicted in only one stage (hereinafter referred to as "absolute CTR"). As a result, without heuristic reward design, it is possible to generate advertisement texts such that the advertisement effect of potential advertisement texts excluding the advertisement effects of elements other than the advertisement text, such as the attributes of the advertisement publisher, the budget amount of the advertisement, and the distribution settings, becomes higher than when predicted in only one stage.

[0036] In the advertisement text automatic generation system according to the eighth aspect, in the advertisement text automatic generation system according to the first aspect, the advertisement has a hierarchical structure of advertisement account, campaign, advertisement group, and advertisement text, and the bias-relaxed advertisement effect is the one in which the bias generated at the hierarchy above the advertisement group is relaxed.

[0037] According to the advertisement text automatic generation system according to the eighth aspect, the search keyword is set for the advertisement group, and the budget and distribution settings are set at the hierarchy above the advertisement group. By relaxing the bias at the advertisement group level generated by these factors, without heuristic reward design, it is possible to generate advertisement texts such that the advertisement effect of potential advertisement texts excluding the advertisement effects of elements other than the advertisement text, such as the attributes of the advertisement publisher, the budget amount of the advertisement, and the distribution settings, becomes higher.

[0038] In the advertisement text automatic generation system according to the ninth aspect, in the advertisement text automatic generation system according to the first aspect, the learning is offline reinforcement learning.

[0039] According to the advertisement text automatic generation system according to the ninth aspect, since learning is performed by offline reinforcement learning, compared with the case of using an autoregressive model learned by maximum likelihood estimation (hereinafter referred to as "MLE") for teacher data, "exposure bias" can be reduced. Also, compared with the case of using online reinforcement learning, efficiency can be improved.

[0040] In text generation, it is mainstream to use an autoregressive model learned by maximum likelihood estimation (MLE) for teacher data, but there is a discrepancy between the function to be optimized (negative log-likelihood) and the evaluation (human judgment) in this method. In addition, during learning, the next word is predicted based on the correct data, while during inference, it is conditioned on the result generated by the model. This is called "exposure bias", which may cause problems such as the generation of repetitions of words and phrases.

[0041] In the advertisement text automatic generation system according to the tenth aspect, in the advertisement text automatic generation system according to the ninth aspect, the offline reinforcement learning is performed offline using only data on the bias-relaxed advertisement effect predicted and estimated from the advertisements distributed in the past and their distribution histories.

[0042] According to the advertisement text automatic generation system according to the tenth aspect, generally, when elements other than the advertisement effect are added to the reward, the nature of the advertisement text generated by the reward design varies greatly, and various ingenuities are required for the reward design. However, since the present invention performs learning using only data on the advertisement effect, it is possible to perform learning efficiently.

[0043] In the advertisement text automatic generation system according to the 11th aspect, in the advertisement text automatic generation systems according to the 1st to 10th aspects, the bias-relaxed advertisement effect is predicted in word units, and the predicted value of the advertisement effect is added to the reward.

[0044] According to the advertisement text automatic generation system according to the 11th aspect, the bias-relaxed advertisement effect is predicted in word units. Usually, an index for estimating an advertisement effect such as CTR is given to the advertisement text rather than each output word, so it is unclear which word has contributed to the advertisement effect, and it is difficult to learn which word should be output to produce a higher advertisement effect. In addition, since the index of the advertisement effect itself also depends on the posting order, it is necessary to remove the bias.

[0045] By predicting this bias-relaxed advertisement effect in word units rather than in sentence units and adding the value to the reward, it is possible to tell the model which word contributes to the CTR, and it is possible to accelerate the convergence of learning.

[0046] The advertisement text automatic generation method according to the 12th aspect includes an information acquisition step of acquiring information necessary for summarizing the advertisement text, a language model adjustment step of adjusting a language model pre-learned by a prepared corpus with the information acquired in the information acquisition step, and using the language model adjusted in the language model adjustment step as an initial value, performing learning in consideration of the bias-relaxed advertisement effect predicted and estimated from past advertisements and their delivery histories, and generating an advertisement text so that the reward used in the learning is maximized.

[0047] Further, the advertisement text automatic generation program according to the 13th aspect causes a computer to function as information acquisition means for acquiring information necessary for summarizing an advertisement text, language model adjustment means for adjusting a language model pre-trained by a corpus using the information acquired by the information acquisition means, and advertisement text generation means for performing learning in consideration of a bias-relaxed advertisement effect predicted and estimated from past advertisements and their distribution histories, with the language model adjusted by the language model adjustment means as an initial value, and generating an advertisement text so that the reward used for the learning becomes maximum.

[0048] According to the advertisement text automatic generation method according to the 12th aspect and the advertisement text automatic generation program according to the 13th aspect, a high-performance advertisement text automatic generation model can be obtained by adding a simple output layer to a model that has learned language characteristics by a corpus and adjusting it, that is, by fine-tuning.

[0049] Then, using the model generated by this fine-tuning as an initial value, learning is performed in consideration of a bias-relaxed advertisement effect. Generally, indicators for estimating advertisement effects such as CTR change depending on factors such as budget and placement position, resulting in bias. By performing learning in consideration of the advertisement effect with this bias relaxed, it becomes possible to estimate the potential advertisement effect and automatically generate advertisement texts without heuristic reward design.

Advantages of the Invention

[0050] The present invention enables the generation of attractive advertising texts by fine-tuning a language model pre-trained with both KW and LP as inputs and performing off-line reinforcement learning. In particular, in reinforcement learning, by using the potential click-through rate (CTR) estimated from the delivery history of text advertisements as a reward, it becomes possible to automatically generate more attractive text advertisements. As described above, the advertising text automatic generation system, advertising text automatic generation method, and advertising text automatic generation program according to the present invention have an excellent effect that, in advertising text generation, they can estimate a bias-relaxed potential advertising effect and automatically generate an advertising text without heuristic reward design.

Brief Description of Drawings

[0051]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Embodiments for Carrying Out the Invention

[0052] The present invention aims to solve the difficulties of conventional advertising text generation and automatically generate advertising texts based on LP and search KW in order to generate more attractive advertising texts (Objective 1: Automatic generation of advertising texts), and for reinforcement learning, to enable the generation of human-like texts without heuristic reward design and to make the learning efficient (Objective 2: Generation of human-like texts by reinforcement learning and improvement of learning efficiency).

[0053] As will be described in detail below, in the present invention, in order to enable the generation of human-like sentences without heuristic reward design by reinforcement learning, a bias-relaxed CTR is used.

[0054] Also, in order to solve the difficulty of reinforcement learning, by using the maximum likelihood estimation (MLE) model for the action policy, it is possible to assign higher weights to words with low occurrence probabilities.

[0055] Furthermore, by making reinforcement learning an offline learning and performing learning using only data on advertising effects, it is possible to perform reinforcement learning more efficiently.

[0056] <First Embodiment> Hereinafter, with reference to FIGS. 1 to 5, an advertisement sentence automatic generation system according to a first embodiment of the present invention will be described.

[0057] The advertisement sentence automatic generation system according to the present invention pre-learns a language model from a corpus and fine-tunes it to create a model capable of automatically generating advertisement sentences (shown as process A in FIG. 2 and hereinafter referred to as "Method A"). Then, for learning considering the bias-relaxed advertising effect, for example, in this embodiment, a bias-relaxed CTR is predicted and used as a reward for offline reinforcement learning (shown as process B in FIG. 2 and hereinafter referred to as "Method B").

[0058] Here, the "advertising effect" refers to the reach of an advertisement and the influence it has on the psychology and decision-making of consumers and on the sales of products. In this embodiment, the CTR is used as an advertising effect or an index thereof, but it is not limited to this. For example, it may be the number of impressions, which is the number of times the advertisement is displayed, or the conversion rate obtained by dividing the number of conversions (such as material requests or product purchases), which are the final products obtained by clicking on the advertisement and transitioning to the LP, by the number of clicks.

[0059] The "biased-relaxed advertising effect" refers to the advertising effect of potential advertising texts excluding the advertising effects of elements other than the advertising text.

[0060] FIG. 1 is a functional block diagram for explaining a configuration example of an advertising text automatic generation system according to an embodiment of the present invention. Specifically, as shown in FIG. 1, an advertising text automatic generation system 100 according to the present embodiment includes a corpus acquisition functional means 16 for acquiring a corpus 36 (see FIG. 2 described later) prepared in advance, a pre-training functional means 18 for learning language characteristics by the corpus 36, a KW·LP acquisition functional means 20 for acquiring KW and LP, an adjustment functional means 22 for fine-tuning a language model, an advertising delivery result acquisition functional means 24 for acquiring the advertising delivery results, an advertising delivery feature quantity extraction functional means 26 for extracting feature quantities independent of the advertising text, a CTR estimation functional means 28 for estimating the CTR, a bias calculation functional means 30 for calculating the bias of the CTR caused by the budget, the placement position, etc., an advertising text generation functional means 32 for generating an advertising text by a model in which reinforcement learning is performed with the CTR with the bias relaxed as a reward, and a database (DB) 34 in which the acquired information is stored.

[0061] Here, the KW·LP acquisition functional means 20 for acquiring the above KW and LP functions as information acquisition means for acquiring information necessary for summarizing the advertising text in the advertising text automatic generation system according to the present invention. Further, the adjustment functional means 22 for fine-tuning the above language model functions as a language model adjustment means for adjusting the language model pre-learned by a prepared corpus in advance with the information acquired by the information acquisition means in the advertising text automatic generation system according to the present invention. Furthermore, the advertising text generation functional means 32 for generating an advertising text by a model in which reinforcement learning is performed with the CTR with the bias relaxed as a reward functions as an advertising text generation means for performing learning considering the biased-relaxed advertising effect predicted and estimated from the past delivered advertisements and their delivery histories, with the language model adjusted by the language model adjustment means as an initial value, and generating an advertising text so that the reward used for the learning becomes maximum in the advertising text automatic generation system according to the present invention.

[0062] That is, the advertisement text automatic generation system according to the present invention includes an information acquisition means for acquiring information necessary for summarizing the advertisement text, a language model adjustment means for adjusting a language model pre-trained by a corpus prepared in advance according to the information acquired by the information acquisition means, and using the language model adjusted by the language model adjustment means as an initial value, learning is performed considering the bias-relaxed advertisement effect predicted and estimated from the advertisements distributed in the past and their distribution histories, and advertisement text generation means for generating an advertisement text so that the reward used for the learning is maximized.

[0063] In the aspect shown in FIG. 1, since the KW·LP acquisition function means 20, which is an information acquisition means, acquires KW and LP as information necessary for summarizing the advertisement text, in the advertisement text automatic generation system according to the present invention, the information acquired by the information acquisition means can be the keyword and the landing page.

[0064] Next, with reference to FIG. 2, a process (method A) of creating a model capable of automatically generating an advertisement text by pre-training a language model with a corpus and fine-tuning it by the advertisement text automatic generation system according to the present invention, and, as an example of learning considering the bias-relaxed advertisement effect, a process (method B) of predicting the bias-relaxed CTR and performing offline reinforcement learning using it as a reward will be described.

[0065] [Method A: Automatic generation of advertisement text using a pre-trained model] Next, the automatic generation of advertisement text (method A) using a pre-trained model will be described with reference to FIG. 2.

[0066] FIG. 2 is a block diagram for explaining an example of the processing in the advertisement text automatic generation system of the present invention, and in this figure, an overall image of an offline reinforcement learning method for generating advertisement text is conceptually shown. The example shown in this figure is the processing of automatic generation of advertisement text using pre-trained Japanese T5, and Wikipedia (38), OSCAR (40), and CC-100 (42) are used for the pre-trained Japanese corpus 36. A tensor-form dataset is created from these corpora.

[0067] In the present embodiment, T5 is adopted as the language processing model for the ease of pre-training. Here, T5 is a machine learning model that utilizes "transfer learning" which transfers a pre-trained model in one area to another area, and it is an Encoder-Decoder model using Transformer that performs learning in two stages of pre-training and fine-tuning. The input text is randomly masked and then input to the encoder, and the input is converted into an internal status. The decoder predicts what the mask was, and the internal status is converted into an output. Since the problem setting of advertisement text generation is Sequence-to-sequence (Seq2seq), either T5 or UNILM (Unified Language Model) is suitable as the language processing model.

[0068] (Pre-training) T5, which is an Encoder-Decoder model using Transformer of the deep learning model, has five models: small, base, large, 3B, and 11B according to its size. Compared with the standard base model, the small model has half the number of layers, and the large model has twice the number of layers. The 3B model and the 11B model are models with even more increased dimensions and numbers of hidden layers. As described above, in the present embodiment, considering the constraints on computing resources and inference speed during generation, the T5-base model is used as the language processing model for the ease of pre-training.

[0069] (Fine-tuning) For generating an advertisement text, maximum likelihood estimation (MLE) based on the following Equation 1 is used. That is, using the pre-trained model of T5-base, fine-tuning is performed to generate an advertisement text from the information necessary for summarizing the advertisement text (in this embodiment, LP and search keywords obtained by the KW·LP acquisition function 20). In Equation 1, L(θ) is the loss function of MLE when p human (y|x) is the data generation distribution, and the generation model is the conditional probability model p θ (y|x) parameterized by θ (Equation 2 below). For optimization, L(θ) is estimated using the training data, and θ is repeatedly updated by the gradient method to do.

[0070] [Number]

[0071] [Number]

[0072] The model obtained by MLE (MLE model) is used as the initial value for subsequent reinforcement learning. Thereby, it is possible to avoid the problem that no reward is actually obtained during learning and the policy gets stuck.

[0073] [Method B: Offline Reinforcement Learning Using CTR with Relaxed Bias] Next, with reference to FIGS. 2 to 5, offline reinforcement learning using CTR with relaxed bias (Method B) will be described.

[0074] (Dataset) In this embodiment, a dataset of search keywords, LPs, and advertisement texts collected by scraping is used. The number of cases is 741,506 for training data, 62,320 for validation data, and 20,000 for test data. Also, for the training of CTR, 373,932 pieces of delivery performance data are used, and the training:validation:test data is randomly split at a ratio of 3:1:1.

[0075] (Hyperparameter) Also, in this embodiment, unless otherwise specified, in all learning, the learning rate of Adam is grid-searched and determined from {4×10 -5 , 1×10 ―4 , 1×10 -3}. For the number of epochs, it is determined by performing early stopping on the validation data. For the batch size, it is determined by performing grid search from {64, 256, 512}.

[0076] In this embodiment, a more general offline reinforcement learning without using a mask is adopted. Specifically, in addition to the GOLD offline reinforcement learning algorithm proposed for language generation, offline reinforcement learning is performed by using the MLE model as the behavior policy.

[0077] In GOLD, for the weight of importance sampling, it is approximated assuming that the behavior policy is uniform. However, as shown in FIG. 3, in this embodiment, the behavior policy is estimated using the MLE model.

[0078] That is, the estimated value of the MLE model shown in the following Equation 3 is also used as the estimated value of the behavior policy of the importance weight used for off-policy importance sampling, as shown in the following Equation 4.

[0079]

Equation

[0080]

Number

[0081] As a result, it is possible to consider the occurrence probability of samples that were not considered in GOLD. It becomes possible to assign higher weights to samples with low occurrence probabilities, and it becomes possible to learn samples that are rare cases but have high rewards.

[0082] Regarding reward design, use the bias-relaxed CTR. Here, the bias-relaxed CTR means the difference between the logit of the actually predicted CTR and the logit of the CTR predicted from feature quantities that do not depend on the ad text such as the distribution setting, budget, campaign, and ad group, predicted from the ad text.

[0083] (Estimation of relative CTR and bias relaxation) Next, the details of the estimation of relative CTR and bias relaxation will be described.

[0084] Figure 4 is a conceptual diagram showing an example of the process of estimating relative CTR and bias relaxation in the ad text automatic generation system shown in Figure 1. Here, what is marked as "GBDT" in Figure 4 is an abbreviation for Gradient Boosting Decision Tree. Also, what is marked as "TD" is an abbreviation for Title & Description, meaning ad text such as the title and description text.

[0085] FIG. 5 is a conceptual diagram showing the hierarchical structure of an advertisement in the advertisement text automatic generation system shown in FIG. 1. As shown in FIG. 5, the advertisement has a hierarchical structure of an advertisement account 58, a campaign 60, an advertisement group 62, and an advertisement text 64. The top-level advertisement account 58 is a set of campaigns 60 and refers to something that can be grouped by only one advertising medium. The campaign 60 is a set of advertisement groups 62 and refers to something that can manage the daily budget. The advertisement group 62 refers to a set of advertisements, that is, the advertisement text 64 actually displayed on the advertising medium, a set of keywords, and a set of LP URLs.

[0086] Referring to FIG. 4, in terms of relative CTR, in the first stage, the CTR is estimated from feature quantities that do not depend on advertisement texts such as the delivery setting, budget 46, campaign 60, and advertisement group 62. Also, it is generally known that the CTR changes depending on the placement position and the placement rank. Since the budget 46 and the quality score that determine this placement rank are associated with the campaign 60 and the advertisement group 62, they are treated as proxy variables for the unobtainable placement rank. As the model, LightGBM, which is a type of Gradient Boosting Decision Tree (GBDT), is used.

[0087] Thereafter, the CTR in the second stage is predicted using the pre-trained encoder of T5. To suppress bias, only the predicted value in the second stage is used as the reward in the reinforcement learning.

[0088] As an example thereof, the following processing is described in FIG. 4. That is, in the first stage, a CTR baseline estimation based on covariates is performed using the bidding strategy 44, the budget 46, and the advertisement group 62. In the second stage, the difference from the evaluation value for the pure TD48 excluding biases such as the bidding strategy 44, the budget 46, and the placement position is predicted by T5, and the difference from the CTR baseline 50 (relative CTR 52) is predicted.

[0089] Furthermore, it is also possible to configure the learning such that the logarithm of both the 5-fold Out-of-Fold (OOF) prediction value (OOF prediction value) in the first stage and the relative CTR52 are added together (predicted CTR54) to match the observed CTR (actual CTR56).

[0090] (Advertisement Hierarchical Structure and Bias Mitigation) As described with reference to FIG. 5, the advertisement has a hierarchical structure of advertisement account 58, campaign 60, ad group 62, and ad text 64. The top-level advertisement account 58 is a set of campaigns 60.

[0091] The search keyword is set for the ad group 62, and the budget 46 and the distribution setting are set at a level higher than the ad group 62. Therefore, bias at the ad group 62 level is likely to occur with respect to the CTR.

[0092] For example, in relation to the budget 46, when the budget 46 is small, the advertisement is preferentially distributed to users who are likely to click on the medium, so the CTR tends to be high. On the other hand, when the budget 46 is large, the advertisement is distributed including users who are difficult to click in order to consume the budget 46, so the CTR tends to be low. Also, in relation to the distribution setting, depending on the pre-set distribution setting, the target users are different and the CTR is biased.

[0093] Since these effects do not depend on the ad text, if the CTR is predicted without mitigating the bias, the CTR may be estimated to be high for advertisements with a small budget 46 by chance, and even if a reward is given to such ad texts, the actual CTR does not necessarily increase.

[0094] Therefore, for the estimation of the CTR in the first stage, it is desirable to use the advertisement account ID, campaign ID, ad group ID, search keyword ID, daily budget, and distribution setting, which are common explanatory variables at the ad group 62 level or higher, to suppress the bias occurring at the ad group 62 level or higher.

[0095] Next, the operation and effects of the advertisement text automatic generation system according to the present embodiment will be described.

[0096] Tables 1 to 3 show the results of the comparative evaluation performed manually. For the same search KW·LP, advertisement texts were generated (the texts made by humans were used as they were), and it was asked which one would be more likely to be clicked. That is, Tables 1 to 3 are the verification results of the effectiveness of offline reinforcement learning using relative CTR. 1,000 pieces were extracted from the test data, and a manual comparative evaluation was performed on 64 advertisement texts made by humans, 64 advertisement texts generated by the MLE model (without performing reinforcement learning) (referred to as "MLE" in the table), 64 advertisement texts generated by the offline reinforcement learning model using relative CTR as the reward (referred to as "RL (relative CTR)" in the table), and 64 advertisement texts generated by the offline reinforcement learning model using absolute CTR as the reward (referred to as "RL (absolute CTR)" in the table).

[0097] In Table 1, it was asked which of the text made by humans and the generated text of the RL (relative CTR) model was more likely to be clicked. The result shows that the generated text by RL (relative CTR) is statistically significantly "more likely to be clicked". That is, it can be seen that the advertisement text generated by reinforcement learning using relative CTR is evaluated to be more likely to be clicked than the one made by humans.

[0098]

Table 1

[0099] In Table 2, it was asked which of the generated text of the MLE model and the generated text of the RL (relative CTR) model was more likely to be clicked. The result shows that the generated text by RL (relative CTR) is statistically significantly "more likely to be clicked".

[0100]

Table 2

[0101] In Table 3, it was investigated which of the generated sentences of the RL (absolute CTR) model and the RL (relative CTR) model was more likely to be clicked. As a result, it was found that the generated sentences by RL (relative CTR) were statistically significantly "more likely to be clicked".

[0102]

Table 3

[0103] Thus, from Tables 2 and 3, it is evaluated that the advertisement sentences output by the MLE model without reinforcement learning or the reinforcement learning model using absolute CTR are less likely to be clicked than the advertisement sentences of the reinforcement learning model using relative CTR.

[0104] Table 4 shows the results of the absolute evaluation performed manually.

[0105] In Question Q1, fluency was compared. The proportion of "no problem" in the sentences created by humans was the lowest. The proportion of "no problem" in the MLE model was also comparable to that of the sentences created by humans, while the proportion of "obviously problematic" was about twice that of the sentences created by humans. This suggests that the fluency when the generation is successful is comparable to that of the sentences created by humans, but there is a possibility of failure in the output. For the two models using reinforcement learning, the fluency of both was comparable. The proportion of the answer "no problem" was higher than that of humans and the MLE model, indicating that the fluency has not been impaired by reinforcement learning.

[0106] In Question Q2, it was asked whether the advertisement sentences looked as if they were written by humans, and the responses of "yes" and "no" were tabulated. Among the sentences created by humans, the proportion of "yes" was the lowest even though they were actual displayed advertisement sentences. The two models of the MLE model and RL can be said to be more human-like advertisement sentences than those created by humans. Due to the RL reward, the proportion of "yes" did not change significantly.

[0107] In Question Q3, regarding the relevance to the search query, the responses of "Yes" and "No" were tabulated. Since the sentences created by humans were the advertisements displayed for the actually given search query (search keyword), the percentage of "Yes" was the highest. For the MLE model, the percentage of "Yes" was the lowest. Next, RL using absolute CTR was the second lowest, and RL using relative CTR was more relevant than the sentences created by humans.

[0108] In the bottom row of Table 4, the average after scaling the responses to each question item of Q1 - Q3 above to 0 - 1 was shown. Regarding fluency, 1.0 was assigned to "No problem", 2 / 3 to "Can't say either way", 1 / 3 to "Some problems", and 0.0 to "Obviously a problem". For the others, 1.0 was assigned to "Yes" and 0.0 to "No". The result was that RL (relative CTR), the proposed method, was the highest. Next, RL (absolute CTR) had good results, and MLE and the sentences created by humans had the same score. In the absolute evaluation, the model using relative CTR received a higher evaluation in terms of human - likeness than that using absolute CTR, which was the main reason why the method proposed in the present invention was the best.

[0109]

Table 4

[0110] Thus, it can be said that for the 64 advertisement sentences created by humans, the percentage of "Yes" was the lowest, and the MLE model and the reinforcement learning models all generated advertisement sentences that were more human - like than the 64 advertisement sentences created by humans. On the other hand, the relevance to the search query was the highest for the 64 advertisement sentences created by humans. Also, the average comprehensive score after scaling these questions to 0 - 1 was the highest for the offline reinforcement learning model using relative CTR as the reward, and it showed high values especially for fluency and the generation of human - like sentences.

[0111] As described above, in this embodiment, a high-performance automatic advertisement text generation model can be obtained by adding a simple output layer to the model that has learned language characteristics from the corpus 36 and adjusting it, that is, fine-tuning.

[0112] Then, using the model generated by this fine-tuning as the initial value, learning is performed considering the advertisement effect with bias relaxation. Generally, indicators for estimating advertisement effects such as CTR change depending on the budget 46, placement position, etc., and biases occur. In order to perform learning considering the advertisement effect with this bias relaxed, it is possible to estimate the potential advertisement effect without heuristic reward design and automatically generate the advertisement text 64.

[0113] Also, in this embodiment, since fine-tuning is performed to generate an advertisement text from keywords and LPs using a pre-trained model, it is possible to automatically generate a more attractive advertisement text 64 without heuristic reward design.

[0114] Also, in this embodiment, CTR is used as an indicator of the advertisement effect. Since a click represents the user's action on the web and is also an action that leads to a transition to the LP or a purchase on the page (conversion), it is possible to estimate the advertisement effect using CTR as an objective and useful indicator for estimating the advertisement effect and automatically generate the advertisement text 64.

[0115] Also, in this embodiment, by excluding the advertisement effect caused by elements other than the advertisement text 64 from the advertisement effect, as a result of estimating the potential advertisement effect, it is possible to estimate the potential advertisement effect with bias relaxation without heuristic reward design and automatically generate the advertisement text 64.

[0116] Also, in this embodiment, the difference between the logit of the actual CTR and the logit of the CTR predicted from the feature quantities independent of the advertisement text 64 is used as the objective variable. In order to give rewards excluding elements that affect the CTR regardless of the content of the advertisement text 64 such as the budget 46, the distribution settings, and the placement position, the advertisement effect of potential advertisement texts excluding the advertisement effects of elements other than the advertisement text 64 such as the attributes of the advertisement publisher, the budget amount of the advertisement, and the distribution settings is higher. Thus, it is possible to generate the advertisement text 64 without heuristic reward design.

[0117] Also, in this embodiment, the budget 46 and the quality score that determine the advertisement placement rank are used as proxy variables for the placement rank, and the CTR predicted from the feature quantities independent of the advertisement text 64 is calculated.

[0118] Generally, although the placement rank affects the CTR, the method of explicitly treating the placement position as input data can only be done from the perspective of the advertisement platform side. Users of the advertisement platform cannot even obtain information such as the average placement rank and can only obtain things such as the ratio displayed at the top of the page.

[0119] And since the budget 46 and the quality score that determine the placement rank are associated with feature quantities independent of the advertisement text 64 such as the campaign 60 and the advertisement group 62, the budget 46 and the quality score are treated as proxy variables for the unobtainable placement rank, and thereby the CTR predicted from the feature quantities independent of the advertisement text 64 can be calculated. As a result, it is possible to generate the advertisement text 64 such that the advertisement effect of the potential advertisement text 64 excluding the advertisement effects of elements other than the advertisement text 64 such as the attributes of the advertisement publisher, the budget amount of the advertisement, and the distribution settings becomes higher without heuristic reward design.

[0120] Also, in this embodiment, the bias is relaxed by two-stage learning. The relative CTR can predict the ranking of CTRs in the same setting better than the absolute CTR. As a result, the generation of the advertisement text 64 with a higher advertisement effect of the potential advertisement text 64 that excludes the advertisement effects of elements other than the advertisement text 64 such as the attributes of the advertisement publisher, the budget amount of the advertisement, and the distribution settings becomes possible without heuristic reward design, as compared with the case of prediction with only one stage.

[0121] Also, in this embodiment, the search keyword is set for the advertisement group 62, and the budget 46 and the distribution settings are set at a level higher than the advertisement group 62. By relaxing the bias at the advertisement group 62 level caused by these factors, the generation of the advertisement text 64 with a higher advertisement effect of the potential advertisement text 64 that excludes the advertisement effects of elements other than the advertisement text 64 such as the attributes of the advertisement publisher, the budget amount of the advertisement, and the distribution settings becomes possible without heuristic reward design.

[0122] Also, in this embodiment, since learning is performed by offline reinforcement learning, the exposure bias can be reduced as compared with the case of using an autoregressive model learned by maximum likelihood estimation for the teacher data. Also, efficiency can be achieved as compared with the case of using online reinforcement learning.

[0123] Also, in this embodiment, when elements other than the advertisement effect are generally added to the reward, the nature of the advertisement text 64 generated by the reward design varies greatly, and various ingenuities are required for the reward design. However, since the present invention performs learning using only the data of the advertisement effect, learning can be performed efficiently.

[0124] <Second Embodiment> Next, an advertisement text automatic generation system according to a second embodiment of the present invention will be described.

[0125] In this embodiment, a method called Reward Shaping, which adds an additional value to the normal reward value of reinforcement learning, is used, and the prediction of the CTR is performed for each word of the advertisement text 64 instead of for each advertisement text 64.

[0126] Specifically, when predicting the logit of CTR using the encoder of T5, a forward mask is applied and predictions are made for each word. Then, as the reward, the difference in the reward for each word is used.

[0127] As shown in Table 5, for example When using Reward Shaping for the advertisement text JPEG0007711897000009.jpg1489, the estimated value of CTR for the advertisement text is output in word units, and the reward is the difference thereof. Also, a keyword is input at the beginning of the word sequence, and the relevance to the keyword is considered.

[0128]

Table 5

[0129] This enables the model to be informed of which word contributes to the CTR, and the convergence of learning can be improved.

[0130] As described above, in this embodiment, the advertisement effect with bias relaxation is predicted in word units. Usually, an index for estimating an advertisement effect such as CTR is given to the advertisement text 64 rather than each output word, so it is ambiguous which word contributes to the advertisement effect, and it is difficult to learn which word should be output to produce a higher advertisement effect. In addition, since the index of the advertisement effect itself also depends on the posting order, it is necessary to remove the bias.

[0131] By predicting this advertisement effect with bias relaxation in word units instead of sentence units and adding the value to the reward, the model can be informed of which word contributes to the advertisement effect, and the convergence of learning can be accelerated.

[0132] On the other hand, when Reward Shaping is not used, the estimated CTR after considering up to the last word is used as the reward. While it can reflect the realistic setting where the advertisement text 64 is not completed, the model cannot obtain information on which word contributed to the CTR.

[0133] <Supplementary Explanation of this Embodiment> As described above, this embodiment has been used as one form for implementing the invention. However, the present invention is not limited to such embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the gist of the present invention.

Explanation of Reference Numerals

[0134] 100 Advertisement Text Automatic Generation System 16 Corpus Acquisition Function Means 18 Pre-training Function Means 20 KW·LP Acquisition Function Means 22 Adjustment Function Means 24 Advertisement Delivery Performance Acquisition Function Means 26 Feature Extraction Function Means 28 CTR Estimation Function Means 30 Bias Calculation Function Means 32 Advertisement Text Generation Function Means 34 DB 36 Corpus 38 Wikipedia 40 OSCAR 42 CC-100 44 Bidding Strategy 46 Budget 48 TD 50 CTR Baseline 52 Relative CTR 54 Predicted CTR 56 Actual CTR 58 Advertisement Account 60 Campaign 62 Advertisement Group 64 Advertisement Text

Claims

1. An information acquisition means for acquiring keywords and a landing page as information necessary for the summary of an advertisement text, A language model adjustment means for adjusting a language model pre-learned by a prepared corpus using maximum likelihood estimation according to the information acquired by the information acquisition means, Using the language model adjusted by the language model adjustment means as an initial value, learning is performed in consideration of the bias-relaxed advertisement effect predicted and estimated from past advertisements and their distribution histories. Offline reinforcement learning is adopted for the learning. Regarding the reward, the difference between the advertisement effect predicted from the elements affecting the advertisement effect regardless of the content of the advertisement text and the actual advertisement effect is predicted from the advertisement text and used. An advertisement text generation means for generating an advertisement text so that the reward used for the learning is maximized, An advertisement text automatic generation system comprising:

2. The bias-relaxed advertisement effect is the bias-relaxed click-through rate (CTR). The advertisement text automatic generation system according to Claim 1.

3. The bias-relaxed advertisement effect is the advertisement effect of a potential advertisement text excluding the advertisement effect of elements other than the advertisement text. The advertisement text automatic generation system according to Claim 1.

4. The bias-relaxed CTR is the difference between the logit of the actually predicted CTR and the logit of the CTR predicted from the elements affecting the advertisement effect regardless of the content of the advertisement text, predicted from the advertisement text. The advertisement text automatic generation system according to Claim 2.

5. The CTR predicted from the elements affecting the advertisement effect regardless of the content of the advertisement text is estimated by treating at least one of the budget and the quality score for determining the advertisement placement rank as a proxy variable for the unobtainable advertisement placement rank. The advertisement text automatic generation system according to Claim 4.

6. The bias-relaxed CTR has its bias relaxed by two-stage learning. In the second stage of the two-stage learning, the CTR is predicted using the objective variable using the difference in the logit, and only the predicted value of the CTR is used for the reward. The advertisement text automatic generation system according to Claim 4.

7. The advertisement has a hierarchical structure of an advertisement account, a campaign, an advertisement group, and an advertisement text. The bias-relaxed advertisement effect is the one with the bias occurring at the hierarchy above the advertisement group relaxed. The advertisement text automatic generation system according to Claim 1.

8. The offline reinforcement learning is the reinforcement learning performed offline using only the data of the bias-relaxed advertisement effects predicted and estimated from the advertisements distributed in the past and their distribution histories. The advertisement sentence automatic generation system according to claim 1.

9. The bias-relaxed advertisement effect is predicted in word units, and the predicted value of the advertisement effect is added to the reward. The advertisement sentence automatic generation system according to any one of claims 1 to 8.

10. An advertisement sentence automatic generation method executed by a computer, comprising: an information acquisition step of acquiring a keyword and a landing page as information necessary for summarizing an advertisement sentence; a language model adjustment step of adjusting a language model pre-learned by a corpus prepared in advance using maximum likelihood estimation with the information acquired in the information acquisition step; using, as an initial value, the language model adjusted in the language model adjustment step, performing learning in consideration of the bias-relaxed advertisement effect predicted and estimated from the advertisements distributed in the past and their distribution histories, adopting offline reinforcement learning for the learning, and for the reward, using the difference between the actually predicted advertisement effect and the advertisement effect predicted from the elements affecting the advertisement effect regardless of the content of the advertisement sentence, and generating an advertisement sentence so that the reward used for the learning is maximized. An advertisement sentence automatic generation method comprising the above steps.

11. A computer, information acquisition means for acquiring a keyword and a landing page as information necessary for summarizing an advertisement sentence, language model adjustment means for adjusting a language model pre-learned by a corpus prepared in advance using maximum likelihood estimation with the information acquired by the information acquisition means, and using, as an initial value, the language model adjusted by the language model adjustment means, performing learning in consideration of the bias-relaxed advertisement effect predicted and estimated from the advertisements distributed in the past and their distribution histories, adopting offline reinforcement learning for the learning, and for the reward, using the difference between the actually predicted advertisement effect and the advertisement effect predicted from the elements affecting the advertisement effect regardless of the content of the advertisement sentence, and advertisement sentence generation means for generating an advertisement sentence so that the reward used for the learning is maximized. An advertisement sentence automatic generation program for causing the computer to function as described above.

Images