Method for training a learning model for estimating advertising effectiveness, method for estimating advertising effectiveness, and program

Abstract

To improve rank prediction accuracy in an advertisement group.SOLUTION: A learning method of a learning model for advertisement effect estimation includes the steps of: acquiring data of all advertisements included in one or a plurality of advertisement accounts as learning data (step A); dividing all advertisements including the one or the plurality of advertisement accounts into a plurality of rank learning groups for each of conditions characterizing advertisements (step B); selecting one rank learning group from among a plurality of rank learning groups divided into groups (step C); calculating a loss function value about the selected one rank learning group (step D); and performing ranking learning in which rank learning is performed in one selected rank learning group and rank learning is not performed between different rank learning groups (step E).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technology for estimating advertising effectiveness, and more particularly to a technology for improving the accuracy of ranking advertising effectiveness. [Background technology]

[0002] Advertisements on advertising media such as Google (registered trademark) and Yahoo! (registered trademark) have a hierarchical structure, and are created in the following order from the highest level: 1: Account, 2: Campaign, 3: Ad Group, 4: Ad.

[0003] Advertising media such as Google (registered trademark) and Yahoo! (registered trademark) have an optimization function that prioritizes the delivery of ads within an ad group that are expected to perform better than other ads. This function allows ads with statistically better performance to be delivered preferentially as data accumulates. However, because this optimization function relies on the accumulation of data, there is a problem in that ads with poor performance may be delivered until sufficient data is accumulated. Against this background, technology has been developed to predict the delivery effectiveness of ads in an ad group in order to prioritize the delivery of ads that are expected to perform better than other ads. Note that indicators of ad delivery effectiveness include the number of impressions, the number of clicks, the click-through rate, and the delivery amount.

[0004] For example, Japanese Patent Publication No. 2021-182340 (Patent Document 1) and "Estimating Delivery Priority of Ad Creatives Using Rank Learning" (Non-Patent Document 1) propose technology for determining delivery priority for ad creations by predicting the average daily delivery amount, which is one of the indicators of the quality of ad creations, before delivery. Specifically, based on the idea that determining delivery priority does not require strict estimation of the actual numerical value of the effectiveness of ads, it is sufficient to know the order of advertising effectiveness for a given group of ads, a method is proposed that uses rank learning technology to recommend ad creations in ascending order of quality. Note that this method calculates the score of ad creatives using pairwise rank learning. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-182340

[0006] [Non-Patent Document 1] "Estimating the Delivery Priority of Ad Creatives Using Rank Learning" (Yuki Iwasaki, Kazuteru Taniguchi: Proceedings of the 2020 National Conference of the Japanese Society for Artificial Intelligence, Vol. JSAI2020, 34th (2020), Session ID: 1H4-OS-12b-04, Published: June 19, 2020) Summary of the Invention [Problem to be solved by the invention]

[0007] The conventional prediction model for ad delivery effectiveness, as described above, learns the ranking of all ads in a dataset. However, in advertising operations, it is not necessary to know the relative merits of all ads; it is sufficient to determine the relative merits of some ads that you want to compare. Here, conditions for "some ads that you want to compare" include delivery settings such as ad groups, image sizes, and user attributes. For example, if you can determine the relative merits of ads that are included in the same ad group and have the same image size, it becomes possible to determine the relative merits of ad images displayed in similar positions within the ad group, and to determine the ad images that should be delivered.

[0008] In this way, to determine the advertising images to be delivered, it is only necessary to compare the merits and demerits of some of the advertisements that one wishes to compare. However, the methods described in Patent Document 1 and Non-Patent Document 1 determine the merits and demerits of all advertisements. As is commonly known as Simpson's paradox, the correlation in a population does not match the correlation in groups into which the population is divided. For this reason, the conventional method that learns the "merits and demerits of all advertisements" has the problem of low prediction accuracy for the "merits and demerits of the delivery effects of some of the advertisements that one wishes to compare."

[0009] For example, consider the example of distribution effectiveness between advertisements shown in Figure 1. This figure is intended to simply explain an example in which the correlation between feature values and advertising effectiveness in a population does not match the correlation between feature values and advertising effectiveness in groups obtained by dividing the population.

[0010] Figure 1(a) shows four example advertisement images (advertising IDs 1 to 4), and Figure 1(b) is a scatter plot showing the correlation between the feature values of these four advertisement images and the effectiveness of the advertisements. Advertisement IDs 1 and 2 are advertisement images for the sale of "cars," and belong to the same group (group A) in the sense that they are both selling "cars." On the other hand, ad IDs 3 and 4 are advertisement images for the sale of "houses," and belong to the same group (group B) in the sense that they are both selling "houses." In this example, the effectiveness of the advertisements is the click-through rate (CTR), and the feature value of the advertisements is the font size of the characters. Within group A, the feature value (font size) of the advertisement with advertisement ID 2 is larger than that of advertisement ID 1, and within group B, the feature value (font size) of the advertisement with advertisement ID 4 is larger than that of advertisement ID 3.

[0011] In this example, in group A, ad ID 2, which has larger features than ad ID 1, has a higher CTR, and in group B, ad ID 4, which has larger features than ad ID 3, has a higher CTR, so there is a positive correlation between font size and CTR within the same group. On the other hand, when comparing ad ID 2 and ad ID 3, which belong to different ad groups, ad ID 3, which has smaller ad features, has a higher CTR than ad ID 2, which has larger ad features, so there is a negative correlation between font size and CTR for the four ad images as a whole. In this way, if learning is done overall without taking ad groups into consideration, the opposite correlation (negative correlation) will be learned instead of the positive correlation that we originally wanted to learn, resulting in a problem of reduced rank prediction accuracy within the group that we want to rank.

[0012] In view of such problems, the present inventors came up with the idea that if rank learning is performed for each advertisement group, it will be possible to correctly estimate the quality of advertisement creation, and have thus come up with the present invention. [Means for solving the problem]

[0013] In order to solve the above problem, a learning method for a learning model for estimating advertising effectiveness according to the present invention is characterized by including the following steps A to E. Step A: Obtaining data of all advertisements contained in one or more advertising accounts as learning data; Step B: grouping all advertisements included in the plurality of advertisement accounts into a plurality of rank learning groups according to conditions characterizing the advertisements; Step C: selecting one rank learning group from the plurality of rank learning groups; Step D: calculating a loss function value for the selected one rank learning group; Step E: A step of performing rank learning within the selected one rank learning group, while not performing rank learning between different rank learning groups.

[0014] In one embodiment, steps D and E are steps of performing rank learning to correct the learning model so as to minimize the loss function value of the selected one rank learning group.

[0015] In a preferred embodiment, the method further comprises step F of executing steps C to E for all of the plurality of rank learning groups, and execution of step F corrects the learning model for all of the learning data.

[0016] In one aspect, the step F includes a sub-step of setting the sum of the loss function values of the rank learning groups as the loss function value of the learning model.

[0017] For example, the conditions that characterize the advertisement include at least one of a distribution setting, an image size, a distribution date and time, and an advertisement group.

[0018] Furthermore, for example, the loss function value is calculated using a pairwise or listwise loss function.

[0019] The method for estimating advertising effectiveness according to the present invention includes the steps of: creating a ranking group consisting of multiple advertisements that have been delivered or are yet to be delivered and are to be ranked based on conditions that characterize the advertisements; calculating a score for each of the multiple advertisements included in the ranking group using a learning model trained by the learning method for a learning model for estimating advertising effectiveness according to the present invention; and determining a delivery priority based on the calculated score for each advertisement.

[0020] Furthermore, a program according to the present invention is a program for causing a computer to execute a method for training a learning model for estimating advertising effectiveness, the method including the following steps A to E. Step A: Obtaining data of all advertisements contained in one or more advertising accounts as learning data; Step B: grouping all advertisements included in the one or more advertisement accounts into a plurality of rank learning groups according to conditions characterizing the advertisements; Step C: selecting one rank learning group from the plurality of rank learning groups; Step D: calculating a loss function value for the selected one rank learning group; Step E: A step of performing rank learning within the selected one rank learning group, while not performing rank learning between different rank learning groups.

[0021] In one aspect of the program according to the present invention, steps D and E are steps of performing rank learning to correct the learning model so as to minimize the loss function value of the selected one rank learning group.

[0022] In a preferred embodiment, the method further comprises step F of executing steps C to E for all of the plurality of rank learning groups, and execution of step F corrects the learning model for all of the learning data.

[0023] In one aspect, the step F includes a sub-step of setting the sum of the loss function values of the rank learning groups as the loss function value of the learning model.

[0024] For example, the conditions that characterize the advertisement include at least one of a distribution setting, an image size, a distribution date and time, and an advertisement group.

[0025] Furthermore, for example, the loss function value is calculated using a pairwise or listwise loss function.

[0026] Furthermore, the program of the present invention is a program for causing a computer to execute a method for estimating advertising effectiveness, the method including the steps of: creating a ranking group consisting of multiple advertisements that have been delivered or are not yet delivered, based on conditions that characterize the advertisements; calculating a score for each of the multiple advertisements included in the ranking group using a learning model trained by the learning method for a learning model for estimating advertising effectiveness of the present invention; and determining a delivery priority based on the calculated score for each advertisement. [Effects of the Invention]

[0027] In the present invention, rank learning for estimating advertising effectiveness is performed for each group to be ranked, thereby improving the accuracy of rank prediction within the group to be ranked. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 10 is a diagram for simply explaining an example in which the correlation between the feature amount and the advertising effectiveness in a population does not match the correlation between the feature amount and the advertising effectiveness in a population divided from the population. [Figure 2] FIG. 2 is a block diagram for conceptually explaining the hierarchical structure of advertisements. [Figure 3] FIG. 10 is a diagram showing an example of a plurality of advertising accounts that constitute learning data to be acquired. [Figure 4] FIG. 10 is a diagram conceptually illustrating an example in which all advertisements contained in multiple advertising accounts acquired as learning data are grouped into multiple rank learning groups according to conditions that characterize the advertisements. [Figure 5] 1 is a flowchart for explaining an outline of processing executed in a learning method for a learning model for estimating advertising effectiveness according to the present invention. [Figure 6] 10 is a flowchart illustrating an example of a procedure for completing one learning model for all acquired advertising data. [Figure 7] 1 is a flowchart illustrating an example of ranking prediction using a learning method for a learning model for estimating advertising effectiveness according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0029] Hereinafter, a learning method for a learning model for estimating advertising effectiveness, a method for estimating advertising effectiveness, and a program according to the present invention will be described with reference to the drawings.

[0030] Figure 2 is a block diagram that conceptually explains the hierarchical structure of advertising. As shown in this diagram, advertising has a hierarchical structure, where an "advertising account" is a collection of "campaigns," each "campaign" is a collection of "ad groups," and each "ad group" is a collection of "ads." An "ad" is an image that is actually displayed on an advertising medium, and the one that is considered appropriate for that medium is displayed from among the "ad groups."

[0031] In the example shown in this diagram, the "Ad Account" is a collection of two "Campaigns," with "Campaign 1" being a collection of "Ad Group A" and "Ad Group B," "Ad Group A" being a collection of "Ad a," "Ad b," and "Ad c," and "Ad Group B" being a collection of "Ad d," "Ad e," and "Ad f." The same is true for "Campaign 2," with "Campaign 2" being a collection of "Ad Group C" and "Ad Group D," "Ad Group C" being a collection of "Ad g," "Ad h," and "Ad i," and "Ad Group D" being a collection of "Ad j," "Ad k," and "Ad l."

[0032] In the learning method for a learning model for estimating advertising effectiveness according to the present invention, first, data on all advertisements (advertising data) contained in multiple advertising accounts consisting of one or multiple advertising groups is obtained as learning data.

[0033] Figure 3 is a diagram showing an example of one or more advertising accounts (data) that make up the learning data to be acquired. In the example shown in this figure, data for all advertisements contained in each of the advertising accounts of companies A to N will be acquired as learning data.

[0034] In the learning method for a learning model for estimating advertising effectiveness according to the present invention, all advertisements contained in one or more advertising accounts acquired as learning data are grouped into multiple rank learning groups based on conditions that characterize the advertisements.

[0035] 4 is a conceptual diagram showing an example of grouping all advertisements contained in one or more advertising accounts acquired as learning data into multiple rank learning groups according to conditions that characterize the advertisements. In the example shown in this figure, data on all advertisements contained in the advertising accounts of companies A to N is acquired as learning data, and this learning data is grouped into four "learning groups" (A to D), with each "learning group" being a collection of multiple "advertisements."

[0036] 5 is a flowchart outlining the processing executed in the learning method for a learning model for estimating advertising effectiveness according to the present invention. As shown in this figure, as described above, in step S101, data on all advertisements is acquired as learning data (data set). Then, in step S102, the acquired learning data is divided into a plurality of rank learning groups according to conditions characterizing the advertisements.

[0037] Next, one rank learning group is selected from the plurality of rank learning groups (step S103), a loss function value is calculated for the selected rank learning group (step S104), and based on this loss function value, rank learning is performed within the selected rank learning group (step S105), and the learning model is corrected (step S106).

[0038] In the present invention, the rank learning in step S105 is performed within one selected rank learning group, but rank learning is not performed between different rank learning groups. As a result, the superiority or inferiority of advertisements within the same advertisement group is learned, but the superiority or inferiority of advertisements between different advertisement groups is not learned, thereby solving the problem of "Simpson's Paradox" described above.

[0039] In the example shown in FIG. 5, rank learning in step S105 is performed to correct the learning model so as to minimize the loss function value of one selected rank learning group.

[0040] As described above, the training method for a learning model for estimating advertising effectiveness according to the present invention includes the steps of: acquiring data on all advertisements included in one or more advertising accounts consisting of one or more advertising groups as training data (Step A); grouping all advertisements included in the one or more advertising accounts into a plurality of rank training groups based on conditions characterizing the advertisements (Step B); selecting one rank training group from the plurality of rank training groups (Step C); calculating a loss function value for the selected rank training group (Step D); and performing rank learning within the selected rank training group while not performing rank learning between different rank training groups (Step E). In Step E, rank learning is performed within the selected rank training group while rank learning between different rank training groups is not performed. By adopting such a configuration, the superiority or inferiority of advertisements within the same advertising group is learned, but the superiority or inferiority of advertisements between different advertising groups is not learned, thereby resolving the problem of "Simpson's Paradox."

[0041] As described above, in one aspect of the present invention, rank learning may be performed so as to minimize the loss function value of one selected rank learning group, thereby correcting the learning model.

[0042] By modifying the learning model as described above, it is theoretically possible to optimize the learning model for a single selected rank learning group. However, if the selected rank learning group does not contain a sufficiently large amount of training data, the learning model that is supposed to be "optimized" may not be optimal. Furthermore, a learning model "optimized" for a single selected rank learning group is generally not "optimized" for the entire dataset obtained as training data.

[0043] Therefore, in the learning method for a learning model for estimating advertising effectiveness according to the present invention, in order to obtain an "optimized" learning model for the entire data set acquired as learning data, the following steps may be included in addition to the steps shown in FIG. 5, so that an "optimized" learning model for the entire acquired data set, i.e., one learning model for all acquired advertising data, can be completed.

[0044] FIG. 6 is a flowchart illustrating an example of a procedure for completing one learning model for all of the acquired advertising data.

[0045] After "learning within the selected learning group" is completed according to the procedure shown in Fig. 5, another learning group is selected (step S201). Then, a loss function value within this selected learning group is calculated (step S202), and based on this loss function value, rank learning is performed within the selected rank learning group (step S203), and the learning model is corrected (step S204). Note that in the example shown in Fig. 6, rank learning in step S203 is performed to correct the learning model so as to minimize the loss function value of the selected rank learning group.

[0046] Following step S204, it is determined whether "an unprocessed learning group exists" (step S205), and if "an unprocessed learning group exists" (Yes), the processing from step S201 onwards is repeated. If "an unprocessed learning group does not exist" (No), the learning of the learning model is terminated.

[0047] As described above, the learning method for a learning model for estimating advertising effectiveness according to the present invention may, in a preferred embodiment, include a step (step F) of performing the above-described steps C to E for all of the plurality of divided rank learning groups to obtain a learning model that is "optimized" for the entire data set obtained as learning data, and the learning model for the entire learning data may be corrected by performing step F. Even if the learning model obtained in this manner is not "optimized" for each individual "rank learning group," it is "optimized" for the entire data set obtained as learning data, and there is no problem that the supposedly "optimized" learning model may not be optimal if a selected rank learning group does not contain a sufficiently large amount of learning data.

[0048] The above-mentioned step F may include a sub-step of setting the sum of the loss function values of the rank learning group as the loss function value of the learning model.

[0049] In the present invention, the conditions that characterize an advertisement include, for example, at least any of the distribution settings, image size, distribution date and time, and advertisement group.

[0050] In the present invention, the loss function value is calculated using, for example, a pairwise or listwise loss function.

[0051] By utilizing the learning method of the learning model for estimating advertising effectiveness according to the present invention, advertising effectiveness can be estimated (ranking prediction).

[0052] FIG. 7 is a flowchart illustrating an example of ranking prediction using the learning method for a learning model for estimating advertising effectiveness according to the present invention.

[0053] First, a ranking group for which the advertising effectiveness is to be ranked is created (step S301), and a score for each advertisement in the ranking group is calculated using a trained learning model (step S302).Then, a ranking is predicted based on the calculated score, and distribution priority is determined (ranking) (step S303).

[0054] Thus, the method for estimating advertising effectiveness according to the present invention is a method for estimating advertising effectiveness, which includes the steps of creating a ranking group consisting of multiple advertisements that have been delivered or are not yet delivered, based on conditions that characterize the advertisements; calculating a score for each of the multiple advertisements included in the ranking group using a learning model trained by the learning method for a learning model for estimating advertising effectiveness according to the present invention; and determining a delivery priority based on the calculated score for each advertisement.

[0055] The method for training a learning model for estimating advertising effectiveness according to the present invention can be executed by a computer using a program.

[0056] That is, the program according to the present invention is a program for causing a computer to execute a method for training a learning model for estimating advertising effectiveness, and the method is characterized by including the following steps A to E. Step A: Obtaining data of all advertisements contained in one or more advertising accounts as learning data; Step B: grouping all advertisements included in the one or more advertisement accounts into a plurality of rank learning groups according to conditions characterizing the advertisements; Step C: selecting one rank learning group from the plurality of rank learning groups; Step D: calculating a loss function value for the selected one rank learning group; Step E: A step of performing rank learning within the selected one rank learning group, while not performing rank learning between different rank learning groups.

[0057] In one aspect of the program according to the present invention, steps D and E are steps of performing rank learning to correct the learning model so as to minimize the loss function value of the selected one rank learning group.

[0058] In a preferred embodiment, the method further comprises step F of executing steps C to E for all of the plurality of rank learning groups, and execution of step F corrects the learning model for all of the learning data.

[0059] In one aspect, the step F includes a sub-step of setting the sum of the loss function values of the rank learning groups as the loss function value of the learning model.

[0060] For example, the conditions that characterize the advertisement include at least one of a distribution setting, an image size, a distribution date and time, and an advertisement group.

[0061] Furthermore, for example, the loss function value is calculated using a pairwise or listwise loss function.

[0062] Furthermore, the program of the present invention is a program for causing a computer to execute a method for estimating advertising effectiveness, the method including the steps of: creating a ranking group consisting of multiple advertisements that have been delivered or are not yet delivered, based on conditions that characterize the advertisements; calculating a score for each of the multiple advertisements included in the ranking group using a learning model trained by the learning method for a learning model for estimating advertising effectiveness of the present invention; and determining a delivery priority based on the calculated score for each advertisement. [Example]

[0063] In the following examples, the effectiveness of rank learning that takes into account the structure of advertisements according to the present invention will be confirmed by comparing it with rank learning that does not take into account the structure of advertisements. Note that the average rank correlation coefficient was used as an indicator for evaluating the effectiveness.

[0064] [Create data] The data obtained as learning data includes the "impressions," "image size" (image width and image height), "date," "image" (image_id), and "ad group ID" (ad_group_id) for each ad.

[0065] Ten months' worth of data was extracted from the accumulated data and data was created that took into account the ad structure. The specific procedure is as follows: Of the 10 months' worth of data, the first eight months' data were used as training data, and the following two months' data were used as test data, and the data was divided into training data and test data. This data was then divided into monthly data sets, and the resulting monthly data sets were grouped by records with the same "ad group ID (ad_group_id)," "image width" (image_width), and "image height" (image_height). The reason "image width" (image_width) and "image height" (image_height) are taken into consideration when grouping is because the relative effectiveness of images of the same size is important when determining ad submission priorities. Next, groups with only one image (unique number of image_ids) were removed, and groups containing two or more images were extracted. The "impressions" (impressions) for each "image" (image_id) within each group were then totaled. This creates multiple groups with "image_id" and "impressions" as columns. "image_id" and "impressions" are then used for learning and testing.

[0066] On the other hand, the procedure for creating data that does not take into account the ad structure is as follows. First, prepare training data for "data that takes into account the ad structure." Then, ignore "group" and add up "impressions" for each "image" (image_id). This creates data with "image" (image_id) and "impressions" as columns. Then, use "image" (image_id) and "impressions" for training.

[0067] [Learning Method] "Impressions" is used as an index for rank learning. The learning procedure taking into account the structure of advertisements is as follows. First, as described above, training data is prepared that has multiple groups with "image" (image_id) and "impressions" as columns. Next, the groups are extracted one by one in order, and two rows of records are randomly extracted from each extracted group. Then, pairwise learning is performed with the extracted records. As in Non-Patent Document 1, RankNet is used as the loss function. When records have been extracted once from all groups, one learning epoch is considered to be complete. This learning epoch is repeated 10 times.

[0068] On the other hand, the procedure for learning without considering the ad structure is as follows. First, as described above, prepare learning data with the columns "image" (image_id) and "impressions" (impressions). Next, randomly extract two records. Then, perform pairwise learning with the extracted records. When the same number of records as the number of groups in "learning with consideration for ad structure" has been extracted, one learning epoch is completed. This learning epoch is repeated 10 times.

[0069] The test procedure is as follows: After 10 epochs of training, the average rank correlation coefficient for the test data is calculated and recorded. To calculate the rank correlation coefficient, first calculate the model output (score) for all images, then calculate the rank correlation coefficient between that score and the target impression for training for each group and average them.

[0070] The results of the training experiment are shown in Table 1. The values shown in this table are the average rank correlation coefficients for the test data after 10 epochs of training.

[0071] [Table 1]

[0072] As can be seen from these results, the average rank correlation coefficient for the test data is higher for training that takes into account the ad structure (0.2349) than for training that does not (0.1026). This result shows that training that takes into account the ad structure can predict rankings more accurately than training that does not take into account the ad structure.

[0073] As described above, rank learning that takes into account the structure of advertisements according to the present invention improves rank prediction accuracy compared to rank learning that does not take into account the structure of advertisements. As a result, it becomes possible to more accurately estimate the quality of advertisement creation within a group for which it is desired to rank the advertising effectiveness.

[0074] As described above, the present invention employs a method for performing rank learning of the delivery priority of ad creations on a group-by-group basis, and during learning, ads within the same group are extracted as pairs and learned. Such group-by-group learning improves rank prediction accuracy compared to predicting ranks within the entire data set. As a result, it becomes possible to more accurately estimate the quality of ad creations within a group for which it is desired to rank advertising effectiveness. [Industrial Applicability]

[0075] According to the present invention, the accuracy of rank prediction within a group for which it is desired to rank advertising effectiveness is improved.

Claims

1. A method for learning a learning model for estimating advertising effectiveness, comprising the following steps A to E: Step A: acquiring data of all advertisements contained in one or more advertising accounts as learning data; Step B: grouping all advertisements included in the one or more advertisement accounts into a plurality of rank learning groups according to conditions characterizing the advertisements; Step C: selecting one rank learning group from the plurality of rank learning groups; Step D: calculating a loss function value for the selected one rank learning group; Step E: a step of performing rank learning within the selected one rank learning group, while not performing rank learning between different rank learning groups; Steps D and E are steps of performing rank learning using the loss function value of the selected one rank learning group to correct the learning model; The method further comprises a step F of executing steps C to E for all of the plurality of rank learning groups, and the execution of step F corrects a learning model for all of the learning data. How the learning model is trained.

2. 2. The learning method for a learning model according to claim 1, wherein steps D and E are steps of performing rank learning to correct the learning model so as to minimize the loss function value of the selected one rank learning group.

3. 3. The method for learning a learning model according to claim 1, wherein step F comprises a substep of setting the sum of the loss function values of the rank learning groups as the loss function value of the learning model.

4. The method for learning a learning model according to claim 1 or 2, wherein the conditions characterizing the advertisement include at least one of a distribution setting, an image size, a distribution date and time, and an advertisement group.

5. The method for learning a learning model according to claim 1 or 2, wherein the loss function value is calculated using a pairwise or listwise loss function.

6. creating a ranking group consisting of a plurality of advertisements that have been delivered or have not yet been delivered, based on conditions that characterize the advertisements; Calculating a score for each of the plurality of advertisements included in the ranking group using a learning model trained by the method according to claim 1 or 2; determining a delivery priority based on the calculated score for each advertisement; How to estimate advertising effectiveness.

7. A program for causing a computer to execute a method for training a learning model for estimating advertising effectiveness, the method comprising: Step A: acquiring data of all advertisements contained in one or more advertising accounts as learning data; Step B: grouping all advertisements included in the one or more advertisement accounts into a plurality of rank learning groups according to conditions characterizing the advertisements; Step C: selecting one rank learning group from the plurality of rank learning groups; Step D: calculating a loss function value for the selected one rank learning group; Step E: a ranking learning step of performing rank learning within the selected one rank learning group, while not performing rank learning between different rank learning groups; Steps D and E are steps of performing rank learning using the loss function value of the selected one rank learning group to correct the learning model; The method further comprises a step F of executing steps C to E for all of the plurality of rank learning groups, and the execution of step F corrects a learning model for all of the learning data. program.

8. 8. The program according to claim 7, wherein steps D and E are steps of performing rank learning to correct a learning model so as to minimize a loss function value of the selected one rank learning group.

9. 9. The program according to claim 7, wherein step F comprises a substep of setting a sum of the loss function values of the rank learning groups as the loss function value of the learning model.

10. The conditions characterizing the advertisement include at least a distribution setting, an image size, a distribution date and time, an advertisement group, and the like.

9. The program according to claim 7, further comprising: a loop.

11. The program according to claim 7 or 8, wherein the loss function value is calculated using a pairwise or listwise loss function.

12. A program for causing a computer to execute a method for estimating advertising effectiveness, the method comprising: creating a ranking group consisting of a plurality of advertisements that have been delivered or have not yet been delivered, based on conditions that characterize the advertisements; Calculating a score for each of the plurality of advertisements included in the ranking group using a learning model trained by the method according to claim 1 or 2; determining a delivery priority based on the calculated score for each advertisement; program.

Images