Advertisement monitoring device, information processing method, and program

Abstract

This technology provides solutions to problems that may arise when determining the timing of replacements based on the performance monitoring results of advertising materials. [Solution] An advertising monitoring device relating to one aspect of this disclosure continuously monitors the performance of the first advertising material being distributed as the target advertisement, and after the adjustment period of the first advertising material has elapsed, determines whether or not the replacement conditions are met according to the performance monitoring results, and outputs information for processing the replacement of the first advertising material with the second advertising material if the replacement conditions are met.

Description

Technical Field

[0001] The present disclosure relates to an advertisement monitoring device, an information processing method, and a program.

Background Art

[0002] Advertisements are used in various places such as the web to inform people about target products or services. In recent years, technologies have been developed to improve the performance of advertising materials used for such advertisements. For example, Non-Patent Document 1 proposes a system for predicting the performance of advertising materials using a trained model obtained by machine learning.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The performance of advertising materials is high at the beginning of use (submission) and often decreases over time. The deteriorated performance of this advertisement can be restored by replacing the advertising material used. Therefore, conventionally, the replacement of advertising materials may be performed manually according to the monitoring results of the performance of advertising materials. In order to reduce the cost of manual labor, a scenario is assumed in which at least part of the process related to the replacement of this advertising material is automated. In this scenario, the inventors of the present case have found that the following problems may occur.

[0005] In other words, for a period after the ad material is submitted, delivery attributes such as the target audience, the devices used, and the time of day may be adjusted. During this adjustment period, the performance of the ad material tends to be unstable. If the timing of ad material replacement is determined based on the performance observed during this adjustment period, there is a risk of replacement problems occurring, such as replacing high-performing ad material too early.

[0006] This disclosure is made, in part, in light of these circumstances, and one of its purposes is to provide a technology to resolve potential problems that may arise when determining the timing of replacements based on the performance monitoring results of advertising materials. [Means for solving the problem]

[0007] This disclosure adopts the following configuration to solve the aforementioned problems. Note that the following configurations can be combined as appropriate.

[0008] An advertising monitoring device relating to one aspect of this disclosure includes a control unit. The control unit is configured to continuously monitor the performance of a first advertising material being delivered as a target advertisement, to determine whether the replacement conditions are met according to the performance monitoring results after the adjustment period of the first advertising material has elapsed, and to output information for processing the replacement of the first advertising material with a second advertising material if the replacement conditions are met.

[0009] In this configuration, after the adjustment period has elapsed, a decision is made on whether or not to replace the ad materials based on the performance monitoring results. This is because unstable performance during the adjustment period This avoids using performance monitoring results as an indicator for replacement. Therefore, this configuration is expected to eliminate potential problems that may arise when determining the timing of replacement based on the performance monitoring results of advertising materials.

[0010] In the advertising monitoring device relating to the above aspect, monitoring performance may include measuring the value of one or more first indicators related to performance. The control unit may be configured to further determine whether the adjustment period for the first advertising material has elapsed, based on the fluctuations in the measured values ​​of one or more first indicators. If it is determined that the adjustment period for the first advertising material has elapsed, the control unit may determine whether the replacement conditions are met. During the adjustment period, the performance of the advertising material tends to be unstable; that is, the measured indicator values ​​tend to fluctuate. On the other hand, once the adjustment period has elapsed, fluctuations in indicator values ​​often subside. With this configuration, it is possible to appropriately determine the elapsed period based on the fluctuations in indicator values ​​(values ​​of the first indicators). This makes it possible to appropriately resolve problems that may occur when determining the timing of replacement according to the performance monitoring results of the advertising material.

[0011] In the advertising monitoring device relating to the above aspect, monitoring performance may further include measuring the value of one or more second indicators related to performance. Determining whether or not the replacement conditions are met according to the performance monitoring results may consist of determining whether or not the replacement conditions are met according to the measured values ​​of one or more second indicators. One or more first indicators and one or more second indicators may at least partially coincide. With this configuration, the number of indicators to be monitored can be arbitrarily narrowed down by having the indicator used to determine the elapsed adjustment period (first indicator) and the indicator used to determine the replacement conditions (second indicator) overlap. When the number of indicators to be monitored is narrowed down, a reduction in monitoring costs can be expected.

[0012] In the advertising monitoring device relating to the above aspect, the control unit may be configured to further cause the large-scale generation model to generate a second advertising material by giving a generation instruction to the large-scale generation model before outputting information for processing replacement. The replacement may be a replacement with the second advertising material generated by the large-scale generation model. With this configuration, at least a part of the process of preparing the advertising material (second advertising material) can be automated. This can be expected to reduce the cost of generating the advertising material.

[0013] In the advertising monitoring device relating to the above aspect, monitoring performance may further include measuring the values ​​of one or more third indicators related to performance. The generation instruction may include instructional text indicating that a second advertising material should be generated to improve the values ​​of the first advertising material and one or more third indicators. With this configuration, it is possible to expect the generation of advertising materials that can achieve the desired improvements.

[0014] In the ad monitoring device relating to the above aspect, the generation instruction may further include one or more sample sets. Each of the one or more sample sets may include the improvement in the values ​​of a first past ad delivered in the past, a second past ad delivered by replacing the first past ad, and one or more third indicators measured when the first past ad was replaced with the second past ad. Large-scale generative models are known to have acquired the ability to perform in-context learning. With this configuration, by providing sample sets, it is possible to expect an improvement in the accuracy of generating ad materials that can achieve the desired improvement.

[0015] In the advertising monitoring device relating to the above aspect, the control unit may be configured to further select one or more candidates to be used as a second advertising material from a plurality of pre-provided candidates for advertising material before outputting information for processing replacement. The replacement may be a replacement with the selected second advertising material. According to this configuration, candidates are provided in advance. This allows us to clearly define the advertising materials that will be used.

[0016] In the ad monitoring device relating to the above aspect, one or more ads, including the target ad, may be managed on a group basis. The replacement conditions may be replacement conditions linked to the group to which the target ad belongs. With this configuration, a reduction in management costs can be expected compared to managing each ad individually.

[0017] In the advertising monitoring device relating to the above aspect, monitoring performance may further include measuring the values ​​of one or more fourth indicators related to performance. The control unit may be configured to further perform monitoring the performance of the second advertising material being delivered after replacement, and to output information regarding the results of evaluating the degree of improvement in the values ​​of one or more fourth indicators due to the replacement of the first advertising material with the second advertising material, based on the monitoring results of the performance of the first advertising material when it was being delivered and the monitoring results of the performance of the second advertising material. With this configuration, it is possible to evaluate the performance of the advertising material after replacement (second advertising material).

[0018] In the advertising monitoring device relating to the above aspect, one or more fourth indicators may include one or more second indicators. Determining whether or not the replacement conditions are met according to the performance monitoring results may be comprised of determining whether or not the replacement conditions are met according to the measured values ​​of each of the one or more second indicators. Outputting information regarding the results of the evaluation of the degree of improvement may include optimizing the replacement conditions according to the trends of the measured values ​​of each of the one or more fourth indicators in the performance monitoring results of the first advertising material when it was being delivered, and in the performance monitoring results of the second advertising material. With this configuration, it is possible to expect to maintain the performance of the advertising material by optimizing the replacement conditions.

[0019] In the ad monitoring device relating to the above aspect, one or more ads, including the target ad, may be managed on a group basis. The replacement conditions to be optimized may be replacement conditions linked to the group to which the target ad belongs. With this configuration, a reduction in management costs can be expected compared to managing each ad individually.

[0020] In the ad monitoring device relating to the above aspect, the control unit may be configured to perform the following for each of the one or more ads: monitor the performance of the first ad material, determine whether or not the replacement conditions are met, output information for processing the replacement from the first ad material to the second ad material if the replacement conditions are met, monitor the performance of the second ad material being delivered due to the replacement, and output information regarding the results of evaluating the degree of improvement of each of the one or more fourth indicators due to the replacement from the first ad material to the second ad material. The control unit may further be configured to optimize the allocation of each of the one or more ads to a group according to the trend of the measured values ​​of each of the one or more fourth indicators and at least one of the optimized replacement conditions. With this configuration, it is possible to optimize the replacement conditions by optimizing the allocation to groups.

[0021] In the ad monitoring device relating to the above aspect, optimizing the distribution may include merging the first group to which the first ad belongs and the second group to which the second ad belongs, if the optimized replacement conditions are similar. This configuration may reduce the number of groups to be managed. As a result, a reduction in management costs can be expected.

[0022] Furthermore, the form of this disclosure is not limited to the above-mentioned advertising monitoring device (information processing device). As another form of the advertising monitoring device relating to each of the above aspects, one aspect of this disclosure may be an information processing method or a program that implements all or part of the above-mentioned configurations. A machine-readable storage medium storing a program such as this may also be used. Here, the machine-readable storage medium may be a non-transitory medium that stores information such as programs by means of electrical, magnetic, optical, mechanical, or chemical actions. The non-transitory storage medium may include storage media (CDs, DVDs, semiconductor memories, etc.), auxiliary storage devices of computers, external storage devices connected to computers, and the like.

[0023] For example, an information processing method (advertisement monitoring method) according to one aspect of the present disclosure may be executed by a computer. The information processing method may include continuously monitoring the performance of a first advertisement material being distributed as a target advertisement, determining whether or not replacement conditions are satisfied according to the monitoring results of the performance after the elapse of the adjustment period of the first advertisement material, and outputting information for processing the replacement from the first advertisement material to the second advertisement material in response to the satisfaction of the replacement conditions.

[0024] Also, for example, a program (advertisement monitoring program) according to one aspect of the present disclosure may be a program for causing a computer to execute an information processing method. The information processing method may include continuously monitoring the performance of a first advertisement material being distributed as a target advertisement, determining whether or not replacement conditions are satisfied according to the monitoring results of the performance after the elapse of the adjustment period of the first advertisement material, and outputting information for processing the replacement from the first advertisement material to the second advertisement material in response to the satisfaction of the replacement conditions.

Advantages of the Invention

[0025] According to one aspect of the present disclosure, it is possible to expect the elimination of problems that may occur when determining the replacement timing according to the monitoring results of the performance of the advertisement material.

Brief Description of the Drawings

[0026] [Figure 1] FIG. 1 schematically shows an example of a scene to which the present disclosure is applied. [Figure 2]Figure 2 schematically shows one example of a method for preparing the second advertising material. [Figure 3] Figure 3 schematically shows an example of a method for preparing the second advertising material. [Figure 4] Figure 4 schematically shows an example of monitoring the performance of the second advertising material after replacement. [Figure 5] Figure 5 shows an example of how to manage advertisements on a group basis. [Figure 6] Figure 6 schematically illustrates an example of a scenario in which a group is optimized. [Figure 7] Figure 7 schematically shows an example of the hardware configuration of an advertising monitoring device. [Figure 8] Figure 8 schematically shows an example of the software configuration of an advertising monitoring device. [Figure 9] Figure 9 is a flowchart showing an example of the processing procedure for replacing advertising materials using an advertising monitoring device. [Figure 10] Figure 10 is a flowchart showing an example of the processing procedure for group optimization by an advertising monitoring device. [Modes for carrying out the invention]

[0027] Hereinafter, embodiments relating to one aspect of this disclosure will be described with reference to the drawings. However, the embodiments described below are merely illustrative in all respects of this disclosure. Various improvements or modifications may be made without departing from the scope of this disclosure. In implementing this disclosure, specific configurations may be adopted as appropriate depending on the embodiment. In this embodiment, the data appearing is described in natural language, but more specifically, it is specified in pseudo-language, commands, parameters, machine code, electrical signals, etc., that can be recognized by machines such as computers.

[0028] §1 Examples of Application Figure 1 schematically shows an example of a scenario to which this disclosure applies. The advertising monitoring device 1 according to this embodiment monitors the performance of advertising materials and determines whether or not to replace the advertising materials based on the monitoring results. It is one or more computers configured to make such determinations.

[0029] In this embodiment, the ad monitoring device 1 continuously monitors the performance of the first ad material 20 being delivered as the target ad 2. After the adjustment period for the first ad material 20 has elapsed, the ad monitoring device 1 determines whether the replacement conditions are met according to the performance monitoring results. If the adjustment period has not elapsed or the replacement conditions are not met, the ad monitoring device 1 may continue to monitor the performance of the first ad material 20. Depending on whether the replacement conditions are met, the ad monitoring device 1 outputs information 40 (replacement information) for processing the replacement of the first ad material 20 with the second ad material 25. By performing the replacement of the ad material in accordance with the output of this information 40, the second ad material 25 is delivered as the new ad material for the target ad 2. The ad monitoring device 1 may repeat the above series of processes, starting from treating the second ad material 25 as the new first ad material 20 and monitoring its performance.

[0030] In this embodiment, the advertising monitoring device 1 withholds the replacement of advertising materials according to the performance monitoring results until the adjustment period has elapsed. After the adjustment period has elapsed, the advertising monitoring device 1 determines whether or not to replace the advertising material from the first advertising material 20 to the second advertising material 25 according to the performance monitoring results. This avoids using unstable performance monitoring results during the adjustment period as an indicator for replacement. Therefore, according to this embodiment, it is possible to eliminate problems that may occur when determining the timing of replacement according to the performance monitoring results of the advertising material (first advertising material 20).

[0031] [advertisement] An advertisement (Target Advertisement 2) may be any information intended to inform people about at least one of the target products and services. Multiple advertisements (different advertisements from each other) may be prepared for the same product or service for any reason, such as different target audiences. The number of advertisements prepared for the same product or service may be determined as appropriate depending on the embodiment.

[0032] The advertising materials (first advertising material 20, second advertising material 25) may be materials used in the advertisement. The advertising materials may consist of any type of data, such as text, symbols, figures, images (still images, moving images), and sound. One piece of advertising material may constitute part or all of the advertisement. In one example, advertising material may be treated as synonymous with one or more assets. In this case, advertising material may be called an asset. However, the unit of advertising material is not limited to this example. In another example, advertising material does not have to correspond to an asset. The number of advertising materials prepared for a single advertisement is not particularly limited and may be determined as appropriate depending on the embodiment.

[0033] Advertisements (advertising materials) may be delivered by any method. For example, advertisements (advertising materials) may be delivered on the web. Web delivery methods may include any method instructed by a server, such as delivery on a web page, playback within a web video, or playback on digital signage. Advertisements (advertising materials) may be delivered on an advertising platform PF. In this case, the advertising delivery system may consist of an advertising monitoring device 1 and an advertising platform PF. The advertising platform PF may be configured as appropriate to control the delivery of advertisements (advertising materials). The advertising platform PF may consist of any server device, such as a server that provides a search engine (e.g., Google). The advertising platform PF may also consist of a cloud system. The destination of the advertising materials may be at least one of an internal server and an external server. External servers may include, for example, web servers such as e-commerce sites and news sites.

[0034] [Performance Monitoring] The performance of advertising materials (ads) may be monitored in any way. For example, performance may relate to at least one of the effectiveness, quality, and viewing status of the advertising material (ads). Yes. Publicly known methods may be used for monitoring performance. For example, monitoring performance may include measuring the value of one or more performance metrics 3. The types of metrics 3 to be monitored are not particularly limited and may be appropriately selected depending on the embodiment. Publicly known metrics may be used for metric 3. The values ​​of metric 3 may be measured by publicly known methods. For example, one or more metrics 3 may consist of estimated click-through rate (estimated CTR), relevance, impression share (IS), impression share loss rate (IS loss rate), conversion rate (CVR), return on ad spend (ROAS), bounce rate, keyword relevance, or a combination thereof.

[0035] The estimated click-through rate (CTR) may be defined as an indicator showing the estimated percentage of viewers (users) who click on delivered advertisements. The estimated CTR may be calculated using any method. For example, the estimated CTR may be derived from past CTRs and impression counts. The impression count is the number of times the advertisement was displayed. Past CTRs and impression counts may be measured as appropriate. The estimated CTR may be expressed in any format. For example, the estimated CTR may be expressed as a rank such as below average, average, above average, etc.

[0036] The relevance of an ad can be defined as an indicator showing the degree of relevance between the user's search intent and the ad. Relevance can be calculated using any method. For example, relevance can be derived from the degree of match between the search query and the ad when the ad is displayed in a search engine. The relevance of an ad can be expressed in any format. For example, relevance can be expressed as a rank such as below average, average, above average, etc.

[0037] The impression share may be defined as an indicator showing the number of times an ad was actually displayed out of the total number of opportunities to display an ad. Whether or not an ad is displayed may be controlled in any way. For example, whether or not an ad is displayed may be controlled according to at least one of the budget and rank. At least one of the budget and rank may be specified in any way. The impression share may be calculated in any way. For example, the impression share may be calculated as "(number of impressions in which the user acquired an ad) / (total number of impressions in which the user could have acquired an ad)". The impression share may be expressed in any format. For example, the impression share may be expressed as a percentage.

[0038] On the other hand, the impression share loss rate may be defined as an indicator that shows the number of times an advertisement (advertising material) was not actually displayed out of the total number of opportunities to display it. The impression share loss rate may be calculated using any method. For example, the impression share loss rate may be calculated as "1 - (impression share)". The impression share loss rate may be expressed in any format. For example, the impression share loss rate may be expressed as a percentage.

[0039] The conversion rate may be defined as an indicator showing the percentage of users who accessed the media and achieved a desired outcome (such as a purchase or membership registration). The conversion rate may be calculated using any method. For example, the total conversion rate may be calculated as "(total number of conversions) / (total number of sessions)". Also, for example, the unique conversion rate may be calculated as "(number of unique conversions) / (number of unique users)". The conversion rate may be expressed in any format. For example, the conversion rate may be expressed as a percentage.

[0040] ROI may be constructed as an indicator showing the degree of return on advertising investment. ROI may be calculated using any method. For example, ROI may be "(Advertising It may be calculated as (sales of goods or services) / (advertising expenses). The return on advertising cost may be expressed in any format. For example, the return on advertising cost may be expressed as a percentage.

[0041] The bounce rate may be structured as an indicator of landing page quality. For example, the bounce rate may be structured to show the percentage of users who arrived at the target site from an advertisement (advertising material) and left the site after viewing only the first page. In this case, the bounce rate is structured so that the higher the bounce rate, the more likely it is that users will not obtain the information they desire. The bounce rate may be calculated in any way. For example, the bounce rate may be calculated as "(number of bounces) / (number of visits)". The bounce rate may be expressed in any format. For example, the bounce rate may be expressed as a percentage.

[0042] For the purpose of searching for advertisements (advertising materials), each advertisement (advertising material) may have one or more keywords associated with it. Keywords may be directly associated with the advertising materials (first advertising material 20, second advertising material 25). Keywords may also be indirectly associated with the advertising materials by being associated with the advertisement (target advertisement 2). Keywords may be assigned in any way. For example, keywords may be assigned manually. For example, keywords may be assigned automatically, at least partially, by any information processing. As an example of an automatic assignment method, keywords to be associated may be extracted from the advertising materials by analyzing the advertising materials (natural language processing, etc.). Publicly known methods may be used for analysis. A trained model generated by machine learning may be used for analysis. At least some of the one or more keywords may be changed when replacing the first advertising material 20 with the second advertising material 25, or they may continue to be used without change after the replacement. If keywords are associated, monitoring the performance of the advertising materials may include monitoring the relevance of the keywords associated with the advertisements (advertising materials).

[0043] Keyword relevance may be structured as an indicator showing the degree of relevance of each of the one or more keywords associated with the advertisement (advertising material) to the user's search query. For example, keyword relevance may be structured to show the number of hits for the associated keywords (i.e., the number of times it matched the search query). Keyword relevance may be expressed in any format. For example, keyword relevance may be expressed in a ranking format (most hits, fewest hits, etc.) showing the number of hits for each keyword.

[0044] The types of metric 3 used are not limited to the examples above. One or more metric 3 may include any other metrics not listed above. Furthermore, the calculation process for determining the value of metric 3 may be performed on any computer. The calculation process for determining the value of metric 3 may be performed on at least one of the advertising monitoring device 1 and other computers. Other computers may include, for example, computers that constitute the advertising platform PF, server devices on which advertisements (advertising materials) are placed, etc. The value of metric 3 may be calculated on a single computer, or it may be calculated by multiple related computers working together. If at least a part of the calculation process for determining the value of metric 3 is performed on other computers, measuring the value of metric 3 may include obtaining the calculation results from other computers. If all of the calculation process for determining the value of metric 3 is performed on other computers, measuring the value of metric 3 may consist of obtaining the value of metric 3 (calculation result) calculated by other computers. For example, when an advertising platform PF calculates the value of metric 3, the measurement of the value of metric 3 by the advertising monitoring device 1 may consist of requesting the advertising platform PF to provide the measured value of metric 3, and obtaining the measured value of metric 3 from the advertising platform PF.

[0045] [Elapsed time during the adjustment period] The adjustment period is from the start of delivery of the target ad material (ad) until the adjustment of delivery attributes is completed. It may be a period until [a certain date is reached]. Distribution attributes may include, for example, the target audience, the devices to be used, the time of day to be used, etc. If no adjustment period is set, it may be interpreted as the adjustment period having elapsed.

[0046] Whether or not the adjustment period has elapsed can be determined by any method. For example, whether or not the adjustment period has elapsed can be determined based on whether or not a predetermined time has elapsed after the first advertisement material 20 has been submitted and the distribution of the first advertisement material 20 has started. If a predetermined time has elapsed since the start of distribution of the first advertisement material 20, it can be determined that the adjustment period has elapsed. On the other hand, if a predetermined time has not elapsed, it can be determined that the adjustment period has not elapsed. The predetermined time that serves as the criterion for the elapsed adjustment period can be appropriately defined depending on the embodiment.

[0047] Furthermore, during the adjustment period, the performance of advertising materials tends to be unstable due to changes in the attribute values ​​of the delivery. In other words, the measured value of metric 3 is prone to fluctuation. On the other hand, after the adjustment period has passed, the attributes of the delivery are fixed, so the fluctuations in the measured value of metric 3 tend to subside. Therefore, in another example, whether or not the adjustment period has passed can be determined using any metric (first metric 31) included in metric 3. That is, monitoring performance may include measuring the value of one or more first metric 31 related to performance. First metric 31 is metric 3 used to determine whether or not the adjustment period has passed. One or more metrics 3 may include one or more first metrics 31. The type of first metric 31 is not particularly limited and may be appropriately selected depending on the embodiment. In one example, first metric 31 may consist of estimated click-through rate, relevance, impression share, impression share loss rate, conversion rate, return on ad spend, bounce rate, keyword relevance, or a combination thereof. The advertising monitoring device 1 may determine whether the adjustment period for the first advertising material 20 has elapsed, based on the fluctuations in the measured values ​​of one or more first indicators 31. If it is determined that the adjustment period for the first advertising material 20 has elapsed, the device may determine whether the replacement conditions are met.

[0048] The relationship between the fluctuation of the measured value of the first indicator 31 and the criteria for determining whether or not the adjustment period has elapsed may be appropriately defined depending on the embodiment. As long as the system is configured such that a large fluctuation in the measured value makes it easier to determine that the adjustment period has not elapsed, and a small fluctuation in the measured value makes it easier to determine that the adjustment period has elapsed, the calculation for determining whether or not the adjustment period has elapsed according to the fluctuation of the measured value of the first indicator 31 is not particularly limited and may be appropriately defined depending on the embodiment.

[0049] For example, the value of the first indicator 31 may be measured repeatedly within a certain period. The measurement period and number of measurements may be appropriately defined depending on the embodiment. In the case of a ratio-type indicator 3 such as an estimated click-through rate, the advertising monitoring device 1 may calculate the amount of change between the measured values ​​obtained at each adjacent time point (i.e., the amount of change between the measured value at the target time point (k) and the measured value at the next time point (k+1)) for the measured values ​​obtained within a certain period. Subsequently, the advertising monitoring device 1 may calculate the sum of the amounts of change obtained between each adjacent time point within a certain period. The greater the fluctuation of the measured value of the first indicator 31, the higher the sum of the amounts of change will be, and the less the fluctuation, the lower the sum of the amounts of change will be. Therefore, the advertising monitoring device 1 may determine whether the adjustment period has elapsed depending on whether the sum of the obtained amounts of change exceeds a threshold. If the sum of the amounts of change exceeds the threshold, it may be determined that the adjustment period has not elapsed. If the sum of the amounts of change is less than the threshold, it may be determined that the adjustment period has elapsed. If the sum of the changes equals the threshold, it may be determined that the adjustment period has not yet elapsed or that the adjustment period has elapsed. The threshold may be arbitrarily defined.

[0050] On the other hand, in the case of cumulative metrics 3 such as keyword relevance (number of hits, etc.), the ad monitoring device 1 may calculate the cumulative value of the first metric 31 from the repeatedly measured values ​​as a fluctuation in the measured value of the first metric 31. Depending on whether the obtained cumulative value exceeds a threshold, the ad monitoring device 1 will... The system may determine whether or not the adjustment period has elapsed. If the cumulative value is less than the threshold, it may be determined that the adjustment period has not elapsed. If the cumulative value exceeds the threshold, it may be determined that the adjustment period has elapsed. If the cumulative value is equal to the threshold, it may be determined that either the adjustment period has not elapsed or the adjustment period has elapsed. The threshold may be arbitrarily defined.

[0051] When multiple indicators 3 are used as the first indicator 31, the relationship between the number of first indicators 31 that satisfy the conditions and the determination result of whether or not the adjustment period has elapsed may be appropriately defined depending on the embodiment. In one example, the advertising monitoring device 1 may determine that the adjustment period has elapsed when at least one of the multiple first indicators 31 satisfies the conditions. In another example, the advertising monitoring device 1 may determine that the adjustment period has elapsed when all of the multiple first indicators 31 satisfy the conditions.

[0052] This makes it possible to determine whether the adjustment period for the first advertisement material 20 has elapsed, based on the fluctuations in the measured values ​​of one or more first indicators 31. Note that the calculations related to the above determination, such as the calculation of the change in the measured values ​​of the first indicators 31, the calculation of the sum of the change amounts, and the calculation of the cumulative value, do not necessarily have to be performed on the advertisement monitoring device 1. At least a part of the calculations related to the above determination may be performed on another computer, such as an advertisement placement platform PF.

[0053] According to one example of this embodiment, the elapsed time of the adjustment period can be appropriately determined by the fluctuation of the indicator value (the measured value of the first indicator 31). This makes it possible to appropriately resolve problems that may occur when determining the timing of replacement according to the performance monitoring results of the advertising material (first advertising material 20).

[0054] [Replacement] The information 40 for processing the replacement may consist of any information that triggers the replacement automatically or manually. For example, the advertising monitoring device 1 may perform a replacement from the first advertising material 20 to the second advertising material 25. In this case, the information 40 for processing the replacement may include the second advertising material 25. Outputting the information 40 for processing the replacement may include performing a replacement from the first advertising material 20 to the second advertising material 25.

[0055] In another example, another computer, such as an advertising platform PF, may perform the replacement of the first advertising material 20 with the second advertising material 25. In this case, the information 40 for processing the replacement may include a command instructing the replacement with the second advertising material 25. Outputting the information 40 for processing the replacement may include outputting a command instructing the replacement with the second advertising material 25 to the other computer.

[0056] In yet another example, the replacement with the second advertisement material 25 may be performed manually. In this case, the information 40 for processing the replacement may include a notification prompting the replacement with the second advertisement material 25. Outputting the information 40 for processing the replacement may include sending a notification prompting the replacement with the second advertisement material 25 to the user terminal. If the advertisement monitoring device 1 is equipped with an output device, the notification prompting the replacement with the second advertisement material 25 may be output to the output device of the advertisement monitoring device 1, either together with the user terminal or in place of the user terminal.

[0057] The unit of replacement is not particularly limited and may be appropriately selected depending on the embodiment. Replacing the first advertisement material 20 with the second advertisement material 25 may consist of at least partially replacing the first advertisement material 20 with the second advertisement material 25 (i.e., replacing at least a part of one or more first advertisement materials 20 with at least a part of one or more second advertisement materials 25). Replacing the first advertisement material 20 with the second advertisement material 25 may consist of completely replacing the first advertisement material 20 with the second advertisement material 25, and partially replacing the first advertisement material 20 with the second advertisement material 25. This may include replacing it with a target.

[0058] [Replacement conditions] The replacement conditions are not particularly limited and may be defined as appropriate depending on the embodiment, provided that they are configured to trigger a replacement in accordance with the performance monitoring results (especially in response to poor performance).

[0059] For example, if the value of one or more indicators 3 is measured as part of performance monitoring, it may be determined whether the replacement condition is met using any indicator (second indicator 32) included in indicator 3. That is, monitoring performance may include measuring the value of one or more second indicators 32 related to performance. The second indicator 32 is indicator 3 used to determine whether the replacement condition is met. One or more indicators 3 may include one or more second indicators 32. The type of second indicator 32 is not particularly limited and may be appropriately selected depending on the embodiment. For example, the second indicator 32 may consist of estimated click-through rate, relevance, impression share, impression share loss rate, conversion rate, return on ad spend, bounce rate, keyword relevance, or a combination thereof. Determining whether the replacement condition is met according to the performance monitoring results may consist of determining whether the replacement condition is met according to the measured value of one or more second indicators 32.

[0060] The relationship between the measured value of the second indicator 32 and the criteria for determining the replacement condition may be defined as appropriate depending on the embodiment. As long as the system is configured such that it is easier to determine that the replacement condition is met when performance is poor and easier to determine that the replacement condition is not met when performance is good, the calculation for determining whether or not the replacement condition is met according to the measured value of one or more second indicators 32 is not particularly limited and may be defined as appropriate depending on the embodiment. In one example, the replacement condition may be composed of a threshold value for the measured value of the second indicator 32.

[0061] For example, in the case of a ratio-type metric 3 such as an estimated click-through rate, the ad monitoring device 1 may determine whether the replacement condition is met based on the result of comparing the measured value of the second metric 32 with a threshold. If a higher measured value of the second metric 32 indicates better performance, the ad monitoring device 1 may determine that the replacement condition is met, depending on whether the measured value of the second metric 32 is below the threshold. The ad monitoring device 1 may determine that the replacement condition is not met, depending on whether the measured value of the second metric 32 exceeds the threshold. On the other hand, if a lower measured value of the second metric 32 indicates better performance, the ad monitoring device 1 may determine that the replacement condition is not met, depending on whether the measured value of the second metric 32 is below the threshold. The ad monitoring device 1 may determine that the replacement condition is met, depending on whether the measured value of the second metric 32 exceeds the threshold. If the measured value of the second metric 32 is equal to the threshold, the device may determine whether the replacement condition is met or not. The threshold may be arbitrarily defined. In addition, the value of the second metric 32 may be measured repeatedly. In this case, the advertising monitoring device 1 may determine that the replacement condition is met depending on whether the measured value of the repeatedly obtained second indicator 32 is continuously below or above the threshold (i.e., continuously for a predetermined period). The number of times (predetermined period) that the replacement condition is met can be appropriately defined depending on the embodiment.

[0062] In the case of cumulative metrics 3 such as keyword relevance (number of hits, etc.), the ad monitoring device 1 may determine whether the replacement condition is met based on the result of comparing the increase in the measured value from the previous time (t-1) to the current time (t) with a threshold. The ad monitoring device 1 may determine that the replacement condition is not met if the increase in the second metric 32 exceeds the threshold. The ad monitoring device 1 may determine that the replacement condition is met if the increase in the second metric 32 is less than the threshold. If the increase in the second metric 32 is equal to the threshold, the replacement condition is met. The system may determine that the replacement condition is not met, or that the replacement condition is not met. The threshold can be arbitrarily defined. Furthermore, if the value of the second indicator 32 is measured repeatedly, the advertising monitoring device 1 may determine that the replacement condition is met if the amount of increase of the second indicator 32 obtained repeatedly is continuously less than the threshold. The number of times (predetermined period) that serves as the criterion for determining whether the replacement condition is met may be appropriately defined depending on the embodiment.

[0063] Furthermore, if one or more second indicators 32 include the degree of relevance for each of multiple keywords, the degree of relevance for each of the multiple keywords may include the ranking of the number of hits for each keyword. The ranking may be determined by the number of hits aggregated within any period (for example, a certain period). The ranking may be the current ranking obtained from measurements up to the most recent date. The replacement condition may be defined as a change in the ranking of a keyword. That is, the ad monitoring device 1 may determine whether the replacement condition is met depending on whether a change has occurred between the ranking at the previous time (t-1) and the ranking at the current time (t). The ad monitoring device 1 may determine that the replacement condition is met depending on whether a change has occurred in the ranking. The ad monitoring device 1 may determine that the replacement condition is not met depending on whether no change has occurred in the ranking. In addition, the replacement condition may further include the number of keywords for which a change has occurred. That is, the ad monitoring device 1 may determine that the replacement condition is met depending on whether the number of keywords for which a change in ranking has occurred from the previous time to the current time exceeds a threshold. The ad monitoring device 1 may determine that the replacement condition is met if the number of keywords whose ranking has changed from the previous time to the current time is less than a threshold. If the number of keywords whose ranking has changed is equal to the threshold, it may be determined that the replacement condition is met or not met. Note that the ranking to be compared with the current ranking does not necessarily have to be the ranking from the previous time. The ad monitoring device 1 may compare the ranking at any point in the past with the ranking at the current time and evaluate whether or not a change in the ranking has occurred.

[0064] When multiple indicators 3 are used as the second indicator 32, the relationship between the number of second indicators 32 that satisfy the conditions and the determination result of whether or not the replacement conditions are met may be appropriately defined depending on the embodiment. In one example, the advertising monitoring device 1 may determine that the replacement conditions are met if at least one of the multiple second indicators 32 satisfies the conditions. In another example, the advertising monitoring device 1 may determine that the replacement conditions are met if all of the multiple second indicators 32 satisfy the conditions. This makes it possible to determine whether or not the replacement conditions are met according to the measured value of one or more second indicators 32.

[0065] In one example, when the first indicator 31 is used to determine the elapsed adjustment period, one or more first indicators 31 and one or more second indicators 32 may be at least partially identical. That is, the indicator 3 used as the first indicator 31 and the indicator 3 used as the second indicator 32 may overlap. According to one example of this embodiment, the number of indicators 3 to be monitored can be arbitrarily narrowed down by the overlap between the indicator 3 used to determine the elapsed adjustment period (first indicator 31) and the indicator 3 used to determine the replacement condition (second indicator 32). The number of indicators 3 to be monitored does not have to be narrowed down, but if the number of indicators 3 to be monitored is narrowed down, a reduction in monitoring costs can be expected. However, the relationship between the first indicator 31 and the second indicator 32 is not limited to this example. In another example, one or more first indicators 31 and one or more second indicators 32 do not have to be identical.

[0066] [Preparing advertising materials] The advertising materials (first advertising material 20, second advertising material 25) may be generated by any method. The advertising materials may be generated at least partially by human hands, or they may be generated automatically by information processing such as a generative model obtained by machine learning. The advertising materials may be prepared at any time. The second advertising material 25 may be prepared at any time up to the point in time when the replacement of the first advertising material 20 is performed. The second advertising material 25 may be prepared immediately before replacing the first advertising material 20. It may be prepared while the first advertising material 20 is being distributed, or it may be prepared before the first advertising material 20 is distributed. In one example of this embodiment, at least one of the following two methods may be used to prepare the second advertising material 25.

[0067] (1) Method 1 Figure 2 schematically shows an example of a method for preparing the second advertising material 25. In one example, the advertising monitoring device 1 may give a generation instruction I1 to the large-scale generation model M1 before outputting information 40 for processing replacement, thereby causing the large-scale generation model M1 to generate the second advertising material 25. Accordingly, the replacement may be a replacement with the second advertising material 25 generated by the large-scale generation model M1. The generated advertising material may be used as the second advertising material 25 as is, or it may be used as the second advertising material 25 after any verification (user confirmation, application of verification processing, etc.) has been performed. According to this example, at least a part of the process of preparing the advertising material (second advertising material 25) can be automated. This can be expected to reduce the cost of generating advertising material.

[0068] The large-scale generative model M1 may be a trained model generated by machine learning using a large amount of training data. The large-scale generative model M1 may include, for example, a large-scale language model, a large-scale visual language model, a large-scale speech model, etc. The large-scale generative model M1 may also include a multimodal model. The configuration and number of parameters of the large-scale generative model M1 may be appropriately determined depending on the embodiment. The large-scale generative model M1 may include, for example, a Transformer, a multimodal model. It may incorporate any structure, such as a dispersed model. The large-scale generative model M1 may be fine-tuned according to its intended use. For example, known models such as Claude, GPT, and CyberAgentLM may be used for the large-scale generative model M1.

[0069] The large-scale generation model M1 may be deployed within the advertising monitoring device 1, or it may be deployed on another computer (such as an advertising platform PF). For example, if the large-scale generation model M1 is deployed within the advertising monitoring device 1, the advertising monitoring device 1 may generate the second advertising material 25 using the large-scale generation model M1 and transmit the generated second advertising material 25 to the advertising platform PF.

[0070] In another example, if the large-scale generation model M1 is deployed on the advertising platform PF, the advertising monitoring device 1 may cause the large-scale generation model M1 to generate the second advertising material 25 by sending a generation instruction I1 to the advertising platform PF. The advertising monitoring device 1 may provide the generation instruction I1 separately from the information 40 for processing the replacement (a command instructing the replacement with the second advertising material 25), or the command instructing the replacement with the second advertising material 25 may include the generation instruction I1.

[0071] In yet another example, if the large-scale generation model M1 is deployed on a computer other than the advertising monitoring device 1 and the advertising platform PF, the advertising monitoring device 1 may send a generation instruction I1 to the other computer, causing the large-scale generation model M1 to generate the second advertising material 25. The generated second advertising material 25 may be sent to the advertising platform PF via the advertising monitoring device 1, or it may be sent directly to the advertising platform PF.

[0072] The generation instruction I1 constitutes a prompt given to the large-scale generation model M1. The data format of the generation instruction I1 is not particularly limited and may be appropriately selected depending on the embodiment. For example, the generation instruction I1 may consist of data such as text, images, sounds, and tokens. Tokens may be obtained by converting data such as text with a tokenizer. Furthermore, the configuration of the generation instruction I1 is not particularly limited and may be appropriately determined depending on the embodiment, as long as it can instruct the generation of the second advertising material 25.

[0073] For example, if the values ​​of one or more indicators 3 are measured as part of performance monitoring, the target for improvement may be specified in the generation instruction I1 using any indicator (third indicator 33) included in indicator 3. That is, monitoring performance may include measuring the values ​​of one or more third indicators 33 related to performance. The third indicator 33 is an indicator 3 used to specify the target for improvement when generating advertising material. One or more indicators 3 may include one or more third indicators 33. The type of third indicator 33 is not particularly limited and may be appropriately selected depending on the embodiment. For example, the third indicator 33 may consist of estimated click-through rate, relevance, impression share, impression share loss rate, conversion rate, return on ad spend, bounce rate, keyword relevance, or a combination thereof. Accordingly, the generation instruction I1 may include an instruction sentence 331 indicating that the second advertising material 25 should be generated to improve the values ​​of the first advertising material 20 and one or more third indicators 33. According to this example, it is possible to expect the generation of advertising material (second advertising material 25) that can achieve the desired improvements.

[0074] For example, when the first indicator 31 is used to determine the elapsed adjustment period, one or more third indicators 33 may at least partially coincide with one or more first indicators 31, or they may not coincide with one or more first indicators 31. When the second indicator 32 is used to determine the replacement condition, one or more third indicators 33 may at least partially coincide with one or more second indicators 32, or they may not coincide with one or more second indicators 32.

[0075] The data format of the instruction statement 331 is not particularly limited and may be appropriately selected depending on the embodiment. For example, the instruction statement 331 may consist of data such as text, images, sounds, and tokens. The structure of the instruction statement 331 is not particularly limited and may be appropriately determined depending on the embodiment, as long as it can instruct the improvement of the third indicator 33. In one example, the instruction statement 331 may be structured to specifically indicate the degree of improvement, such as "Generate an ad to improve the estimated click-through rate by 5%." Alternatively, the instruction statement 331 may be structured without specifically indicating the degree of improvement, such as "Generate an ad to improve the estimated click-through rate."

[0076] One or more third indicators 33 may include the degree of relevance for each of the multiple keywords. The degree of relevance for each of the multiple keywords may include the ranking of the number of hits for each keyword. The ranking may be determined by the number of hits aggregated within any period (for example, a certain period). The ranking may be the current ranking obtained from measurements up to the most recent date. If there is a change in the ranking of each keyword, instruction statement 331 may include instruction statement 3315 indicating that the second advertisement material 25 should be generated prioritizing one or more first keywords that have newly ranked higher among the multiple keywords.

[0077] Whether a keyword is ranked highly or not may be evaluated according to predetermined criteria. The criteria for evaluating a keyword as being ranked highly may be arbitrarily determined, for example, nth place or higher, within the top n%, etc. The first keyword is a keyword that has not been ranked highly in past rankings but has been ranked highly in the target ranking (for example, the current ranking). As long as it is indicated that the first keyword should be prioritized, the structure of instruction sentence 3315 is not particularly limited and may be determined as appropriate depending on the embodiment. For example, instruction sentence 3315 may consist of a sentence that shows only one or more first keywords that have newly ranked highly. Alternatively, for example, instruction sentence 3315 may consist of a sentence that indicates that one or more first keywords should be prioritized over one or more second keywords that have fallen out of the top ranks due to changes in rankings. According to one example of this embodiment, by including keywords that match the user search trends at the time of ranking measurement in the generation instruction I1, it is possible to expect the generation of a second advertisement material 25 with high search performance (i.e., easily searchable).

[0078] Furthermore, the generation instruction I1 may include other information along with or in place of at least some of the above components. For example, the large-scale generative model M1 is known to have acquired the ability to perform in-context learning. In-context learning is the temporary learning of a task from a given input. To utilize this, the generation instruction I1 may further include one or more sample sets SS. Each of the one or more sample sets SS may include a first past advertisement 21 that was delivered in the past, a second past advertisement 26 that was delivered by replacing the first past advertisement 21, and the degree of improvement 335 of one or more third indicators 33 measured when the first past advertisement 21 was replaced with the second past advertisement 26. The first past advertisement 21 may be an advertisement that was delivered in the past as the first advertisement and is the advertisement before it was replaced with the second past advertisement 26. The second past advertisement 26 may be an advertisement that was delivered in the past as the second advertisement and is the advertisement after it was replaced with the first past advertisement 21. The degree of improvement 335 may be expressed in any format. For example, the degree of improvement 335 may be expressed as the amount of change before and after the replacement, such as "estimated click-through rate increased by 5%." Alternatively, the degree of improvement 335 may be expressed to indicate whether or not an improvement has occurred, such as "estimated click-through rate decreased." According to one example of this embodiment, by providing one or more sample sets SS and promoting in-context learning, it is possible to expect an improvement in the accuracy of generating advertising material (second advertising material 25) that can achieve the desired improvement.

[0079] (2) Second method Figure 3 schematically shows an example of a method for preparing the second advertising material 25. In one example, the advertising monitoring device 1 may select one or more candidates 250 to be used as the second advertising material 25 from a plurality of pre-provided candidates 250 of advertising materials before outputting information 40 for processing replacement. The second advertising material 25 may consist of the selected one or more candidates 250. Replacement may be a replacement with the selected second advertising material 25. According to this example, by providing candidates 250 in advance, the advertising material to be used (second advertising material 25) can be clearly identified.

[0080] The correspondence between candidate 250 and the second advertising material 25 may be determined as appropriate depending on the embodiment. The second advertising material 25 may consist of one candidate 250 or a combination of multiple candidates 250. Each candidate 250 may be generated by any method. Each candidate 250 may be generated at least partially by human effort, or it may be generated automatically by information processing such as a generative model obtained by machine learning. Each candidate 250 may be prepared at any time before being selected for the second advertising material 25.

[0081] Furthermore, the method for selecting one or more candidates 250 to be used as the second advertising material 25 from among multiple candidates 250 is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the second advertising material 25 (one or more candidates 250) may be randomly selected from among multiple candidates 250. In another example, each candidate 250 may be assigned a priority. The priority may be assigned in any way. For example, the priority may be determined according to attributes such as the registration date of each candidate 250 (prioritizing newly registered candidates, prioritizing older registered candidates, etc.). The priority may be specified by the user. The priority may also be determined by information processing such as a trained model (including a large-scale generative model) obtained by machine learning. The second advertising material 25 (one or more candidates 250) may be selected from among multiple candidates 250 according to the assigned priority. That is, one or more candidates 250 with the highest priority among multiple candidates 250 may be used as the second advertising material 25. Furthermore, the first advertising material 20 may be replaced with the second advertising material 25, and after a certain period of time, it may be reused as one or more candidate materials 250.

[0082] [Evaluation of the degree of improvement] In one example, after the second advertisement material 25 is delivered through a replacement process, the advertisement monitoring device 1 then... The performance of the second advertising material 25 may also be monitored. The advertising monitoring device 1 may then evaluate the degree of performance improvement resulting from the replacement of the first advertising material 20 with the second advertising material 25, based on the monitoring results of the second advertising material 25.

[0083] Figure 4 schematically shows an example of monitoring the performance of the second ad material 25 after replacement. In one example, when measuring the value of one or more indicators 3 for performance monitoring, the degree of performance improvement may be evaluated using any indicator (fourth indicator 34) included in indicator 3. That is, monitoring performance may include measuring the value of one or more fourth indicators 34 related to performance. The fourth indicator 34 is an indicator 3 used to evaluate the degree of improvement. One or more indicators 3 may include one or more fourth indicators 34. The type of fourth indicator 34 is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the fourth indicator 34 may consist of estimated click-through rate, relevance, impression share, impression share loss rate, conversion rate, return on ad spend, bounce rate, keyword relevance, or a combination thereof.

[0084] The ad monitoring device 1 may monitor the performance of the second ad material 25 that is being delivered after replacement. The ad monitoring device 1 may output information 45 regarding the results of evaluating the degree of improvement of each of the one or more fourth indicators 34 due to the replacement of the first ad material 20 with the second ad material 25, based on the monitoring results of the performance of the first ad material 20 when it was being delivered and the monitoring results of the performance of the second ad material 25. According to one example of this embodiment, the performance of the ad material after replacement (second ad material 25) can be evaluated. Similar to the first ad material 20, the ad monitoring device 1 may continuously monitor the performance of the second ad material 25. After the adjustment period of the second ad material 25 has elapsed, the ad monitoring device 1 may generate information 45 regarding the results of evaluating the degree of improvement of each of the one or more fourth indicators 34 due to the replacement. This makes it possible to stably evaluate the degree of improvement.

[0085] With regard to the degree of improvement of the fourth indicator 34, the configuration of the information 45 regarding the evaluation results of the degree of improvement is not particularly limited and may be determined as appropriate depending on the embodiment. For example, the information 45 regarding the evaluation results of the degree of improvement may be configured to directly indicate the degree of improvement of the fourth indicator 34, such as "estimated click-through rate increased by 5%." The degree of improvement may be expressed in any format such as arrows, symbols, or images.

[0086] In another example, one or more fourth indicators 34 may include one or more second indicators 32. Determining whether the replacement conditions are met according to the performance monitoring results may consist of determining whether the replacement conditions are met according to the measured values ​​of each of the one or more second indicators 32. In this case, evaluating the degree of improvement may include optimizing the replacement conditions from the trend of the measured values ​​of the fourth indicators 34, including the second indicators 32. That is, outputting information 45 regarding the results of the evaluation of the degree of improvement may include optimizing the replacement conditions according to the trend of the measured values ​​of each of the one or more fourth indicators 34 in the performance monitoring results of the first ad material 20 and the performance monitoring results of the second ad material 25 when they were being delivered. The information 45 regarding the results of the evaluation of the degree of improvement may be configured to show the results of the optimization of the replacement conditions. According to this example, the performance of the ad material can be expected to be maintained by optimizing the replacement conditions.

[0087] The method for optimizing the replacement conditions is not particularly limited and may be determined as appropriate depending on the embodiment, as long as it is configured to obtain replacement conditions that have a high probability of maintaining high performance. Known methods may be used for optimizing the replacement conditions. Optimizing the replacement conditions may include changing the second indicator 32 used to determine the replacement conditions by selecting the second indicator 32 from the fourth indicator 34. Furthermore, if the replacement conditions are composed of threshold values ​​for the measured values ​​of the second indicator 32, the replacement conditions Optimizing the process may include adjusting the threshold.

[0088] Any calculation formula, such as an evaluation function, may be used to optimize the replacement conditions. For example, the advertising monitoring device 1 may continuously acquire measured values ​​of the fourth indicator 34 for advertising materials (first advertising material 20, second advertising material 25). The advertising monitoring device 1 may use the measured values ​​acquired within a certain period to simulate the performance of the advertising materials and identify the replacement conditions that maximize performance. For example, an evaluation function may be prepared for each combination of the fourth indicator 34. The combination of the fourth indicator 34 may include the combination of the second indicator 32 currently used as a replacement condition. The evaluation function may be appropriately configured to derive the estimated performance score from the measured values ​​of the fourth indicator 34 and the threshold when the target fourth indicator 34 is used as the second indicator 32 and the replacement condition is configured with a specified threshold. For example, the evaluation function may be configured to calculate the performance score as the cumulative value (integral value, etc.) of the measured values ​​of the fourth indicator 34. The ad monitoring device 1 may calculate a performance score by providing the measured values ​​and thresholds of the fourth indicator 34 obtained over a certain period to an evaluation function and executing the calculation process of the evaluation function. The ad monitoring device 1 may repeatedly execute the calculation process of the evaluation function while changing the thresholds. Based on the obtained score, the ad monitoring device 1 may identify the optimal combination and threshold of the fourth indicator 34 to be used as the second indicator 32. In this way, the ad monitoring device 1 may optimize the replacement conditions.

[0089] In one example, after replacing the ad material delivered as target ad 2 from the first ad material 20 to the second ad material 25, the ad monitoring device 1 may treat the second ad material 25 as the new first ad material 20 and perform a series of processes related to the replacement. The ad monitoring device 1 may repeatedly perform performance monitoring, determination of replacement conditions, and replacement with the new ad material when the replacement conditions are met. In this way, the ad monitoring device 1 may continuously monitor the delivered ad material. Once a certain amount of historical data on the measured values ​​of the ad material has been accumulated through this continuous monitoring, the ad monitoring device 1 may optimize the replacement conditions. The second ad material 25 may be a new ad material, or it may be an ad material that has been delivered in the past (used ad material). In one example, multiple ad materials may be prepared, and the prepared multiple ad materials may be used sequentially as target ad 2. The multiple ad materials may be used in a loop, such as when the first ad material is used again after the last ad material has been used.

[0090] The relationship between the fourth indicator 34 and the other indicators (first indicator 31, second indicator 32, third indicator 33) may be determined as appropriate depending on the embodiment. When the first indicator 31 is used to determine the elapsed adjustment period, one or more fourth indicators 34 may at least partially coincide with one or more first indicators 31, or they may not coincide with one or more first indicators 31. When the optimization of the replacement conditions is omitted, one or more second indicators 32 may not at least partially coincide with one or more fourth indicators 34. When the third indicator 33 is used to specify the target for improvement during the generation of advertising materials, one or more fourth indicators 34 may at least partially coincide with one or more third indicators 33, or they may not coincide with one or more third indicators 33.

[0091] [Management Unit] In one example, one or more advertisements, including target advertisement 2, may be managed by any computer. Multiple advertisements, including target advertisement 2, may be managed by any computer. The computer managing the advertisements is not particularly limited and may be appropriately selected depending on the embodiment. The computer managing the advertisements may be, for example, at least one of the advertisement monitoring device 1, the advertisement placement platform PF, and other computers.

[0092] Managing the target advertisement may consist of accessing information (advertisement information 125) that identifies the target advertisement and being able to update the accessed information. Each advertisement may be managed on an arbitrary basis. For example, one or more advertisements may be managed individually. In this example, one or more advertisements may be managed on a group basis.

[0093] Figure 5 shows an example of how to manage advertisements in groups. In the example in Figure 5, advertisements A and B belong to group GA, and advertisements C and D belong to group GB. Each of advertisements A to D is an example of an advertisement managed in a group. In the example in Figure 5, the target advertisement 2 may be any of advertisements A to D, or it may be an advertisement that exists separately from advertisements A to D.

[0094] The allocation of each ad to a group may be determined by any method. For example, the allocation of each ad may be determined manually by the user. In another example, the allocation of each ad may be determined automatically by any classification process such as clustering. Any metric may be used for the allocation of each ad. For example, the allocation of each ad may be determined according to the similarity of ad attributes such as the ad's theme, the ad's subject (product, service, etc.), the target audience, and the device on which it is delivered. In other words, ad attributes may be used as the allocation metric. For example, the allocation of each ad may be determined manually by the user using the attributes of each ad. Alternatively, the allocation of each ad may be determined automatically by performing any classification process such as clustering using the values ​​of the attributes of each ad. The similarity of attributes may be evaluated by any method.

[0095] The number of groups is not particularly limited and may be determined as appropriate depending on the embodiment. Groups may be generated manually or automatically in any classification process such as clustering. The number of advertisements belonging to one group and the total number of advertisements managed are also not particularly limited and may be determined as appropriate depending on the embodiment.

[0096] For example, if one or more advertisements, including target advertisement 2, are managed as a group, the replacement conditions applied to the advertising material (first advertising material 20) delivered as target advertisement 2 may be replacement conditions linked to the group to which target advertisement 2 belongs. According to this example, it is possible to manage replacement conditions for all advertisements belonging to a group at once, compared to managing them for each advertisement. Therefore, a reduction in management costs can be expected.

[0097] Furthermore, in one example, when optimizing the replacement conditions described above, the replacement conditions to be optimized may be those linked to the group to which the target advertisement 2 belongs. According to this example, a reduction in management costs can be expected. Also, compared to optimizing the replacement conditions for each advertisement individually, since multiple advertisements can belong to one group, it becomes easier to collect samples (measured values ​​of the fourth indicator) used for optimizing the replacement conditions. As a result, it can be expected that optimization will become easier by obtaining more samples. Note that when multiple groups are established, the second indicator 32 used to determine the replacement conditions may differ between each group, or may at least partially match.

[0098] (Group Information / Advertising Information) If a group can be identified, the structure of the information used to manage the group (group information 120) is not particularly limited and may be determined as appropriate depending on the embodiment. Similarly, if each advertisement can be identified, the structure of the information used to manage each advertisement (advertisement information 125) is not particularly limited and may be determined as appropriate depending on the embodiment.

[0099] For example, group information 120 may include fields for storing information such as a group ID and replacement conditions. The group ID (group identifier) ​​may be used to identify the group. The replacement conditions may be linked to the target group by being included in group information 120.

[0100] In addition, for example, ad information 125 may include fields for storing information such as the ad ID, the group ID of the group to which the ad belongs, the ad material ID of each ad material, the usage period of each ad material, keywords, the history of the measured value (metric value) of metric 3, and the attributes of the ad. The ad ID (ad identifier) ​​may be used to identify the ad. The ad material ID (ad material identifier) ​​may be used to identify the ad material. The ad material may be linked to the target ad by the inclusion of the ad material ID in ad information 125. The usage period of the ad material may be the period during which the ad material was delivered. The usage period may be maintained for each ad material. Keywords may be linked to the target ad (ad material) by being included in ad information 125. The history of the measured value of metric 3 may be obtained as a result of monitoring the performance of the ad material.

[0101] The storage area for storing the group information 120 and the advertising information 125 is not particularly limited and may be appropriately selected depending on the embodiment, as long as the group information 120 and the advertising information 125 can be accessed as appropriate. For example, at least one of the group information 120 and the advertising information 125 may be stored in the memory resources of the advertising monitoring device 1, the memory resources of the advertising platform PF, or other storage area. Other storage areas may include, for example, the memory resources of another computer, an external storage device, a storage medium, etc. Similarly, the storage area for storing each advertising material (first advertising material 20, second advertising material 25) including the candidate 250 is not particularly limited and may be appropriately selected depending on the embodiment.

[0102] The configuration of each piece of information (120, 125) may be modified as appropriate depending on the embodiment. For example, if the replacement conditions are not linked to a group, the field for storing the replacement conditions may be omitted from the group information 120. If the replacement conditions are managed on an ad-by-ad basis, the field for storing the replacement conditions may be provided in the ad information 125.

[0103] (Group optimization) For example, if one or more advertisements, including target advertisement 2, are managed as a group, the advertisement monitoring device 1 may optimize the distribution of one or more advertisements to the group according to the results of monitoring the performance of the advertising materials used as each of the one or more advertisements. In other words, the distribution of advertisements to the group may be changed dynamically.

[0104] Figure 6 schematically shows an example of a scenario in which a group is optimized. In one example, the ad monitoring device 1, for each of the one or more ads, similar to the target ad 2, (a) the puff of the first ad material 20 (b) monitor performance, (c) determine whether the replacement conditions are met, (d) output information 40 for processing the replacement of the first ad material 20 to the second ad material 25 in accordance with whether the replacement conditions are met, and (e) the replacement of the second ad material 25 that is being distributed. (e) monitoring performance and replacing the first ad material 20 with the second ad material 25 The advertising monitoring device 1 may output information 45 regarding the results of evaluating the degree of improvement of each of the one or more fourth indicators 34. The advertising monitoring device 1 may repeatedly perform the series of processes from (a) to (e).

[0105] Furthermore, the ad monitoring device 1 may optimize the allocation of one or more ads to a group according to at least one of the trends in the measured values ​​of one or more fourth indicators 34 and the optimized replacement conditions. The trends in the measured values ​​of one or more fourth indicators 34 may include the degree of improvement in the values ​​of one or more fourth indicators 34 due to replacement. According to one example of this embodiment, optimizing the allocation to groups makes it easier to optimize the replacement conditions. Appropriate replacement conditions can be applied to each ad. Improved efficiency in managing replacement conditions can be expected.

[0106] Optimizing the distribution to groups involves redistributing each of the one or more ads to a group. It may be a matter of sorting. For example, optimizing the allocation to groups may include merging two or more groups (i.e., allocating ads belonging to different groups to the same group). For example, if substitution conditions are used as an indicator for group optimization, optimizing the allocation may include merging the first group to which the first ad belongs and the second group to which the second ad belongs if the optimized substitution conditions are similar. According to this example, it may be possible to reduce the number of groups to manage. This can be expected to reduce management costs. However, optimizing the allocation is not limited to this example. Optimizing the allocation may also include allocating a third ad and a fourth ad belonging to the same group to different groups.

[0107] Advertisements with similar trends in the measured values ​​of the fourth indicator 34 or similar optimized replacement conditions are more likely to be assigned to the same group, and advertisements with dissimilar trends in the measured values ​​of the fourth indicator 34 and similar optimized replacement conditions are more likely to be assigned to different groups. The method for optimizing the assignment to groups is not particularly limited and may be appropriately defined depending on the embodiment. Known classification methods such as clustering may be used as the method for optimizing the assignment.

[0108] For example, the trend of the measured value of the fourth indicator 34 may be evaluated by the distance between the measured values ​​of the ad pairs being compared. Similarly, if the replacement condition is composed of thresholds for the measured value of the second indicator 32, the optimized replacement condition may be evaluated by the distance between the thresholds of the ad pairs being compared. The type of distance may be appropriately selected depending on the embodiment. In one example, known distances such as cosine distance and Euclidean distance may be used for evaluation. The ad monitoring device 1 may optimize the allocation to groups by assigning ads that are evaluated as having a short calculated distance to the same group and assigning ads that are evaluated as having a long calculated distance to different groups.

[0109] The criteria for the distance required to assign ads to the same group may be defined as appropriate depending on the embodiment. For example, the ad monitoring device 1 may decide whether or not to assign ads to the same group by comparing the calculated distance with a threshold. If the calculated distance is less than the threshold, the ad monitoring device 1 may assign the pair of ads being compared to the same group. If the calculated distance exceeds the threshold, the ad monitoring device 1 may assign each of the ads being compared to a different group. If the calculated distance is equal to the threshold, the ad monitoring device 1 may assign each of the ads being compared to the same group or to different groups. The threshold distance for assigning ads to the same group may be defined arbitrarily.

[0110] For example, Ad A measures estimated click-through rate and relevance as the fourth metric 34, while Ad B measures estimated click-through rate and bounce rate. In such cases, the types of fourth metric 34 being monitored may differ at least partially between the ads being compared. In one example, the missing values ​​for the fourth metric 34 may be appropriately filled in. In the example above, Ad A lacks a bounce rate measurement, and Ad B lacks a relevance measurement. The missing values ​​for the fourth metric 34 may be filled in with arbitrary values, such as assigning 0 to the bounce rate measurement for Ad A and the relevance measurement for Ad B. The values ​​to be filled in may be arbitrarily defined. The same applies to the optimized substitution conditions (thresholds for the measurements of the second metric 32).

[0111] In one example, the optimization conditions may consist of both the trend of the measured value of the fourth indicator 34 and the optimized replacement conditions, or they may consist of only one of the trend of the measured value of the fourth indicator 34 and the optimized replacement conditions. In another example, the optimization conditions may further include other conditions other than the trend of the measured value of the fourth indicator 34 and the optimized replacement conditions. The other conditions may be arbitrarily given. For example, the other conditions may be the advertisements belonging to one group Constraints such as a maximum number and a minimum number may be included. This allows for the optimization of group assignments by taking into account factors other than the trend of the measured values ​​of the fourth indicator 34 and the optimized replacement conditions.

[0112] In the example shown in Figure 6, group optimization has resulted in ads A and B belonging to group GA, and ad C belonging to group GB, being allocated to group GC. Group GA is an example of one of the first and second groups, and group GB is an example of the other. Ads A and B are examples of one of the first and second ads, and ad C is an example of the other. As shown in Figure 6, merging the first and second groups may be done by allocating at least some of the ads belonging to the first group and at least some of the ads belonging to the second group to the same group. In one example, all of the ads belonging to the first group and all of the ads belonging to the second group may be allocated to the same group. Also, in the example shown in Figure 6, group optimization has resulted in ads C and D belonging to group GB being allocated to different groups. Ad C is an example of one of the third and fourth ads, and ad D is an example of the other.

[0113] When merging multiple groups, such as in the case of Group GC, the replacement conditions associated with the merged group may be appropriately derived from the replacement conditions associated with each group before the merger. For example, the replacement conditions associated with the merged group may be obtained by integrating the replacement conditions associated with each group before the merger. For example, the second indicator 32 used to determine the replacement conditions may be integrated by logical OR. As a specific example, if estimated click-through rate and relevance are used to determine the replacement conditions for Group GA, and estimated click-through rate and bounce rate are used to determine the replacement conditions for Group GB, then estimated click-through rate, relevance, and bounce rate may be used to determine the replacement conditions for Group GC. Also, for example, if the replacement conditions are composed of thresholds for the measured values ​​of the second indicator 32, and a common second indicator 32 is used to determine the replacement conditions for multiple groups before the merger, the thresholds for the measured values ​​of the common second indicator 32 may be integrated by calculating statistics such as the mean and median. For example, if the replacement condition for group GA includes a threshold AA for the estimated click-through rate, and the replacement condition for group GB includes a threshold BB for the estimated click-through rate, then the replacement condition for group GC may include a statistic derived from thresholds AA and BB as the threshold CC for the estimated click-through rate. The thresholds of the missing second indicators 32 may be used as they are. In the above example, the threshold for association included in the replacement condition for group GA, and the threshold for bounce rate included in the replacement condition for group GB, may be used as they are in the replacement condition for group GC.

[0114] Furthermore, when merging, the merging ratio may be determined according to priority. When merging the threshold of a common second indicator 32, statistics may be derived to prioritize one of the pre-merger groups using a weighted average or the like. Moreover, the merging ratio may include 0. In other words, merging replacement conditions may include using at least one of the pre-merger group replacement conditions as the post-merger replacement condition, according to priority. For example, if group GA has a higher priority than group GB, the replacement condition of group GA may be used as the replacement condition of group GC.

[0115] Priority can be determined in any way. For example, the more ads allocated to the merged group, the higher the priority given to the replacement conditions of the pre-merger group. In the example in Figure 6, more ads are allocated to group GC from group GA than from group GB. Therefore, the merger ratio can be determined so that the replacement conditions of group GA take precedence over those of group GB. Conversely, the fewer ads allocated to the merged group, the higher the priority given to the replacement conditions of the pre-merger group. In one example, the priority of each group before the merger may be specified by the user. In yet another example, the priority of each group before the merger may be determined according to the performance score calculated when optimizing the replacement conditions for the ads belonging to each group. The higher the performance score, the higher the priority may be determined.

[0116] On the other hand, in cases where no group merger occurs, such as with group GD, the replacement conditions linked to the original group may be used as is. In the example in Figure 6, the replacement conditions for group GB may be used as is for group GD.

[0117] Furthermore, when deploying one or more new advertisements, the initial value of the group to which each of the deployed new advertisements belongs may be determined by any method. For example, the advertisement monitoring device 1 may generate one or more new groups and assign each of the one or more new advertisements to one of the generated new groups. In another example, each of the one or more new advertisements may be assigned to an existing group to which an existing advertisement with similar attributes belongs. Attribute similarity may be evaluated by any method. In either case, after deploying the new advertisements, optimization of group assignment is performed, so it can be expected that the deployed new advertisements will be assigned to the appropriate group.

[0118] §2 Example Configuration [Hardware configuration] Figure 7 schematically shows an example of the hardware configuration of the advertising monitoring device 1 according to this embodiment. In one example, the advertising monitoring device 1 may be configured as a computer in which a control unit 11, a storage unit 12, a communication module 13, an input device 14, and an output device 15 are electrically connected.

[0119] The control unit 11 may include a hardware processor such as a CPU, RAM (Random Access Memory), and ROM (Read Only Memory), and is configured to perform information processing based on a program and various data. The control unit 11 (CPU) is an example of a processor resource.

[0120] The storage unit 12 may include, for example, a hard disk drive, a solid-state drive, or semiconductor memory, and is configured to hold arbitrary data. The storage unit 12, RAM, and ROM are examples of memory resources of the advertising monitoring device 1. In one example of this embodiment, the storage unit 12 may store various information such as an advertising monitoring program 81, group information 120, and advertising information 125.

[0121] The advertising monitoring program 81 is a program that causes the advertising monitoring device 1 to perform information processing related to the monitoring of advertising materials (Figures 9 and 10 described later). The advertising monitoring program 81 includes a series of instructions for said information processing. The advertising monitoring program 81 is an example of the program of this disclosure.

[0122] In one example, at least one of the advertising monitoring program 81, group information 120, and advertising information 125 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store various types of information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical means so that a machine such as a computer can read the information. The storage unit 12 and the storage medium 91 are examples of non-temporary storage media. The advertising monitoring device 1 may acquire at least one of the advertising monitoring program 81, group information 120, and advertising information 125 from the storage medium 91. The storage medium 91 may be a disk-type storage medium (CD, DVD, etc.) or a non-disk-type storage medium such as semiconductor memory (flash memory, etc.). Any drive device may be used to read the information stored in the storage medium 91. The type of drive device is... The drive device may be selected according to the medium 91. The drive device may be connected to the advertising monitoring device 1 in any way. The storage medium 91 may include an external storage device.

[0123] The communication module 13 is configured to perform wired or wireless communication over a network. The communication module 13 may consist of, for example, a wired LAN (Local Area Network) module, a wireless LAN module, etc. The network standard is not particularly limited and may be appropriately selected depending on the embodiment. For example, the type of network may be appropriately selected from the Internet, wireless communication network, mobile communication network, telephone network, dedicated network, etc. The advertising monitoring device 1 may perform data communication with other computers (for example, an advertising platform PF, etc.) via the communication module 13.

[0124] The input device 14 is configured to accept information input. The input device 14 may consist of, for example, a mouse, keyboard, or control. The output device 15 is configured to output information. The output device 15 may consist of, for example, a display or speaker. The user can operate the advertising monitoring device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be directly connected to the advertising monitoring device 1, or they may be indirectly connected via at least one of the communication module 13 and the external interface. The external interface may be appropriately configured to connect to an external device via wired or wireless connection, for example, a USB (Universal Serial Bus) port or a dedicated port. The input device 14 and the output device 15 may be integrated at least in part by a touch panel display or the like.

[0125] Furthermore, regarding the specific hardware configuration of the advertising monitoring device 1, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. Hardware processors include microprocessors, FPGAs (field-programmable gate arrays), DSPs (digital signal processors), and GPs. It may consist of a U (Graphics Processing Unit), an ASIC (application-specific integrated circuit), etc. At least one of the input device 14 and the output device 15 may be omitted. This may be omitted. At least one of the ad monitoring program 81, group information 120, and ad information 125 may be stored in an external storage device such as a NAS (Network Attached Storage). An external storage device is also an example of a non-temporary storage medium. The ad monitoring device 1 may consist of multiple computers. In this case, the hardware configuration of each computer may or may not be the same. The ad monitoring device 1 may be a computer designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC (Personal Computer), a notebook PC, a terminal device, etc. The terminal device may be a smartphone, a tablet, etc. This may include user terminals such as network terminals. When the ad monitoring device 1 operates as a server, users may access the ad monitoring device 1 via their user terminals.

[0126] [Software Configuration] Figure 8 schematically shows an example of the software configuration of the advertising monitoring device 1 according to this embodiment. The control unit 11 of the advertising monitoring device 1 executes instructions contained in the advertising monitoring program 81 stored in the storage unit 12 using the CPU. As a result, the advertising monitoring device 1 operates as a computer equipped with a monitoring unit 111, a progress determination unit 112, a replacement determination unit 113, a replacement processing unit 114, an evaluation processing unit 115, and a group optimization unit 116 as software modules. In other words, in this example, each software module of the advertising monitoring device 1 is realized by the control unit 11 (CPU).

[0127] The monitoring unit 111 is configured to continuously monitor the performance of the first advertising material 20 being delivered as the target advertisement 2. The progress determination unit 112 is configured to determine whether the adjustment period for the first advertising material 20 has elapsed. The replacement determination unit 113 determines whether the adjustment period for the first advertising material 20 has elapsed. After the set time has elapsed, the system is configured to determine whether the replacement conditions are met based on the performance monitoring results. The replacement processing unit 114 is configured to output information 40 for processing the replacement of the first advertising material 20 with the second advertising material 25, depending on whether the replacement conditions are met.

[0128] In one example, the replacement processing unit 114 may be configured to cause the large-scale generation model M1 to generate the second advertising material 25 by giving a generation instruction I1 to the large-scale generation model M1 before outputting the information 40 for processing the replacement. In another example, the replacement processing unit 114 may be configured to select one or more candidates 250 to be used as the second advertising material 25 from a plurality of pre-provided advertising material candidates 250 before outputting the information 40 for processing the replacement.

[0129] For example, monitoring performance may include measuring the value of one or more fourth indicators 34 related to performance. The monitoring unit 111 may be further configured to monitor the performance of the second ad material 25 that is being delivered by replacement. The evaluation processing unit 115 may be configured to output information 45 regarding the results of evaluating the degree of improvement of one or more fourth indicators 34 by replacing the first ad material 20 with the second ad material 25, according to the monitoring results of the performance of the first ad material 20 when it was being delivered and the monitoring results of the performance of the second ad material 25.

[0130] In one example, the one or more fourth indicators 34 may include one or more second indicators 32. The replacement determination unit 113 may be configured to determine whether or not the replacement conditions are met according to the measured values ​​of each of the one or more second indicators 32. In this case, the evaluation processing unit 115 may include a condition optimization unit 1151. The condition optimization unit 1151 may be configured to optimize the replacement conditions according to the trend of the measured values ​​of each of the one or more fourth indicators 34 in the monitoring results of the performance of the first ad material 20 and the monitoring results of the performance of the second ad material 25 when they were being delivered.

[0131] In one example, one or more advertisements, including the target advertisement 2, may be managed as a group. For each of the one or more advertisements, the monitoring unit 111, progress determination unit 112, replacement determination unit 113, replacement processing unit 114, and evaluation processing unit 115 may be executed. The group optimization unit 116 may be configured to optimize the allocation of each of the one or more advertisements to the group according to at least one of the trends in the measured values ​​of each of the one or more fourth indicators 34 and the optimized replacement conditions.

[0132] In this example of the embodiment, each software module of the advertising monitoring device 1 is implemented by a general-purpose CPU. However, the method of implementing each of the above modules is not limited to this example. Some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may also be implemented as a hardware module. Regarding the software configuration of the advertising monitoring device 1, modules may be omitted, replaced, and added as appropriate, depending on the embodiment.

[0133] §3 Example of Operation [Replacing advertising materials] Figure 9 is a flowchart showing an example of the processing procedure for replacing advertising materials by the advertising monitoring device 1 according to this embodiment. The following processing procedure is merely an example, and each step may be modified as much as possible. Furthermore, depending on the embodiment, steps in the following processing procedure can be omitted, replaced, and added as appropriate.

[0134] (Step S101) In step S101, the control unit 11 operates as a monitoring unit 111 and monitors the performance of the first advertising material 20 being delivered as the target advertisement 2.

[0135] In one example, the control unit 11 may measure the value of one or more indicators 3 for the first advertising material 20. The one or more indicators 3 may include at least one of one or more first indicators 31, one or more second indicators 32, one or more third indicators 33, and one or more fourth indicators 34. In one example, when the control unit 11 measures the values ​​of one or more first indicators 31 and one or more second indicators 32, the one or more first indicators 31 and one or more second indicators 32 may at least partially coincide. The number of times the measurement value is obtained in one step may be appropriately determined depending on the embodiment. Once the monitoring result of the performance of the first advertising material 20 is obtained, the control unit 11 proceeds to the next step S102.

[0136] (Step S102) In step S102, the control unit 11 operates as a progress determination unit 112 and determines whether or not the adjustment period for the first advertising material 20 has elapsed.

[0137] In one example, the control unit 11 may determine whether the adjustment period for the first advertising material 20 has elapsed based on the fluctuations in the measured values ​​of one or more first indicators 31. If it is determined that the adjustment period for the first advertising material 20 has elapsed, the control unit 11 proceeds to the next step S103. On the other hand, if it is determined that the adjustment period for the first advertising material 20 has not elapsed, the control unit 11 returns to step S101 and executes the process again from step S101.

[0138] (Step S103) In step S103, the control unit 11 operates as a replacement determination unit 113 and, after the adjustment period for the first advertising material 20 has elapsed, determines whether or not the replacement conditions are met according to the performance monitoring results.

[0139] In one example, the control unit 11 may determine whether the replacement condition is met according to the measured value of one or more second indicators 32. In another example, if one or more advertisements, including the target advertisement 2, are managed as a group, the replacement condition may be a replacement condition linked to the group to which the target advertisement 2 belongs.

[0140] If it is determined that the replacement conditions are not met, the control unit 11 returns to step S101 and executes the process again from step S101. In one example, the performance of the first advertising material 20 may be continuously monitored by repeatedly executing the process in step S101 due to the determination processes in steps S102 and S103. On the other hand, if it is determined that the replacement conditions are met, the control unit 11 proceeds to the next step S104.

[0141] (Step S104) In step S104, the control unit 11 operates as a replacement processing unit 114 and outputs information 40 for processing the replacement of the first advertising material 20 with the second advertising material 25, depending on whether the replacement conditions are met.

[0142] In one example, the control unit 11 may operate as a replacement processing unit 114 at any timing before executing the process in step S104, and give a generation instruction I1 to the large-scale generation model M1, causing the large-scale generation model M1 to generate the second advertising material 25. In this case, the replacement may be a replacement with the second advertising material 25 generated by the large-scale generation model M1. The generation instruction I1 may be appropriately configured to cause the large-scale generation model M1 to generate the second advertising material 25. In one example, the control unit 11 measures the values ​​of one or more third indicators 33. If so, the generation instruction I1 may include an instruction statement 331 indicating that the second advertisement material 25 should be generated to improve the values ​​of the first advertisement material 20 and one or more third indicators 33. The generation instruction I1 may also further include one or more sample sets SS. Each of the one or more sample sets SS may include the first past advertisement material 21, the second past advertisement material 26, and the degree of improvement 335 of the values ​​of one or more third indicators 33 measured when the first past advertisement material 21 was replaced with the second past advertisement material 26.

[0143] In one example, the control unit 11 may, at any time before executing the process in step S104, act as a replacement processing unit 114 and select one or more candidates 250 to be used as the second advertising material 25 from a given list of candidates 250 for advertising materials. In this case, the replacement may be a replacement with the selected second advertising material 25 (one or more candidates 250). After outputting information 40 for processing the replacement, the control unit 11 proceeds to the next step S105.

[0144] (Step S105) In step S105, the control unit 11 operates as a monitoring unit 111 and monitors the performance of the second advertising material 25 being distributed by replacement.

[0145] The processing in step S105 may be the same as in step S101 above. In one example, the control unit 11 may measure the value of one or more indicators 3 for the second advertising material 25. The one or more indicators 3 may include at least one of one or more first indicators 31, one or more second indicators 32, one or more third indicators 33, and one or more fourth indicators 34. Once the monitoring results for the performance of the second advertising material 25 are obtained, the control unit 11 proceeds to the next step S106.

[0146] (Step S106) In step S106, the control unit 11 operates as a progress determination unit 112 and determines whether or not the adjustment period for the second advertising material 25 has elapsed.

[0147] The processing in step S106 may be the same as in step S102 above. In one example, the control unit 11 may determine whether the adjustment period for the second advertising material 25 has elapsed according to the fluctuation of the measured values ​​of one or more first indicators 31. If it is determined that the adjustment period for the second advertising material 25 has elapsed, the control unit 11 proceeds to the next step S107. On the other hand, if it is determined that the adjustment period for the second advertising material 25 has not elapsed, the control unit 11 returns to step S105 and executes the processing again from step S105. In one example, the performance of the second advertising material 25 may be continuously monitored by repeatedly executing the processing in step S105 as a result of the determination process in step S106.

[0148] (Step S107) In step S107, the control unit 11 operates as an evaluation processing unit 115 and outputs information 45 regarding the results of evaluating the degree of improvement achieved by replacing the first advertising material 20 with the second advertising material 25.

[0149] In one example, the control unit 11 may output information 45 regarding the results of evaluating the degree of improvement in the value of one or more fourth indicators 34 by replacing the first ad material 20 with the second ad material 25, based on the monitoring results of the performance of the first ad material 20 and the performance of the second ad material 25 when they were being delivered. In another example, the control unit 11 may operate as a condition optimization unit 1151. That is, the control unit 11 may optimize the replacement conditions according to the trends of the measured values ​​of one or more fourth indicators 34 in the monitoring results of the performance of the first ad material 20 and the performance of the second ad material 25 when they were being delivered. In one example, one or more ads, including the target ad 2, are managed as a group. In this case, the replacement condition to be optimized may be the replacement condition associated with the group to which the target ad 2 belongs.

[0150] Furthermore, when evaluating the degree of improvement due to the replacement, optimization of the replacement conditions is not necessarily required. Periodically, after a certain period of time, if the measured value of the fourth indicator 34 satisfies predetermined conditions or any other conditions are met, the control unit 11 may perform a process to optimize the replacement conditions. If the arbitrary conditions are not met, the control unit 11 may omit the execution of the process to optimize the replacement conditions. Once the output of information 45 regarding the results of the evaluation of the degree of improvement due to the replacement is completed, the control unit 11 terminates the processing procedure related to the replacement of advertising materials in this example of operation.

[0151] In one example, the control unit 11 may execute the processes from steps S101 to S107 in real time. In another example, the control unit 11 may treat the second advertisement material 25 delivered by replacement as a new first advertisement material 20 and repeat the execution of the processes from step S101. In yet another example, if one or more advertisements including the target advertisement 2 are provided, the control unit 11 may execute the processes from steps S101 to S107 for each of the one or more advertisements.

[0152] [Group Optimization] Figure 10 is a flowchart showing an example of a processing procedure for group optimization by the advertising monitoring device 1 according to this embodiment. The processing procedures in Figures 9 and 10 are examples of information processing methods (advertising monitoring methods) executed by a computer. However, the processing procedure in Figure 10 is merely an example, and each step may be modified as much as possible.

[0153] Before executing the process in step S201, the control unit 11 may execute the processes in steps S101 to S107 once or more times for each of the one or more advertisements. This may result in obtaining the measured values ​​of one or more fourth indicators 34 and optimized replacement conditions for each of the one or more advertisements.

[0154] In step S201, the control unit 11 operates as a group optimization unit 116. That is, the control unit 11 optimizes the allocation of one or more advertisements to groups according to at least one of the trends in the measured values ​​of one or more fourth indicators 34 and the optimized replacement conditions. For example, optimizing the allocation may include merging the first group to which the first advertisement belongs and the second group to which the second advertisement belongs if the optimized replacement conditions are similar.

[0155] Once the optimization of the distribution is complete, the control unit 11 terminates the processing procedure related to the optimization of the group in this example of operation. The control unit 11 may execute the process of step S201 at any time, such as periodically, by a user trigger, or when the replacement conditions have been optimized.

[0156] [Features] In this embodiment, the advertising monitoring device 1 suspends the replacement of advertising materials according to the performance monitoring results until the adjustment period has elapsed, as determined by the process in step S102. After the adjustment period has elapsed, the advertising monitoring device 1 determines, in the process in step S103, whether or not to replace the advertising material from the first advertising material 20 to the second advertising material 25, according to the performance monitoring results. This avoids using unstable performance monitoring results during the adjustment period as an indicator for replacement. Therefore, according to this embodiment, it is possible to eliminate problems that may occur when determining the timing of replacement according to the performance monitoring results of the advertising material (first advertising material 20).

[0157] §4 Variant While embodiments of this disclosure have been described in detail above, the above description is merely illustrative in all respects. The processes and means described in this disclosure can be freely combined and implemented as long as no technical inconsistencies arise. Furthermore, various improvements or modifications may be made to the above embodiments as appropriate. For example, the following modifications are possible. In the following, the same reference numerals are used for components similar to those in the above embodiments, and explanations of points similar to those in the above embodiments have been omitted as appropriate.

[0158] In the above embodiment, the advertising monitoring device 1 is a separate computer from the advertising platform PF. However, the configuration of the advertising distribution system is not limited to this example. The advertising monitoring device 1 may be a computer that constitutes at least a part of the advertising platform PF. In other words, the advertising monitoring device 1 and the advertising platform PF may be configured as an integrated unit.

[0159] Furthermore, the processing procedure of the advertising monitoring device 1 according to the above embodiment may be omitted, replaced, or added to at least one of the steps. For example, the processing of step S201 may be omitted. Accordingly, the group optimization unit 116 may be omitted from the software configuration of the advertising monitoring device 1. Also, for example, the processing to optimize the replacement conditions may be omitted. Accordingly, the condition optimization unit 1151 may be omitted from the software configuration of the advertising monitoring device 1. Also, for example, the processing of step S106 may be omitted. Also, for example, the processing of steps S105 and S107 may be omitted. Accordingly, the evaluation processing unit 115 may be omitted from the software configuration of the advertising monitoring device 1. [Explanation of Symbols]

[0160] 1…Advertising monitoring device, 11...Control unit, 12...Storage unit, 13...Communication module, 14...Input device, 15...Output device, 81...Advertising monitoring program, 91...Storage medium, 2...Advertising, PF...Advertising platform, 20...First advertising material, 25...Second advertising material, 250... Candidates, 21...First past advertisement, 26...Second past advertisement, 3...Indicators, 31...first indicator, 32...second indicator, 33...Third indicator, 34...Fourth indicator, 40... (Replacement) information, 45... (Evaluation) information, I1…Generation instruction, SS…Sample set, 331... Instructions, 335... Degree of improvement, M1...Large-scale generative model

Claims

1. To continuously monitor the performance of the primary ad material currently being delivered as the target ad, After the adjustment period for the first advertising material has elapsed, it is determined whether or not the replacement conditions are met according to the performance monitoring results, and In response to the fulfillment of the aforementioned replacement conditions, information for processing the replacement of the first advertising material with the second advertising material is output. A control unit configured to perform the following actions: Advertisement monitoring device.

2. Monitoring the performance includes measuring the value of one or more first indicators relating to the performance, The control unit is further configured to determine whether or not the adjustment period for the first advertising material has elapsed, based on the fluctuations in the measured values ​​of each of the one or more first indicators. If it is determined that the adjustment period for the first advertising material has elapsed, a determination is made as to whether or not the replacement conditions are met. The advertising monitoring device according to claim 1.

3. Monitoring the performance further includes measuring the value of one or more second indicators relating to the performance, Determining whether the replacement conditions are met according to the performance monitoring results is comprised of determining whether the replacement conditions are met according to the measured values ​​of each of the one or more second indicators. The advertising monitoring device according to claim 2, wherein the one or more first indicators and the one or more second indicators are at least partially identical.

4. The control unit is configured to further cause the large-scale generation model to generate the second advertising material by giving a generation instruction to the large-scale generation model before outputting the information for processing the replacement. The aforementioned replacement is a replacement with the second advertising material generated by the large-scale generation model. The advertising monitoring device according to claim 1.

5. Monitoring the performance further includes measuring the value of one or more third indicators relating to the performance, The aforementioned generation instruction is: The first advertising material, and Instructions indicating that the second advertising material will be generated in such a way that the values ​​of each of the one or more third indicators mentioned above are improved. including, The advertising monitoring device according to claim 4.

6. The generation instruction further includes one or more sample sets, Each of the one or more sample sets mentioned above is: Previously distributed advertisements, The second past advertising material, which was distributed by replacing the first past advertising material, and The degree of improvement in the value of each of the one or more third indicators measured when the first past advertising material is replaced with the second past advertising material, including, The advertising monitoring device according to claim 5.

7. The control unit is configured to further select one or more candidates to be used as the second advertising material from a plurality of pre-given candidates for advertising material before outputting information for processing the replacement. The aforementioned replacement is a replacement with the selected second advertising material. The advertising monitoring device according to claim 1.

8. One or more advertisements, including the aforementioned advertisement, are managed on a group basis. The aforementioned replacement conditions are replacement conditions linked to the group to which the aforementioned advertisement belongs. The advertising monitoring device according to claim 1.

9. Monitoring the performance further includes measuring the value of one or more fourth indicators relating to the performance, The control unit, The performance of the second advertising material being distributed as a result of the aforementioned replacement, and Based on the monitoring results of the performance of the first advertising material while it was being distributed, and the monitoring results of the performance of the second advertising material, information is output regarding the results of evaluating the degree of improvement in each of the one or more fourth indicators resulting from replacing the first advertising material with the second advertising material. It is configured to perform further actions. The advertising monitoring device according to claim 1.

10. The aforementioned one or more fourth indicators include one or more second indicators, Determining whether the replacement conditions are met according to the performance monitoring results is comprised of determining whether the replacement conditions are met according to the measured values ​​of each of the one or more second indicators. Outputting information regarding the results of the evaluation of the degree of improvement includes optimizing the replacement conditions according to the trends of the measured values ​​of each of the one or more fourth indicators in the monitoring results of the performance of the first advertising material while it was being distributed, and in the monitoring results of the performance of the second advertising material. The advertising monitoring device according to claim 9.

11. One or more advertisements, including the aforementioned advertisement, are managed on a group basis. The replacement conditions to be optimized are replacement conditions linked to the group to which the target advertisement belongs. The advertising monitoring device according to claim 10.

12. The control unit, for each of the one or more advertisements, To monitor the performance of the first advertising material, To determine whether the aforementioned replacement conditions are met, In response to the fulfillment of the replacement conditions, information for processing the replacement of the first advertising material with the second advertising material is output. The replacement allows for monitoring the performance of the second advertising material being distributed, and To output information regarding the results of evaluating the degree of improvement in the value of each of the one or more fourth indicators resulting from replacing the first advertising material with the second advertising material. It is configured to perform, The control unit analyzes the trend of the measured values ​​of each of the one or more fourth indicators, and optimizes them. It is further configured to optimize the distribution of each of the one or more advertisements to the group according to at least one of the replacement conditions. The advertising monitoring device according to claim 11.

13. A method of information processing performed by a computer, To continuously monitor the performance of the primary ad material currently being delivered as the target ad, After the adjustment period for the first advertising material has elapsed, it is determined whether or not the replacement conditions are met according to the performance monitoring results, and In response to the fulfillment of the aforementioned replacement conditions, information for processing the replacement of the first advertising material with the second advertising material is output. including, Information processing methods.

14. Monitoring the performance further includes measuring the value of one or more fourth indicators relating to the performance, The aforementioned information processing method is: The performance of the second advertising material being distributed as a result of the aforementioned replacement, and Based on the monitoring results of the performance of the first advertising material while it was being distributed, and the monitoring results of the performance of the second advertising material, information is output regarding the results of evaluating the degree of improvement in each of the one or more fourth indicators resulting from replacing the first advertising material with the second advertising material. This also includes, The information processing method according to claim 13.

15. A program that causes a computer to execute an information processing method, The aforementioned information processing method is: To continuously monitor the performance of the primary ad material currently being delivered as the target ad, After the adjustment period for the first advertising material has elapsed, it is determined whether or not the replacement conditions are met according to the performance monitoring results, and In response to the fulfillment of the aforementioned replacement conditions, information for processing the replacement of the first advertising material with the second advertising material is output. including, program.

16. Monitoring the performance further includes measuring the value of one or more fourth indicators relating to the performance, The aforementioned information processing method is: The performance of the second advertising material being distributed as a result of the aforementioned replacement, and Based on the monitoring results of the performance of the first advertising material while it was being distributed, and the monitoring results of the performance of the second advertising material, information is output regarding the results of evaluating the degree of improvement in each of the one or more fourth indicators resulting from replacing the first advertising material with the second advertising material. This also includes, The program according to claim 15.

Images