Post-campaign analysis and automation system

Abstract

The present invention relates to a system (1) for evaluating campaign outcomes quickly and accurately for telecommunications companies and other industries that run mass campaigns.

Description

[0001] DESCRIPTION

[0002] POST-CAMPAIGN ANALYSIS AND AUTOMATION SYSTEM

[0003] Technical Field

[0004] The present invention relates to a system for evaluating campaign outcomes quickly and accurately for telecommunications companies and other industries that run mass campaigns.

[0005] Background of the Invention

[0006] Today, there are applications for assessing the impact of advertising campaigns on the sales of a product. During the pre-advertising period, a model with KPIs, such as net sales, number of interactions, etc. as target variables is established, and the data set for the previous period is fed to the model for training; this learned model is then applied to the data set for the next period. The values predicted with the data set for the next period are compared with the actual values, and a simulation is run on how sales would have been affected by the variables for the previous period and how sales would proceed without the advertisement. To do so, the model runs many simulations, and the different outcomes are averaged and compared with the actual value. Thus, it jointly assesses the impact of advertisements made through different channels. However, there are no applications that compare the before and after periods of subscribers who accept and reject the campaign with the Difference in Difference (DiD) method; score the subscribers who are most similar to them in the pre-campaign period on a scale of 0-1 for subscribers who accepted the campaign; apply Supervised Machine Learning methodology; feeds the model with subscribers who accept and reject the campaign, teaches the model the variables that most affect the acceptance of the campaign, and scores the subscribers among those who reject the campaign; constitutes the synthetic control group of the subscribers who scored the highest, i.e., the subscribers who were most similar to the experimental group; shows the campaign effect of the difference between the before and after periods of these two groups; and utilizes first a stratified sampling method and then models that employ classifiers, i.e. decision tree methodologies, such as XGBoost, Light GBM.

[0007] Therefore, considering the studies and deficiencies included in the current technique, it is understood that there is a need for a system that finds the most similar subscribers who do not benefit from the campaign and compares their before-and-after differences using a distance-based stratified sampling approach followed by supervised machine learning algorithms instead of time series models.

[0008] The United States patent document no. US2023267499, an application included in the state of the art, discloses a system for assessing the impact of advertising campaigns on the sale of a product. The invention being subject to the said United States patent document assesses the impact of advertising campaigns on the sale of a product -one of the most challenging problems faced by marketing professionals. One of the main reasons for the insufficient allocation of expenditure is the incorrect analysis of the effectiveness of the advertising campaign. Here, an approach to determine the effectiveness of an advertising campaign more accurately and reliably by using machine learning to unveil the true impact of an advertisement is introduced. This approach can be implemented by a data analytics platform that can train a machine learning algorithm using company- specific data as part of a training operation and apply the generated machine learning model as part of an inference operation.

[0009] Summary of the Invention The object of the present invention is to realize a system which is developed with the aim of evaluating campaign outcomes quickly and accurately for telecommunications companies and other industries that run mass campaigns.

[0010] Another object of the present invention is to realise a system which is developed with the aim of calculating the subscriber-based impacts of the campaign automatically with the synthetic control group created; and understanding quickly the successes of campaigns and the differences between different groups.

[0011] Another object of the present invention is to realise a system which is developed with the aim of assessing the outcomes of campaigns more quickly and precisely, while also taking into account regulatory requirements and customer satisfaction; and saving time by automating assessments done with manual processes.

[0012] Another object of the present invention is to realise a system developed which is developed with the aim of making campaign management more efficient and shaping future campaigns better.

[0013] Detailed Description of the Invention

[0014] “A Post-Campaign Analysis and Automation System” realized to fulfd the objectives of the present invention is shown in the figure, in which:

[0015] Figure 1 is a schematic view of the inventive system.

[0016] The components illustrated in the figure are individually numbered, where the numbers refer to the following:

[0017] 1. System

[0018] 2. Electronic Device

[0019] 3. Interface 4. Database

[0020] 5. Server

[0021] The inventive system (1) developed with the aim of evaluating campaign outcomes quickly and precisely comprises:

[0022] - at least one electronic device (2) which is configured to exchange data and run at least one application thereon by using any remote communication protocol;

[0023] - at least one interface (3) which is configured to be run on the electronic device (2); to enter the campaign name, the audience who joined the campaign to be assessed, the period to be analysed, the previous and next periods, the sample size for simulation, the characteristics to be simulated, the model to be used and the threshold value for the synthetic control group, and the model; and to view reports;

[0024] - at least one database (4) which is configured to keep records of the campaign name, the audience who joined the campaign to be assessed, the period to be analysed, the previous and next periods, the sample size for simulation, the characteristics to be simulated, the model to be used and threshold value for the synthetic control group, model details, such as Catboost, LightGBM and the data of the identified customer group, and the data of the subscribers in the campaign audience therein;

[0025] - at least one server (5) which is configured to establish a connection with the electronic device (2) using by any communication protocol and access the interface (3) thereon; to connect with the database (4) by using any communication protocol; to access and save data thereon; to access the data entered on the interface (3) and save them on the database; to run the relevant queries on the database (4) for the experiment and the sample to be taken from the database (4) based on the inputs entered on the interface (3); to run queries in the database (4) for details of the simulated control and experimental groups, such as their demographic characteristics, past habits of use, digital footprints, and mobility; to select similar customers among the millions of subscribers who have not joined the campaign by using a stratified sampling method according to critical variables, such as revenue per subscriber, usage, additional packages, overage, digital habits in the previous period in the sample rate entered through the interface (3) and record them on the database (4); to eliminate outliers first according to the characteristics to be simulated, fill in missing data and create layers / strata; to select subscribers with values closest to the averages of these strata and save them on the database (4); to collect the data of the identified customer group and the data of the subscribers in the campaign audience; to improve data quality by following the pre-processing steps of filling in missing or incorrect data in the dataset, eliminating extreme values, and organizing inconsistent data; to convert the data received through the database (4) into a suitable format; label the target audience expected to learn in the model, i.e. the campaign audience, as “1”; to label randomly selected and data-enriched customers from a similar customer base as “0”; to train a supervised machine learning model with the generated dataset; to identify subscribers from the unlabelled synthetic control group for whom the probability of 1 is greater than a certain threshold value and to create a synthetic control group; to run LGBM, XGBoost, CatBoost, Random Forrest algorithms; to improve on this synthetic control group with Cosine Similarity distance-based simulation methods if there is not enough similarity in the previous period according to the desired characteristics such as revenue per subscriber, usage; to calculate the revenue change and churn trend of the campaign audience after joining the campaign in comparison with the synthetic control group after creating a synthetic control group; to compare the experimental group and the synthetic control group for the analysis period and previous periods according to many characteristics such as demographics, customer behaviour, digitality, etc. for making sure that the synthetic control group is similar; to automatically report the previous and next period subscriberbased effects and detail characteristics of the synthetic group created for comparison with the experimental group; and to automatically generate the total profit and loss statements, and detailed reports of the campaign with unit effects for the final synthetic control group and experimental group and present them to the user as output on the interface (3).

[0026] The electronic device (2) included in the inventive system (1) is configured to exchange data and run at least one application thereon by using any remote communication protocol. The electronic device (2) is a device such as tablet, a laptop, and / or a desktop computer. The electronic device (2) is configured to run the interface (3) thereon. The electronic device (2) is configured to establish a connection with the server (5) by using any remote communication protocol included in the state of the art.

[0027] The interface (3) included in the inventive system (1) is configured to be run on the electronic device (2). The interface (3) is configured to allow the user to enter the campaign name, the audience who joined the campaign to be assessed, the period to be analysed, the previous and next periods, the sample size for simulation, the characteristics to be simulated, the model to be used and threshold value for the synthetic control group, and the model (Catboost, LightGBM). The interface (3) is configured to view the total profit and loss statements, and detailed reports of the campaign with unit effects for the final synthetic control group and experimental group.

[0028] The database (4) included in the inventive system (1) is configured to connect with the server (5) by using any communication protocol in the state of the art. The database (4) is configured to keep records of the campaign name, the audience who joined the campaign to be assessed, the period to be analysed, the previous and next periods, the sample size for simulation, the characteristics to be simulated, the model to be used and threshold value for the synthetic control group, and model details, such as Catboost, LightGBM therein. The database (4) is configured to keep records of the data of the identified customer group, and the data of the subscribers in the campaign audience therein.

[0029] The server (5) included in the inventive system (1) is configured to establish a connection with the electronic device (2) by using any communication protocol included in the state of the art, and to access the interface (3) thereon. The server (5) is configured to connect with the database (4) by using any communication protocol included in the state of the art, and to access the data entered on the interface (3) and save them on the database. The server (5) is configured to access the campaign name, the audience who joined the campaign to be assessed, the period to be analysed, the previous and next periods, the sample size for simulation, the characteristics to be simulated, the model to be used and threshold value for the synthetic control group, and the Catboost, LightGBM, etc. that are entered on the interface (3), and save them on the database. The server (5) is configured to run the relevant queries on the database (4) for the experiment and the sample to be taken from the database (4) based on the inputs entered on the interface (3). The server (5) is configured to run queries in the database (4) for details of the simulated control and experimental groups, such as their demographic characteristics, past habits of use, digital footprints, and mobility. The server (5) is configured to select similar customers among the millions of subscribers who have not joined the campaign by using a stratified sampling method according to critical variables, such as revenue per subscriber, usage, additional packages, overage, digital habits in the previous period in the sample rate entered through the interface (3) and save them on the database (4). The server (5) is configured to eliminate outliers first according to the characteristics to be simulated, fill in missing data and create layers / strata; to select subscribers with values closest to the averages of these strata and save them on the database (4). The server (5) is configured to collect the data of the identified customer group and the data of the subscribers in the campaign audience; to improve data quality by following the pre-processing steps of filling in missing or incorrect data in the dataset, eliminating extreme values, and organizing inconsistent data. The server (5) is configured to convert the data received through the database (4) into a suitable format; label the target audience expected to learn in the model, i.e. the campaign audience, as “1”; to label randomly selected and data-enriched customers from a similar customer base as “0”; to train a supervised machine learning model with the generated dataset; to identify subscribers from the unlabelled synthetic control group for whom the probability of 1 is greater than a certain threshold value and to create a synthetic control group. The server (5) is configured to run LGBM, XGBoost, CatBoost, Random Forrest algorithms for predicting the most similar subscribers. The server (5) is configured to improve on this synthetic control group with Cosine Similarity distance-based simulation methods if there is not enough similarity in the previous period according to the desired characteristics, such as revenue per subscriber, usage. The server (5) is configured to calculate the revenue change and churn trend of the campaign audience after joining the campaign in comparison with the synthetic control group after creating a synthetic control group. The server (5) is configured to compare the experimental group and the synthetic control group for the analysis period and previous periods according to many characteristics, such as demographics, customer behaviour, digitality, etc. for making sure that the synthetic control group is similar. The server (5) is configured to automatically report the previous and next period subscriber-based effects and detail characteristics of the synthetic group created for comparison with the experimental group. The server (5) is configured to automatically generate the total profit and loss statements, and detailed reports of the campaign with unit effects for the final synthetic control group and experimental group and present them to the user as output on the interface (3).

[0030] Industrial Application of the Invention

[0031] In the inventive system (1), he server (5) accesses the campaign name, the audience who joined the campaign to be assessed, the period to be analysed, the previous and next periods, the sample size for simulation, the characteristics to be simulated, the model to be used and threshold value for the synthetic control group, and the Catboost, LightGBM, etc. that are entered on the interface (3), and saves them on the database. The server (5) runs the relevant queries on the database (4) for the experiment and the sample to be taken from the database (4) based on the inputs entered on the interface (3). The server (5) runs queries database (4) for details of the simulated control and experimental groups, such as their demographic characteristics, past habits of use, digital footprints, and mobility. The server (5) selects similar customers among the millions of subscribers who have not joined the campaign by using a stratified sampling method according to critical variables, such as revenue per subscriber, usage, additional packages, overage, digital habits in the previous period in the sample rate entered through the interface (3) and saves them on the database (4). The server (5) eliminates outliers first according to the characteristics to be simulated, fill in missing data and create layers / strata; selects subscribers with values closest to the averages of these strata and saves them on the database (4). The server (5) collects the data of the identified customer group and the data of the subscribers in the campaign audience; to improve data quality by following the pre-processing steps of filling in missing or incorrect data in the dataset, eliminating extreme values, and organizing inconsistent data. The server (5) converts the data received through the database (4) into a suitable format; labels the target audience expected to leam in the model, i.e. the campaign audience, as “1”; labels randomly selected and data-enriched customers from a similar customer base as “0”; trains a supervised machine learning model with the generated dataset; identifies subscribers from the unlabelled synthetic control group for whom the probability of 1 is greater than a certain threshold value and to create a synthetic control group. The server (5) runs LGBM, XGBoost, CatBoost, Random Forrest algorithms for predicting the most similar subscribers. The server (5) improves on this synthetic control group with Cosine Similarity distance-based simulation methods if there is not enough similarity in the previous period according to the desired characteristics, such as revenue per subscriber, usage. The server (5) calculates the revenue change and churn trend of the campaign audience after joining the campaign with the synthetic control group after creating a synthetic control group. The server (5) compares the experimental group and the synthetic control group for the analysis period and previous periods according to many characteristics, such as demographics, customer behaviour, digitality, etc. for making sure that the synthetic control group is similar. The server (5) automatically reports the previous and next period subscriber-based effects and detail characteristics of the synthetic group created for comparison with the experimental group. The server (5) automatically generates the total profit and loss statements, and detailed reports of the campaign with unit effects for the final synthetic control group and experimental group and presents them to the user as output on the interface (3). Thus, it makes campaign management more efficient and shapes future campaigns better.

[0032] The electronic device (2), interface (3), database (4), and server (5) included in the inventive system (1) inform the user in accordance with the principles of data privacy and present consent and operate under the Law on the Protection of Personal Data (LPPD).

[0033] Within these basic concepts; it is possible to develop various embodiments of the inventive “A Post-Campaign Analysis and Automation System (1)”; the invention cannot be limited to examples disclosed herein and it is essentially according to claims.

Claims

CLAIMS1. A system (1) which is developed with the aim of evaluating campaign outcomes quickly and precisely; comprising at least one electronic device (2) which is configured to exchange data and run at least one application thereon by using any remote communication protocol; at least one interface (3) which is configured to be run on the electronic device (2); to enter the campaign name, the audience who joined the campaign to be assessed, the period to be analysed, the previous and next periods, the sample size for simulation, the characteristics to be simulated, the model to be used and the threshold value for the synthetic control group, and the model; and to view reports; at least one database (4) which is configured to keep records of the campaign name, the audience who joined the campaign to be assessed, the period to be analysed, the previous and next periods, the sample size for simulation, the characteristics to be simulated, the model to be used and threshold value for the synthetic control group, model details, such as Catboost, LightGBM and the data of the identified customer group, and the data of the subscribers in the campaign audience therein; and characterized by at least one server (5) which is configured to establish a connection with the electronic device (2) using by any communication protocol and access the interface (3) thereon; to connect with the database (4) by using any communication protocol; to access and save data thereon; to access the data entered on the interface (3) and save them on the database; to run the relevant queries on the database (4) for the experiment and the sample to be taken from the database (4) based on the inputs entered on the interface (3); to run queries in the database (4) for details of the simulated control and experimental groups, suchas their demographic characteristics, past habits of use, digital footprints, and mobility; to select similar customers among the millions of subscribers who have not joined the campaign by using a stratified sampling method according to critical variables, such as revenue per subscriber, usage, additional packages, overage, digital habits in the previous period in the sample rate entered through the interface (3) and record them on the database (4); to eliminate outliers first according to the characteristics to be simulated, fill in missing data and create layers / strata; to select subscribers with values closest to the averages of these strata and save them on the database (4); to collect the data of the identified customer group and the data of the subscribers in the campaign audience; to improve data quality by following the preprocessing steps of filling in missing or incorrect data in the dataset, eliminating extreme values, and organizing inconsistent data; to convert the data received through the database (4) into a suitable format; label the target audience expected to learn in the model, i.e. the campaign audience, as “1”; to label randomly selected and data- enriched customers from a similar customer base as “0”; to train a supervised machine learning model with the generated dataset; to identify subscribers from the unlabelled synthetic control group for whom the probability of 1 is greater than a certain threshold value and to create a synthetic control group; to run LGBM, XGBoost, CatBoost, Random Forrest algorithms; to improve on this synthetic control group with Cosine Similarity distance-based simulation methods if there is not enough similarity in the previous period according to the desired characteristics such as revenue per subscriber, usage; to calculate the revenue change and churn trend of the campaign audience after joining the campaign in comparison with the synthetic control group after creating a synthetic control group; to compare the experimental group and the synthetic control group for the analysis period and previous periods according to many characteristics such as demographics,customer behaviour, digitality, etc. for making sure that the synthetic control group is similar; to automatically report the previous and next period subscriber-based effects and detail characteristics of the synthetic group created for comparison with the experimental group; and to automatically generate the total profit and loss statements, and detailed reports of the campaign with unit effects for the final synthetic control group and experimental group and present them to the user as output on the interface (3).

2. A system (1) according to Claim 1; characterized by the electronic device (2) which is configured to exchange data and run at least one application thereon by using any remote communication protocol; and which is a device such as tablet, a laptop, and / or a desktop computer.

3. A system (1) according to Claim 1 or 2; characterized by the electronic device (2) which is configured to run the interface (3) thereon.

4. A system (1) according to Claim 3; characterized by the electronic device (2) which is configured to establish a connection with the server (5) by using any remote communication protocol.

5. A system (1) according to any one of the preceding claims; characterized by the interface (3) which is configured to be run on the electronic device (2).

6. A system (1) according to any one of the preceding claims; characterized by the interface (3) which is configured to allow the user to enter the campaign name, the audience who joined the campaign to be assessed, the period to be analysed, the previous and next periods, the sample size for simulation, the characteristics to be simulated, the model to be used andthreshold value for the synthetic control group, and the model (Catboost, LightGBM).

7. A system (1) according to any one of the preceding claims; characterized by the interface (3) which is configured to view the total profit and loss statements, and detailed reports of the campaign with unit effects for the final synthetic control group and experimental group.

8. A system (1) according to any one of the preceding claims; characterized by the database (4) which is configured to connect with the server (5) by using any communication protocol.

9. A system (1) according to any one of the preceding claims; characterized by the database (4) which is configured to keep records of the campaign name, the audience who joined the campaign to be assessed, the period to be analysed, the previous and next periods, the sample size for simulation, the characteristics to be simulated, the model to be used and threshold value for the synthetic control group, and model details, such as Catboost, LightGBM therein.

10. A system (1) according to any one of the preceding claims; characterized by the database (4) which is configured to keep records of the data of the identified customer group, and the data of the subscribers in the campaign audience therein.

11. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to establish a connection with the electronic device (2) by using any communication protocol, and to access the interface (3) thereon.

12. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to connect with the database (4) by using any communication protocol in the state of the art, and to access the data entered on the interface (3) and save them on the database.

13. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to access the campaign name, the audience who joined the campaign to be assessed, the period to be analysed, the previous and next periods, the sample size for simulation, the characteristics to be simulated, the model to be used and threshold value for the synthetic control group, and the Catboost, LightGBM, etc. that are entered on the interface (3), and save them on the database.

14. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to run the relevant queries on the database (4) for the experiment and the sample to be taken from the database (4) based on the inputs entered on the interface (3).

15. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to run queries in the database (4) for details of the simulated control and experimental groups, such as their demographic characteristics, past habits of use, digital footprints, and mobility.

16. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to select similar customers among the millions of subscribers who have not joined the campaign by using a stratified sampling method according to critical variables, such as revenue per subscriber, usage, additional packages, overage, digital habits in the previous period in the sample rate entered through the interface (3) and save them on the database (4).

17. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to eliminate outliers first according to the characteristics to be simulated, fill in missing data and create layers / strata; to select subscribers with values closest to the averages of these strata and save them on the database (4).

18. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to collect the data of the identified customer group and the data of the subscribers in the campaign audience; to improve data quality by following the pre-processing steps of filling in missing or incorrect data in the dataset, eliminating extreme values, and organizing inconsistent data.

19. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to convert the data received through the database (4) into a suitable format; label the target audience expected to learn in the model, i.e. the campaign audience, as “1”; to label randomly selected and data-enriched customers from a similar customer base as “0”; to train a supervised machine learning model with the generated dataset; to identify subscribers from the unlabelled synthetic control group for whom the probability of 1 is greater than a certain threshold value and to create a synthetic control group.

20. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to run LGBM, XGBoost, CatBoost, Random Forrest algorithms for predicting the most similar subscribers.

21. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to improve on this synthetic control group with Cosine Similarity distance-based simulation methods if there isnot enough similarity in the previous period according to the desired characteristics, such as revenue per subscriber, usage.

22. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to calculate the revenue change and churn trend of the campaign audience after joining the campaign in comparison with the synthetic control group after creating a synthetic control group.

23. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to compare the experimental group and the synthetic control group for the analysis period and previous periods according to many characteristics, such as demographics, customer behaviour, digitality, etc. for making sure that the synthetic control group is similar.

24. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to automatically report the previous and next period subscriber-based effects and detail characteristics of the synthetic group created for comparison with the experimental group.

25. A system (1) according to any one of the preceding claims; characterized by the server (5) which is configured to automatically generate the total profit and loss statements, and detailed reports of the campaign with unit effects for the final synthetic control group and experimental group and present them to the user as output on the interface (3).