Performance anomaly detection and smart alert

Abstract

The invention relates to performance anomaly detection and smart alerts. Techniques disclosed herein relate to computer-implemented methods and computer systems that utilize a performance anomaly detection model to identify performance metrics (e.g., cost-per-click data) above an anomaly threshold. For example, a communication session may be established with one or more servers hosted by one or more third parties for receiving performance metrics for a first entity. Performance metrics received for a first entity may be used for anomaly detection by a performance anomaly detection model, the performance anomaly detection model may be trained using historical performance metrics (e.g., a historical performance metric of the first entity, a historical performance metric of the first entity during a particular time period, a historical performance metric of the first entity for a particular geographic location). Based on the one or more anomaly detections, one or more notifications or particular displays may be provided to the user device.

Description

Technical Field

[0001] The present disclosure relates to performance metric anomaly detection using a performance anomaly detection model, and in particular to a computer method and a computer system for performance metric anomaly detection using a performance anomaly detection model. Background Art

[0002] Websites, applications, and other service providing platforms can provide recommendations associated with digital marketing to users of multiple e-commerce platforms. For example, a website can provide item recommendations or service recommendations for users to purchase. These recommendations are sometimes provided by different sellers, and these recommendations can also include, for example, images or videos. With the increasing use of recommendations through various platforms, improvements in computing operations associated with these recommendations can provide more efficient and accurate analytical data processing. In addition, it is expected that improvements will be made to user functions and navigation associated with these recommendations. Summary of the invention

[0003] At a high level, aspects described herein relate to performance metric anomaly detection using a performance anomaly detection model. The performance anomaly detection model can detect anomalies above one or more anomaly thresholds based on performance metric historical trends and / or seasonal considerations of the performance metric (e.g., trends of historical performance metrics during a specific time period, trends of historical performance metrics at specific times during specific dates, trends of historical performance metrics for a specific geographic region). For example, the methods, systems, and media described herein can detect problems that trigger anomalies, and provide early detection of these problems. As another example, the techniques described herein can provide alerts, other types of notifications, and automatic remediation for one or more of the detected problems.

[0004] In some example embodiments, a communication session (e.g., a communication session between user devices or servers) may be established with a first server hosted by a third party, a second server hosted by the third party, a third server hosted by another third party, or one or more combinations thereof. Based on the establishment of one or more communication sessions, performance data (e.g., cost-per-click data) of the first entity may be automatically received from each server. A performance anomaly detection model may be applied to the performance data (e.g., a time series including performance data) for anomaly detection based on one or more anomaly thresholds. The performance anomaly detection model may be trained using historical cost-per-click data (e.g., historical cost-per-click data of the first entity for a specific geographic area corresponding to a specific time period) and other historical performance data. As a non-limiting example, the performance anomaly detection model may be trained using historical cost-per-click data of the first entity for a specific city within a specific country during a specific time period on a specific day of the week.

[0005] Upon anomaly detection (based on detecting an anomaly using a performance anomaly detection model), one or more alerts, other types of notifications, automatic remediation, or one or more combinations thereof may be initiated. For example, based on determining that the cost per click data is above an anomaly threshold, a notification may be provided to a user device. As another example, anomaly detection may be indicated on a display (e.g., of a user device) within a graph or table displayed on the user device. In some embodiments, certain automatic remediation may include pausing certain actions associated with the corresponding server and the first entity.

[0006] The Summary of the Invention section is intended to introduce in simplified form a selection of concepts that are also described in the Detailed Description section of the present disclosure. The Summary of the Invention section is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter. Additional objects, advantages, and novel features of the present technology will be set forth in part in the description that follows, and will become apparent in part to those skilled in the art upon examination of the present disclosure or through practice of the present technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the present technology are described in detail below with reference to the accompanying drawings, in which:

[0008] Figure 1 including an example operating environment suitable for employing a performance anomaly detection model according to embodiments described herein;

[0009] Figure 2 is another example operating environment suitable for employing a performance anomaly detection model according to embodiments described herein;

[0010] Figure 3 is another example operating environment suitable for employing a performance anomaly detection model according to embodiments described herein;

[0011] Figure 4 shows example graphs and tables generated using a performance anomaly detection model according to embodiments described herein;

[0012] Figure 5 Another example table generated using a performance anomaly detection model according to embodiments described herein is shown, the table including alert functions associated with anomaly detection;

[0013] Figure 6 shows an example graph including one or more anomalies detected using a performance anomaly detection model, a snapshot of historical performance metrics, and a weekly anomaly summary according to embodiments described herein;

[0014] Figure 7shows an example graph including anomalies detected using a performance anomaly detection model, snapshots of historical performance metrics, and a weekly anomaly summary according to embodiments described herein;

[0015] Figure 8 shows example application programming interface (API) requests and responses associated with a performance anomaly detection model according to embodiments described herein;

[0016] Fig. 9 An example of an anomaly detection service API according to the embodiments described herein is shown;

[0017] Fig.10 depicts an example embodiment of a performance anomaly detection model for anomaly detection according to embodiments described herein;

[0018] Figure 11 to Figure 12 shows example performance model metrics for a performance anomaly detection model according to embodiments described herein;

[0019] Fig.13 shows example alerts for different countries when a performance anomaly detection model detects an anomaly according to embodiments described herein;

[0020] Fig.14 shows an example flow chart corresponding to anomaly detection according to embodiments described herein;

[0021] Fig.15 shows another example flow chart corresponding to anomaly detection according to embodiments described herein; and

[0022] Fig.16 is an example user device suitable for implementing the described techniques in accordance with the embodiments described herein. DETAILED DESCRIPTION

[0023] Websites, applications, and other service providing platforms have been widely used to provide recommendations (e.g., recommendations associated with digital marketing) to users of these websites, applications, or other service providing platforms. For example, a third-party server may provide recommendations associated with digital marketing for a first entity (e.g., an e-commerce provider) that is a separate entity from the third party, such that the recommendations provided to the users of the third-party server may accept offers for items (e.g., goods, software products) or services provided by the first entity. To further illustrate, a particular application may provide recommendations for items or services provided by the first entity for users of the application to purchase when using an application provided by an entity separate from the first entity.

[0024] Recommendations associated with digital marketing provided by third parties are sometimes provided by different sellers (e.g., multiple entities separate from the third-party application). The increasing use of digital recommendations through various platforms allows businesses and other entities to efficiently deliver content to users of third-party platforms. For example, an entity that provides an item or service via a digital recommendation through a third-party website, application, or other service offering platform will pay the third party based on user clicks on the recommendation. The cost per click can be determined by dividing the cost of advertising via the third party by the number of clicks generated by the digital advertisement for the item or service provided by the first entity.

[0025] Sometimes, when recommendations are provided on a third-party website, application, or other service providing platform, specific data points associated with these recommendations may spike or otherwise change dramatically for a variety of reasons. For example, sometimes these spikes or dramatic changes are simply software errors of previous methods or systems, improper model detection or insufficient model training of previous methods or systems, or excessive or inappropriate data inputs to these previous models. For example, when these previous systems detect spikes or changes, these systems cannot determine or detect that these spikes or changes are caused by strategic changes made by the business operations team, which the previous model or software does not know or cannot detect or otherwise consider. As an example, these previous systems and methods have been unable to detect or recognize that a bidding strategy has been implemented (e.g., managed by a machine learning engine or operations team). Examples of these bidding strategies can include increasing specific prices for individual items or services, changing target return on advertising spend (ROAS) that increases or decreases specific costs, and changing conversion values ​​(e.g., changing the incremental value of users in New York City from 20% to 22%). Previous software or models generally do not transmit or detect these expected bidding strategy changes.

[0026] In these regards, prior systems have fallen short in actual and true anomaly detection. In yet another example, when a data outage or service disruption at a particular server managed by a third party causes a price increase for a single item or service, these prior systems (e.g., software or models) have been unable to detect or realize that the data outage or service disruption was the actual cause of the change or spike. Furthermore, in situations where a malicious actor (or machine) has committed click fraud to influence a particular entity's referral costs (e.g., advertising budget), these prior systems have been unable to determine that this was the true cause of the anomaly (e.g., a spike or drastic change in particular monitored data).

[0027] In these regards, existing methods and systems for monitoring digital referral costs have various shortcomings and deficiencies in terms of effective monitoring and remediation based on any identified triggers related to recommendations (e.g., referral costs). For example, existing methods and systems fail to identify a specific cause (e.g., a change in bidding strategy or a data outage) that has triggered an increase in digital referral costs. In instances where a data outage has caused an increase in digital referrals, computer network communications between computing devices associated with the first entity may be adversely affected, computer network communications between computing devices associated with the first entity and computing devices associated with a third party may be adversely affected, and computer network communications between computing devices associated with the first entity and computing devices associated with users of third-party digital services may be adversely affected.

[0028] In addition, some existing system architectures that include machine learning models have not trained the machine learning models on specific performance metrics (e.g., historical cost-per-click data for the first entity associated with the digitally recommended item or service during a specific time period). Instead, these existing systems and methods have used general rule-based approaches that generate too many false positives or fail to detect serious business-impacting issues early enough to take appropriate remedial actions. For example, existing systems set bid adjustments for digital recommendations based solely on target sales amounts and target spend amounts. Other existing system architectures only set bid limits to avoid overspending. As a result, these existing systems may incorrectly characterize or fail to identify cost increases, identify inaccurate reasons for the claimed increases, and provide too many false positives.

[0029] As another example, each time an additional bid adjustment or bid limit is applied, the associated content or payload (e.g., the payload associated with the metadata from each rule-based limit that is applied) is increased due to the additional processing required to analyze the associated database (including the limit or adjustment to the bid). In this way, the additional processing and metadata sent over the network can cause throughput and latency issues. In some cases, when the stored bid factors are incorrectly or inaccurately identified or labeled, the additional processing can also increase storage device I / O (e.g., excessive physical read / write head movement on a non-volatile disk) due to unnecessary information being processed or stored. In doing so, the computing system often must contact the storage device to perform the read or write operation, which is time consuming, error-prone, and can eventually wear out components, such as the read / write head.

[0030] Therefore, methods and systems for accurately monitoring, detecting, and remediating abnormal referral costs (e.g., detecting a specific cause (e.g., a change in bidding strategy or a data outage) that has triggered a specific increase in digital referral costs) are desired. It is also desired to enhance computer network communications between computing devices associated with a first entity, enhance computer network communications between computing devices associated with the first entity and computing devices associated with a third party, and enhance computer network communications between computing devices associated with the first entity and computing devices associated with users using third-party digital services. Therefore, the technology described herein achieves these goals and provides various improvements to the problems specific to the above-mentioned existing systems.

[0031] For example, in an embodiment of the present disclosure, a computer-implemented method begins by establishing a communication session with a server hosted by a third party. In some embodiments, one or more additional communication sessions may be established with one or more other servers hosted by a third party. Additionally or alternatively, one or more additional communication sessions may be established with one or more other servers hosted by different third parties. A server hosted by a third party may include one or more computing devices / user devices, servers, databases, or one or more combinations thereof configured to generate digital content (e.g., a website or application) that can be accessed via a network. In some embodiments, an application server associated with a first entity and a server hosted by a third party establish a communication session so that the application server can provide items or services on a third-party application or a third-party website.

[0032] Based on establishing the communication session, cost per click data for the first entity may be received from a server hosted by a third party. The cost per click data for the first entity relates to a price for the first entity each time a user interacts with content provided by the first entity through a third-party digital service (e.g., a webpage or application) (e.g., a user selects or clicks on an item or service provided by the first entity through a third-party digital service). Additionally or alternatively, other performance metrics corresponding to the first entity may be received from the third-party server, such as cost per action (the price of the first entity each time a user performs a specific action on content provided by the first entity through a third-party digital service (e.g., purchases an item or service provided by the first entity, downloads software provided by the first entity, fills out a form provided by the first entity)), cost per thousand (cost per thousand impressions, or cost to send one thousand email messages), click-through rate (ratio of clicks to impressions), browse rate (ratio of the number of times a user does not skip an ad to the number of times an ad is displayed (i.e., presented)), another type of performance metric, or one or more combinations thereof.

[0033] The performance metrics received from the third-party server can be used to identify one or more anomalies that are above an anomaly threshold of one or more performance metrics. As an example, a performance anomaly detection model can be applied to the cost per click data to generate an output that is compared with the anomaly threshold. The performance anomaly detection model can be trained using historical cost per click data. For example, the performance anomaly detection model can be trained using historical cost per click data of a first entity, which is grouped based on a specific date and a specific time period during a specific date (e.g., a first grouping of historical cost per click data of the first day (e.g., Monday before a specific holiday) during a first time period (e.g., from 12:00:00pm EST (Eastern Standard Time) to 1:00:00pm EST), a second grouping of historical cost per click data of the first day during a second time period (e.g., from 5:00:00pm EST to 7:00:00pm EST), and a third grouping of historical cost per click data of the second day during the first time period). In some embodiments, the performance anomaly detection model can be trained using historical cost per click data of the first entity for a specific geographic area (e.g., a specific city in a specific country, a specific state in a specific country, a specific town in a specific country).

[0034] In an embodiment, based on applying the performance anomaly detection model to a performance metric (e.g., cost per click data), an output from the performance anomaly detection model may be compared to an anomaly threshold. The anomaly threshold may be based on the type of performance metric, a specific performance metric about the first entity, a spending cap associated with the first entity, the location of a server hosted by a third party, or one or more combinations thereof, and in some embodiments, the anomaly threshold may vary for different geographic locations (e.g., for different cities within the same state, for different countries). Based on determining that the output is above the anomaly threshold, a notification may be provided to a user device. In some embodiments, the detected anomaly may be indicated and displayed on the user device in graphical form or in the form of a table.

[0035] Having provided some example scenarios, techniques suitable for performing these examples are described in more detail with reference to the accompanying drawings.It will be appreciated that additional systems and methods for providing improved search results and navigation may be derived from the following description of the present technology.

[0036] Now turn to Figure 1 , Figure 1 An example operating environment 100 is shown in which embodiments of the present disclosure may be employed. Specifically, Figure 1 A high-level architecture of an operating environment 100 having components according to an embodiment of the present disclosure is shown. Figure 1The components and architecture of are intended to be examples, as noted at the end of the Detailed Description section.

[0037] The example operating environment 100 includes: a user device 102; a dashboard 104 communicatively coupled to a plurality of sites 106A-106C (each site including a cost per click (CPC) component 108A-108C, a cost component 110A-110C, and a click component 112A-112C); a performance anomaly detection model 114 having one or more models 116, a mobile metric detector burst model 118, and a plurality of business analytics 120, the business analytics 120 including a spend cap 122 and a cost per click cap 122. 4; a remediation engine 126, including a suspend expenditure function 128A for site 106A, a suspend expenditure function 128B for site 106B, and a suspend expenditure function 128C for site 106C; an alert engine 130 for issuing alerts to user devices 102 via email 132, a messaging application 134, or a software as a service (SaaS) platform 136; a third party server 140; and a scheduler 142, including a distributed file system 144 and an enterprise resource planning (ERP) system 146.

[0038] The user device 102 may be a device capable of using a wireless communication network, and may also be referred to as a "computing device," "mobile device," "user device," "wireless communication device," or "UE." In some embodiments, the user device may take various forms, such as a PC, a laptop, a tablet, a mobile phone, a PDA, a server, or any other device capable of communicating with other devices using wireless communication (e.g., by sending or receiving signals). In a broad sense, the user device 102 may include a computer-readable medium storing computer-executable instructions executed by at least one computer processor. An example of a user device includes the user device described herein. Fig.16 A computing device 1600 is described.

[0039] As shown in the example environment 100, the user device 102 may be capable of displaying a dashboard 104 and may be configured to communicate with the performance anomaly detection model 114 and other components of the example operating environment 100 (e.g., the alert engine 130). Other embodiments of the example environment 100 may include additional user devices that are capable of displaying a dashboard (e.g., the dashboard 104) and are configured to communicate (e.g., by sending or receiving signals) with the performance anomaly detection model 114, the remediation engine 126, the alert engine 130, the third-party server 140, and the scheduler 142. The user device 102 may be operated by a user (e.g., one or more of a human, a machine, a robot, other user device operators, or one or more combinations thereof). The user of the user device 102 may be associated with a first entity (e.g., an entity different from the entity associated with the third-party server 140).

[0040] The dashboard 104 may include performance metrics for each of the plurality of sites 106A-106C (e.g., corresponding to CPC components 108A-108C, cost components 110A-110C, and click components 112A-112C). In an embodiment, one or more of the sites 106A-106C displayed on the dashboard 104 (e.g., displayed to a user of a user device 102 associated with a first entity) may provide an e-commerce offer to a third-party server 140 for advertising on a third-party's webpage, website, or application. For example, the third-party server 140 hosted by a third party may include one or more computing devices / user devices, servers, databases, or one or more combinations thereof configured to generate digital content (e.g., a website or application), which may be accessed, for example, over a network (e.g., via an application server associated with the first entity) based on an established communication session, so that the application server associated with the first entity may provide items or services on a third-party application or a third-party website via the third-party server 140. In some embodiments, site 106B may correspond to a core server (e.g., a central monitoring system) associated with an e-commerce offering provided by the first entity, and site 106C may correspond to a vertical site for the e-commerce offering and may include hardware components for additional processing systems (e.g., for extending capabilities associated with the first entity to other locations).

[0041] In a non-limiting example, the first site 106A may correspond to an e-commerce offer provided by a first entity via a third-party web page (e.g., including a search engine) associated with a first city in a first state in the United States, the second site 106B may correspond to an e-commerce offer provided by the first entity via a third-party web page associated with a second city in the first state in the United States, and the third site 106C may correspond to a third-party web page associated with a third city in the European Union. In another example embodiment, first site 106A may correspond to a third-party application (e.g., a social media application) associated with a first city in a first state in the United States (e.g., wherein CPC component 108A, cost component 110A, and click component 112A include cost per click data, cost data, and click data for multiple items or services advertised on the third-party application for users located within the first city), second site 106B may correspond to a third-party application associated with the first state in the United States (e.g., wherein CPC component 108B, cost component 110B, and click component 112B include cost per click data, cost data, and click data for multiple items or services advertised on the third-party application for users located throughout the first state), and third site 106C may correspond to a third-party application associated with a second city within the United States. In yet another example embodiment, the first site 106A may include cost per click data, cost data, and click data for multiple items or services advertised on a third-party digital service provider for a specific geographic area, the second site 106B may include cost per click data, cost data, and click data for multiple items or services advertised on third-party digital service providers for multiple specific geographic areas (including the specific geographic area), and the third site 106C may include cost per click data, cost data, and click data for multiple items or services advertised on different third-party digital service providers for a specific geographic area.

[0042] In some embodiments, cost per click data, cost data, and click data for one or more items or services advertised on one or more third-party digital service providers for one or more specific geographic regions can be grouped based on time periods. For example, in some embodiments, cost per click data, cost data, and click data can be tracked hourly over multiple days to generate historical cost per click data. In some embodiments, cost per click data can be tracked additionally or alternatively and grouped based on specific seasons (e.g., holiday season, winter). In some embodiments, the cost per click data can include timestamps for constructing a time series of the cost per click data (e.g., based on time of day, season, specific geographic location).

[0043] In some embodiments, dashboard 104 may include other performance metrics for each of sites 106A to 106C, such as cost per action (the price paid by the first entity each time a user performs a particular action on content provided by the first entity through a third-party digital service (e.g., purchasing an item or service provided by the first entity, downloading software provided by the first entity, filling out a form provided by the first entity)), cost per thousand (the cost per thousand impressions, or the cost of sending one thousand email messages), click-through rate (the ratio of clicks to impressions), view-through rate (the ratio of the number of times an ad is not skipped by a user to the number of impressions (i.e., the ad is presented)), another type of performance metric, or one or more combinations thereof.

[0044] In embodiments, dashboard 104 may be a graphical user interface that provides analytics and management tools for tracking, monitoring, analyzing, and displaying various information related to digital advertisements for items or services provided by the first entity to users of third-party digital services (e.g., third-party search engines, third-party applications), including performance metrics for those digital advertisements provided through the third-party digital services. In embodiments, dashboard 104 may provide one or more visual illustrations of data sets collected from multiple third-party sources (e.g., Figure 4 ) and analysis of the data (e.g., performance metrics such as cost per click) for anomaly detection (e.g., anomalies in the cost per click time series), and for providing determinations related to one or more causes of anomaly detection (e.g., data outages at a particular site, click fraud at a particular site), and for providing remediation actions (e.g., remediation actions generated by remediation engine 126).

[0045] The performance anomaly detection model 114 can be configured to communicate with the user device 102 (and the dashboard 104), the remediation engine 126, and the alert engine 130 (e.g., over a network, such as the Internet, a local area network, another wide area network, another type of wireless network, or one or more combinations thereof). For example, the performance anomaly detection model 114 can receive various rules and inputs associated with the sites 106A-106C for anomaly detection. In some embodiments, the performance anomaly detection model 114 receives cost per click data for each of the sites 106A-106C and historical cost per click data for each of the sites 106A-106C. In some embodiments, the performance anomaly detection model 114 receives other types of performance metrics for each of the sites 106A-106C and historical performance metrics for each type of performance metric for each of the sites 106A-106C. In some embodiments, the performance anomaly detection model 114 may receive performance metrics (e.g., for digital advertisements provided by a first entity) from a third-party server 140 or based on user device interactions (e.g., clicks) of users using a digital service provided by the third-party server 140, where the digital bids are provided by the first entity.

[0046] In an example embodiment, the performance anomaly detection model 114 may be applied to the first grouping of historical cost per click data such that the output generated by the performance anomaly detection model 114 is generated based on the cost per click data input and the first grouping of historical cost per click data. As a non-limiting illustration, the performance anomaly detection model 114 may receive (e.g., in real time or near real time) a time series of cost per click data for the first site 106A and may generate an anomaly detection output based on the time series cost per click data and based on the grouping of historical cost per click data for the first site 106A during the same time period and on the same day in the past few weeks or the same day in the same month in the previous year (e.g., a historical per click time series generated from 12:01:00 pm EST to 12:31:00 pm EST every Monday in the past few weeks or every Monday in October of the previous year). In this manner, the output being generated by the performance anomaly detection model 114 may be based on a time period corresponding to the cost per click data received for the first entity (e.g., corresponding to one or more of the sites 106A to 106C).

[0047] In some embodiments, one or more models 116 of the performance anomaly detection model 114 may include a machine learning model, a deep learning model, a neural network, a hidden Markov model, a statistical language model, a natural language processing model, a task flow model, a decision tree model, a regression model, a random forest model, another type of model, or one or more combinations thereof for anomaly determination and cause identification. For example, one or more models 116 and the mobile metric detector burst model 118 may be used to detect anomalies associated with a data outage at a particular site (e.g., site 106A) or an updated business analysis 120 (e.g., a change in the spending cap 122 at site 106B). In an embodiment, the mobile metric detector burst model 118 may be used to generate an output of a time series of cost per click, and one or more models 116 may be used to compare user-generated or artificial intelligence-generated feedback with true or false alarms (these alarms are generated via the alarm engine 130 in response to the output from the mobile metric detector burst model 118). As another example, one or more models 116 may be used to compare expenditure values ​​associated with first site 106A over a particular time period such that these expenditure values ​​may be used to generate a time series of cost per click data for input at mobile metric detector burst model 118. In some embodiments, one or more models 116 may be used to determine anomaly thresholds.

[0048] In an embodiment, the mobile metric detector burst model 118 may include a decomposer for decomposing each time series of click cost data into trend, seasonality, and residual components. For example, in an embodiment, the mobile metric detector burst model 118 may include a mobile metric decomposer that uses a median value associated with the historical click cost data to extract trends and seasonality in the time series click cost data without using labels in the historical click cost data. As a non-limiting example, the mobile metric detector burst model 118 may utilize a mobile metric decomposer and a median value of historical click cost data associated with a particular city or town for each non-holiday Saturday during a particular time period of multiple previous Saturdays corresponding to the second site 106B to extract trends or seasonality in one or more associated time series of click cost data without using labels in the historical click cost data. As another non-limiting example with respect to analyzing the time series of click cost data for first site 106A, mobile metric detector burst model 118 may utilize a mobile metric decomposer and a median value of historical click cost data corresponding to second site 106B and a median value of historical click cost data corresponding to third site 106C, where the historical click cost data for both second site 106B and third site 106C are associated with a particular time period that is also associated with the time series of click cost data for first site 106A (e.g., considering the median values ​​of the other two sites if the first site does not have sufficient data). In some embodiments, mobile metric detector burst model 118 may be Fig.10 In some embodiments, the mobile metric detector burst model 118 may receive an application programming interface (API) and send a burst model such as Figure 8 API responses are shown (eg, for communications with the user device 102, the dashboard 104, the remediation engine 126, and the alert engine 130).

[0049] The performance anomaly detection model 114 may analyze the cost per click data, other performance data, or one or more combinations thereof based on one or more of the multiple business analyses 120 including the spending cap 122 and the cost per click cap 124. For example, the performance anomaly detection model 114 may determine whether a data point associated with a time series of cost per click data exceeds a predetermined range based on the spending cap 122, the cost per click cap 124, or one or more combinations thereof. In some embodiments, the user determines one or more of the spending cap 122 and the cost per click cap 124 via the user device 102. In other embodiments, one or more models 116 may be used to determine one or more of the spending cap 122 and the cost per click cap 124. For example, one or more models 116 may determine the spending cap 122 or the cost per click cap 124 based on historical spending cap values, historical cost per click cap values, other business analyses, or one or more combinations thereof. In addition, the cost per click cap 124 of a particular time series of cost per click data associated with a particular time period may be determined based on the historical cost per click cap values ​​associated with the particular time period. In yet another example, the spend cap 124 for a particular time series of cost-per-click data associated with a particular location may be determined based on historical spend cap values ​​associated with the particular location.

[0050] The remediation engine 126 may be configured to communicate with the performance anomaly detection model 114 and the third-party server 140 (e.g., to pause digital advertising of the first entity associated with the detected anomaly). For example, the remediation engine 126 may automatically pause spending of one or more digital advertisements of the first site 106A via the pause spending function 128A based on the detected anomaly associated with the cost per click data (or other performance metric data) of the first site 106A, automatically pause spending of one or more digital advertisements of the site 106B via the pause spending function 128B based on the detected anomaly associated with the cost per click data of the site 106B, and automatically pause spending of one or more digital advertisements of the site 106C via the pause spending function 128C based on the detected anomaly associated with the cost per click data of the site 106C. The pause spending functions 128A-128C may suspend spending based on communication with the user device 102, the third-party server 140, or one or more combinations thereof.

[0051] The alert engine 130 can provide an alert to the user device 102 based on detecting an anomaly using the performance anomaly detection model 114. For example, the alert engine 130 can provide an alert using an email 132, a messaging application 134, a SaaS platform 136 (e.g., PagerDuty™), another type of computer-based service, or one or more combinations thereof. For example, the messaging application 134 can send an alert or notification to the user device 102b based on a messaging protocol used by the user device 102 or the messaging application 134. As another example, the SaaS platform 136 can provide an alert or notification to the user device 102 via a cloud infrastructure system. In some embodiments, the logic for triggering an alert or notification is decoupled from the logic for determining which of a plurality of user devices should receive the alert or notification and the content of the notification.

[0052] The scheduler 142 may be configured to communicate with the third-party server 140, and the scheduler 142 may include a distributed file system 144 and an ERP system 146. For example, the distributed file system 144 may include a Hadoop distributed file system (HDFS). For example, the distributed file system 144 may store a large-scale data set including cost-per-click data and other performance metrics associated with each of the sites 106A to 106C, and stream these data sets to the performance anomaly detection model 114 (or another user application, for example, based on certain permissions) at high bandwidth. In addition, the ERP system 146 may be an enterprise software application that employs one or more databases or structured content to collect cost-per-click data, other performance metrics associated with one or more sites 106A to 106C, other enterprise resource planning data associated with the first entity (e.g., employee data, other financial information of the first entity, asset data, etc.), or one or more combinations thereof.

[0053] In addition to communicating with third-party server 140 (e.g., via the communication session), scheduler 142 may also establish a second communication session with a second server hosted by another third party, establish one or more additional communication sessions with third-party server 140 (e.g., one session for cost-per-click data for the first set of digital advertisements, a second session for cost-per-click data for the second set of digital advertisements, a third session for spend cap and cost-per-click cap data, etc.), establish one or more additional communication sessions with third-party servers hosted by other third parties, etc. In an embodiment, scheduler 142 may automatically receive cost-per-click data, other performance metrics associated with one or more sites 106A-106C, other enterprise resource planning data associated with the first entity, or one or more combinations thereof based on the established communication session.

[0054] Now turn to Figure 2 , Figure 2 An example timeline 200 is shown regarding implementations in which the present disclosure may be employed. Figure 2 The components and architecture of are intended as examples. For example, the timeline at about T+22 minutes may include more or fewer reports 210, or other types of reports.

[0055] The example timeline 200 depicts a third-party server 202 receiving an application programming interface (API) 206 request to transmit cost-per-click data associated with a first entity different from the third-party server 202. For example, the third-party server 202, which is hosted by a third party, may include one or more computing devices / user devices, servers, databases, or one or more combinations thereof configured to generate digital content (e.g., a website or application) that can be accessed over a network (e.g., via a computing device associated with the first entity) based on an established communication session so that the first entity can provide items or services on a third-party application, a third-party website, or another digital service provided by the third-party server 202. In some embodiments, based on the API request 204, at about T+3 minutes in one or more data files 206, the API request 204 to the third-party server 202 can initiate the collection of performance metrics and metadata from various platforms provided by the third-party server (e.g., from product listing ads, text ads (e.g., SlickText), display ads provided by a display network (e.g., Google Display Network)).

[0056] In an embodiment, one or more API requests 204 may be triggered by a computing device of a first entity to receive information corresponding to a digital offer provided by the first entity on a third-party digital service hosted by a third-party server 202. The API request 204 may trigger the collection of specific information from one or more API endpoints associated with the third-party server 202. The one or more data files 206 may correspond to a specific time period (e.g., a start time and an end time). For example, the one or more data files 206 collected based on the API 204 may include click data, cost data, spend cap data, cost per click data, other types of performance metrics associated with one or more digital offers of the first entity or one or more groups of digital offers of the first entity (e.g., cost per impression), or one or more combinations thereof.

[0057] In an embodiment, one or more data files 206 may include multiple digital advertising formats. For example, one or more of the multiple digital advertising formats may correspond to product listing ads (PLAs) that display additional information in addition to text-based ads (e.g., digital shopping ads on Google Search, Google Shopping, Google Image Search, dynamic product ads on Facebook and Instagram, etc.), and may also include metadata for grouping specific items or groups of items according to specific digital advertising groups (e.g., grouping based on item features, categories, or specific sellers). As another example, one or more of the data files 206 may correspond to text-based ads or display network ads (e.g., Google display network ads, online auction applications, etc., in which ads participating in online auctions are input to an ad server to determine which of those ads will win the auction), and may also include metadata for grouping specific items or services or groups of items or services according to specific digital advertising groups. In an embodiment, each of the data files 206 may include cost per click data for a digital offer of a first entity. In some embodiments, metadata (e.g., metadata corresponding to cost per click data) may be extracted from each of the data files 206. In some embodiments, one or more data files 206 may correspond to different third parties or different third-party servers.

[0058] In an embodiment, (e.g., at about T+17 minutes) one or more campaign performance reports 208 may be generated based on the data file 206. For example, the campaign performance report 208 may be generated based on metadata of the data file 206 corresponding to cost per click data. In some embodiments, the campaign performance report 208 may be generated based on a first grouping of metadata for a first set of items with a first seller. In some embodiments, the campaign performance report 208 may be generated based on a first grouping of metadata for a first set of items in a first category (e.g., infant formula). In an embodiment, the campaign performance report 208 may include one or more of: a campaign name, an offer type associated with the campaign (e.g., text ads for items, text ads for services, sponsored updates, display network ads, video ads), audience size, daily budget, total budget, bids, duration, number of active ads, number of content items or services, cost per click data and corresponding timestamps, other types of performance metrics, another type of campaign performance report data, or one or more combinations thereof. In a non-limiting example, the campaign performance report 208 may indicate that a particular digital advertisement with impressions or user interactions based on a keyword (e.g., "moon") had 15,000 impressions (i.e., presented to users of the third-party digital service provider 15,000 times), 918 clicks, and 15 conversions.

[0059] In an embodiment, a plurality of reports 210 are generated using the activity performance reports 208 (e.g., at approximately T+22 minutes). In an embodiment, an expense report of the plurality of reports 210 may be generated based on the activity performance reports 208 associated with one or more of the data files 206, a PLA report may be generated based on the activity performance reports 208 associated with one or more of the data files 206, a text report may be generated based on the activity performance reports 208 associated with one or more of the data files 206, and a display network report (e.g., Google Display Network) may be generated based on the activity performance reports 208 associated with one or more of the data files 206. In some embodiments, the plurality of reports 210 may be generated based on an aggregation of one or more of the data files 206 (e.g., from multiple sites (e.g., Figure 1 One or more of the plurality of reports 210 may be generated based on an aggregation of cost-per-click data for sites 106A to 106C of the first entity or an aggregation of cost-per-click data from data file 206. In some embodiments, one or more of the plurality of reports 210 may be generated based on metadata extracted from the plurality of reports 210 (e.g., metadata corresponding to the cost-per-click data). In some embodiments, one or more of the plurality of reports 210 may be generated based on click data, cost data, spend cap data, cost-per-click data, other types of performance metrics associated with one or more digital quotes for the first entity or one or more groups of digital quotes for the first entity, or one or more combinations thereof extracted from campaign performance report 208. For example, one or more of the plurality of reports 210 may be generated based on an aggregation of cost-per-click data for a particular time period, the cost-per-click data associated with one or more digital quotes for the first entity or one or more groups of digital quotes for the first entity.

[0060] One or more of the plurality of reports 210 or specific information extracted therefrom can be used as input to the performance anomaly detection model 220 for anomaly detection (e.g., at approximately T1+1 hour or approximately T2+30 minutes corresponding to the timeline) and for issuing an alert via the alert engine (e.g., at approximately T1+1.75 hours or approximately T2+1.25 minutes corresponding to the timeline). In some embodiments, the performance anomaly detection model 220 can perform the same Figure 1 As another example, in some embodiments, the alert engine of the performance anomaly detection model 220 is similar to the performance anomaly detection model 114. Figure 1In an embodiment, ERP 212 and open source platform (OSP) 216 may process one or more of plurality of reports 210 before providing input to performance anomaly detection model 220 .

[0061] In an embodiment, ERP 212 may include a software application or software application suite that collects, stores, manages, and further processes one or more reports 210 in the plurality of reports 210 and one or more databases for storing operations performed on the plurality of reports 210 or portions thereof. In some embodiments, ERP 212 may (e.g., based on Figure 1 ERP 212 may also merge data sets (e.g., associated with specific cost-per-click data) from each of the plurality of reports 210. In some embodiments, OSP 216 (e.g., a Grafana visualization or a Prometheus visualization) may be additionally applied to one or more of the plurality of reports 210 to further manage or process specific cost-per-click data sets. For example, a first set of cost-per-click data corresponding to a first predetermined time period may be identified, and a second set of cost-per-click data corresponding to a second predetermined time period (a time period later than the first predetermined time period) may be identified to apply specific rules (e.g., Figure 1 The business analysis 120 of the first group is applied to each of the first group and the second group. In this way, the first group can be provided to the performance anomaly detection model 220 as input for anomaly detection (e.g., at approximately T1+1 hour corresponding to the timeline), and the second group can be provided to the performance anomaly detection model 220 as input for anomaly detection (e.g., at approximately T2+30 minutes corresponding to the timeline).

[0062] In an example embodiment, the performance anomaly detection model 220 may detect anomalies based on hourly inputs for one or more performance metrics corresponding to one or more of the plurality of reports 210 using one or more of the plurality of reports 210. In some embodiments, anomaly detection via the performance anomaly detection model 220 may occur for approximately twenty-five minutes. Based on anomaly detection for performance metrics via the performance anomaly detection model 220, the performance anomaly detection model 220 may provide one or more alerts or notifications to a user device associated with the first entity, as well as provide a report for specific cost-per-click data 222.

[0063] In some embodiments, the performance anomaly detection model 220 is trained using the grouping of historical cost-per-click data for anomaly detection of cost-per-click data associated with one or more of the plurality of reports 210. As an example, the performance anomaly detection model 220 may be trained using a first group of historical cost-per-click data for display network ads (e.g., Google display network ads, online auction applications, etc., in which ads participating in an online auction are input to an ad server to determine which of those ads will win the auction) corresponding to one or more historical campaign performance reports, and a second group of historical cost-per-click data for product listing ads (PLAs) that display additional information in addition to text-based ads (e.g., digital shopping ads on Google Search, Google Shopping, Google Image Search, dynamic product ads on Facebook and Instagram, etc.) corresponding to one or more historical campaign performance reports.

[0064] As another example, the performance anomaly detection model 220 may be trained using a first set of historical cost-per-click data (e.g., a historical time series including cost-per-click data) for text-based advertisements corresponding to a third-party digital service and corresponding to one or more historical campaign performance reports for a first season (e.g., a holiday season or a season associated with weather), and a second set of historical cost-per-click data for text-based advertisements corresponding to a different third-party digital service and corresponding to one or more historical campaign performance reports for the first season, and a third set of historical cost-per-click data for text-based advertisements corresponding to the third-party digital service for a second season. Additionally or alternatively, the performance anomaly detection model 220 may be trained using historical cost-per-click data corresponding to a particular geographic region. For example, the performance anomaly detection model 220 may be trained using a first set of historical cost-per-click data corresponding to a first entity and for a particular time period (e.g., every Friday for a particular group of weeks) and corresponding to a third-party digital service provided by a third-party server located in a first town or city, and a second set of historical cost-per-click data corresponding to the first entity and for a particular time period and corresponding to a third-party digital service provided by a third-party server located in a second town or city (different from the first town or city).

[0065] Based on utilizing one or more of the plurality of reports 210, (e.g., using cost per click data from historical campaign performance reports) training a performance anomaly detection model 220, and based on utilizing the performance anomaly detection model 220 (e.g., using a moving average over a particular time period, as described herein with reference to Fig.10As described above, the performance anomaly detection model 220 can (e.g., via the alert engine) be faster (e.g., existing systems and methods 214 that use only general rule-based methods to set bid adjustments take up to three hours to detect based on alerts, which results in a significant increase in spending before detection) and more efficient (e.g., existing systems and methods that use static rules to generate alerts generate too many false positives and the accuracy of detection is reduced by 3.4 times, which can be achieved using the data from Fig.13 Example Table 1200 of the example formulas to determine) generates alerts without false positives.

[0066] Now turn to Figure 3 , Figure 3 An example operating environment 300 is shown in which embodiments of the present disclosure may be employed. The example operating environment 300 includes data sources 302, anomaly detection as a service 304, MMD model 306, infrastructure 308, and alert engine 310. Figure 3 The components and architecture of are intended to be illustrative. For example, data source 302 may include multiple data sources.

[0067] The data source 302 may include a non-unique / non-structured query language (NoSQL) database that supports a SQL-like query language or runs in parallel with one or more SQL databases in a polyglot persistence architecture. For example, a NoSQL database may be an organized collection of data (e.g., cost per click data, time series of cost per click data, time series of other performance metrics) modeled in a form other than table relationships, which is accessible via a network. For example, an API call may send a query to a NoSQL database to extract specific cost per click data or other performance metrics associated with a quote provided by a first entity through a third-party digital service provider. In one embodiment, a user device sends a request for a specific document to a NoSQL database. In another embodiment, the NoSQL database may utilize a query language similar to SQL, or may run in parallel with a relational model database (e.g., including tables with data corresponding to relationships between queries of an API call) of the data source 302 in a polyglot persistence architecture.

[0068] In some embodiments, the data source 302 may correspond to a distributed computing platform for data processing, such as Apache Fink®, Apache Storm, Apache Spark®, Apache Kafka® Streams, Apache Hadoop®, Apache Hive, Splunk®, Amazon Kinesis®, SQLstream®, Elastisearch, Statistical Analysis System (SAS®), Information Bus Company (TIBCO) StreamBase®, International Business Machines Corporation (IBM) InfoSphere® Streams, another type of distributed computing platform, or one or more combinations thereof. In an example embodiment, the data source 302 may include a Hadoop® Distributed File System (HDFS) (e.g., which provides horizontal data partitioning that can reduce latency and increase throughput). In another example, the data source 302 may be configured with a distributed framework (e.g., Hadoop®, etc.) to perform real-time processing or batch-based parallel processing of cost-per-click data, cost-per-click time series data, or other performance metrics that may be provided by one or more components of the example operating environment 300.

[0069] Anomaly detection as a service 304 may store data at and extract data from data sources 302. Anomaly detection as a service 304 may include metric injection 304A, hybrid detection models, and rules 304B (e.g., Figure 1 One or more models 116 or multiple business analyses 120), an alert component 304C (eg, configured to communicate with an alert engine 310, which can communicate with Figure 1 ), feedback 304D, API entry point 304E, metadata storage component 304F (e.g., for storing Figure 2 ), a result storage component 304G (e.g., for storing anomaly detections), and a dashboard component 304H (e.g., for communicating with Figure 1 dashboard 104 communications).

[0070] For example, in an embodiment, the anomaly detection service 304 can utilize the metric injection 304A to consume the collected metrics (e.g., cost per click data, cost per click time series data, or other performance metrics) into a specified format and store the formatted metrics in the data source 302 or the metadata storage component 304F of the anomaly detection service 304. In the illustrative example, the anomaly detection service 304 can utilize the metric injection 304A to format the Figure 2For example, the anomaly detection service 304 can utilize the metric injection 304A to format the cost per click time series or the cost per click time series of the plurality of reports 210. Figure 2 Specific cost-per-click data 222 generated by a performance anomaly detection model of the present invention (e.g., formatted based on one or more campaign names, types of offers associated with the campaign, audience size, daily budget, total budget, bids, duration (e.g., one hour during a particular date), number of active ads at a particular site during a particular time period, number of content items or services associated with active ads, or one or more combinations thereof).

[0071] In an embodiment, the hybrid detection model and rules 304B of the anomaly detection as a service 304 can fine-tune the MMD model 306 for a specific performance metric. For example, the hybrid detection model and rules 304B can be used to fine-tune the MMD model 306 for the cost per click data based on the timestamp corresponding to the cost per click data, based on the location of the third party that collects the cost per click data, based on the daily or hourly budget for the click associated with the cost per click data, based on the type of digital advertisement associated with the cost per click data, based on the specific items or services provided associated with the cost per click data, or one or more combinations thereof. In some embodiments, the hybrid detection model and rules 304B of the anomaly detection as a service 304 can be used to group specific historical cost per click data or other historical performance data. For example, the historical cost per click data can be grouped based on one or more of the timestamp or duration of the digital advertisement associated with the cost per click data. In some embodiments, the historical cost per click data grouping can be based on a specific geographic location associated with the cost per click data (e.g., a specific country, state, town, city, province, region, street, etc.).

[0072] In an example embodiment, the alarm component 304C may generate an abnormal alarm, such as Figure 5 304D. In an embodiment, a user may generate feedback 304D via a user device or an artificial intelligence model. For example, an artificial intelligence model may be updated via an AI cloud service that uses machine learning to identify false anomaly detections and updates MMD model 306 based on the false anomaly detections. For example, an artificial intelligence model may be trained to identify false anomalies using a training data set (e.g., stored in data source 302) that is generated based on historical indications from a user device that an anomaly detection is a false anomaly detection. In some embodiments, feedback 304D may also include an indication from a user device that an anomaly detection is a false anomaly detection (e.g., such as Fig.13In some embodiments, a false anomaly detection may correspond to Fig.12 One or more of the true positive rate, false positive rate, specific true negative rate, or false positive rate.

[0073] In some embodiments, the API entry point 304E may correspond to Figure 8 API requests and responses, or Fig. 9 In some embodiments, dashboard 304H may provide a custom API as shown in example environment 900 (e.g., including an API for obtaining all anomaly detection metadata or historical anomaly detection data). Figures 4 to 7 as well as Fig.13 In some embodiments, dashboard 304H may be generated based on updating or fine-tuning MMD model 306. In some embodiments, MMD model 306 may be updated by changing the symmetric moving average of the previous set of cost-per-click data time series based on different times and dates corresponding to the second set of cost-per-click data time series.

[0074] In embodiments, the infrastructure 308 may include an authentication and certification service (e.g., NuData) for authenticating and authenticating the transmission of cost-per-click data or other performance data (e.g., between the anomaly detection as a service 304 and the data source 302). In some embodiments, the infrastructure 308 may provide third-party digital service provider services (e.g., via Apache Airflow), which may include one or more of the following: authentication, authorization, control, service registration, security, monitoring, alerting, start / stop, scaling, restart, certification revocation, backup / restore, or one or more combinations thereof. In some embodiments, some of these provided third-party digital service provider services may be provided by scripts executed in the provision of services. In these environments, continuous security and compliance may be enabled and enforced (e.g., by using managed encryption (managed keys), automated controls, advanced auditing (e.g., instance-level vulnerability scanning and intrusion detection, centralized security and log aggregators with advanced dashboards), and centralized network control of egress and policies associated with firewalls).

[0075] Figure 4An example dashboard 400 is shown (e.g., a dashboard generated based on output provided by a performance anomaly detection model such that anomaly detections are displayed on a user device via tables and graphs). As shown in the graphical depiction of the performance anomaly detection model output, anomaly detections associated with cost per click data are determined. Each of the detected anomalies is associated with an upper bound (e.g., a "uBound" column) and a lower bound (e.g., a "lBound" column) and a seasonal minimum. In some embodiments, the performance anomaly detection model may process the cost per click data in hourly intervals. In some embodiments, a user may pause one or more specific alerts associated with the example dashboard 400.

[0076] Figure 5 An example email alert 500 is shown with an embedded dashboard table corresponding to anomaly detection associated with cost per click (CPC) data. The embedded dashboard table includes a selectable icon that, upon selection, can quiet or silence the alert for a specific time period (e.g., silence the alert for the next one or more hours). The embedded dashboard table may also include a "mark error" selectable icon such that upon selection of the "mark error" icon, a signal may be sent to notify another user device (e.g., a user device associated with the first entity). In some embodiments, selecting the "mark error" icon may be used to generate further input or training data into the performance anomaly detection model to update the model. In an embodiment, the example email alert 500 may include an optional link, wherein upon selection of the link, a web page or application is launched to provide additional details of the performance metrics and further details for the metrics. In addition, each CPC data in the email alert 500 may have corresponding upper and lower limits and a timestamp. The example email alert 500 may be associated with hourly generated CPC data for a specific third-party platform within a specific geographic region (e.g., Australia).

[0077] Figure 6 to Figure 7 Example dashboards 600 and 700 are shown, which include anomaly detection for CPC data received hourly for a specific third-party platform associated with a specific geographic region (e.g., the United States), and anomaly detection for click and spend data received hourly. The graphs include predicted upper and lower boundaries. Sharp points within the graphs represent anomalies. Dashboards 600 and 700 also include a weekly summary of each detected anomaly. As described above, Figure 8 shows example API requests and responses associated with a performance anomaly detection model, and Fig. 9The example environment 900 includes customized APIs that include anomaly detection historical data responses, anomaly detection metadata requests, anomaly detection triggers (e.g., for a given feature, such as anomalies above a threshold or anomalies for a specific set of provided items), and other APIs.

[0078] Fig.10 An example MMD model 1000 is shown, and Fig.11 A diagram 1100 is included that compares the differences between the prior art and the currently disclosed art. For example, the example MMD model 1000 includes an anomaly detection algorithm known as a moving metric detector (MMD) that is capable of identifying potential alerts for specific anomalies associated with cost per click data or other performance metrics with distribution agnostic criteria. The MMD model 1000 can decompose the cost per click data time series to identify anomalies in patterns (e.g., based on trends or specific seasons or time periods) as well as residuals.

[0079] In some embodiments, to identify true anomalies, Chebyshev's Inequality can be used to determine the anomaly criteria to control the overall false positive detection rate. The MMD model 1000 can learn patterns in the time series and use them as predictions to test whether the last observation is within various ranges of cost per click data (e.g., Figures 4 to 5 as well as Figure 6 to Figure 7 (The upper and lower boundaries shown are associated) For a cost-per-click data time series X, an estimated trend L can be determined using a symmetric moving average with a sliding time window (w), which can be the number of CPC observations for a particular time series X. To determine the window (w) for each CPC metric, the ESPRIT method can be used to determine a single signal frequency estimate (e.g., estimating signal parameters via a rotational invariant technique (ESPRIT) for determining parameters of a mixture of sinusoids within background noise). Using the trend L, seasonality can be determined to reduce the variance of X. To extract robust seasonality (S), in some embodiments, the median of the historical period values ​​of the detrended time series L' can be used instead of the mean, which may be more sensitive to outliers. As Fig.10 As shown, (w) is the number of observations per period, (i) is the number of periods far from the observation at (t), and τ is the number of periods close to t, which is used to estimate the seasonality (S t ). In addition, τ can be set to the length of the input CPC time series X.

[0080] As mentioned above, Fig.12 An example table 1200 is included for comparing the precision and accuracy of anomaly detection of the present technique with detection by the prior art. Fig.13An example email alert 1300 is included with an embedded single data point graph including movement metrics associated with cost per click data within Canada, Australia, and Germany (eg, including an invalidation indication of a detected anomaly).

[0081] Figure 14 to Figure 15 Includes example flow chart.

[0082] Fig.14 Flowchart 1400 begins at step 1402, where cost per click data for a first entity is automatically received from a server hosted by a third party. Figure 1 A third-party server 140 or Figure 2 The third-party server 202 automatically receives (for example, using Figure 3 In an embodiment, the cost per click data is automatically received at periodic intervals (e.g., within an hourly time frame). Historical cost per click data for the first entity may also be received from a server hosted by a third party.

[0083] At step 1404, the performance anomaly detection model is applied to the cost per click data to generate an output. In an embodiment, the performance anomaly detection model includes a time series model, and a maximum average difference may also be applied. For example, the maximum average difference may be a frequency measure of similarity between CPC distributions, which may be used to train one or more generative models of the performance anomaly detection model. The performance anomaly detection model may also be applied to the cost per click data received from a second server associated with a third party to generate a second output (e.g., to generate a Fig.13 Output for different countries shown).

[0084] At step 1406, the output provided by the performance anomaly detection model is compared to an anomaly threshold. In an embodiment, based on historical cost per click data or Figure 1 The output may be analyzed by other business 120 to determine an abnormal threshold. As an example, the abnormal threshold may be based on a spending cap associated with the first entity in a particular country. In addition, based on determining that the output is above the abnormal threshold, at step 1408, a notification may be provided to the user device. For example, the notification may correspond to Figure 1 The remediation engine 126 or the alert engine 130 .

[0085] Fig.15Flowchart 1500 of begins at step 1502, where a performance anomaly detection model is trained using a first set of historical cost-per-click data and a second set of historical performance metrics. In an embodiment, the first set and the second set each correspond to a different geographic location. In some embodiments, the first set includes specific cost-per-click data for a first set of items provided on a digital platform, and the second set includes specific cost-per-click data for a second set of items. In some embodiments, the first group is grouped based on a specific date (e.g., the first Monday of each month during the summer). In some embodiments, the first group includes historical cost-per-click data for display network ads (e.g., Google display network ads, online auction applications, etc., in which ads participating in online auctions are input to an ad server to determine which of those ads will win the auction), and the second group includes historical cost-per-click data for product listing ads (PLAs) (e.g., digital shopping ads on Google Search, Google Shopping, Google Image Search, dynamic product ads on Facebook and Instagram, etc.).

[0086] Step 1504 includes fine-tuning the performance anomaly detection model for a particular performance metric. In an embodiment, the performance anomaly detection model may be fine-tuned based on a timestamp corresponding to the cost per click data, based on the location of a third party that sent the cost per click data, based on a daily or hourly budget in which clicks associated with the cost per click data occurred, based on a type of digital advertisement associated with the cost per click data, based on a particular item or service offered in association with the cost per click data, another type of fine-tuning parameter, or one or more combinations thereof. For example, the performance anomaly detection model may be fine-tuned for items offered on a product listing advertisement based on a product listing advertisement in Canada that belong to a particular category (e.g., dog food). As another example, the performance anomaly detection model may be fine-tuned for items offered on a display network advertisement based on a display network advertisement in Germany that are sold by a particular seller.

[0087] At step 1506, a performance anomaly detection model may be applied to the first set of performance metrics. For example, the performance anomaly detection model may be applied to the cost per click data from the third party server for anomaly detection, the training performance anomaly detection model being trained using the first entity's historical cost per click data corresponding to different specific geographic regions. In some embodiments, the performance anomaly detection model may be Figure 1 Performance of anomaly detection model 114 or Figure 2 The performance anomaly detection model 220. At step 1508, based on applying the performance anomaly detection model to the first set of performance metrics, it can be determined that the output is above an anomaly threshold.

[0088] Having described an overview of embodiments of the present technology, an example operating environment in which embodiments of the present technology may be implemented is described below to provide a general context for various aspects. Fig.16 , an example operating environment for implementing embodiments of the present technology is shown and generally designated as computing device 1600. Computing device 1600 is merely an example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the present technology. Neither should computing device 1600 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated.

[0089] The technology of the present disclosure may be described in the general context of computer code or machine-usable instructions, which include computer-executable instructions (e.g., program modules) executed by a computer or other machine (e.g., a personal data assistant or other handheld device). Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs a specific task or implements a specific abstract data type. The technology may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The technology may also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked through a communications network.

[0090] refer to Fig.16 , computing device 1600 includes bus 1602 that directly or indirectly couples the following devices: memory 1604, one or more processors 1606, one or more presentation components 1608, input / output ports 1610, input / output components 1612, and schematic power supply 1614. Bus 1602 represents one or more buses (e.g., an address bus, a data bus, or a combination thereof). Although for clarity, Fig.16 The various boxes of an MCU are represented by lines, but in reality, it is not so clear to depict the various components, and metaphorically, the lines would more accurately be gray and fuzzy. For example, a presentation component (e.g., a display device) can be considered an I / O component. As another example, a processor may also have memory. This is the nature of the art, and again, Fig.16 The figures are merely examples of computing devices that can be used in conjunction with one or more embodiments of the present technology. There is no distinction between categories such as "workstation," "server," "laptop," "handheld device," etc., as all of these categories are in the same Fig.16 and refers to “computing devices”.

[0091] The computing device 1600 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 1600, and includes volatile and non-volatile media, and removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0092] Computer storage media includes volatile and nonvolatile, and removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the computing device 1600. Computer storage media themselves do not include signals.

[0093] Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in the form of a modulated data signal (such as a carrier wave or other transmission mechanism), and includes any information transmission medium. The term "modulated data signal" refers to a signal that is set or changed in a manner that encodes the information in the signal, and the signal has one or more characteristics. By way of example and not limitation, communication media include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0094] Memory 1604 includes computer storage media in the form of volatile memory or non-volatile memory. Memory 1604 can be removable, non-removable, or a combination thereof. Example hardware devices include solid-state memory, hard drives, optical drives, etc. Computing device 1600 includes one or more processors that read data from various entities such as memory 1604 or I / O components 1612. Presentation component 1608 presents data indications to a user or other device. Examples of presentation components include display devices, speakers, printing components, vibration components, etc.

[0095] I / O ports 1610 allow computing device 1600 to be logically coupled to other devices including I / O components 1612, some of which may be built-in. Illustrative components include microphones, joysticks, game pads, satellite dishes, scanners, printers, wireless devices, and the like.

[0096] The above-described embodiments may be combined with one or more specifically described alternatives. Specifically, a claimed embodiment may include references to more than one other embodiment in an alternative. A claimed embodiment may specify additional limitations of the claimed subject matter.

[0097] The subject matter of the present technology is described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. Instead, the inventors have contemplated that the claimed or disclosed subject matter may also be embodied in other ways to include different steps or combinations of steps similar to the steps described in this document in conjunction with other prior art or future technologies. In addition, although the terms "step" or "box" may be used herein to refer to different elements of the method employed, the terms should not be interpreted as implying any particular order between or among the various steps disclosed herein, unless and in addition to the order of the various steps being explicitly described.

[0098] For purposes of this disclosure, the words "including" or "having" have the same broad meaning as the word "comprising," and the word "accessing" includes "receiving," "referencing," or "retrieving." Additionally, the word "communicating" has the same broad meaning as the words "receiving" or "sending" facilitated by a software or hardware based bus, receiver, or transmitter using a communications medium.

[0099] Furthermore, unless otherwise specified, words such as "a", "an" include the plural as well as the singular. Thus, for example, where there are one or more features, the constraint of "a feature" is satisfied. Furthermore, the term "or" includes conjunctions, disjunctions, and both (thus a or b includes a or b, and a and b).

[0100] For the purposes of the above detailed discussion, embodiments of the present technology are described with reference to a distributed computing environment; however, the distributed computing environment described herein is only one example. Components may be configured to perform novel aspects of the embodiments, where the term "configured to" or "configured to" may refer to being "programmed to" perform a particular task or to implement a particular abstract data type using code. Furthermore, while embodiments of the present technology may generally refer to a distributed data object management system and the schematic diagrams described, it should be understood that the described technology may be extended to other implementation contexts.

[0101] From the foregoing, it can be seen that the present technology is well suited to achieve all of the above-mentioned goals and objectives, including other advantages that are obvious or inherent to the structure. It will be understood that some features and sub-combinations are useful and can be adopted without reference to other features and sub-combinations. This is contemplated by the claims and is within the scope of the claims. Since many possible embodiments of the described technology can be made without departing from the scope, it should be understood that all matter described herein or shown in the drawings should be interpreted as illustrative and not restrictive.

[0102] Some example aspects of the technology that can be implemented from the foregoing disclosure include the following:

[0103] Aspect 1: A computer-implemented method comprising: establishing a communication session with a server hosted by a third party; automatically receiving cost-per-click data for a first entity from the server hosted by the third party; applying a performance anomaly detection model trained using historical cost-per-click data to the cost-per-click data to generate an output; comparing the output provided by the performance anomaly detection model to an anomaly threshold; and based on determining that the output is above the anomaly threshold, broadcasting a notification (e.g., to a user device).

[0104] Aspect 2: According to aspect 1, wherein the cost-per-click data is automatically received at periodic intervals.

[0105] Aspect 3: According to Aspect 1 or 2, it also includes: receiving historical cost-per-click data of the first entity from a server hosted by the third party; grouping the historical cost-per-click data based on a predetermined time period; and using a first grouping of the historical cost-per-click data to train a performance anomaly detection model, wherein the first grouping is based on a location associated with the historical cost-per-click data.

[0106] Aspect 4: According to aspect 1, 2 or 3, it also includes: identifying the location of the server; receiving a timestamp for each of the historical cost-per-click data of the first group; and adjusting the performance anomaly detection model based on the location of the server and the received timestamps.

[0107] Aspect 5: According to aspect 1, 2, 3 or 4, the performance anomaly detection model includes a time series model.

[0108] Aspect 6: According to aspect 1, 2, 3, 4 or 5, wherein the performance anomaly detection model applies the maximum average difference.

[0109] Aspect 7: According to Aspect 1, 2, 3, 4, 5 or 6, it also includes: establishing a second communication session with a second server hosted by another third party; automatically receiving cost-per-click data of the first entity from the second server hosted by the other third party; applying a performance anomaly detection model to the cost-per-click data received from the second server to generate a second output; comparing the second output provided by the performance anomaly detection model with an anomaly threshold; and providing a second notification based on determining that the second output is higher than the anomaly threshold.

[0110] Aspect 8: According to aspect 1, 2, 3, 4, 5, 6 or 7, wherein the abnormality threshold is based on a spending cap associated with the first entity.

[0111] Aspect 9: According to Aspect 1, 2, 3, 4, 5, 6, 7 or 8, it also includes: receiving historical cost-per-click data of the first entity from a server hosted by the third party; grouping the historical cost-per-click data into at least a first group based on a first predetermined time period and a second group based on a second predetermined time period, the second predetermined time period being a time period later than the first predetermined time period; using the first group of historical cost-per-click data and the second group of historical cost-per-click data to train a performance anomaly detection model; and applying the trained performance anomaly detection model to the cost-per-click data to generate an output, which is generated based on a time period corresponding to the cost-per-click data of the first entity received from the server hosted by the third party, wherein, with respect to the first predetermined time period and the second predetermined time period, the output is generated based on the time period corresponding to the cost-per-click data.

[0112] Aspect 10: A computer system comprising: a processor; and a computer storage medium storing computer-usable instructions which, when used by the processor, cause the computer system to perform operations comprising: establishing a communication session with a server hosted by a third party; automatically receiving a plurality of cost-per-click data of a first entity from the server hosted by the third party; applying a performance anomaly detection model to the plurality of cost-per-click data for anomaly detection, the performance anomaly detection model being trained using historical cost-per-click data; determining, based on the application of the performance anomaly detection model, that a first cost-per-click data among the plurality of cost-per-click data is above an anomaly threshold; and broadcasting a notification based on determining that the first cost-per-click data is above the anomaly threshold.

[0113] Aspect 11: According to Aspect 10, the performance anomaly detection model is trained using historical cost per click data of the first entity, historical cost per click data, and cost per click data corresponding to a specific geographic area.

[0114] Aspect 12: According to Aspect 10 or 11, it also includes: establishing a communication session with another server; automatically receiving cost-per-click data of the first entity from the other server, wherein the cost-per-click data from the other server corresponds to different specific geographic areas; applying a performance anomaly detection model to the cost-per-click data from the other server for anomaly detection, the performance anomaly detection model being trained using historical cost-per-click data of the first entity corresponding to different specific geographic areas; based on applying the performance anomaly detection model to the cost-per-click data from the other server, determining that the cost-per-click data from the other server is higher than a second anomaly threshold; and broadcasting a second notification based on determining that the cost-per-click data from the other server is higher than the second anomaly threshold.

[0115] Aspect 13: According to Aspect 10, 11 or 12, wherein the multiple cost-per-click data include multiple time series of cost-per-click data, and wherein the performance anomaly detection model utilizes a symmetric moving average based on the time and date corresponding to each time series of the multiple time series of cost-per-click data to determine that the first cost-per-click data is above an anomaly threshold.

[0116] Aspect 14: According to aspect 10, 11, 12 or 13, wherein the performance anomaly detection model utilizes estimation of signal parameters via a rotational invariance technique to determine that the first cost-per-click data is above an anomaly threshold.

[0117] Aspect 15: One or more computer storage media storing computer-usable instructions that, when used by a computing device, cause the computing device to perform operations, the operations comprising: automatically receiving a plurality of time series, each time series comprising cost-per-click data for a first entity from a first server, wherein the cost-per-click data from the first server corresponds to a first geographic region; applying the performance anomaly detection model to each of the plurality of time series for anomaly detection, each time series comprising the cost-per-click data, the performance anomaly detection model being trained using historical time series including historical cost-per-click data of the first entity corresponding to the first geographic region; based on applying the performance anomaly detection model to each of the plurality of time series, displaying an output provided by the performance anomaly detection model on a user device; based on the output provided by the performance anomaly detection model, identifying cost-per-click data above an anomaly threshold in the plurality of time series; and based on identifying cost-per-click data above the anomaly threshold, displaying anomaly detection corresponding to the cost-per-click data above the anomaly threshold on a user device.

[0118] Aspect 16: According to Aspect 15, wherein the output and anomaly detection provided by the performance anomaly detection model are displayed on the user device in a graphical form.

[0119] Aspect 17: According to aspect 15 or 16, wherein the output and anomaly detection provided by the performance anomaly detection model are displayed on the user device in the form of a table.

[0120] Aspect 18: According to Aspect 15, 16 or 17, the operations further include: based on causing the output and anomaly detection provided by the performance anomaly detection model to be displayed on the user device, receiving an indication from the user device that the anomaly detection is a false anomaly detection, and updating the performance anomaly detection model based on the false anomaly detection.

[0121] Aspect 19: According to Aspect 15, 16, 17 or 18, these operations also include: automatically receiving a second plurality of time series from the first server, each of the second plurality of time series including cost-per-click data of the first entity from the first server, wherein the cost-per-click data of the second plurality of time series corresponds to the first geographic area; based on updating the performance anomaly detection model, applying the updated performance anomaly detection model to each time series in the second plurality of time series; based on applying the updated performance anomaly detection model, displaying the output provided by the updated performance anomaly detection model on a user device; identifying cost-per-click data above an anomaly threshold from the second plurality of time series; and displaying anomaly detection corresponding to the cost-per-click data above the anomaly threshold in the second plurality of time series on the user device.

[0122] Aspect 20: According to Aspect 15, 16, 17, 18 or 19, these operations also include: automatically receiving a second plurality of time series from a second server, each of the second plurality of time series including cost-per-click data of the first entity from the first server, wherein the cost-per-click data of the second plurality of time series corresponds to a second geographic area different from the first geographic area; applying the performance anomaly detection model to each time series in the second plurality of time series; causing output for the second plurality of time series provided by the updated performance anomaly detection model to be displayed on a user device; identifying cost-per-click data above a second anomaly threshold from the second plurality of time series; and causing anomaly detection corresponding to the cost-per-click data above the second anomaly threshold in the second plurality of time series to be displayed on the user device.

Claims

1. A computer-implemented method comprising: Establishing communications sessions with servers hosted by third parties; automatically receiving cost-per-click data for the first entity from a server hosted by the third party; applying a performance anomaly detection model trained using historical cost per click data to the cost per click data to generate an output; comparing an output provided by the performance anomaly detection model to an anomaly threshold; as well as Based on determining that the output is above the abnormal threshold, a notification is broadcast.

2. The computer-implemented method of claim 1 , wherein: The cost-per-click data is automatically received at periodic intervals.

3. The computer-implemented method of claim 1 , further comprising: receiving historical cost-per-click data for the first entity from the server hosted by the third party; grouping the historical cost-per-click data based on a predetermined time period; as well as The performance anomaly detection model is trained using a first grouping of the historical cost-per-click data, wherein the first grouping is based on locations associated with the historical cost-per-click data.

4. The computer-implemented method of claim 3, further comprising: identifying the location of the server; receiving a timestamp for each of the cost-per-click data; as well as The performance anomaly detection model is adjusted based on a location of the server and a timestamp of each of the received cost-per-click data.

5. The computer-implemented method of claim 1 , wherein: The performance anomaly detection model includes a time series model.

6. The computer-implemented method of claim 5, wherein: The performance anomaly detection model applies the maximum mean difference.

7. The computer-implemented method of claim 1 , further comprising: establishing a second communication session with a second server hosted by another third party; automatically receiving cost-per-click data for the first entity from a second server hosted by the other third party; applying the performance anomaly detection model to the cost-per-click data received from the second server to generate a second output; comparing the second output provided by the performance anomaly detection model to the anomaly threshold; as well as Based on determining that the second output is above the abnormal threshold, a second notification is broadcast.

8. The computer-implemented method of claim 1 , wherein: The anomaly threshold is based on a spending cap associated with the first entity.

9. The computer-implemented method of claim 1 , further comprising: receiving historical cost-per-click data for the first entity from the server hosted by the third party; grouping the historical cost-per-click data into at least a first group based on a first predetermined time period and a second group based on a second predetermined time period, the second predetermined time period being a time period later than the first predetermined time period; training the performance anomaly detection model using the first set of historical cost-per-click data and the second set of historical cost-per-click data; as well as Applying a trained performance anomaly detection model to the cost-per-click data to generate an output, the output being generated based on a time period corresponding to the cost-per-click data for the first entity received from the server hosted by the third party, wherein the output is generated based on a time period corresponding to the cost-per-click data relative to the first predetermined time period and the second predetermined time period.

10. A computer system comprising: processor; as well as A computer storage medium storing computer usable instructions that, when used by the processor, cause the computer system to perform operations comprising: Establishing communications sessions with servers hosted by third parties; automatically receiving a plurality of cost-per-click data for a first entity from said server hosted by said third party; applying a performance anomaly detection model to the plurality of cost-per-click data to perform anomaly detection, the performance anomaly detection model being trained using historical cost-per-click data; Based on applying the performance anomaly detection model, determining that a first cost-per-click data among the plurality of cost-per-click data is above an anomaly threshold; and Based on determining that the first cost-per-click data is above the abnormal threshold, a notification is broadcast.

11. The computer system according to claim 10, wherein: The performance anomaly detection model is trained using cost per click data and historical cost per click data corresponding to a particular geographic area, and historical cost per click data for the first entity.

12. The computer system of claim 11, further comprising: establishing a communication session with another server; automatically receiving cost-per-click data for the first entity from the other server, wherein the cost-per-click data from the other server corresponds to a different specific geographic region; applying the performance anomaly detection model to the cost per click data from the other server to perform anomaly detection, the trained performance anomaly detection model being trained using historical cost per click data of the first entity corresponding to the different specific geographic region; determining, based on applying the performance anomaly detection model to the cost-per-click data from the other server, that the cost-per-click data from the other server is above a second anomaly threshold; and Based on determining that the cost-per-click data from the other server is above the second anomaly threshold, a second notification is broadcasted.

13. The computer system according to claim 10, wherein: The multiple cost-per-click data include multiple time series of cost-per-click data, and wherein the performance anomaly detection model utilizes a symmetric moving average based on the time and date corresponding to each of the multiple time series of cost-per-click data to determine that the first cost-per-click data is above the anomaly threshold.

14. The computer system of claim 13, wherein: The performance anomaly detection model determines that the first cost-per-click data is above the anomaly threshold using an estimate of a signal parameter via a rotational invariance technique.

15. One or more non-transitory computer storage media storing computer usable instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: automatically receiving a plurality of time series, each time series comprising cost-per-click data for a first entity from a first server, wherein the cost-per-click data from the first server corresponds to a first geographic region; applying a performance anomaly detection model to each of the plurality of time series for anomaly detection, each time series including the cost-per-click data, the performance anomaly detection model being trained using historical time series including historical cost-per-click data for the first entity corresponding to the first geographic region; Based on applying the performance anomaly detection model to each of the plurality of time series, causing an output provided by the performance anomaly detection model to be displayed on a user device; Based on output provided by the performance anomaly detection mode, identifying cost per click data from the plurality of time series that is above an anomaly threshold; and Based on identifying the cost-per-click data above the anomaly threshold, causing display on the user device of the anomaly detection corresponding to the cost-per-click data above the anomaly threshold.

16. The one or more non-transitory computer storage media of claim 15, wherein: The output provided by the performance anomaly detection model and the anomaly detection are displayed in graphical form on the user device.

17. The one or more non-transitory computer storage media of claim 15, wherein: The output provided by the performance anomaly detection model and the anomaly detection are displayed on the user device in the form of a table.

18. The one or more non-transitory computer storage media of claim 15, the operations further comprising: Based on causing display on the user device of the output provided by the performance anomaly detection model and the anomaly detection, receiving from the user device an indication that the anomaly detection is a false anomaly detection, and updating the performance anomaly detection model based on the false anomaly detection.

19. The one or more non-transitory computer storage media of claim 18, the operations further comprising: automatically receiving from the first server a second plurality of time series, the second plurality of time series each comprising cost-per-click data for the first entity, wherein the cost-per-click data of the second plurality of time series corresponds to the first geographic area; based on updating the performance anomaly detection model, applying the updated performance anomaly detection model to each time series in the second plurality of time series; Based on applying the updated performance anomaly detection model, causing an output provided by the updated performance anomaly detection model to be displayed on the user device; identifying cost-per-click data from the second plurality of time series that is above the anomaly threshold; and Anomaly detections corresponding to cost-per-click data in the second plurality of time series that are above the anomaly threshold are caused to be displayed on the user device.

20. The one or more non-transitory computer storage media of claim 15, the operations further comprising: automatically receiving from a second server a second plurality of time series, the second plurality of time series each comprising cost-per-click data for the first entity, wherein the cost-per-click data of the second plurality of time series corresponds to a second geographic area different from the first geographic area; applying the performance anomaly detection model to each time series in the second plurality of time series; causing outputs for the second plurality of time series provided by the updated performance anomaly detection model to be displayed on the user device; identifying cost-per-click data from the second plurality of time series that is above a second anomaly threshold; and Anomaly detections corresponding to the cost-per-click data in the second plurality of time series that are above the second anomaly threshold are caused to be displayed on the user device.

Images