Systems, computer implementations, and computer programs (association of disturbance events with accidents or tickets)

Abstract

To provide systems and methods for training machine learning models to generate useful data for performing predictions and forecasts.SOLUTION: A method for incorporating statistical change point analysis for time series events corresponding to service tickets in relevant geographic sub-regions is provided. The method includes calibrating to separate a normal period from an environmental disturbance period under the assumption that disturbances significantly increase the rate of loss-causing events. In order to acquire the probability that a given loss-causing event is associated with an environmental disturbance, the method includes exploiting the difference between the rate of events expected in the absence of any disturbance (baseline) and the rate of events actually observed. In the analysis, the method includes identifying local disturbances and acquiring estimated values of their duration and magnitude.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] This application relates in general to computers and computer applications, more specifically to systems and methods for generating data useful for training machine learning models to perform predictions and forecasts. [Background technology]

[0002] Environmental disturbances such as weather storms, blizzards, and electromagnetic hazards often cause asset failures or malfunctions, as well as associated power outages, leading to service quality issues. However, such failures can also occur when there are no environmental disturbances. In other words, failures are generally not identifiable as being caused by data-driven disturbances, such as those resulting from limited information available when data is being compiled, time constraints are present, or staff training is insufficient. For better business management, it is always desirable that all failures be appropriately labeled or classified, for example, as a catastrophe.

[0003] Examples of such disturbances can often be found in the case of infrastructure companies, power infrastructure for electric companies supplying electricity (weather storms), and sensor networks deployed in the wild for chemical manufacturing processes (electromagnetic hazards).

[0004] Modern critical analyses require clean data for those malfunction or failure events. Therefore, events need to be automatically labeled to separate them into normal and environmental disturbance cases.

[0005] From a data quality perspective, it is necessary to validate existing labels and fill in any missing labels. [Overview of the project] [Problems that the invention aims to solve]

[0006] For public utility companies, understanding the relationship between external disturbances (weather conditions) and the resulting power outages is essential. [Means for solving the problem]

[0007] The system, method, and computer program product provides the ability to reliably and automatically label asset malfunctions, failures, or power outages as being caused by external disturbances (e.g., environmental disturbances).

[0008] Since machine learning (ML) models can be trained and executed by hardware processors to make predictions and generate forecasts related to the input, systems, methods, and computer program products are provided that process the data to appropriately label or classify (e.g., as a catastrophe) or a combination thereof before it is used as training data for the model training set, so as to improve the quality of the training data.

[0009] Systems, methods, and computer program products typically automatically label or classify (as catastrophics) or a combination thereof as trouble tickets associated with infrastructure failure events involving environmental disturbances in areas that cause malfunctions or failures in specific assets or result in power outages.

[0010] Systems, methods, and computer program products provide probabilistic labeling to associate asset malfunctions, failures, or power outages with disturbances, which may be known or unknown based on available data.

[0011] The system, method, and computer program product implement statistical change point analysis for time-series events corresponding to service tickets in relevant geographical sub-regions.

[0012] The framework of the system and method automatically labels events and separates the events into normal or out-of-environment disturbance cases before being used as training data for the model training set.

[0013] The framework of the system and method provides automated data processing to "clean" data on their malfunctions or failure events in various use case domains before being used as training data for the model training set.

[0014] The framework of the system and method further validates existing labels and fills in missing labels in the model training set, which is automated to greatly reduce the time required to be invested by human validators. Finally, the system and method identify all storms and label tickets as either related to the storm or not.

[0015] In some examples, a system for probabilistic labeling is provided. The system includes memory and a processor, which includes hardware and is configured to communicate with the memory. The processor is configured to receive a set of service records from a data source, one or more service records corresponding to events representing multiple disturbances in a specified region occurring during a period, and one or more service records that do not have labels associated with mislabeled or associated disturbances; to determine the observed actual event rate of events representing disturbances for the period; to determine the periodically aggregated baseline mean rate of expected service-related records under non-disturbance conditions for the period; to determine a set of standardized daily scores as a function of the periodically aggregated baseline mean rate and the observed actual event rate during the period; to identify disturbance time windows based on changes in the standardized periodically aggregated scores detected during the period; to generate disturbance-related probabilities for service records corresponding to the identified disturbance time windows; and to reassign labels to service tickets as associated with the identified disturbance time windows based on the generated probabilities.

[0016] In several examples, methods for probabilistic labeling are provided. A computer implementation method includes the steps of: receiving a set of service records from a data source, one or more service records corresponding to events representing multiple disturbances in a specified region occurring during a period, and one or more service records that do not have labels associated with mislabeled or associated disturbances, by a hardware processor; determining the observed actual event rate of events representing disturbances for the period by the hardware processor; determining the periodically aggregated baseline mean rate of expected service-related records under non-disturbance conditions for the period by the hardware processor; determining a set of standardized periodically aggregated scores as a function of the periodically aggregated baseline mean rate and the observed actual event rate during the period by the hardware processor; identifying a disturbance time window based on changes in the standardized periodically aggregated scores detected during the period by the hardware processor; generating disturbance-related probabilities for service tickets corresponding to the identified disturbance time window by the hardware processor; and reassigning labels to service records as associated with the identified disturbance time window based on the generated probabilities.

[0017] In several examples, computer program products for probabilistic labeling are generally described. These computer program products may include a computer-readable storage medium in which program instructions are embodied. The program instructions may be executable by the device's processing elements to cause the device to perform one or more of the methods described herein.

[0018] The following sections will describe in detail the further features and the structure and operation of various embodiments, with reference to the attached drawings. In the drawings, similar reference numerals indicate identical or functionally similar elements. [Brief explanation of the drawing]

[0019] [Figure 1] This document provides a general overview of a system framework for probabilistic labeling of events, and for the detection and identification of disturbances.

[0020] [Figure 2A] The service ticket dataset used in data quality analysis before restoration is shown below.

[0021] [Figure 2B] The output of the restored and corrected dataset includes records containing information on detected storms and also includes probabilistic labels.

[0022] [Figure 3] This specification provides a computer system for implementing a framework and methodology for improving the performance of a trouble ticket dataset according to the embodiments described herein.

[0023] [Figure 4] This method is implemented by a supervisory program that implements an automated task to (probabilistically) more accurately label trouble ticket data with disturbances (e.g., storms) performed by the system in Figure 3.

[0024] [Figure 5] A table is shown illustrating the characteristics of a ticket representing a storm in an exemplary embodiment.

[0025] [Figure 6] The system shown in Figure 3 illustrates the overall system architecture for identifying disturbances, probabilistic labeling, and assigning events (tickets) to previously known disturbances (known storms in specific sub-regions).

[0026] [Figure 7]This chart shows a time series plot of the period-aggregated SRT count rate (Y-axis) of a location-specific sub-region over time (X-axis).

[0027] [Figure 8A] A table summarizing the notation and exemplary default values ​​used in the change point analysis according to the embodiments of this specification is provided.

[0028] [Figure 8B] A table is provided that outlines the multiple time-series data used in the analysis according to the embodiments of this specification.

[0029] [Figure 9] This shows the lifetime of disturbances such as storms, which have a life cycle involving three states.

[0030] [Figure 10A] An exemplary CUSUM control plot of calculated CUSUM values ​​plotted against time is shown to illustrate an embodiment of applying a calibrated change point algorithm used to identify the boundaries of local disturbances.

[0031] [Figure 10B] An exemplary CUSUM control plot of the calculated CUSUM value in the first alternative instance is shown.

[0032] [Figure 10C] An exemplary CUSUM control plot of the calculated CUSUM value in a second alternative instance is shown.

[0033] [Figure 10D] An exemplary CUSUM control plot of the calculated CUSUM value in a third alternative instance is shown.

[0034] [Figure 11] An exemplary pseudocode of a calibrated change-point method is shown, where time points are acquired as shown in Figure 10A to establish the storm boundary.

[0035] [Figure 12] This diagram shows a subset of data used in the verification process, specifically illustrating the categorization of tickets for a public utility company according to the embodiment of this specification.

[0036] [Figure 13] This figure shows the key metrics used in the verification process according to the embodiments of this specification.

[0037] [Figure 14] This flowchart illustrates the processes that may be executed by the processor to implement preventative measures using a machine learning model according to the embodiment.

[0038] [Figure 15] A schematic diagram of an exemplary computer or processing system that may implement probabilistic labeling of service tickets in one embodiment of the present disclosure is shown. [Modes for carrying out the invention]

[0039] In one embodiment, the system framework implements a method for probabilistic labeling of records or data related to the detection and identification of events (e.g., asset malfunctions, equipment failures, and power outages) and disturbances.

[0040] The ability to reliably label asset malfunctions, failures, and power outages as being caused by external disturbances substantially improves data quality and the openness of additional analytical techniques, which ultimately leads to better power outage detection, prediction, and management.

[0041] As an example, the system framework is described in the exemplary context of managing weather-related disturbances (implemented, for example, by a utility company). A particular concern is storm-related disturbances, and this type of company tends to emphasize preparedness, minimizing the number of affected customers, and resource planning. Therefore, understanding the relationship between disturbances (weather conditions) and their power outages is essential for them.

[0042] To aid in the explanation, some definitions of terms are provided in the context of managing weather-related power outages.

[0043] "Service" is the act of performing work for a customer in accordance with several contractual obligations. Examples include the provision of electricity or computing support. In one example, attention is focused on the interests of a service provider, such as a utility company.

[0044] A “disturbance” is an interruption of the stable or normal conditions of a service. Disturbances may be entirely or partially predictable. Their effects involve altering or exacerbating the infrastructure supporting the service to the extent that it causes quality-of-service problems, such as interruptions. In the analysis of weather-related power outages, the disturbance corresponds to a storm. In this context, the impact of disturbances is often related to the power line infrastructure and its characteristics (e.g., demographics or topography) in a local sub-domain. Another example of a disturbance in a different domain or context is a disturbance such as a change in a computer operating system, for example, in the help desk service domain.

[0045] An "event" is a maintenance or repair request to provide corrective measures to ensure the quality of service. An event may or may not be related to a disturbance. In storm analysis, an event often corresponds to a power restoration request in the form of a blackout repair ticket.

[0046] A "Disturbance-Revealing Event (DRE)" is a specific family of events that tend to occur more frequently in the presence of a disturbance than in the absence of one. Such events can be identified through knowledge of previous subjects or through statistical analysis. In the case of storm-related power outage analysis, one type of DRE would correspond to a power outage caused by a fallen tree, while another type would be a power outage caused by the need to remove a tree leaning against a power line. In the context of weather-related power outages, DREs are associated with service tickets referred to herein as storm-revealing tickets (SRTs).

[0047] A "prior-known disturbance" is a set of disturbances that were known before an event was assigned to them. In meteorological applications, these are known as "named storms" or "known storms."

[0048] "Prior-labeled events" are the events assigned to each disturbance. In weather-related examples, they represent the service tickets assigned to the storm before they are resolved.

[0049] The "Disturbance-Related Probability" is the probability of a DRE event being associated with a disturbance. In the case of a weather-related power outage, this is the probability that a given SRT is associated with a storm.

[0050] "Labeling" is the process of assigning a DRE (or SRT) to a disturbance. Such assignments can be deterministic (i.e., TRUE / FALSE) or probabilistic. In meteorological applications, system frameworks provide probabilistic labels that assign SRTs to either known or newly discovered storms.

[0051] "Verification" is the process of establishing the consistency of probabilistic labels based on a set of previously labeled events and the periods corresponding to known and newly discovered storms. Events and disturbances are often location-specific. For example, in a given storm blackout analysis, tickets are subdivided into sub-regions due to the ticket initiation process and the spatial coverage of the storm. In this case, local disturbances are defined as disturbances identified for a particular geographical location.

[0052] Weather-related power outages are a major cause of concern for utility companies. Figure 1 shows a general overview of a system framework 50 for probabilistic labeling of events, detection and identification of disturbances. The first processing stage, implemented by the computer system, is access to initial data 55 obtained from a trouble ticket database 60, which contains a set of data related to a stream of service tickets 70 for a specific area, including data related to service outages such as those caused by weather-related disturbances. These tickets may contain information that associates a particular power outage with an identified storm. Thus, at least two sets of data are required: tickets or logs and weather (or environmental data). The tickets may contain information related to a family of disturbances, some of the tickets know a storm ID (used for verification), and typically none of them are labeled as non-storm-related. Additional data included may include, but is not limited to, weather data such as pressure per hour, wind speed, gusts, and precipitation.

[0053] Further processing steps 75 implemented by a computer system include evaluating the quality of the SRT data, e.g., 1) checking for the percentage of missing labels, and 2) checking for tickets that are poorly labeled (e.g., having temporal or spatial (geographical) inconsistencies). Such quality decisions may be performed by validators who analyze the data and generate a quality report containing recommendations, or combinations thereof, with explanations, pass / fail ratings, scores, or any details or visualizations. A restorer may analyze the quality report, restore the data report, and generate a restoration report. Alternatively, an iterative process may be employed, along with a workflow that receives the input service ticket data, implements a knowledge graph to learn and configure constraints, generates a data profile, and presents the results in a generated quality report used by the restorer to apply constraints and restore the data in the restoration process.

[0054] Therefore, in Figure 1, when the SRT data quality is deemed unacceptable at 75 (e.g., not meeting the quality threshold), a repair step is then performed in accordance with the embodiment described herein. The repair step is performed at 80 and updates and sorts the dataset (i.e., identifies missing storms), assigns appropriate individual disturbances from events representing each disturbance, and provides probabilities to quantify the confidence of such assignments. Such a repair step efficiently fills in the missing labels (probabilistic labels) and identifies appropriate storms / storm periods. Probabilistic labels are generated, all missing storms are identified, and as a result the dataset is updated, increasing its fit for training a machine learning model.

[0055] Figure 2A shows a service ticket dataset 60 used in data quality analysis before remediation. In Figure 2A, the dataset 60 consists of utility service records (tickets) 70 corresponding to the handling of actual and predicted outages. Each record 70 is self-contained and contains multiple attributes. There is no limit to the amount of records in the dataset 60, and the dataset can contain hundreds of thousands of records covering multiple substations and spanning several years. As shown, each record 70 contains attributes used for analysis, including a ticket ID field 63, an estimated start time of the outage 65 which is timestamped at the ticket start time, and a valid substation field 68 which indicates the affected coverage area. Additionally included are a storm ID field 73, which is filled only when the outage was caused by a storm, a cause description field 75 which indicates the cause of the specific outage, and a field 78 which indicates the number of affected customers. There may be additional fields (not shown) corresponding to the end time, which include a timestamp filled at the end of the ticket. While the assigned storm ID may be included in the ticket, as shown in dataset 60, prior to the remediation, there were many storm-related ticket records (e.g., ticket 72) where this associated storm ID field was missing or empty. Furthermore, there were tickets where this field was incorrect. Therefore, as shown in dataset 60 in Figure 2A, supplementary and remedial measures are necessary with respect to these fields / records.

[0056] Figure 3 shows a computer system 100 for implementing the framework and a methodology for improving the performance of a trouble ticket dataset used for machine model training, specifically illustrating the association of service tickets with disturbance events according to the embodiments herein. In some embodiments, the system 100 may include a computing device, a mobile device, or a server. In some embodiments, the computing device 100 may include, for example, a personal computer, a laptop, a tablet, a smart device, a smartphone, or any other similar computing device.

[0057] The computing system 100 includes one or more hardware processors 152A, 152B, memory 150 (for example, for storing operating system and application program instructions), a network interface 156, a display device 158, an input device 159, and any other functions common to computing devices. In some embodiments, the computing system 100 may be any computing device configured to communicate with one or more websites 125, including a web or cloud-based server 120, over a public or private communication network 99, for example. For example, the exemplary embodiments described herein are in the context of a utility company providing services and managing trouble tickets that focus on weather-related disturbances, while the website 125 may include a database resource 130 that provides utility service records (tickets).

[0058] Since the framework of this specification is applicable to many contexts, database 130 may include trouble ticket data maintained by other companies in the context of disturbance management in other domains. Examples of such domains include technology (in the analysis of computer network overload and other phenomena occurring in QoS assurance), insurance (where disturbances are typically associated with an unusually large number of claims), or finance where events correspond to transactions.

[0059] Furthermore, as shown as part of system 100, a local memory 160 is provided that is useful for storing selected parameters or default parameters or both used in a data processing framework, such as storing and processing tickets and ticket subsets 161, or evaluating data quality, or both, as well as other metrics 162. Such a device may include a connected memory storage device 160 or a remote memory storage device (e.g., a database accessible via a remote network connection for input to system 100).

[0060] In the embodiment shown in Figure 3, processors 152A and 152B may include, for example, a microcontroller, a field-programmable gate array (FPGA), or any other processor configured to perform various operations. In addition, the shown communication channels 140 are wired connections, such as data bus lines, address bus lines, and input / output (I / O) data lines, which route signals between various components of the system 100. Processors 152A and 152B are configured to execute instructions in the manner described below. These instructions may be stored, for example, as programmed modules in an associated memory storage device 150.

[0061] Memory 150 may include, for example, non-temporary computer-readable media in the form of volatile memory such as random access memory (RAM) or cache memory or a combination thereof, or other such media. Memory 150 may also include, for example, other removable / non-removable, volatile / non-volatile storage media. In non-limiting examples only, memory 150 may include portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any preferred combination of the foregoing.

[0062] The network interface 156 is configured to transmit data or information to and receive data or information from the website server 120, for example, via a wired or wireless connection. For example, the network interface 156 may utilize wireless technologies and communication protocols such as Bluetooth®, WiFi (e.g., 802.11a / b / g / n), cellular networks (e.g., CDMA, GSM®, M2M, and 3G / 4G / 4G LTE), near-field communication systems, satellite communications, etc., via a local area network (LAN), a wide area network (WAN), or any other form of communication that enables the computing device 100 to transmit information to or receive information from the server 120.

[0063] The display 158 may include, for example, a computer monitor, a television, a smart television, or a display screen integrated into a personal computing device such as a laptop, smartphone, smartwatch, virtual reality headset, or smart wearable device, or any other mechanism for displaying information to a user. In some embodiments, the display 158 may include a liquid crystal display (LCD), an e-paper / e-ink display, an organic LED (OLED) display, or other similar display technology. In some embodiments, the display 158 may be touch-sensitive and may function as an input device.

[0064] The input device 159 may include, for example, a keyboard, mouse, touch sensor display, keypad, microphone, or any other similar input device or any other input device that can be used alone or together to provide the user with the ability to interact with the computing device 100.

[0065] As mentioned, the memory 150 of the computer system 100 further stores processing modules containing programmed instructions adapted to invoke actions to ensure more accurate labeling of asset malfunctions, such as power outages, as being caused by external disturbances, such as storms, and to substantially improve data quality so that the data can be used to train machine learning models, thereby enabling additional analytical techniques that result in better detection, prediction, and management of power outages.

[0066] In one embodiment, one of the programmed processing modules stored in the associated memory 150 includes a module 165 which provides computer-readable instructions, data structures, program components, and application interfaces for taking in, collecting, or processing or a combination thereof of trouble tickets 70 received from a trouble ticket database 130 or a similar data source.

[0067] Furthermore, the programmed processing module includes a disturbance ticket categorization module 170 which provides computer-readable instructions, data structures, program components, and application interfaces for preparing tickets for categorization, which helps (a) ensure the accuracy of the SRT and (b) identify mislabeled tickets.

[0068] The programmed processing module further includes a probabilistic labeling module 175, which is provided with computer-readable instructions, data structures, program components, and application interfaces for interacting with the SRT and automating the probabilistic labeling process of trouble ticket events as described herein. This module establishes a family of tickets representing storms (or, more generally, events representing disturbances) and performs a method for serving as a basis for the detection and diagnosis of meteorological storms (or disturbances). Further probabilistic labeling methods are performed to assign the probability of associated events to the type of disturbance. The approach involves a combination of statistical methodology, data analysis, and domain knowledge expertise. In embodiments, baseline process features using robust estimation methods and complements are obtained for a completion time range, then disturbance periods are identified and their features are contrasted against the baseline. Probabilistic labeling is then obtained using the form of the contrast.

[0069] Furthermore, the programmed processing module includes a disturbance identification module 180 provided with computer-readable instructions, data structures, program components, and an application interface for identifying the duration of disturbances, specifically by performing one or more size-calibrated change point detection methods to identify disturbance cycles. The processing provides separate modeling of non-disturbance (baseline) behavior, including baseline development using trimmed estimates corrected for constraints and biases, as described herein, and modeling of the ticket stream under disturbance (storm) conditions. This technique can be used for the efficient and automated generation of probabilistic labels (PLs) under conditions where disturbances do not dominate the dataset.

[0070] Furthermore, the programmed processing module includes a verification module 185 that provides computer-readable instructions, data structures, program components, and application interfaces, and provides a workable procedure for (a) rediscovering and processing labeling for “known storms,” and (b) discovering key parts of missing storms and labeling corresponding tickets. Verification in (b) at this stage may be assisted by human experts and may involve adjusting parameters to achieve customer-satisfactory metrics.

[0071] In one embodiment, a separate programmed processing module 190, stored in an associated memory 150 of the system framework 100, employs instructions that constitute a system for building and running a trained supervised machine learning model to correlate meteorological disturbances with specific power outages or meteorological conditions or combinations thereof, based on labeled data. For example, the framework addresses data quality issues by achieving SRT labeling to previous existing storms. However, often one primary objective of a utility company is to achieve the best possible predictive capability for power outages associated with upcoming storms. Based on data from previous storm events and the expected duration of the storm, the framework obtains a distribution of the number of tickets for various causes (for any given storm and sub-regions). The quality of the prediction depends on a policy of assignment probabilities for previously observed tickets. Combined with meteorological observations, the machine learning model is also trained to predict the number of affected customers, costs, or resources required to mitigate the effects of the storm within a timeframe of concern. For example, the trained model is then used to answer questions such as how infrastructure factors (number of utility poles, transformers, miles of cable) contribute to the cost of a blackout, how geographical features contribute, and how weather-related variables (precipitation, wind speed, gusts) affect the outcome.

[0072] This power outage prediction model 195 may include deep learning neural network (DNN) models such as convolutional neural networks (CNNs), graph CNNs, multilayer perceptrons (MLPs), or recurrent neural networks (RNNs). Deep learning models can be written to Python using the TensorFlow library.

[0073] Such machine learning models may be trained with the resulting datasets, including more accurately relabeled data, according to the methods herein, and the trained models may be used to forecast the disruption of weather-related services for a given set of weather conditions. Alternatively or additionally, machine learning models trained with such relabeled datasets may be run to facilitate long-term planning and resource allocation, making infrastructure more resilient to storms. For example, in the case of any approaching storm, the framework may suggest preventive measures to mitigate the consequences of the storm. For instance, when it has been found that utility pole fires are more likely to occur during storm periods, measures to remove potentially threatening debris from connectors before the storm may be helpful in this regard, and when leaning trees are the cause of an excessive number of tickets during a storm, utility companies may also take proactive measures to address this threat as well.

[0074] As further shown in Figure 3, the memory 150 includes a supervisory program 110 having instructions for configuring the computing system 100 to call various modules and to perform the methods described herein to “clean” trouble ticket data, i.e., to achieve probabilistic labeling of all trouble ticket data having disturbances (e.g., storms).

[0075] Figure 4 shows a method 200 performed by a supervisory program 110 that implements an automated task to (probabilistically) more accurately label trouble ticket data with disturbances (e.g., storms) performed by system 100. For example, Prob[Ticket is related to storm] = 0.8.

[0076] In method 200 of Figure 4, the first step 202 includes establishing a family of events (tickets) representing disturbances. That is, in embodiments, the framework implements an approach that relies on the existence of a family of tickets having specific characteristics: they tend to occur more frequently when disturbances are present than when they are absent. In exemplary embodiments herein, in the context of a utility company, the disturbance is a storm that causes the issuance of trouble tickets related to a power outage event. Thus, in 202, a family of tickets representing storms (Storm-Revealing Tickets, SRTs) is identified. In embodiments, the family is developed based solely on subjective knowledge. Experts may be consulted to create the SRT family, and independent sources and underlying scientific evidence may be analyzed. In embodiments, the family is determined based on observed characteristics (observed manually or from Internet of Things (IoT) sensors).

[0077] In an embodiment, the SRT family is selected based on domain knowledge, and this selection is then verified using statistical analysis. Figure 5 shows Table 90, which illustrates the characteristics of tickets representing storms in an exemplary embodiment. In an embodiment, for example, the SRT family 91 may be selected first based on domain knowledge, and this selection may then be verified using statistical analysis. As shown in Table 90, two columns 94 and 96 represent the first and second sets of tickets created, respectively, 1) the (percentage, order) of tickets with a pre-assigned storm ID, and 2) the (percentage, order) of all tickets. The first column 92 of Table 90 shows the cause description for an exemplary ticket family, for example, a tree falling onto a main power line, a tree falling onto a secondary power line, equipment malfunction caused by a storm resulting in material failure, equipment malfunction caused by a storm, or a tree or branch falling onto a power line. Based on the percentages shown in Table 90, considerable differences can be expected in the distribution of cause descriptions. To prevent signal dilution, the number of causes (representing storms) in a trouble ticket family is a priori limited, for example, as shown in the six causes in Table 90 of Figure 5.

[0078] Returning to Figure 4, the next step 205 involves checking the data quality of tickets issued during the disturbance. The check analyzes the labeling coverage of tickets belonging to a family of events representing the disturbance, and this coverage is an indicator of data quality.

[0079] Several significant data quality issues may arise in the ticket dataset described above. Some of these issues are, in principle, unavoidable and therefore represent a "missing data" problem rather than a quality problem in itself. The main data issues are: 1) Missing fields: In the database, SRTs lack storm ID assignments at a significant rate, even if they fall within a known storm time window. Using the techniques described herein, it has been statistically proven that the problem of missing labels exists and can be repaired through completion. Possible reasons for this problem include the inability to establish a causal relationship with the storm at the time the ticket is completed, or insufficient information about the storm itself. 2) Mislabeling: Non-SRTs can be assigned to known storms (and are actually caused by storms). However, the assignment may be applied very broadly, and in some cases, there may be no other reason than the ticket falling within a known storm time window. Here again, the existence of mislabeling can be statistically proven using the framework approach and repair described herein. 3) Missing Storms - Based on the analysis, the list of disturbances that are “known storms” is incomplete: there are storms of varying magnitudes that are not recorded in the ticket database. These storms generate identifiable clusters of SRTs, which often cover several sub-regions. For machine learning and any other reasoning activity, system 100 identifies such storms and assigns them the relevant tickets, thereby creating a database image with overall consistency of the impacts associated with the storms.

[0080] In an embodiment, at 205 in Figure 4, a data quality check is applied to 1) determine the percentage of missing labels and 2) check for missing labels that have temporal or spatial (geographical) mismatches. In an embodiment, the data quality check may involve creating a metric that defines the coverage of labeled events (tickets) included in a set of known disturbance periods and quantifying the metric value. In an embodiment, the metric is the ratio between the number of labeled SRT tickets included in the set of known disturbance periods and all SRT tickets included in the set of known disturbance periods. Subsequently, at 208 in Figure 4, a decision is made to determine whether the data quality is acceptable, for example, by determining whether a poor quality signal results when the coverage is below a threshold. When the data quality is deemed acceptable, the process may return to terminate the process, to process newly received trouble tickets, or both. Acceptable data quality ensures improved accuracy of deployed blackout forecasting models trained on such data.

[0081] Alternatively, returning to 208, if the data quality is deemed unacceptable, the process proceeds to 210, where the system performs a remediation according to the embodiments described herein to fill in the missing data values. Then, in 215, the method generates a remediation summary including 1) the number of tickets assigned or reassigned to distribution IDs, and 2) improvements in data quality. The results of the remediation and filling in of missing values ​​are re-evaluated, and a data quality check is rerun to determine whether further remediation is needed. If further remediation is needed, the steps in Figure 4 may be rerun.

[0082] In the basic recovery approach, SRT assignment is achieved in two phases. The first phase focuses on the time series of SRT tickets for individual sub-regions, which are typically identified by the name of their corresponding service substation. Using the time series, the storm-associated probability of an SRT is calculated for each day. This accomplishes the task of probabilistic labeling. By applying thresholds to the calculated probabilities, pre-existing storms can be associated with each SRT that has a high probability of representing a storm.

[0083] In an embodiment, the system performs a calibrated change point analysis to separate normal periods from periods of environmental disturbance (storms) in a first phase. Such separation is completed under the assumption that environmental disturbances significantly increase the probability of events resulting in losses for affected days. Next, the difference between the expected number of SRTs under non-storm conditions (reflecting the non-storm event rate) and the number of tickets observed for any identified storm period (actual event rate) is calculated. This difference provides a basis for calculating the probability of representing a storm.

[0084] In the second phase, the system improves the quality of ticket assignments to storm IDs by using a set of tickets for which storm IDs have already been assigned in the database. This information or knowledge can then be used to further validate the probabilistic labeling and model fit.

[0085] Figure 6 shows the overall system architecture 300 for identifying disturbances, probabilistic labeling, and assigning events (tickets) to previously known disturbances (known storms in a particular sub-region) using the framework 100 of Figure 3. As shown in the system architecture 300 of Figure 6, a sequence of all sub-region-specific trouble tickets 301 is taken. From this sequence of tickets, 302 the system generates a baseline daily rate of SRT for each year, month by month, and for each sub-region.

[0086] To generate a baseline daily rate from this data, the system performs the task of dividing the chronological order of the daily SRT counts into sets of time series corresponding to individual sub-regions. Figure 7 is Chart 350 plotting the time series of the period aggregated rate (SRT count) (Y axis) of the location-specific sub-regions over time (X axis). In embodiments, aggregation is performed on a weekly or daily basis. In addition to the basic assumption that an SRT family exists, it is assumed that the arrival of SRTs follows a non-homogenous Poisson process (NHPP) for all sub-regions. Two main sources of non-homogeneity exist. First, there are “normal” seasonal effects, including, for example, the effects of El Niño years, and even year-end or quarterly considerations that may affect planned ticket processing. Second, there are storm periods during which the SRT rate increases. From the type of data shown in Figure 7, the method can identify the baseline rate 360, and then the method can identify storm periods.

[0087] Returning to the system architecture 300 in Figure 6, in the baseline generation stage 302, it is assumed that the non-storm period provides sufficient information to estimate the daily baseline rate of the SRT, i.e., the rate that would have been observed if there were no storms. The baseline rate is location-specific and incorporates the seasonality coefficients described above. Incorrectly pre-labeled tickets form a small portion of the overall pre-labeled tickets. Subsequently, in 305 in Figure 6, the calibrated change point algorithm is implemented to identify local storms (i.e., identify the scale, lifetime, and dynamic status (start, intermediate, and end) of the disturbance). At such a time, the storm ID dictionary generator process 315 is further invoked to record new storm IDs for newly identified events (e.g., storms) for a particular location or sub-area. Table 400, shown in Figure 8A, summarizes the notation and exemplary default values ​​used in the change point analysis. Multiple time series used in these analyses are summarized in Table 402 in Figure 8B.

[0088] For baseline generation, the system displays the baseline rate of the SRT's NHPP as λ(t). For example, over a given period such as one month, the system generates the daily count of SRT tickets using a random variable vector X:X={x i It is obtained as}.

[0089] Here, index i is, for example, a date within one month. In embodiments, estimation is applied to months with complete one-month data (thus the available date indices are i=1,2,…,D, where D≧28). Furthermore, the system indicates the average daily rate (under non-storm conditions) for the month under consideration by λ. One objective is to obtain a robust estimation based on X.

number

[0090]

number

[0091] 1) Set the trimming number r (e.g., select the default, r=10) to remove the daily rates of the highest and lowest r from the monthly data. A trimming number r=10 is understood to establish a breakdown value of the estimate: it can handle up to 10 storm days in any given month without generating bias induced by outliers. Generally, the trimmed mean gives an unbiased estimate of the central tendency for a symmetric distribution. The Poisson distribution is skewed to the right, and therefore conventional trimmed estimates are biased downward overall, as points removed from the top have a greater influence than points removed from the bottom. This bias is limited and tends to limit b as λ→∞.

[0092] 2) By averaging the remaining data points in X, the trimmed average is obtained.

number

[0093] 3) Apply an additional adjustment coefficient (bias correction) b (for example, b=0.15),

number

[0094] 4) The robust estimate is corrected by preventing it from becoming too small. For this purpose, a lower threshold β1 is applied (e.g., β1 = 0.2), where, on average, under non-storm conditions, at least one SRT is expected every 5 days for every service sub-region.

number

number

[0095] Further separation of storm periods is performed using baselines available for each month of each year and for all sub-regions. To achieve this goal, all daily rates are standardized, and as a result they form a sequence of variables that are slightly distributed with a mean of 0 and a standard deviation of 1 under non-storm conditions. For a given month, the standardized daily rates

number

number

[0096] Here, i = 1, 2, ..., D,

number

number

number

[0097] In an alternative embodiment, instead of calculating the daily rates under non-storm conditions for each month and assigning those values ​​to each day of the month, a sliding window may be used in which the same operation can be performed and the baseline is calculated as a moving average. That is, the procedure is applied to a sliding window (e.g., of length = D days) in which the same operation can be performed. The baseline value calculated based on this window is assigned to the day i = the midpoint of the window. Thus, sliding the window provides a new value to the baseline curve.

[0098] Returning to step 305 in Figure 6, further identification and isolation of storm periods are performed. Baseline values ​​for each month are created and converted into a standardized time series of scores (1) so that the system is positioned for localized storm identification. In embodiments, a calibrated change point algorithm is used to identify localized storms. One goal is to automate the labeling of tickets as storm-related or non-storm-related. As shown in Figure 9, a storm has a lifecycle that includes three states: start (S), middle (M), and end (E). For any given day, as shown in Figure 9, there exists a corresponding state selected from the set [N,S,M,E], where N represents a normal (non-storm) day. Referring back to Figure 2A, a record 70 corresponding to a single ticket may include additional storm state attributes selected from the set [N,S,M,E].

[0099] For a time series of scores corresponding to a given sub-region, multiple storms may be observed. These result in unusually high scores (relative to a standardized baseline score of 0) and persist throughout their respective periods. To organize an efficient process for identifying storm periods, one criterion for declaring a “storm” condition is when the mean of the sequence of scores in equation (1) deviates from the nominal level of 0–1, i.e., when the sequence underlying the daily rate is shifted by one standard deviation. If 0 is specified as the acceptable level for the mean of the process of scores, the next thing to apply is a cumulative sum (CUSUM) control scheme that triggers an alarm when the onset of a storm is detected. A one-sided CUSUM process suitable for detecting the upward process mean requires transforming the sequence of scores (1), represented by {y(i)}(i=1,2,…), into a sequence of control scheme values ​​{s(i)}(i=1,2,…) using a process that follows equations 2a, 2b as follows:

number

number

[0100] Furthermore, the signal is triggered at a first time i where s(i) > h, where h > 0 is a appropriately selected control limit. In an embodiment, s0 = 0. However, it can be any other value greater than 0. In the above equation, k is a reference value, such as an anchor, and is usually selected midway between an acceptable level and an unacceptable level, in one case being as follows:

[0101]

number

[0102] for example,

number

[0103] In the embodiment, a threshold h is selected to achieve the desired trade-off between the false alarm rate and sensitivity. In the embodiment, the value h may be equal to 5. However, since the score Y in equation (1) shows some positive distortion and a continuous correlation, a higher threshold is required to achieve reasonable protection against false alarms. Therefore, a somewhat higher threshold, h=6, is used in the embodiment.

[0104] FIG. 10A shows an exemplary CUSUM control plot 500 of calculated CUSUM values {s(i)} plotted against time, showing a manner of applying a calibrated change point algorithm used to identify the boundaries of local disturbances (e.g., storms). In an embodiment, as shown in Table 400 of FIG. 8A, a CUSUM time series is calculated based on Y and two parameters (k = 1 and h = 6). In the case of a substation, the algorithm starts from the start date of the recorded ticket, and the system creates a CUSUM time series of values 501 (i.e., S = {s i}) for disturbance identification and identifies the initial indicators of the storm. Using the knowledge of the CUSUM time series and the identified storm indicators, the system establishes the start and end dates of each storm period.

[0105] As shown in the calculated CUSUM value control plot 500 of FIG. 10A, for a known storm indicator corresponding to date T510, the system first defines the neighborhood of the storm indicator as [T - m, T + n] as shown along the x-axis. s T-m The CUSUM value 505 of s = 0 is the last zero value scheme point before date T510. The value s T+d = 0 is the first zero value scheme point 520 after date T (e.g., here 0 ≦ n < d). The case of n = 0 only exists when d = 1. Considering equation (2b), the CUSUM value s i does not creep below the value of 0 under non-storm conditions as shown in FIG. 10A. When the value of Y becomes systematically greater than 1 during storm conditions (e.g., Y ≧ 2), the CUSUM value s i 501 increases. In an embodiment, as shown in FIG. 10A, when the CUSUM value exceeds the threshold value h at time T at 510, a declaration of storm conditions is triggered. In an alternative embodiment, the start point of the disturbance at that time can be determined based on additional information (e.g., additional information about wind speed) that can function after a certain amount of additional time has elapsed, or the evaluation of the start point can be re-evaluated as the situation develops.

[0106] The system reliably detects the first storm and its origin point in the time series in order to declare the "end of the storm," and thus the state s as shown in Figure 10A. i While enabling the generation of both start and end points for any storm window resulting in the labeling of a storm, a modified CUSUM procedure is implemented that, among other things, includes the automatic restart of the CUSUM process (2a, 2b) at any point in the signal. That is, in the case of CUSUM calculation, this current CUSUM value for T will be reset to zero for the next date point. In an embodiment, in Figure 10A, a restart occurred at point T510 of the storm declaration, but no other restarts occurred before the declaration of “end of episode” at point 520. At that point, T+n515 was declared as the last day of the storm.

[0107] In the modified CUSUM process, the procedure for identifying the storm period follows equation (3a, 3b) as follows:

[0108]

number

[0109]

number

[0110] In Equation (3b), the first event of exceeding the threshold (h) is considered as a signal that a storm has started in the sub-region under consideration. Here, 0 ≦ s0 < h. Therefore, the process s(i) generates cycles (all ending at the exceeding of the threshold h), and several cycles may exist as long as the disturbance continues. Eventually, there is a cycle 525 where the value s(i) reaches 0 before reaching the threshold, and at this point, the system declares that the storm has ended somewhere in this cycle. Therefore, this will be the decision point for "the end of the episode" (e.g., time T + d). Next, the method starts searching for the point within the last cycle where the storm is considered to have actually ended. In FIG. 10A, this is time 515.

[0111] That is, after the first event of exceeding the threshold (h) as a signal that a storm has started in the sub-region under consideration, subsequent threshold violations (if reasonably occurring immediately after the preceding violation) will serve as an indicator of the continuing storm. Eventually, the temporal cluster of threshold violations stops, and this will function as a criterion for declaring a storm episode. At this point, the storm boundary may be declared. Specifically, for a known storm indicator corresponding to date T, therefore, the storm period is the temporal neighborhood [T - m, T + n]. The CUSUM value of s(T - m) = 0 is the last zero-value scheme point before T. Making s(T + d) = 0 the first zero-value scheme point after the discovery of the storm at time T and corresponding time (T + n) 515 to the maximum CUSUM value in the period P[T, T + d] 525, then, since the CUSUM trajectory indicates that the tick rate after that time matches the baseline process, the storm end date is defined as (T + n). Therefore, the length of the identified storm is l = m + n + 1.

[0112] In one embodiment, as long as the storm persists, the CUSUM value will tend to increase and will exceed the threshold value for k=1. Thus, the CUSUM value will tend to increase until it exceeds the threshold (an alarm is triggered). During such a time, the s(i) value is reset after the alarm is triggered. In one embodiment, it may be reset to a value s0 between 0 and the threshold h. Thus, the CUSUM process and the resetting of the s(i) value will continue as long as the storm conditions persist and their values ​​remain in an increasing state above the threshold. Resetting s(i) after the alarm is triggered may occur multiple times (due to the reset) if the disturbance conditions persist. Once the score Y becomes small, the trajectory will begin to decline to a time such as time T+d520, and the criterion for declaring the end of the storm disturbance period is time T+n515, after which the value will tend to decrease.

[0113] In an alternative embodiment, considering the right end point of the disturbance window, the decision may be made at time T+d (where point T+n is then identified as the preferred end point). However, several alternative policies exist. For example, additional data may be collected before determining that the disturbance episode has ended.

[0114] For example, in the first alternative instance, the estimation of the disturbance boundary is based on the variables

number

number

number

number

[0115] In the second alternative instance, the estimation of the disturbance boundary is given by s(i) and s with variables u. * (i) is based on the use of a "twin" process, that is, after a reboot in s0, process s(i) of equation (2b) may be executed. However, as shown in Figure 10C, s * (i) In addition to calculating the value 511, the system further calculates the current cycle (s * max ) s in 545 * (i) The maximum value and s* (i) Calculate the additional difference between (s * max )-s * If (i) > u (where u > 0 is one of several preferred thresholds), then the end of the disturbance episode may be declared, and the method may be to search for the maximum value of s(i) in Figure 10B to establish the end point. Here, time 541(T+d1) is the “end of episode” determination point 540 where the disturbance ended in the second cycle 535, and becomes the last point if time T+n515 is established. On the other hand, it should be noted that s(i) may run as usual, exceeding h, and a new cycle may be triggered at the restart point 512.

[0116] In a third alternative instance, the estimation of the disturbance boundary is s(i) and s with importance level threshold α. * (i) is based on the use of a "twin" process, that is, after a reboot of s0, the system s * max Monitor the position of s * max The system evaluates the point y(i) corresponding to the subsequent process. The "end of episode" determination is made only if these points show a sufficiently low measurement of the central trend (e.g., mean), as confirmed by statistical testing. Next, the system searches for the maximum value of s(i) as shown in Figure 10D and may establish the end point. That is, to search whether the "end of episode" should be declared at time 542T+d2, the system searches the data segment SEG as shown in Figure 10D. d In addition to extracting data y(i) corresponding to 550, for example, 1) Hypothesis H0: Segment SEG d The average of y(i) within 550 is the segment SEG d For every point in the comparison between internal and alternative, a hypothesis is made that the baseline target area is within a certain level of confidence, and Hypothesis H1: Segment SEG d1) Test the hypothesis that the mean of y(i) is within the region corresponding to the disturbance for at least one point within 550, and 2) determine whether H0 cannot be rejected at some predetermined level of importance value α, then declare the “end of the episode” and establish T+n as the last point of the disturbance, thus ending the cycle (i.e., returning to the baseline condition), thereby performing a statistical test to confirm the “end of the episode” with a certain level of confidence. In another exemplary embodiment, this may be stylized as follows: Hypothesis H1 for all subsegments:μ Y The last data segment SEG compared to ≤ε d Given test hypothesis H0:μ for several subsegments in Y >ε≧0. ​​When H0 can be rejected at importance level α, then “end of episode” is declared, T+n is established as the end point, and the cycle ends. It will be understood that other measures of central tendency may be used instead of the mean.

[0117] Generally, other factors, including external information, may be considered in determining the end of a disturbance episode. However, generally speaking, it is the maximum value of s(i) in the final cycle that determines the end point of the disturbance. This point, adjusted to the right, may include other points where an elevated score y(i)>0 was observed.

[0118] In this embodiment, estimating the disturbance start point Tm as the first point of the signal trigger trajectory introduces a positive bias. This is because the data value Y(i) is assumed to be an increased rate, for example, aY(i)>μ0, or b. The disturbance hypothesis is supported against the baseline (T+d, s * max (And a similar process to the one used to establish T+n can be used, but it moves to the left.) As long as this hypothesis is supported, it can be addressed by extending the starting point to the left by including additional points (sequentially).

[0119] In the embodiment, in the case of dynamic boundary adjustment, the system is not obligated to set a starting point at detection time T. That is, the disturbance boundary may be adjusted as new information (e.g., new information corresponding to observed tickets) comes in at a given time in a month, and through the examination of time series of baselines and seasonal effects, a reassessment of the baseline for that month one year prior may occur. Analysis using the revised baseline may then lead to a revision of the storm boundary for subsequent observations.

[0120] Further improvements are possible based on region-specific disturbance patterns. For example, in the case of storm disturbances, it may be known that the effect appears within a short time but gradually diminishes.

[0121] Furthermore, in a further embodiment, covariates may be incorporated into the method, for example, through baseline adjustment.

[0122] Figure 11 shows an exemplary pseudocode 600 of the calibrated change point method, where time points are taken as shown in Figure 10A to establish the storm boundary. As shown, after initializing s_previous=s(0)=s0, a FOR loop is entered at 601, and s(i) is recalculated sequentially at 610 for each index i (day of the month) as a function of the difference between the y(i) score and k reference values, without decreasing to a value below 0, as in equations (3a) and (3b). At 620, a decision is made as to whether the s(i) value exceeds the threshold h. If the s(i) value does not exceed the threshold h, then s(i) takes its previous value at 625. At such a time the threshold is exceeded, the storm condition is declared, the procedure is restarted, and s_previous may be reset to s0 at 622. However, in an alternative embodiment, when the threshold is exceeded at 622, the CUSUM value s_previous can be assigned a value somewhere between 0 and the threshold h. Thus, the reset of s(i) in the case (s>h) does not necessarily have to be restarted with the value 0, but it does have to be restarted with an intermediate value s0 between 0 and the threshold h.

[0123] Returning to Figure 6, after identifying storms in subregions (e.g., localized storms) and establishing the duration of the storms using the CUSUM process S={s(i)} in 305, the next task in 310 is to provide the storm-associated probabilities of the SRT corresponding to the declared storm window. In embodiments, a model-driven assignment formula is used for events, assigning the probability of an event to be associated with one of the disturbances. To provide the probability that the SRT is associated with a storm, a fundamental property of the NHPP process is that it itself has its own intensity function {λ j Considering that it is a superposition of NHPP subprocesses driven by (t) (where j=1,2,...J), then the probability that an event occurs, if it occurs, is that it occurs by subprocess j, λ j It is given by (t) / λ(t), where λ(t) is the total intensity. The probability P that SRT emitted on day i is related to the disturbance and can be calculated according to equation (4) as follows.

[0124]

number

[0125] Here,

number

[0126] Furthermore, in Figure 6, after assigning probabilistic labels to SRTs in 320, the next step is to assign SRTs to known storms. This can be achieved using a derived dataset of known storms. For any known storm ID, this dataset provides the total number of tickets assigned to it, the estimated start and duration of the storm, and the number of affected sub-regions. Using the probability (p) of the storm calculated for each SRT, any given ticket can be associated with a known storm as follows. Given that the duration of the storm is [T1, T2], the set of SRTs belonging to this storm is then defined according to equation (5) as follows.

[0127]

number

[0128] Here, (ε1,ε2) provides an uncertainty-related buffer and default to (1,1) and defines an extended storm period, as shown in Table 400 of Figure 8A, and p0 is a probability threshold. In this embodiment, the value p0 = 0.5 is a preferred implementation of the probability threshold.

[0129] Furthermore, returning to Figure 6, in 325, after assigning the SRT to a known storm, the next step is to verify the results of the probabilistic labeling and the storm identification procedure. In embodiments, a human expert may be used to verify whether the detected storm should actually be classified as a storm and whether the high storm-related probability assigned to the ticket is guaranteed.

[0130] In the embodiment, a workable procedure is implemented that includes (a) rediscovering and processing labeling for known storms, and (b) discovering the main parts of missing storms and labeling the corresponding tickets. Verification of (b) can be achieved with the help of human experts and by adjusting the parameters in Table 400 and Figure 8A. That is, manually adjusting the parameters listed in Table 400 in Figure 8A is performed to achieve metrics that satisfy the customer.

[0131] Figure 12 shows a subset of data used in the validation process, specifically Figure 700 which shows the categorization of tickets for public utility companies. Figure 13 shows the main metrics 710 used in the validation process. Of high importance is the subset H715 of SRTs with high storm association probabilities, i.e., p≧p0 for some threshold p0. H is used to validate probabilistic labeling. The validation process can identify subsets of H containing tickets with missing storm IDs and find matching storm IDs using the method of equation (5). Next, we focus on subsets of H that cannot be associated with any known storm. This subset allows for an assessment of the degree of completeness of the list of known storms and further helps in identifying and characterizing missing local storms. Tickets in set L can be used for a fully automated part of the validation process. SRTs in L satisfy the properties i) having a probability p≧p0 representing a high storm and ii) having labels assigned based on (5). This part of the verification is based on a comparison between L and V (i.e., the entire set of pre-labeled SRTs). In an embodiment, first,

number

[0132]

number

[0133] Here, the divergence measure is

number

number

[0134] In one embodiment, to declare the procedure successful, a low value

number

[0135] Therefore, regarding the stage of labeling SRTs with missing storm IDs, Figure 13 is a chart showing the key metrics of the verification process. As shown in Figure 13,

number

[0136]

number

[0137] The analysis using the default parameters in Table 400 of Figure 8A is a newly discovered storm-related ticket associated with known storm periods.

number

[0138] Finally, there is another class of newly discovered storm-related tickets with a high storm probability p≧p0, but which cannot be associated with any known storm using the process of equation (5). This subset of SRT is G as shown in Figure 13. b As shown by 725, the ability to find missing storms and associated tickets is thus characterized by the following equation (8):

[0139]

number

[0140] This analysis is,

number

[0141] Subsequently, the service tickets are relabeled based on the generated probability and whether they fall within a known disturbance period. That is, as a result of the process in Figure 6, as shown in Figure 2B, the system outputs a revised record that includes a probabilistic label along with information about the detected storm. That is, as shown in Figure 2B, the system returns an exemplary record 90 that includes additional pre-filled fields, including a field 93 into which the identified storm with the substation and corresponding time is entered, a storm status field 95 containing the display from the set [N,S,M,E], a calculated probabilistic label (e.g., a value of 0.95) shown in field 96, and a known storm ID field 98. As shown in some exemplary returned records 90, high-probability tickets not associated with any known storm that are not indicated by any known storm ID label 97A and high-probability tickets associated with a known disturbance (storm) with the storm ID label shown in 97b are found.

[0142] Figure 14 is a flowchart showing a process 800 that may be executed by the processor to implement a preventative measure using a machine learning model according to the embodiment. The exemplary process may include one or more actions, measures, or functions, as indicated by one or more blocks. Although shown as individual blocks, various blocks may be divided into additional blocks and, depending on the desired implementation, combined into fewer blocks, removed, or executed in parallel.

[0143] Processing may begin in block 802. In block 802, the processor may receive an improved dataset containing records with probabilistic labeling obtained using the process described herein from a data source corresponding to a particular domain. Processing may continue to block 804. In block 804, the processor may train a machine learning model to correlate the effects of disturbances on one or more assets. The machine learning model may be capable of generating forecasts related to performance metrics of one or more assets based on the trained machine learning model. In one example, the machine learning model may be a neural network with hidden layers.

[0144] Processing may continue from block 804 to block 806. In block 806, the processor may receive an input indicating a predicted disturbance (e.g., a storm) or weather condition / pattern. Processing may continue from block 806 to block 808. In block 808, the processor may run a machine learning model to generate a forecast related to the asset's performance metrics or to a specific impact on the asset when considering the forecast disturbance, weather condition, or pattern. In some examples, the processor may further run a machine learning model to generate an interpretation of the forecast, where the interpretation may indicate its importance to the service company's assets.

[0145] In embodiments, for the probabilistic labeling of disturbance-related events, the example is taken from the management domain of weather-related power outages to illustrate the problem. For example, environmental disturbances (e.g., weather storms, blizzards, and electromagnetic hazards) can significantly affect the service capacity and quality of infrastructure systems (utility systems, sensor networks). However, the scale of environmental disturbances depends on location-specific weather or electromagnetic conditions (e.g., wind speed, gusts, total precipitation). Furthermore, the robustness of the infrastructure plays a crucial role in whether services are interrupted or devices are damaged by such environmental hazards.

[0146] In non-limiting exemplary embodiments, the general framework provided herein uses location-specific maintenance or fault tickets in conjunction with meteorological data (or monitored electromagnetic data) to identify environmental disturbance events and to use them for future planning and infrastructure improvements (e.g., underground cabling, electromagnetic shielding mechanisms, etc.).

[0147] As described herein, labeling SRTs to previously existing storms addresses data quality issues; however, in many cases, the primary objective of utility companies is to achieve the best possible predictive capability for power outages associated with upcoming storms. Based on data from previous storm events and the expected duration of the storm, the system obtains a distribution of the number of tickets for various causes (for any given storm and sub-region). The quality of the prediction depends on the assignment probability policy for previously observed tickets. When combined with meteorological observations, the model can also be used to predict the number of affected customers, costs, or resources required to mitigate the effects of the storm within a timeframe of concern.

[0148] The general framework provided herein further facilitates long-term planning and resource allocation to ensure infrastructure is more resilient to storms. For example, in the case of any approaching storm, a trained model can be used to suggest preventative measures to mitigate the storm's consequences. For instance, if it has been found that utility pole fires are more likely to occur during storms, taking measures to remove potentially threatening debris from connectors before the storm hits could be helpful in this regard, and if leaning trees are the cause of an excessive number of tickets during a storm, proactive measures can also be taken to address this threat.

[0149] The general framework further handles disturbances such as accident tickets, maintenance, and spikes in QoS log data. In one embodiment, the processing results of the general framework enable utility companies to always be prepared for weather storms (disturbances). The method involves automatically assigning appropriate storm IDs (e.g., disturbance IDs) to all tickets, and further, if a set of data quality issues sharing a common abstraction exists, the general framework implements a method for assigning disturbances as the cause of tickets or QoS log events.

[0150] Therefore, in the embodiment, the system 100 and the probabilistic labeling framework for utility data resulting in automatic labeling of blackout tickets and identification of storm periods may ultimately lead to answering questions of the following types: how many storm-related tickets are expected by substations during each period; how do infrastructure factors (number of poles, transformers, miles of cable) contribute to the cost of blackouts; how do geographical features contribute; and what effects do weather-related variables (precipitation, wind speed, gusts) have?

[0151] While the framework is common, numerous applications exist that have similar data structures and are driven by similar probabilistic models. Therefore, the analysis can be expanded to cover a broader range of model and system functions. For example, modeling the service ticket arrival process as an NHPP can be expanded to cover a range of possible components, covariates, and dependencies, once a sufficiently rich and reliable data source is available. Specifically, in the problem of managing weather-related power outages, additional data reflecting meteorological conditions in the affected sub-region can be utilized, such as wind speed and direction, gust intensity and duration, barometric pressure measurements, or precipitation, often available from weather stations located close to the service sub-region. However, the task of incorporating such data is quite complex and remains a subject of ongoing research.

[0152] A set of applications arises in the deployment, management, and maintenance of infrastructure companies. In the case of power operators described herein, it can be adapted to address the problems of companies managing transportation networks, water resources, or construction projects. In manufacturing, the methodology can be used to manage sensor networks deployed by IoT to serve the chemical and related industries.

[0153] Other possible application areas include, but are not limited to, use cases such as technology (e.g., in the analysis of computer network overload and other phenomena occurring in QoS assurance), insurance claims (where disturbances are typically associated with an unusually large number of claims), or financial services where events correspond to transactions. These use cases share similar challenges and require solutions involving a system framework that implements algorithms to correlate tickets or log events with disturbances.

[0154] Therefore, in an exemplary use case of computer services, the system receives data records (i.e., tickets or log events) including service QoS logs and tickets for services (e.g., service delays, I / O errors, and service failures). Metrics include a) router network traffic, b) storage latency, c) CPU utilization, d) memory utilization, e) service traffic, and f) TTL (Time To Live). Specific disturbances may include, but are not limited to, network attacks, expected or unexpected spikes in service activity, web service attacks, network failures, CPU overuse, intensive I / O errors due to storage malfunctions, and internet route failures (communication delays). Problem statements addressed using such a framework may include: 1) how to associate QoS tickets with known disturbances, and 2) identifying and classifying disturbances based on a stream of tickets or event logs.

[0155] In an exemplary use case for an insurance application, the system receives data records (i.e., tickets or log events) including service requests such as claims, registrations, and cancellations of life and term insurance. Specific disturbances may include, but are not limited to, pandemics (e.g., (coronavirus)), economic crises, and changes in the local economy (e.g., business or factory closures or openings). Problem statements addressed using such a framework may include: 1) identifying the root cause of a service request; identifying the impact of such a cause; and 2) identifying unknown disturbances affecting the business.

[0156] In an exemplary use case of a financial application, the system receives data records (i.e., tickets or log events) that include requirements for buying or selling specific stocks or adjusting a portfolio, but are not limited to those mentioned. Specific disturbances may include, but are not limited to, positive or negative news, and market disturbances from international, domestic, political, or economic events. Problem statements addressed using such a framework may include: 1) linking the motivation for financial activity to the requirement through the association of disturbances, and 2) identifying the undiscovered reasons for unusual financial activity (e.g., hidden disturbances).

[0157] In an exemplary use case for a help desk service, the system receives data records (i.e., tickets or log events) that include, but are not limited to, tickets (e.g., a service company provides a call service; for example, each call is tagged with a new patch applied to an existing window deployment; a particular disturbance could include, but is not limited to, an unusually concentrated stream of incoming tickets indicating that many of them are related to a common cause), new software applications, service failures, and new patches applied. Problem statements addressed using such a framework might include: the help center needs to understand the cause of each ticket (time and nature); therefore, to improve operational efficiency, it is necessary to identify and classify disturbances based on data.

[0158] Figure 15 shows a schematic diagram of an exemplary computer or processing system that may implement probabilistic labeling of service tickets in one embodiment of the present disclosure. This computer system is merely an example of a suitable processing system and is not intended to imply any limitation on the scope or functionality of the embodiments of the methodology described herein. The processing system shown may operate in the environment or configuration of many other general-purpose or dedicated computing systems. Examples of well-known computing systems, environments, configurations, or combinations that may be suitable for use with the processing system shown in Figure 6 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, supercomputers, and distributed cloud computing environments that include any of the above systems or devices.

[0159] A computer system may be described in the general context of computer system executable instructions, such as program modules, that are executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform a specific task or implement a specific abstract data type. A computer system may run in a distributed cloud computing environment where tasks are performed by remote processing devices linked over a communication network. In a distributed cloud computing environment, program modules may reside in both local computer system storage media and remote computer system storage media, including memory storage devices.

[0160] The components of the computer system may include, but are not limited to, one or more processors or processing units 12, system memory 16, and a bus 14 connecting various system components, including the system memory 16, to the processor 12. The processor 12 may include modules (e.g., machine learning modules) that perform the methods described herein. The modules may be programmed into the integrated circuit of the processor 12, or they may be loaded from the memory 16, a storage device 18, or a network 24, or a combination thereof.

[0161] Bus 14 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of the various bus architectures. Examples, but not limited to, such architectures include the Industry Standard Architecture (ISA) bus, the Microchannel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0162] A computer system may include various computer system-readable media. Such media can be any available media accessible by the computer system and may include both volatile and non-volatile media, as well as removable and non-removable media.

[0163] System memory 16 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM), cache memory, or a combination thereof, or other such media. The computer system may further include other removable / non-removable volatile / non-volatile computer system storage media. As merely an example, a storage system 18 may be provided for reading from and writing to a non-removable non-volatile magnetic medium (e.g., a “hard drive”). Not shown, a magnetic disk drive may be provided for reading from and writing to a removable non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive may be provided for reading from and writing to a removable non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such cases, each may be connected to the bus 14 by one or more data medium interfaces.

[0164] The computer system may communicate with one or more external devices 26, such as a keyboard, pointing device, or display 28; one or more devices that allow a user to interact with the computer system; or any device (e.g., a network card, modem, etc.) or a combination thereof that allows the computer system to communicate with one or more other computing devices. Such communication may occur via an input / output (I / O) interface 20.

[0165] Furthermore, the computer system can communicate with one or more networks 24, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via the network adapter 22. As illustrated, the network adapter 22 communicates with other components of the computer system via the bus 14. It should be understood that other hardware components or software components, or combinations thereof, can be used in conjunction with the computer system, although these are not shown. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0166] The present invention may be a system, method, or computer program product or combination thereof in an integration of any possible level of technical detail. The computer program product may include a computer-readable storage medium (or multiple mediums) having computer-readable program instructions for causing a processor to perform an aspect of the present invention.

[0167] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may, for example, be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any preferred combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes, namely, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures recording instructions, and any preferred combination thereof. The computer-readable storage media used herein should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0168] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0169] The computer-readable program instructions that perform the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, the one or more programming languages ​​including object-oriented programming languages ​​such as Smalltalk®, C++, etc., and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or on a server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or it may be connected to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, to carry out aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute a computer-readable program instruction by customizing the electronic circuit using state information of the computer-readable program instruction.

[0170] Aspects of the invention are described herein with reference to flowcharts or block diagrams or combinations thereof of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram or combination thereof, and each combination of blocks in a flowchart or block diagram or combination thereof, can be implemented by computer-readable program instructions.

[0171] By providing these computer-readable program instructions to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device and manufacturing a machine, instructions executed via the processor of the computer or other programmable data processing device may create means for implementing functions / operations specified in one or more blocks of a flowchart or block diagram or a combination thereof. Furthermore, these computer-readable program instructions may be stored in a computer-readable storage medium, which can instruct a computer, programmable data processing device, or other device or a combination thereof to function in a specific manner, resulting in the computer-readable storage medium containing the instructions containing a product that includes instructions for implementing modes of functions / operations specified in one or more blocks of a flowchart or block diagram or a combination thereof.

[0172] Computer-readable program instructions may be loaded onto a computer, other programmable data processing device, or other device to execute a series of operational steps on the computer, other programmable device, or other device, thereby generating a computer implementation process, so that the instructions executed on the computer, other programmable device, or other device implement a function / operation specified in one or more blocks of a flowchart or block diagram or a combination thereof.

[0173] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this context, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may be performed in an order different from that shown in the diagram. For example, depending on the related functions, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may be executed in reverse order. It should also be noted that each block in a block diagram or flowchart, or any combination thereof, and any combination of blocks in a block diagram or flowchart, or any combination thereof, may be implemented by a dedicated hardware-based system that performs a specified function or action or executes a combination of dedicated hardware and computer instructions.

[0174] The technical terms used herein are solely for the purpose of describing specific embodiments and are not intended to limit the invention. Where used herein, the singular forms "a," "an," and "the" are intended to include the plural form unless otherwise clearly indicated in the context. Furthermore, it will be further understood that the terms "comprises" or "comprising," or both, where used herein, specify the existence of a particular function, integer, stage, operation, element, or component or combination thereof, but do not exclude the existence or addition of one or more other functions, integers, stages, operations, elements, components or groups thereof or combination thereof.

[0175] All means or steps and (if any) corresponding structures, materials, actions, and equivalents of functional elements in the following claims are intended to include any structures, materials, or actions for performing a function in combination with other claimed elements, as specifically claimed. The description of the invention is presented for illustrative and explanatory purposes, but is not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments are selected and described in order to best illustrate the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in terms of various embodiments with various modifications to suit specific possible uses.

Claims

1. Memory and Processor including hardware Equipped with, The processor is configured to communicate with the memory, and the processor is configured to communicate with the memory. Receiving a set of service records from a data source that includes one or more service records corresponding to events representing multiple disturbances in a specified area occurring over a certain period, and one or more service records that do not have labels associated with mislabeled or associated disturbances, To determine the observed actual event rate of events representing disturbances during the aforementioned period, To determine the periodically aggregated rate of the baseline average value of expected service-related records under non-disturbance conditions for the aforementioned period, Determine a set of standardized, periodically aggregated scores as a function of the periodically aggregated rate of the baseline mean and the rate of observed actual events during the period, Identifying disturbance time windows based on changes in standardized, periodically aggregated scores detected during the aforementioned period, To generate disturbance-related probabilities for the service record corresponding to the identified disturbance time window, Based on the generated disturbance-related probabilities, the service ticket is relabeled as being related to the identified disturbance time window. A system configured to perform the following actions.

2. To determine the periodically aggregated rate of the baseline average value of expected service-related records under non-disturbance conditions for the aforementioned period, the processor further: Obtain a vector of random variables containing a large number of periodically aggregated rate values ​​from the aforementioned service-related records, The vector is trimmed by removing a predetermined number of the highest periodically aggregated rate values ​​and a predetermined number of the lowest periodically aggregated rate values ​​from the predetermined period. By averaging the remaining data points in the aforementioned vector, a periodically aggregated rate estimate of the trimmed average is calculated, Applying an additional bias adjustment factor to the trimmed mean estimate, By applying a lower threshold, the trimmed mean estimate is prevented from falling below the lower threshold. The system according to claim 1, configured to perform the following:

3. To determine the normalized set of daily scores, the processor further: To determine the difference between the periodically aggregated rate of the baseline average of expected service-related tickets and the rate of the observed actual events, By applying standardized normalization to the determined difference, each standardized periodically aggregated score is obtained. The system according to claim 2, configured to perform the following:

4. In order to identify the disturbance time window, the processor further, By applying a calibrated change point analysis, the non-disturbant conditions for the aforementioned period are separated from the disturbant conditions for the aforementioned period. The system according to claim 2, configured as follows.

5. To apply the calibrated change point analysis, the processor further: The process involves generating an automated restart cumulative sum chart based on the standardized set of periodically aggregated scores, reference parameters, restart points, and sensitivity threshold parameters. Determining the termination of an episode condition that establishes a limit on the termination time of the aforementioned disturbance condition, Based on the value of the automatic restart cumulative sum chart, the end time of the disturbance time window is determined. The system according to claim 4, configured to perform the following:

6. To apply the calibrated change point analysis, the processor further: Determining the start of an episode condition that establishes a limit on the start time of the aforementioned disturbance condition, Based on the value of the automatic restart cumulative sum chart, the start time of the disturbance time window is determined. The system according to claim 5, configured to perform the following:

7. The aforementioned processor further, Determining the labeling coverage for each service ticket in the set of service tickets during the aforementioned period, wherein the labeling coverage includes the percentage of missing labels or labels with temporal or spatial inconsistencies. Based on the labeling coverage, check the data quality of the service tickets generated during the period. The system according to any one of claims 1 to 6, configured to perform the following:

8. The aforementioned processor further, The system according to claim 6, configured to establish data quality checks using a metric that defines coverage of the labeled events included in a set of known disturbance periods.

9. In order to relabel, the processor further, The system according to claim 6, configured to assign an environmental disturbance identifier (ID) to the service record.

10. The hardware processor receives a set of service records from a data source, which includes one or more service records corresponding to events representing multiple disturbances in a specified area occurring during a certain period, and one or more service records that are mislabeled or do not have labels associated with the disturbances. The hardware processor determines the observed actual event rate of events representing disturbances during the period, The hardware processor determines a periodically aggregated rate of the baseline average value of expected service-related records under non-disturbance conditions for the period, The hardware processor determines a set of standardized, periodically aggregated scores as a function of the periodically aggregated rate of the baseline average and the rate of observed actual events during the period. The hardware processor identifies a disturbance time window based on the standardized, periodically aggregated score changes detected during the period, The hardware processor generates disturbance-related probabilities for the service record corresponding to the identified disturbance time window, Based on the generated disturbance-related probabilities, the step of reassigning labels to service tickets as being associated with the identified disturbance time window. A computer implementation method comprising the above.

11. The step of determining the periodically aggregated rate of the baseline mean of expected service-related records under non-disturbance conditions for the aforementioned period is: The hardware processor obtains a vector of random variables containing a large number of periodically aggregated rate values ​​of the service-related records. The hardware processor performs a step of trimming the vector by removing a predetermined number of the highest periodically aggregated rate values ​​and a predetermined number of the lowest periodically aggregated rate values ​​from the predetermined period. The hardware processor calculates a periodically aggregated rate estimate of the trimmed average by averaging the remaining data points in the vector. The hardware processor applies an additional bias adjustment coefficient to the trimmed mean estimate. The hardware processor applies a lower threshold to prevent the trimmed mean estimate from falling below the lower limit. A computer implementation method according to claim 10, comprising:

12. The step of determining the standardized, periodically aggregated set of scores is: The hardware processor determines the difference between the periodically aggregated rate of the baseline average value of expected service-related tickets and the rate of the observed actual events. The hardware processor applies standardized normalization to the difference determined to obtain each standardized, periodically aggregated score. A computer implementation method according to claim 11, comprising:

13. The step of identifying the disturbance time window is, The computer implementation method according to claim 11 or 12, further comprising the step of separating the non-disturbant conditions of the period from the disturbant conditions of the period by applying a calibrated change point analysis using the hardware processor.

14. The step of applying the calibrated change point analysis is: The hardware processor generates an automated restart cumulative sum chart based on the standardized set of periodically aggregated scores, reference parameters, restart points, and sensitivity threshold parameters. A step of determining the termination of an episode condition that establishes a limit on the termination time of the aforementioned disturbance condition, The hardware processor determines the end time of the disturbance time window based on the value of the automatic restart cumulative sum chart. A computer implementation method according to claim 13, comprising:

15. The step of applying the calibrated change point analysis further, A step of determining the start of an episode condition that establishes a limit on the start time of the aforementioned disturbance condition, The steps include determining the start time of the disturbance time window based on the value of the automatic restart cumulative sum chart, and A computer implementation method according to claim 14, comprising:

16. A computer program for probabilistic labeling, wherein the processor... A procedure for receiving a set of service records from a data source, which includes one or more service records corresponding to events representing multiple disturbances in a specified area occurring over a certain period, and one or more service records that do not have labels associated with mislabeled or associated disturbances. A procedure for determining the observed actual event rate of an event representing a disturbance during the aforementioned period, A procedure for determining the periodically aggregated rate of the baseline mean of expected service-related records under non-disturbance conditions for the aforementioned period, A procedure for determining a set of standardized, periodically aggregated scores as a function of the periodically aggregated rate of the baseline mean and the rate of observed actual events during the period, A procedure for identifying a disturbance time window based on changes in standardized, periodically aggregated scores detected during the aforementioned period, A procedure for generating disturbance-related probabilities for the service record corresponding to the identified disturbance time window, A procedure for relabeling service tickets as being related to the identified disturbance time window based on the generated disturbance-related probabilities, and A computer program designed to execute something.

17. To determine the periodically aggregated rate of the baseline average value of expected service-related records under non-disturbance conditions for the aforementioned period, the computer program instructs the processor to: A procedure for obtaining a vector of random variables containing a large number of periodically aggregated rate values ​​from the aforementioned service-related records, A procedure for trimming the vector by removing a predetermined number of the highest periodically aggregated rate values ​​and a predetermined number of the lowest periodically aggregated rate values ​​from the predetermined period, A procedure for calculating a periodically aggregated rate estimate of the trimmed average by averaging the remaining data points in the aforementioned vector, A procedure for applying an additional bias adjustment factor to the trimmed mean estimate, A procedure to prevent the trimmed mean estimate from falling below the lower limit by applying a lower threshold. A computer program according to claim 16, which causes to execute [the specified action].

18. To determine the set of standardized daily scores, the computer program instructs the processor to: A procedure for determining the difference between the periodically aggregated rate of the baseline average of expected service-related tickets and the rate of the observed actual events, The procedure involves applying standardized normalization to the determined difference to obtain each standardized, periodically aggregated score. A computer program according to claim 17, which causes to execute [the specified function].

19. In order to identify the disturbance time window, the computer program instructs the processor to: The procedure for applying a calibrated change point analysis is to perform a procedure to separate the non-disturbant conditions of the period from the disturbant conditions of the period, and the procedure for applying a calibrated change point analysis is: A procedure for generating an automated restart cumulative sum chart based on the aforementioned standardized set of daily scores, reference parameters, restart points, and sensitivity threshold parameters, A procedure for determining the termination of an episode condition that establishes a limit on the termination time of the aforementioned disturbance condition, A procedure for determining the end time of the disturbance time window based on the value of the automatic restart cumulative sum chart, and A computer program according to claim 17 or 18, including the computer program according to claim 17 or 18.

20. To apply the calibrated change point analysis, the computer program instructs the processor to: A procedure for determining the start of an episode condition that establishes a limit on the start time of the aforementioned disturbance condition, A procedure for determining the start time of the disturbance time window based on the value of the automatic restart cumulative sum chart, A computer program according to claim 19, which causes to execute [the specified action].

Images