Estimation of complex events through temporal logic

Abstract

A computer-implemented method for discovering a structure of composite duration events through temporal logic includes identifying a plurality of temporally related atomic events from a duration data trajectory of a multivariate dataset according to a definition of an atomic event predicate. At least one composite event having a duration event structure of at least some of the plurality of temporally related atomic events is discovered by machine learning. An action selected from a predefined list associated with the composite event is executed.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for dynamic multivariate data, and more particularly to the discovery of composite events having a temporal relationship.

Background Art

[0002] With the advent of artificial intelligence (AI), there has been an increasing interest in predictive analytics across many fields, including but not limited to, video comprehension and audio recognition. One such example is the estimation of composite events based on dynamic multivariate temporal data over an interval (e.g., [0, Ti] of trajectory i) that includes multiple (M) variables. Within the interval, there are persistent events called atomic events. Atomic events correspond to sub-intervals (e.g., [0, T i within) having start and end times. Atomic events typically have a signature of timestamp data in a sub-interval spanning a subset of the M variables. A set of atomic events that are combined with a logical temporal relationship spanning the set is known as the composite event. Although attempts have been made to provide the estimation of composite events by deep learning to support real-time decision-making, they have not been successful.

[0003] For example, conventional approaches to composite event estimation using existing technologies, such as deep learning, often rely on the use of black-box models and fail to provide a constructive structural and interpretable view of the composite event. These conventional approaches also cannot accommodate any domain-specific knowledge or inductive biases that may be introduced. The generally poor performance of conventional approaches on benchmark tasks, including composite event detection, hinders their ability to support real-time decision-making. [Overview of the Initiative] [Means for solving the problem]

[0004] According to one embodiment, a computer-implemented method for discovering a composite durational event structure through temporal logic includes identifying multiple temporally related atomic events from temporal data trajectories of a multivariate dataset according to the definition of an atomic event predicate. At least one composite event having the durational event structure of at least some of the multiple temporally related atomic events is discovered by machine learning. An action selected from a predetermined list associated with the composite event is performed. This method enables the identification of multiple atomic events and the discovery of composite events, and makes it possible to correct or prevent undesirable results.

[0005] According to one embodiment, the action performed is performed on a specified entity from the list associated with the composite event. The fact that the complex event occurred This includes notifying the designated entity, which may be a monitoring system that further analyzes the complex event for a response.

[0006] According to one embodiment, the period data trajectory of the multivariate dataset from which the multiple atomic events are identified is identified by determining an input trajectory having at least two variables numerically measured over one time interval.

[0007] According to one embodiment, the atomic event predicate is determined based on domain-specific knowledge. This domain-specific knowledge can be used to connect and identify high-level patterns using pre-processed period data.

[0008] According to one embodiment, the multivariate dataset is a video dataset, and processing the raw time series data of the multivariate dataset includes identifying multiple temporally related atomic events; and further including identifying individual atomic events having a duration of one sub-interval of one time interval of the raw time series data. The raw time series data of the multivariate dataset is used for data acquisition operations to analyze complex data.

[0009] According to one embodiment, the atomic event predicate is further determined by processing the raw time-series data of the multivariate dataset. Determining the atomic predicate by processing the raw time-series data provides more accurate results.

[0010] According to one embodiment, the multivariate dataset includes raw data, and the method further includes learning a corresponding rule structure based on the labeling of the raw data. The labeling of the raw data facilitates the use of machine learning to construct predicate-based rules.

[0011] According to one embodiment, the multivariate dataset includes raw data, and the method further includes capturing temporal trajectories within the raw data of the multivariate dataset, and storing the captured temporal trajectories as temporal trajectories in an automatically selected state space construction. The captured temporal trajectories are used to provide a more accurate state space storage of the multiple atomic and composite events.

[0012] According to one embodiment, at least one composite event is identified by constructing a timeline of multiple temporally related atomic events along the temporal relationships between the multiple atomic events. The timeline is used to assist in temporal logic analysis when identifying atomic events and discovering composite events.

[0013] According to one embodiment, the timeline is annotated based on the atomic event predicate, and one or more sub-intervals are localized on the timeline corresponding to the multiple atomic events. The annotation assists in identifying atomic events and discovering composite events.

[0014] According to one embodiment, the time-event structure of the at least one composite event is learned in a supervised learning operation using the localized sub-intervals on the timeline. Machine learning provides more accurate results by using domain-specific knowledge and processing of raw data by machine learning to provide more accurate results.

[0015] According to one embodiment, the multivariate dataset comprises at least one of a video dataset and an audio dataset, and the time-event structure of the at least one composite event is learned in a reinforcement learning operation using the localized sub-intervals on the timeline. The learning of the time-event structure is used to construct predicate-based rules to help identify atomic events and discover composite events. Composite events in the video dataset, the audio dataset, or a combination thereof are advantageous in fields such as video comprehension and audio recognition.

[0016] In one embodiment, the analysis of real-time streaming data is performed. The discovered composite event structure is applied to detect the evolving progression of a particular composite event by determining the multiple constituent atomic events and verifying their temporal relationships. By detecting the evolving progression of a particular composite event, fatal failures can be prevented and current failures can be mitigated.

[0017] According to one embodiment, the combined event includes a power system failure. The time-related atomic events include sensor data provided by one or more components of the power system. This application demonstrates several advantages of the present disclosure, such as mitigating and, in some cases preventing, power grid outages.

[0018] According to one embodiment, a computing device for discovering the structure of a composite period event through temporal logic comprises a processor and memory connected to the processor. The memory stores instructions and causes the processor to perform actions including discovering the structure of a composite period event through temporal logic. The computing device includes identifying a plurality of temporally related atomic events from the period data trajectory of a multivariate dataset according to the definition of an atomic event predicate. At least one composite event has a period event structure of at least some of the atomic events among the plurality of temporally related atomic events. An action selected from a predetermined list associated with the composite event is performed. The computing device enables the identification of the plurality of atomic events and the discovery of the composite event, and corrects or prevents undesirable results.

[0019] In one embodiment, the instruction causes the processor to perform an additional action to identify at least one composite event by constructing a timeline of a plurality of temporally related atomic events along the temporal relationships between the plurality of atomic events. The timeline is annotated based on the atomic event predicate, and a plurality of sub-intervals are localized on the timeline corresponding to the plurality of atomic events. The timeline is used to assist in temporal logic analysis when identifying the plurality of atomic events and discovering the plurality of composite events.

[0020] According to one embodiment, a non-temporary computer-readable storage medium is provided that actually executes computer-readable program code having computer-readable instructions, which, when executed, cause a computer device to execute a method for discovering the structure of a composite period event through temporal logic. The method includes identifying a plurality of temporally related atomic events from a period data trajectory of a multivariate dataset according to the definition of an atomic event predicate, and discovering, by machine learning, at least one composite event having the period event structure of at least some of the temporally related atomic events. An action selected from a predetermined list associated with the composite event is performed. The method makes it possible to identify atomic events, discover composite events, and correct or prevent undesirable results.

[0021] These and other features will become apparent from the following detailed description of exemplary embodiments, which should be read in conjunction with the attached drawings.

[0022] The drawings are illustrative embodiments. They do not illustrate all embodiments. Other embodiments may be added or used instead. Obvious or unnecessary details may be omitted to save space or for more effective explanation. Some embodiments may be carried out using additional components or processes, without using all of the illustrated components or processes, or in combination thereof. When the same number appears in different drawings, it refers to the same or similar component or process. [Brief explanation of the drawing]

[0023] [Figure 1] Figure 1 is a schematic diagram showing complex temporal data consistent with an exemplary embodiment. [Figure 2] Figure 2 is a conceptual block diagram of a computer-implemented method for estimating complex events through temporal logic, consistent with an exemplary embodiment. [Figure 3] Figure 3 shows a time series capture that conforms to an exemplary embodiment. [Figure 4] Figure 4 shows a state space construction that conforms to an exemplary embodiment. [Figure 5] Figure 5 shows the identification of composite events / atomic events that conform to an exemplary embodiment. [Figure 6] Figure 6 is an explanatory diagram of power grid data monitored for event identification that conforms to an exemplary embodiment. [Figure 7] Figure 7 shows the power grid of Figure 6 that conforms to an exemplary embodiment. [Figure 8] Figure 8 shows some images of individual atomic events of a synthetic video dataset that conforms to an exemplary embodiment. [Figure 9] Figure 9 is a flowchart diagram showing a computer-implemented method for discovering the structure of composite duration events through temporal logic that conforms to an exemplary embodiment. [Figure 10] Figure 10 is an explanatory diagram of a functional block diagram of a computer hardware platform that conforms to an exemplary embodiment. [Figure 11] Figure 11 illustrates an exemplary cloud computing environment that conforms to an exemplary embodiment. [Figure 12] Figure 12 illustrates a set of functional abstraction layers provided by a cloud computing environment that conforms to an exemplary embodiment.

Mode for Carrying Out the Invention

[0024] Overview In the following detailed description, numerous specific details are set forth as examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent that the present teachings may be practiced without such details. In other instances, well-known methods, procedures, components, or circuits, or combinations thereof, are described at a relatively high level in order to avoid unnecessarily obscuring aspects of the present teachings.

[0025] In summary, the process begins with identifying the individual unobserved events that make up a composite event observed within a timeline. For example, raw time-series data is processed to identify individual events within the time series. Logical rules are constructed through a basic set of predicate rules to classify the individual events. Through machine learning, these logical rules are further combined to identify which of the individual events constitute the observed composite event. These logical rules provide an interpretable view of how underlying temporal events (e.g., individual events) can give rise to the observed composite event. Machine learning is used to construct the composite rules and to provide tuning with new data over time.

[0026] There are transient sources of information that include many persistent events. A composite event is a temporal logical combination of individual atomic events. This disclosure provides computer-implemented methods and computing devices specifically configured to discover the basic structure / make-up of a composite event. To discover the composite event, identification is provided of each of the corresponding atomic events and their logical temporal relationships when combined to form the composite event.

[0027] In this disclosure, the underlying complex rules are revealed through the use of labeled data. For example, a temporal relational network can be used by first learning the temporal relationships between multiple atomic events, and then learning a given rule structure from labeled complex events.

[0028] Figure 1 is an overview illustrating complex period data consistent with an exemplary embodiment. It should be understood that Figure 1 is provided for illustrative purposes only and without limitation. In Figure 1, the complex period data illustrates a composite event which is a compilation of atomic events. In this case, automobile 101 has a windshield 145 that is shattered by a baseball 110. However, the shattering of the windshield can be considered a composite event of four atomic events. In the first atomic event 105, a pitcher throws the baseball 110 towards a batter. In the second atomic event 115, the batter swings at the baseball 110. In the third atomic event, the baseball 110 is hit by the bat and reverses direction. In the fourth atomic event, the baseball 110 travels over a fence and collides with automobile 101 in a parking lot, shattering the windshield 145. Although only four atomic events are shown in Figure 1, the number of atomic events constituting the composite event depends in part on the complexity of the composite event, and substantially any number of atomic events may exist. A start time and an end time exist for the composite event and for each of the multiple atomic events.

[0029] The aforementioned composite event in Figure 1 clearly presents multiple atomic events, but in the real world, a considerable amount of "noise" may exist that hinders the discovery of composite events. It should be noted that in the case of complex time series, such complex time series may have intrinsic entropy and may include multiple time scales. Therefore, analyzing raw data, such as raw datasets, and using domain-specific knowledge and machine learning to analyze the raw data, discover composite events, and identify the atomic events that constitute these composite events has been previously unknown. Raw datasets contain many different types of information, such as event data, and discovering specific events in a temporal information source containing many persistent events requires complex analysis.

[0030] The approaches used to discover composite events may include, but are not limited to, time series ingestion, state space construction, or identification of composite / atomic events or combinations thereof. According to this disclosure, composite event detection is performed by integrating and handling both the raw period data targeted by a purely learning-based model and the current domain knowledge provided to connect and identify high-level patterns in the preprocessed period data. Detecting composite events using both approaches has been shown to be more accurate than using either method alone.

[0031] The computer methods and computer devices implemented in this disclosure advantageously provide an interpretable construction of complex event definitions and associated domain knowledge. The above-mentioned benchmark task performance evaluation tests, such as the detection of complex events, have improved results in terms of speed and accuracy compared to conventional techniques using black-box models, thus improving the detection of complex events.

[0032] The computer-implemented methods and computing devices of this disclosure also offer many improvements in the field of predictive analysis by discovering the structures of various composite labels previously unknown in the art, in terms of each atomic event and their temporal relationships with one another. Furthermore, the teachings of this disclosure provide the use of acquired structural knowledge for interpretable real-time detection and interpretable real-time mitigation of ongoing composite events in complex dynamic systems. The teachings of this specification can result in reductions in processing overhead and storage, as well as reductions in power consumption.

[0033] Additional advantages of the computer-implemented methods and devices of this disclosure are disclosed herein.

[0034] Exemplary Embodiments Figure 2 shows a conceptual block diagram 200 in operation 205, where individual events in a time series are defined by processing raw time-series data. This raw time-series data may be substantially arbitrary data having a start time and an end time. For example, motion data, video data, audio data, digitally logged data, streaming sensor data, or temperature data are just a few examples of the non-limiting data that can constitute a time series. Identifying individual events in such a time series is particularly useful, for example, in the fields of video understanding and audio recognition. However, those skilled in the art should understand that the computer-implemented methods and apparatus of this disclosure are applicable to many other fields.

[0035] In operation 225, logical rules are constructed through a basic set of predicate rules. Each event that occurs has a basic logic that determines its existence or non-existence. Such predicate rules may have been previously created, for example, by a management application, or through an administrator or subject matter expert (SME), or a combination thereof. The management application or SME, or a combination thereof, can define the configuration of the basic set of predicate rules that identify these events given information available from the state space. These logical rules are used to classify individual events identified in chronological order.

[0036] Regarding an example from the basic set of predicate rules, if power grid operation is observed, a possible composite event is a transformer failure. A base predicate that can be applied is that an excess current exceeding a certain threshold leads to a transformer failure. An additional data modality, such as a temperature sensor, could be that a heat wave uses more energy for cooling, thus leading to a potential malfunction of the power grid. Also, the transformer temperature may rise to a level associated with equipment failure, which could be another rule in the basic set of predicate rules.

[0037] In operation 245, machine learning is used to further combine multiple logical rules to identify which of the individual events constitute the observed composite event. In this exemplary embodiment, the machine learning is supervised with some representative labeled event data. However, reinforcement learning (RL) may also be used. RL is a learning agent that interacts with an environment and can be configured to observe basic behavior. RL can be used to perform various operations, such as exploration, exploitation, Markov decision processes, deep learning, policy learning, value learning, etc.

[0038] Figure 3 shows a time-series ingestion (300) consistent with an exemplary embodiment. Some attributes of a time-series ingestion are that some time-series modality can be ingested (310). Time-series data can be arranged on a timeline in many forms. For example, timeline data may include, but is not limited to, data logs, streaming sensors, audio, or video (320).

[0039] Continuing with Figure 3, a composite event detection device according to this disclosure can operate with one or more data modalities for a given data preprocessor. For example, a data preprocessor can be run on an acquired data modality (330) and used to convert raw time-series data into a queryable format. In the case of continuous-time sensor data (340), such sensor data may be binned by day, week, or month. This binning converts the raw sensor data into a searchable format. The raw sensor data is a value stream, e.g., temperature measurement, pressure measurement, etc. Data preprocessing can be used to aggregate temperature or pressure by minute, hour, day, etc. Furthermore, in the case of video data (350), object identification and localization can be performed within the frame. Acquired data modalities from security video can also be preprocessed. It should be understood that the non-limiting examples described above are provided for illustrative purposes only. There are many types of acquired data that can be preprocessed. The data preprocessor is typically modality-specific.

[0040] Figure 4 shows a state-space construction (400) consistent with an exemplary embodiment. The state-space database represents the overall point state of the system over a period of time. In the case of preprocessed time-series data (discussed with respect to Figure 3), such preprocessed data is combined into a single database that is searchable (e.g., queryable) by rules, e.g., a state-space database in aggregate of all preprocessed data modalities aligned (405) on a continuous timeline. The state-space can be queried to identify where along the timeline an event occurred. For evaluation of the state-space database, there exists an expert rule identification (420) in which a set of predicate rules is defined (425). The predicate rules are building blocks for constructing event rules about items to be monitored. Event detection (430) can be used to predict events, e.g., equipment failure, and the predicate rules are applied on the state-space database to obtain information about the event. Events of interest occur over intervals in the timeline and are identified (435) for which candidate rules are applied. It should be understood that events of interest are detected in order to execute the candidate rule, avoiding the application of the candidate rule across the entire state space, which could potentially become infinitely large. Thus, savings in computation time and power are achieved.

[0041] Figure 5 shows the identification of composite / atomic events (500) consistent with an exemplary embodiment. Given a state space construction, for example, a given state space construction as discussed with respect to Figure 4, there are given predicates and event intervals used to identify atomic events and composite events (505). Multiple atomic events of interest occur within the identified interval (510). Such events are evaluated by construction rules using predefined basic predicates (515). These multiple atomic events may be implicitly defined based on knowledge and may not have records within the timeline (520).

[0042] The composite event is essentially determined by sequentially identifying multiple atomic events on the timeline (525). The composite event is determined, for example, by utilizing supervised learning or reinforcement learning. The system associates the most likely combination and order of events to explain the identification of the composite event. In the case of event prediction, the computer framework will apply previously defined rules to the detected events. Such rules may be predefined by subject experts.

[0043] Figure 6 is an illustrative diagram of power grid data monitored for event identification (600), consistent with an exemplary embodiment. A basic predicate that can be applied is an overcurrent exceeding a certain threshold that causes a transformer malfunction. Additional data modalities, such as temperature sensing, indicate that more energy is used for cooling during a heatwave, thus potentially causing a malfunction. The power grid (as shown, for example, in Figure 7) may be monitored for temperature and current values ​​to detect or predict operational faults and to enable the implementation of corrective or preventive measures.

[0044] Continuing to refer to Figure 6, it is shown that in 605, the temperature (56°C to 57°C) and current (3kA) values ​​in the power lines of the power grid are measured by sensors. The temperature data and current data are two modalities taken in by the preprocessor 610. The preprocessor 610 converts the raw temperature and current data into a format that can be searched in the state-space database 630. In this example, the power grid monitoring has at least two predicate rules. High current is a current value greater than 5kA in 4 minutes. High temperature is a temperature greater than 87°C in 3 minutes. The event detection operation 635 detects events according to the predicate rules, for example, multiple atomic events 640. The event identification operation 650 is performed based on rules in which faults are defined as high temperature and high current at values ​​specified by the predicates. A composite learner analyzes multiple atomic events 640 and, based on rules, identifies a failure 660 which is a composite event of multiple atomic events 640. Another action, such as notification, alarm activation, corrective action, or preventive action, may then be taken.

[0045] Figure 7 shows the power grid 700 of Figure 6, consistent with an exemplary embodiment. A tower 705 is used to support power lines and includes a sensor 710 capable of monitoring items such as temperature and current (kA). Regarding the identification interval of the power grid, there may be a radio transmitter 715 for sensor data to be provided to a server 720 for data acquisition, as previously discussed herein.

[0046] Figure 8 illustrates multiple images of several individual atomic events in a synthetic video dataset 800, consistent with an exemplary embodiment. While this disclosure is shown in this embodiment as applicable to synthetic video data for teaching purposes, it should be understood that it can be applied to multivariate video datasets, audio datasets, or combinations thereof by identifying individual events as atomic events of composite events. For example, a composite event may be a combination of temporally related audio atomic events and video atomic events. Three of the multiple actions exemplified in the synthetic video include pick-place 805 (e.g., pick up and place), slide 810, and contain 815. Some of the items in the video are a cone, a sphere, and a snitch. Task 1 is an atomic recognition event 820, and is performed by comparing the synthetic frames slide 810, place-place 805, and contain 815, as shown therein. The multiple atomic events are combinations of objects with different motion and shape, as well as temporal predicates, e.g., before, during, and after. Task 2 is a composite recognition event 825, e.g., the composite recognition event 825 which includes performing a location occupancy action during a slide and performing a containment action after the slide action. The composite recognition event 825 consists of temporal predicates between two atomic events. Task 3 demonstrates identifying snitch localization 830 on a graph. The synthetic video dataset 800 is an example of a multi-label classification problem that aims to recover the underlying atomic events leading to composite events.

[0047] Exemplary process Given the above overview of the exemplary architecture, it may be useful here to consider a higher-level discussion of the exemplary method. For this purpose, Figure 9 illustrates flowchart 900, which shows various perspectives of a computer-implemented method consistent with the exemplary embodiment. Figure 9 is shown as a collection of blocks in logical order, representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions that perform the mentioned operations when executed by one or more processors. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, etc., that perform functions or implement abstract data types. In each method, the order in which the operations are described is not intended to be interpreted as restrictive, and any number of described blocks can be combined in any order, in parallel, or in any combination thereof to implement the method.

[0048] Figure 9 is a flowchart 900 showing a computer-implemented method for discovering the structure of a complex period event through temporal logic, consistent with the illustrated embodiment.

[0049] In operation 905, multiple temporally related atomic events are identified from the time-series data trajectories of a multivariate dataset according to the definition of an atomic event predicate. The atomic event predicate is determined based on domain-specific knowledge, which may be provided in advance by an expert. The atomic event predicate may be additionally determined by processing the raw time-series data of the multivariate dataset. If supervised data is available for the atomic events, the atomic event predicate can be automatically learned over raw time-series events of interest without input from a subject expert (SME).

[0050] By using raw period data (raw time-series data) and current domain knowledge to detect complex events, and by leveraging the ability to connect and identify high-level patterns with preprocessed period data, this approach provides higher accuracy than either approach alone. A data preprocessor can be used to transform the raw data into a format that can be queried by rules within the system.

[0051] In operation 915, machine learning is used to discover at least one composite event having a time event structure of at least some of the time-related atomic events. The type of machine learning used may be supervised learning or reinforcement learning. The machine learning may also be used to construct composite rules based on the atomic predicates, and may also be used to further refine the rules in response to receiving new data or a combination thereof, either periodically or in combination thereof.

[0052] Operation 925 performs an action selected from a predetermined list associated with the composite event. Such an action may be specific to the composite event. For example, the action may notify a designated entity that the event has occurred, or warn that a composite event may occur in the near future based on multiple atomic events identified within a certain time frame. The applications are substantially unlimited, and as previously stated, a power grid failure is one non-limiting example. The action may be to remember the discovered composite event, or to take action in response to a composite event, which may involve much more than just notification, or a combination of both. Operations 905 to 925 may be performed repeatedly.

[0053] A particularly configured exemplary computer hardware platform Figure 10 provides an explanatory diagram of a functional block diagram of a computer hardware platform 1000. In particular, Figure 10 illustrates a specially configured network or host computer platform 1000 that may be used to implement the method shown in Figure 9.

[0054] The computer platform 1000 may include a central processing unit (CPU) 1004 connected to a system bus 1002, a hard disk drive (HDD) 1006, random access memory (RAM) or read-only memory (ROM) 1008, a keyboard 1010, a mouse 1012, a display 1014, and a communication interface 1016. The HDD 1006 may include a data store.

[0055] In one embodiment, the HDD 1006 has the function of storing a program that can perform various processes, such as machine learning, predictive modeling, classification, and updating model parameters. The ML model generation module 1040 is configured to generate a machine learning model based on at least one of the generated candidate machine learning pipelines.

[0056] Continuing to refer to Figure 10, various modules exist, shown as discrete components for the sake of clarity. However, it should be understood that the functionality and number of such modules may be fewer or more than shown. The composite event estimation module 1040, consistent with the exemplary embodiment, comprises an atomic event identification module 1042 configured to identify multiple temporally related atomic events from the time-data trajectory of a multivariate dataset according to the definition of an atomic event predicate. The composite event discovery module 1044 is configured to discover composite events having a time-event structure of at least some of the multiple temporally related atomic events identified by the atomic event identification module 1042.

[0057] The composite event action module 1050 is configured to perform actions associated with a composite event. These actions may be selected from a predetermined list of actions. The time series ingestion module 1052 is configured to ingest any time series modality, including but not limited to data logs, streaming sensors, audio data, or video data, or combinations thereof, and to perform data preprocessing on the ingested modality. This data preprocessing transforms the raw time series data into a form that can be searched via queries.

[0058] The state space construction module 1054 is configured to provide expert rule identification by aligning preprocessed modalities on a unified timeline and defining a set of base predicate rules that can be evaluated in a state space database, and to perform event detection when events of interest occur over the interval of the timeline.

[0059] The domain-specific knowledge module 1056 contains domain-specific knowledge loaded into storage regarding the subject of identification associated with the data being analyzed for atomic and composite events. The supervised learning module 1058 and the reinforcement learning module 1060 consist of their respective types of machine learning used for discovering composite events, creating and refining rules, and updating them based on new data being analyzed. It should be noted that there are no capability requirements for both supervised and reinforcement learning in the composite event estimation module 1040. The communication interface module 1062 is configured to communicate with a source providing, for example, streaming data, such as from sensors.

[0060] Exemplary cloud platform As previously stated, functions related to the low-bandwidth transmission of high-resolution video data may include cloud computing. While this disclosure includes a detailed description of cloud computing as described later herein, it will be understood that the implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of this disclosure can be implemented in combination with any other type of computing environment that is currently known or will be developed in the future.

[0061] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0062] The features are as follows:

[0063] On-demand self-service: Cloud consumers can unilaterally provision computing functions, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0064] Broad network access: The functionality is available over a network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0065] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, and various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally do not have control or knowledge of the exact location of the resources provided, but can identify the location at a higher level of abstraction (e.g., country, state, or data center), making it location-independent.

[0066] Rapid Adaptability: Features can be provisioned quickly and flexibly, and in some cases automatically, they can scale out quickly, be released quickly, and scale in quickly. For consumers, the features available for provisioning are often unlimited and can be purchased at any amount at any time.

[0067] Measured Services: Cloud systems automatically control and optimize resource usage by employing metric functions at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0068] The service model is as follows:

[0069] Software as a Service (SaaS): This refers to the functionality provided to consumers for using a provider's applications running on a cloud infrastructure. These applications are accessible from various client devices through a thin client interface, such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, such as the network, servers, operating system, storage, or even the underlying cloud infrastructure encompassing individual application functions, with the possible exception of limited user-specific application configuration settings.

[0070] Platform as a Service (PaaS): A service provided to a consumer to deploy applications they have created or acquired, generated using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, such as the network, servers, operating system, or storage, but has control over the deployed applications and, in some cases, the application hosting environment configuration.

[0071] Infrastructure as a Service (IaaS): This is a service provided to a consumer to provision processing, storage, networking, and other basic computing resources, enabling the consumer to deploy and run any software, including operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has limited control over the operating system, storage, deployed applications, and, in some cases, network components (e.g., the host's firewall).

[0072] The deployment models are as follows:

[0073] Private Cloud: A cloud infrastructure is operated solely for a specific organization. This cloud infrastructure may be managed by that organization or a third party, and may reside on-premises or off-premises.

[0074] Community Cloud: Cloud infrastructure is shared by several organizations and supports a specific community that shares common interests (e.g., mission, security requirements, policies, and compliance considerations). The cloud infrastructure may be managed by the organization or a third party and may reside on-premises or off-premises.

[0075] Public cloud: Cloud infrastructure is available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0076] Hybrid Cloud: Cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain separate entities but are brought together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application migration.

[0077] Cloud computing environments are oriented services that focus on statelessness, low coupling, modularity, and semantic interoperability. The heart of cloud computing is the infrastructure, which includes a network of interconnected nodes.

[0078] Referring to Figure 11, a cloud computing environment 1100 utilizing cloud computing is illustrated. As illustrated, the cloud computing environment 1100 comprises one or more cloud computing nodes 1110 that can communicate with local computing devices used by cloud consumers, such as personal digital assistants (PDAs) or mobile phones 1154A, desktop computers 1154B, laptop computers 1154C, or automotive computer systems 1154N, or a combination thereof. The nodes 1110 can communicate with each other. The nodes 1110 can be physically or virtually grouped into one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or a combination thereof, as described herein (not shown). This enables the cloud computing environment 1100 to provide infrastructure, platforms, or software, or a combination thereof, as a service that does not require cloud consumers to maintain resources on their local computing devices. It is understood that the types of computing devices 1154A to 1154N shown in Figure 11 are intended to be illustrative only, and that the cloud computing node 1110 and the cloud computing environment 1100 can communicate with any type of computerized device via any type of network or network addressable connection or a combination thereof (for example, using a web browser).

[0079] Referring to Figure 12, an abstraction model layer 1200 provided by the cloud computing environment 1200 (Figure 12) is shown. It should be understood that the components, layers, and functions shown in Figure 12 are intended to be illustrative only, and that embodiments of this disclosure are not limited thereto. As illustrated, several layers and corresponding functions are provided below.

[0080] The hardware and software layer 1260 includes hardware and software components. Examples of hardware components include a mainframe 1261, a RISC (Reduced Instruction Set Computer) architecture-based server 1262; a server 1263; a blade server 1264; a storage device 1265; and network and networking components 1266. In some embodiments, the software components include network application server software 1267 and database software 1268.

[0081] The virtualization layer 1270 provides an abstraction layer from which the following examples of virtual entities may be provided: namely, virtual servers 1271; virtual storage 1272; virtual networks 1273, including the above virtual network 1273, for example, a virtual private network; virtual applications and operating systems 1274; and virtual clients 1275.

[0082] In one example, the management layer 1280 may provide several functions as described below: Resource provisioning 1281 provides the dynamic procurement of computing and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 1282 provides cost tracking when resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identification and verification of cloud consumers and tasks, and protection for data and other resources. The user portal 1283 provides access to the cloud computing environment for consumers and system administrators. Service level management 1284 provides the allocation and management of cloud computing resources to ensure that the required service levels are met. Service level agreement (SLA) planning and execution 1285 provides the pre-placement and procurement of cloud computing resources for which future requirements are anticipated according to the SLA.

[0083] The workload layer 1290 provides examples of several functions for which a cloud computing environment may be utilized. Examples of several workloads and functions that may be provided from this layer include mapping and navigation 1291; software development and lifecycle management 1292; provision of virtual classroom education 1293; data analysis processing 1294; transaction processing 1295; and a composite event estimation module 1296 configured to perform the identification of multiple temporally related atomic events and the discovery of composite events consisting of multiple atomic events, as described herein.

[0084] conclusion

[0085] The various embodiments described in this instruction are presented for illustrative purposes only and are not intended to be exhaustive or limitful to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the embodiments described herein. The terms used herein have been selected to best describe the principles of the embodiments, the practical application or technical improvement to the art found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0086] The above describes what is considered to be the best-case scenario, other examples, or combinations thereof, but various modifications may be made, the subject matter disclosed herein may be implemented in various forms and examples, and these teachings may be applied in numerous applications, only a portion of which are described herein. Any and all applications, modifications, and variations that fall within the true scope of these teachings are intended to be claimed by the attached claims.

[0087] The components, processes, features, objectives, benefits, and advantages discussed herein are merely illustrative. None of them, nor any discussions relating to them, are intended to limit the scope of protection. While various advantages have been discussed herein, it will be understood that not all embodiments necessarily include all of these advantages. Unless otherwise noted, all measurements, values, ratings, locations, sizes, and other specifications specified herein, including in the appended claims, are approximate and not precise. They are intended to be within a reasonable range that is consistent with the functions and art to which they pertain.

[0088] Numerous other embodiments are also being considered. These include embodiments having fewer, additional, different, or combinations thereof of components, processes, features, purposes, benefits, and advantages. These also include embodiments in which components or processes or combinations thereof are arranged or sequenced differently, or arranged and sequenced.

[0089] The flowcharts and block diagrams in the figures herein illustrate the architecture, functionality, and operation of possible implementations according to various embodiments of this disclosure.

[0090] While the above is described in relation to exemplary embodiments, the term “exemplary” should be understood to mean merely one example, and not the best or optimal. Except as stated above, nothing described or illustrated, whether or not it is included in the claims, is intended to provide to the general public any component, process, feature, purpose, benefit, advantage, or equivalent.

[0091] Words and expressions used herein will be understood to have the general meanings given to such words and expressions in relation to their respective corresponding areas of inquiry and study, unless a specific meaning is otherwise specified herein. Words indicating relationships such as first and second may be used solely to distinguish entities or actions from one another and do not necessarily require or suggest any actual relationship or order between such entities or actions. The words “comprises,” “comprising,” or any other variant thereof are intended to include non-exclusive inclusion, and a process, method, article, or apparatus containing a list of elements may not only have those elements but may also include other elements not explicitly listed or inherent in such process, method, article, or apparatus. The element following one ("a" or "an") does not, without further constraint, exclude the presence of additional identical elements in a process, method, article, or apparatus containing that element.

[0092] This abstract of the disclosure is provided to enable the reader to quickly confirm the nature of the technical disclosure. It is submitted with the understanding that it is not to be used to interpret or limit the claims or their meaning. In addition, it is found that, for the purpose of simplifying the disclosure, various features are grouped together in various embodiments in the forms for which the invention is to be carried out as described herein. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments have more features than those expressly described in each claim. Rather, as reflected in the appended claims, the subject matter of the invention is not found in all the features of the single embodiment disclosed. Accordingly, the appended claims are incorporated into the forms for which the invention is to be carried out as described herein, and each claim exists independently as separately claimed subject matter.

Claims

1. A computer-implemented method for discovering the structure of complex period events through temporal logic, wherein the method is Identifying multiple temporally related atomic events from the time-series data trajectories of a multivariate dataset according to the definition of atomic event predicates; To discover, by machine learning, at least one composite event having a time-series event structure of at least some of the aforementioned time-related atomic events; and, Execute an action selected from a predetermined list associated with the aforementioned complex event. Includes, The method described above is Identifying the at least one composite event by constructing a timeline of multiple temporally related atomic events along the temporal relationships between the multiple atomic events; Annotating the aforementioned timeline based on the aforementioned atomic event predicates; and, Localizing multiple sub-intervals on the timeline corresponding to the multiple temporally related atomic events. This also includes, The aforementioned method.

2. A computer-implemented method for discovering the structure of complex period events through temporal logic, wherein the method is Identifying multiple temporally related atomic events from the time-series data trajectories of a multivariate dataset according to the definition of atomic event predicates; To discover, by machine learning, at least one composite event having a time-series event structure of at least some of the aforementioned time-related atomic events; and, Execute an action selected from a predetermined list associated with the aforementioned complex event. Includes, The method described above is Identifying the at least one composite event by constructing a timeline of multiple temporally related atomic events along the temporal relationships between the multiple atomic events; Annotating the aforementioned timeline based on the aforementioned atomic event predicates; and, Localizing multiple sub-intervals on the timeline corresponding to the aforementioned multiple temporally related atomic events; The supervised learning operation further includes learning the time-event structure of the at least one composite event using the localized sub-intervals on the timeline, The aforementioned method.

3. A computer-implemented method for discovering the structure of complex period events through temporal logic, wherein the method is Identifying multiple temporally related atomic events from the time-series data trajectories of a multivariate dataset according to the definition of atomic event predicates; To discover, by machine learning, at least one composite event having a time-series event structure of at least some of the aforementioned time-related atomic events; and, Execute an action selected from a predetermined list associated with the aforementioned complex event. Includes, The method described above is By analyzing real-time streaming data, apply the period event structure of the at least one composite event discovered; and, By detecting constituent atomic events and verifying the temporal relationships between the multiple atomic events, the evolving development of a particular composite event can be detected. This also includes, The aforementioned method.

4. A computer-implemented method according to any one of claims 1 to 3, wherein the action performed includes notifying a specified entity from the list associated with the composite event that the composite event has occurred.

5. A computer-implemented method according to any one of claims 1 to 3, wherein the period data trajectory of the multivariate dataset from which the plurality of atomic events are identified is identified by determining an input trajectory that includes at least two variables measured numerically over one time interval.

6. A computer-implemented method according to any one of claims 1 to 5, wherein the atomic event predicate is determined based on domain-specific knowledge.

7. The multivariate dataset includes a video dataset, and the ability to identify multiple temporally related atomic events is Processing the raw time series data of the aforementioned multivariate dataset; and, Identifying individual atomic events having a duration of one sub-interval of one time interval in the aforementioned raw time series data. A computer-implemented method according to claim 6, further comprising:

8. A computer-implemented method according to claim 7, further comprising determining the atomic event predicates by processing the raw time-series data of the multivariate dataset.

9. The multivariate dataset includes raw data; and, The method further includes learning the corresponding rule structure based on the labeling of the raw data. The method according to claim 6, implemented on a computer.

10. To capture the temporal trajectory within the raw data of the multivariate dataset; and, The captured temporal trajectory is stored as the temporal trajectory in the automatically selected state space construction. A computer-implemented method according to claim 9, further comprising:

11. A computer-implemented method according to any one of claims 2 to 3, further comprising identifying the at least one composite event by constructing a timeline of a plurality of temporally related atomic events along the temporal relationships between the plurality of atomic events.

12. Annotating the aforementioned timeline based on the aforementioned atomic event predicates; and, To localize one or more sub-intervals on the timeline corresponding to the plurality of atomic events. A computer-implemented method according to claim 11, further comprising:

13. A computer-implemented method according to claim 2, further comprising learning the time-event structure of the at least one composite event using the localized sub-intervals on the timeline in a reinforcement learning operation.

14. By analyzing real-time streaming data, apply the period event structure of the at least one composite event discovered; and, By detecting constituent atomic events and verifying the temporal relationships between the multiple atomic events, the evolving development of a particular composite event can be detected. A computer-implemented method according to claim 1 or 2, further comprising:

15. The aforementioned at least one composite event includes a power system failure; and, The aforementioned time-related plurality of atomic events include sensor data provided by one or more components of the power system. A computer-implemented method according to any one of claims 1 to 14.

16. A computing device configured to discover the structure of a complex period event through temporal logic, wherein the computing device Processor; Memory connected to the processor () It is equipped with, the memory stores instructions, and the processor, To perform the method described in any one of claims 1 to 15, The aforementioned computing device.

17. A non-temporary computer-readable storage medium that, when executed, actually executes computer-readable program code having computer-readable instructions causing a computer device to perform a method for discovering the structure of a complex period event through temporal logic, wherein the method includes the method according to any one of claims 1 to 15.

Images