Method for detecting complex event using stream data pattern analysis and apparatus therefor

Abstract

The present disclosure relates to a technique of detecting a complex event by using stream data pattern analysis and provides an apparatus and method, the apparatus comprising: a data receiving unit; a stream data conversion unit for inputting unstructured data into one or more preset artificial intelligence models and converting the unstructured data into stream data; a stream data purification unit for performing delay compensation and synchronous mapping on stream data called from a data storage management object and generating purified stream data; an event processing unit for extracting complex event information by using the purified stream data; and a monitoring unit for determining whether a complex event occurs.

Description

Method and device for detecting complex events using stream data pattern analysis

[0001] The present disclosure relates to a technique for detecting complex events using stream data pattern analysis.

[0002] With advancements in artificial intelligence and deep learning, video analysis technology is improving, leading to an increase in the adoption of this technology in the field of industrial accident prevention. Interest in strengthening industrial safety, such as through the enactment of the Serious Accident Punishment Act, is growing, and CCTV systems being recently introduced in the field require a high level of detection, automation, and efficiency.

[0003] However, smart CCTV's functionality and performance are limited due to solutions that operate only on specific sites and various on-site variables. Furthermore, the nature of disaster scenarios makes it difficult to secure large amounts of training data, making it difficult to utilize deep learning algorithms that require big data-based training.

[0004] In other words, although artificial intelligence technology is being applied in various industrial fields, the characteristics and requirements of each industrial field are different, so there are limitations to its universality, and there are also difficulties in securing specialized learning data for learning.

[0005] Additionally, there is a problem that the processing time of the artificial intelligence model becomes long when detecting or predicting the occurrence of complex events using image data through the artificial intelligence model.

[0006] In this way, the demand for detecting the occurrence of various events using image data is increasing in industrial sites, but there are limitations to practical application due to increased processing time of artificial intelligence models, limitations in environmental specialization, and difficulties in securing learning data.

[0007] Therefore, it is necessary to develop a technology that uses a general-purpose artificial intelligence model, can be quickly applied in various environments, and can also quickly and efficiently process tuning for detection events.

[0008] The present disclosure seeks to provide a technique for detecting complex events using stream data pattern analysis.

[0009] In one aspect, the present embodiments provide a complex event detection device using stream data pattern analysis, including a data receiving unit that receives unstructured data, a stream data conversion unit that inputs the unstructured data into one or more preset artificial intelligence models and converts each output data output from each artificial intelligence model into stream data and transmits it to a data storage management object, a stream data purification unit that performs delay compensation and synchronous mapping on stream data called from the data storage management object to generate refined stream data, an event processing unit that inputs the refined stream data into one or more preset pattern algorithms to extract pattern information for each event sequence and inputs the pattern information into one or more preset complex event algorithms to extract complex event information, and a monitoring unit that monitors the complex event information to determine whether a complex event has occurred.

[0010] In another aspect, the present embodiments provide a method for detecting complex events using stream data pattern analysis, comprising a data receiving step of receiving unstructured data, a stream data conversion step of inputting the unstructured data into one or more preset artificial intelligence models and converting each output data output from each artificial intelligence model into stream data and transmitting the stream data to a data storage management object, a stream data purification step of performing delay compensation and synchronous mapping on stream data called from the data storage management object to generate refined stream data, an event processing step of inputting the refined stream data into one or more preset pattern algorithms to extract pattern information for each event sequence and inputting the pattern information into one or more preset complex event algorithms to extract complex event information, and a monitoring step of monitoring the complex event information to determine whether a complex event has occurred.

[0011] According to the present disclosure, a technique for detecting complex events using stream data pattern analysis can be provided.

[0012] Figure 1 is a drawing for explaining an operation of detecting a complex event according to a conventional technology.

[0013] FIG. 2 is a diagram illustrating a configuration of a composite event detection device according to one embodiment.

[0014] FIG. 3 is a diagram for explaining an operation of monitoring an event using image data according to one embodiment.

[0015] FIG. 4 is a diagram for explaining a configuration for storing stream data in a data storage management object according to one embodiment.

[0016] FIG. 5 is a diagram for explaining a stream data processing operation according to an event occurrence time according to one embodiment.

[0017] FIG. 6 is a diagram for explaining an operation of refining stream data using watermark information according to one embodiment.

[0018] Figure 7 is a diagram for explaining the operation of a pattern algorithm according to one embodiment.

[0019] FIG. 8 is a diagram for explaining a composite event detection operation according to one embodiment.

[0020] FIG. 9 is a diagram illustrating a step-by-step composite event detection operation according to one embodiment.

[0021] FIG. 10 is a diagram for explaining a complex event detection method according to one embodiment.

[0022] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When adding reference numerals to components in each drawing, identical components may have the same numerals as much as possible even if they are shown in different drawings. In addition, when describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the technical idea of ​​the present invention, the detailed description may be omitted. When "includes," "has," "consists of," etc. are used in this specification, other parts may be added unless "only" is used. When a component is expressed in the singular, it may include a case in which the plural is included unless specifically stated otherwise.

[0023] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the present disclosure. These terms are only intended to distinguish the components from other components, and the nature, order, sequence, or number of the components are not limited by the terms.

[0024] In a description of the positional relationship of components, when it is described that two or more components are "connected," "combined," or "connected," it should be understood that the two or more components may be directly "connected," "combined," or "connected," but that the two or more components may also be further "interposed" with another component to be "connected," "combined," or "connected." Here, the other component may be included in one or more of the two or more components that are "connected," "combined," or "connected" to each other.

[0025] In the description of the temporal flow relationship related to components, operation methods, or manufacturing methods, for example, when the temporal or flow relationship is described as “after”, “following”, “next to”, “before”, etc., it may also include cases where it is not continuous, unless “immediately” or “directly” is used.

[0026] Meanwhile, when numerical values ​​or corresponding information (e.g., levels, etc.) for components are mentioned, even without separate explicit description, the numerical values ​​or corresponding information may be interpreted as including an error range that may occur due to various factors (e.g., process factors, internal or external impact, noise, etc.).

[0027] The embodiments are described in detail with reference to the drawings below.

[0028] Below, the devices and methods according to the present disclosure are described in more detail with reference to the drawings. Each algorithm described below is provided as an example, and various algorithms that perform the same purpose and function can be applied to the present disclosure. Furthermore, programs capable of performing the functions of the present embodiments are also included in the present disclosure, and recording media containing the programs are also construed as being included in the present disclosure.

[0029]

[0030] Figure 1 is a drawing for explaining an operation of detecting a complex event according to a conventional technology.

[0031] Referring to Figure 1, with the advancement of artificial intelligence and deep learning technologies and the strengthening of policies such as the Serious Disaster Punishment Act, interest in accident prevention in industrial settings is growing. In particular, development is actively underway on technologies that utilize artificial intelligence to detect and prevent hazardous situations in advance.

[0032] For example, technologies are being developed that capture industrial sites, such as smart CCTVs, and input the captured video data into AI models to detect various event situations. However, smart CCTVs are trained specifically for specific industrial sites, making them difficult to quickly apply to a wider range of situations. Furthermore, in situations where various events coexist with a wide field of view, such as smart CCTVs, the video data processing speed is limited in quickly detecting dangerous events. Furthermore, AI models require a massive amount of training data to detect various events, and even during actual operation, they have limitations in predicting all events in real time.

[0033] For example, when detecting complex events using an artificial intelligence model, training data suitable for the task of each event to be detected is required. In addition, when image data is input, there is a problem that performance dependency appears between each task because it has a shared network (100) structure. For example, various tasks such as a segmentation head (110), a depth head (120), an optical flow (130), and an object detection head (140) may exist, and these tasks are connected by a shared network (100) structure. Therefore, performance dependency appears between each task.

[0034] This structure increases the size of AI models and increases inference time. Specifically, to detect and prevent all the various events that can occur in industrial settings, the number of tasks can increase infinitely, leading to exponential increases in inference time.

[0035] In addition, when the system is installed in an industrial site and needs to be modified, such as when the industrial site changes (e.g., adding new equipment, changing the process line, etc.) or a new risk situation is defined, it may be necessary to retrain the large artificial intelligence model or modify all tasks by considering the interdependence through the Shared Network (100).

[0036] Therefore, there are limitations in applying artificial intelligence models in conventional technology to industrial settings.

[0037] The present disclosure, which was developed to solve the aforementioned problems, uses an artificial intelligence model to detect complex events in industrial sites, etc., and can provide increased detection speed, easy response to new situations, and quick event detection functions.

[0038] For example, the present embodiments can derive only general-purpose result values ​​from an AI model and convert them into stream data to quickly detect complex events using an event detection algorithm at the later stage. This operation allows for the addition or modification of event detection algorithms to respond to various environments, and has the advantage of eliminating unnecessary operations such as AI model modification learning. Below, the operation of the complex event detection device according to the present disclosure is described in detail with reference to the drawings.

[0039]

[0040] FIG. 2 is a diagram illustrating a configuration of a composite event detection device according to one embodiment.

[0041] Referring to FIG. 2, a complex event detection device (200) using stream data pattern analysis may include a data receiving unit (210) that receives non-standard data.

[0042] The data receiving unit (210) can receive unstructured data from one or more sensors. For example, the unstructured data may include image data generated from one or more sensors. Alternatively, the unstructured data may include various sensing information, such as temperature, humidity, and speed, received from the sensors. The unstructured data may include real-time data received continuously, or data generated periodically or upon the occurrence of an event.

[0043] Taking industrial settings as an example, images or video data captured by cameras installed in industrial sites can be considered unstructured data. Unstructured data contains a variety of information, and can be used to detect complex events. However, because unstructured data is not structured, event detection requires the use of artificial intelligence models.

[0044] Unstructured data can be received via wired and / or wireless networks. Unstructured data can be received via private networks or general-purpose networks such as LTE / NR.

[0045] The complex event detection device (200) may include a stream data conversion unit (220) that inputs non-standard data into one or more preset artificial intelligence models, converts each output data from each artificial intelligence model into stream data, and transmits the converted data to a data storage management object.

[0046] For example, the stream data conversion unit (220) can generate preprocessed data based on input data preprocessing criteria set for each of one or more pre-set artificial intelligence models, and input the preprocessed data into each artificial intelligence model to extract output data. One or more artificial intelligence models can be used for complex event detection. The artificial intelligence models can be trained in advance to be specialized for each function.

[0047] When unstructured data is received, the stream data conversion unit (220) performs the preprocessing operation used in the training of each artificial intelligence model to preprocess the unstructured data so that it can be input to each artificial intelligence model. For example, when image data is input, the stream data conversion unit (220) divides the frames of the image data and inputs them to each artificial intelligence model for image processing. In this process, the stream data conversion unit (220) preprocesses the image data so that the desired output value can be output from each artificial intelligence model.

[0048] In addition, the stream data conversion unit (220) combines metadata with the output data output through the artificial intelligence model and converts it into stream data. For example, the stream data conversion unit (220) may combine metadata including at least one of event occurrence time information, sensor identification information, frame information, and data format information with the output data. In addition, the stream data conversion unit (220) may convert the output data combined with metadata into a preset format to generate stream data. Unstructured data is continuously generated through sensors, and data can be collected through multiple sensors. Therefore, in order to process such information in real time, it is necessary to combine metadata with each artificial intelligence output data. In addition, by converting various data information into a preset format and converting it into stream data, the speed and consistency of subsequent data processing can be increased. The preset format may include, but is not limited to, JSON.

[0049] The stream data conversion unit (220) transfers stream data to a data storage management object when it is generated. For example, the data storage management object refers to a real-time data streaming service object that receives stream data, separates and stores it according to preset criteria, and provides stream data when a stream data call occurs. For example, the data storage management object may refer to a real-time data streaming service object such as Kafka or Kinesis. However, it is not limited to the services described above, and the meaning of an object may refer to not only a physical object but also a form in which multiple physical / logical objects are combined for providing a service.

[0050] Stream data is stored separately according to the criteria of each data storage management object, and the corresponding stream data can be provided when called by a consumer.

[0051] In addition, the stream data conversion unit (220) can check whether data can be retrieved from the data storage management object at preset time intervals to confirm whether stream data has been normally input into the data storage management object. For example, the stream data conversion unit (220) can perform data retrieval from the data storage management object using a consumer function to confirm whether stream data has been normally input.

[0052] Meanwhile, the complex event detection device (200) may include a stream data purification unit (230) that performs delay compensation and synchronous mapping on stream data called from a data storage management object to generate purified stream data.

[0053] The stream data purification unit (230) can perform purification operations as a preliminary task for detecting events by calling stream data stored in a data storage management object. In a situation where a large amount of stream data is collected and stored in real time, purification operations must be performed on the stream data to ensure accurate event detection.

[0054] For example, the stream data purification unit (230) can perform delay compensation by extracting event occurrence time information from the stream data and using the watermark information to compensate for the delay time required to generate each output data. Even if the stream data is generated with the same event occurrence time information, a delay may occur depending on the transmission delay and the difference in processing speed for each artificial intelligence algorithm. Therefore, in order to accurately detect an event, the stream data generated at the same time must be used. Accordingly, the stream data purification unit (230) can perform delay compensation using the event occurrence time information of the stream data (e.g., the time of generation of non-standard data). For this purpose, watermark information can be used, and the use of watermark information will be described below.

[0055] Additionally, the stream data purification unit (230) can extract frame information and event occurrence time information from the stream data to perform synchronous mapping between stream data. For example, in the case of video data, it can be continuously generated in an analog manner. Therefore, using the event occurrence time information and frame information of the stream data, stream data mapping can be performed at a more accurate point in time.

[0056] The complex event detection device (200) may include an event processing unit (240) that inputs purified stream data into one or more preset pattern algorithms to extract pattern information for each event sequence, and inputs pattern information into one or more preset complex event algorithms to extract complex event information.

[0057] For example, the event processing unit (240) can input refined purified stream data into one or more preset pattern algorithms to extract pattern information. The pattern information refers to basic information used for extracting complex events and can be defined in various ways depending on the system configuration. Once the pattern information is generated, the event processing unit (240) can use the pattern information to extract more complex complex event information. To extract the complex event information, one or more preset complex event algorithms can be used.

[0058] For example, the event processing unit (240) can extract a plurality of pattern information based on refined stream data and key factor information set for each preset pattern algorithm. For example, the event processing unit (240) can input refined stream data into a pattern algorithm set to extract simple defeat information. In this case, each pattern algorithm can have the necessary key factor information set in advance, and the event processing unit (240) can use the key factor information to distinguish and use refined stream data for each pattern algorithm. Pattern information refers to relatively simple information. For example, pattern information can refer to simple basic information such as distance information between objects in image data, distance information between people, and person posture information.

[0059] The event processing unit (240) can input pattern information into a composite event algorithm to extract more complex composite events. A composite event uses pattern information as the basis for specific situations and complex events. For example, a composite event can be an event that can be generated using pattern information, such as whether safety equipment is worn, the appropriateness of the work area, or measuring the time required for heavy work. Such composite event information can be extracted not only using pattern information but also using composite events extracted by other composite event algorithms.

[0060] For example, the event processing unit (240) can store pattern information and complex event information in a data storage management object. Once the pattern information and complex event information are extracted, they are stored again in the data storage management object and can be used to extract other complex event information.

[0061] For example, the event processing unit (240) may extract other complex events by using at least one of the N stored pattern information and M stored complex event information as input to one or more preset complex event algorithms. Here, N and M are natural numbers. Through this feedback operation, the event processing unit (240) may extract more complex complex events.

[0062] Furthermore, when adding a complex event algorithm, modifications and additions are freely possible, as previously stored pattern information and complex event information can be utilized. Furthermore, if pattern information is required, a newly defined pattern algorithm can be added. Similarly, if new complex event detection is required, a complex event algorithm can be added.

[0063] In this way, the complex event detection device (200) can detect various complex events without retraining the artificial intelligence model. Furthermore, the processing speed of the complex event detection device (200) can also be increased because it performs simple algorithmic operations at each stage.

[0064] If necessary, the complex event detection device (200) may include a monitoring unit (250) that monitors complex event information to determine whether a complex event has occurred.

[0065] For example, the monitoring unit (250) can check the event log for complex event information, and if it is determined that a complex event satisfying preset criteria has occurred according to each event level, it can generate a linked control signal. The linked control signal can also transmit a preset notification signal to a manager, the relevant worker, or the relevant work site. The monitoring unit (250) can check whether an event level exceeding the existing level has occurred by checking the event log in which the complex event information is stored at a preset cycle. The event level can be calculated according to each complex event, and can also be set by separating it according to risk level or situation classification, etc.

[0066] Through the above-described operation, the complex event detection device (200) can provide a complex event detection technology that is fast and easy to tune.

[0067] Below, the operation of a composite event detection device according to the present disclosure is described in detail with reference to various embodiments of the drawings. The description below is provided using examples for ease of understanding and is not limited to the examples.

[0068]

[0069] FIG. 3 is a diagram for explaining an operation of monitoring an event using image data according to one embodiment.

[0070] Referring to FIG. 3, the non-standard data may be image data. When image data is input, a preprocessing operation (300) may be performed to input the image data into an artificial intelligence model (310).

[0071] The AI ​​model (310) can be configured in various ways depending on the settings. For example, the AI ​​model (310) can be configured as a model for object recognition and tracking, a model for performing semantic image segmentation, a model for recognizing object behavior, and a text recognition model. Additionally, the AI ​​model (310) can be added, modified, or deleted as needed. Therefore, optimization can be performed by deploying an AI model (310) suited to each industrial site.

[0072] As described above, the stream data conversion unit can convert output data from the artificial intelligence model (310) into stream data (320). To this end, the stream data conversion unit can add metadata to each output data and change it into stream data.

[0073] For example, the stream data conversion unit can combine metadata including at least one of event occurrence time information at which image data was generated, sensor identification information for identifying the sensor that generated the image data, frame information input during the image preprocessing process, and data format information with output data output from the object recognition and tracking model.

[0074] Once output data and metadata are combined, they can be converted into a stream data format. For example, stream data can be converted into a predefined format, such as JSON. This format can also be determined by linking to a data storage management object.

[0075] Once converted to stream data, the stream data is input (330) into a data storage management object. The stream data is used for complex event detection through real-time stream data processing and complex event processing (340) according to the call. The monitoring unit can monitor events (350) by querying the complex event log.

[0076] Additionally, real-time stream data processing and / or complex event, pattern information, etc. can be fed back as stream data input and reused as input values ​​for other complex event detection.

[0077] FIG. 4 is a diagram for explaining a configuration for storing stream data in a data storage management object according to one embodiment.

[0078] Referring to Figure 4, the data storage management object can be composed of various real-time data streaming service objects. Two service objects are described here as examples. A real-time data streaming service object refers to an object that provides a system capable of collecting and processing data streams in real time.

[0079] Referring to 490 in FIG. 4, producer (400) refers to an object that generates data. There may be one or more producers (400) and they provide stream data to a data storage management object (410). Therefore, they may also be the aforementioned stream data conversion unit.

[0080] The data storage management object (410) stores stream data in each topic as it is received, and the topics can be distributed and stored across multiple brokers. Distributed topics can be called partitions.

[0081] A consumer (420) refers to an object that calls and uses stream data. The complex event detection device of the present disclosure can also be viewed as a consumer (420) in that it detects complex events using stream data stored in a data storage management object (410).

[0082] Looking at 495 in Fig. 4, producer (405) refers to an object that generates data. There may be one or more producers (405) and they provide stream data to a data storage management object (415).

[0083] Data processing of the data storage management object (415) starts at a unit called a shard, and each shard transmits stream data to a consumer (425) in an ec2 instance connected in parallel.

[0084] As shown in 490 and 495 of Figure 4, the data storage management object performs the function of storing or appropriately transmitting stream data. Any service system that provides such functions can be applied to this embodiment, and there are no limitations. For example, a Kafka or Kinesis service system can serve as the data storage management object.

[0085] Meanwhile, the stream data conversion unit can check whether data can be retrieved from the data storage management object at a preset time interval to confirm whether stream data has been normally input into the data storage management object.

[0086] For example, the stream data transformation unit can use the producer function to ingest the generated stream data into a data storage management object such as Kafka or Kinesis, and use the data record function to check whether the data can be retrieved based on a specific time to confirm whether it has been transmitted.

[0087]

[0088] FIG. 5 is a diagram for explaining a stream data processing operation according to an event occurrence time according to one embodiment.

[0089] Referring to Figure 5, since stream data has different parameter counts and inference times for each AI model (e.g., a vision deep learning model), the time it takes to be imported into the data storage management object must be taken into account, and network transmission delays may occur. To prevent this, the complex event detection device extracts the event occurrence time from each stream data during stream data purification and includes watermark information. The complex event detection device considers messages arriving later than the watermark as delayed messages and processes them.

[0090] Time in stream data processing can be defined in various ways. Event time can refer to a data-dependent timestamp, meaning the time of data generation within the data. Because it is data-dependent, the value used as the timestamp varies, but the time of the event occurrence is commonly used. Processing time refers to the time it takes for the processing engine that actually processes the data stream to process the data. In other words, it refers to the time the processing server receives and processes the stream data. Ingestion time (ingestion time) can refer to the time when data is first ingested into the stream data processing engine (e.g., a data storage management object). In other words, it can refer to the time when stream data is ingested.

[0091] For example, a source generating unstructured data, such as 500, might generate three data points at 13 seconds and 16 seconds. Assume that two data points are generated at 13 seconds. That is, two events with key "a" occur at 13 seconds, and one event occurs at 16 seconds.

[0092] If unstructured data is converted to stream data and arrives at the data storage object on time, the stream data can be collected and processed, as in 550. For example, if the window size is 10 and slides every 5 seconds, events with two a keys are collected in the first window, three events with a key are collected in the second window, and one event with a key is collected in the third window.

[0093] Thus, stream data collection and processing can operate ideally when network latency and intermediate data processing times are equal or absent. However, in reality, delays can occur due to network latency, differences in the number of parameters in the aforementioned AI model, and processing times.

[0094] FIG. 6 is a diagram for explaining an operation of refining stream data using watermark information according to one embodiment.

[0095] Referring to FIG. 6, an event having a single a key may be processed by another artificial intelligence model and may be entered into the data storage management object at a different time than an event having a key that occurred simultaneously due to a difference in processing time.

[0096] For example, if a message, such as 600, that was supposed to occur at 13 seconds arrives at 19 seconds, with a 6-second delay, this could cause problems in a time-based system. For example, the key agreement result for the first window might be 1, while the result for window 3 might be 2, resulting in a different outcome than expected. Data delays like this can result in different results than expected.

[0097] Therefore, the stream data purification unit must process stream data based on event occurrence time information. For example, the stream data purification unit can extract event occurrence time information from the stream data and purify the stream data based on this information. However, while processing stream data simply based on event occurrence time information can partially resolve problem 600 in Figure 6, it still cannot completely resolve it.

[0098] For example, when processing based on event occurrence time information, the a key that arrived delayed in window 3 may be judged as delayed and removed based on the event occurrence time information. Therefore, windows 2 and window 3 are set to the same as normal 550 processing. However, window 1 receives the a key delayed outside the window period, so the consensus result is still 1, resulting in a difference in value from window 1 of 550.

[0099] To address this, the stream data purification unit utilizes watermark information. That is, the stream data purification unit can perform delay compensation by compensating for the delay time required to generate each output data using watermark information.

[0100] For example, the event occurrence time can be extracted from stream data, such as 650. Through this, delayed data in window 3 can be removed. In addition, the watermark (630) information can be used to change the window to additionally process one a key that arrives delayed in window 1. For example, the watermark can be set to 5 seconds to allow a delay up to the watermark (630) section. In other words, the watermark represents a single timestamp. Through this watermark, the results can be processed assuming that messages delayed later than the watermark will not arrive.

[0101] Meanwhile, the stream data refinement unit extracts frame information and event occurrence time information from stream data, enabling synchronous mapping between stream data. This allows for mapping of stream data for identical unstructured data and identical preprocessed data, enabling subsequent extraction of pattern information and complex event information.

[0102] A complex event detection device can extract pattern information using a preset pattern algorithm using refined stream data. The pattern algorithm can be designed to extract each pattern piece using refined stream data of the output data of an AI model. Pattern algorithms can be added, modified, or deleted as needed.

[0103] Figure 7 is a diagram for explaining the operation of a pattern algorithm according to one embodiment.

[0104] Referring to Figure 7, an example of defining a pattern algorithm using Flink CEP code is shown.

[0105] For example, the Flink CEP library can be utilized for refined stream data processing and complex event processing. This is an example, and code structure can be structured with the same concept even when using engines with similar concepts other than the library. By pre-defining a data POJO class (700), specific data can be read from various incoming stream data and used for subsequent event processing based on internal property values. App Property is defined as a part that defines the argument values ​​to be passed during execution (710). Consumer data connector (720) contains information about input data. For example, when using Kinesis, detailed information such as Region information, data stream name, and EFO (Enhanced Fan-out) is required. CEP pattern definition and aggregation processing (730) defines the content to be extracted through a pattern algorithm. For example, detection logic is designed and defined for patterns or events to be detected using functions such as numeric counts, repetition counts, whether an event occurred within a certain period of time, and whether a subsequent event occurred. The output data stream generation unit (740) defines a Sink method for transmitting output results, such as transmitting detection results to another data storage management object (e.g., a data stream) or storing an event log in a database. The Producer data connector (750) inputs detailed settings for the defined Sink method.

[0106] You can write pattern algorithms by writing code with this structure.

[0107] FIG. 8 is a diagram for explaining a composite event detection operation according to one embodiment.

[0108] Referring to Figure 8, the complex event detection device can convert output data from an AI model into stream data and transmit it to a data storage management object. For example, an object recognition and tracking algorithm can output data at 30 frames per second (FPS). An action recognition algorithm can output data at 15 frames per second (FPS). Each output data is converted into stream data format, formatted, and then input to a data storage management object.

[0109] We assume that there are two pattern algorithms: a sitting decision algorithm and a standing decision algorithm. The sitting decision algorithm and the standing decision algorithm each configure consumer #1 to receive stream data from a data storage management object. Using the stream data, a pattern processing operation is performed to extract pattern information from the pattern algorithm. The pattern information generated by the pattern processing operation is fed back to the data storage management object for storage and management.

[0110] Additionally, a composite event algorithm can be configured to extract more complex events. For example, a sit-stand judgment algorithm and a weight-lifting judgment algorithm can be configured. Each composite event algorithm can extract the results of determining whether a sit-stand event occurred or weight-lifting occurred by calling and processing pattern information stored in the data storage management object through consumer #1. For example, the sit-stand judgment algorithm can generate an event by determining that a transition from sitting to standing or from standing to sitting within a certain time period is a sit-stand event. Furthermore, if the result of the sit-stand judgment is stored in the data storage management object, the weight-lifting judgment algorithm can count the number of times this transition occurred within a certain time period to determine whether weight-lifting occurred. The extracted output results are fed back to the data storage management object and stored, where they can be used to detect other weight-lifting events. In other words, the composite event detection device utilizes this feedback structure to generate derived events, enabling high-level event detection.

[0111] Thus, a pattern algorithm can be defined as an algorithm that directly utilizes refined stream data from the output data of an AI model. Furthermore, a complex event algorithm can refer to an algorithm that extracts output data using the output results of a pattern algorithm and / or a complex event algorithm. However, this distinction is for convenience of explanation; any algorithm that extracts events can be understood as either a pattern algorithm or a complex event algorithm.

[0112] Accordingly, the event processing unit can store pattern information and composite event information in a data storage management object. The event processing unit can extract another composite event by using at least one of the stored N pieces of pattern information and M pieces of composite event information as input to one or more preset composite event algorithms. Here, N and M are natural numbers.

[0113] Figure 9 illustrates the event detection operation using a more exemplary situation.

[0114] FIG. 9 is a diagram illustrating a step-by-step composite event detection operation according to one embodiment.

[0115] Referring to FIG. 9, the event processing unit can extract a plurality of the above pattern information based on refined stream data and key factor information set for each preset pattern algorithm.

[0116] This table details examples of sub-events and their expanded parent events based on unsafe condition and behavior detection scenarios. Key factors are selected based on the resulting images and elements used in event detection for each vision deep learning model. The first event is a low-level event detected through simple pattern logic as the output of a general-purpose vision deep learning model. By combining these low-level events, complex events suitable for the safety domain can be generated without additional training of the deep learning model for specific sites or situations.

[0117] For example, based on unsafe condition and behavior detection scenarios, detailed examples of sub-events and their extended upper-level events are explained. The result images of each vision deep learning model and the elements used in event detection are selected as key factors. The first pattern information is a low-level event detected through a simple pattern algorithm using the output of a general-purpose vision deep learning model. The composite event information is a high-level event that can be extracted through an algorithm by combining the low-level event pattern information. In other words, a composite event suitable for the safety domain can be generated by combining low-level events (pattern information) without additional training of the deep learning model for a specific site or situation.

[0118]

[0119] Through the above operations, even when the scene changes or detection of a different situation is required, complex events can be detected by utilizing the relationships between events without additional training of the deep learning model. Furthermore, according to the present disclosure, by merging multiple deep learning models and pattern algorithms, rapid event detection is possible with less computational effort compared to multi-task learning models. Furthermore, according to the present disclosure, since pattern detection and decision-making are performed using the results of multiple deep learning models, the false detection rate can be reduced through a larger amount of information. Furthermore, according to the present disclosure, a highly scalable structure can be provided that can be expanded to detect complex events by gradually increasing the event level through a feedback structure utilizing simple event output results.

[0120] Below, the aforementioned embodiments and the operation of the composite event detection device are re-explained in a time-series flowchart. Each step described below can be merged, split, or reordered. Furthermore, to avoid unnecessary duplication, specific examples are omitted, and the content described above can be applied to each step in any combination.

[0121]

[0122] FIG. 10 is a diagram for explaining a complex event detection method according to one embodiment.

[0123] Referring to FIG. 10, a complex event detection method using stream data pattern analysis includes a data receiving step of receiving unstructured data (S1000).

[0124] The data reception step may receive unstructured data from one or more sensors. For example, the unstructured data may include image data generated from one or more sensors. Alternatively, the unstructured data may include various sensing information, such as temperature, humidity, and speed, received from sensors. Unstructured data may include real-time data received continuously, or data generated periodically or upon the occurrence of an event.

[0125] Unstructured data can be received via wired and / or wireless networks. Unstructured data can be received via private networks or general-purpose networks such as LTE / NR.

[0126] The complex event detection method may include a stream data conversion step of inputting unstructured data into one or more preset artificial intelligence models, converting each output data from each artificial intelligence model into stream data, and transmitting the converted data to a data storage management object (S1010).

[0127] The stream data conversion step, when unstructured data is received, performs the preprocessing operations used in the training of each AI model to preprocess the unstructured data so that it can be input to each AI model. For example, when image data is input, the stream data conversion step divides the image data into frames and inputs them to each AI model for image processing. During this process, the stream data conversion step preprocesses the image data so that the desired output value can be output from each AI model.

[0128] Additionally, the stream data conversion step combines metadata with output data generated through an artificial intelligence model and converts it into stream data. For example, the stream data conversion step may combine metadata including at least one of event occurrence time information, sensor identification information, frame information, and data format information with the output data. Furthermore, the stream data conversion step may convert the output data combined with metadata into a preset format to generate stream data.

[0129] The stream data transformation step transfers the generated stream data to a data storage management object. For example, a data storage management object refers to a real-time data streaming service object that receives stream data, separates and stores it according to preset criteria, and provides the stream data when a call is made. For example, a data storage management object may refer to a real-time data streaming service object such as Kafka or Kinesis. Stream data is separated and stored according to the criteria of each data storage management object, and the corresponding stream data can be provided when called by a consumer.

[0130] Additionally, the stream data conversion step can check whether data is available for retrieval from the data storage management object at preset time intervals to ensure that stream data has been successfully imported into the data storage management object. For example, the stream data conversion step can use the consumer function to perform data retrieval from the data storage management object to ensure that stream data has been successfully imported.

[0131] The complex event detection method can perform a stream data purification step of generating purified stream data by performing delay compensation and synchronous mapping on stream data called from a data storage management object (S1020).

[0132] The stream data cleansing step can perform cleansing operations as a preliminary step for event detection by calling stream data stored in a data storage management object. In situations where a large amount of stream data is collected and stored in real time, cleansing operations on the stream data are essential for accurate event detection.

[0133] For example, the stream data refinement step can perform delay compensation by extracting event occurrence time information from the stream data and using the watermark information to compensate for the delay time required to generate each output data. Even if stream data is generated with the same event occurrence time information, delays may occur due to transmission delays and differences in processing speeds between AI algorithms. Therefore, for accurate event detection, stream data generated at the same time must be used. Accordingly, the stream data refinement step can perform delay compensation using the event occurrence time information of the stream data (e.g., the time of generation of unstructured data). For this purpose, watermark information can be used.

[0134] Additionally, the stream data refinement step can extract frame information and event occurrence time information from the stream data, enabling synchronous mapping between stream data. For example, video data can be generated continuously in an analog manner. Therefore, using the event occurrence time information and frame information in the stream data, more accurate stream data mapping can be performed.

[0135] The complex event detection method may include an event processing step of inputting purified stream data into one or more preset pattern algorithms to extract pattern information for each event sequence, and inputting pattern information into one or more preset complex event algorithms to extract complex event information (S1030).

[0136] For example, the event processing step can extract pattern information by inputting refined stream data into one or more preset pattern algorithms. Pattern information refers to basic information used to extract complex events and can be defined in various ways depending on the system architecture. Once pattern information is generated, the event processing step can use the pattern information to extract more complex complex event information. To extract complex event information, one or more preset complex event algorithms can be used.

[0137] For example, the event processing step can extract multiple pattern information based on refined stream data and key factor information set for each preset pattern algorithm. For example, the event processing step can input refined stream data into a pattern algorithm set to extract simple defeat information. In this case, each pattern algorithm can have the necessary key factor information set in advance, and the event processing step can use the key factor information to distinguish and use refined stream data for each pattern algorithm. Pattern information refers to relatively simple information. For example, pattern information can refer to simple basic information such as distance information between objects in image data, distance information between people, and person posture information.

[0138] The event processing stage can extract more complex composite events by inputting pattern information into a composite event algorithm. A composite event uses pattern information as the foundation to represent specific situations and complex events. For example, a composite event can be generated using pattern information, such as whether safety equipment is worn, the appropriateness of the work area, or the measurement of weight-bearing work time. This composite event information can be extracted not only from pattern information but also from composite events extracted by other composite event algorithms.

[0139] For example, the event processing step can store pattern information and composite event information in a data storage management object. Once the pattern information and composite event information are extracted, they are stored in the data storage management object again, where they can be used to extract other composite event information.

[0140] For example, the event processing step can extract other composite events by using at least one of the N stored pattern information and M pieces of composite event information as input to one or more preset composite event algorithms, where N and M are natural numbers. Through this feedback operation, the event processing step can extract more complex composite events. In addition, when adding a composite event algorithm, modification and addition are free in that the pre-stored pattern information and composite event information can be utilized. In addition, when pattern information is required, a pattern algorithm can be newly defined and added. Similarly, when new composite event detection is required, a composite event algorithm can be added.

[0141] The composite event detection method may include a monitoring step of monitoring composite event information to determine whether a composite event has occurred (S1040).

[0142] For example, the monitoring step can query the event log for complex event information and, if it determines that a complex event that satisfies preset criteria for each event level has occurred, generate a linked control signal. The linked control signal can also transmit a preset notification signal to the manager, the relevant worker, or the relevant work area. The monitoring step can query the event log containing complex event information at a preset interval to determine if an event level exceeding the existing level has occurred. The event level can be calculated for each complex event and can also be set separately based on risk level or situation classification.

[0143]

[0144] The above description is merely an illustrative example of the technical idea of ​​the present disclosure, and those skilled in the art to which the present disclosure pertains will appreciate that various modifications and variations can be made without departing from the essential characteristics of the technical idea of ​​the present disclosure. In addition, the present embodiments are not intended to limit the technical idea of ​​the present disclosure but rather to explain it, and therefore the scope of the technical idea of ​​the present disclosure is not limited by these embodiments. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included within the scope of the rights of the present disclosure.

[0145]

[0146] CROSS-REFERENCE TO RELATED APPLICATION

[0147] This patent application claims priority under 35 USC § 119(a) to Korean Patent Application No. 10-2023-0180494, filed December 13, 2023, the entire contents of which are incorporated herein by reference. Furthermore, this patent application claims priority in countries other than the United States for the same reasons, the entire contents of which are incorporated herein by reference.

Claims

1. A data receiving unit that receives non-standard data; A stream data conversion unit that inputs the above-mentioned unstructured data into one or more preset artificial intelligence models, converts each output data from each artificial intelligence model into stream data, and transmits it to a data storage management object; A stream data purification unit that performs delay compensation and synchronous mapping on the stream data called from the data storage management object to generate purified stream data; An event processing unit that inputs the purified stream data into one or more preset pattern algorithms to extract pattern information for each event sequence, and inputs the pattern information into one or more preset composite event algorithms to extract composite event information; and A complex event detection device using stream data pattern analysis, including a monitoring unit that monitors the above complex event information to determine whether a complex event has occurred.

2. In paragraph 1, The above irregular data is, A composite event detection device comprising image data generated from one or more sensors.

3. In paragraph 1, The above stream data conversion unit, A complex event detection device that generates preprocessed data according to input data preprocessing criteria set for each of the above-described one or more artificial intelligence models from the above-described unstructured data, and inputs the preprocessed data into each of the artificial intelligence models to extract the output data.

4. In paragraph 1, The above stream data conversion unit, A composite event detection device that combines metadata including at least one of event occurrence time information, sensor identification information, frame information, and data format information with the output data and converts it into a preset format to generate the stream data.

5. In paragraph 1, The above data storage management object is, A complex event detection device, which is a real-time data streaming service object that receives the above stream data, separates and stores the stream data according to preset criteria, and provides the stream data when a call for the stream data occurs.

6. In paragraph 5, The above stream data conversion unit, A composite event detection device that checks whether data can be retrieved from the data storage management object at preset time intervals to confirm whether the above stream data has been normally input into the data storage management object.

7. In paragraph 1, The above stream data purification unit, A composite event detection device that extracts event occurrence time information from the above stream data and performs the delay compensation by compensating for the delay time required to generate each output data using watermark information.

8. In paragraph 1, The above stream data purification unit, A composite event detection device that extracts frame information and event occurrence time information from the above stream data and performs synchronous mapping between the above stream data.

9. In paragraph 1, The above event processing unit, A composite event detection device that extracts a plurality of said pattern information based on the above refined stream data and key factor information set for each of the above preset pattern algorithms.

10. In paragraph 1, The above event processing unit, Store the above pattern information and the above complex event information in the data storage management object, A complex event detection device that uses at least one of the N stored pieces of pattern information and the M pieces of complex event information as input to one or more preset complex event algorithms, wherein N and M are natural numbers.

11. In paragraph 1, The above monitoring unit, A complex event detection device that generates a linked control signal when it is determined that a complex event satisfying preset criteria for each event level has occurred by querying an event log for the above complex event information.

12. In a method for detecting complex events using stream data pattern analysis, Data receiving step for receiving unstructured data; A stream data conversion step of inputting the above-mentioned unstructured data into one or more preset artificial intelligence models, converting each output data from each artificial intelligence model into stream data, and transmitting it to a data storage management object; A stream data purification step for generating purified stream data by performing delay compensation and synchronous mapping on the stream data called from the data storage management object; An event processing step of inputting the purified stream data into one or more preset pattern algorithms to extract pattern information for each event sequence, and inputting the pattern information into one or more preset composite event algorithms to extract composite event information; and A method for detecting a complex event using stream data pattern analysis, comprising a monitoring step of monitoring the above complex event information to determine whether a complex event has occurred.

13. In paragraph 12, The above stream data conversion step is, A composite event detection method for generating the stream data by combining metadata including at least one of event occurrence time information, sensor identification information, frame information, and data format information with the output data and converting it into a preset format.

14. In paragraph 12, The above stream data conversion step is, A composite event detection method for checking whether data can be retrieved from the data storage management object at preset time intervals to confirm whether the above stream data has been normally input into the data storage management object.

15. In paragraph 12, The above stream data purification step is, A composite event detection method for extracting event occurrence time information from the above stream data and performing the delay compensation by compensating for the delay time required to generate each output data using watermark information.

16. In paragraph 12, The above stream data purification step is, A composite event detection method for extracting frame information and event occurrence time information from the above stream data and performing synchronous mapping between the above stream data.

17. In paragraph 12, The above event processing step is, A composite event detection method for extracting a plurality of said pattern information based on the above refined stream data and key factor information set for each of the above preset pattern algorithms.

18. In paragraph 12, The above event processing step is, Store the above pattern information and the above complex event information in the data storage management object, A method for detecting a complex event, wherein at least one of the N stored pieces of pattern information and the M pieces of complex event information is used as input to one or more preset complex event algorithms, wherein N and M are natural numbers.

19. In paragraph 12, The above monitoring steps are: A composite event detection method for generating a linked control signal when it is determined that a composite event satisfying preset criteria has occurred according to each event level by querying an event log for the above composite event information.

Images