Method and apparatus for detecting complex events using stream data pattern analysis

Abstract

The present disclosure relates to a technology for detecting a complex event using stream data pattern analysis, and provides an apparatus and a method, the apparatus including a data receiving part, a stream data converting part for inputting unstructured data into one or more preset artificial intelligence models and converting the same into stream data, a stream data refining part for performing delay compensation and synchronization mapping on stream data called from a data storage management object to generate refined stream data, an event processing part for extracting complex event information using the refined stream data, and a monitoring part for determining whether a complex event occurs.

Description

Technical Field

[0001] This disclosure relates to a technique for detecting compound events using streaming data pattern analysis. Background Technology

[0002] With the development of artificial intelligence and deep learning technologies, image analysis technology is constantly advancing, and its application in industrial disaster prevention is increasing. With the promulgation of laws such as the "Major Disaster Punishment Law," there is a growing focus on strengthening industrial safety, and recently deployed closed-circuit television (CCTV) systems are required to possess high levels of detection capabilities, automation, and operational efficiency.

[0003] However, because the solution relies solely on a specific site and is affected by various site variables, the functionality and performance of smart CCTV have limitations. Furthermore, due to the nature of disaster cases, it is difficult to obtain large amounts of training data, making it difficult to apply deep learning algorithms that require training on large datasets.

[0004] In other words, although artificial intelligence technology is being applied to various industries, its universality is limited due to the different characteristics and requirements of each industry, and it is difficult to obtain dedicated training data for training.

[0005] Furthermore, if image data is used to detect or predict complex events through artificial intelligence models, there is also the problem of excessively long processing times for these models.

[0006] As mentioned above, although the demand for using image data to detect various events is growing in fields such as industrial sites, its practical application is still limited due to the increased processing time of artificial intelligence models, limited environmental adaptability, and difficulty in obtaining training data.

[0007] Therefore, there is a need to develop a technology that can utilize general artificial intelligence models, adapt quickly to various environments, and optimize the detected events rapidly and efficiently. Summary of the Invention

[0008] (a) Technical problems to be solved This disclosure aims to provide a technique for detecting compound events using streaming data pattern analysis.

[0009] (II) Technical Solution According to one aspect, this embodiment provides a composite event detection device that utilizes streaming data pattern analysis. The composite event detection device includes: a data receiving unit for receiving unstructured data; a streaming data conversion unit for inputting the unstructured data into one or more preset artificial intelligence models, converting the output data of each artificial intelligence model into streaming data, and transmitting it to a data storage management object; a streaming data refining unit for performing delay compensation and synchronization mapping on the streaming data called from the data storage management object to generate refined streaming data; an event processing unit for inputting the refined streaming data into one or more preset pattern algorithms to extract pattern information of each event sequence, and inputting the pattern information into one or more preset composite event algorithms to extract composite event information; and a monitoring unit for monitoring the composite event information to determine whether a composite event has occurred.

[0010] According to another aspect, a composite event detection method is provided, which utilizes streaming data pattern analysis. The composite event detection method includes: a data receiving step, receiving unstructured data; a streaming data conversion step, inputting the unstructured data into one or more preset artificial intelligence models, converting the output data of each artificial intelligence model into streaming data, and transmitting it to a data storage management object; a streaming data refining step, performing delay compensation and synchronization mapping on the streaming data retrieved from the data storage management object to generate refined streaming data; an event processing step, inputting the refined streaming data into one or more preset pattern algorithms to extract pattern information of each event sequence, and inputting the pattern information into one or more preset composite event algorithms to extract composite event information; and a monitoring step, monitoring the composite event information to determine whether a composite event has occurred.

[0011] (III) Beneficial Effects According to this disclosure, a technique for detecting compound events using streaming data pattern analysis can be provided. Attached Figure Description

[0012] Figure 1 This is a diagram used to illustrate the operation of detecting compound events according to existing technology.

[0013] Figure 2 This is a diagram illustrating the configuration of a composite event detection apparatus according to one embodiment.

[0014] Figure 3 This is a diagram illustrating the operation of monitoring events using image data according to one embodiment.

[0015] Figure 4 This is a diagram illustrating a configuration for storing streaming data in a data storage management object according to one embodiment.

[0016] Figure 5This is a diagram illustrating a streaming data processing operation based on the event occurrence time according to one embodiment.

[0017] Figure 6 This is a diagram illustrating the operation of refining streaming data using watermark information according to one embodiment.

[0018] Figure 7 This is a diagram illustrating the operation of a pattern algorithm according to one embodiment.

[0019] Figure 8 This is a diagram illustrating a composite event detection operation according to one embodiment.

[0020] Figure 9 This is a diagram illustrating a phased composite event detection operation according to one embodiment.

[0021] Figure 10 This is a schematic diagram illustrating a composite event detection method according to an embodiment. Implementation

[0022] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the exemplary accompanying drawings. When assigning reference numerals to the constituent elements of each drawing, the same reference numerals will be used as much as possible for the same constituent elements, even if they are shown in different drawings. Furthermore, when describing this embodiment, if it is determined that a detailed description of a related well-known structure or function would obscure the essence of the technical concept, its detailed description may be omitted. Expressions such as "comprising," "having," and "consisting of" used in this specification may include additional components unless the word "only" is used. When constituent elements are described in a singular form, a plural form may be included unless otherwise expressly stated.

[0023] Furthermore, in describing the constituent elements of this disclosure, terms such as first, second, A, B, (a), and (b) may be used. These terms are used only to distinguish one constituent element from other constituent elements and do not limit the nature, order, sequence, or quantity of the constituent element.

[0024] When describing the positional relationship between constituent elements, if two or more constituent elements are described as "connected," "combined," or "joined," it should be understood that the two or more constituent elements can be directly "connected," "combined," or "joined," but other constituent elements can also be further "inserted" between the two or more constituent elements to be "connected," "combined," or "joined." Here, other constituent elements can be included in one or more of the two or more constituent elements that are "connected," "combined," or "joined" to each other.

[0025] When describing time sequence relationships related to constituent elements, operating methods, or manufacturing methods, for example, when using terms such as "after," "next," "following," or "before" to describe time sequence or process sequence, non-continuous cases may also be included unless "immediately" or "directly" is used.

[0026] On the other hand, when referring to the numerical values ​​of constituent elements or their corresponding information (such as levels), even if there is no separate explicit record, it can be interpreted that the numerical values ​​or their corresponding information include the range of errors that may arise due to various factors (such as process factors, internal or external impacts, noise, etc.).

[0027] The embodiments will now be described in detail with reference to the accompanying drawings.

[0028] The apparatus and method according to this disclosure will now be described in more detail with reference to the accompanying drawings. The algorithms described below are exemplary representations, and various algorithms capable of performing the same purpose and function can be applied to this disclosure. Furthermore, it should be understood that programs capable of performing the functions of this embodiment are also included in this disclosure, as are recording media containing programs.

[0029] Figure 1 This is a diagram used to illustrate the operation of detecting compound events according to existing technology.

[0030] Reference Figure 1 With the development of artificial intelligence and deep learning technologies, and the strengthening of policies such as the "Major Disaster Punishment Law," there is an increasing focus on disaster prevention at industrial sites. In particular, the development of technologies that utilize artificial intelligence to detect and prevent dangerous situations in advance is actively underway.

[0031] For example, a technology is being developed that uses smart CCTV to capture images of industrial sites and feeds the captured data into an artificial intelligence model to detect various events. However, because smart CCTV is trained specifically for particular industrial scenarios, it is difficult to quickly apply it to a wide range of situations. Furthermore, in situations like smart CCTV, which have a wide field of view and involve multiple events simultaneously, the image data processing speed is limited in its ability to quickly detect dangerous events. Additionally, for the artificial intelligence model to detect various events, it must build a massive amount of training data, and its ability to predict all events in real time during actual operation also has limitations.

[0032] As an example, when using an artificial intelligence model to detect compound events, training data that matches the tasks for each event to be detected is required. Furthermore, when inputting image data, the shared network 100 structure introduces performance dependencies between tasks. For example, there may be various tasks such as a segmentation head 110, a depth head 120, an optical flow head 130, and an object detection head 140, all connected through the shared network 100 structure. Therefore, performance dependencies exist between the tasks.

[0033] This structure leads to massive AI models and increases inference time. In particular, the number of tasks can increase infinitely in order to detect and prevent various events that may occur in industrial settings, in which case inference time may grow exponentially.

[0034] Furthermore, once installed in an industrial setting, modifications may be required due to changes in the industrial setting (such as the addition of new equipment, changes in production lines, etc.) or the need to define new hazardous situations. This may necessitate retraining the massive AI model or modifying all tasks to account for the interdependencies generated through the shared network 100.

[0035] Therefore, the application of existing artificial intelligence models in industrial settings has limitations.

[0036] This disclosure aims to solve the above-mentioned problems. This disclosure utilizes artificial intelligence models to detect complex events in industrial settings, etc., and can provide faster detection speed, flexible response to new situations, and rapid event detection functions.

[0037] For example, in this embodiment, the artificial intelligence model can only export general result values, convert them into streaming data, and then use an event detection algorithm in the backend to quickly detect compound events. This operation allows for the addition and modification of event detection algorithms to cope with various environments, and has the advantage of eliminating the need for unnecessary operations such as modifying or retraining the artificial intelligence model. The operation of the compound event detection apparatus according to this disclosure will now be described in detail with reference to the accompanying drawings.

[0038] Figure 2 This is a diagram illustrating the configuration of a composite event detection apparatus according to one embodiment.

[0039] Reference Figure 2 The composite event detection device 200 that utilizes streaming data pattern analysis may include a data receiving unit 210 for receiving unstructured data.

[0040] The data receiving unit 210 can receive unstructured data from one or more sensors. For example, the unstructured data may include image data generated by one or more sensors. Alternatively, the unstructured data may also include various sensor information such as temperature, humidity, and speed received from the sensors. The unstructured data may include continuously received real-time data, or data generated periodically or when an event occurs.

[0041] Taking industrial sites as an example, images or video data captured by cameras installed in industrial sites can be considered unstructured data. Unstructured data contains various kinds of information and can be used to detect complex events. However, because unstructured data is not in a structured form, artificial intelligence models and other technologies must be used to perform event detection.

[0042] Unstructured data can be received via wired and / or wireless means. It can be received via private networks or general-purpose networks such as LTE / NR.

[0043] The composite event detection device 200 may include a streaming data conversion unit 220, which is used to input unstructured data into one or more preset artificial intelligence models, convert the output data of each artificial intelligence model into streaming data, and transmit it to the data storage management object.

[0044] For example, the streaming data conversion unit 220 can generate preprocessed data from unstructured data according to input data preprocessing standards set for one or more preset artificial intelligence models, and input the preprocessed data into each artificial intelligence model to extract output data. One or more artificial intelligence models can be used to detect complex events. Each artificial intelligence model can be specialized for its own function and pre-trained.

[0045] When the streaming data conversion unit 220 receives unstructured data, it performs preprocessing operations used during the training of each artificial intelligence model to preprocess the unstructured data so that it can be input into each artificial intelligence model. For example, when image data is input, the streaming data conversion unit 220 divides the image data into frames and inputs them into the artificial intelligence models used for image processing. In this process, the streaming data conversion unit 220 preprocesses the image data so that each artificial intelligence model can output the desired output value.

[0046] Furthermore, the streaming data conversion unit 220 combines the output data from the artificial intelligence model with metadata and converts it into streaming data. For example, the streaming data conversion unit 220 can combine metadata containing at least one of event occurrence time information, sensor identification information, frame information, and data format information with the output data. Additionally, the streaming data conversion unit 220 can convert the output data combined with metadata into a preset format to generate streaming data. Unstructured data is continuously generated by sensors, and data from multiple sensors can be collected. Therefore, in order to process this information in real time, it is necessary to combine each artificial intelligence output data with metadata. Furthermore, by converting various data information into a preset format and then into streaming data, the speed and consistency of subsequent data processing can be improved. The preset format can be JSON, but is not limited to this.

[0047] After the streaming data is generated, the streaming data conversion unit 220 transmits it to the data storage management object. For example, the data storage management object refers to a real-time data stream service object, which receives streaming data, stores it according to preset standards, and provides streaming data when it is requested. For example, the data storage management object can refer to real-time data stream service objects such as Kafka and Kinesis. However, it is not limited to the above services; "object" can refer not only to physical objects but also to a combination of multiple physical / logical objects to provide services.

[0048] Streaming data is stored according to the standard classification of each data storage management object, and can provide the corresponding streaming data when it is accessed by consumers.

[0049] Furthermore, to confirm whether the streaming data has been correctly introduced into the data storage management object, the streaming data conversion unit 220 can check whether the data can be retrieved from the data storage management object at a preset time period. For example, the streaming data conversion unit 220 can use the consumer function to perform a data query in the data storage management object to confirm whether the streaming data has been correctly introduced.

[0050] On the other hand, the composite event detection device 200 may include a streaming data refining unit 230, which performs delay compensation and synchronization mapping on streaming data called from the data storage management object to generate refined streaming data.

[0051] The streaming data refining unit 230 can call up streaming data stored in the data storage management object and perform refining operations as a preparatory job for event detection. In situations where large amounts of streaming data are collected and stored in real time, refining operations must be performed on the streaming data in order to accurately detect events.

[0052] For example, the streaming data refinement unit 230 can extract event occurrence time information from the streaming data and use watermark information to compensate for the delay time required to generate each output data, thereby performing delay compensation. Even if the streaming data is generated under the same event occurrence time information, delays may occur due to transmission delays and differences in the processing speed of various artificial intelligence algorithms. Therefore, in order to accurately detect events, streaming data generated at the same time must be used. Therefore, the streaming data refinement unit 230 can use the event occurrence time information of the streaming data (such as the generation time of unstructured data) to perform delay compensation. For this purpose, watermark information can be used, and the use of watermark information will be explained below.

[0053] Furthermore, the streaming data refining unit 230 can extract frame information and event occurrence time information from the streaming data to perform synchronous mapping between streaming data. For example, image data can be continuously generated in an analog manner. Therefore, by utilizing the event occurrence time information and frame information of the streaming data, streaming data mapping can be performed at more accurate time points.

[0054] The composite event detection device 200 may include an event processing unit 240, which inputs refined stream data into one or more preset pattern algorithms to extract pattern information of each event sequence, and inputs the pattern information into one or more preset composite event algorithms to extract composite event information.

[0055] For example, the event processing unit 240 can input refined stream data into one or more preset pattern algorithms to extract pattern information. Pattern information refers to basic information used for complex event extraction and can be defined in various ways according to the system architecture. After generating pattern information, the event processing unit 240 can use this pattern information to extract more complex complex event information. To extract complex event information, one or more preset complex event algorithms can be used.

[0056] For example, the event processing unit 240 can extract multiple pattern information based on refined stream data and key factor information set for each preset pattern algorithm. For instance, the event processing unit 240 can input refined stream data into a pattern algorithm set to extract simple pattern information. In this case, each pattern algorithm can pre-set the required key factor information, and the event processing unit 240 can use the key factor information to differentiate the refined stream data used by each pattern algorithm. Pattern information refers to relatively simple information. For example, pattern information can refer to simple basic information such as distance information between objects, distance information between people, and human posture information in image data.

[0057] The event processing unit 240 can input pattern information into a composite event algorithm to extract more complex composite events. Composite events refer to specific situations and complex events that use pattern information as the basis. For example, a composite event is an event that can be calculated using pattern information such as whether safety equipment is worn, the suitability of the work area, and the measurement of heavy-load operation time as input. This type of composite event information can be extracted not only using pattern information but also by further utilizing composite events extracted by other composite event algorithms.

[0058] For example, the event processing unit 240 can store pattern information and composite event information in a data storage management object. After retrieving the pattern information and composite event information, it can store them again in the data storage management object and use them to retrieve other composite event information.

[0059] For example, the event processing unit 240 can also extract other composite events by using at least one of the stored N pattern information and M composite event information as input to one or more preset composite event algorithms. Here, N and M are natural numbers. Through this feedback operation, the event processing unit 240 can extract more complex composite events.

[0060] Furthermore, the system offers flexibility in modifying and adding algorithms because it can utilize pre-stored pattern and composite event information when adding composite event algorithms. Additionally, new pattern algorithms can be defined and added when pattern information is needed. Similarly, composite event algorithms can be added when new composite events need to be detected.

[0061] As described above, the composite event detection device 200 can detect various composite events even without retraining the artificial intelligence model. Furthermore, since the composite event detection device 200 performs only simple algorithmic operations at each stage, its processing speed can also be improved.

[0062] As needed, the composite event detection device 200 may include a monitoring unit 250, which monitors composite event information to determine whether a composite event has occurred.

[0063] For example, the monitoring unit 250 can query the event log containing composite event information. When it determines that a composite event has occurred that meets the preset criteria for each event level, it can generate an associated control signal. The monitoring unit 250 can then send preset alarm signals to the administrator, relevant operators, or relevant work sites based on the associated control signals. The monitoring unit 250 can query the event log containing composite event information at preset intervals to confirm whether any event exceeding the existing level has occurred. Event levels can be calculated based on each composite event, or they can be categorized and set according to hazard level or situation classification.

[0064] Through the above operations, the composite event detection device 200 can provide a fast and easily tunable composite event detection technology.

[0065] Hereinafter, various embodiments of the operation of the composite event detection apparatus according to the present disclosure will be described in detail with reference to the accompanying drawings. The following description is merely an exemplary description for ease of understanding and is not limited to these examples.

[0066] Figure 3 This is a diagram illustrating the operation of monitoring events using image data according to one embodiment.

[0067] Reference Figure 3 Unstructured data can be image data. When image data is input, preprocessing operations (300) can be performed to input the image data into the artificial intelligence model 310.

[0068] The artificial intelligence model 310 can be configured in various ways according to settings. For example, the artificial intelligence model 310 can consist of models for object recognition and tracking, models for performing semantic image segmentation, models for recognizing object behavior, and text recognition models. Furthermore, the artificial intelligence model 310 can be added, modified, or deleted as needed. Therefore, it can be optimized by configuring artificial intelligence models 310 to suit various industry scenarios.

[0069] As described above, after the artificial intelligence model 310 calculates the output data, the streaming data conversion unit can convert it into streaming data (320). To do this, the streaming data conversion unit can add metadata to each output data to convert it into streaming data.

[0070] For example, the streaming data conversion unit can combine the output data from the object recognition and tracking model with metadata containing at least one of the following: event occurrence time information of the image data generation, sensor recognition information of the sensor used to identify the sensor that generated the image data, frame information input during image preprocessing, and data format information.

[0071] The output data, combined with metadata, can be converted into a streaming data format. For example, streaming data can be converted into a preset format such as JSON. This format can also be determined in association with a data storage management object.

[0072] After being converted into streaming data, the streaming data is input (330) to the data storage management object. Depending on the call, the streaming data undergoes real-time streaming data processing and composite event processing (340) for composite event detection. The monitoring department can monitor events (350) by querying the composite event log.

[0073] In addition, real-time streaming data processing and / or composite events, pattern information, etc., can be fed back as streaming data inputs and used again as input values ​​for the detection of other composite events.

[0074] Figure 4 This is a diagram illustrating a configuration for storing streaming data in a data storage management object according to one embodiment.

[0075] Reference Figure 4 Data storage management objects can consist of various real-time data stream service objects. Here, we will use two types of service objects as examples. Real-time data stream service objects refer to objects that provide a system capable of collecting and processing data streams in real time.

[0076] observe Figure 4 In section 490, Producer 400 refers to the object that generates data. One or more Producers 400 may exist and provide streaming data to the data storage management object 410. Therefore, Producer 400 can be the aforementioned streaming data conversion unit.

[0077] After streaming data is introduced, the data storage management object 410 stores it in various topics, which can be distributed across multiple broker nodes. Distributed topics can also be called partitions.

[0078] Consumer 420 refers to an object that invokes and uses streaming data. Since the composite event detection apparatus of this disclosure uses streaming data stored in the data storage management object 410 to detect composite events, it can also be regarded as consumer 420.

[0079] observe Figure 4 495, Producer 405 refers to the object that generates data. There can be one or more producers 405, which provide streaming data to the data storage management object 415.

[0080] Data processing of the data storage management object 415 begins with a unit called a "shard". Each shard transmits streaming data to the consumer 425 through parallel connected EC2 instances.

[0081] like Figure 4 As shown in 490 and 495, the data storage management object performs the function of storing or appropriately transmitting streaming data. Any service system that provides such functionality can be applied to this embodiment, and there is no limitation. For example, service systems like Kafka or Kinesis can perform the functions of the data storage management object.

[0082] On the other hand, in order to confirm whether the streaming data has been properly introduced into the data storage management object, the streaming data conversion unit can check whether the data can be queried from the data storage management object at a preset time period.

[0083] For example, the streaming data conversion department can use the producer function to import the generated streaming data into data storage management objects such as Kafka and Kinesis. In order to confirm whether the data has been transmitted, the data logging function can be used to confirm whether the data can be queried based on a predetermined time.

[0084] Figure 5 This is a diagram illustrating a streaming data processing operation based on the event occurrence time according to one embodiment.

[0085] Reference Figure 5 Because different AI models (such as visual deep learning models) have different numbers of parameters and inference times, the time it takes for streaming data to be introduced into the data storage and management object must be considered, and network transmission delays may occur. To prevent this, the composite event detection device extracts the event occurrence time from each stream of data during the refinement of the streaming data, and includes information about the watermark. The composite event detection device treats messages arriving after the watermark as delayed messages.

[0086] In streaming data processing, time can have multiple definitions. Event time can refer to a timestamp that depends on the data. That is, it can have the same meaning as the time the data was generated. Because it depends on the data, the specific value used as a timestamp may vary, but the time when the event occurred is the primary one used. Processing time refers to the time that the processing engine actually processes the data stream. That is, the time taken by the server to receive and process the streaming data. Collection time (introduction time) can refer to the time when data is first collected into the streaming data processing engine (such as a data storage management object). That is, it can refer to the time when the streaming data is introduced.

[0087] For example, such as Figure 5 As shown in Figure 500, the source generating the unstructured data may generate three data points at seconds 13 and 16. We can assume that two data generation events occur at second 13. That is, the first is two events with the key 'a' occurring at second 13, and the second is one event occurring at second 16.

[0088] If unstructured data is converted into streaming data and arrives at the data storage object on time, the streaming data can be like... Figure 5The 550 events shown are collected and processed. For example, if the window size is 10 and it slides once every 5 seconds, then 2 events with the 'a' key are collected in the first window, 3 events with the 'a' key are collected in the second window, and 1 event with the 'a' key is collected in the third window.

[0089] As mentioned above, the collection and processing of streaming data can ideally function if network latency and the time required for intermediate data processing are equal or nonexistent. However, in reality, latency can occur due to network latency and differences in the number of parameters and processing time of the aforementioned artificial intelligence models.

[0090] Figure 6 This is a diagram illustrating the operation of refining streaming data using watermark information according to one embodiment.

[0091] Reference Figure 6 Events with the same 'a' key are processed in another artificial intelligence model. Due to the difference in processing time, they may be introduced into data storage management objects at different times than events with the same 'a' key that occur at the same time.

[0092] For example, such as Figure 6 As shown in Figure 600, assuming a message occurring at second 13 arrives at second 19 due to a 6-second delay, problems may arise in a time-based system. As an example, the key aggregation result for window 1 is 1, while window 3's result is 2, which could lead to a different outcome than expected. As mentioned above, data delays can produce results different from what is expected.

[0093] Therefore, the streaming data refining department must process streaming data based on event occurrence time information. For example, the streaming data refining department can extract event occurrence time information from the streaming data and refine the streaming data based on this. However, simply processing streaming data based on event occurrence time information can partially solve the problem. Figure 6 The problem shown in 600 is still not completely solved.

[0094] For example, when processing based on event occurrence time information, a delayed 'a' key in window 3 might be judged as delayed and removed due to the event occurrence time information processing. Therefore, windows 2 and 3 are set to be processed the same as normal 550. However, because the 'a' key is received delayed outside the window period, the aggregation result of window 1 is still 1, which differs from the value of window 1 in 550.

[0095] To address this issue, the streaming data refining department utilizes watermarking information. Specifically, the department can use watermarking information to compensate for the latency required to generate each output data segment, thereby performing latency compensation.

[0096] For example, such as Figure 6 As shown in 650, the event occurrence time can be extracted from the streaming data. In this way, delayed data in window 3 can be removed. Furthermore, by using the watermark 630 information to modify the window, the delayed arrival of one 'a' key in window 1 can be additionally processed. For example, the watermark can be set to 5 seconds, allowing delays up to the watermark 630 interval. That is, the watermark refers to a timestamp. Using this watermark, it can be assumed that messages later than the watermark will not arrive, and the calculation results can be processed accordingly.

[0097] On the other hand, the streaming data refining unit can extract frame information and event occurrence time information from the streaming data to perform synchronous mapping between streaming data. In this way, streaming data with the same unstructured data and the same preprocessed data can be mapped for subsequent extraction of pattern information and composite event information.

[0098] The composite event detection device can utilize refined streaming data and employ pre-defined pattern algorithms to extract pattern information. The pattern algorithms can be designed to extract pattern information from refined streaming data derived from the output of artificial intelligence models. Pattern algorithms can be added, modified, or deleted as needed.

[0099] Figure 7 This is a diagram used to illustrate the operation of a pattern algorithm according to one embodiment.

[0100] Reference Figure 7 This example demonstrates the operation of the Flink CEP code definition pattern algorithm.

[0101] For example, the Flink CEP library can be used to process refined streaming data and composite events. This is just an example; even if using an engine with similar concepts outside of this library, the code configuration can be based on the same concepts. A predefined data POJO class (700) is used to read specific data from various incoming streaming data and use it for subsequent event processing based on internal attribute values. The App Property (710) defines the parameter values ​​passed during execution. The Consumer data connector (720) contains information about the input data. For example, when using Kinesis, details such as Region information, data stream name, and EFO (Enhanced Fan-out) are required. The CEP pattern definition and aggregation processing (730) section defines the content to be extracted by the pattern algorithm. For example, using functions such as digit counting, repetition measurement, whether an event occurred within a predetermined time, and whether subsequent events occurred, the detection logic is designed and defined for the patterns or events to be detected. The output data stream generation unit (740) defines the sink method for transmitting output results, such as transmitting detection results to other data storage management objects (e.g., data streams) or storing event logs in a database. The producer data connector (750) inputs detailed settings for the defined sink method.

[0102] By writing code with this configuration, pattern algorithms can be built.

[0103] Figure 8 This is a diagram illustrating a composite event detection operation according to one embodiment.

[0104] Reference Figure 8 The composite event detection device can convert the output data from the artificial intelligence model into streaming data and transmit it to the data storage management object. For example, object recognition and tracking algorithms can output data at 30 FPS, while behavior recognition algorithms can output data at 15 FPS. The output data is converted into streaming data format and then introduced into the data storage management object after format conversion.

[0105] Assume there are "sitting posture judgment algorithm" and "standing posture judgment algorithm" as pattern algorithms. Each algorithm is configured with consumer #1 to receive streaming data from the data storage management object. The streaming data is used to perform pattern processing operations to extract pattern information from the pattern algorithms. The pattern information generated by the pattern processing operations can be fed back to the data storage management object for storage and management.

[0106] Furthermore, composite event algorithms can be constructed to extract more complex events. For example, a "sitting-standing transition judgment algorithm" and a "heavy-load operation judgment algorithm" can be constructed. Each composite event algorithm retrieves and processes pattern information stored in the data storage management object through consumer #1, thereby extracting the judgment results of "whether a sitting-standing transition event has occurred" and "whether it is a heavy-load operation". For example, the sitting-standing transition judgment algorithm can determine the state of changing from "sitting down" to "standing up" or from "standing up" to "sitting down" within a predetermined time as a "sitting-standing transition event" and generate an event. In addition, after the result of the "sitting-standing transition" judgment is stored in the data storage management object, the heavy-load operation judgment algorithm can determine whether it is a heavy-load operation by counting the number of times the event occurs within a predetermined time. The output results extracted in this way can be fed back to the data storage management object for storage and used for other composite event detection. That is, the composite event detection device can use this feedback structure to generate derivative events, thereby achieving a high level of event detection.

[0107] As mentioned above, pattern algorithms can be defined as algorithms that directly utilize refined streaming data from the output data of artificial intelligence models. Furthermore, composite event algorithms can refer to algorithms that extract output data using the output results of pattern algorithms and / or composite event algorithms. However, this distinction is merely for illustrative purposes; any algorithm that extracts events can be understood as either a pattern algorithm or a composite event algorithm.

[0108] Therefore, the event processing unit can store pattern information and composite event information in a data storage management object. The event processing unit can extract another composite event by using at least one of the stored N pattern information and M composite event information as input to one or more preset composite event algorithms. Here, N and M are natural numbers.

[0109] Figure 9 Let's illustrate the event detection operation with a more exemplary scenario.

[0110] Figure 9 This is a diagram illustrating a phased composite event detection operation according to one embodiment.

[0111] Reference Figure 9 The event processing unit can extract multiple pattern information based on refined stream data and key factor information set for each preset pattern algorithm.

[0112] The following table illustrates detailed examples of lower-level events and their extended higher-level events based on insecure state and behavior detection scenarios. The resulting images from each visual deep learning model and the elements used in event detection are selected as key factors. The first event is a low-level event detected through simple pattern logic as the output of a general visual deep learning model. Composite events suitable for the security domain can be generated by combining low-level events without requiring additional training of deep learning models for specific scenarios or situations.

[0113] For example, based on unsafe conditions and behavior detection scenarios, detailed examples of low-level events and their extended high-level events are illustrated. The resulting images from various visual deep learning models and the elements used in event detection are selected as key factors. The first pattern information consists of low-level events detected using a simple pattern algorithm based on the output of a general visual deep learning model. Composite event information is high-level events extracted by an algorithm by combining pattern information that serves as low-level events. That is, without requiring additional training of deep learning models for specific scenarios or situations, composite events suitable for the security domain can be generated by combining low-level events (pattern information).

[0114] Through the above operations, even if the situation changes or needs to detect conditions different from the existing ones, composite events can be detected by utilizing the relationships between events without additional training of deep learning models. Furthermore, according to this disclosure, by merging multiple deep learning models and pattern algorithms, fast event detection can be achieved with less computation compared to multi-task learning models. Moreover, according to this disclosure, since pattern detection and decision-making are performed using the results of multiple deep learning models, the false detection rate can be reduced with more information. Furthermore, according to this disclosure, a highly scalable structure can be provided, which, by utilizing a feedback structure of simple event output results, can progressively increase the event level and expand to complex event detection.

[0115] The operation of the above embodiments and the composite event detection device will be explained again below using a timing flowchart. The steps described below can be combined, split, or changed in order. In addition, to avoid unnecessary repetition, specific embodiments will be omitted, and the above content can be applied to each step in any combination.

[0116] Figure 10 This is a schematic diagram illustrating a composite event detection method according to an embodiment.

[0117] Reference Figure 10The composite event detection method utilizing streaming data pattern analysis includes a data receiving step (S1000) for receiving unstructured data.

[0118] The data receiving step can receive unstructured data from one or more sensors. For example, unstructured data may include image data generated by one or more sensors. Alternatively, unstructured data may also include various sensory information such as temperature, humidity, and speed received from the sensors. Unstructured data may include continuously received real-time data, or data generated periodically or during events.

[0119] Unstructured data can be received via wired and / or wireless means. It can be received via private networks or general-purpose networks such as LTE / NR.

[0120] The composite event detection method may include a streaming data conversion step (S1010): inputting unstructured data into one or more preset artificial intelligence models, converting the output data of each artificial intelligence model into streaming data and transmitting it to the data storage management object.

[0121] Upon receiving unstructured data, the streaming data transformation step performs the preprocessing operations used during the training of each AI model to preprocess the unstructured data, making it suitable for input into the AI ​​models. For example, when inputting image data, the streaming data transformation step segments the image data into frames and inputs them into the AI ​​models used for image processing. During this process, the streaming data transformation step preprocesses the image data so that each AI model can output the desired values.

[0122] Furthermore, the streaming data conversion step combines the output data from the artificial intelligence model with metadata and converts it into streaming data. For example, the streaming data conversion step can combine metadata containing at least one of the following: event occurrence time information, sensor identification information, frame information, and data format information, with the output data. Additionally, the streaming data conversion step can convert the output data combined with metadata into a preset format to generate streaming data.

[0123] The streaming data transformation step transmits the generated streaming data to a data storage management object. For example, a data storage management object is a real-time data streaming service object that receives streaming data, stores it according to preset standards, and provides the streaming data when it is invoked. Examples of such data storage management objects include Kafka and Kinesis. The streaming data is stored according to the standards of each data storage management object, and can be provided to consumers when invoked.

[0124] In addition, to confirm whether the streaming data has been successfully introduced into the data storage management object, the streaming data conversion step can check whether the data can be retrieved from the data storage management object at preset time intervals. For example, the streaming data conversion step can use the consumer function to perform a data query in the data storage management object to confirm whether the streaming data has been successfully introduced.

[0125] The composite event detection method can perform a streaming data refining step (S1020): perform latency compensation and synchronization mapping on the streaming data called from the data storage management object to generate refined streaming data.

[0126] The streaming data refining step can call upon streaming data stored in the data storage management object and perform refining operations as a preparatory job for event detection. In situations where large amounts of streaming data are collected and stored in real time, refining the streaming data is essential for accurate event detection.

[0127] For example, the streaming data refinement step can extract event occurrence time information from the streaming data and use watermark information to compensate for the latency required to generate each output data, thereby performing latency compensation. Even if the streaming data is generated under the same event occurrence time, latency may occur due to transmission delays and differences in the processing speed of various artificial intelligence algorithms. Therefore, to accurately detect events, streaming data generated at the same time must be used. Thus, the streaming data refinement step can utilize the event occurrence time information of the streaming data (such as the generation time of unstructured data) to perform latency compensation. For this purpose, watermark information can be used.

[0128] Furthermore, the streaming data refining step can extract frame information and event occurrence time information from the streaming data to perform synchronous mapping between streaming data. For example, image data can be continuously generated in an analog manner. Therefore, by utilizing the event occurrence time information and frame information of the streaming data, streaming data mapping can be performed at more accurate time points.

[0129] The composite event detection method may include an event processing step (S1030): inputting refined stream data into one or more preset pattern algorithms to extract pattern information of each event sequence, and inputting the pattern information into one or more preset composite event algorithms to extract composite event information.

[0130] For example, the event processing step can input refined streaming data into one or more preset pattern algorithms to extract pattern information. Pattern information refers to the basic information used for extracting complex events and can be defined in various ways depending on the system architecture. After generating pattern information, the event processing step can use this pattern information to extract more complex complex event information. To extract complex event information, one or more preset complex event algorithms can be used.

[0131] For example, the event processing step can extract multiple pattern information based on refined stream data and key factor information set for each preset pattern algorithm. For instance, the event processing step can input refined stream data into a pattern algorithm configured to extract simple pattern information. In this case, each pattern algorithm can pre-set the required key factor information, and the event processing step can use this key factor information to differentiate the refined stream data used by each pattern algorithm. Pattern information refers to relatively simple information. For example, pattern information can refer to simple basic information such as distance information between objects, distance information between people, and human posture information in image data.

[0132] The event processing steps can input pattern information into composite event algorithms to extract more complex composite events. Composite events refer to specific situations and complex events that use pattern information as the basic information. For example, a composite event is an event that can be calculated using pattern information such as whether safety equipment is worn, the suitability of the work area, and the measurement of heavy-load operation time as input. This type of composite event information can not only be extracted using pattern information, but can also be further extracted using composite events extracted by other composite event algorithms.

[0133] For example, the event processing step can store pattern information and composite event information in a data storage management object. After retrieving the pattern information and composite event information, it can be stored again in the data storage management object and used to retrieve other composite event information.

[0134] For example, the event processing step can also extract other composite events by using at least one of the stored N pattern information and M composite event information as input to one or more pre-defined composite event algorithms. Here, N and M are natural numbers. Through this feedback operation, the event processing step can extract more complex composite events. Furthermore, since pre-stored pattern information and composite event information can be utilized when adding composite event algorithms, there is flexibility in modification and addition. Similarly, when pattern information is needed, new pattern algorithms can be defined and added. Likewise, when new composite events need to be detected, new composite event algorithms can be added.

[0135] The composite event detection method may include a monitoring step (S1040): monitoring composite event information to determine whether a composite event has occurred.

[0136] For example, the monitoring process can query the event log containing composite event information. When a composite event is determined to have occurred that meets the preset criteria for each event level, a corresponding control signal can be generated. Based on the associated control signal, preset alarm signals can be sent to the administrator, relevant operators, or relevant work sites. The monitoring process can query the event log containing composite event information at preset intervals to confirm whether an event exceeding the existing level has occurred. Event levels can be calculated based on each composite event, or they can be classified and set according to hazard level or situation classification.

[0137] The above description is merely illustrative of the technical concept of this disclosure. Those skilled in the art should understand that various modifications and variations can be made without departing from the essential characteristics of the technical concept of this disclosure. Furthermore, these embodiments are intended to illustrate the technical concept of this disclosure and not to limit it; therefore, the scope of the technical concept of this disclosure is not limited by these embodiments. The scope of protection of this disclosure should be interpreted in accordance with the appended claims, and should be construed as including all technical concepts within the equivalent scope of this disclosure.

[0138] Cross-references to related applications This application claims priority to patent application No. 10-2023-0180494, filed in Korea on December 13, 2023, pursuant to Section 119(a) of the United States Patent Act (35 U.S.SC §119(a)), the entire contents of which are incorporated herein by reference. Furthermore, for the same reason, if this application claims priority in any country other than the United States, the entire contents of that application are also incorporated herein by reference.

Claims

1. A composite event detection device that utilizes streaming data pattern analysis, the composite event detection device comprising: The data receiving unit is used to receive unstructured data; The streaming data conversion unit is used to input the unstructured data into one or more preset artificial intelligence models, convert the output data of each artificial intelligence model into streaming data, and transmit it to the data storage management object. The streaming data refining unit performs latency compensation and synchronization mapping on the streaming data called from the data storage management object to generate refined streaming data; The event processing unit inputs the refined stream data into one or more preset pattern algorithms to extract pattern information of each event sequence, and inputs the pattern information into one or more preset composite event algorithms to extract composite event information. as well as The monitoring department monitors the composite event information to determine whether a composite event has occurred.

2. The composite event detection device according to claim 1, wherein, The unstructured data includes image data generated by one or more sensors.

3. The composite event detection device according to claim 1, wherein, The streaming data conversion unit generates preprocessed data from the unstructured data according to the input data preprocessing standards set for each of the preset one or more artificial intelligence models, and inputs the preprocessed data into each artificial intelligence model to extract output data.

4. The composite event detection device according to claim 1, wherein, The streaming data conversion unit combines metadata containing at least one of event occurrence time information, sensor identification information, frame information, and data format information with the output data and converts it into a preset format to generate the streaming data.

5. The composite event detection device according to claim 1, wherein, The data storage management object is a real-time data stream service object, which is used to receive the streaming data, classify and store it according to preset standards, and provide the streaming data when it is called.

6. The composite event detection device according to claim 5, wherein, In order to confirm whether the streaming data has been properly introduced into the data storage management object, the streaming data conversion unit checks whether the data can be retrieved from the data storage management object at a preset time period.

7. The composite event detection device according to claim 1, wherein, The streaming data refining unit extracts event occurrence time information from the streaming data and uses watermark information to compensate for the delay time required to generate each output data, thereby performing the delay compensation.

8. The composite event detection device according to claim 1, wherein, The streaming data refining unit extracts frame information and event occurrence time information from the streaming data to perform synchronous mapping between the streaming data.

9. The composite event detection device according to claim 1, wherein, The event processing unit extracts multiple pattern information based on the refined stream data and key factor information set for each of the preset pattern algorithms.

10. The composite event detection device according to claim 1, wherein, The event processing unit stores the pattern information and the composite event information in the data storage management object, and uses at least one of the stored N pattern information and M composite event information as input to the one or more preset composite event algorithms, where N and M are natural numbers.

11. The composite event detection device according to claim 1, wherein, The monitoring unit queries the event log of the composite event information, and when it determines that a composite event has occurred that meets the preset standards according to each event level, it generates an associated control signal.

12. A method for detecting composite events, which utilizes streaming data pattern analysis, the method comprising: The data receiving step involves receiving unstructured data. The streaming data conversion step involves inputting the unstructured data into one or more preset artificial intelligence models, converting the output data of each artificial intelligence model into streaming data, and transmitting it to the data storage management object. The streaming data refining step involves performing latency compensation and synchronization mapping on the streaming data retrieved from the data storage management object to generate refined streaming data. The event processing step involves inputting the refined stream data into one or more preset pattern algorithms to extract pattern information of each event sequence, and inputting the pattern information into one or more preset composite event algorithms to extract composite event information. as well as The monitoring step involves monitoring the composite event information to determine whether a composite event has occurred.

13. The composite event detection method according to claim 12, wherein, The streaming data conversion step combines metadata, including at least one of event occurrence time information, sensor identification information, frame information, and data format information, with the output data and converts it into a preset format to generate streaming data.

14. The composite event detection method according to claim 12, wherein, The streaming data conversion step is to confirm whether the streaming data has been properly introduced into the data storage management object, and to confirm whether the data can be queried from the data storage management object at a preset time period.

15. The composite event detection method according to claim 12, wherein, The stream data refining step extracts event occurrence time information from the stream data and uses watermark information to compensate for the delay time required to generate each output data, thereby performing the delay compensation.

16. The composite event detection method according to claim 12, wherein, The streaming data refining step extracts frame information and event occurrence time information from the streaming data to perform synchronous mapping between the streaming data.

17. The composite event detection method according to claim 12, wherein, The event processing step extracts multiple pattern information based on the refined stream data and key factor information set for each of the preset pattern algorithms.

18. The composite event detection method according to claim 12, wherein, The event processing step stores the pattern information and the composite event information in the data storage management object, and uses at least one of the stored N pattern information and M composite event information as input to the one or more preset composite event algorithms, where N and M are natural numbers.

19. The composite event detection method according to claim 12, wherein, The monitoring step queries the event log of the composite event information, and when it is determined that a composite event has occurred that meets the preset standards according to each event level, an associated control signal is generated.

Images