Systems and methods for identifying anomalous constructs through cognitive abstraction in multi-modal sensory fusion ecosystem

Abstract

The invention provides a method for identifying an abnormal structure in real time based on multi-modal monitoring data. The identification is completed by using a multi-modal sensory and cognitive abstract module. A meta descriptor is generated for multi-modal monitoring data, and then a meta descriptor embedding for the meta descriptor is generated, where the meta descriptor embedding is used by an exception assertion model to detect an exceptional structure in the monitoring data.

Description

Technical Field

[0001] This application relates to a method for real-time identification of abnormal structures based on multi-modal monitoring data, where the identification is accomplished using a cognitive abstraction module in a multi-modal sensory integration ecosystem. Meta-descriptors are generated for the multi-modal monitoring data, and subsequently, meta-descriptor embeddings are generated using multiple algorithmic strategies for the meta-descriptors, enabling the identification of abnormal structures in the continuous stream of multi-modal monitoring data. Background Art

[0002] The task of a monitoring system is to monitor a large amount of monitoring data streams composed of video, audio, temperature, smell, air pressure, etc. Such monitoring encounters numerous challenges. A prominent problem is that a large amount of manpower is required to monitor the large number of panels, screens, and optical devices presenting this information. Human operators need to continuously observe and analyze the manifestations of each data stream, which poses huge requirements on human operators. In addition, operators are also vulnerable to human errors and fatigue, especially during long-term operations. Therefore, maintaining a fully staffed command center to handle the huge amount of information becomes impractical and costly. In addition, the huge amount of data can lead to information overload, making it extremely challenging to quickly identify and respond to critical events.

[0003] To address the challenges faced by monitoring systems in processing a large amount of and diverse monitoring information sources, those skilled in the art have proposed various methods to deal with these problems. One such method is to utilize an analysis system to perform real-time analysis on the monitoring information sources and conduct out-of-bounds escalation. In such a system, data streams from audio sensors, temperature sensors, smell sensors, air pressure sensors, and video streams will be continuously collected and monitored. When a specific event or pattern is detected in the collected data, triggers or alarms will subsequently be generated according to predefined rules or thresholds. For example, in the field of digital video monitoring, it has been proposed to use video analysis software to perform real-time object recognition, tracking, and behavior analysis. The purpose of such software is to filter out non-critical events and only present important alarms to the operator. Such software can also be adopted to analyze monitoring data including audio, temperature, smell, or air pressure measurements. However, such systems face data integration and compatibility issues because the data obtained from various sources each have their own formats and structures. In addition, setting appropriate thresholds and predefined criteria for anomaly assertions is also challenging because it requires in-depth knowledge of the system, and incorrect settings may lead to false positives or missed anomalies.

[0004] Another approach proposed by those skilled in the art is to develop an alarm system that prioritizes alarms based on the importance of the perceived events, where critical events should receive immediate attention while low-priority events can be queued for review. Another approach proposed by those skilled in the art is to simply utilize multiple panels or screen displays and video walls to integrate monitoring information sources from different sources onto a smaller number of screens, making it easier for operators to manage the simultaneous monitoring of multiple information sources.

[0005] While the above solutions provide valuable improvements to monitoring systems for handling large volumes of data streams, it should be noted that most of these solutions are fundamentally rule-based approaches. The methods and systems proposed above rely on predefined criteria and if-then-else conditions. This approach has its limitations as it is inherently reactive rather than proactive. In other words, rule-based systems can only identify events that match predefined patterns or conditions and may miss new or unexpected events that have never occurred before. Additionally, adapting these rules to changing circumstances and complex scenarios can be a challenging and resource-intensive task. Therefore, those skilled in the art have been seeking more advanced, self-adaptive and self-learning systems that can go beyond rule-based methods to effectively address the complexity of modern security and monitoring challenges. Summary of the Invention

[0006] The present disclosure has revolutionized monitoring technology by leveraging machine learning techniques to autonomously handle routine contemporary physical security events. The main objective is to reduce human involvement by empowering machines to effectively handle known scenarios or events, thereby reducing the need for continuous human supervision. The system is built to operate independently, enabling the system to manage daily or familiar events without the need for continuous human supervision.

[0007] According to one aspect, the present disclosure provides a method for identifying anomalous constructs from multi-modal Binary Large Objects (BLOBs) of monitoring data using a cognitive abstraction module. The disclosed method includes the following steps: receiving multi-modal BLOBs of monitoring data, generating a meta-descriptor for each of the multi-modal BLOBs of monitoring data, generating a cumulative frequency distribution of all meta-descriptors, generating a meta-descriptor embedding for each of the meta-descriptors, and using an anomaly assertion model to identify anomalous constructs from the meta-descriptor embeddings, and validating the anomalous constructs by asserting the anomalous constructs against the cumulative frequency distribution of all meta-descriptors.

[0008] According to another aspect, the present disclosure provides a system for identifying abnormal structures from multi-modal binary large objects (BLOBs) of monitoring data. The disclosed system includes a computer processor; and a non-volatile medium readable by the computer processor, the medium storing instructions which, when executed by the computer processor, cause the computer processor to perform the following operations: receiving multi-modal BLOBs of monitoring data; generating a meta-descriptor for each of the multi-modal BLOBs of monitoring data; generating a cumulative frequency distribution of all the generated meta-descriptors; generating a meta-descriptor embedding for each of the generated meta-descriptors; and using an anomaly assertion model to identify abnormal structures from the meta-descriptor embeddings, and validating the abnormal structures by asserting the abnormal structures against the cumulative frequency distribution of all the meta-descriptors. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Various embodiments of the present disclosure are described below with reference to the following drawings:

[0010] Figure 1 A block diagram showing a cognitive abstraction module for real-time identification of abnormal structures from monitoring data according to an embodiment of the present disclosure;

[0011] FIG. 2 shows a block diagram of a processing system for implementing an embodiment of the present disclosure;

[0012] FIG. 3 shows a flowchart of a process for real-time identification of abnormal structures based on monitoring data according to an embodiment of the present disclosure; and

[0013] FIG. 4 shows a cumulative frequency distribution graph that classifies meta-descriptors according to the occurrence of meta-descriptors and their time distribution. DETAILED DESCRIPTION

[0014] With reference to the drawings, the details and embodiments of the present disclosure are described in detail below for illustrative purposes. Features described in the context of an embodiment may be correspondingly applicable to the same or similar features in other embodiments, even if not explicitly described in these other embodiments. Additions and / or combinations and / or alternatives described for features in the context of an embodiment may be correspondingly applicable to the same or similar features in other embodiments.

[0015] In the context of various embodiments, the articles "a", "an", and "the" used with respect to a feature or element include references to one or more of the feature or element.

[0016] In the context of various embodiments, the term "about" or "approximately" applied to a numerical value includes the exact value and a reasonable range of variation commonly understood in the relevant technical field, for example, within 10% of the specified value.

[0017] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0018] As used herein, "comprising" means including but not limited to whatever follows the word "comprising". Thus, the use of the term "comprising" indicates that the listed elements are required or mandatory, but other elements are optional and may or may not be present.

[0019] As used herein, "consisting of" means including and limited to whatever follows the phrase "consisting of". Thus, the use of the phrase "consisting of" indicates that the listed elements are required or mandatory and that no other elements can be present.

[0020] As used herein, a "meta-descriptor" refers to, including but not limited to, a series of alphanumeric characters used to describe or characterize an object, document, image, data file, audio recording, or any type of digital file that can be processed by a computing device.

[0021] Furthermore, those skilled in the art will recognize that throughout the specification, certain functional units in this specification are labeled as modules. Those skilled in the art will also recognize that a module can be implemented as a circuit, a logic chip, or any kind of discrete component. Additionally, those skilled in the art will also recognize that a module can be implemented in software, which can then be executed by various processor architectures. In embodiments of the present disclosure, a module can also include computer instructions or executable code that can direct a computer processor to execute a sequence of events based on received instructions. The choice of how to implement the module is left as a design choice to those skilled in the art and does not limit the scope of the claimed subject matter in any way.

[0022] Furthermore, those skilled in the art will also recognize that the detailed working principles, internal structures, techniques, and training data sets of the machine learning models, neural networks, and anomaly assertion models mentioned in the present disclosure have not been disclosed or shown in detail because this information is known to those skilled in the art and is therefore omitted for the sake of brevity.

[0023] A command center designed to monitor a large range of information in real time continuously receives a steady stream of monitoring data, including binary large objects (BLOBs), which can include video, audio, temperature measurements, pressure measurements, odor measurements, images, text data, and / or non-text data from numerous data sources. In embodiments of the present disclosure, video data can be obtained using an image capture device, such as a surveillance camera set to capture and record real-time video information sources; audio data can be obtained using an audio capture device, such as a microphone and audio sensor set to capture real-time audio information sources, audio recordings, or ambient sound data; temperature / pressure data can be obtained through a measuring device, such as a temperature or pressure sensor set to provide real-time temperature / pressure readings; and text data can be obtained from various sources, such as reports or event logs, text messages, and / or emails.

[0024] Specifically, the video stream provides a visual perspective of the monitored area, while the audio information source provides auditory cues and context. In embodiments of the present disclosure, the audio information source can be digitally embedded in the video stream so that each BLOBs contains the same timestamp. At the same time, the temperature and pressure sensors provide critical environmental data to ensure the safety and functionality of the monitored system. In addition, the text data source provides insights and reports related to the monitoring domain. The robust setup of the command center enables the seamless integration and analysis of all these data streams, providing a comprehensive view of the monitored ecosystem to ensure that any potential anomalies or violations are not overlooked.

[0025] When an existing monitoring system detects an anomaly, it quickly triggers an alarm and notification to alert the command center operator. The operator can then immediately take action to verify the importance of the event and activate the corresponding response protocol. Depending on the nature and severity of the detected anomaly, this response may involve notifying the relevant authorities, deploying security personnel, or initiating an automated system response. The command center, as the central nervous system of the monitoring network, ensures a quick and effective response to situations that deviate from the expected standards, maintains the integrity and security of the monitored area, and effectively responds to changing circumstances.

[0026] Figure 1 A block diagram showing a cognitive abstraction module for real-time identification of abnormal structures from monitoring data according to an embodiment of the present disclosure. In embodiments of the present disclosure, the cognitive abstraction module 110 can be provided as a precursor or preparatory stage to the command center, where the BLOBs of the multimodal monitoring data are first received and processed by the cognitive abstraction module 110 before the processed data is provided to the command center.

[0027] In an embodiment of the present disclosure, the cognitive abstraction module 110 may be configured to identify abnormal constructs based on the BLOBs of the received multimodal monitoring data, where the abnormal constructs include combinations of unusual patterns, structures, or elements. It should be understood that these abnormal constructs do not necessarily correspond to a single event in the BLOBs of the monitoring data, but may include unusual associations of various types of monitoring data clusters, or groupings of various types of monitoring data points.

[0028] In an embodiment of the present disclosure, an "abnormal construct" represents a paradigm shift in how to interpret the monitoring data. Different from the traditional method of visual monitoring that only identifies objects and quantifies their occurrences, the multimodal method allows the focus to be placed on grasping the broader context, interactions, and understanding of the ongoing actions, and capturing potential situations. Therefore, it is a construct rather than an event in a background without potential situations.

[0029] This innovative method prioritizes identifying the potential meaning and context that define the events occurring in an image or video, going beyond the mechanical output of simple answers such as object counting. It delves deeper into the complexity of the visual content, focusing on the relationships, interactions, and dynamics occurring between objects and their environment.

[0030] Substantially, an "abnormal construct" encompasses a more comprehensive understanding of the interactions of the monitoring situations and events described in the available data. It enables the machine to go beyond basic object recognition and includes a profound understanding of the visual content and the activities unfolding therein. This shift in perspective has particular significance in applications where assertions about the detection of anomalies, unusual events, or complex activities are crucial.

[0031] Therefore, it can be said that the cognitive abstraction module 110 is configured to detect abnormal relationships or structures in the BLOBs of the multimodal monitoring data, which may not manifest as isolated events but as deviations resulting from various combinations of the multimodal monitoring data. In an embodiment of the present disclosure, the cognitive abstraction module 110 may be provided in the cloud, in a command center, and / or at a remote location far from the command center, and is configured to communicate with the command center via a wireless or wired connection. In other embodiments of the present disclosure, the cognitive abstraction system may include multiple cognitive abstraction modules 110 such that each of the modules is configured to process its own BLOBs of the multimodal monitoring data, thereby allowing multiple BLOBs of the multimodal monitoring data to be processed in parallel by each respective module 110 in the system.

[0032] Different from the process of separately analyzing multimodal monitoring data adopted by existing command centers, the cognitive abstraction module 110 is configured to simplify, generalize, and extract key information from diverse and complex data sources before providing this summary information to the command center. The process performed by the cognitive abstraction module 110 involves converting raw data from multiple modalities, such as video streams, image frames, audio segments, text segments, and measurement segments, into a more manageable and understandable form. Through the processing and analysis of multimodal monitoring data, the cognitive abstraction module 110 can establish relationships between objects and elements in the scene and / or activities occurring in the scene, and can understand the roles and purposes of objects and / or their interactions and behaviors.

[0033] To achieve the above object, the cognitive abstraction module 110 is configured to continuously monitor and receive BLOBs of multimodal monitoring data transmitted to the module 110 from various sources. The BLOBs of multimodal monitoring data may include, but are not limited to: a series of image frames 103 obtained from the image capture device 102, a series of audio segments 105 obtained from the audio capture device 104, a series of measurements 107 obtained from the measurement device 106, and a series of text segments 109 obtained from the computing module 108. By processing the BLOBs of monitoring data from multiple modalities, the cognitive abstraction module 110 provides information and data to the command center, so that the human operator in the command center can have a more comprehensive view of the monitored area or system, thus allowing the human operator to gain a more comprehensive understanding of the ongoing activities.

[0034] From Figure 1 As can be seen, the cognitive abstraction module 110 includes a meta-descriptor generator module 112, a meta-descriptor embedding module 114, a frequency distribution module 115, and an anomaly assertion module 116. The meta-descriptor generator module 112 is configured to receive and store BLOBs of multimodal monitoring data from various sources (image frames 103, audio segments 105, measurement results 107, text segments 109, and / or any type of digital data), and generate meta-descriptors for each different type of monitoring data as long as the monitoring data is in digital format. The meta-descriptors are then provided to the meta-descriptor embedding module 114 and the frequency distribution module 115.

[0035] In embodiments of the present disclosure, a meta-descriptor database (not shown) may be configured to receive and store meta-descriptors generated by various other technical means, and may include, but is not limited to, software modules or hardware modules configured to generate meta-descriptors from various types of digital data. The meta-descriptor database may include a server, a computing device, a cloud server, a remote database, or any other similar storage device, and may be located very close to the cognitive abstraction module 110, or may also be located at a remote location. The meta-descriptor database may then be configured to transmit the received and / or stored meta-descriptors to the cognitive abstraction module 110 for further processing as described in the following sections.

[0036] The meta-descriptor embedding module 114 is configured to create a meta-descriptor embedding for each of the generated meta-descriptors. In embodiments of the present invention, these meta-descriptor embeddings may include, but are not limited to, dense vector representations of meta-descriptors, where meta-descriptors are processed based on their semantics rather than just their individual words and / or terms. It should be noted that dense vectors include vectors that mainly contain non-zero values and are used to represent numerical data in a compact form. In the above context, dense vectors typically contain high-dimensional representations of meta-descriptors, where each dimension of the dense vector corresponds to a feature or property of the meta-descriptor. These dense vectors capture complex relationships and semantic information in the meta-descriptors, enabling the anomaly assertion module 116 to effectively process the data. For example, dense vectors can be used to represent various features of meta-descriptors, where each element has a weight assigned to a specific term or concept. When combined together, a set of dense vectors can form part of a more complex model, such as a word embedding model, where each element in the dense vector can correspond to a specific linguistic property, such as context, semantics, or syntactic information. Those skilled in the art will recognize that meta-descriptor embeddings may include other types of vectors representing meta-descriptors and are not limited to the above embodiments.

[0037] Meanwhile, the frequency distribution module 115 is configured to continuously generate a cumulative frequency distribution of all meta-descriptors generated by the meta-descriptor module 112. In embodiments of the present disclosure, the cumulative frequency distribution may be generated using a cumulative frequency distribution technique that classifies meta-descriptors based on the frequency of their occurrence. In another embodiment of the present disclosure, the frequency distribution module 115 may be configured to generate a cumulative frequency distribution of meta-descriptors and its temporal distribution using a cumulative frequency distribution technique that classifies meta-descriptors based on their occurrence and its temporal distribution. Such an embodiment, for example Figure 4As shown. The time interval can be defined in hours, minutes, or seconds according to the granularity of the timestamps of the meta-descriptors. It should be noted that the cumulative frequency distribution is generated within a large time frame, e.g., days, weeks, months, and depends on the scenario or area being monitored. The cumulative frequency distributions of all meta-descriptors can then be provided to the anomaly assertion module 116.

[0038] The anomaly assertion module 116 then identifies anomalous constructs within each time interval based on the meta-descriptor embeddings of each of the meta-descriptors received from the meta-descriptor embedding module 114, where each time interval can be defined in minutes or hours. Then, the anomaly assertion module 116 then determines whether the identified anomalous constructs are true anomalies or false positives based on the anomalous constructs and the cumulative frequency distributions of all meta-descriptors. It is worth noting that the cumulative frequency distributions of all meta-descriptors generated within a longer time frame (days, weeks, months) have a higher weight compared to the shorter time frame (minutes, hours) of the anomaly assertion module. Thus, when the anomalous constructs generated by the anomaly assertion module 116 are compared and superimposed with the cumulative frequency distributions of all meta-descriptors, this provides an insight into whether the anomalous constructs are true anomalies or false positives, thereby validating the anomalous constructs. In an embodiment of the present disclosure, the anomalous constructs can be verified by asserting the anomalous constructs according to the cumulative frequency distribution, and this can include the step of evaluating the degree of match of these anomalous constructs in the overall distribution of the data or occurrences to determine whether these anomalous constructs significantly deviate from the expected or typical patterns represented by the cumulative frequency distribution.

[0039] It is worth noting that the functions performed by the anomaly assertion module 116 are not limited to anomaly detection. After unusual patterns or data points deviating from the expected behavior are detected, the anomaly assertion module 116 verifies whether these detected anomalies are real based on the continuous frequency distributions obtained from the frequency distribution module 115. Generally speaking, the anomaly assertion completes the verification of the identified anomalies and determines their certainty. This procedure helps to distinguish true anomalies that may indicate potential problems or security risks from incorrect alarms that are merely normal fluctuations or harmless anomalies. The anomaly assertion simplifies the laborious and disruptive manual verification process when dealing with true anomalies, ultimately improving the accuracy and efficiency of the anomaly detection system.

[0040] Once this is done, the monitoring data associated with these verified anomaly constructs is then marked and / or highlighted by the anomaly assertion module 116, and the monitoring data can include video, audio, temperature measurements, pressure measurements, odor measurements, images, text data, and / or non-text data. The verified anomaly constructs and their associated monitoring data can then be provided as a data file 120 to trigger a notification or application programming interface (API) to existing monitoring techniques or devices in the command center, thereby allowing the operator in the command center to perform faster and more accurate automated event classification accordingly, without spending futile time viewing the real-time data information sources on the screens and panels.

[0041] In an embodiment of the present disclosure, the results obtained based on the remedial measures triggered or taken by a human operator for the received anomaly constructs will then be fed back to the anomaly assertion model 116 through an API call to allow for continuous self-learning of the model, thereby improving the accuracy of future anomaly construct assertions. In particular, the anomaly assertion model 116 can undergo a reinforcement learning process, where the results of the triggered or taken remedial measures are used to allow for continuous improvement of the accuracy of the anomaly assertion model 116. In an embodiment of the present disclosure, this can also be accomplished by retraining the model based on the received results.

[0042] Utilizing the cognitive abstraction module 110 as described above is beneficial because the meta-descriptors associated with various types of monitoring data are valuable sources of information that enrich the raw monitoring data stored in the data lake. Different from the raw data that usually includes a continuous stream of various types of monitoring data, the meta-descriptors can provide an environmental context, enabling a deeper understanding of the specific moments captured in the monitoring data, providing insights into each of the BLOBs captured by the monitoring data, helping to identify anomalies, and facilitating event description.

[0043] In addition, the meta-descriptors contribute to the organization and classification of related data, providing descriptions, markings, and labels for each frame and / or segment of the monitoring data. This improves the search and retrieval efficiency of the module 110, thereby improving event response, forensic analysis, and data management in the monitoring field.

[0044] In an embodiment of the present disclosure, the image capture device 102 may be configured to capture and record a real-time video information source. These video information sources may be in various formats, such as MP4, MKV, MOV, AVI, etc. The video information source may subsequently be converted into a series of image frames 103 using video processing software and may be in formats such as JPEG, TIFF, BMP, PNG, etc. During the generation of the image frames 103, metadata containing the spatio-temporal information of the generated images (e.g., time stamps or position indicators) may be obtained from the video information source and embedded into each of the image frames 103. The video information source and / or the series of image frames 103 are then transmitted to the cognitive abstraction module 110 for further processing.

[0045] In an embodiment of the present disclosure, the audio capture device 104 may be configured to capture and record a real-time audio information source. These audio information sources may be in various formats, such as MP3, WAV, AAC, etc. The audio capture device 104 may subsequently convert the audio information source into a series of audio segments using audio processing software, where each audio segment contains a predetermined period of the audio signal. Similar to the processing of image frames, during the generation of the audio segments 105, metadata containing the spatio-temporal information of the generated audio segments (e.g., time stamps or position indicators) may be obtained from the audio information source and embedded into each of the audio segments 105. The series of audio information sources and / or the series of audio segments 105 are then transmitted to the cognitive abstraction module 110 for further processing. In an embodiment of the present disclosure, the image and audio capture devices 102 and 104 may be combined to generate image frames 103 with the audio signal 105 embedded therein. The series of image frames with the embedded audio signal (along with the embedded metadata) may then be transmitted to the cognitive abstraction module 110.

[0046] In an embodiment of the present invention, the measurement device 106 may include various types of sensors configured to capture various measurements such as temperature, odor, and / or pressure readings. The captured measurements may be stored in digital format and may be embedded with metadata containing spatio-temporal information, specific environmental conditions, or events related to the measurements. The measurement device 106 may then convert the measurements into a series of measurements 107 based on the information contained in the metadata. The metadata may similarly be embedded into the series of measurements 107 and then transmitted to the cognitive abstraction module 110.

[0047] In an embodiment of the present invention, the computing module 108 may include a computing device configured to retrieve text data from messages, emails, structured and unstructured data, etc. using application programming interfaces (APIs). The computing module 108 may preprocess the text data by removing irrelevant characters and by converting the text into a standardized format. The text may then be segmented and embedded with metadata containing spatio-temporal information about the segmented text. A series of text segments 109 with embedded metadata is then transmitted to the cognitive abstraction module 110.

[0048] The meta-descriptor generator module 112 then receives multi-modal monitoring data from various sources and generates meta-descriptors for each different type of monitoring data. Notably, the meta-descriptors are generated simultaneously for each of the different BLOBs (i.e., different types of monitoring data) of the monitoring data, and the timestamp of each of the BLOBs is embedded into each of the meta-descriptors such that when the meta-descriptors are processed by the anomaly assertion module 116, the module 116 will be able to establish correlations between the various meta-descriptors.

[0049] In an embodiment of the present invention, when the monitoring data includes a sequence or series of image frames in the form of BLOBs, the meta-descriptor generator module 112 may randomly select BLOBs from the series of BLOBs and may then proceed to generate meta-descriptors for each of the randomly selected BLOBs. The meta-descriptor generator module 112 may include a generative artificial intelligence (AI) model that includes a deep learning neural network configured to generate meta-descriptors for image frames by processing visual data. The generative AI model may include, but is not limited to, a generative transformer model, a universal image-text representation model, an object-semantic alignment model, or a contrastive learning model. The detailed working principles of these models are omitted for brevity as they are known to those skilled in the art.

[0050] In an embodiment of the present disclosure, the generative AI may be configured to make full use of complex machine learning models to create new data highly similar to its training data. These models are typically based on neural networks and learn patterns and structures in large datasets during the training phase. Once trained, the generative AI model can then generate entirely new and original data samples.

[0051] For example, a generative AI model can be trained on a corpus of text documents or a dataset of images, etc. During training, the generative AI model learns the underlying structure and patterns by using neural networks such as recurrent neural networks (RNNs), convolutional neural networks (CNNs), or more recently, transformer-based models (such as GPT). Once the underlying model is trained, the generative AI model can subsequently generate new data similar to the training data. The generative model can be further adjusted based on the feedback of its output or based on an automated matrix. For the sake of brevity, the detailed working principle of the generative AI model is omitted in this disclosure.

[0052] In an embodiment of the present invention, when the monitoring data includes a sequence of audio segments or a series of audio segments in the form of BLOBs, the meta-descriptor generator module 112 can randomly select BLOBs from the series of BLOBs and can subsequently continue to generate meta-descriptors for each of these randomly selected BLOBs. The meta-descriptor generator module 112 can include a speech recognition model for transcribing the spoken content in the audio segment into text, thereby forming a set of meta-descriptors for the audio segment, and the transcribed text can be further processed using a natural language model such as, but not limited to, a language transformer to form a set of meta-descriptors for the audio segment. The language transformer can be pre-trained based on a large and detailed text corpus, thereby allowing the language transformer to capture context information, semantics, and patterns from the text provided to the language transformer. Since the detailed workings and training of these models are known to those skilled in the art, they are omitted for the sake of brevity.

[0053] In an embodiment of the present disclosure, when the monitoring data includes a sequence of measurements or a series of measurements, such as, but not limited to, temperature and pressure measurements, or a series of text segments in the form of BLOBs, the meta-descriptor generator module 112 can randomly select BLOBs from the received series of BLOBs and subsequently generate meta-descriptors for each of these randomly selected BLOBs. The meta-descriptor generator module 112 can use a natural language model such as, but not limited to, a language transformer to generate a set of meta-descriptors for each of these randomly selected elements, and since the detailed workings and training of these models are known to those skilled in the art, they are omitted for the sake of brevity.

[0054] In an embodiment of the present disclosure, the meta-descriptor generator module 112 may include any combination of meta-descriptor generation models, and the type of model used by module 112 depends on the type of monitoring data being received. It is noted that there is no requirement that the meta-descriptors must take the form of coherent or structured latent meanings, i.e., the meta-descriptors may include incoherent latent meanings. When the meta-descriptors take the form of structured latent meanings, these meta-descriptors will provide a certain degree of awareness and insight into the information being conveyed, as it includes latent meanings that may be contained in an image or audio recording. Conversely, when the meta-descriptors take the form of incoherent latent meanings, although the same degree of awareness and insight may not be obtainable from the image or recording, the module is still able to identify abnormal constructs from these incoherent latent meanings based on the analysis performed on these meta-descriptors, as described in subsequent sections.

[0055] Once the meta-descriptors are generated for each of the different types of monitoring data, the meta-descriptors are subsequently provided to the meta-descriptor embedding module 114 and the frequency distribution module 115.

[0056] In an embodiment of the present disclosure, the frequency distribution module 115 is configured to continuously generate a cumulative frequency distribution of all the meta-descriptors generated by the meta-descriptor module 112, and the cumulative frequency distribution may be a one-dimensional frequency distribution or a two-dimensional frequency distribution, where the time data of each of the meta-descriptors is also utilized.

[0057] In an embodiment of the present disclosure, the meta-descriptor embedding module 114 is configured to create a meta-descriptor embedding for each of the generated meta-descriptors. The generated meta-descriptors may be initially preprocessed by a tokenization process, a lowercasing process, and / or a text preprocessing process. A language converter may subsequently be used to convert each of the meta-descriptors into a dense vector representation, where the dense vector representation represents the semantic information of the meta-descriptor. In an embodiment of the present disclosure, the meta-descriptor embedding module 114 may be configured to generate meta-descriptor embeddings for various digital techniques for meta-descriptors that are capable of generating multimodal BLOBs. By assimilating meta-descriptors from various techniques, the meta-descriptor embedding module 114 is able to generate a comprehensive set of meta-descriptor embeddings for various types of multimodal BLOBs.

[0058] In an embodiment of the present disclosure, the anomaly assertion module 116 includes an anomaly assertion model that is configured to identify data points or patterns that significantly deviate from the "normal" or expected behavior of the meta descriptor embedding. In operation, the model will collect and process data from various sources, such as the meta descriptor embeddings generated by the meta descriptor generator module 112. Common techniques for anomaly assertion include statistical methods (e.g., Z-score), machine learning methods (e.g., One-Class Support Vector Machine), clustering methods (k-means clustering or local outlier factor models), or other deep learning methods (e.g., variational autoencoders). Since the detailed operation and training of these models are known to those skilled in the art, they are omitted for brevity.

[0059] In an embodiment of the present disclosure, the anomaly assertion module 116 may include an unsupervised algorithm integration that combines multiple unsupervised clustering models, multiple statistical models, multiple deep learning models, or combinations thereof to improve the quality and reliability of clustering results when the data source contains complex and diverse data patterns. Each algorithm in the integration may employ different techniques, initialization methods, or parameters to introduce diversity during the clustering process. The goal of the integration is to mitigate the limitations and biases of individual algorithms and provide a more robust, stable, and accurate clustering solution. This integration method typically produces better results than any individual algorithm because the collective decisions of multiple algorithms are fully utilized simultaneously to generate the results. In an embodiment of the present invention, the anomaly assertion module 116 may include any combination of the following models, but is not limited to One-Class Support Vector Machine, k-Nearest Neighbors, Support Vector Machine, Local Outlier Factor, RandomForest, Isolation Forest, Mahalanobis Distance-based model, Histogram-based Outlier Detection, and / or Clustering-based Anomaly assertion.

[0060] This integration typically involves a two-step process: aggregation and consensus. In the aggregation step, each algorithm is set to generate its clustering results, predictions, and / or scores (where the output depends on the type of algorithm used), and these clustering results, predictions, and / or scores are then combined using techniques such as voting, averaging, or probability methods. In the consensus step, the integration attempts to find a common agreement or clustering structure in the aggregated results. This consensus clustering helps filter out noise, increase the stability of the clusters, and produce a more reliable final partition of the data. In an embodiment of the present disclosure, the scores, results, and / or predictions obtained from each algorithm are stored in a database. The information stored in this database can then be used as a weighted average or directly aggregated. For example, to determine whether a meta-descriptor embedding contains a part of an abnormal structure, the results of various algorithms related to a specific meta-descriptor embedding are analyzed using weighted voting or averaging techniques.

[0061] In an embodiment of the present disclosure, the unsupervised algorithm integration may include a combination of the following models: the k-nearest neighbor (K-NN) model, which focuses on classifying data points based on the majority class of its nearest neighbors; the support vector machine (SVM) model, which aims to find an optimal hyperplane for separating different classes; and the random forest (Random Forest) model, which uses an ensemble of decision trees for prediction. The combination of these three models is advantageous because this combination enhances the accuracy and robustness of the anomaly assertion module. However, those skilled in the art will recognize that other types of machine learning models, classification models, and / or grouping techniques can be used in the unsupervised algorithm integration, and the unsupervised algorithm integration is not limited to the models and / or techniques mentioned in the present disclosure.

[0062] In an embodiment of the present disclosure, the K-NN model can be set to determine the class of a deep vector representation (such as obtained from a meta-descriptor embedding) based on the classes of its k nearest neighbors in the feature space. A value is specified for "K", which represents the number of nearest neighbors to consider when making a classification decision, and the choice of "K" affects the performance of the algorithm. The distance between data points in the feature space can be obtained using the Euclidean distance metric, which is defined as:

[0063] … (1)

[0064] where x and x′ represent two data points in the feature space, and there are "n" data points in the feature space.

[0065] The k-nearest neighbor "A" with the minimum Euclidean distance to x is then selected. To determine the class of the data point x in the K-NN classification scenario, the classes of its k nearest neighbors need to be evaluated. The data point x is then assigned to the class with the highest probability, which is calculated by counting the number of k nearest neighbors in each class. This probability function can be defined as:

[0066] … (2)

[0067] where represents the probability that the input x belongs to class j, K represents the number of nearest neighbors considered, and is an indicator function that verifies whether the class of the i-th nearest neighbor matches class j.

[0068] In an embodiment of the present disclosure, an SVM model is used to find the optimal hyperplane that separates the data into different classes. Given a set of "n" data points, "x n ", and the corresponding class labels "y i ", where x i is defined as a feature vector in a high-dimensional space, and y i is defined as for binary classification, the hyperplane can be defined by the following equation:

[0069] … (3)

[0070] where is defined as a weight factor, is defined as a bias term. The goal of the SVM model is to determine the values such that the margin is maximized, where the margin represents the perpendicular distance from the hyperplane to the nearest data point, while correctly classifying all data points according to their labels. Therefore, this optimization problem can be expressed as:

[0071]

[0072] This optimization problem can then be solved using quadratic programming techniques, where the data points closest to the hyperplane and involved in defining the hyperplane are called support vectors.

[0073] For the classification of data points, given a new data point x, the SVM model will then determine the class of the new data point x by calculating (w⋅x + b). If the result is greater than 0, the SVM model determines that the data point belongs to one class; if the result is less than 0, the data point belongs to another class. This can be expressed as:

[0074] … (4)

[0075] Note that the flexibility of the SVM model comes from using the kernel function (K(xᵢ, xⱼ)) to map data points into a high-dimensional space, which enables the SVM model to capture complex non-linear relationships, and common kernel functions may include, but are not limited to, linear, polynomial, and radial basis function (RBF).

[0076] In an embodiment of the present disclosure, the random forest algorithm is a method that combines the predictions of multiple decision trees to improve the overall accuracy and robustness of the model. It operates on a dataset containing "n" data points, each represented by a feature set. The algorithm creates a forest of decision trees through a process of resampling and random feature selection. For each tree in the forest, a random subset of the data points is sampled with replacement from the original dataset, and a random subset of the features is considered at each split. This introduces randomness and diversity into the individual trees, making them less prone to overfitting.

[0077] The decision trees in the random forest are designed to make predictions by recursively partitioning the feature space. At each node of the tree, the algorithm (from the random subset) selects the best feature and split point to minimize impurity. For classification tasks, impurity can be measured using Gini impurity or entropy; for regression tasks, impurity can be measured using mean squared error. The prediction of each tree is obtained by averaging (for regression) or voting (for classification) the results of all individual trees. Mathematically, this can be expressed as:

[0078] For classification:

[0079] … (5)

[0080] For regression:

[0081] … (6)

[0082] Where represents the prediction made by the i-th decision tree.

[0083] In embodiments of the present disclosure, to ensure that the meta - descriptor embeddings have a suitable format for each of the above - mentioned models, the embeddings can be scaled and normalized. Each of the models is then applied separately to the embeddings to identify potential anomalies. The results obtained from each method are then combined using an ensemble technique (such as but not limited to resampling, random forests, boosting, adaptive boosting, stacking, etc.) or a voting system to produce a unified anomaly score for each data point. By integrating these methods in a parallel manner (based on the timestamps of the embeddings), the advantages of each model can be exploited while compensating for their respective drawbacks. This integrated framework allows for a more comprehensive and accurate identification of anomalies in the data, leading to a more effective anomaly assertion process.

[0084] After the anomaly assertion module 116 identifies an anomalous structure based on the meta - descriptor embeddings of each of the meta - descriptors from the meta - descriptor embedding module 114, the unsupervised algorithm ensemble can utilize the cumulative frequency distribution from the frequency distribution module 115 to verify the authenticity of the identified anomalous structures to determine whether these identified anomalous structures are true anomalies or false positives. It should be noted that the anomaly assertion module 116 identifies anomalous structures from the meta - descriptor embeddings at a predetermined time interval (e.g., hours or minutes), while the cumulative frequency distribution includes the occurrences of the meta - descriptors over a larger time period (e.g., days, weeks, months). Thus, once the anomaly assertion module 116 determines the anomalous structures of the meta - descriptor set (for a specific time interval), the module 116 will then utilize the cumulative frequency distribution to evaluate whether the identified anomalous structures are indeed true anomaly assertions or false positives.

[0085] In an exemplary embodiment of the present disclosure, when the anomaly assertion module 116 determines that the meta - descriptor embedding group contains an anomalous structure, the module 116 will then retrieve the meta - descriptors associated with this meta - descriptor embedding group. Then, the module 116 uses the cumulative frequency distribution to evaluate whether the retrieved meta - descriptors are true anomalous structures or false positives. In embodiments of the present disclosure, if it is determined from the cumulative frequency distribution that these meta - descriptors are part of an existing pattern (visible from the historical data of the cumulative frequency distribution) or these meta - descriptors repeatedly appear in the cumulative frequency distribution, the retrieved meta - descriptors are determined to be false positives.

[0086] Once this is done, the monitoring data associated with these identified anomalies is then classified as anomalous structures by the anomaly assertion module 116. The anomalous structures can then be provided as a data file 120 to trigger a notification or an application programming interface (API) call to the command - center operation technology, thus allowing the human operator at the command center to perform faster automatic classification accordingly. The results obtained based on the remedial measures taken or triggered by the human operator can undergo a reinforcement learning process to allow for continuous improvement of the anomaly assertion model through API calls.

[0087] According to an embodiment of the present invention, the algorithm for real-time identification of abnormal structures based on monitoring data is summarized in Algorithm 1 below.

[0088] Algorithm 1: Cognitive Abstraction Algorithm

[0089] Data: Monitoring data (image frames, audio segments, measurements, text segments)

[0090] Result:

[0091] Monitoring data related to anomalies.

[0092] Start

[0093] 1. Generate or integrate a set of meta-descriptors for each of the BLOBs of multi-modal monitoring data.

[0094] 2. Generate meta-descriptor embeddings for each of the meta-descriptors.

[0095] 3. Continuously generate the cumulative frequency distribution of all meta-descriptors.

[0096] 4. Identify abnormal structures from the meta-descriptor embeddings.

[0097] 5. Use the cumulative frequency distribution to evaluate whether the meta-descriptors labeled as abnormal structures are truly abnormal structures or false positives.

[0098] End

[0099] According to an embodiment of the present disclosure, Figure 2 FIG. shows a block diagram representing components of a processing system 200, and the components of the processing system 200 may be provided within a cognitive abstraction module 110 or in any module provided within the cognitive abstraction module 110 to perform any of the above functions. Those skilled in the art will recognize that the exact arrangement of each processing system provided within these modules may vary, and the exact configuration of the processing system 200 may vary, and Figure 2 the arrangement shown in is provided only as an example.

[0100] In an embodiment of the present invention, the processing system 200 may include a controller 201 and a user interface 202. The user interface 202 is arranged to enable manual interaction between a user and the computing module as needed, and for this purpose includes user input instructions to provide the input / output components required to update each of these modules. Those skilled in the art will recognize that the components of the user interface 202 may vary according to the embodiment, but generally include one or more of a display 240, a keyboard 235, and an optical device 236.

[0101] The controller 201 communicates data with the user interface 202 via the bus 215, and includes a memory 220, a processor 205 mounted on a circuit board (for processing instructions and data for executing the method of this embodiment), an operating system 206, an input / output (I / O) interface 230 for communicating with the user interface 202, and a communication interface (a network card 250 in this embodiment). The network card 250 can be used, for example, to send data from these modules to other processing devices via a wired or wireless network, or to receive data via a wired or wireless network. The wireless networks that can be utilized by the network card 250 include, but are not limited to, Wi-Fi, Bluetooth, Near Field Communication (NFC), cellular networks, satellite networks, telecommunication networks, Wide Area Networks (WAN), etc.

[0102] The memory 220 and the operating system 206 communicate data with the CPU 205 via the bus 210. The memory components include volatile and non-volatile memories, and more than one of each type of memory, including Random Access Memory (RAM) 223, Read-Only Memory (ROM) 225, and a mass storage device 245, the latter including one or more Solid State Drives (SSD). Those skilled in the art will recognize that the above memory components include non-transitory computer-readable media and should be understood to encompass all computer-readable media, but not including transient propagated signals. Generally, instructions are stored in the memory components as program code, but they can also be hard-coded. The memory 220 can include kernels and / or programming modules, such as software applications that can be stored in volatile or non-volatile memory.

[0103] In this document, the term "processor" is used generically to refer to any device or component that can process such instructions, and can include: microprocessors, microcontrollers, programmable logic devices, or other computing devices. That is, the processor 205 can be provided by any suitable logic circuit for receiving inputs, processing the inputs according to instructions stored in the memory, and generating outputs (e.g., output to the memory components or the display 240). In this embodiment, the processor 205 can be a single-core or multi-core processor with an addressable space. In one example, the processor 205 can be multi-core, including, for example, an 8-core CPU. In another example, it can be a cluster of CPU cores operating in parallel to accelerate computations.

[0104] Figure 3 Illustrates a process for real-time identification of abnormal structures based on monitoring data according to an embodiment of the present disclosure, where the process 300 can be executed by a cognitive abstraction module.

[0105] Process 300 begins at step 302, where the cognitive abstraction module receives BLOBs of multimodal monitoring data from multiple monitoring sources, and the received BLOBs of multimodal monitoring data can include sequences of image frames, sequences of audio clips, sequences of measurements, and / or sequences of text.

[0106] At step 304, process 300 then generates meta-descriptors for each of the BLOBs of multimodal monitoring data, i.e., for each image frame in the sequence of image frames, each audio clip in the sequence of audio clips, each text segment in the sequence of text segments, and / or each measurement in the sequence of measurements and / or any type of digital monitoring data. In some embodiments, the BLOBs can be randomly or in a predetermined order selected from all the BLOBs of the received multimodal monitoring data. For this embodiment, process 300 will then generate meta-descriptors only for each of the selected series of BLOBs.

[0107] In embodiments of the present disclosure, process 300 can generate meta-descriptors for any BLOBs of digital data received by process 300, and this will occur at step 304. Additionally, it is noted that process 300 can utilize various methods and techniques to generate meta-descriptors of BLOBs, and these methods are not limited to the techniques mentioned in the present disclosure.

[0108] Process 300 then continuously generates a cumulative frequency distribution of all the meta-descriptors at step 306. At step 308, process 300 then generates a meta-descriptor embedding for each of the meta-descriptors. Then, process 300 will identify anomalous configurations from the generated meta-descriptor embeddings at step 310. Then, process 300 utilizes the cumulative frequency distribution generated at step 306 to determine whether the meta-descriptors labeled as anomalous configurations are true anomalous configurations or whether they are false positives. The true anomalous configurations and their labeled meta-descriptors and / or the corresponding multimodal monitoring data can then be provided by process 300 as data files to trigger a notification or an application programming interface (API) to a command center, thereby allowing a human operator at the command center to perform classification accordingly. In embodiments of the present disclosure, process 300 can receive triggers post-asserted by a human operator at the command center, and this information is used by process 300 to retrain the anomaly assertion model to improve the accuracy of the model.

[0109] In embodiments of the present disclosure, process 300 can classify the monitoring data associated with the verified anomalous configurations as anomalous monitoring data.

[0110] In an embodiment of the present disclosure, process 300 may further provide the command center with anomaly monitoring data and verified anomaly constructs related to the anomaly monitoring data, and after receiving the results obtained from the command center based on the remedial measures taken on the anomaly monitoring data and the verified anomaly constructs, process 300 may subsequently use these received results to fine-tune the anomaly assertion model.

[0111] In an embodiment of the present disclosure, when at least one of the multimodal BLOBs of the monitoring data includes a sequence of image frames, the step of generating a meta descriptor for at least one of the multimodal BLOBs of the monitoring data includes process 300 of using a generative artificial intelligence model to generate a meta descriptor for each of the image frames in the sequence of image frames.

[0112] In an embodiment of the present disclosure, the step of generating a meta descriptor for each of the image frames includes process 300 of randomly selecting image frames from the series of image frames and generating a meta descriptor for each of the randomly selected image frames.

[0113] In an embodiment of the present invention, the generative artificial intelligence model may include a generative transformer model, a general image-text representation model, an object-semantic alignment model, or a contrastive learning model.

[0114] In an embodiment of the present disclosure, the anomaly assertion model may include an unsupervised algorithm integration.

[0115] In an embodiment of the present disclosure, the unsupervised algorithm integration may include a combination of an unsupervised clustering model, a statistical model, and a deep learning model.

[0116] In an embodiment of the present disclosure, the unsupervised algorithm integration may be set by process 300 to identify anomaly constructs from the outputs of the unsupervised clustering model, the statistical model, and the deep learning model using integration techniques.

[0117] In an embodiment of the present disclosure, the sequence of image frames may include image frames obtained from a video data file.

[0118] In an embodiment of the present disclosure, when at least one of the multimodal BLOBs of the monitoring data further includes a sequence of audio segments, process 300 may subsequently use an alphanumeric character generation model to generate a meta descriptor for each of the audio segments in the sequence of audio segments.

[0119] In an embodiment of the present disclosure, when at least one of the multimodal BLOBs of the monitoring data further includes a sequence of measurements, process 300 may subsequently use an alphanumeric character generation model to generate a meta descriptor for each of the measurements.

[0120] In an embodiment of the present disclosure, the cumulative frequency distribution of all generated meta-descriptors may include the occurrences of the meta-descriptors and the temporal distribution of each of the meta-descriptors.

[0121] In an embodiment of the present invention, the anomalous construction of multiple streams of multi-modal binary large objects (BLOBs) from monitoring data may be obtained by a system comprising multiple cognitive abstraction modules through process 300, where each cognitive abstraction module is arranged to receive one of the multiple streams of multi-modal BLOBs of monitoring data, and where each cognitive abstraction module in the system is arranged to perform the processes performed by process 300.

[0122] Those skilled in the art may determine many other changes, substitutions, variations, and modifications, and the intention of this application covers all changes, substitutions, variations, and modifications within the scope defined by the appended claims.

Claims

1. A method for identifying abnormal structures from multimodal binary large objects of monitoring data using a cognitive abstraction module, the method comprising: Receiving the multimodal binary large objects of the monitoring data; Generating a meta-descriptor for each of the multimodal binary large objects of the monitoring data; Generating a cumulative frequency distribution of all the meta-descriptors; Generating a meta-descriptor embedding for each of the meta-descriptors, and Using an anomaly assertion model to identify abnormal structures from the meta-descriptor embeddings, and verifying the abnormal structures by asserting the abnormal structures against the cumulative frequency distribution of all the meta-descriptors.

2. The method according to claim 1, further comprising the following steps: Classifying the monitoring data related to the verified abnormal structures as abnormal monitoring data.

3. The method according to claim 1, wherein before the step of generating the cumulative frequency distribution of all the meta-descriptors, the method further comprises the step of receiving meta-descriptors from a meta-descriptor database.

4. The method according to claim 2, further comprising the following steps: Providing the abnormal monitoring data and the verified abnormal structures related to the abnormal monitoring data to a command center; And Receiving, from the command center, a result obtained based on a remedial measure taken for the abnormal monitoring data and the verified abnormal structures, wherein the received result is used to fine-tune the anomaly assertion model.

5. The method according to any one of claims 1 to 4, wherein when at least one of the multimodal binary large objects of the monitoring data includes an image frame sequence, the step of generating a meta-descriptor for each of the multimodal binary large objects of the monitoring data comprises the step of using a generative artificial intelligence model to generate a meta-descriptor for each frame image in the image frame sequence.

6. The method according to claim 5, wherein the step of generating a meta-descriptor for each frame image in the image frame sequence comprises the following steps: Randomly selecting image frames from the image frame series in each discrete time model; And Generating a meta-descriptor for each of the randomly selected image frames.

7. The method according to claim 5, wherein the generative artificial intelligence model comprises a generative transformer model, a general image-text representation model, an object-semantic alignment model or a contrast learning model.

8. The method according to claim 1, wherein the anomaly assertion model comprises an unsupervised algorithm ensemble.

9. The method according to claim 8, wherein the unsupervised algorithm ensemble comprises a combination of an unsupervised clustering model, a statistical model and a deep learning model.

10. The method according to claim 9, wherein the unsupervised algorithm ensemble is configured to identify abnormal structures from the outputs of the unsupervised clustering model, the statistical model and the deep learning model using an ensemble technique.

11. The method according to claim 5, wherein the image frame sequence comprises image frames obtained from a video data file.

12. The method according to claim 5, wherein when at least one of the multimodal binary large objects of the monitoring data further includes a sequence of audio segments, the step of generating a meta-descriptor for each of the multimodal binary large objects of the monitoring data includes the step of using an alphanumeric character generation model to generate a meta-descriptor for each of the audio segments in the sequence of audio segments.

13. The method according to any one of claims 5 or 12, wherein when at least one of the multimodal binary large objects of the monitoring data further includes a series of measurements, the step of generating a meta-descriptor for each of the multimodal binary large objects of the monitoring data includes the step of using an alphanumeric character generation model to generate a meta-descriptor for each of the measurements.

14. The method according to claim 1, wherein the cumulative frequency distribution of all the generated meta-descriptors includes the meta-descriptors that appear with the time distribution of each meta-descriptor.

15. A method for identifying abnormal structures from multiple multimodal binary large object streams of monitoring data using a system including a plurality of cognitive abstraction modules, wherein each cognitive abstraction module is arranged to receive one of the multiple multimodal binary large object streams of monitoring data, and wherein each cognitive abstraction module in the system is arranged to perform the method according to claim 1.

16. A system for identifying abnormal structures from multimodal binary large objects of monitoring data, the system comprising: a computer processor; and a non-volatile medium readable by the computer processor, the medium storing instructions which, when executed by the computer processor, cause the computer processor to: receive the multimodal binary large objects of the monitoring data; generate a meta-descriptor for each of the multimodal binary large objects of the monitoring data; generate a cumulative frequency distribution of all the meta-descriptors; generate a meta-descriptor embedding for each of the meta-descriptors; and use an anomaly assertion model to identify abnormal structures from the meta-descriptor embeddings, and verify the abnormal structures by asserting the abnormal structures against the cumulative frequency distribution of all the meta-descriptors.

17. The system according to claim 16, further comprising instructions that direct the computer processor to: classify the monitoring data related to the verified abnormal structures as abnormal monitoring data.

18. The system according to claim 16, wherein before the instructions direct the computer processor to generate the cumulative frequency distribution of all the meta-descriptors, the system further includes instructions that direct the computer processor to receive meta-descriptors from a meta-descriptor database.

19. The system according to claim 17, further comprising instructions that direct the computer processor to perform the following operations: provide the abnormal monitoring data and the verified abnormal structures related to the abnormal monitoring data to a command center; and Receiving, from the command center, a result obtained based on a remedial action taken with respect to the anomaly monitoring data and the verified anomaly constructs, wherein the received result is used to fine-tune the anomaly assertion model.

20. The system according to any one of claims 16 to 18, wherein when at least one of the multimodal binary large objects of the monitoring data includes a sequence of image frames, the instructions for guiding the computer processor to generate the meta-descriptor for the at least one of the multimodal binary large objects of the monitoring data include instructions for guiding the computer processor to use a generative artificial intelligence model to generate a meta-descriptor for each frame image in the sequence of image frames.

21. The system according to claim 20, wherein the instructions for guiding the computer processor to generate a meta-descriptor for each of the image frames further include instructions for guiding the computer processor to perform the following operations: Randomly selecting an image frame from the series of image frames; and Generating a meta-descriptor for each of the randomly selected image frames.

22. The system according to claim 20, wherein the generative artificial intelligence model includes a generative transformer model, a general image-text representation model, an object-semantic alignment model, or a contrastive learning model.

23. The system according to claim 16, wherein the anomaly assertion model includes an unsupervised algorithm ensemble.

24. The system according to claim 23, wherein the unsupervised algorithm ensemble includes a combination of an unsupervised clustering model, a statistical model, and a deep learning model.

25. The system according to claim 24, wherein the unsupervised algorithm ensemble is configured to identify anomaly constructs from the outputs of the unsupervised clustering model, the statistical model, and the deep learning model using an ensemble technique.

26. The system according to claim 20, wherein the sequence of image frames includes image frames obtained from a video data file.

27. The system according to claim 20, wherein when at least one of the multimodal binary large objects of the monitoring data further includes a sequence of audio segments, the instructions for guiding the computer processor to generate the meta-descriptor for the at least one of the multimodal binary large objects of the monitoring data include instructions for guiding the computer processor to use an alphanumeric character generation model to generate a meta-descriptor for each of the audio segments.

28. The system according to any one of claims 20 or 27, wherein when at least one of the multimodal binary large objects of the monitoring data further includes a sequence of measurements, the instructions for guiding the computer processor to generate the meta-descriptor for the at least one of the multimodal binary large objects of the monitoring data include instructions for guiding the computer processor to use an alphanumeric character generation model to generate a meta-descriptor for each of the measurements.

29. The system according to claim 16, wherein the cumulative frequency distribution of all the generated meta-descriptors includes the meta-descriptors that occur with the time distribution of each meta-descriptor.

30. A system infrastructure for identifying abnormal structures from multiple multi-modal binary large object streams of monitoring data, the system infrastructure including a plurality of the systems according to claim 16, wherein each of the plurality of systems is configured to receive one of the multiple multi-modal binary large object streams of the monitoring data.

Images