Anomaly detection method, electronic device, computer readable medium and computer program product

Abstract

The invention provides an anomaly detection method. The method comprises the following steps: converting to-be-detected original data into a first-order logic FOL set by adopting a large language model LLM; extracting a minimum unsatisfactory subset (MUS) of the FOL set by adopting an SMT solver, wherein the MUS is used for indicating mutually conflicting FOLs; and performing conflict set analysis based on the MUS to obtain abnormal points of the original data. The invention further provides electronic equipment, a computer readable medium and a computer program product.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method for anomaly detection, an electronic device, a computer-readable medium, and a computer program product. Background Technology

[0002] In recent years, large language models (LLMs) have made significant progress in their ability to process natural language data, including text and images.

[0003] Large language models suffer from poor accuracy when performing anomaly analysis and inference on large amounts of natural language data. Summary of the Invention

[0004] This disclosure provides a method for anomaly detection, an electronic device, a computer-readable medium, and a computer program product.

[0005] In a first aspect, embodiments of this disclosure provide a method for anomaly detection, comprising:

[0006] The Large Language Model (LLM) is used to convert the raw data to be detected into a set of first-order logical FOLs.

[0007] The minimum unsatisfiable subset (MUS) of the FOL set is extracted using the Satisfiability Modular Theory (SMT) solver. The MUS is used to indicate conflicting FOLs.

[0008] Based on the MUS, conflict set analysis is performed to obtain the outliers in the original data.

[0009] Secondly, embodiments of this disclosure provide an electronic device, which includes a memory and a processor; the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, it implements any one of the anomaly detection methods of embodiments of this disclosure.

[0010] Thirdly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the anomaly detection methods of embodiments of this disclosure.

[0011] Fourthly, embodiments of this disclosure provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the anomaly detection methods of embodiments of this disclosure.

[0012] In this embodiment, LLM is first used to convert the raw data to be detected into a set of FOOLs (Falls of Interest). Then, an SMT (Surface Mount Technology) solver is used to extract the MUS (Matches of Conflicts) of this set of FOOLs. Based on the MUS, conflict set analysis is performed to obtain outliers in the raw data. On the one hand, the natural language data understanding capabilities of LLM are used to extract information from the FOOLs without requiring deep reasoning by LLM. On the other hand, the method of using an SMT solver to extract MUS and perform conflict set analysis to determine outliers is precise and transparent, thus improving the accuracy of anomaly detection. Attached Figure Description

[0013] In the accompanying drawings of the embodiments disclosed herein:

[0014] Figure 1 A flowchart of an anomaly detection method provided in this embodiment of the disclosure;

[0015] Figure 2 A block diagram illustrating the composition of an electronic device provided in an embodiment of this disclosure;

[0016] Figure 3 A block diagram illustrating the composition of a computer-readable medium provided in accordance with embodiments of this disclosure;

[0017] Figure 4 This is a schematic diagram of an anomaly detection system provided in an embodiment of the present disclosure;

[0018] Figure 5 A schematic diagram of a main control system provided in an embodiment of this disclosure;

[0019] Figure 6 A schematic diagram illustrating a knowledge base construction process provided in an embodiment of this disclosure;

[0020] Figure 7 A schematic diagram of an anomaly analysis process provided in an embodiment of this disclosure;

[0021] Figure 8 This is a schematic diagram illustrating the relationship between data objects in the knowledge base and the reasoning conclusion base in an embodiment of this disclosure;

[0022] Figure 9 A schematic diagram illustrating another knowledge base construction process provided in this embodiment of the disclosure;

[0023] Figure 10 A schematic diagram of another anomaly analysis process provided in an embodiment of this disclosure;

[0024] Figure 11 An exemplary schematic diagram of system prompts provided in embodiments of this disclosure;

[0025] Figure 12A schematic diagram of a legal clause conflict detection system provided in this embodiment of the disclosure;

[0026] Figure 13 This illustration shows a relationship diagram of the various relational tables within the database provided in this embodiment of the disclosure;

[0027] Figure 14 A schematic diagram of reference SQL code for database table creation in an embodiment of this disclosure is shown;

[0028] Figure 15 A schematic diagram illustrating sample input data from an embodiment of this disclosure is shown.

[0029] Figure 16 An exemplary schematic diagram of system prompts in an embodiment of this disclosure is shown. Detailed Implementation

[0030] To enable those skilled in the art to better understand the technical solutions of this disclosure, the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0031] The present disclosure will be described more fully below with reference to the accompanying drawings; however, the embodiments shown may be embodied in different forms, and the present disclosure should not be construed as limited to the embodiments set forth below. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of the disclosure.

[0032] The accompanying drawings of the embodiments disclosed herein are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the detailed embodiments to explain this disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the description of the detailed embodiments with reference to the accompanying drawings.

[0033] This disclosure may be described with reference to plan and / or cross-sectional views using the ideal schematic diagrams of this disclosure. Therefore, the example illustrations may be modified according to manufacturing techniques and / or tolerances.

[0034] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0035] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. The term "and / or" as used in this disclosure includes any and all combinations of one or more of the associated enumerated entries. The singular forms "a" and "the" as used in this disclosure are also intended to include the plural forms, unless the context clearly indicates otherwise. The terms "comprising," "made of," etc., as used in this disclosure specify the presence of the stated feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0036] Unless otherwise specified, all terms used in this disclosure (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined in this disclosure.

[0037] This disclosure is not limited to the embodiments shown in the accompanying drawings, but includes modifications to the configuration based on the manufacturing process. Therefore, the areas illustrated in the drawings are schematic, and the shapes of the areas shown illustrate specific shapes of the areas of an element, but are not intended to be limiting.

[0038] Technical Terminology Explanation

[0039] Unless otherwise specified in this disclosure, the following technical terms shall be interpreted as follows:

[0040] First-order logic (FOL) is a logical system used for formal reasoning. It includes constants, variables, function symbols, predicate symbols, quantifiers (universal and existential quantifiers), and logical connectives. FOL can be divided into atomic formulas and complex formulas. Atomic formulas are formed by applying predicates to constants or variables, while complex formulas are formed by combining atomic formulas using logical connectives. FOL is widely used in mathematics, computer science, artificial intelligence, and linguistics for precise expression and reasoning.

[0041] Satisfiability Modulo Theories (SMT) solvers: SMT solvers are tools used to determine whether a logical formula is satisfiable under a specific theory (such as integers, real numbers, arrays, bit vectors, etc.). They combine Boolean satisfiability (SAT) solving techniques with specific theory solving techniques, decomposing complex logical formulas into Boolean and theoretical parts, processing them separately, and then synthesizing the results. SMT solvers are widely used in software verification, hardware verification, optimization problems, and automated reasoning, and can efficiently handle logical problems involving complex data structures and constraints.

[0042] Minimal Unsatisfiable Subset (MUS): A MUS is a subset of a set of logical formulas where each subset is unsatisfiable, meaning there is no interpretation that makes all the formulas in the subset true simultaneously. A MUS possesses the minimality property: removing any formula from the subset makes the remaining subset satisfiable. In computer science and artificial intelligence, MUS is used to optimize problems, verify and debug logical systems, and help identify and resolve inconsistencies. By finding the MUS, the core conflict of a problem can be effectively located.

[0043] Large Language Models (LLMs), such as GPT-4, have made significant progress in their ability to process natural language data, including text and images.

[0044] In Natural Language Processing (NLP) tasks, LLM can understand complex grammatical and semantic structures, process various languages, such as translation, summarization, and sentiment analysis; it can also understand and analyze image content and extract descriptive or summarizing natural language text from images.

[0045] In image recognition, LLM can identify different objects in an image and generate associated text descriptions. For example, it can identify people, animals, and objects in an image. LLM can also understand the relationships between different objects in an image; for example, it can identify whether someone is pulling a dog's leash or whether a box is placed directly above a table.

[0046] In text processing, LLM can identify named entities (such as people's names, place names, and organization names) and other important information. LLM can also extract relationships between named entities from text; for example, it can extract the brotherhood between John and Mary from the sentence "John is Mary's brother".

[0047] LLM also possesses logical reasoning, common-sense reasoning, and cross-modal reasoning capabilities. LLM can perform a certain level of logical reasoning, answer complex question-and-answer questions, or perform reasoning tasks. LLM can use common-sense knowledge learned during training to reason, such as understanding the common-sense notion that "people use umbrellas when it rains." LLM can combine information from different modalities to reason, such as inferring whether the text matches the content of an image based on a given picture and a text description.

[0048] Because of the powerful capabilities mentioned above, people are increasingly using LLM in related fields to perform tasks that could only be done manually, such as automated data annotation, knowledge graph construction, and summary analysis, thereby improving efficiency and accuracy.

[0049] In some related technologies, LLM (Limited Language Management) is used for anomaly detection in natural language data. However, when using LLM for anomaly analysis and inference on large amounts of natural language data, the following problems still exist:

[0050] LLM has limited deep reasoning capabilities: LLM is not very accurate when performing complex multi-step reasoning tasks, and its accuracy drops rapidly as the length of the context and the depth of reasoning increase.

[0051] The context length of LLM is limited: Due to internal design factors, the length of the input sequence of LLM increases approximately quadratically with memory usage. Therefore, the longer the context that LLM processes, the greater the cost (whether it is API overhead or hardware overhead when self-deploying). In practice, this often limits the maximum context length of LLM, which is generally much smaller than the amount of context data that needs to be referenced simultaneously when performing complex inference.

[0052] The analysis process of LLM is opaque: LLM has poor interpretability and transparency and the model behavior is difficult to predict. Relevant personnel cannot fully understand how such a huge model containing hundreds of billions (or even trillions) of parameters in a timely manner arrives at the final result step by step from the context. The information in this intermediate process is often necessary evidence for anomaly detection tasks. Otherwise, people cannot believe and verify the final conclusion.

[0053] LLMs suffer from hallucination: they frequently fabricate facts or arrive at logically flawed conclusions, making their answers unreliable. Therefore, if LLMs are used for anomaly detection tasks, manual verification of the final results is essential; otherwise, a large number of false positives will inevitably occur.

[0054] In summary, while current LLM systems possess powerful capabilities for processing natural language data, they suffer from limitations in anomaly analysis and reasoning, including limited depth reasoning, restricted context length, opacity, and the potential for illusions, resulting in lower accuracy in anomaly detection. However, as a software system, LLM's data processing efficiency far surpasses that of humans, making it impractical to have humans manually verify its output.

[0055] In this embodiment, LLM is first used to convert the raw data to be detected into a set of FOOLs (Falls of Interest). Then, an SMT (Surface Mount Technology) solver is used to extract the MUS (Matches of Conflicts) of this set of FOOLs. Based on the MUS, conflict set analysis is performed to obtain outliers in the raw data. On the one hand, the natural language data understanding capabilities of LLM are used to extract information from the FOOLs without requiring deep reasoning by LLM. On the other hand, the method of using an SMT solver to extract MUS and perform conflict set analysis to determine outliers is precise and transparent, thus improving the accuracy of anomaly detection.

[0056] In a first aspect, embodiments of this disclosure provide a method for anomaly detection.

[0057] The method of this disclosure is performed by an electronic device to achieve anomaly detection of the raw data to be detected.

[0058] In this embodiment of the disclosure, the electronic device can be an independent server or a server cluster consisting of multiple servers.

[0059] In this embodiment of the disclosure, the electronic device can communicate with other devices, such as through Long Term Evolution (LTE) systems, Universal Mobile Telecommunications System (UMTS), 5G New Radio (NR) systems and their evolved communication systems, and also through wireless networks such as Wi-Fi. This application does not limit the communication methods between the electronic device and other devices.

[0060] Reference Figure 1 The anomaly detection method of this disclosure embodiment may include:

[0061] S101, LLM is used to convert the raw data to be detected into a FOL set.

[0062] The raw data to be detected can represent natural language data that requires anomaly detection, such as text and images. The raw data can be natural language data from various scenarios, such as legal documents, medical diagnostic records, contracts, and interrogation transcripts. This disclosure does not limit the raw data to be detected.

[0063] Raw data can be used as input data for an LLM. LLM possesses natural language data understanding and controllable generation capabilities, thus enabling information extraction. In this embodiment, an LLM can extract Forms of Analytical Writing (FOLs) from raw data. An electronic device can use an LLM to convert the raw data to be detected into one or more FOLs, which can then form a set of FOLs.

[0064] In this embodiment of the disclosure, the electronic device first inputs raw data into the LLM, which then outputs a set of FOLs (Outliers). For each batch of raw data input into the LLM by the electronic device, the LLM outputs a set of FOLs. The electronic device can then further process the FOL sets to identify outliers in the raw data.

[0065] As can be seen, in this embodiment of the disclosure, the information extraction capability of LLM is used to extract the FOL set. However, in the subsequent further processing of the FOL set, LLM is no longer involved, meaning there is no need to use LLM for deep inference. Therefore, this embodiment of the disclosure has low requirements for LLM capabilities, allowing the use of LLMs with weaker capabilities and smaller parameter sizes. This does not affect the accuracy of anomaly detection and reduces implementation costs.

[0066] Furthermore, since the LLM is no longer involved in the subsequent processing of the FOL set, the LLM is not required to handle context reasoning in this embodiment. Therefore, when processing data in batches, the amount of data in each batch (i.e., the amount of original data input to the LLM each time) can be small, thus reducing implementation costs. This embodiment also eliminates the need for retraining or fine-tuning the LLM, further reducing implementation costs.

[0067] It should be understood that in this embodiment, the LLM is treated as a black box, meaning its internal implementation, interface, and deployment method are not of concern. The selection of the LLM, interface calls, parameter configuration, and deployment method in this embodiment can be referred to relevant technologies, and will not be repeated here.

[0068] S102, use the SMT solver to extract the MUS of the FOL set.

[0069] MUS can be used to indicate conflicting FOLs. A MUS can include two or more FOLs that are logically conflicting.

[0070] In this embodiment of the disclosure, the electronic device can use an SMT solver to extract the MUS of the FOL set. The FOL set includes one or more FOLs, which may or may not be logically consistent. The SMT solver can efficiently analyze whether a large number of FOLs are logically consistent and extract logically inconsistent (i.e., conflicting) FOLs separately to form the MUS. Therefore, the MUS can be used to indicate conflicting FOLs. The process of extracting the MUS of the FOL set using the SMT solver can be referred to related technologies and will not be elaborated here.

[0071] In this embodiment, the SMT solver is used to extract the MUS of the FOL set. This process is precise and transparent, allowing users to directly observe how the solver reasones step by step to arrive at the final conclusion. Users can also reverse-index the relevant original data from the FOLs contained in the MUS and make manual judgments based on the analysis conclusions, making the subsequent reasoning results precise, transparent, and easy to verify.

[0072] S103, perform conflict set analysis based on the MUS to obtain the outliers in the original data.

[0073] In this embodiment of the disclosure, the electronic device can perform collision set analysis based on MUS to obtain outliers in the original data.

[0074] It should be understood that in many cases, a few anomalous Forms of Logic (FOLs) can lead to a large number of logical conflicts, and these FOLs are what are known as root cause data. One root cause data can lead to multiple Conflict Sets (MUSs), and the root cause data is also a member of these MUSs. Therefore, electronic devices can perform conflict set analysis based on MUSs, statistically analyze the FOLs contained in the MUSs, and thus reverse-engineer the root cause data. Since there is a correspondence between the original data and the FOL set, finding the root cause data allows us to locate the original data that exhibited the anomaly, thereby obtaining the anomaly point in the original data.

[0075] In this embodiment, LLM is first used to convert the raw data to be detected into a set of FOOLs (Falls of Interest). Then, an SMT (Surface Mount Technology) solver is used to extract the MUS (Matches of Conflicts) of this set of FOOLs. Based on the MUS, conflict set analysis is performed to obtain outliers in the raw data. On the one hand, the natural language data understanding capabilities of LLM are used to extract information from the FOOLs without requiring deep reasoning by LLM. On the other hand, the method of using an SMT solver to extract MUS and perform conflict set analysis to determine outliers is precise and transparent, thus improving the accuracy of anomaly detection.

[0076] In practical applications, due to the massive scale of data and the intricate underlying logical relationships, it is often difficult to predict the general situation of anomalies in advance, making it difficult to initiate appropriate query requests to the LLM (vague requests can only yield vague answers, while precise requests place too high demands on humans). However, the embodiments disclosed in this disclosure complete autonomous reasoning without human intervention, automatically providing users with valuable anomaly information, demonstrating a high degree of autonomy.

[0077] In this embodiment of the disclosure, after the electronic device acquires the raw data, it first needs to use LLM to convert the raw data into a FOL set. The following conversion process is described. In some embodiments, converting the raw data to be detected into a FOL set using LLM (S101) may include:

[0078] S1011, Obtain a system prompt word that matches the data type of the original data.

[0079] As one embodiment of this disclosure, a system prompt word can be used to indicate the method of extracting FOLs, and can serve as behavior control data for LLM. The electronic device can first obtain a system prompt word that matches the data type of the original data, and then convert the original data into a set of FOLs according to the method of extracting FOLs indicated by the system prompt word.

[0080] As one embodiment of this disclosure, the data type of the raw data can be preset as needed. For example, the data type may include text and images. In this case, text corresponds to a system prompt word, and images correspond to a system prompt word. As another example, the data type may include legal documents, medical diagnostic records, and contract documents. In this case, legal documents correspond to a system prompt word, and medical diagnostic records correspond to a system prompt word. It should be understood that the above are merely exemplary descriptions of data types and are not intended to limit the data types.

[0081] As one embodiment of this disclosure, system prompts may include: data processing requirements, input examples, and output examples.

[0082] The data processing requirements can be used to instruct the LLM on data restrictions during data processing, such as data type, internal data format specifications, and FOL conversion instructions. An input example and an output example can be referred to as an input example and output example pair. System prompts can include one or more input example and output example pairs. A system prompt containing one input example and output example pair can be called "One-Shot-Learning," and a system prompt containing multiple input example and output example pairs can be called "Few-Shot-Learning." Exemplary examples of system prompts will be provided later and will not be repeated here.

[0083] S1012, input the original data and the system prompt words into the LLM, and output the FOL set corresponding to the original data.

[0084] Electronic devices can input raw data and system prompts matching its data type into an LLM (Local Management Module). The LLM then outputs a set of Forms of Entries (FOLs) corresponding to the raw data. The system prompts, once input into the LLM, help it quickly adjust to the raw data to be processed, thus improving the analysis of the raw data and outputting the FOL set in a standardized format for easier subsequent processing.

[0085] As one embodiment of this disclosure, by informing the LLM of the data processing requirements, such as the data type (e.g., text or image), data structure (e.g., what objects it may contain), and general application scenario (e.g., medical diagnosis or student information), in the system prompts, the LLM can better understand the data and complete the information extraction and transformation work with high quality, which is beneficial to improving the accuracy and efficiency of anomaly detection.

[0086] In this embodiment of the disclosure, after the electronic device obtains the FOL set, it needs to use an SMT solver to extract the MUS of the FOL set. This extraction process is described below. In some embodiments, extracting the MUS of the FOL set using an SMT solver (S102) may include:

[0087] S1021, In the knowledge base, the original data and the FOL set corresponding to the original data are stored together.

[0088] As one embodiment of this disclosure, the electronic device may first associate the raw data and the corresponding set of FOLs, and then store the association between the two in a knowledge base. As the amount of raw data processed by the electronic device increases, the number of FOL sets also increases, as does the amount of raw data and FOL sets stored in the knowledge base. Subsequently, the electronic device can obtain the MUS based on the FOL sets stored in the knowledge base.

[0089] It should be understood that since the original data and the corresponding FOL sets are stored in association, if the original data is known, the corresponding FOL set can be found in the knowledge base; similarly, if the FOL set is known, the corresponding original data can be found in the knowledge base. For example, if the knowledge base stores original data 1 and FOL set 1 in association, and also stores original data 2 and FOL set 2 in association, then based on original data 1, FOL set 1 can be found in the knowledge base, and based on FOL set 2, original data 2 can be found in the knowledge base.

[0090] S1022, in response to the first triggering condition, extract the FOL set stored in the knowledge base.

[0091] As one embodiment of this disclosure, an electronic device can, in response to a first triggering condition, extract a set of FOLs stored in a knowledge base, and then use an SMT solver to solve the extracted set of FOLs to obtain a MUS. It should be understood that the number of MUS obtained after solving the set of FOLs each time is determined by the number of logical contradictions contained in the corresponding original data. If there are no logical contradictions in the original data, no MUS will be obtained; if the original data includes one logical contradiction, one MUS will be obtained; if the original data includes multiple unrelated logical contradictions, multiple MUS will be obtained. For example, at time t1, if the knowledge base stores 10 FOLs (these 10 FOLs can be called a set of FOLs), then at time t1, the electronic device can obtain one MUS based on the 10 FOLs (it should be noted that obtaining one MUS may result in no MUS being obtained, or it may result in one MUS, or it may result in multiple MUS). Suppose that between time t1 and time t2, 5 new FOLs are added to the knowledge base. Then at time t2, the electron can acquire MUS again based on 15 FOLs (including the 10 FOLs already existing at time t1 and the 5 new FOLs added between time t1 and time t2, which can be called a FOL set).

[0092] As one embodiment of this disclosure, the first triggering condition can represent the conditions that the electronic device needs to meet to obtain the MUS, or it can represent the conditions that the electronic device needs to meet to solve the FOL set using SMT. When the first triggering condition is met, the electronic device can extract the FOL set stored in the knowledge base, and then use the SMT solver to solve the extracted FOL set to obtain the MUS.

[0093] As one embodiment of this disclosure, the first triggering condition includes at least one or more of the following: the time interval between the current time and the previous time of obtaining MUS reaches a first threshold; the current time reaches a first preset time; the number of FOL sets stored in the knowledge base reaches a second threshold; the original data stored in the knowledge base is a first preset data; and a first triggering instruction is received, the first triggering instruction being used to indicate that the user has performed the operation of determining MUS.

[0094] The first threshold can be used to indicate the period for obtaining the MUS. The first threshold can be set as needed, for example, it can be 1 minute, 10 minutes, or 1 hour, etc., and this embodiment of the disclosure does not impose any limitations on this. If the time interval between the current moment and the previous moment of obtaining the MUS reaches the first threshold, it indicates that the MUS should be obtained again. Therefore, the electronic device can extract the FOL set stored in the knowledge base and use the SMT solver to solve the extracted FOL set to obtain the MUS. This method enables the periodic acquisition of the MUS.

[0095] The first preset time can represent a fixed, pre-set trigger time. The first preset time can be set as needed, such as a specific date or time, and this embodiment does not impose any limitations on this. This method can achieve the acquisition of MUS at a fixed date or time.

[0096] The first preset data can represent pre-set data. The first preset data can be data of a certain type, data including certain keywords, or data representing certain meanings. For example, the first preset data can be "student." When the knowledge base stores raw data including "student," the electronic device acquires MUS. In this embodiment, the first preset data is not limited. This approach facilitates the rapid discovery of sensitive content, thereby improving the efficiency of anomaly detection.

[0097] The second threshold can represent a pre-set amount of data. The second threshold can be set as needed; for example, it can be 10, 200, or 1000, etc. This embodiment does not impose any limitations on this. When the number of FOL sets stored in the knowledge base reaches the second threshold, it indicates that a large number of FOLs have been added. At this point, the possibility of logical contradictions is relatively high, so MUS can be retrieved. This method can improve efficiency.

[0098] The first trigger command can be used to indicate that the user has performed a specific MUS (Mobile Information System) operation. In other words, when the user performs this operation, the electronic device receives the first trigger command and, in response, can acquire the MUS. This method is a manual triggering method, which allows for forced acquisition of the MUS by the user, improving user-friendliness.

[0099] S1023, the SMT solver is used to solve the extracted FOL set to obtain the MUS.

[0100] As one embodiment of this disclosure, after the electronic device extracts the FOL set from the knowledge base, it can use an SMT solver to solve the extracted FOL set to obtain the MUS. It should be understood that the process of using an SMT solver to solve the FOL set for MUS can be referred to related technologies, and will not be described in detail here.

[0101] As one embodiment of this disclosure, by first associating and storing the original data and the FOL set in the knowledge base, and then extracting the FOL set from the knowledge base and solving the MUS when the first triggering condition is met, the flexibility of obtaining the MUS can be improved, making the obtained MUS more in line with user needs.

[0102] In this embodiment of the disclosure, after obtaining the MUS, the electronic device can perform conflict set analysis based on the MUS to obtain outliers in the original data. The conflict set analysis process is described below. In some embodiments, performing conflict set analysis based on the MUS to obtain outliers in the original data (S103) may include:

[0103] S1031, Store the MUS in the reasoning conclusion library.

[0104] The electronic device can store one or more acquired MUS into an inference theory library after each acquisition. The inference theory library can represent a database used to store the MUS. This disclosure does not limit the type of database used for the inference theory library.

[0105] S1032, in response to the second triggering condition, extract the MUS stored in the reasoning conclusion library.

[0106] The electronic device can perform conflict set analysis based on the MUS (Must-Have Analyses) in the inference conclusion library, under the condition that a second triggering condition is met, thereby identifying outliers in the original data. The second triggering condition determines whether to extract the MUS from the inference theory library. If the second triggering condition is met, the electronic device can perform the action of extracting the MUS from the inference theory library; if the second triggering condition is not met, the electronic device will not perform the action of extracting the MUS from the inference theory library.

[0107] As one embodiment of this disclosure, the second triggering condition may include at least one or more of the following: the time interval between the current time and the previous time when root cause data was obtained reaches a third threshold; the current time reaches a second preset time; the number of MUS stored in the inference conclusion library reaches a fourth threshold; the MUS stored in the inference conclusion library is second preset data; a second triggering instruction is received, the second triggering instruction being used to indicate that the user has performed the operation of obtaining root cause data.

[0108] The third threshold can be used to indicate the period for detecting anomalies. The third threshold can be set as needed, for example, it can be 1 minute, 10 minutes, or 1 hour, etc., and this embodiment does not impose any limitations on this. If the time interval between the current moment and the moment when root cause data was previously obtained reaches the third threshold, it indicates that anomalies should be re-detected. Therefore, the electronic device can extract the MUS stored in the inference conclusion library and perform conflict set analysis on the extracted MUS. This method enables periodic anomaly detection.

[0109] The second preset time can represent a pre-set fixed trigger time point. The second preset time can be set as needed, and it can be referred to the first preset time, so it will not be elaborated here. This method can realize conflict set analysis on a fixed date or time point.

[0110] The fourth threshold represents a pre-set amount of data. This threshold can be set as needed and is similar to the second threshold, so it will not be elaborated further here. When the number of MUS stored in the reasoning conclusion database reaches the fourth threshold, it indicates that a significant number of new MUS have been added, potentially leading to numerous logical contradictions. Therefore, conflict set analysis can be performed. This approach can improve efficiency.

[0111] The second preset data can represent pre-set data. This second preset data can be of a certain type, or it can include certain keywords or represent certain meanings. The second preset data can be referenced from the first preset data, and will not be elaborated further here. This method is beneficial for quickly identifying logical contradictions, thereby improving the efficiency of anomaly detection.

[0112] The second trigger command indicates that the user has performed an operation to retrieve root cause data. In other words, when the user performs an operation to retrieve root cause data (or an anomaly detection operation, or a conflict set analysis operation), the electronic device receives the second trigger command. In response to this command, the electronic device can retrieve the root cause data and perform subsequent analysis. The second trigger command is similar to the first trigger command and will not be elaborated here. This method is manual triggering, which allows for human intervention, improves user-friendliness, and increases flexibility.

[0113] S1033, perform conflict set analysis on the extracted MUS to obtain root cause data.

[0114] After extracting the MUS, electronic devices can perform conflict set analysis on the extracted MUS to obtain root cause data, i.e., conflicting FOLs.

[0115] As one embodiment of this disclosure, the root cause data may include a list of binary objects and a list of MUS identifiers; the list of binary objects includes at least one binary object, and each binary object includes an identifier and weight of a FOL.

[0116] The MUS identifier list can be used to identify a unique MUS; that is, based on the MUS identifiers included in the root cause data, it can be determined which MUS the root cause data originates from. The FOL identifier can identify a unique FOL; based on the FOLs included in the root cause data, the specific FOL can be determined. The FOL weights can be used to evaluate the probability that a FOL is root cause data. For example, the FOL weights can be used to rank the FOLs; the higher the ranking, the greater the probability, and the lower the ranking, the less likely the probability.

[0117] As one embodiment of this disclosure, the electronic device can count the total number of occurrences of each root cause data (i.e., conflicting FOLs) in all MUS, and then sort all FOLs in descending order using this count as a weight. The higher the FOL's ranking, the greater the probability that it is a root cause data (which is what follows). Figure 8 (The source of FOL_And_Weights in the RootCause data object). The total number of times FOL appears here can be used as the weight of that FOL. It should be understood that other conflict set analysis methods can also be used to find root cause data. For example, linear regression, KNN clustering, and PCA principal component analysis can be used to determine the weight of FOL, thereby finding root cause data. This disclosure does not limit the embodiments in this way.

[0118] S1034, Based on the root cause data, determine the outliers in the original data.

[0119] Since the original data is stored in the knowledge base in association with the FOL sets, there is a correspondence between the original data and the FOL sets. Therefore, after identifying the root cause data, the electronic device can first look up the FOL set to which the root cause data belongs, and then look up the original data corresponding to that FOL set, thereby identifying the outliers in the original data.

[0120] In this embodiment, the natural language data understanding and controllable generation capabilities of LLM are first utilized to convert the raw data into FOL (Factory Object) form using prompt engineering, gradually constructing and improving the knowledge base (KB). Then, the efficient processing capability of the SMT (Simplified Language Management) solver is used to solve for the minimum unsatisfiable subset (MUS) of the FOL set extracted from the knowledge base, obtaining conflicting FOLs. This process is repeated, continuously expanding the raw data and FOL set in the knowledge base, while the SMT solver outputs more MUS. Conflict set analysis is performed on these MUS to identify FOLs that frequently appear in multiple MUS, thus completing the potential root cause data analysis of anomalies within a period.

[0121] It should be understood that the specific design decisions and details that may be adopted during implementation (e.g., choosing different LLMs, different LLM deployment methods, choosing different solvers, using different implementation languages ​​and frameworks, different interaction interface design and processing methods, different hardware selection and design methods, etc.) should all be regarded as different specific implementations under this method framework.

[0122] The anomaly detection method provided in this disclosure can automatically analyze logical anomalies in large amounts of natural language data with low cost and high accuracy. Furthermore, it has low requirements for users and offers good transparency and auditability.

[0123] Secondly, referring to Figure 2 This disclosure provides an electronic device, which includes a memory and a processor; the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, it implements any of the anomaly detection methods of this disclosure.

[0124] Thirdly, referring to Figure 3 This disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements any of the anomaly detection methods of this disclosure.

[0125] Fourthly, embodiments of this disclosure provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the anomaly detection methods of embodiments of this disclosure.

[0126] Among them, the processor is a device with data processing capabilities, including but not limited to the central processing unit (CPU); the memory is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, enabling information exchange between the memory and the processor, including but not limited to the data bus (Bus).

[0127] It should be understood that the anomaly detection methods, electronic devices, computer-readable media, and computer program products provided in this disclosure can be applied to business scenarios that require automatic anomaly reasoning and analysis of large amounts of specific types of natural language data. For example, scenarios where open-source or commercial versions of LLM are used, and customized training or fine-tuning is not desired; scenarios where the system is expected to automatically discover contradictions and anomalies in the data without human guidance; and scenarios where the system is expected to provide reasoning processes and contextual clues related to anomalies for manual verification.

[0128] It should be understood that the anomaly detection methods, electronic devices, computer-readable media, and computer program products provided in the embodiments of this disclosure have no special requirements on the hardware and software of the application environment, and are applicable to all scenarios where it is desirable to automatically analyze and obtain logical anomalies in large amounts of natural language data. Examples include legal document review scenarios, medical diagnostic record review scenarios, contract document review scenarios, and inquiry or interrogation record review scenarios.

[0129] This disclosure also provides an anomaly detection system, and the anomaly detection method of this disclosure can be applied to the anomaly detection system. The anomaly detection system will be described exemplarily below with reference to Example 1, and the application of the anomaly detection method in the anomaly detection system will be described exemplarily below with reference to Example 2.

[0130] Example 1

[0131] This disclosure provides an anomaly detection system, referring to... Figure 4 The anomaly detection system provided in this embodiment may include a main control system, system prompts, an LLM (Local Level Management), a knowledge base, an SMT (Surface Mount Technology) solver, and an inference conclusion base. It should be understood that the anomaly detection system may also include other electronic devices, which will not be described in detail here.

[0132] Main control system: Receives the raw data to be detected as input data and provides it to LLM for information extraction and transformation, i.e., converts the raw data into a set of FOLs (corresponding to S101); responsible for updating, maintaining, and applying system prompts; responsible for updating and maintaining the knowledge base, such as storing raw data and FOL sets in association in the knowledge base (corresponding to S1021); responsible for responding to the first trigger condition, extracting the FOL sets stored in the knowledge base (corresponding to S1022), and using the SMT solver to solve the extracted FOL sets to obtain MUS (corresponding to S1023); responsible for performing conflict set analysis based on MUS to obtain outliers in the raw data (corresponding to S103).

[0133] System prompts: Their content is maintained and updated by the master control system; they are sent by the master control system to the LLM, thereby controlling the LLM's output behavior in S101. When the system prompts are provided to the LLM, they enable it to convert the raw data into several FOLs, i.e., a set of FOLs.

[0134] LLM: Provides the system with the ability to extract and transform information from natural language data; the main control system sends raw data to it by calling its interface and obtains the transformation results, i.e., the FOL set; the main control system can provide matching system prompt words for raw data of different data types, so that it can complete the information extraction and transformation work as expected.

[0135] Knowledge Base: The knowledge base contains the original data for each batch and the corresponding set of FOLs (Files of Order) for that batch. The master control system is responsible for associating the original data with the corresponding FOL sets and storing them in the knowledge base. The master control system can index the original data based on the FOLs, and vice versa. The master control system can maintain the knowledge base as needed, such as deleting old records or adding category tags to records.

[0136] SMT Solver: The SMT solver can be selected as needed, such as Z3, CVC4, etc.; the main control system is responsible for providing the FOL set in the knowledge base to the SMT solver to obtain MUS. These MUS represent outliers present in the original data.

[0137] Inference Conclusion Library: After the main control system obtains several MUS from the SMT solver, it performs conflict set analysis on them to obtain the FOL (i.e., root cause data) that appears most frequently and has the widest impact among all MUS, thus completing the root cause analysis. Based on the root cause data, the main control system indexes the corresponding batch of raw data from the knowledge base and outputs it to the inference conclusion library along with the root cause data for manual use or review.

[0138] Considering Figure 4The main control system shown interacts extensively with other modules within the system. For ease of understanding, the internal logical architecture of the main control system is explained below. (Refer to...) Figure 5 , Figure 4 The main control system shown may include a business logic submodule, a system prompt word management submodule, an LLM interface call submodule, a knowledge base management submodule, an SMT solver submodule, and a root cause analysis submodule. Among these, the system prompt word management submodule, the knowledge base management submodule, the SMT solver submodule, and the root cause analysis submodule are modules provided in the embodiments of this disclosure. Wherein:

[0139] The business logic submodule is responsible for implementing all business logic processes of this system (corresponding to S101 to S103). It drives the prompt word management submodule, LLM interface call submodule, knowledge base management submodule, SMT solver submodule, and root cause analysis submodule to complete corresponding actions. These actions include updating system prompt word data, calling LLM, calling the SMT solver, storing FOL sets in the knowledge base, retrieving FOL sets from the knowledge base, and performing conflict set analysis.

[0140] The prompt word management submodule is responsible for... Figure 4 The system prompt word module shown interacts with the interface to store and retrieve multiple system prompt word data (i.e., the specific system prompt word content). The system prompt word data can be found in the following text. Figure 11 Different types of raw data generally require different system prompts to achieve the best information extraction and transformation results. System administrators need to pre-write appropriate system prompts based on the data characteristics of the raw data (such as data type) and save them through this module. When actually sending raw data to the LLM, a matching system prompt must be selected and attached.

[0141] The LLM interface call submodule is responsible for implementing the interface for calling the LLM. The business logic submodule sends interactive data (i.e., raw data and system prompts) to the LLM through this interface and obtains the response (i.e., the FOL set). This module can be referenced from relevant technologies, and will not be elaborated here.

[0142] The knowledge base management submodule is responsible for storing and retrieving the raw data and the corresponding FOL set. The business logic submodule associates the raw data and the FOL set returned by LLM and submits it to the knowledge base management submodule for storage. When performing anomaly analysis (i.e. conflict set analysis), it extracts the FOL data and passes it to the SMT solver submodule (SMT solver submodule).

[0143] The SMT solution submodule is responsible for solving the FOL set to obtain the MUS. The business logic submodule then passes the MUS to the root cause analysis submodule for further analysis and persistence. This module can be referenced from relevant technologies, and will not be elaborated upon here.

[0144] The root cause analysis submodule is responsible for persistently storing all MUS and performing conflict set analysis on the MUS to obtain root cause data. The business logic submodule is responsible for providing it with all MUS, triggering analysis actions, and returning the analysis results to the user, including root cause data and anomalies in the original data.

[0145] Example 2

[0146] The anomaly detection method of this disclosure embodiment can be applied to... Figure 4 and Figure 5 The system shown in the figure contains two key processes: a knowledge base construction process and an anomaly analysis process.

[0147] The following is based on Figure 4 The system shown provides a detailed explanation of the knowledge base construction process. (Refer to...) Figure 6 The knowledge base construction process may include:

[0148] 501, The main control system receives input data.

[0149] This step corresponds to Figure 4 In step 1, the input data is the original data to be detected.

[0150] 502. The master control system sets a system prompt word for the LLM to match the data type of the input data, and then sends the input data and system prompt word to the LLM so that the LLM can convert the input data into a FOL set.

[0151] This step corresponds to Figure 4 Steps 2 and 3 in the process.

[0152] 503. The main control system obtains the output FOL set from the LLM, associates the FOL set with the input data, and stores them together in the knowledge base.

[0153] This step corresponds to Figure 4 Step 4 in the process.

[0154] The following is based on Figure 4 The system shown illustrates the anomaly analysis process. (Refer to...) Figure 7 The anomaly analysis process may include:

[0155] 601. The main control system extracts the FOL set from the knowledge base and calls the SMT solver to solve MUS.

[0156] This step corresponds to Figure 4 Step 5 in the process.

[0157] 602, the main control system stores the obtained MUS into the reasoning conclusion database.

[0158] This step corresponds to Figure 4 Step 6 in the process.

[0159] 603. In response to the first triggering condition, the main control system acquires MUS multiple times and stores the acquired MUS into the inference conclusion database.

[0160] As the knowledge base construction process is executed multiple times, the content of the knowledge base will continuously expand. The aforementioned steps of this process can be executed multiple times within a certain period of time based on the first triggering condition, resulting in a continuous increase in the number of MUS stored in the reasoning conclusion base.

[0161] 604. In response to the second triggering condition, the main control system performs conflict set analysis on multiple MUS in the inference conclusion database to obtain root cause data, and also stores the root cause data in the inference conclusion database.

[0162] Figure 6 and Figure 7 This illustration shows the execution process of the anomaly detection method provided in this embodiment within an anomaly detection system. In this embodiment, LLM is first used to convert the raw data to be detected into a set of Forms of Occurrence (FOLs). Then, an SMT solver is used to extract the Means of Conflict (MUS) of this FOL set. Based on the MUS, conflict set analysis is performed to obtain anomalies in the raw data. On one hand, the natural language data understanding capabilities of LLM are utilized for FOL information extraction without requiring deep reasoning by LLM; on the other hand, the method of using an SMT solver to extract MUS and perform conflict set analysis to determine anomalies is precise and transparent, thus improving the accuracy of anomaly detection.

[0163] Reference Figure 8 This illustrates the relationship between the data objects in the knowledge base and the reasoning conclusion base in an embodiment of this disclosure.

[0164] RawData represents raw data. Each piece of raw data (i.e., raw data in a batch) should contain a unique Id (i.e., an identifier, such as a number) and Content (i.e., its own data content). Here, int represents an integer type, string represents a string type, and RawData_Id represents the identifier of the raw data.

[0165] FOL represents a first-order logical representation extracted from raw data. A single piece of raw data can correspond to a set of FOLs, which includes one or more FOLs. Each FOL should contain a unique Id (i.e., an identifier, such as a number), a FOL (i.e., its own content), and the Id of the corresponding raw data (i.e., RawData_Id). FOL_Ids represents a list of identifiers for FOLs.

[0166] MUS represents the smallest unsatisfiable subset obtained by the SMT solver from the set of FOLs. Each MUS should contain a unique Id (i.e., identifier, such as number) and a FOL_Ids (i.e., a list of identifiers of FOLs, since each MUS is essentially a set of several FOLs). MUS_Ids represents the list of identifiers of the MUS.

[0167] RootCause represents root cause data, which is the product of conflict set analysis of multiple Mutual Roots (MUS). Each RootCause should contain a unique ID (i.e., identifier, such as a number), a list of tuple objects, and a MUS_Ids. Each element in the list of tuples is a tuple containing a FOOL ID and a weight (i.e., FOL_And_Weight). Root cause analysis essentially involves performing conflict set analysis on several MUS to obtain several target FOOLs, and using the total frequency of these FOOLs in the MUS as weights to measure the contribution of different FOOLs in the root cause data. Here, tuple indicates the tuple type.

[0168] For the knowledge base, it only involves RawData and FOL data objects; for the reasoning conclusion base, it only involves MUS and RootCause data objects.

[0169] The two key system processes described above (knowledge base construction process and anomaly analysis process) are processes of continuously instantiating these data objects.

[0170] When the system is running, users can index and read any specific data they want using the information contained in these data object instances.

[0171] It should be understood that the anomaly detection system provided in this disclosure can also be applied to business scenarios that require automatic anomaly reasoning and analysis of a large amount of natural language data of preset types.

[0172] In one possible implementation, the anomaly detection system of this disclosure includes two roles: a system administrator and a user. The system administrator is responsible for implementing and managing the entire system; and for writing appropriate system prompts based on data provided by the user. The user provides the system administrator with relevant information about the raw data (such as data type) in advance to facilitate the writing of appropriate system prompts; inputs the raw data into the system; and retrieves intermediate results (MUS) and final results (root cause data, anomalies) from the system. Depending on the application scenario, these two roles can be combined.

[0173] The following is based on Figure 5 The system shown provides a detailed explanation of the knowledge base construction process. (Refer to...) Figure 9The knowledge base construction process may include:

[0174] 901, The business logic submodule has received the raw data to be analyzed at the user's end, and the process has begun.

[0175] 902, The business logic submodule obtains system prompts from the system prompt management submodule.

[0176] 903, The business logic submodule sends the system prompt word along with the raw data to the LLM.

[0177] 904, LLM extracts and transforms information from the original data and returns a FOL set.

[0178] 905. The business logic submodule submits the original data and the corresponding FOL set to the knowledge base management submodule for persistent data storage.

[0179] 906, End.

[0180] Once the knowledge base contains a certain amount of content, the anomaly analysis process can be initiated. The triggering conditions can be flexibly selected based on the user's actual needs, such as the first triggering condition mentioned above.

[0181] The following is based on Figure 5 This section provides a detailed explanation of the anomaly analysis process. (Refer to...) Figure 10 The anomaly analysis process may include:

[0182] 1001, The triggering conditions for the anomaly analysis process have been met, and the process has begun.

[0183] 1002, The business logic submodule extracts the FOL set from the knowledge base management submodule.

[0184] 1003, The business logic submodule sends the FOL set to the SMT solver for solving and obtaining the MUS.

[0185] 1004. The business logic submodule sends the obtained MUS to the root cause analysis submodule for persistent storage.

[0186] 1005. Determine if the second trigger condition is met. If it is, proceed to step 1006; otherwise, proceed to step 1007.

[0187] 1006, the root cause analysis submodule performs conflict set analysis on multiple MUS to obtain root cause data.

[0188] 1007, End.

[0189] visible, Figure 9 and Figure 10The execution process of the anomaly detection method provided in the embodiments of this disclosure in the main control system of the anomaly detection system is illustrated.

[0190] To facilitate understanding of the system prompts and MUS, Example 3 is provided below to illustrate the process of analyzing the FOL set using the system prompts and the SMT solver in this embodiment of the present disclosure.

[0191] Example 3

[0192] Application Scenario: Users will be provided with a batch of text data related to school teaching. The user's task is to extract as much FOL (Forms of Order) as possible from this text and represent it using SMTLIB format. For this application scenario, users need to define data types, predicates, and functions, and assert the logical relationships within the text data. System prompts for the above application scenario can be found in [reference needed]. Figure 11 The system prompt text shown is an example.

[0193] like Figure 11 As shown, the system prompts include data processing requirements, specifically 1. Define types (define types for the designed entities (such as teachers, students, courses, assignments, etc.), 2. Define predicates and functions (define appropriate predicates and functions based on the description in the text), and 3. Assert logical relationships (assert logical relationships in the text using the defined types, predicates, and functions).

[0194] As can be seen, the system prompts inform the LLM about the data type (such as text or image), data structure (such as what objects it may contain), and general application scenario. This allows the LLM to better understand the data and complete information extraction and transformation tasks with high quality.

[0195] SMTLIB (Satisfiability Modulo Theories Library) is a standardized language and library for describing and exchanging SMT problems. Its primary goal is to provide a unified input format for SMT solvers, thereby promoting interoperability and comparison between different solvers. Many popular SMT solvers (such as Z3, CVC4, and Yices) support the SMTLIB format. These solvers can read and solve SMTLIB formatted files.

[0196] In this example, the LLM is required to output in SMTLIB format to allow for flexible use with different SMT solvers. If a solver has already been selected in a real-world scenario, the LLM can also be required to output a specific input format supported by that solver.

[0197] Figure 11The example input and output data are also shown. The example input is a sample of the text data to be input, and the example output is a sample of the format (FOL) to be output. For example, "Student roster" is the text data to be input, and (assert(StudentA)) is the format (FOL) to be output. The same applies to other data, which will not be elaborated here.

[0198] based on Figure 11 The system prompts shown indicate that the SMT solver will obtain a MUS after analyzing the SMTLIB output of the LLM, which includes the following FOLs: (assert(Student A)); (assert(forall((x Person)(yCourse))(=>(and(Student x)(Registered xy))(CompleteAssignments xy)))); (assert(Registered A Math)).

[0199] The MUS described above indicates a logical inconsistency among these three FOLs (Forms of Occurrence). A is a student (first FOL), and all students should complete assignments for all their registered courses (second FOL). A then registered for a math course (third FOL), but no FOL showing A completing math assignments can be found in the available information. Thus, through the MUS obtained through automatic system analysis, people can identify anomalies in large amounts of data and perform anomaly analysis.

[0200] The following example, using the legislative review of legal provisions as an example, describes a specific example of the anomaly detection system provided in this disclosure, as well as the implementation process of the anomaly detection method provided in this disclosure in the system.

[0201] Example 4: Legal text legislative review scenario. In this scenario, there are several types of participants: legislators and reviewers.

[0202] Legislators: They design new legal provisions based on actual needs.

[0203] Given the large number of existing and proposed legal provisions, and the potentially complex relationships between them, new legal provisions put forward by the legislature may conflict with other provisions.

[0204] Reviewing party: The reviewing party needs to review the new legal provisions proposed by the legislator, check whether they conflict with other provisions, and ensure that all legal provisions are logically consistent.

[0205] Given the large number of existing and proposed legal provisions, and the potentially complex relationships between them, manual review would be both difficult and costly. Therefore, the reviewing body uses a legal provision conflict detection system to assist in the analysis.

[0206] It should be noted that, considering that the application scenario of Example 4 is the legislative review of legal provisions, the anomaly detection system provided in this disclosure embodiment is referred to as the legal provision conflict detection system in Example 4.

[0207] Reference Figure 12 This legal clause conflict detection system may include: a main control program, a Z3 solver, a MySQL database, and the Tongyi Qianwen large model. The main control program is... Figure 4 The example of the main control system shown has the Z3 solver as follows: Figure 4 The examples shown are of SMT solvers, MySQL databases as knowledge bases and reasoning conclusion bases, and Tongyi Qianwen large models as LLM examples.

[0208] The main control program obtains access to the Tongyi Qianwen large model through the Python API client library provided by the Tongyi Qianwen vendor.

[0209] The main control program obtains access to the MySQL database through the Python API client library provided by MySQL. All data that needs to be persistently stored in the system is stored in different tables in the MySQL database (such as conflict sets, FOL tables, root cause tables, legal text data tables, and system prompt word tables).

[0210] The main control program obtains the ability to call the Z3 solver through the Python API of the Z3 solver.

[0211] The main control program receives legal text data provided by the reviewer as input data through the HTTP API of the HTTP server, and outputs the analysis results of conflicting texts.

[0212] It should be understood that, in addition to Python, the main control program can also be implemented using other mainstream programming languages ​​(such as Golang, C / C++, Rust, etc.) as needed, and can also use databases other than MySQL (such as Postgres, etc.). All mainstream LLM, databases, and SMT solvers provide API libraries for common programming languages ​​for integration. This embodiment only provides one possible design and implementation.

[0213] Reference Figure 13 , showed Figure 12 A diagram showing the relationships between the various tables in a MySQL database. Figure 13 You can refer to Figure 8For reference only, further details will not be provided here.

[0214] The table design (table name, field names, etc.) in this embodiment is only one possible design scheme and is for reference only. In practice, other similar designs can be selected according to requirements.

[0215] The reference SQL code for database table creation in this embodiment can be found by referring to... Figure 14 .

[0216] Note: Some table and field names in the SQL code above use Chinese characters for ease of understanding. Although MySQL supports Unicode table and field names, it is generally recommended to use English characters for compatibility reasons. The code here is for reference only.

[0217] The input data for this system is intellectual property-related legal provisions. Sample data can be found by referring to... Figure 15 .

[0218] Since this embodiment is specifically designed for legal text analysis and uses a fixed SMT solver, its system prompts are essentially fixed, similar to... Figure 16 The system prompts shown are as follows. Among them, "You will see...Z3 program" indicates a data processing requirement, "Example legal text" indicates an input example, and "Example output" indicates an output example.

[0219] Based on system prompts, LLM outputs the input legal provisions as FOL rules. The main control module writes them into the FOL table in the database (the original data is written into the legal provisions table). At the same time, when the triggering conditions are met, the FOL content is extracted from the data table and submitted to the SMT solver to obtain MUS (conflict set).

[0220] The key steps in the reference Python code for submitting FOL content to the Z3 solver and solving MUS are: passing the FOL definition code output by LLM as input parameters to the program; importing the passed-in FOL definition into the Z3 solver instance; and using the Z3 solver to determine if there are any conflicts. If there are no conflicts, the empty set is output directly; if there are conflicts, the unsat_core output by the Z3 solver (the set of conflicting FOLs, but which may not be the smallest set) is obtained and further filtered to remove redundant FOL definitions, finally obtaining and returning MUS.

[0221] The main control module saves the non-empty conflict set data returned by SMT to the conflict set table. When the trigger condition is met, it extracts the MUS content from the conflict set table and performs conflict set analysis to attempt to obtain root cause data. There are various methods for conflict set analysis, which will not be elaborated here.

[0222] This embodiment includes two main processes: the construction of a legal provisions knowledge base and the analysis of provisions conflicts.

[0223] The core logic of the above code is to count the total number of occurrences of each FOL in all MUS sets, and finally output all FOLs in descending order of occurrence. The higher the FOL ranks, the greater the probability that it is the root cause of the conflict.

[0224] After obtaining root cause FOL data through conflict set analysis, the main control module stores it in the root cause table for review. Reviewers can query and obtain the FOL information contained in any conflict set (MUS) and root cause, as well as the corresponding original legal provisions, thereby knowing the logical conflicts and conflict points in the legal provisions, and then conducting manual review or correction.

[0225] The main control module provides two key external functions in the form of a standard HTTP interface: inputting legal text data to be reviewed; and querying MUS or root cause data obtained from system analysis. There are multiple options for the specific design and implementation of the external interface, which will not be elaborated here.

[0226] Those skilled in the art will understand that all or some of the steps, systems, and devices disclosed above, as functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0227] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be executed by several physical components working together.

[0228] Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only optical disc (CD-ROM), digital versatile disc (DVD) or other optical disc storage; magnetic cartridges, magnetic tapes, disk storage or other magnetic storage; and any other media that can be used to store desired information and can be accessed by a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0229] This disclosure has disclosed exemplary embodiments, and although specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A method for anomaly detection, comprising: The Large Language Model (LLM) is used to convert the raw data to be detected into a set of first-order logical FOLs. The minimum unsatisfiable subset (MUS) of the FOL set is extracted using the Satisfiability Modular Theory (SMT) solver. The MUS is used to indicate conflicting FOLs. Based on the MUS, conflict set analysis is performed to obtain the outliers in the original data.

2. The method according to claim 1, wherein, The process of using a Large Language Model (LLM) to convert the raw data to be detected into a first-order logistic (FOL) set includes: Obtain a system prompt word that matches the data type of the original data; the system prompt word is used to indicate the method for extracting FOL. The raw data and the system prompt words are input into the LLM, and the FOL set corresponding to the raw data is output.

3. The method according to claim 2, wherein, The system prompts include at least the data processing requirements, input examples, and output examples.

4. The method according to claim 1, wherein, The extraction of the minimum unsatisfiable subset (MUS) of the FOL set using the Satisfiability Modular Theory (SMT) solver includes: In the knowledge base, the original data and the corresponding FOL set are stored together. In response to the first triggering condition, the FOL set stored in the knowledge base is extracted; The extracted FOL set is solved using the SMT solver to obtain the MUS.

5. The method according to claim 4, wherein, The first triggering condition includes at least one or more of the following: The time interval between the current moment and the previous moment when MUS was obtained reaches the first threshold; The current time reaches the first preset time; The number of FOL sets stored in the knowledge base has reached the second threshold; The original data stored in the knowledge base is the first preset data; Upon receiving a first trigger instruction, the first trigger instruction indicates that the user has performed a certain MUS operation.

6. The method according to any one of claims 1 to 5, wherein, The collision set analysis based on the MUS detection, to obtain outliers in the original data, includes: The MUS is stored in the reasoning conclusion database; In response to the second triggering condition, the MUS stored in the reasoning conclusion library is extracted; Conflict set analysis was performed on the extracted MUS to obtain root cause data; The outliers in the original data are determined based on the root cause data.

7. The method according to claim 6, wherein, The second triggering condition includes at least one or more of the following: The time interval between the current moment and the previous moment when root cause data was obtained reaches the third threshold. The current time reaches the second preset time; The number of MUS stored in the reasoning conclusion library has reached the fourth threshold. The MUS stored in the reasoning conclusion database is the second preset data; A second trigger instruction is received, which indicates that the user has performed an operation to obtain root cause data.

8. The method according to claim 6, wherein, The root cause data includes a list of binary objects and a list of MUS identifiers; the list of binary objects includes at least one binary object, and each binary object includes an identifier and weight of a FOL; the list of MUS identifiers is used to identify the MUS that determines the root cause data.

9. An electronic device comprising a memory and a processor; the memory storing a computer program executable by the processor, wherein the computer program, when executed by the processor, implements the anomaly detection method according to any one of claims 1 to 8.

10. A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the anomaly detection method according to any one of claims 1 to 8.

Images