Apparatus and method for auditing rule-based command risk assessment system

Abstract

The present invention provides an apparatus (1001) for auditing rules in a rule-based command risk assessment system, the apparatus communicatively connected to one or more data stores (406, 407, 1003), the data stores storing one or more commands and a pre-existing set of rules for classifying the commands; the apparatus comprises one or more processors (1002) for, for each of the at least one command: receiving (901) a first predictive classification for the respective command, the first predictive classification determined based on the pre-existing rule set; forming (902) a second predictive classification for the respective command based on an external document of the respective command; if the first prediction is different from the second prediction, a notification is output (903). This may provide a supervisory system that enables rules to correctly reflect the risk of executed commands over time. The risk assessment quality can be improved, and the accident occurrence frequency is reduced.

Description

TECHNICAL FIELD

[0001] The present invention relates to a risk assessment system, and in particular to auditing rules in a command risk assessment system. BACKGROUND

[0002] The growth of security threats has prompted organizations to develop effective security solutions to protect digital applications, data, and resources. The goal of these solutions is to prevent incidents and malicious attacks. To prevent failures and threats in large IT systems due to operators executing high-risk commands in command-line interfaces, it is necessary to automatically identify high-risk commands and, where possible, prevent high-risk commands.

[0003] In large IT systems, operations and maintenance (O&M) personnel can access production systems for maintenance operations through remote terminals. Mistaken or malicious operations can cause significant system failures and data loss. In this case, it is necessary to use a security solution that classifies risk operations before they are executed to prevent the execution of potentially risky operations. One example of risk classification is shown in the table of Figure 1 , where the operations (column on the left) are divided into a "risk" group and a "safe" group.

[0004] Rule-based command risk assessment systems can apply a whitelist / blacklist rule set to classify and process commands. Figure 2 A common solution for the above scenario is schematically shown, in which a command interception system 203, deployed on a network gateway between the operator 201 and the production host 202, contains a rule-based classifier in which IF-THEN-ELSE rules define which commands are allowed (or whitelisted) and which commands are blocked (or blacklisted). These rules are stored in a rules database 204, which can be defined by the O&M operator based on professional knowledge and the history of the executed commands. In large-scale environments, this approach requires periodic maintenance and updating of existing rules to reflect the dynamic changes in the environment. In some cases, new, unknown commands can be executed, and existing rules can not be suitable for these commands. In addition, the risk level assigned to a command by the operator based on their professional knowledge can differ from the inherent risk of the command as described in the command technical documentation.

[0005] Therefore, as shown in 205, an auditing system can be used in the command risk assessment system to assess whether the risk assignment performed by the rule-based classifier is correct.

[0006] Previous solutions in the field of auditing systems can be generally divided into two categories: direct auditing systems and classification-based auditing systems.

[0007] Direct audit systems rely on human expertise (e.g. in the form of an administrator 206) to confirm that the risk assignments performed by a rule-based classifier are correct. If these systems are based entirely on human feedback, it is not possible to assess whether past judgments by operators were incorrect, as the operator’s judgment represents the ground truth. Therefore, such systems require high quality labels and are unable to correct past incorrect judgments. Furthermore, when existing rules do not precisely match a current command, it is not possible to assess the risk and human intervention is required.

[0008] When existing rule-based classifiers are unable to provide rules or the provided rules are incorrect, classification-based audit systems apply machine learning classifiers to predict the risk of a command. By providing more accurate classifications, it is possible to leverage the classifier knowledge to create and update rules. However, this approach does not generalize to unknown commands that are significantly different from the commands used to train the classifier. This presents a challenge for the audit task. Furthermore, the adopted design often does not allow the classifier to analyze the temporal and contextual relationships between command labels, which are very important to understand the behavior of a command. Classification by similarity matching is very sensitive to the choice of similarity function and availability of a large number of labeled commands, and can not be able to analyze all commands executed in a real environment.

[0009] Therefore, it would be desirable to develop a method that overcomes at least some of the above problems. SUMMARY

[0010] According to a first aspect, there is provided an apparatus for auditing rules in a rule-based command risk assessment system, the apparatus being communicably connected to one or more data stores storing one or more commands and a pre-existing rule set for classifying the commands; the apparatus comprising one or more processors configured to, for each of the at least one command: receive a first predicted classification for the respective command, the first predicted classification being determined based on the pre-existing rule set; form a second predicted classification for the respective command based on external documentation of the respective command; output a notification if the first prediction is different from the second prediction.

[0011] By leveraging prior knowledge from an existing classification rule set and additionally leveraging external sources of knowledge from a command file to assess the inherent risk of a command, it is possible to challenge and revise the existing classification rule set. This can provide a supervised system that allows rules to correctly reflect the risk of executed commands over time. This can improve the quality of risk assessment and reduce the frequency of incidents.

[0012] For example, the data store can be a data management system or database system or the like, can be stored in the cloud or hardware, can be a computer readable medium, such as a non-transitory computer readable medium. The external documentation can be stored in the data store. The data store can be in communication with a data network, such as the internet, to periodically update the external documentation stored in the data store. This can keep the external documentation up to date in terms of code or software updates or the like. The rule set can comprise one or more rules, preferably a plurality of rules.

[0013] The one or more commands are historical commands issued by an operator of a target host in accordance with the pre-existing rule set, the target host being controlled by an access control system implementing the rule-based command risk assessment system, each historical command having a corresponding first predicted classification determined based on the pre-existing rule set. This enables the apparatus to audit the rule set used in the rule-based risk assessment system to determine whether the rules are suitable for classifying commands. The rule set can be updated if it does not provide suitable classifications.

[0014] The first classification and the second classification can each comprise an indication of a risk level associated with executing the respective command. For example, the classification can be "safe" or "dangerous". This enables the system to classify risky operations before they are executed to prevent potentially risky operations from being executed.

[0015] The external documentation can comprise one or more of: documentation describing standard operating system commands, files or system calls; internally defined scripts, programs or aliases; third party programs, tools or internal file names. This enables the system to take into account information contained in the external documentation relating to the command when determining whether the rule set is suitable for correctly classifying the command.

[0016] The one or more processors can be configured to process the external documentation or information extracted therefrom (e.g. a cleaned or compressed version of the extracted information) and the command to form the second predicted classification. This enables the system to take into account information relating to the command and a description of the command included in the external documentation when determining the second prediction.

[0017] The one or more processors can be configured to: retrieve a relevant document for the respective command from the external documentation; extract information relating to the respective command from the relevant document; form a command description for the respective command based on the extracted information. The command description can be an explanatory summary of the respective command. This enables the operation of the command to be expressed.

[0018] The command description can be in natural language format. The command can be in computer programming language format. The one or more processors can be configured to input the command description to an artificial intelligence model and form the second predicted classification from an output of the artificial intelligence model. By extracting information from the document, using a language model (such as BERT or GPT3) to classify the command, the context of the command can be captured. The language model can generalize to unknown commands, subcommands, and flag combinations and preserve the sequence of executed commands, thereby taking into account the context of using flags and subcommands. This can improve the accuracy of the classification.

[0019] The artificial intelligence model can include a first head configured to process the command description. The artificial intelligence model can include a second head configured to process the respective command. For example, the first head can be an English head and the second head can be a BERT head, such as a BERT model in a programming language such as Bash. Other models and languages can also be used. This way the context of the command can be captured.

[0020] The one or more processors can be configured to form the second predicted classification from a third classification prediction, the third classification prediction being formed from one or more risk indications corresponding to one or more rules in the set of pre-existing rules selected based on a similarity to the respective command. The third classification prediction can be formed by aggregating risk indications corresponding to rules retrieved based on a similarity to the respective command. The third classification can be an output of a similarity search classification system. This allows the second classification to be formed using similar rules when an existing rule does not precisely match a current command.

[0021] The one or more processors can be configured to form the second predicted classification by aggregating the output of the artificial intelligence model and the third classification prediction. This can improve the accuracy of the final prediction.

[0022] The one or more processors can be configured to output respective notifications of a plurality of commands. The notifications can be pre-grouped by similarity and reported to an administrator of the system. This can improve the efficiency of the notification process.

[0023] The one or more processors can be configured to update one or more rules of the set of pre-existing rules based on the notifications. This way the set of rules can be continually updated, ensuring that the set of rules is suitable for classifying commands.

[0024] According to a second aspect, there is provided a method for auditing rules in a rule-based command risk assessment system, the method causing a device to be communicably connected to one or more data stores storing one or more commands and a pre-existing set of rules for classifying the commands; for each of the at least one command, the method comprising: receiving a first predicted classification for the respective command, the first predicted classification determined based on the pre-existing set of rules; forming a second predicted classification for the respective command based on external documentation of the respective command; outputting a notification if the first prediction is different from the second prediction.

[0025] By leveraging prior knowledge from an existing set of classification rules, and additionally leveraging external sources of knowledge from command files to assess the inherent risk of a command, the method can challenge and revise the existing set of classification rules so that the rules can accurately reflect the risk of commands that have been executed over a period of time. This can improve the quality of risk assessment and reduce the frequency of incidents.

[0026] The above method can be a computer-implemented method implemented on a device (e.g. on a laptop or a smartphone) or externally (e.g. on a cloud-based service).

[0027] According to another aspect, there is provided a computer program comprising instructions which, when executed by a computer, cause the computer to perform the above method.

[0028] According to another aspect, there is provided a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a computer (e.g. comprising one or more processors), cause the computer to perform the above method. The computer-readable storage medium can be a non-transitory computer-readable storage medium. The computer can be implemented as a system comprising interconnected devices. BRIEF DESCRIPTION OF DRAWINGS

[0029] The application will now be described by way of example with reference to the accompanying drawings.

[0030] In the drawings:

[0031] Figure 1 showing the division of operations into a “dangerous” group and a “safe” group;

[0032] Figure 2 schematically showing an application scenario of a rule-based command risk assessment system and a corresponding auditing system;

[0033] Figure 3 schematically showing an exemplary system of an artificial intelligence-based auditing system to which embodiments of the application can be applied;

[0034] Figure 4 An overview of an example audit system and its interaction with an existing rule-based risk assessment system is illustratively shown;

[0035] Figure 5 An example document system of an audit system is illustratively shown;

[0036] Figure 6 An example classifier of an audit system is illustratively shown;

[0037] Figure 7 An example similarity search system that can optionally be implemented by an audit system is illustratively shown;

[0038] Figure 8 An example sequence diagram of the operation of an audit system described herein is shown;

[0039] Figure 9 An example of a method for auditing rules in a rule-based command risk assessment system is shown;

[0040] Figure 10 An example of an apparatus for auditing rules in a rule-based command risk assessment system and some associated components thereof is shown. DETAILED DESCRIPTION

[0041] The present invention relates to a system for auditing a rule-based command risk assessment system, which can be executed in the cloud or other large-scale computing infrastructure. The system can analyze historical risk predictions to validate whether past block / allow decisions for commands were correct according to additional external knowledge. The system can report possible discrepancies to the system administrator. The system compares the output of the risk assessment system with the audit predictions to assess the quality of the predictions provided by the existing system. If the predictions differ, the event is reported to the system administrator for modification.

[0042] In the embodiments described below, command risk classification is performed using Natural Language Processing (NLP) techniques. Common NLP problems include speech recognition, text summarization, and sentiment analysis, which is a category of text classification used to determine the polarity of a given text, where polarity can represent different aspects of language, such as positive / negative, objective / subjective, or hate / supportive content. Recent advances in neural network-based methods, also known as deep learning, have driven the development of some neural architectures specifically tailored for speech and text, such as transformers and other transformer-based large language models (LLMs), such as Bidirectional Transformers for Language Understanding (BERT) (see Devlin, Jacob, et al., "Bert: Pre-training of deep bidirectional transformers for language understanding", arXiv preprint arXiv:1810.04805 (2018)) or GPT3 (see Brown, Tom, et al., "Language models are few-shot learners", Advances in neural information processing systems 33 (2020): 1877-1901”). Benefiting from large amounts of text data and effective representation learning, these models are able to capture relationships between different sub-units of text, which are called tokens (e.g., words, punctuation, or letters). Transformer-based architectures can also be applied to document text and CLI commands. Similar to other sentiment analysis tasks, LLMs can be applied to document text to assess the polarity of dangerous / safe behavior aspects, i.e., if the English document of a command exhaustively describes the behavior of the command, then the LLM is able to classify whether the command poses a risk to the host system. Similar variable parameters also apply to the remote command itself, as remote commands, being instances of programming languages, have predefined lexical, syntactic, contextual, and structural dependency relationships, which are similar in any form to natural languages such as English.

[0043] Figure 3A use case diagram for an example risk assessment system 300 is shown schematically. An operator 301 accesses a production system through an agent 302 for maintenance. In this example, the agent intercepts an operation to be performed on a target host 303 from a Secure Shell (SSH) bastion and forwards the operation to a risk assessment system 304, which assesses the operational risk represented by the command.

[0044] The risk assessment system 304 estimates the operational risk with internal configuration resources, such as a rules management system 305. If the request for the operation is safe, it is forwarded to the target host 303 and the results of the operation are forwarded back to the operator 301. However, if the requested operation is risky, the operation is blocked and the operator is notified that the requested operation is not allowed. The risk prediction classification and the corresponding command are stored in a command database 306 for offline analysis.

[0045] In embodiments of the invention, an audit system 350 retrieves historical risk prediction classifications and current command pairs and estimates a new risk level from the command documentation, thereby analyzing the command pairs. If the historical prediction is inconsistent with the new risk level, the command is reported. The command can be submitted to an expert reviewer (e.g., a system administrator) for modification. The expert reviewer can then decide to take action to modify the runtime behavior of the risk assessment system to better reflect the correct risk level reported by the audit system. To do so, the expert reviewer can modify the rules management system that provides the prediction capability to the risk assessment system. The reported anomalous predictions can be grouped by similarity and the notifications can be grouped by similarity as well. Similar predictions can be reported together.

[0046] Figure 4 A complete schematic diagram of an example operational environment 400 is described.

[0047] In this example, an operator 401 is an engineer performing a maintenance task. The engineer can perform the action on a terminal (not shown) approved by an access control system 402. A target host 403 can be a machine in the production environment where the operator performs the task. The access to the machine is controlled by the access control system 402. An administrator 404 can be an engineer in the maintenance department. The administrator can be manually responsible for keeping the risk assessment system accurate and up to date. The system can also be updated automatically based on the notifications output by the system.

[0048] The access control system 402 manages the execution of commands to the target host 403. The access control system receives commands from the operator terminal and requests the rule-based classifier 405 to make a risk prediction. If the command is classified as "safe", the command is forwarded to the target host 403 and the output of the command is forwarded back to the operator terminal. Otherwise, the command is blocked and an error message is reported to the operator. In any case, the access control system 402 stores the command and the corresponding prediction in the command database 406.

[0049] The rule-based classifier 405 evaluates the risk associated with each individual command. The rule-based classifier 405 can use a predefined set of rules to evaluate the risk associated with each command. The rules database 407 stores the classification rules used to evaluate commands. These rules can be IF-THEN-ELSE rules that define which commands are allowed (or whitelisted) and which commands are blocked (or blacklisted). Each rule consists of a pattern and a label (e.g., safe or dangerous). If a command matches the rule pattern, it is assigned the corresponding rule label.

[0050] Rules are retrieved from the rules database 407 and compared to the input command to find a rule match for the command. Based on the matched rule, the command is classified. The classification gives an indication of the risk level associated with executing the corresponding command. A risk label corresponding to the classification is selected. For example, the label can be safe or dangerous. For commands that do not match, a default behavior can be defined (e.g., "dangerous").

[0051] The command database 406 stores historical risk predictions determined based on the rule set in the rules database 407, including the corresponding commands and their corresponding predicted classifications. The command-prediction pairs can be used to perform offline audits of the rule-based classifier.

[0052] The documentation system 408 includes a command knowledge base (in the form of a document database 409 in this example) for storing external documentation of terminal executable programs; a command analysis module (in the form of a command descriptor system 410 in this example) for generating an interpreted summary of terminal commands, as shown in 411.

[0053] The documentation system 408 extracts relevant information about the executed commands from the command documentation and summarizes this information in the form of natural language command descriptions. In this example, the natural language is English, but other languages can also be used.

[0054] The Al classifier 412 takes as input the original command and its generated textual command description and predicts the classification of the command. The classification can have a corresponding risk label that infers the risk level of the executed command. The Al model of the Al classifier can be trained based on historical commands retrieved from the command database 406.

[0055] The system compares the outputs of the rules-based system 405 and the Al-based classifier 412 to audit the rules.

[0056] In this example, the similarity search module 413 also classifies the command through similarity analysis with existing rules. The classification rules are retrieved from the rules database 407. The most similar rules to the input command are identified through a similarity function. Using the labels of the most similar rules, the risk prediction is obtained through aggregating the responses, for example using an averaging method or a voting method. The system integrates the existing set of rule knowledge into the audit system and provides a fallback solution in case of missing documentation. The similarity search system can overcome the potential limitations in the documentation knowledge base by leveraging the existing system rules to generalize the risk predictability in neighboring cases.

[0057] The responses from the similarity search system and the Al-based classifier are aggregated to generate the final output response. The response aggregator 414 receives the outputs of the similarity search system 413 and the Al classifier 412 and aggregates these responses to generate the final risk prediction for the command.

[0058] In the absence of the similarity search system, the final risk prediction is the output of the Al classifier 412.

[0059] The prediction comparator 415 retrieves the historical predictions of the rule-based prediction values and compares them with the final prediction generated by the response aggregator 414. If the predictions are different, the administrator 404 is notified of the issue to inform potential erroneous predictions, as shown in 416.

[0060] Figure 5 An example of the documentation system 408 is described. The documentation database 409 is a document-oriented database in which documentation files of terminal executable programs can be stored in a structured format. A single documentation file can detail how to use a single program, including syntax usage, program behavior, and available options. The documentation can be organized in sections, so each section can describe general usage or specific options and / or variable parameters. The documentation can be imported in an initial setup step using an import script that takes as input the standardized documentation files. Documentation that does not conform to the standardized format required by the import script can be converted. To improve the quality of the audit, internally defined commands can also be added. These commands can be documented directly in the standardized format or converted.

[0061] Examples of using different data sources can be summarized as follows:

[0062] —A document page describing standard operating system commands, files, and system calls;

[0063] —Internally defined scripts, programs, and aliases;

[0064] —Other third-party programs and tools commonly used in the IT environment;

[0065] —Other third-party or internal file names commonly used in the IT environment.

[0066] The documentation system acts as a command descriptor, a software module that generates an explanatory summary of terminal commands. Upon receiving a command, the command descriptor retrieves relevant documentation pages from the documentation database, extracts information related to the current command instance, and generates a short text describing the command's function in English.

[0067] For example, the external command documents contained in document database 409 may include any of the following:

[0068] —Management pages describing commonly used Linux commands (“manpages”), which are available online for free and can be directly imported into a documentation database;

[0069] —Other third-party commands in the documentation can be converted to manpages format;

[0070] —Scripts, internal commands, and aliases can be recorded directly in manpages format to improve audit quality;

[0071] —Common filenames.

[0072] exist Figure 5 In the example shown, document system 408 includes three sub-modules.

[0073] The first module is the parser program 501, which generates an abstract syntax tree (AST) representation of the input command 500, where the command markers can be tree nodes with program, option, or variable parameter markers.

[0074] The second module is the matcher program 502, which associates elements in the AST with documents in the document database by matching AST element names with document file titles and aliases. After identifying relevant documents, it extracts relevant paragraphs for each document by matching the AST's options and variable parameters with the document's paragraphs. A general description paragraph of the program (summary, usage, main functions) should ideally be included in the final selection.

[0075] The third module is a post-processing program 503 that connects the selected passages and / or cleans the extracted text, for example by removing HTML tags, links, author and copyright information, and / or by removing trailing spaces and normalizing the text. The final output is a single string describing the input command behavior in natural language format. The command description is specific to the specified program, options, and variable argument set. The description is sent to the AI model as an additional source of information for risk classification. If no relevant document is found, no command description is returned.

[0076] Thus, the input command is parsed to identify named entities. The output is an AST with entities (programs, options, variable arguments) as nodes. The document corresponding to the matching entity is retrieved. The system queries the document database using the program name found in the command and returns the exact match. The relevant textual information is extracted and a concise command description is constructed. The elements in the AST are associated with the database match. From the match, the following can be selected:

[0077] — Basic description (usage, main functionality)

[0078] — If a program, associated with the specified flags and arguments (e.g. -l, -r).

[0079] The extracted text can be cleaned and merged to obtain a concise command description. The cleaning process can remove redundant information (such as links, copyright, author information). The extracted text can be concatenated, for example by merging the selected passages into a single text block. The command description is sent to the AI model as an additional source for risk classification prediction. The natural language model (in this case an English language model) can learn to extract semantic patterns from the text, for example "dangerous" keywords (erase, delete, stop, disable) and "safe" keywords (query, print, return), which help to inform the risk of the command.

[0080] Figure 6 The main elements of the exemplary AI classifier 412 are summarized. In this example, the AI classifier is a neural network model based on the BERT model. Other model architectures can also be used to classify commands according to external documents.

[0081] In this example, the AI classifier includes a BERT head 601 pre-trained on an English corpus (referred to herein as an English BERT head). In other implementations, the head can implement another natural language. The first head 601 is used to process the document text (command description) generated by the document system. This head is used in conjunction with a full tokenizer 602. In this example, the full tokenizer 602 combines base tokenization (e.g., based on spacing and punctuation) with WordPiece tokenization (see https: / / ai.googleblog.com / 2021 / 12 / a-fast-wordpiece-tokenization-system.html). The full tokenizer 602 can efficiently encode English text into a numerical representation.

[0082] The classifier also includes a BERT head 603 pre-trained on a Bash command corpus (referred to herein as a Bash BERT head). In other implementations, the head can implement another computer programming language. The second head 603 is used to process the command string. This head is used in conjunction with a byte-pair encoder (BPE) tokenizer 604, which can adjust the encoding token set based on the input provided during pre-training and the desired vocabulary size. This head is able to preserve the sequential nature of the executed commands, thereby preserving the context of variable arguments, options, and flags. Thus, the AI-based classifier employs byte-pair tokenization and encoding that is specific to the Bash language. This head takes into account the relationships between tokens that often occur together, thereby improving the classification results.

[0083] The setup of the AI classifier can be divided into two steps: pre-training and fine-tuning, as shown in Figure 6 During the pre-training process, the BERT heads are pre-trained on a corresponding large corpus of unlabeled text (English head for English documents, Bash head for Bash commands). An example task used for pre-training is masked token prediction and next sentence prediction (see Devlin, Jacob, et al. "BERT: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018)). In the same step, the BPE tokenizer 604 can be trained on the Bash corpus to identify and encode tokens that occur in Bash commands, while the full tokenizer 602 for English can not require a training step.

[0084] The pre-trained model is then able to capture complex relationships between tokens and learn the grammatical structure of both languages (English and Bash in this case). During fine-tuning, the classification network is connected to the BERT heads to learn the classification task of determining the command classification (e.g., “dangerous” and “safe”). The labeled commands stored in the command database can be used for training. The pre-trained model will show the command, the corresponding command description, and its associated label. The model can then be trained during the fine-tuning phase to distinguish between “safe” and “dangerous” commands using a backpropagation algorithm.

[0085] During execution, the method of using the AI classifier 412 is as follows. The corresponding command to be classified is retrieved from the command database 406. The command is tokenized using the BPE tokenizer 604 and fed into the pre-trained Bash BERT head 603. The command description is retrieved from the documentation system 409. If the command description is available, it is tokenized using the full tokenizer 602 and fed into the pre-trained English BERT head 601.

[0086] An exemplary classification network (shown as 607) connecting the two BERT heads includes a fully connected layer that maps the BERT outputs 605, 606 to class predictions, as shown in Figure 6 “risk” of the particular command. The output vectors 605 and 606 produced by the two BERT heads are concatenated. A fully connected layer then applies a linear transformation that maps the concatenated input to the number of predicted classes (e.g., “safe” or “dangerous”). A SoftMax activation can be used to scale the output to obtain class probabilities. In the case where there is no relevant documentation for the particular command, then the Bash BERT head can be run with the English Bert head disabled, as described below.

[0087] In the case where there is a command description, the BERT heads 601, 603, and the classification network 607 process the tokenized inputs 605, 606 to output a classification prediction. In the case where there is no command description, only the Bash BERT head 603 and the classification network 607 process the tokenized input 605 to output a classification prediction. In this case, the classification network input from the English BERT head is set to 0.

[0088] Thus, this approach utilizes two language models (Bash for commands and English for documentation) to classify executed commands, e.g., to classify them as “dangerous” or “safe”.

[0089] Alternatively, a similarity search system can be used to overcome potential limitations in the document knowledge base (which may arise in some implementations) by leveraging existing system rules to generalize risk predictability in approximate scenarios. Thus, similarity search provides a fallback based on the most similar rule, ensuring that risk classification is always available in the audit system.

[0090] A schematic diagram of an exemplary similarity search system is shown below. Figure 7 As shown.

[0091] In this example, the similarity search system 413 implements the k-Nearest-Neighbour (KNN) algorithm to classify input commands based on rules stored in a rule database 407. Each rule is associated with a block / allow policy and can be used as a training label for the classification algorithm. In one implementation, during training, database rules are retrieved and an efficient tree-like (kd-tree) data structure is employed to enable fast neighbor queries during the inference phase. During inference, the input command is used as a query to return the most similar matching rule. To evaluate similarity, a similarity distance (e.g., Jaccard, Hamming, or Levenshtein distance) can be used to measure the minimum number of insertion, deletion, or substitution changes required to convert the command into a potential match. After retrieving the top k matches, the associated labels of these matches are used to predict the category of the input command through majority voting.

[0092] Therefore, rules similar to the input command can be identified, and the risk of the input command can be predicted by combining the risk classification of the identified rules.

[0093] When predictions from both the AI-based classifier 412 and the similarity search system 413 are available, the responses are aggregated to generate a single prediction. Different aggregation methods can be defined depending on the objectives of the auditing system.

[0094] When using a weighted aggregation method, each classification prediction ( ) is considered proportional to its weight:

[0095]

[0096] in:

[0097]

[0098] in , and It is the probability vector for each class. and It is a scalar constant with units and sum. Besides the predictive class ( In addition, the class probability of each prediction needs to be tracked .

[0099] Alternatively, the prediction class with the highest risk is selected using a recall-first weighted aggregation method:

[0100]

[0101] By default, recall-first aggregation can be enabled.

[0102] After the final prediction is determined, it is compared to the historical predictions of the same command performed by the rule-based system stored in the command database 406.

[0103] If the two classification predictions are inconsistent, an alert is outputted notifying the system administrator of the anomalous prediction. The administrator can choose to intervene on the rules database to improve the prediction quality of the existing risk assessment system. Alternatively, the rules database can also be updated automatically.

[0104] Figure 8 An operational sequence diagram of the exemplary system described above is shown. The numbering of the system components is as described previously.

[0105] At 801, a prediction job is initiated and an instruction is sent to the command database 406 to retrieve the command and its corresponding classification (determined according to the rules in the rules database 407) of a historical command. At 802, the command is sent to the similarity search system 413, which queries the rules database 407 at 803.

[0106] At 804, the similar rules are returned to the similarity search system 413. At 805, the similarity predictions are sent to the response aggregator 414.

[0107] At 806, the command is also sent to the command descriptor 410. At 807, the documents of the document database 409 are queried and at 808, the documents matching the command are sent to the document system. After the command description is formed, the command description is sent to the AI classifier 412 at 809.

[0108] The AI classifier 412 also receives the command at 810. At 811, the classification prediction formed by the AI classifier 412 is sent to the response aggregator 414. At 812, the final audit classification prediction is sent to the prediction comparator 415. The prediction comparator also receives the rule-based classification prediction from the command database 406 at 813.

[0109] At 814, the two classification predictions (the first determined based on rules and the second determined by the AI classifier described herein) are compared. If they are different, a notification is outputted at 815.

[0110] Figure 9 A flowchart showing steps of an exemplary method 900 for auditing rules in a rules-based command risk assessment system. An apparatus implementing the method can be communicatively connected to one or more data stores to store one or more commands and a pre-existing set of rules for classifying the commands. For each of at least one command, the method includes the following steps: at step 901, the method includes receiving a first predicted classification for the respective command, the first predicted classification determined based on the pre-existing set of rules. At step 902, the method includes forming a second predicted classification for the respective command based on external documentation for the respective command. At step 903, the method includes outputting a notification if the first prediction is different from the second prediction.

[0111] Figure 10 An example of an apparatus 1001 for performing the methods described herein is shown. The apparatus 1001 can include at least one processor (e.g., processor 1002) and at least one memory (e.g., memory 1003). The memory stores, in a non-transient manner, code executable by the processor to implement the apparatus in the manner described herein. The apparatus can be associated with multiple processors, and each processor can be used to perform different functions. For example, one or more of the processors or memories associated with the apparatus can be based in the cloud.

[0112] Internal rules in a risk assessment system are traditionally based on the knowledge of the operator over time, and can not cover all variable parameters and / or option combinations of past commands, and / or new or unknown commands. Furthermore, even for commands covered and classified by existing rules, the traditional classification risk described by the whitelist / blacklist rules can not be the same as the specific risk of the command. In this case, either a dangerous command is treated as a safe command, allowing the operator to run and cause damage to the system, or a safe command is unreasonably blocked, resulting in unnecessary blocking operations that can hinder the prompt operation.

[0113] The methods described herein advantageously utilize prior knowledge from an existing set of classification rules, and additionally utilize external sources of knowledge from the documentation of the command to assess the inherent risk of the command, thereby challenging and revising the existing classification rules. In embodiments of the invention, by evaluating the prediction made by a rules-based system and comparing it to an internal prediction calculated using external knowledge bases, risk predictions that are considered inaccurate can be reported, and the corresponding rules can be updated accordingly.

[0114] Accordingly, the approach to rule auditing in the rule-based risk classification system described herein combines external information from different sources and embedded knowledge to provide more accurate command classification. The approach provides a mechanism to dynamically discover and revise security rules to manage cloud infrastructure and employs an auditing mechanism to evaluate existing rules and suggest improvements. As a result, the quality of existing classification rules is improved, the effort required to revise existing rules is reduced, and the frequency of incidents is reduced.

[0115] The approach combines document and operator risk knowledge to classify commands through a sequential neural network model that can capture relationships and context between commands expressed in CLI language. The system can generate more accurate command classification, thereby reducing dangerous operations performed. The language model can generalize to unknown commands, subcommands, and flag combinations.

[0116] Existing rules can be reused to predict the risk of unknown commands. This enables the system to cover cases where no documentation can be found or the risk of a command cannot be assessed otherwise. The ability of the system to reuse existing knowledge enables the full utilization of existing resources and reduces the need for human intervention and retraining.

[0117] The approach can self-improve over time by training on new data samples observed from real environments. The approach can provide a higher level of security for target hosts. Furthermore, if new application command line interfaces (CLIs) are newly introduced in the environment, the approach can also adapt to these interfaces.

[0118] During the auditing process, the method discovers both false positive and false negative predictions by comparing all predictions for all samples, not just positive predictions or specific rules.

[0119] The method can handle all valid commands, including commands resulting from multiple commands connected through pipes, and other commands generated by command line language (&, xargs, etc.).

[0120] Embodiments of the invention also overcome several limitations of existing audit systems. Previous ML-based direct command classifiers are trained on a large number of labeled commands but fail to generalize to commands outside of those that are sufficiently similar to the ones shown in the training phase. This is because the classifier learns to recognize dangerous patterns in command strings. When the string changes significantly, the classifier fails to match the corresponding risk pattern and thus fails to classify accurately. Furthermore, a classifier trained on operator-provided labels agrees with the operator’s opinion and thus cannot be signaled to disagree. When existing rules do not apply and no risk assessment is available, a human-in-the-loop audit system reports to a human. Such a system thus requires constant human intervention to provide feedback and improve the existing classification system. Embodiments of the invention can reduce the number of reported revised commands because a command is only reported if it is deemed necessary to take a specific revision action (because of a predicted discrepancy) and because of its fallback mechanism it can always generate a risk classification, thus avoiding over-reporting due to lack of knowledge.

[0121] Previous audit schemes also typically fail to capture context effectively because the algorithms applied do not consider the sequence of tokens and their relationship in the sequence and cannot discover past misjudgments, especially false negative predictions. Due to the existence of a generalization gap, and the lack of applicable rules or lack of adjacent commands / patterns for similarity classification, previous audit schemes fail to assess unknown commands correctly. In implementations of the invention, by extracting information from the document, using a language model (e.g. BERT or GPT3) to classify the command, the context of the command can be captured. The language model can generalize to unknown commands, subcommands, and flag combinations and preserve the sequentiality of the executed commands, thus considering the context of using flags and subcommands. Using bytes for tokenization and encoding (considering the correlation between tokens) and transformer architecture also helps to model and process text sequences effectively before classifying the executed commands.

[0122] The scheme can also be extended to other CLI tools such as databases, interactive shell languages, and network device management, not just UNIX CLI and cloud environments.

[0123] In other implementations, the system can be deployed to prevent employees from running commands that can damage Windows operating systems, etc. Dangerous commands can be blocked and require additional permissions to execute. This approach can also be implemented in proxy servers, firewall systems, and databases, etc. and can be used to automatically review access control policies, including exceptions and white-list operations (defined as rules in access points) to prevent harmful connections from being initiated. This approach can also be used as part of standard operating procedure (SOP) verification. Given a natural language description of a SOP (as a document) and a list of commands or operations performed (as commands), the system can be used to verify that the SOP was followed. The system can also be used in the medical industry to automatically review diagnostic tests to verify that important diagnoses were not missed. A medical review system can take diagnostic test results as input and correct medical staff decisions to prevent false negative (FN) diagnoses of high-risk diseases.

[0124] Applicant hereby discloses all individually described features and any combination of two or more such features described herein. Such features or combinations of features can be implemented based on the knowledge of one of ordinary skill in the art having regard to the disclosure herein, without regard to whether such features or combinations of features solve any problem disclosed herein; and without regard to whether the scope of the claims is limited to such features or combinations of features. Applicant shows that aspects of the application can consist of any of such individual features or combinations of features. In view of the descriptions herein, various modifications can be made to the application as will be apparent to those skilled in the art.

Claims

1. An apparatus (1001) for auditing rules in a rule-based command risk assessment system, characterized in that, The apparatus is communicably connected to one or more data stores (406, 407, 1003) storing one or more commands and a pre-existing rule set for classifying the commands, and comprises one or more processors (1002) configured to, for each of at least one command: receive (901) a first predicted classification for the respective command, the first predicted classification determined based on the pre-existing rule set; form (902) a second predicted classification for the respective command based on external documentation of the respective command; output (903) a notification if the first prediction is different from the second prediction.

2. The apparatus (1001 ) according to claim 1, characterized by The one or more commands are historical commands issued by an operator (301, 401) of a target host (303, 403) in accordance with the pre-existing rule set, the target host (303, 403) being controlled by an access control system (302, 402) implementing the rule-based command risk assessment system (304, 405), each historical command having a corresponding first predicted classification determined based on the pre-existing rule set.

3. The apparatus (1001 ) according to any one of the preceding claims, characterized in that The first classification and the second classification each comprise an indication of a risk level associated with executing the respective command.

4. The apparatus (1001 ) according to any one of the preceding claims, characterized in that The external documentation comprises one or more of: documentation describing standard operating system commands, files or system calls; internally defined scripts, programs or aliases; third party programs, tools or internal file names.

5. The apparatus (1001 ) according to any one of the preceding claims, characterized by The one or more processors are configured to process the external documentation or information derived therefrom and the command to form the second predicted classification.

6. The apparatus (1001 ) according to any one of the preceding claims, characterized by The one or more processors are configured to: retrieve relevant documentation of the respective command from the external documentation; extract information related to the respective command from the relevant documentation; form a command description of the respective command based on the extracted information.

7. The apparatus (1001 ) according to claim 6, characterized by The command description is in a natural language format, the command being in a computer programming language format.

8. The apparatus (1001 ) according to claim 6 or 7, characterized by The one or more processors are configured to input the command description to an artificial intelligence model (412) and form the second predicted classification based on an output of the artificial intelligence model.

9. The apparatus (1001 ) according to claim 8, characterized by The artificial intelligence model comprises a first head (601) configured to process the command description and a second head (603) configured to process the respective command.

10. The apparatus (1001 ) according to any one of the preceding claims, characterized by The one or more processors are configured to form the second predicted classification based on a third classification prediction, the third classification prediction being formed based on one or more risk indications corresponding to one or more rules of the rule set selected based on a similarity to the respective command.

11. The apparatus (1001) according to claim 8 or claim 9 as dependent on claim 10, characterized in that, The one or more processors are configured to form the second predicted classification by aggregating the output of the artificial intelligence model (412) and the third classification prediction.

12. The apparatus (1001 ) according to any one of the preceding claims, characterized by The one or more processors are configured to output respective notifications of a plurality of commands, wherein the notifications are pre-grouped based on a similarity and reported to an administrator of the system.

13. The apparatus (1001 ) according to any one of the preceding claims, characterized by The one or more processors are configured to update one or more rules of the pre-existing rule set based on the notifications.

14. A method (900) for auditing rules in a rules-based command risk assessment system, characterized by, The method makes a device (1001) communicatively connectable to one or more data stores (406, 407, 1003) storing one or more commands and a pre-existing rule set for classifying the commands, the method comprising, for each of the at least one command: receiving (901) a first predicted classification for the respective command, the first predicted classification determined based on the pre-existing rule set; forming (902) a second predicted classification for the respective command based on external documentation of the respective command; if the first prediction is different from the second prediction, outputting (903) a notification.

15. A computer readable storage medium (1003), characterized in that, a computer readable medium having stored thereon computer readable instructions which, when executed by a computer, cause the computer to carry out the method (900) of claim 14.

Images