Automatic security test system using multi-layer language model

Abstract

Provided is an automatic security test system using a multi-layer language model. A cost-effective cybersecurity risk countermeasure system and method enable an LLM-based automatic security test for a managed cybersecurity service without leaking confidential information on a target system. In an embodiment, this is achieved by utilizing a flexible local language model that identifies and filters information specific to the target system when communicating with a public large language model to obtain a highly accurate security test pattern.

Description

Automated Security Testing System Using Multi-Layer Language Models

[0001] The present disclosure relates generally to security systems, and more particularly to a system and method for automated security testing using multi-layer language models.

[0002] A significant number of operational technology (OT) departments face budget and manpower constraints for conducting cybersecurity testing. Automation is a potential solution to security testing challenges. Emerging technologies such as generative artificial intelligence (AI) and large-scale language models (LLMs) hold promise for automating various tasks across industries. However, LLM-based automated security testing faces challenges. For example, generating effective security test patterns requires detailed input information about the target system's configuration, which may unnecessarily expose sensitive system configuration details. Therefore, addressing potential information leakage issues is crucial for successful implementation of LLM-based automated penetration testing in industries.

[0003] Public LLMs, such as ChatGPT (GPT-4) and Google Gemini, generate responses to user prompts with increasing accuracy because these models are constantly updated (ChatGPT, Google, and GEMINI are registered trademarks). Public LLMs are freely available and relatively inexpensive. In comparison, local LMs are user-customized versions of LLMs. To generate results specific to an environment, local LMs must be fine-tuned. However, local LMs are not automatically updated and must be individually customized. Therefore, compared to public LLMs, local LMs have relatively low accuracy in responding to general prompts. LLM-based penetration testing methodologies use a variety of tools to collect information from real systems. The collected information is provided in various formats, making it difficult to identify sensitive information using standard pattern matching algorithms. Existing approaches for automating LLM-based penetration testing typically include a user interaction function, an LLM, a tool operating unit, and a test environment, i.e., a target system. However, these approaches often only incorporate a single LLM and do not consider the criticality of information obtained from the target environment. Therefore, it is clear that improved systems and methods are needed that overcome these limitations.

[0004] Some aspects of the present disclosure automatically generate security test patterns suitable for a target system while preventing external leakage of system-related confidential information, such as system configuration information. The cybersecurity testing system includes a high-precision large-scale language model (public LLM) running on a public cloud and a domain-specific language model (local LM) running in a local environment. A scan module interacts directly with the target system to acquire data related to configuration or security information, such as vulnerabilities of the target system. A management server interacts with the scan module and manages the target system's configuration information and security test patterns. A fine-tuning module uses data generated by the scan module to teach the local LM the target system's configuration and security information. The management server sends prompts to the local LM, which include instructions and command requests for security tests that include confidential information about the system configuration. Upon receiving the prompts from the management server, the local LM converts the confidential system configuration information included in the prompts received from the management server into different words based on information about the target system learned from the data obtained from the scan module, generates prompts for the public LLM, and obtains corresponding results from the public LLM. When the management server receives the response from the public LLM, it performs a reverse translation process to generate command information that the security module can recognize and process.

[0005] In some aspects of the present disclosure, a method for conducting cybersecurity testing includes scanning a target system to obtain results including target system information; using the target system information to train a first language model to recognize sensitive information in the target system information; identifying a set of security test patterns for evaluating the results in a first phase of the security testing; creating a first prompt including the target system information for the security test patterns, which may be retrieved from a database; and communicating the first prompt to the first language model to perform the following steps: evaluating the first prompt to determine whether it contains sensitive data; In response to determining that the sensitive data is included, causing the first language model to perform a filtering process to generate a second prompt including a filtered set of commands that does not include the sensitive data, and communicating the second prompt to a security test manager; in response to receiving the second prompt, communicating the second prompt to the second language model to obtain a first model response; communicating a third prompt including the sensitive data to the first language model to obtain a second model response including a test command, where the test command may include user-provided input and may be stored in a database, for example after verification; and executing the test command to initiate a security test session.

[0006] The target system information may include configuration information of the target system, network information of the target system, or component information of the target system. The obtained results are used to generate structured system data associated with the configuration information, and the structured system data is converted into a format recognizable by a fine-tuning module including a first language model.

[0007] In some aspects, identifying the set of security test patterns includes obtaining a set of test conditions provided by a user or identifying a current location of the scan module relative to the structured system data. The test conditions and the current location can be used to determine the security test patterns. In some aspects, scanning the target system can include generating and communicating to the tool library commands for operating a toolset. Some aspects further include eliminating a command from the commands in response to determining that the command deviates from a predetermined standard.

[0008] Some aspects relate to an automated cybersecurity testing system that includes a first language model trained without using sensitive information of a target system and a second language model trained using information that includes sensitive information, wherein the second language model is configured to receive a first prompt that includes the sensitive information and return non-sensitive information, and the first language model is configured to generate a first model response in response to receiving a second prompt that includes the non-sensitive information.

[0009] The automated cybersecurity testing system may further include a scan module configured to scan a target system for sensitive information; an administration server configured to generate a first prompt, the first prompt including the sensitive information and a security testing request; and a fine-tuning module configured to train a second language model to learn the sensitive information, wherein the scan module may include a test tool library including a file system or database system for managing security testing tools, wherein the security testing tools include at least one of a network scanning tool, a vulnerability scanning tool, and a penetration testing tool.

[0010] In some aspects, the second language model, in response to receiving the first prompt, converts the sensitive information to non-sensitive information and communicates the non-sensitive information to the administrative server in a second prompt, and the second language model can be configured to obtain the sensitive information from at least one of a fine-tuning module, a scanning module, and user-provided data, and the fine-tuning module can automatically receive the input data or information in a machine-readable format.

[0011] Some aspects of the present disclosure include a method for scanning a target system to obtain results including target system information, a method for using the target system information to train a first language model to recognize sensitive information in the target system information, a method for identifying a set of security test patterns for evaluating results in a first phase of security testing, and a method for generating a first prompt including the target system information for the security test patterns, communicating the first prompt to the first language model, and performing the following steps: evaluating the first prompt to determine whether it contains sensitive data; In response to determining that the first language model contains the sensitive data, the system may include means for causing the first language model to perform the steps of: executing a filtering process to generate a second prompt including a filtered set of commands that does not contain the sensitive data; and communicating the second prompt to a security test manager; and in response to receiving the second prompt, communicating the second prompt to the second language model to obtain a first model response; communicating a third prompt including the sensitive data to the first language model to obtain a second model response that includes the test command; and means for executing the test command to initiate a security test session.

[0012] 1 illustrates an exemplary system for automated security testing, according to various embodiments of the present disclosure.

[0013] 1 illustrates a management server according to various embodiments of the present disclosure.

[0014] 1 illustrates a scan module according to various embodiments of the present disclosure.

[0015] 1 is a flowchart illustrating a process for automated security testing according to various embodiments of the present disclosure.

[0016] 1 is a flowchart illustrating a process for learning a target system structure according to various embodiments of the present disclosure.

[0017] 6 is a data flow diagram illustrating an implementation of the process of FIG. 5 in a test system according to various embodiments of the present disclosure.

[0018] 1 is a flowchart illustrating a process for determining a security test pattern according to various embodiments of the present disclosure.

[0019] 8 is a data flow diagram illustrating an implementation of the process of FIG. 7 in a test system according to various embodiments of the present disclosure.

[0020] 10 is a flowchart illustrating a process for creating a security test pattern prompt according to various embodiments of the present disclosure.

[0021] 10 is a data flow diagram illustrating an implementation of the process of FIG. 9 in a test system according to various embodiments of the present disclosure.

[0022] 10 is a flowchart illustrating a process for creating a security test command according to various embodiments of the present disclosure.

[0023] 12 is a data flow diagram illustrating an implementation of the process of FIG. 11 in a test system according to various embodiments of the present disclosure.

[0024] 1 is a flowchart illustrating a process for performing security testing and feedback according to various embodiments of the present disclosure.

[0025] 14 is a data flow diagram illustrating an implementation of the process of FIG. 13 in a test system according to various embodiments of the present disclosure.

[0026] 1 illustrates an exemplary computing environment having exemplary computer devices suitable for use with various embodiments of the present disclosure.

[0027] The following detailed description provides details of the figures and exemplary implementations of the present application. Reference numerals and redundant element descriptions between figures have been omitted for clarity. Terms used throughout this description are provided by way of example and are not intended to be limiting. For example, the use of the term "automatic" may include fully automatic or semi-automatic implementations with user or administrator control over certain aspects of the implementation, depending on the desired implementation of those skilled in the art practicing implementations of the present application. Selection can be performed by a user through a user interface or other input means, or through a desired algorithm. The exemplary implementations as described herein may be utilized alone or in combination, and the functionality of the exemplary implementations may be implemented through any means according to the desired implementation. In this document, the terms "target network system" and "target system" are used interchangeably.

[0028] 1 illustrates an exemplary system for automated security testing according to various embodiments of the present disclosure. The system 100 includes a public LLM 102, a local LM 104, a fine-tuning module 106, an administration server 108, a database 110, and a scan module 112 that interacts with a target network system 120. It is understood that the scan module 112, the administration server 108, the fine-tuning module 106, and the local LM 104 can be integrated into a single device.

[0029] The public LLM 102 is a pre-trained language model with more parameters than the local LM 104. The public LLM 102 does not have a pre-trained model that uses specific data or information about the target network system 120 to be tested. The public LLM 102 is configured to analyze prompts from the management server 108 and send results back to the management server 108. The public LLM 102 is characterized by its high accuracy and cost-effectiveness for common prompts, and the inputs available for training.

[0030] The local LM 104 is a user-customizable language model that typically has fewer parameters than the public LLM 102 and is trained using specific data or information about the target network system 120. The local LM 104 may be implemented as an LLM such as LlaMa2, Vicuña, or WizardLM, or an SLM such as TinyLlaMa, and can learn the target system / network architecture through the constraints of the fine-tuning module 106 and the scanning module 112. The local LM 104 can analyze prompts from the management server 108, generate prompts for the public LLM 102, and send the results back to the management server 108. The local LM 104 is configured to be trained to the requirements of a particular environment and the computing resources possessed by the user. Implementations of the LLM and / or SLM can run, for example, on an edge computer.

[0031] The fine-tuning module 106 is configured to provide training data to the local LM 104 to update its language model and tune the local LM 104 to learn the particular environment and configuration of the target system.

[0032] The management server 108 manages the scan modules 112. The management server 108 can visualize system and network information, fetch data from the managed scan modules 112, generate an overall system / network architecture picture from the fetched data, manually enter data, plan security test patterns, and generate prompts for the local LM 104 and for the public LLM 102 based on the results from the local LM 104.

[0033] Database 110 represents a database management system for managing and storing data and information including system and network structural information and security test patterns (adversarial databases).

[0034] The scan module 112 interacts with the target system 120 to be tested. The scan module 112 manages the test tools (tool library), stores the tool execution results and edge-specific data (data store), executes commands using the test tools (command manager), and generates prompts that are sent to the local LM 104 (command manager).

[0035] FIG. 2 illustrates a management server according to various embodiments of the present disclosure. The management server 200 includes a command operator 202, a system data manager 204, a security test manager 206, a database 208, an interface allocator 210, an HID controller database 212, a network interface control (NIC) device 214, an HID device 220, and an Internet Protocol (IP) network 222. The command operator 202 is a program configured to execute commands generated by the security test manager 206. The commands can be executed (e.g., via a tool library) on a scan module, such as the one shown in FIG. 1. The command operator 202 also receives results of command execution from the scan module and stores them in the database 208. The system data manager 204 is a program responsible for generating and managing structured system and network information data about the target system. This data is based on both input data and scan data fetched by the scan module. As discussed in more detail below, the system data manager 204 can generate prompts for the LLM and update the structured system information based on responses to these prompts.

[0036] Security test manager 206 is a program that generates and manages security test plans and specific security test patterns (including tool commands). Similar to system data manager 204, security test manager 206 can generate prompts for the LLM and update structured system information based on responses to these prompts. Database 208 is a system for storing and managing data related to system and network structure, information about target systems, and information about security tests, such as test plans and specific commands. Database 208 can be located on the same hardware device as the management server (shown in FIG. 1) or on a different device.

[0037] 3 illustrates a scan module 300 according to various embodiments of the present disclosure. The scan module 300 includes a human machine interface (HMI) 302, a HID controller database 212, a NIC device 214, a tool library 308, and an interface allocator / pipe 210.

[0038] The HMI 302 includes features such as a web browser, a GUI including the X Window System, and provides a user-friendly interface for users to interact with the management server as depicted in FIG. 1 . The HID controller database 212 may include software that controls the physical HID devices 220 and can facilitate user interaction with the system. The NIC device 214 is used to connect the system to a TCP / IP network 222 to enable network communication for the system. The tool library 308 is a file system or database system that manages security testing tools such as network scanning tools (e.g., nmap), vulnerability scanning tools (e.g., GVM, NESSUS), penetration testing tools (e.g., Metasplot, Empire), and runtime environments for executing security testing tools (e.g., Python, C libraries, Powershell). The interface allocator / pipe 210 allocates and manages network sockets or serial interfaces to each program running on the device. The interface allocator / pipe 210 can be implemented as a function of an operating system (eg, MS Windows, Linux) (NESSUS, Metasplot, Python, Powershell, Windows, and Linux are each registered trademarks).

[0039] FIG. 4 is a flowchart illustrating a process for automated security testing according to various embodiments of the present disclosure. Process 400 begins at step 402, where a target system structure is learned, as discussed in further detail with reference to FIGS. 5 and 6. At step 404, a security test pattern is determined, as discussed in further detail with reference to FIGS. 7 and 8. At step 406, a security test pattern prompt is created, as discussed in further detail with reference to FIGS. 9 and 10. At step 408, a security test command is created, as discussed in further detail with reference to FIGS. 11 and 12. Finally, at step 410, security testing and feedback is performed, as discussed in further detail with reference to FIGS. 13 and 14. Those skilled in the art will recognize that (1) certain steps may be performed optionally, (2) steps are not limited to the particular order described herein, (3) certain steps may be performed in different orders, and (4) certain steps may occur simultaneously.

[0040] FIG. 5 is a flowchart illustrating a process for learning a target system structure according to various embodiments of the present disclosure. Process 500 begins at step 502, where a command set can be generated by a command operator using, for example, pre-built tools or an LLM. The command operator can use tools available in a tool library. In addition, the command operator can incorporate user knowledge or expertise and / or additional information about the system configuration into the security testing process, for example, via a human-machine interface (HMI). At step 504, the generated command set is executed to perform one or more scans on the target system and obtain scan results. At step 506, the scan results can be stored in a database, for example, for future reference or analysis. At step 508, a system data manager can use the results to generate structured system information related to the system configuration, representing a comprehensive overview of the target system's configuration. At step 510, a fine-tuning module converts the structured system information into a readily recognizable format. In step 512, the transformed information is used to update the local LM's large scale language model (LLM) to ensure that the LLM uses the most up-to-date system configuration information to generate accurate and relevant security test patterns.

[0041] 6 is a data flow diagram illustrating an implementation of the process of FIG. 5 in a test system according to various embodiments of the present disclosure. Test system 600 includes a user 602, a target system 604, a scan module 606, a management server 622, a fine-tuning module 640, and a local LM 646. As depicted, scan module 606 includes an HMI 608 and a tool library 610. Management server 622 includes a database 624, a system data manager 626, a command operator 628, and a security test manager 630. Fine-tuning module 640 includes a local LM tuner 642 that accesses local LM 646, which includes local LLM 648.

[0042] In operation, a command operator 628 generates commands using a number of pre-built tools (or LLMs) in a tool library 610 and / or user-entered data, which may be stored in a database 624 via an HMI 608. When executed by a scan module 606, these commands perform a scan of the target system 604, the results of which may be stored in a database 624. A system data manager 626 uses these results to generate structured system information 638 related to the system configuration of the target system 604. A fine-tuning module 640 converts the structured system information 638 into a format recognizable by a local LM tuner 642. The converted information is then used to update the LLM 648 in a local LM 646.

[0043] FIG. 7 is a flowchart illustrating a process for determining a security test pattern according to various embodiments of the present disclosure. Process 700 begins in step 702, where a current location of a scan module relative to structured system information is identified based on collected information. In step 704, user-defined security test conditions, such as test conditions and termination conditions, are received via the HMI. In step 706, based on the current location of the scan module and the input information, a security test manager determines a security test flow and / or security test pattern and initial test phase. FIG. 8 is a data flow diagram illustrating an implementation of the process of FIG. 7 in a test system according to various embodiments of the present disclosure. Numbers that are the same as in FIG. 6 indicate similar elements. For the sake of brevity, descriptions or their functions will not be repeated here.

[0044] FIG. 9 is a flowchart illustrating a process for creating a prompt for a security test pattern, according to various embodiments of the present disclosure. Once the initial test phase has been determined, the security test manager determines the test pattern to be executed in the associated phase. In making this determination, the security test manager generates a prompt to query the local LM for a command to be executed based on the current phase, current location, and structured system information, in step 902. In step 904, the security test manager communicates the generated prompt to the local LM. In step 906, upon receiving the prompt, the local LM interprets its content and determines whether the prompt contains sensitive information about the target system. In step 908, if the verification process reveals that the prompt contains such sensitive information, the local LM performs a filtering process on the relevant information. Then, in step 910, the local LM responds to the security test manager by communicating an updated prompt containing the filtered information. In step 912, the updated prompt, now without the sensitive information, is sent to the public LLM to ensure that the public LLM receives and processes only non-sensitive information that is safe for public access. FIG. 10 is a data flow diagram illustrating an implementation of the process of FIG. 9 in a test system according to various embodiments of the present disclosure.

[0045] 10 includes a user 602, a target system 604, a scan module 606, an administration server 622, a local LM 646 having a language model 648, and a public LLM 1002 having an LLM 1004. Similar to FIG. 6, the scan module 606 includes an HMI 608 and a tool library 610. The administration server 622 includes a database 624, a system data manager 626, a command operator 628, and a security test manager 630.

[0046] FIG. 11 is a flowchart illustrating a process for creating security test commands according to various embodiments of the present disclosure. Process 1100 begins at step 1102, where a security test manager can receive a prompt from a public LLM containing the response results generated in the flowchart of FIG. 9. At step 1104, the prompt is converted into a command format containing information specific to the target system to ensure the command conforms to the specific requirements and characteristics of the target system. At step 1106, the security test manager communicates the converted prompt to the local LM and obtains a result including the adapted command. At step 1108, in response to receiving the command from the local LM, the security test manager validates the command. If a command is found to deviate from predetermined criteria, the command is rejected to ensure that only valid and relevant commands are executed during the security testing process. At step 1110, the command is stored in a database as a security test pattern to be accessed and executed during the security testing process. FIG. 12 is a data flow diagram illustrating an implementation of the process of FIG. 11 in a test system according to various embodiments of the present disclosure.

[0047] FIG. 13 is a flowchart illustrating a process for performing security testing and feedback according to various embodiments of the present disclosure. Process 1300 begins at step 1302, where a command operator accesses a database to obtain a list of security test patterns and can initiate a security test session. In step 1304, based on the test patterns, the command operator operates a suite of tools in a tool library to execute commands configured to probe the target system for potential vulnerabilities. In step 1306, the results of the tool execution are stored in a database. In step 1308, the security test manager determines whether the goals of the current phase have been achieved based on the results of the tool execution, for example, in an iterative process, and evaluates any additional information obtained about the target system. In step 1310, if the goals of the current phase have been achieved, the next test pattern is determined based on the security test flow information. Otherwise, a different test pattern is generated, and the process returns to step 406 of FIG. 4. FIG. 14 is a data flow diagram illustrating an implementation of the process of FIG. 13 in a test system according to various embodiments of the present disclosure.

[0048] Once the process of running the security tests (see, e.g., step 410 of FIG. 4) is complete, the results of the security tests are generated and stored in a database within the management server. Note that the results may be stored in raw data format or may be partially extracted from the original results from the scan module.

[0049] Any headings used herein are for organizational purposes only and should not be used to limit the scope of the present disclosure.

[0050] Structured System Information As used herein, structured system information may include information about the architecture, configuration, and security vulnerabilities of a target test system. This includes information describing the components and attributes of the target system, such as a list of assets, a list of logical networks, and a list of physical locations where each asset is deployed. A local LM as disclosed herein can learn the structure of the system and network from this system structure information.

[0051] Each asset information entry may include a unique asset name and attribute information about the characteristics of the asset, information about the logical network to which the asset is connected, information about the physical location where the asset is installed, and identified security vulnerability information, which may include associated vulnerabilities specific to the hardware or software, or vulnerabilities identified during security testing.

[0052] The structured system information may further include information defining the characteristics of each logical network and physical location. Each logical network entry may have a unique name and attribute information about the logical network, such as a media type. Each physical location entry may have a unique name and attribute information about the physical location, such as a security level.

[0053] The structured system information can be defined in a structured and machine-readable format, such as JSON, XML, and a relational database. The information can be managed in a database that the management server is configured to interact with. A fine-tuning module for a local LM as described herein can read the structured system information, for example, to train the local LM. The local LM can recognize what assets are in the target system, network structure, and physical structure.

[0054] Table 1 shows an example of structured system information. In this example, the information is written in JSON format. As will be appreciated by those skilled in the art, any suitable machine-readable format may be employed, such as CSV or XML. The underlined text in Table 1 indicates actual or potentially sensitive system-specific information that the local LM should convert to non-sensitive information.

[0055] Confidential Information As used herein, confidential information refers to details of a target system that its owner does not want to be exposed to the public. Examples of such information include names and any other identifiers of assets, network structure, physical structure, system user information, authentication information, elements of assets, information about custom applications running on the internal network (such as source code), user-provided information, and security vulnerabilities in the target system environment and any assets.

[0056] In embodiments, certain information can be hidden. The management server generates prompts based on the security test plan and structured system information about the target system. These raw prompts contain information about the target system defined in the structured system information, as shown in Table 2.

[0057] The local LM identifies and suggests specific words and generates modified prompts. Examples of possible results are shown in Table 3.

[0058] The management server can parse the response from the local LM and forward the modified prompt to the public LLM, as shown in Table 4.

[0059] Once the management server receives the response from the public LLM, it generates a prompt for the local LM to obtain the applicable command for the target system. This allows for more precise security test commands to be obtained from the public LLM while hiding information specific to the target system. An example is shown in Table 5.

[0060] It should be noted that while relatively simple pattern matching algorithms may be used to extract sensitive information, language models are typically more effective in determining sensitive information from input sources and offer greater flexibility in extracting potentially sensitive information. An example of a local LM is shown in Table 6.

[0061] In an embodiment, a prompt can be created to test specific attributes to create a detailed security test pattern for a specific component, such as "WINDOWS 10 Professionals Version 10.0.22621.xxxx," as a result of scanning the target system. The "WINDOWS 10 Professionals Version 10.0.22621.xxxx" information itself is not confidential information unless the target system or owner information is disclosed. Therefore, a specific security test pattern can be fetched from the public LLM without disclosing confidential information. An image of this is shown in Table 7.

[0062] 15 illustrates an exemplary computing environment having an exemplary computing device suitable for use with various embodiments of the present disclosure. The computing device 1505 in computing environment 1500 may include one or more processing units, cores, or processors 1510, memory 1515 (e.g., RAM, ROM, and / or the like), internal storage 1520 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interface 1525, any of which may be coupled over a communication mechanism or bus 1530 for communicating information or may be embedded in the computing device 1505. The I / O interface 1525 may also be configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0063] The computing device 1505 may be communicatively coupled to an input / user interface 1535 and an output device / interface 1540. One or both of the input / user interface 1535 and the output device / interface 1540 may be wired or wireless interfaces and may be detachable. The input / user interface 1535 may include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touchscreen interface, keyboard, pointing / cursor control, microphone, camera, Braille, motion sensor, optical reader, and / or the like). The output device / interface 1540 may include a display, television, monitor, printer, speaker, Braille, or the like. In some example implementations, the input / user interface 1535 and the output device / interface 1540 may be embedded in the computing device 1505 or physically coupled to the computing device 1505. In other example implementations, other computing devices can function as or provide the functionality of input / user interface 1535 and output device / interface 1540 of computing device 1505 .

[0064] Examples of computing devices 1505 may include highly mobile devices (e.g., smartphones, devices mounted on vehicles or other machines, devices carried by humans or animals, and the like), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, and the like), and devices not designed for mobile use (e.g., desktop computers, other computers, information kiosks, televisions with and / or coupled to one or more processors, radios, and the like).

[0065] Computing device 1505 may be communicatively coupled (e.g., via I / O interface 1525) to external storage 1545 and a network 1550 for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 1505, or any connected computing device, may function as, provide services for, or be referred to as, a server, a client, a thin server, a general-purpose machine, a special-purpose machine, or another label.

[0066] I / O interface 1525 may include wired and / or wireless interfaces using any communications or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMax, modem, cellular network protocols, and the like) for communicating information to and from at least all connected components, devices, and networks in computing environment 1500. Network 1550 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, and the like).

[0067] The computing device 1505 may use and / or communicate using computer-usable or computer-readable media, including transitory and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, and the like. Non-transitory media include magnetic media (e.g., disks and tape), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0068] The computing device 1505 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some exemplary computing environments. The computer-executable instructions can be retrieved from transitory media or stored on and retrieved from non-transitory media. The executable instructions can be written in one or more of any programming, scripting, and machine language, such as C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others (Java and Perl are registered trademarks).

[0069] The processor 1510 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, including a logic unit 1560, an application programming interface (API) unit 1565, an input unit 1570, an output unit 1575, and an inter-unit communication mechanism 1595 through which different units communicate with each other, with the OS, and with other applications (not shown). The described units and elements may vary in design, function, configuration, or implementation and are not limited to the provided description. The processor 1510 can be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units.

[0070] In some example implementations, information or instructions to execute, once received by API unit 1565, may be communicated to one or more other units (e.g., logic unit 1560, input unit 1570, output unit 1575). In some examples, logic unit 1560 may be configured to control the flow of information between units and direct the services provided by API unit 1565, input unit 1570, and output unit 1575 in some example implementations described above. For example, the flow of one or more processes or implementations may be controlled by logic unit 1560 alone or in cooperation with API unit 1565. Input unit 1570 may be configured to obtain inputs for the computations described in the example implementations, and output unit 1575 may be configured to provide outputs based on the computations described in the example implementations.

[0071] The processor 1510 may be configured to execute a method or computer instructions that may include, for example, as shown in Figures 1 and 10, scanning a target system such as that shown in Figure 1 to obtain results including target system information, using the target system information to train a first language model to recognize sensitive information in the target system information, identifying a set of security test patterns for evaluating the results in a first phase of security testing, creating a first prompt for the security test patterns including the target system information, and communicating the first prompt to the first language model.

[0072] As a result, the first language model performs steps such as, for example, as shown in Figures 1 and 10, evaluating the first prompt to determine whether it contains sensitive data, performing a filtering process in response to determining that the first prompt contains sensitive data to generate a second prompt including a filtered set of commands that does not include the sensitive data, and communicating the second or updated prompt to the security test manager.

[0073] The processor 1510 may be configured to execute methods or computer instructions that may include, for example, in response to receiving a second prompt, communicating the second prompt to a second language model to obtain a first model response, as illustrated in FIGS. 1 and 10, and communicating a third prompt including sensitive data to the first language model to obtain a second model response including a test command, as illustrated in FIGS. 1 and 10.

[0074] The processor 1510 may be configured to execute methods or computer instructions that may include, for example, executing a test command to initiate a security test session as shown in FIG. 1 and / or FIG. 14 .

[0075] Some portions of the detailed descriptions are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithms and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovations to others skilled in the art. An algorithm is a sequence of defined steps leading to a desired end state or result. In exemplary implementations, the steps performed require specific quantities of physical manipulations to achieve a specific result.

[0076] Unless otherwise specifically stated, and as will be apparent from the discussion, throughout this description, discussions utilizing terms such as "processing," "operating," "calculating," "determining," "displaying," or similar terms will be understood to include operations and processes of a computer system or other information processing device that manipulate and transform data represented as physical (electronic) quantities in the computer system's registers and memory into other data similarly represented as physical quantities in the computer system's memory, registers, or other information storage, transmission, or display devices.

[0077] Example implementations may also relate to apparatuses for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. Computer-readable storage media may include tangible media such as optical disks, magnetic disks, read-only memory, random-access memory, solid-state devices, drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation containing instructions for performing the operations of a desired implementation.

[0078] Various general-purpose systems may be used with programs and modules in accordance with the examples herein, or it may prove convenient to construct more specialized apparatus to perform the desired method steps. In addition, the exemplary implementations are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages ​​can be used to implement the techniques of the exemplary implementations as described herein. Instructions in a programming language may be executed by one or more processing devices, for example, a central processing unit (CPU), processor, or controller.

[0079] As is known in the art, the operations described above can be performed by hardware, software, or a combination of software and hardware. Various aspects of exemplary implementations can be implemented using circuits and logic devices (hardware), while other aspects can be implemented using instructions stored on a machine-readable medium (software), which, when executed by a processor, cause the processor to perform methods for implementing implementations of the present application. Furthermore, some exemplary implementations of the present application can be performed solely in hardware, while other exemplary implementations can be performed solely in software. Furthermore, the various functions described can be performed in a single unit or distributed across multiple components in any number of ways. When performed by software, the methods can be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions can be stored on the medium in a compressed and / or encrypted format.

[0080] Furthermore, other implementations of the present application will be apparent to those skilled in the art from consideration of this specification and practice of the techniques of the present application. Various aspects and / or components of the described exemplary implementations may be used alone or in any combination. It is intended that the specification and exemplary implementations be considered as examples only, with the true scope and spirit of the present application being indicated by the following claims.

[0081] 100 System 102 Public LLM 104 Local LM 106 Fine-tuning module 108 Management server 110 Database 112 Scan module 120 Target network system 200 Management server 202 Command operator 204 System data manager 206 Security test manager 208 Database 210 Interface allocator 212 HID controller database 214 NIC device 220 HID device 222 IP network 302 HMI 308 Tool library 600 Test system 602 User 604 Target system 606 Scan module 608 HMI 610 Tool library 622 Management server 624 Database 626 System data manager 628 Command operator 630 Security test manager 638 Structured system information 640 Fine-tuning module 642 Local LM tuner 646 Local LM 648 LM 648 LLM 800 Test system 1000 Test system 1002 Public LLM 1004 LLM 1200 Test system 1400 Test system 1500 Computing environment 1505 Computer device 1510 Processor 1515 Memory 1520 Internal storage 1525 I / O interface 1530 Bus 1535 Input / user interface 1540 Output device / interface 1545 External storage 1550 Network 1560 Logical unit 1565 API unit 1570 Input unit 1575 Output unit 1595 Inter-unit communication mechanism

Claims

1. A method for conducting cybersecurity testing, comprising: scanning a target system to obtain results including target system information; using the target system information to train a first language model to recognize sensitive information in the target system information; identifying a set of security test patterns for evaluating the results in a first phase of security testing; creating a first prompt including the target system information for the security test patterns; communicating the first prompt to the first language model to cause the first language model to perform the following steps: evaluating the first prompt to determine whether it includes sensitive data; in response to determining that the first prompt includes the sensitive data, performing a filtering process to generate a second prompt including a filtered set of commands that does not include the sensitive data; and communicating the second prompt to a security test manager; in response to receiving the second prompt, communicating the second prompt to a second language model to obtain a first model response; and communicating a third prompt including the sensitive data to the first language model to obtain a second model response including test commands. executing the test command to initiate a security test session.

2. The method of claim 1, wherein the target system information includes at least one of configuration information of the target system, network information of the target system, and component information of the target system.

3. The method of claim 2, further comprising using the results to generate structured system data associated with the configuration information.

4. The method of claim 3, further comprising converting the structured system data into a format recognizable by a fine-tuning module that includes the first language model.

5. The method of claim 3, wherein identifying the set of security test patterns further comprises obtaining a set of test conditions provided by a user.

6. The method of claim 5, wherein identifying the set of security test patterns further comprises identifying a current location of a scan module relative to the structured system data.

7. The method of claim 6, further comprising: determining the security test pattern using the set of test conditions and the current location.

8. The method of claim 6, wherein at least some of the commands include user-provided input.

9. The method of claim 1, wherein scanning the target system includes generating and communicating commands to a tool library for operating a toolset.

10. The method of claim 1, further comprising, in response to determining that a command among the commands deviates from a predetermined standard, eliminating the command.

11. The method of claim 1, further comprising validating and storing the test command in a database.

12. The method of claim 11, wherein the set of test patterns is retrieved from the database.

13. An automated cybersecurity testing system comprising: a first language model trained without using classified information of a target system; and a second language model trained using information including the classified information, wherein the second language model is configured to receive a first prompt including the classified information and return non-classified information; and the first language model is configured to generate a first model response in response to receiving a second prompt including the non-classified information.

14. The system of claim 13, further comprising: a scanning module configured to scan a target system for the sensitive information; an administrative server configured to generate the first prompt, the first prompt including the sensitive information and a request for security testing; and a fine-tuning module configured to train the second language model to learn the sensitive information.

15. The system of claim 14, wherein the second language model, in response to receiving the first prompt, converts the sensitive information into non-sensitive information and communicates the non-sensitive information to the administrative server in the second prompt.

16. The system of claim 14, wherein the second language model is configured to derive the sensitive information from at least one of the fine-tuning module, the scanning module, and user-provided data.

17. The system of claim 14, wherein the fine-tuning module is configured to automatically receive input data or information in a machine-readable format.

18. The system of claim 14, further comprising: a database configured to store information about the target system, wherein the information includes at least one of network information, security test information, and test pattern results.

19. The system of claim 14, wherein the scan module includes a test tool library including a file system or database system for managing security testing tools, the security testing tools including at least one of a network scanning tool, a vulnerability scanning tool, and a penetration testing tool.

20. The system of claim 13, wherein the first language model includes more parameters than the second language model.

Images