Rule-based compliance detection script automatic generation method and system

Abstract

The application provides a rule-based compliance detection script automatic generation method and system, relates to the fields of software engineering and automation technology, and comprises the following steps: receiving user input information, analyzing the user input information, and generating a structured requirement specification; based on the structured requirement specification, searching for related code templates and best practice information from a preset professional knowledge base; generating prompt information based on the structured requirement specification, the code templates, the best practice information and the user input information, inputting the prompt information into a large language model, and outputting an initial compliance detection script; performing multi-layer quality verification on the initial compliance detection script; if the verification is passed, the initial compliance detection script is output as a final compliance detection script; and if the verification is not passed, correction information is generated and returned to the large language model for iterative correction until the verification is passed. The application solves the problems of poor flexibility and unstable quality of the script generated by the prior art.

Description

Technical Field

[0001] This invention relates to the fields of software engineering and automation technology, and in particular to a method and system for automatically generating rule-based compliance detection scripts based on a large language model. Background Technology

[0002] With the rapid development of cloud computing, containerization technology, and microservice architecture, the complexity and dynamism of modern IT infrastructure have reached unprecedented levels. Enterprises face increasingly severe security challenges during their digital transformation and must ensure that their systems and applications strictly comply with various international security standards and compliance requirements, such as authoritative specifications like the Center for Internet Security Benchmarks (CIS Benchmarks) and Service Organization Control Reporting (SOC2).

[0003] Against this backdrop, some open-source declarative infrastructure testing tools, such as Chef InSpec (Chef Infrastructure Specification), are used to describe and verify system security configurations and compliance rules. They employ a domain-specific language (DSL) based on Ruby syntax, allowing security experts and operations personnel to describe the system's security baseline requirements in a human-readable manner and automatically verify whether the system configuration conforms to these predetermined compliance policies.

[0004] However, the traditional work of writing compliance detection scripts relies entirely on security experts to do it manually. This approach not only requires the writer to have a deep understanding of system architecture and security, but also to be proficient in the specific syntax of InSpec, resulting in problems such as insufficient flexibility, low level of intelligence, high maintenance costs, limited coverage, and unstable quality of generated scripts.

[0005] To improve script generation efficiency, existing technologies have proposed some automated or semi-automated solutions, but they all have significant drawbacks: (1) Template-based method: Scripts are generated through predefined code templates. While this reduces repetitive coding to some extent, it is extremely inflexible and cannot dynamically adapt to ever-changing system environments and personalized business needs. When faced with new security requirements or unique system configurations, extensive manual intervention is still required for modification and adaptation.

[0006] (2) Rule engine method: Generates code based on a series of pre-configured "if-then" rules. Although it is more intelligent than the template method, it is expensive to maintain. Whenever a new technology stack, a new security threat, or a new compliance clause appears, experts need to manually write and update a large number of rules, which is difficult to keep up with the rapid evolution of technology and threat landscape. In addition, its reasoning ability is limited and it is difficult to handle complex scenarios that require deep semantic understanding.

[0007] (3) Hybrid approach based on Large Language Model (LLM). For example, although some methods combine large language model and knowledge base retrieval, their core task is to generate general test cases and executable SQL statements, and they cannot directly generate executable scripts that conform to the syntax specifications of declarative security testing frameworks; moreover, the quality verification of this approach relies entirely on actually executing the generated SQL script in the sandbox for result comparison, while for security compliance scripts, the script may involve checking the configuration of the production system, which is not suitable and should not be simulated in the sandbox; in addition, when script defects are detected, it still relies on users to manually submit correction feedback.

[0008] In summary, existing technologies generally suffer from insufficient flexibility, low intelligence, high maintenance costs, and a lack of reliable quality assurance. This results in unstable quality of the generated compliance detection scripts, which may contain errors or omissions, leading to false positives or false negatives in security compliance checks and posing potential risks. Therefore, there is an urgent need for a rule-based automated method for generating compliance detection scripts that is highly flexible, intelligent, low-maintenance, and possesses reliable quality assurance. Summary of the Invention

[0009] To address the aforementioned problems, this invention provides an automated method and system for generating rule-based compliance detection scripts based on a large language model. It integrates deep semantic understanding, knowledge enhancement generation, pre-processing multi-layer static verification, and iterative correction, thus solving the problems of poor flexibility and unstable quality in compliance detection scripts generated by existing technologies.

[0010] On one hand, the present invention provides a rule-based method for automatically generating compliance detection scripts, the method comprising: Receive user input information, parse the user input information, and generate a structured requirements specification. Based on the structured requirements specification, relevant code templates and best practice information are retrieved from a pre-defined professional knowledge base; Based on the structured requirements specification, the code template, the best practice information, and the user input information, a prompt message is generated and input into the large language model to output the initial compliance detection script; The initial compliance detection script undergoes multi-layer quality verification; if the verification passes, it is output as the final compliance detection script; if the verification fails, correction information is generated and returned to the large language model for iterative correction until the verification passes.

[0011] Furthermore, the user input information includes natural language descriptions and / or structured configuration files; parsing the user input information specifically includes: The user's natural language requirements are obtained by identifying key entities and user intent in the natural language description using a natural language processor. And / or, the configuration information in the structured configuration file is extracted by the configuration parser to obtain the system configuration status; Based on the user's natural language requirements and / or the system configuration status, a structured requirements specification is generated.

[0012] Furthermore, the structured requirements specification is generated, specifically including: The identified key entities are mapped to standardized technical terms; Based on the aforementioned technical terms, the system configuration status, and the preset knowledge graph, rule reasoning is performed to transform the user's natural language requirements into one or more control points; Based on the control points, a machine-readable structured requirements specification is generated.

[0013] Furthermore, the preset professional knowledge base includes a vector database storing code snippets and a knowledge graph storing best practice metadata.

[0014] Furthermore, relevant code templates and best practice information are retrieved from a pre-defined professional knowledge base, specifically including: Obtain the control points from the structured requirements specification; Based on the control points, semantic similarity matching is performed in the vector database to retrieve relevant code snippets and obtain code templates; Based on the control points, the relevant best practice metadata is queried in the knowledge graph to obtain best practice information.

[0015] Furthermore, the large language model is a domain-specific large model obtained by performing domain fine-tuning using a compliance detection script.

[0016] Furthermore, the multi-layer quality verification includes at least: Syntax validation: The syntax correctness of the initial compliance detection script is verified by constructing an abstract syntax tree of the initial compliance detection script; Standard verification: The initial compliance detection script is checked using static code analysis tools to see if it conforms to the coding standards for compliance detection scripts; Logical verification: The abstract syntax tree of the initial compliance detection script is compared with the structured requirements specification to ensure semantic consistency between the two.

[0017] Furthermore, generating the correction information specifically includes: The error type and location are identified during verification, and a text description guiding the large language model to make corrections is generated based on the error type, thus generating the correction information.

[0018] Furthermore, while outputting the compliance detection script, a metadata report is generated; wherein, the metadata report includes the source of the compliance detection script, the applicable environment, the compliance standards, and deployment recommendations.

[0019] On the other hand, the present invention also provides a rule-based compliance detection script automated generation system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0020] In summary, this invention provides a rule-based method and system for automatically generating compliance detection scripts, which achieves the following advantages compared to existing technologies: (1) This invention deeply integrates requirement analysis, knowledge retrieval, intelligent generation of large language models, and multi-layer quality verification to construct an end-to-end automated script generation method; it completely changes the inefficient mode of relying entirely on security experts to manually write scripts, reducing the work that may take hours or even days to minutes; at the same time, the automated generation of scripts avoids human negligence, fundamentally improves the accuracy and consistency of scripts, solves the problems of poor flexibility and unstable quality of existing compliance detection scripts, and significantly improves the efficiency and reliability of large-scale, dynamic compliance audits of enterprises.

[0021] (2) This invention combines multimodal input understanding with knowledge enhancement generation, which not only understands the user's natural language needs, but also combines structured configuration files and uses professional knowledge base for retrieval enhancement generation; so that the generated script is not a simple template filling, but a high-precision script that deeply integrates the specific system environment, compliance standards and industry best practices, and conforms to the business context, ensuring the flexibility and practicality of the script in actual execution.

[0022] (3) By introducing multi-layer quality verification and iterative correction, this invention adds deterministic quality assurance to the output of probabilistic large language models. Through the three progressive checks of syntax verification, specification verification and logic verification, it can effectively intercept and automatically correct various potential errors. This intelligent closed loop of generation-verification-feedback-regeneration ensures that the final output is a reliable script that has undergone strict quality inspection, solves the pain point of unstable output quality of general code, and greatly reduces the security risk of false alarms or missed alarms caused by script errors. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the method steps of a rule-based compliance detection script automated generation method and system provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0026] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method, step, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the method, step, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of additional identical elements in the method, step, or apparatus that includes the element.

[0027] To address the issues of poor flexibility and inconsistent quality in existing compliance testing scripts, this invention provides a rule-based automated method and system for generating compliance testing scripts. By deeply analyzing the grammatical patterns, structural features, and best practices of a large number of existing high-quality rule-based compliance testing scripts, and fully leveraging the significant advantages of large language models in language understanding, pattern recognition, and code generation, the system automatically generates high-quality rule-based compliance testing scripts that conform to industry best practices based on user-input system configuration requirements and security baseline requirements. This not only reduces reliance on technical personnel but also significantly improves the efficiency, accuracy, and intelligence of security compliance testing.

[0028] Specifically, such as Figure 1 As shown, the method includes: S100: Receives user input information, parses the user input information, and generates a structured requirement specification.

[0029] It should be noted that user input information is used to describe security baseline requirements and system configuration requirements, including natural language descriptions and / or structured configuration files.

[0030] Natural language description refers to users expressing security testing requirements through text input, such as: "Please generate a script to check the SSH configuration of the RHEL8 system and ensure that root login is disabled." Structured configuration files refer to existing system configuration files uploaded by users, such as sshd_config and httpd.conf. These files store configuration parameters in a specific format (such as INI, YAML, JSON, or system-specific formats).

[0031] After receiving user input, the system can first identify the information type. Specifically, if the input includes text content, it is identified as a natural language description; if the input includes file upload and the file extension matches the preset configuration file type, such as .conf, .yml, .json, etc., it is identified as a structured configuration file; if both types of information are included, they will be merged later.

[0032] Preferably, user input includes both natural language descriptions and structured configuration files. Multimodal input allows for the simultaneous understanding and processing of various input formats, including natural language descriptions, structured configuration files, and system architecture diagrams. This not only enables a more comprehensive understanding of user intent and system status, improving the accuracy and flexibility of requirement understanding, but also allows for the selection of the most convenient input format based on actual conditions, lowering the barrier to entry and significantly enhancing the convenience of user interaction and the applicability of the system.

[0033] As one example, user input information includes natural language descriptions and / or structured configuration files; parsing the user input information specifically includes: By using a natural language processor to identify key entities and user intents in natural language descriptions, user natural language requirements can be obtained. And / or, the configuration information in the structured configuration file is extracted by the configuration parser to obtain the system configuration status; Generate a structured requirements specification based on the user's natural language requirements and / or system configuration status.

[0034] When user input includes natural language descriptions, a Natural Language Processor (NLP) performs deep parsing of these descriptions. This processor employs a pre-trained model based on the Transformer architecture, combined with Named Entity Recognition (NER) technology, to identify key entities and user intent from the natural language description. By deeply parsing the natural language description, key entities and user intent can be accurately identified, avoiding potential misunderstandings that may arise from traditional keyword matching and enhancing the accuracy of understanding user needs.

[0035] Optionally, key entities may be identified, including one or more of the following: operating system identification, service component identification, configuration information identification, expected value identification, compliance standard identification, and specific requirement identification.

[0036] Specifically, it identifies the operating system, such as RHEL8, Ubuntu 20.04, CentOS 7, and other operating system names and version information.

[0037] Service component identification, recognizing service names such as SSH, Apache, Nginx, and MySQL.

[0038] Configuration information recognition, including configuration item names such as Protocol, Permit Root Login, and Max Auth Tries.

[0039] Expected value identification: Identify configuration expected values ​​such as 2, no, 3, etc.

[0040] Identify compliance standards, such as CIS baseline, NIST SP 800-53, Cybersecurity Classified Protection 2.0, ISO 27001, and other security compliance frameworks or benchmarks that users are required to follow.

[0041] Specific requirement identification involves recognizing the user's detailed requirements for specific configurations or security controls. This requires combining configuration information identification with expected value identification to understand the user's operational purpose, such as prohibition, enabling, restriction, or requirement. It typically includes technical details or operational instructions.

[0042] User intent recognition determines whether a user wants to perform an operation such as generating an audit script, verifying a configuration, or repairing a configuration.

[0043] By identifying key entities and user intents in natural language descriptions, unstructured natural language descriptions are transformed into structured user natural language requirements. User natural language requirements can be represented in JSON format and may include fields such as target operating system, service components, configuration information, expected values, compliance standards, specific requirements, and operation types.

[0044] When user input includes a structured configuration file, the configuration parser parses the structured configuration file. The configuration parser supports multiple configuration file formats: INI format, which parses key-value pairs (e.g., Protocol 2); YAML format, which parses key-value pairs and nested structures in YAML format; JSON format, which parses configuration parameters in JSON objects; and system-specific formats, which are specifically parsed for configuration files used by specific systems (e.g., SSH, Apache).

[0045] The configuration parser extracts all valid configuration information from the configuration files, forming a set of key-value pairs. Simultaneously, it infers the operating system, service components, and configuration information based on the filename or file content; for example, the `sshd_config` file corresponds to the SSH service, and the `httpd.conf` file corresponds to the Apache service. The extracted configuration information constitutes the system configuration status, thus reflecting the current actual configuration of the system.

[0046] Generate a unified structured requirement specification based on user natural language requirements and / or system configuration status.

[0047] When only natural language is required, the user's natural language requirements are directly converted into control points, and necessary default checks are supplemented based on the preset knowledge graph.

[0048] For example, when a user inputs a natural language description: "[Please help me generate an InSpec script to check if the RHEL8 system complies with CIS baseline requirements, especially to ensure that SSH prohibits the use of insecure protocol version 1.]", NLP parses it, identifies and extracts key entities and user intent through NER, and obtains the user's natural language requirement: "[Target operating system: RHEL8; Service component: SSH; Compliance standard: CIS baseline; Specific requirement: Prohibit protocol version 1; User intent: Generate script (Action: Generate)]". Next, a knowledge graph is used to infer the user's natural language requirement. The inference reveals that RHEL8 belongs to the Red Hat family, the CIS baseline corresponds to a specific set of control points, and prohibiting protocol version 1 is equivalent to requiring protocol version 2. Finally, all the parsed and inferred information is organized into a structured JSON or similar object and used as a structured requirements specification.

[0049] When only the system configuration status is available, the current configuration information is compared with the preset audit compliance benchmark to verify whether the current configuration meets the audit compliance benchmark. The user intent is then identified based on the verification results, and a structured requirement specification is generated.

[0050] For example, a user uploads an sshd_config file. The configuration parser reads the configuration file and extracts all valid configuration information, such as a line containing Protocol 2. Then, it compares all the extracted configuration information with the audit compliance benchmark. If they match, it means the current configuration information conforms to the audit compliance benchmark, and the user's intention is to generate an audit script, thus generating a requirements specification for auditing. If they don't match, such as Protocol 1 in the file, further analysis is performed to determine whether the uploaded file is for detecting and reporting this non-compliance or is a template that needs to be corrected. This further infers whether the user's intention is to generate a verification configuration or a repair configuration, thus generating a requirements specification for verification or repair.

[0051] When both user natural language requirements and system configuration status are included, these requirements can be processed in parallel, and the resulting requirement specifications can then be merged. Preferably, user natural language requirements are prioritized, with system configuration status as a supplement. When there is a conflict between the two, user natural language requirements take precedence, and system configuration status is discarded. For example, if a user requests that root login be disabled, but the configuration file shows "Permit Root Login yes," then a control point requiring "Permit Root Login no" will be generated. Including both user natural language requirements and system configuration status allows for the comprehensive utilization of both information sources, resulting in more accurate and comprehensive requirement specifications.

[0052] As a preferred embodiment, a structured requirements specification is generated, specifically including: The identified key entities are mapped to standardized technical terms; Based on technical terms, system configuration status, and a pre-set knowledge graph, rule reasoning is performed to transform user natural language requirements into one or more control points; Generate machine-readable structured requirements specifications based on control points.

[0053] By using rule-based reasoning, fuzzy natural language requirements are transformed into precise control points, achieving intelligent conversion from unstructured requirements to structured specifications and ensuring the accuracy of subsequent script generation.

[0054] It should be noted that the mapping process maps identified key entities to standardized identifiers. This can be implemented using a pre-defined entity mapping table and stored in key-value pairs. Specifically, the mapping includes one or more of the following: operating system mapping, service component mapping, configuration information mapping, expected value mapping, compliance standard mapping, and specific requirement mapping.

[0055] Optionally, the mapping process can combine exact matching and fuzzy matching. First, exact matching is performed. If no perfect match is found, the most similar standardized term is found through semantic similarity calculation (such as cosine similarity). After mapping, all entities use a unified, machine-readable standardized identifier. Standardized mapping eliminates ambiguity and diversity in natural language, ensuring the system accurately understands user intent and avoiding misunderstandings caused by inconsistent terminology.

[0056] Knowledge graphs can be built using graph databases such as Neo4j and Janus Graph, and include node types, relationship types, and attribute information. Node types include operating system nodes, service component nodes, configuration information nodes, compliance standard nodes, control point nodes, and best practice nodes. Relationship types include "applies to" (meaning a control point applies to a specific operating system), "contains" (meaning a control point contains a certain configuration information), "requires" (meaning a control point requires a parameter of a certain configuration information to have a specific value), and "best practice" (meaning a configuration value is the best practice), etc. Attribute information for each node includes rich attribute information, such as the control point's description, severity level, remediation suggestions, and reference links. Based on a pre-defined security compliance knowledge graph, it can deeply understand the security logic of user needs, transforming vague natural language descriptions into precise control points, ensuring accurate understanding of user intent.

[0057] For example, the entity "RHEL 8" extracted by NER is queried into the knowledge graph and mapped to standard terms (e.g., $SystemProfile: \{family: 'red hat', release: '8'\}$). Next, rule reasoning is performed, combining $Service$ "SSH" and $Rule$ "prohibit...protocol version 1", and referencing $Compliance Framework: 'cis'$, to deduce that "prohibit protocol version 1" is equivalent to the specific control point "requires protocol version 2". Finally, a detailed $Requirement Specification$ JSON object containing all control points and target configurations is generated and used as a structured requirements specification.

[0058] In addition, in order to generate a specification that meets the benchmark, after generating the structured requirements specification, the following may also be included: performing a completeness check on the requirements specification, and initiating a clarification inquiry to the user if key parameters are missing.

[0059] The generated structured requirements specification is in a machine-readable format, such as JSON Schema. Preferably, the structured requirements specification may include one or more of the following: target operating system information (operating system type, version), control point list information (each control point includes service components, configuration information, expected values, and specific requirements), compliance standards (such as CIS, NIST, etc.), operation type information (audit, remediation, verification), and metadata information (generation time, source). This method of generating a structured, machine-readable format from unstructured user input demonstrates a deep understanding of user intent and lays a solid foundation for subsequently generating high-quality, accurately matched compliance detection scripts.

[0060] S200: Based on the structured requirements specification, retrieve relevant code templates and best practice information from a pre-defined professional knowledge base.

[0061] It should be noted that rule-based compliance detection scripts can be Chef InSpec scripts, Open SCAP scripts, or custom compliance detection scripts; code templates and best practice information include the syntax rules of the corresponding script framework, compliance control point templates, and industry best practices.

[0062] Based on the structured requirements specification, the most relevant InSpec codes and rules are retrieved from a pre-defined professional knowledge base. It should be noted that this invention retrieves relevant information from a professional, structured knowledge base before generating the large language model, ensuring that the generated content is based on the latest and most accurate professional knowledge (such as CIS baseline requirements and InSpec best practices), thus solving the illusion problem of large language models and the problem of knowledge lag.

[0063] Optionally, the default professional knowledge base adopts a dual-storage architecture, including two core components: a vector database and a knowledge graph, which store different types of professional knowledge respectively.

[0064] As one example, the pre-defined professional knowledge base includes a vector database storing code snippets and a knowledge graph storing best practice metadata.

[0065] Vector databases are used to store code snippets and can employ vector-based similarity retrieval techniques such as FAISS, Milvus, or ChromaDB. For example, the Chef InSpec script code is first parsed into an abstract syntax tree to extract structural features; then, a pre-trained language model (such as CodeBERT or InCoder) is used to encode the code snippets into high-dimensional vector representations; next, metadata about the code snippets is extracted, including applicable operating systems, service components, control point types, compliance standards, etc.; finally, the encoded vectors and corresponding code snippets are stored in a vector database, and a vector index is built to support fast similarity retrieval.

[0066] Knowledge graphs are used to store best practice metadata and can be built using graph databases such as Neo4j and JanusGraph. This has already been explained above and will not be repeated here.

[0067] By combining vector databases and knowledge graphs, we can quickly retrieve similar code snippets and acquire best practice knowledge, providing comprehensive knowledge support for script generation.

[0068] As one example, relevant code templates and best practice information are retrieved from a pre-defined professional knowledge base, specifically including: Obtain control points from the structured requirements specification; Based on control points, semantic similarity matching is performed in the vector database to retrieve relevant code snippets and obtain code templates; Based on control points, query the knowledge graph for related best practice metadata to obtain best practice information.

[0069] By leveraging semantic similarity matching and knowledge graph queries, the most relevant code templates and best practice information can be accurately located from massive knowledge bases, improving the accuracy and efficiency of knowledge retrieval.

[0070] For example, the knowledge base retrieval module utilizes structured requirements specifications. The retrieval process specifically includes: First, using the semantic vector of a control point such as "ssh-protocol-version", a similarity search is performed in the vector database to find the most matching, validated InSpec code snippet ($ControlSnippet$), such as `describesshd_config do its('Protocol') { should cmp '2'} end`. Then, the knowledge graph is queried to obtain best practice metadata ($BestPractice$) related to `cis-rhel8-benchmark`, such as `{ 'include_impact': true, 'include_title': true, 'default_impact': 0.7}`, providing context for code generation.

[0071] Based on control points, semantic similarity matching is performed in a vector database to retrieve relevant code snippets. The specific process may include: The semantic information of the control points (including control point descriptions, configuration information, expected values, etc.) is encoded into query vectors using the same language model as the code snippet; Perform a k-nearest neighbor (k-NN) search on the query vector in the vector database, calculate the cosine similarity or Euclidean distance between the query vector and all code fragment vectors, and obtain the similarity score; The search results are sorted based on similarity scores, and the top-k most relevant code snippets are selected. The search results are filtered a second time based on the metadata of the control points (such as operating system, service components, and compliance standards) to ensure that the retrieved code snippets match the current scenario. Combine the retrieved code snippets or select the most suitable snippet as a code template.

[0072] Vector databases perform retrieval based on semantic similarity, which can accurately understand the semantic intent of control points and retrieve the most semantically relevant code fragments, avoiding the limitations of traditional keyword matching and improving retrieval accuracy.

[0073] Based on control points, the system queries related best practice metadata in the knowledge graph to obtain best practice information, which may include: Locate the corresponding control point node in the knowledge graph; if the control point does not exist, find the most similar control point node based on the semantic information of the control point. Starting from the control point node, traverse along the node path of the relation type and query all related nodes; Extract the attribute information of all nodes on the query path to obtain metadata information; the metadata information may include a detailed description and severity level of the control point, best practice values ​​for configuration parameters, remediation suggestions and operation steps, reference links to compliance standards, and descriptions of related security threats and risks, etc. All metadata information is aggregated to obtain structured best practice information.

[0074] Best practice information can be represented in JSON format, including fields such as control point ID, best practice value, documentation, and reference links.

[0075] The graph structure of knowledge graphs can express rich semantic relationships, support complex multi-hop queries, and retrieve comprehensive best practice information related to control points, including configuration requirements, remediation suggestions, and compliance standards. This enhances knowledge coverage and the professional generation capabilities of large language models. By constructing a professional knowledge graph, it is ensured that the generated scripts not only comply with technical specifications but also with industry best practices and security standards. This knowledge enhancement mechanism greatly improves the professionalism and accuracy of script generation.

[0076] It's important to note that the retrieved code templates originate from a verified vector database, ensuring the quality and security of the generated scripts and avoiding potential errors introduced by manual coding. Furthermore, the retrieved best practice information seamlessly integrates rich industry experience and security baseline requirements into the generation process, ensuring the professionalism and authority of the output scripts. In addition, retrieving and querying control points in the structured requirements specification based on a deep understanding of the system environment and business scenarios enables the generation of highly targeted and personalized test scripts, ensuring a perfect match with the actual application environment. Moreover, the vector database and knowledge graph offer excellent scalability, allowing for easy addition of new code snippets and best practice knowledge without modifying the core retrieval logic, increasing the flexibility of the generated compliance testing scripts.

[0077] S300: Generates prompts based on structured requirements specifications, code templates, best practice information, and user input, inputs the prompts into the large language model, and outputs the initial compliance detection script.

[0078] The prompts can be in a structured template format, which may include system role definitions, task descriptions, context information, best practice guidelines, constraints, user input information, etc.

[0079] The system role definition clarifies that the role of the large language model is "rule-based compliance detection script generation expert", requiring it to generate scripts in accordance with the syntax specifications and best practices of compliance detection scripts such as Chef InSpec and Open SCAP.

[0080] The task description is used to describe the generated task in detail, including target operating system information, compliance standards, and a list of control points that need to be checked.

[0081] Contextual information is used to refer to the retrieved code templates as examples, helping the model understand the syntax and coding style of compliance detection scripts such as ChefInSpec and Open SCAP. The code templates are filtered and deduplicated to ensure the quality and diversity of the examples.

[0082] Best practice guidance is used to integrate best practice metadata retrieved from the knowledge graph, including correct values ​​for configuration parameters, common error examples, performance optimization suggestions, etc., to guide the model to generate code that conforms to industry standards.

[0083] Constraints are used to specify generation requirements, such as the need to use Chef InSpec DSL syntax, the need to include necessary comments, and the need to conform to specified coding standards.

[0084] User input information is used to preserve the user's natural language description, helping large language models to better understand the user's true intentions and context.

[0085] The prompts can be organized in JSON format to ensure clear organization and highlight key points, making it easier for large language models to understand and execute.

[0086] Optionally, Retrieval-Enhanced Generation (RAG) technology is employed to integrate structured requirements specifications, code templates, best practice information, and user input into a rich prompt, which is then fed into the generation engine of the large language model. Upon receiving this complete prompt, the large language model generates an initial compliance check script. This script not only includes the retrieved code template but also completes contextual information such as `control`, `impact`, and `title` according to best practice information. This dynamic generation method ensures that the generated script is not only syntactically correct but also highly practical and effective in real-world applications, exhibiting exceptional flexibility to adapt to various complex and personalized needs.

[0087] As an example, the large language model is a domain-wide model obtained by fine-tuning the domain using a compliance detection script.

[0088] It should be noted that a large language model that has been fine-tuned and trained with a large number of high-quality compliance detection scripts is preferred. This allows the model to deeply understand the syntax, common writing styles, control structures, and best practices of compliance detection scripts such as Chef InSpec and Open SCAP. As a result, the generated scripts are far superior to general models in terms of grammatical standardization, professionalism, and accuracy.

[0089] Furthermore, a Transformer neural network architecture specifically optimized for languages ​​relevant to compliance inspection scripting domains such as Chef InSpec and Open SCAP can be employed. The input encoder integrates a multimodal input fusion layer, enabling unified processing of different types of input information, such as text, configuration files, and graphics. Through positional encoding and multi-head attention mechanisms, it can accurately capture complex semantic relationships and contextual dependencies in the input information. The contextual semantic understanding layer further enhances the ability to deeply understand business scenarios and the technical environment.

[0090] Alternatively, an autoregressive generative model similar to GPT can be used as the large language model to generate content with high fluency and strong adaptability to new scenarios.

[0091] Optionally, the fine-tuning process for a large language model can be as follows: First, we collected a large amount of high-quality compliance testing script code, covering script examples from different operating systems, service components, and compliance standards; then we cleaned the data to remove duplicate, erroneous, and incomplete code snippets. Then, the compliance detection script is parsed into an abstract syntax tree (AST), code structure features and semantic information are extracted, and instruction-response pairs are constructed; where the instruction is the control point requirement described in natural language, and the response is the corresponding compliance detection script code; Finally, general-purpose large language models based on the Transformer architecture (such as Codex, CodeT5, and InCoder) were selected as the base model. Supervised fine-tuning of the base model was performed using instruction-response pairs corresponding to compliance detection scripts such as Chef InSpec and Open SCAP. During the fine-tuning process, efficient parameter fine-tuning techniques such as Low-Rank Adaptation (LoRA) were employed to enhance the model's specific capabilities in Chef InSpec, Open SCAP, and other related fields while maintaining its general-purpose capabilities.

[0092] Furthermore, the decoder of a large language model can also include a syntax constraint layer, a control flow generation module, and a best practice fusion layer. The syntax constraint layer ensures that the generated code strictly adheres to the syntax specifications and coding standards of the compliance testing script, achieving a qualitative leap from descriptive test plans to executable security testing scripts; the control flow generation module is used to construct reasonable program control structures and logical flows; and the best practice fusion layer automatically integrates mature industry experience and security standards into the generated code.

[0093] The fine-tuned domain-specific model demonstrates a significantly superior ability to generate compliance testing scripts compared to the general model. It can accurately understand security testing requirements and generate syntactically correct, logically sound script code that conforms to best practices.

[0094] S400: Perform multi-layer quality verification on the initial compliance detection script; if the verification passes, output it as the final compliance detection script; if the verification fails, generate correction information and return it to the large language model for iterative correction until the verification passes.

[0095] This invention introduces an automated verification closed loop, where the generated script undergoes a multi-level rigorous verification process, thereby ensuring the quality of the compliance detection script.

[0096] As an example, multi-layer quality verification includes at least: Syntax validation: Verify the syntactic correctness of the initial compliance check script by constructing an abstract syntax tree of the initial compliance check script; Standard verification: Use static code analysis tools to check whether the initial compliance check script conforms to the ChefInSpec coding standards; Logical verification: The abstract syntax tree of the initial compliance detection script is compared with the structured requirements specification to ensure semantic consistency between the two.

[0097] Through three layers of verification—syntax, specification, and logic—we ensure that the generated script is syntactically correct, conforms to coding standards, and is logically sound.

[0098] It should be noted that if the verification fails, correction information is generated and the large language model is used to make corrections. This process is iterated continuously to form an iterative optimization loop until the verification is successful, generating a compliance detection script that meets the requirements.

[0099] Multi-layered quality verification can proactively identify and locate various static defects after script generation and before delivery, without actual execution. More importantly, this invention further transforms verification failures into structured correction guidance information and automatically feeds it back to the large language model for online iterative correction. This ensures that the final output script has syntactic correctness, specification compliance, and logical accuracy, achieving deterministic and reliable high-quality output and meeting the stringent requirements of security and compliance scenarios for scripts.

[0100] As one example, generating correction information specifically includes: The system identifies the error type and location for verification, generates a text description based on the error type to guide the large language model in making corrections, and uses the text description as correction information.

[0101] Optional, locate the error type and location for verification, specifically including: Syntax errors are identified through an abstract syntax tree, including missing end keywords, mismatched parentheses, indentation errors, Ruby syntax violations, etc., and the specific line number, column number, and error type code of the error are recorded.

[0102] Identify specification errors using static code analysis tools (such as the RuboCop with InSpec plugin); these include missing title or impact fields, non-compliant descriptions, and non-standard variable naming; and identify the rule number and severity level of the violation.

[0103] The logic checker identifies logical errors, including checking for inconsistencies between logic and requirements specifications (e.g., the requirement specifies Protocol 2, but the script checks Protocol 1), incorrect resource paths, and errors in conditional judgment logic, and records the differences between expected and actual values.

[0104] Based on the error type, generate text descriptions to guide the large language model in making corrections, and use the text descriptions as correction information so that the large language model can accurately understand the correction requirements and improve correction efficiency.

[0105] Optionally, generate text descriptions based on error types to guide the large language model in making corrections, specifically including: Syntax error correction, providing standard syntax templates and correction examples; Corrections to specification errors have been made, referencing the InSpec coding specification document and providing code examples that conform to the specification. Correct logical errors by comparing the requirements specification and pointing out any inconsistencies.

[0106] When multiple error types exist, a priority handling strategy is adopted: syntax errors take precedence, logical errors take second, and specification errors take last.

[0107] Preferably, based on the corrected information and the structured requirements specification, new prompts are obtained and input into the large language model, which then re-outputs the compliance detection script. It should be noted that after each iteration and correction, multi-layered quality verification is re-executed.

[0108] This invention can automatically locate errors and generate correction guidelines, forming an intelligent closed loop of generation-verification-correction, solving the problem of unstable output of large language models, and improving the automated correction capability of compliance detection scripts.

[0109] For example, the generated initial compliance check script undergoes multiple layers of quality verification. First, syntax verification is performed using an abstract syntax tree to ensure there are no Ruby or InSpec syntax errors, such as missing `end`. Second, a static code interpolation is run to check if the script conforms to InSpec coding standards, such as the existence of the `title` and `impact` fields. Finally, a logic checker performs semantic verification between the initial compliance check script's abstract syntax tree and the abstract syntax tree, ensuring that the script's checking logic, such as checking if `sshd_config`'s `Protocol` is `'2'`, is completely consistent with user requirements. If any verification fails, the system generates correction information and returns to the large language model for iterative correction, forming a closed loop of generation and verification. Once the initial compliance check script passes multiple layers of quality verification, it is marked as the final compliance check script ($FinalScript$) and output as a standard `.rb` file.

[0110] This invention replaces sandbox execution verification by adding multi-level, automated, multi-layer quality verification to the code generation process. It not only checks the correctness of syntax, code style, and business logic, but also triggers automatic iterative correction without human intervention if the verification fails. This forms an intelligent closed loop of generation-verification-feedback-regeneration, ensuring the high reliability and determinism of the output script.

[0111] Optionally, multi-layered quality verification can also involve building a parser and rule engine to check the script's syntax, syntax, and logic, thereby effectively ensuring script quality. Multi-layered quality verification can also be achieved through a sandbox environment, where the script is actually executed and its quality is verified by analyzing the results.

[0112] As another example, a metadata report can be generated simultaneously with the output of the compliance testing script. This metadata report includes a mapping of security control points covered by the compliance testing script, compliance standard comparisons, and a summary of verification results. The automatic generation of the metadata report ($MetadataReport$) provides complete information for the use, management, and auditing of the script, reducing operational costs and improving traceability and ease of use.

[0113] On the other hand, the present invention also provides a rule-based compliance detection script automated generation system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0114] This invention employs a hierarchical system architecture to achieve an end-to-end intelligent processing flow from user requirements to final script output. This system architecture includes a user input module, a requirements analysis engine, a knowledge base retrieval module, a prompt message generation module, a large language model, a quality verification module, and a script output module.

[0115] The user input module is used to receive user input information, including natural language descriptions and / or structured configuration files; The requirements analysis engine is used to parse user input information and generate structured requirements specifications; it uses a natural language processor and configuration parser to perform deep semantic analysis on user input information to accurately extract key parameters such as system type and security requirements.

[0116] The knowledge base retrieval module is used to retrieve relevant code templates and best practice information from a pre-defined professional knowledge base, including vector databases and knowledge graphs, based on structured requirements specifications.

[0117] The prompt message generation module is used to generate prompt messages based on structured requirements specifications, code templates, best practice information, and user input.

[0118] A large language model is used to input prompts and output the initial compliance check script.

[0119] The quality verification module is used to perform multi-layer quality verification on the initial compliance testing script to ensure the accuracy and standardization of the final compliance testing script.

[0120] The script output module delivers the validated and optimized final script to the user in a standard format, along with detailed test reports and deployment recommendations.

[0121] In addition, the system may include a professional knowledge base, a system configuration pattern library, a best practice rule library, and a database of common errors and remediation suggestions. The professional knowledge base is used to compile core control requirements from various security frameworks and standards; the system configuration pattern library is used to store typical configuration patterns and parameter settings in different system environments; the best practice rule library is used to integrate the rich experience and successful cases accumulated by industry experts; and the database of common errors and remediation suggestions is used to help the system avoid generating defective code.

[0122] The consistency of the system's technical features and methods will not be elaborated upon here.

[0123] In summary, this invention employs requirements analysis, knowledge retrieval, and a large language model. The large language model is directly responsible for generating the core code and proactively injects professional knowledge through Retrieval Enhancement (RAG) before generation, rather than passively modifying it after generation. This demonstrates extremely high flexibility, enabling the dynamic generation of personalized scripts based on specific needs, rather than being limited by static templates. Simultaneously, a multi-layered quality verification system is constructed to ensure the high quality and reliability of the generated scripts. Compared to existing methods, this invention can comprehensively detect syntax errors, coding style violations, and logical defects without executing the scripts. Furthermore, when verification fails, the system automatically generates a Feedback Report containing error location and correction suggestions, which is fed back to the LLM generation engine to trigger automatic correction, avoiding manual correction and greatly reducing the cost of manually maintaining rules. Through deep learning, it automatically adapts to new requirements and scenarios, significantly improving the generation quality and reliability of security test scripts.

[0124] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0125] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed methods or systems can be implemented in other ways. For example, the embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0127] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0128] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A rule-based method for automatically generating compliance detection scripts, characterized in that, The method includes: Receive user input information, parse the user input information, and generate a structured requirements specification. Based on the structured requirements specification, relevant code templates and best practice information are retrieved from a pre-defined professional knowledge base; Based on the structured requirements specification, the code template, the best practice information, and the user input information, a prompt message is generated and input into the large language model to output the initial compliance detection script; The initial compliance detection script undergoes multi-layer quality verification; if the verification passes, it is output as the final compliance detection script; if the verification fails, correction information is generated and returned to the large language model for iterative correction until the verification passes.

2. The method for automatically generating rule-based compliance detection scripts according to claim 1, characterized in that, The user input information includes natural language descriptions and / or structured configuration files; Parsing the user input information specifically includes: The user's natural language requirements are obtained by identifying key entities and user intent in the natural language description through a natural language processor. And / or, the configuration information in the structured configuration file is extracted by the configuration parser to obtain the system configuration status; Based on the user's natural language requirements and / or the system configuration status, a structured requirements specification is generated.

3. The method for automatically generating rule-based compliance detection scripts according to claim 2, characterized in that, The generation of the structured requirements specification includes: The identified key entities are mapped to standardized technical terms; Based on the aforementioned technical terms, the system configuration status, and the preset knowledge graph, rule reasoning is performed to transform the user's natural language requirements into one or more control points; Based on the control points, a machine-readable structured requirements specification is generated.

4. The method for automatically generating rule-based compliance detection scripts according to claim 3, characterized in that, The pre-defined professional knowledge base includes a vector database storing code snippets and a knowledge graph storing best practice metadata.

5. The method for automatically generating rule-based compliance detection scripts according to claim 4, characterized in that, Retrieve relevant code templates and best practice information from a pre-defined professional knowledge base, specifically including: Obtain the control points from the structured requirements specification; Based on the control points, semantic similarity matching is performed in the vector database to retrieve relevant code snippets and obtain code templates; Based on the control points, the relevant best practice metadata is queried in the knowledge graph to obtain best practice information.

6. The method for automatically generating rule-based compliance detection scripts according to claim 1, characterized in that, The large language model is a domain-specific large model obtained by fine-tuning the domain using a compliance detection script.

7. The method for automatically generating rule-based compliance detection scripts according to claim 1, characterized in that, The multi-layer quality verification includes at least: Syntax validation: The syntax correctness of the initial compliance detection script is verified by constructing an abstract syntax tree of the initial compliance detection script; Standard verification: The initial compliance detection script is checked using static code analysis tools to see if it conforms to the coding standards for compliance detection scripts; Logical verification: The abstract syntax tree of the initial compliance detection script is compared with the structured requirements specification to ensure semantic consistency between the two.

8. The method for automatically generating rule-based compliance detection scripts according to claim 7, characterized in that, Generating the correction information specifically includes: The error type and location are identified during verification, and a text description is generated based on the error type to guide the large language model in making corrections. This text description is then used as correction information.

9. The method for automatically generating rule-based compliance detection scripts according to claim 1, characterized in that, While outputting the compliance detection script, a metadata report is generated; wherein the metadata report includes the source of the compliance detection script, the applicable environment, the compliance standards, and deployment recommendations.

10. A rule-based compliance detection script automated generation system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images