Software off-line automatic testing method and device based on local large language model and RAG enhancement

Abstract

The invention discloses a software off-line automatic testing method and device based on a local large language model and RAG enhancement. The method comprises the steps that test information is obtained, and the test information is used for indicating a software code and / or demand description of target software; generating a matched test case set by utilizing a local large language model (LLM) and combining an RAG technology; the test cases in the test case set are sequentially executed in an isolated test environment, an execution log is generated, the isolated test environment is a test environment constructed based on a container and a hardware simulation technology, and the execution log is obtained after real-time monitoring in the execution process; on the basis of execution log generation, a large language model (LLM) is finely adjusted locally by using a low-rank adaptation LoRA technology, and a local knowledge base and system test strategy configuration are optimized, so that a self-learning closed-loop test system is formed. The technical problem that potential safety hazards exist when software testing is carried out through a cloud scheme in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of software testing technology, and more specifically, to a method and apparatus for offline automated software testing based on a local large language model and RAG enhancement. Background Technology

[0002] This section is intended to provide background or context for the content set forth in the claims or specification, and the content described herein is not acknowledged as prior art simply because it is included in this section.

[0003] With the deepening innovation in information technology applications, security software based on specific target operating systems (such as various domestically developed operating systems) has been widely used in key sectors such as finance, government affairs, and energy. This type of software typically involves highly sensitive business logic, access control, and data encryption, making its stability and security paramount. Therefore, thorough and efficient testing before software release is a crucial step in ensuring its quality.

[0004] Currently, automated testing is the main means to improve testing efficiency. However, existing technologies have the following obvious shortcomings when dealing with software testing that has high security requirements and is adapted to specific environments: many advanced intelligent testing tools rely on cloud computing power and knowledge bases, uploading sensitive code containing business logic and potential vulnerabilities to the cloud for analysis, which poses a risk of data leakage and cannot meet the testing needs of highly confidential scenarios such as military and financial industries or strict offline environments.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a method and apparatus for offline automated software testing based on a local large language model and RAG enhancement, so as to at least solve the technical problem of security risks in software testing through cloud solutions in related technologies.

[0007] According to one aspect of the embodiments of this application, a method for offline automated software testing based on a local large language model and RAG enhancement is provided, comprising: acquiring test information, wherein the test information is used to indicate the software code and / or requirement description of target software, the target software being security software that needs to run on a target operating system; using a locally deployed large language model (LLM) and combined with retrieval enhancement generation RAG technology based on a local knowledge base to generate a set of test cases that match the test information and the security environment of the target operating system, wherein the local knowledge base stores a security vulnerability rule base and a test template base of the target operating system; sequentially executing the test cases in the set of test cases in an isolated test environment and generating an execution log of the set of test cases, wherein the isolated test environment is a test environment built based on container and hardware simulation technology that matches the target operating system and its hardware architecture, and the execution log is obtained after real-time monitoring of the resource status of the isolated test environment and the integrity of the test object during execution; and, based on the execution log generation, fine-tuning the large language model (LLM) locally using low-rank adaptation (LoRA) technology, optimizing the local knowledge base and system test strategy configuration, thereby forming a self-learning closed-loop testing system.

[0008] Optionally, a set of test cases matching the test information and the security environment of the target operating system is generated using a locally deployed Large Language Model (LLM) and combined with Retrieval Enhancement Generation (RAG) technology based on a local knowledge base. This includes: performing format validation on the input test information; if the format validation passes, guiding the local LLM to parse the validated test information and extract test requirements using a preset first prompt word template; based on the test requirements, retrieving relevant security vulnerability rules and test templates from the local knowledge base using RAG technology; injecting the retrieved security vulnerability rules and test templates into a preset second prompt word template to guide the local LLM to generate structured test cases, wherein the generated test cases include at least a test description, operation steps, expected results, and confidence level assessment; adding test cases with a confidence level higher than a set threshold to the test case set; if the number of test cases in the test case set does not meet the requirements, optimizing the second prompt word template and guiding the local LLM to regenerate structured test cases.

[0009] Optionally, after adding test cases with confidence levels higher than a set threshold to the test case set, the method further includes: storing the test cases with confidence levels higher than the set threshold, the corresponding test requirements, and the security vulnerability rules generated this time as new knowledge entries in the local knowledge base; and dynamically adjusting the weight or relevance of the new knowledge entries in the local knowledge base based on feedback from subsequent test audit reports regarding the effectiveness of test cases in the new knowledge entries.

[0010] Optionally, test cases in the test case set are executed sequentially in an isolated test environment, and an execution log of the test case set is generated. This includes: creating an offline test environment image corresponding to the system version of the target operating system using container technology, and simulating the hardware architecture environment of the target operating system within the container using a hardware simulator; injecting the test cases in the test case set into the isolated test environment for execution to simulate security vulnerability attack scenarios; during the execution of the test cases, collecting resource usage indicators of the test environment and target software at fixed time intervals through a system resource monitoring library and comparing them with dynamically adjusted resource thresholds; calculating the digest value of the tested file using a hash algorithm before and after the execution of the test cases for integrity verification; determining in real time whether the resource usage indicators exceed the resource threshold or whether the file integrity has been compromised: if so, interrupting the execution of the current test case and recording the abnormal event to the execution log; if not, successfully executing and recording the execution log after normal completion; and after all test cases have been executed, summarizing and generating a structured execution log containing resource monitoring records, integrity verification results, and details of abnormal events.

[0011] Optionally, after interrupting the execution of the current test case, the method further includes: recording the exception type that caused the interruption, the instantaneous value of the resource consumption indicator, and the integrity verification result to the execution log; re-executing the interrupted test case after adjusting the test environment parameters or clearing the test state according to a preset retry strategy, wherein the preset retry strategy is used to indicate the number of retries or the retry duration; if the retry still fails according to the preset retry strategy, the final failure status is recorded, and the test case is skipped and subsequent test cases are executed.

[0012] Optionally, based on the execution logs, the large language model (LLM) is fine-tuned locally using low-rank adaptation (LoRA) technology to optimize the local knowledge base and system test strategy configuration. This includes: using the execution logs as training data, fine-tuning the local large language model (LLM) using low-rank adaptation (LoRA) technology to optimize its test case generation capability; generating a test audit report containing test coverage and vulnerability analysis results based on the parsing of the execution logs; generating test strategy optimization suggestions based on the analysis conclusions in the test audit report; updating the system test strategy configuration based on the test strategy optimization suggestions; and storing the test audit report, model fine-tuning information, and updated system test strategy configuration related to this test in the local knowledge base for optimizing subsequent testing processes.

[0013] Optionally, the test strategy optimization suggestions include two types of suggestions. One type of suggestion is used to update the execution parameters of the system test strategy configuration, and the other type of suggestion is used to fine-tune the prompt word optimization instructions or training data labels of the local large language model LLM.

[0014] According to another aspect of the embodiments of this application, an offline automated software testing device based on a local large language model and RAG enhancement is also provided, comprising: an acquisition unit, configured to acquire test information, wherein the test information is used to indicate the software code and / or requirement description of target software, the target software being security software that needs to run on a target operating system; and a testing unit, configured to utilize a locally deployed large language model (LLM) and combine it with retrieval enhancement generation RAG technology based on a local knowledge base to generate a set of test cases that match the test information and the security environment of the target operating system, wherein the local knowledge base stores a security vulnerability rule base and test templates of the target operating system. The system includes a library and a log unit, used to sequentially execute test cases in the test case set within an isolated test environment and generate an execution log for the test case set. The isolated test environment is a test environment built based on container and hardware simulation technology and matched to the target operating system and its hardware architecture. The execution log is obtained after real-time monitoring of the resource status of the isolated test environment and the integrity of the test object during execution. An optimization unit is used to fine-tune the Large Language Model (LLM) locally using Low-Rank Adaptive (LoRA) technology based on the execution log, and to optimize the local knowledge base and system test strategy configuration, thereby forming a self-learning closed-loop test system.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the storage medium including a stored program that executes the above-described method when the program is run.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor performs the above-described method through the computer program.

[0017] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the embodiments of the methods described above.

[0018] In this embodiment, test information is obtained, which indicates the software code and / or requirement description of the target software, which is security software that needs to run on the target operating system. A set of test cases matching the test information and the security environment of the target operating system is generated using a locally deployed Large Language Model (LLM) and combined with Retrieval Enhanced Generation (RAG) technology based on a local knowledge base. The local knowledge base stores the security vulnerability rule base and test template library of the target operating system. Test cases in the set are executed sequentially in an isolated test environment, and an execution log of the test case set is generated. The isolated test environment is a test environment built based on container and hardware simulation technology that matches the target operating system and its hardware architecture. The execution log is obtained after real-time monitoring of the resource status of the isolated test environment and the integrity of the test object during execution. Based on the execution log, the Large Language Model (LLM) is fine-tuned locally using Low-Rank Adaptive (LoRA) technology, and the local knowledge base and system test strategy configuration are optimized, thereby forming a self-learning closed-loop test system. This solution requires no connection to an external network and does not upload sensitive code containing business logic and potential vulnerabilities to the cloud, thus eliminating the risk of data leakage and resolving the security risks associated with cloud-based software testing in related technologies. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of an optional offline automated software testing method based on a local large language model and RAG enhancement, according to an embodiment of this application. Figure 2 This is a schematic diagram of an optional offline automated software testing scheme based on a local large language model and RAG enhancement, according to an embodiment of this application. Figure 3 This is a schematic diagram of an optional requirements parsing and use case generation scheme according to an embodiment of this application; Figure 4 This is a schematic diagram of an optional offline execution and monitoring scheme according to an embodiment of this application; Figure 5 This is a schematic diagram of an optional intelligent auditing and iteration scheme according to an embodiment of this application; Figure 6 This is a schematic diagram of an optional closed-loop feedback scheme according to an embodiment of this application; Figure 7 This is a schematic diagram of an optional offline automated software testing apparatus based on a local large language model and RAG enhancement, according to an embodiment of this application; and, Figure 8 This is a structural block diagram of a terminal according to an embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows: LLM: Large Language Model, a deep learning-based natural language processing model used for code analysis and test case generation, supporting optimized inference for domestically produced chips.

[0023] RAG: Retrieval-Augmented Generation, a technique that combines external knowledge base retrieval with a generative model to improve the accuracy and credibility of generated content and ensure that test cases meet security standards.

[0024] LoR (Low-Rank Daptation) is an efficient fine-tuning technique used to quickly optimize model parameters based on test logs.

[0025] psutil is a Python system and process utilization library used for real-time monitoring of CPU / memory resources.

[0026] According to one aspect of the embodiments of this application, a method embodiment of a software offline automated testing method based on a local large language model and RAG enhancement is provided.

[0027] Figure 1 This is a flowchart of an optional offline automated software testing method based on a local large language model and RAG enhancement, according to an embodiment of this application. The entire process of this solution does not require connection to an external network, does not upload sensitive code containing business logic and potential vulnerabilities to the cloud, and therefore poses no risk of data leakage. It solves the technical problem of security vulnerabilities in related technologies that use cloud-based software testing. Figure 1 As shown, the method may include the following steps: Step S102: Obtain test information. The test information is used to indicate the software code and / or requirements description of the target software. The target software is security software that needs to run on the target operating system.

[0028] The target software is security software that needs to run on a specific operating system (such as OpenEuler).

[0029] Step S104: Utilize the locally deployed Large Language Model (LLM) and combine it with Retrieval Enhancement Generation (RAG) technology based on a local knowledge base to generate a set of test cases that match the test information and the security environment of the target operating system. The local knowledge base stores the security vulnerability rule base and test template base of the target operating system.

[0030] The system calls a locally deployed large language model (LLM) (such as a finely tuned CodeLLaMA) and drives the RAG engine. The RAG engine retrieves relevant information from the local knowledge base (an offline database containing rules and test templates for common CVE vulnerabilities, security configuration specifications, attack patterns, etc. of the target operating system) and works in conjunction with the LLM to generate a set of structured test cases for the current test information and the security environment of the target operating system. The entire process does not require a connection to an external network.

[0031] Step S106: Execute the test cases in the test case set sequentially in the isolated test environment and generate the execution log of the test case set. The isolated test environment is a test environment built based on container and hardware simulation technology and matched with the target operating system and its hardware architecture. The execution log is obtained after monitoring the resource status of the isolated test environment and the integrity of the test object in real time during the execution process.

[0032] For execution and monitoring in an isolated environment, the system utilizes Docker container technology to create a clean test environment consistent with the target operating system version. A hardware simulator such as QEMU is used within this container to simulate the target hardware architecture (e.g., AArch64). Subsequently, the system sequentially executes the generated test cases within this isolated environment. During execution, libraries such as psutil are used to monitor the container's CPU and memory usage in real time, and hash values ​​such as SHA-256 are calculated to verify the integrity of critical files. All monitoring data, execution results, and exceptions are recorded in real time, forming a detailed execution log.

[0033] Step S108: Based on the execution log generation, the large language model LLM is fine-tuned locally using low-rank adaptation (LoRA) technology, and the local knowledge base and system test strategy configuration are optimized to form a self-learning closed-loop test system.

[0034] The system analyzes execution logs locally to generate test audit reports (such as coverage reports and vulnerability lists). Simultaneously, using Low-Rank Adaptation (LoRA) technology and log data as training samples, it performs lightweight and efficient fine-tuning of the local LLM (Locally Ranked Model) to optimize the accuracy of its generated test cases. Furthermore, it optimizes the system's test strategy configuration based on the analysis findings. Finally, the fine-tuned model information, optimized strategies, and the experience gained from this test (such as new effective test cases and failure scenarios) are fed back as new knowledge and stored in the local knowledge base. This completes a self-learning closed loop of generation, execution, analysis, and optimization.

[0035] This embodiment achieves truly offline, highly secure automated testing, completely eliminating the risk of data leakage. Through the combination of LLM, RAG, and a local knowledge base, it enables intelligent and targeted test case generation for deep security scenarios of the target operating system, significantly improving the breadth and depth of test coverage. Combined with container isolation and real-time monitoring, the testing process itself possesses security awareness and anomaly detection capabilities, improving the reliability and auditability of the tests. Most importantly, through LoRA-based model fine-tuning and dynamic knowledge base updates, the system has the ability to continuously learn from testing practice and self-optimize, allowing testing efficiency and accuracy to continuously improve with increased usage, solving the problem of capability decay in traditional static testing systems.

[0036] The following is combined Figure 1 The steps shown further detail the technical solution of this application: In the technical solution provided in step S102, test information is obtained.

[0037] Testers can input software code and / or requirements documents in natural language using the Big Prophet model.

[0038] In the technical solution provided in step S104, a set of test cases matching the test information and the security environment of the target operating system is generated by utilizing the locally deployed Large Language Model (LLM) and combining it with the retrieval enhancement generation RAG technology based on the local knowledge base.

[0039] The above embodiments can be implemented in the following ways: 1) Validate the format of the input test information.

[0040] The system first performs basic format checks on the input test information (such as Java source code files) such as UTF-8 encoding to ensure that the data is clean and to avoid invalid input affecting subsequent analysis.

[0041] 2) If the format verification passes, the local large language model LLM is guided by the preset first prompt word template to parse the verified test information and extract the test requirements.

[0042] Requirements Analysis: After successful validation, a pre-defined first prompt template guides the local LLM to perform code / requirements analysis. For example, the prompt might be: "Analyze the following Java Web controller code and extract all security-related test requirements involving HTTP request processing, database interaction, and file operations." Based on this, the LLM outputs structured test requirement highlights.

[0043] 3) Based on testing requirements, relevant security vulnerability rules and test templates are retrieved from the local knowledge base using RAG technology.

[0044] Rule retrieval: Based on the test requirements extracted from the LLM (such as "CORS configuration check" and "SQL parameterized query verification"), the RAG engine retrieves highly similar related security vulnerability rule descriptions and corresponding test step templates from the local knowledge base.

[0045] 4) Inject the retrieved security vulnerability rules and test templates into the preset second prompt word template to guide the local large language model LLM to generate structured test cases. The generated test cases should include at least a test description, operation steps, expected results and confidence assessment.

[0046] Test Case Generation: The specific rules and templates retrieved in the previous step are injected into a second prompt template that focuses more on test case generation, guiding the LLM again. For example: "Based on the following security rule: CORS not configuring the 'Access-Control-Allow-Origin' header may lead to cross-domain attacks, and the test template, generate a specific end-to-end test case for [code snippet to be tested], including description, steps, expected results, and evaluate the confidence level of the generated content." The LLM then outputs a structured JSON format test case.

[0047] 5) Add test cases with confidence scores higher than the set threshold to the test case set. If the number of test cases in the test case set does not meet the requirements, optimize the second prompt word template and guide the local large language model LLM to regenerate structured test cases.

[0048] Quality Filtering and Looping: The system checks the confidence evaluation value of the generated test cases and adds test cases with a value higher than a set threshold (e.g., 0.85) to the final set. If the number of generated test cases is insufficient, the system automatically analyzes the reasons, optimizes the second prompt words (e.g., adding examples, refining requirements), and triggers LLM regeneration until the requirements are met or the maximum number of loops is reached.

[0049] In this embodiment, the intelligent generation process is standardized and streamlined. Through a multi-stage prompting engineering process of "parsing-retrieval-injection generation," it is ensured that the generated test cases not only stem from the general code understanding capabilities of LLM but also closely integrate domain-specific security knowledge, greatly improving the relevance and professionalism of the generated test cases. Confidence thresholds and loop retry mechanisms guarantee the controllable quality and stability of the output results, reducing the interference of invalid or low-quality test cases on subsequent tests.

[0050] In the above scheme, after adding test cases with confidence levels higher than a set threshold to the test case set: 1) Store the test cases with confidence levels higher than the set threshold, the corresponding test requirements, and the security vulnerability rules generated this time as new knowledge entries in the local knowledge base; Knowledge Accumulation: The system automatically associates the test case, its corresponding original test requirements (or code snippets), and the security vulnerability rules that triggered its generation, packages them into a new "knowledge entry," and stores it in the local knowledge base. This enriches the instance library of the knowledge base.

[0051] 2) Based on the feedback in the subsequent test audit report regarding the effectiveness of test cases in new knowledge entries, dynamically adjust the weight or relevance of new knowledge entries in the local knowledge base.

[0052] Knowledge optimization: In subsequent test cycles, after the generated test cases are executed and audit reports are produced, the system will provide feedback based on the reports. For example, if a test case derived from a new knowledge item is proven to be effective and discovers a vulnerability, the system will increase the weight or relevance of that item in RAG retrieval, making it easier to retrieve in the future; conversely, if test cases associated with an item are repeatedly executed ineffectively, its weight will be reduced.

[0053] In this embodiment, the local knowledge base evolves from a static rule repository into a dynamic, self-optimizing experience base. It not only accumulates knowledge but also evaluates the value of that knowledge based on practical feedback and dynamically adjusts its application priority. This enables the system's "intelligent" component (RAG retrieval) to become increasingly accurate with use, effectively avoiding the problems of knowledge base "congestion" and "obsolescence," and is a key link in achieving a continuous self-learning closed loop.

[0054] In the technical solution provided in step S106, test cases in the test case set are executed sequentially in the isolated test environment, and an execution log of the test case set is generated.

[0055] The above embodiments can be implemented in the following ways: 1) Use container technology to create an offline test environment image that corresponds to the system version of the target operating system, and use a hardware simulator to simulate the hardware architecture environment of the target operating system within the container.

[0056] Environment setup: Pull the corresponding Docker base image according to the target operating system version specified in the test plan. When running the container, simulate a specific hardware architecture environment inside the container using QEMU command parameters (such as -cpu cortex-a57) to ensure a high degree of consistency between the test environment and the real deployment environment.

[0057] 2) Inject the test cases in the test case set into the isolated test environment in sequence for execution to simulate security vulnerability attack scenarios.

[0058] Test case injection and execution: The test scheduler converts each test case in the test case set into a command or script that can be executed within the container, and injects it into the container to run, simulating a real attack or abnormal operation process.

[0059] 3) During the execution of test cases, resource usage indicators of the test environment and target software are collected at fixed time intervals through the system resource monitoring library and compared with dynamically adjusted resource thresholds.

[0060] Resource monitoring: During test case execution, a separate monitoring process samples the container and the processes under test within it at fixed time intervals (e.g., per second) to measure metrics such as CPU usage and memory consumption.

[0061] 4) Before and after the test cases are executed, the digest value of the file under test is calculated using a hash algorithm to perform integrity verification.

[0062] Integrity verification: Before and after test case execution, calculate hash values ​​for the key executable files and configuration files of the software under test, and compare the two results to see if they are consistent.

[0063] 5) In real time, determine whether the resource usage index exceeds the resource threshold or whether the file integrity is compromised: if so, interrupt the execution of the current test case and record the exception event in the execution log; if not, the execution will be completed normally and the execution log will be recorded.

[0064] Anomaly detection and logging: Real-time comparison of resource metrics to see if they exceed dynamic thresholds (e.g., CPU consistently >80%), or if hash values ​​are inconsistent. Upon detection of an anomaly, immediately terminate the current test case and write an error log entry containing the anomaly type, the resource snapshot at that moment, and the hash comparison result. If execution completes normally, record a success log entry including execution time and final resource status.

[0065] 6) After all test cases have been executed, a structured execution log is generated, which includes resource monitoring records, integrity verification results, and details of abnormal events.

[0066] Log aggregation: After all test cases are executed, the system aggregates all success logs, error logs, resource monitoring time series data and integrity verification reports for the entire process into a structured and complete execution log file (such as JSON format).

[0067] The above embodiments provide a complete and reliable offline test execution solution. Containerization ensures environmental purity and repeatability, while hardware simulation addresses specific environment dependencies. More importantly, it transforms traditional "black-box" function execution into a deeply observable "white-box" or "grey-box" process. Real-time resource monitoring can proactively detect performance defects and resource leaks; proactive integrity checks can identify accidental file tampering that may occur during testing. This significantly enhances the depth and reliability of the testing process, and the generated execution logs provide a rich data foundation for subsequent in-depth analysis.

[0068] In the above embodiment, after interrupting the execution of the current test case: 1) Record the exception type that caused the interruption, the instantaneous value of the resource usage indicator, and the integrity verification result to the execution log.

[0069] Detailed records: Not only are simple events like "CPU exceeds threshold" recorded, but also the precise instantaneous values ​​when the threshold is exceeded (such as CPU utilization of 92.5%), the test step ID being executed when the exception occurred, and the file hash value calculated at that time, providing sufficient context for problem localization.

[0070] 2) Based on the preset retry strategy, after adjusting the test environment parameters or clearing the test status, the interrupted test cases are re-executed. The preset retry strategy is used to indicate the number of retries or the retry duration.

[0071] Strategic Retry: The system queries the preset retry policies. Policies may specify: for resource exceedance exceptions, wait 10 seconds to release resources, then retry once with reduced concurrency; for temporary service unresponsive exceptions, retry twice immediately. The system automatically cleans up the test environment (e.g., restarting the tested service subprocess) or adjusts environment parameters according to the policy before re-executing the test case.

[0072] 3) If the test case still fails after retrying according to the preset retry strategy, the final failure status is recorded, and the test case is skipped and the subsequent test cases are executed.

[0073] Final Failure Handling: If the test case still fails after retrying, the system marks it as a final failure and logs the detailed retry process. Subsequently, the test scheduler will not stall but will automatically skip this test case and continue executing the next test case in the test set, ensuring the overall progress of the test task.

[0074] This embodiment significantly enhances the robustness and fault tolerance of the automated testing system. Faced with unavoidable occasional fluctuations in the testing environment (such as sudden resource spikes or network interruptions), the system no longer simply stops at the first error, but can strategically retry like an experienced tester, thereby distinguishing between false failures caused by environmental noise and true failures caused by real defects. This reduces test task interruptions due to environmental instability and a large amount of ineffective manual troubleshooting work, improving the automation level and execution efficiency of the testing process.

[0075] In the technical solution provided in step S108, based on the generation of execution logs, the large language model LLM is fine-tuned locally using low-rank adaptation (LoRA) technology, and the local knowledge base and system test strategy configuration are optimized, thereby forming a self-learning closed-loop test system.

[0076] The above embodiments can be implemented in the following ways: 1) Using execution logs as training data, the local large language model LLM is fine-tuned using low-rank adaptation (LoRA) technology to optimize its test case generation capability.

[0077] Model fine-tuning: The system converts structured execution logs (especially information about test case pass / fail and vulnerability types discovered) into training data pairs. Then, using efficient LoRA technology, the underlying LLM model is fine-tuned locally. For example, by pairing a large amount of data with "code patterns that lead to SQL injection vulnerabilities" and "SQL injection test cases that should be generated," the model learns to generate such tests more accurately.

[0078] 2) Based on the parsing of the execution logs, a test audit report containing test coverage and vulnerability analysis results is generated. Based on the analysis conclusions in the test audit report, test strategy optimization suggestions are generated. The system test strategy configuration is updated according to the test strategy optimization suggestions. The test audit report, model fine-tuning information and updated system test strategy configuration related to this test are fed back and stored in the local knowledge base for optimization of subsequent test processes.

[0079] Audit report generation: The system analyzes logs, calculates code coverage and branch coverage, counts the number and distribution of various vulnerabilities, and automatically generates graphic test audit reports (such as Allure reports).

[0080] Strategy Optimization and Knowledge Feedback: Based on the analysis conclusions of the audit report, the system automatically generates strategy optimization suggestions (e.g., "For testing the target interface, the resource monitoring threshold should be lowered from 80% to 70%"). These suggestions are used to update the system's test strategy configuration file. Simultaneously, all valuable information from this test—the audit report, incremental model parameters resulting from fine-tuning (such as LoRA weights), and the updated strategy configuration—is stored as a "test cycle snapshot" in the local knowledge base. When the next test starts, the RAG engine can retrieve this historical "experience" to optimize prompts or adjust the test plan.

[0081] This embodiment achieves a leap from "automation" to "intelligence" in the testing system. Through continuous fine-tuning of LLM using LoRA, the core capabilities of the system's "brain" are enhanced in a targeted manner, making the generated test cases increasingly accurate. The automatic analysis and generation of audit reports frees up the workload of manual report preparation. Most importantly, it structures and digitizes the "experience" from each test, storing it in a knowledge base, giving the entire system memory and evolutionary capabilities, forming a continuously self-reinforcing positive cycle—a feature completely lacking in traditional testing tools.

[0082] The aforementioned test strategy optimization suggestions fall into two categories. One category updates the execution parameters of the system test strategy configuration, such as, "Test case A often experiences a peak memory usage of 1.5GB; it is recommended to increase the memory limit for a single container from 1GB to 2GB." This type of suggestion directly translates into modifications to system runtime configurations such as Docker runtime parameters and resource monitoring thresholds. The other category fine-tunes the prompt word optimization instructions or training data labels of the local large language model (LLM), for example, "The success rate of test case generation for the 'file upload bypass' vulnerability is low." Before the next round of testing begins, the system synchronously applies the updated runtime configuration and loads the LLM service integrating the new prompt word template and the fine-tuned model to ensure that strategy optimization and model evolution work synergistically.

[0083] This embodiment achieves refined management and precise deployment of optimization strategies. It breaks down macro-level optimization suggestions into specific operational instructions for different system components (runtime engine vs. intelligent model), avoiding a one-size-fits-all optimization approach. This ensures that the system can quickly adjust execution-level parameters to adapt to current software characteristics while continuously deepening its intelligent kernel's cognitive and generative capabilities. This collaborative optimization mechanism makes the system's adaptive evolution more efficient and systematic, simultaneously improving the overall performance of the closed-loop testing system from two dimensions.

[0084] The solution of this invention is essentially an intelligent pipeline of "input-processing-output-feedback," running entirely offline on a domestically developed OS (such as OpenEuler), packaged into an independent small box using Docker containers for easy deployment and isolation. The entire system acts like a circular chain: after testing, the report is automatically fed back to the AI ​​model, allowing it to learn from the experience and become smarter in the next test.

[0085] With the increasing prevalence of domestically developed operating systems (such as OpenEuler and Kylin OS) in security system software, software testing is as crucial as a house security check. It helps identify problems early, ensuring software stability and resistance to hacking. However, current automated testing tools, such as MeterSphere and Cover-Agent, while capable of generating basic test cases (like checking code execution) in offline environments, fall short when dealing with the unique challenges of domestically developed security software. This software often involves access control, data encryption, and specific hardware architectures (such as the AArch64 processor), making testing particularly complex. The limitations of this technology include: 1) Significant security risks: Many tools require uploading code to the cloud for AI analysis, which can easily leak confidential company data, especially in high-security offline scenarios; 2) Weak intelligent generation: Tools do not automatically design test scenarios for security vulnerabilities (such as privilege escalation or SQL injection), relying solely on manual auditing, which is inefficient and prone to missing issues unique to domestic systems (such as configuration vulnerabilities in cross-domain access CORS); 3) Incomplete monitoring: During test runtime, it is impossible to monitor resource usage in real time (such as sudden CPU or memory exhaustion), and it is unclear how to quickly verify data integrity (such as using SHA256 hash values ​​to check if files have been tampered with), resulting in audit report coverage often being less than 80%; 4) Inability to self-improve: After generating test cases, the model does not automatically adjust based on test results, leading to repetitive work and low test accuracy.

[0086] The aforementioned audit report coverage, often below 80%, is based on data from the applicant's internal testing: On OpenEuler version 23.03, using MeterSphere to test a typical security audit module (containing 10 Java interfaces), the generated test case coverage was only 72% (reaching 78% after manual auditing supplementation). The test logs showed that three CORS-related vulnerabilities were missed. A risk example specific to domestic operating systems: Under OpenEuler, CORS configuration vulnerabilities may manifest as the absence of the "Access-Control-Allow-Origin" header in WebConfig, leading to cross-domain requests bypassing permissions (such as malicious JavaScript accessing local file audit APIs from external domains). Example code snippet: @RequestMapping(" / audit / file") public ResponseEntity <string>auditFile(@RequestParam String path){ / / CORS not validated, vulnerable to cross-domain injection return ResponseEntity.ok(fileService.query(path)); } This vulnerability is more likely to trigger an "Illegal instruction" error under the AArch64 architecture.

[0087] To address the aforementioned shortcomings, this invention constructs a fully offline closed-loop system using a local large model (such as a finely tuned version of CodeLLaMA) and the RAGFlow / FastGPT framework, thus resolving these deficiencies: It employs a local RAG-enhanced security rule base to avoid cloud uploads and achieve zero leakage; AI automatically injects vulnerability scenarios to generate targeted test cases, improving intelligent coverage; it integrates a dynamic monitoring module to sample resources in real time and verify SHA256 digests to ensure anomaly capture; and it introduces a LoRA adaptive fine-tuning mechanism, using test logs to provide feedback for model optimization, forming a closed-loop learning process.

[0088] It can achieve the following functions: 1) Requirement parsing and test case generation: Local AI parses input code / requirements, RAG retrieves domestic OS rule bases, outputs JSON format test cases, and supports confidence threshold control. 2) Offline execution and monitoring: Docker container + QEMU simulates the AArch64 environment, executes test cases one by one, psutil samples resources in real time, hashlib verifies integrity, and retryes in case of abnormal interruption. 3) Intelligent auditing and iteration: FastGPT parses logs, supports SQL fuzzy query, and generates Allure reports; LoRA fine-tunes the model (dataset 200 records / round), and outputs optimization suggestions. 4) Closed-loop feedback: Report data is injected into the knowledge base, the next round of prompts is automatically optimized, and YAML configuration is persisted.

[0089] In practical applications, the following effects are achieved: 1) Fully offline vulnerability test case generation: After inputting the code, FastGPT parses the requirements using a preset prompt, and RAGFlow pulls CORS / SQL injection rules from the local library (150+ templates) to generate 10 JSON test cases (<10s / test case, confidence >0.85), with a measured coverage of 95.2% (based on OpenEuler 24.03, 50 interface datasets, vs. MeterSphere 7.5%), ensuring zero internet privacy protection. 2) Real-time anomaly audit closed loop: When executing test cases, psutil samples CPU / memory every 5s (threshold dynamically 80%→70%), and hashlib calculates SHA256 verification files; logs are stored in SQLite, and after FastGPT parses them, LoRA fine-tunes them for one round (<5min, RTX 3060 GPU), providing feedback suggestions such as "add to CORS whitelist". After iteration, the audit time is reduced by 50% (1.2h / round, F1-score 0.92).

[0090] As an optional embodiment, the following is combined with Figure 2 The technical solution of this application is further described in detail with reference to specific embodiments: 1) Requirements Analysis and Use Case Generation This feature addresses a weakness in intelligent generation by using local AI to parse input code / requirements, retrieving rules from a domestic OS's rule base using RAG, and outputting JSON-formatted test cases with support for confidence threshold control. This achieves targeted test case generation with zero internet privacy protection, increasing coverage to 95.2% (vs. baseline 78.5%). It serves as the starting point of the process, outputting execution inputs directly connected to function point 2. See also... Figure 3 : Explanation of the flowchart: 1.1) Input preparation: Receive security software code / requirements (such as CORS configuration fragments from Java WebConfig). The system automatically verifies UTF-8 encoding (using the Python codecs library). If invalid, it returns "Invalid input format, please try again" to ensure that the data is clean and can be used for subsequent parsing, avoiding manual intervention.

[0091] The pseudocode example is as follows: def generate_cases(input_code): prompt = f"Generate 10 test cases based on code:{input_code}..." response = fastgpt.generate(prompt) if response.confidence < 0.85: prompt += "Detailed Explanation" response = fastgpt.generate(prompt) return json.loads(response.json_output) 1.2) Requirements Analysis: Load the local FastGPT model, parse it using the preset prompt template, and the complete prompt is: "You are a security testing expert." Based on the following Java code: [Insert code], generates 10 end-to-end test cases, focusing on CORS cross-domain issues and SQL injection. Each test case includes a description, steps, and expected values ​​(including SHA256 checksum), and is compatible with OpenEuler AArch64. Output JSON. Duration <3 seconds, bridging rule retrieval.

[0092] 1.3) Rule retrieval and injection: RAGFlow retrieves matching rules (similarity > 0.7) from the local rule base (150+ templates, source company logs + offline CVEs) and injects them into the prompt to generate JSON test cases.

[0093] The pseudocode example is as follows: {"use_cases": [{"id":1, "description":"CORS Cross-Origin Test", "steps":["Cross-Origin Request", "Response Header Check"], "expected":"Access Denied", "confidence":0.92}]} Runtime < 5 seconds.

[0094] 1.4) Validation and Output: If the confidence level is >0.85, output and store in SQLite (INSERT INTO knowledge); otherwise, retry the prompt. Exception Handling: If less than 5 results are generated, retry 3 times and then fallback to the default template. Closed-Loop Connectivity: Output is fed into the next step, and the knowledge base supports subsequent iterations.

[0095] 2) Offline execution and monitoring This feature addresses a deficiency in "incomplete monitoring" by using a Docker container + QEMU to simulate an AArch64 environment, executing test cases one by one, using psutil to sample resources in real time, using hashlib to verify integrity, and retrying in case of exceptions. This achieves secure isolated execution and real-time exception handling, reducing audit time by 50% (1.2 hours per round). It receives the test case output from step 1 as the core of execution. See details... Figure 4 : 2.1) Input and Startup: Receive JSON test cases, run the offline image (openEuler:offline) using Docker, simulate AArch64 using QEMU (qemu-system-aarch64 -cpu cortex-a57 -m 2G), and deploy in isolation for <10 seconds.

[0096] 2.2) Test case execution: Inject test cases one by one, subprocess.run(['java', '-jar', 'audit.jar', '--case', json.dumps(use_case)]), simulating CORS and other scenarios, with a required time of <30s / batch.

[0097] 2.3) Real-time sampling: psutil.cpu_percent(interval=5) / virtual_memory().percent, the threshold is dynamically adjusted (the first 3 times the threshold is reduced by 70%), which solves the blind spot of resource monitoring.

[0098] 2.4) Integrity verification: hashlib.sha256(file.read()).hexdigest() compares the file and marks the risk of tampering.

[0099] 2.5) Anomaly Detection and Branching: If CPU exceeds the threshold or does not match, interrupt the process (os.kill(pid, SIGTERM)) + SQLite logging (INSERT INTO logs('CPU exceeds threshold', 'high')); otherwise, log normally. Retry <3 times (Illegalinstruction switch - CPU host). Anomaly handling: Over-limit fallback CPU mode.

[0100] 2.6) Output Log: The complete log set bridges to the next step. Closed-loop connectivity: The log supports iterative feedback, with a total duration of <1 minute per batch.

[0101] 3) Intelligent Auditing and Iteration This feature addresses a "lack of self-improvement" deficiency by using FastGPT to parse logs, support SQL fuzzy queries, and generate Allure reports; LoRA fine-tuning of the model (200 records / round of data) and outputting optimization suggestions. This achieves log analysis and model adaptation, improving the F1-score to 0.92. It processes log output, forming the basis for optimization. See details... Figure 5 : 3.1) Input Log: Receive the SQLite log set from function point 2 and export fragment bridging parsing.

[0102] 3.2) Log parsing: FastGPT prompt: "Analyze log: [fragment], calculate coverage (pass / total 100), identify high CORS risk." Parsing time < 1 minute.

[0103] 3.3) Fuzzy query: sqlite3 executes SELECT * FROM logs WHERE message LIKE '%CORS%' ORDER BY risk_level DESC, and feeds the result back to the prompt to improve targeting.

[0104] 3.4) Coverage Calculation and Reporting: The formula is: use case / total 100, Allure generate --report-dir. / report to generate PD.

[0105] The pseudocode example is as follows: Coverage: 95% High risk: CORS (1 location) Recommendation: Add Access-Control-Allow-Origin.

[0106] 3.5) Model fine-tuning: LoRA (r=16, alpha=32), dataset with 200 log entries, RTX 3060 GPU < 2 min, early stopping val_loss > 1.05*min_loss.

[0107] 3.6) Optimize output: Suggestions such as "reduce risk by 30%", anomaly handling: fallback Drools for confidence levels <0.8. Closed-loop connectivity: feed the output into the next step, total time <5min / round.

[0108] 4) Closed-loop feedback This function integrates the first three steps, achieving an adaptive loop through "report data injection into the knowledge base, automatic optimization of the next prompt, and persistent YAML configuration," ensuring the system "learns from its mistakes" and iterates continuously. It serves as the process endpoint, feeding back to the starting point to form a closed loop. See details... Figure 6 : 4.1) Input Report: Receive Allure reports / LoRA models / recommendations, pre-bridging process.

[0109] 4.2) Knowledge base injection: UPDATE knowledge SET optimized_data = report_json to ensure historical accumulation.

[0110] 4.3) Prompt optimization: update_prompt(suggestions, input_code), for example, add "CORS_fix priority" to support the next round of operation.

[0111] 4.4) Configure persistence: Load the YAML file {"suggestions":["CORS_fix"], "lora_r":16} into docker-compose.yml.

[0112] 4.5) Iterative output: Triggers the next round of closed_loop_pipeline. Exception handling: Configuration conflict fallback to the default YAML. Closed-loop connectivity: E feeds back to the overall starting point, achieving self-learning. Total duration <1min.

[0113] The technical solution in this application achieves the following: 1) Fully offline AI security fusion: Unlike cloud tools that require internet access, this invention uses a local AI + RAG library to automatically "fill in" domestic vulnerability scenarios, with zero risk of leakage, making it particularly suitable for high-security environments. Distinguishing it from Cover-Agent: Cover-Agent does not support RAG injection of security rules (only general code coverage), and its test case generation speed is <10s for this invention / 15s for test cases compared to Cover-Agent (company test, dataset with 50 interfaces). 2) Real-time monitoring closed-loop sentinel: During testing, it not only runs the code but also checks resources and hash values ​​simultaneously, stopping and recording any issues immediately. Traditional tools lack this "liveness check." 3) Self-learning evolution mechanism: It uses test results to quickly fine-tune the model (LoRA method, simple and efficient), making the system smarter with use, achieving 20% ​​higher accuracy in covering domestically specific risks (accuracy definition: F1-score, tested: this invention 0.92 vs. baseline 0.76, based on binary classification of 1000 log samples: vulnerability / non-vulnerability). 4) Domestic container adaptation: Docker+QEMU can run OpenEuler seamlessly, deployment is like building blocks, it is compatible with the company's security interfaces (such as measurement policies), and it is extremely practical.

[0114] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0116] According to another aspect of the embodiments of this application, a software offline automated testing apparatus based on a local large language model and RAG enhancement is also provided for implementing the above-described software offline automated testing method based on a local large language model and RAG enhancement. Figure 7 This is a schematic diagram of an optional offline automated software testing device based on a local large language model and RAG enhancement, according to an embodiment of this application. Figure 7 As shown, the device may include: The acquisition unit 701 is used to acquire test information, wherein the test information is used to indicate the software code and / or requirement description of the target software, and the target software is security software that needs to run on the target operating system; Test unit 703 is used to generate a set of test cases that match the test information and the security environment of the target operating system by utilizing the locally deployed Large Language Model (LLM) and combining it with the retrieval enhancement generation (RAG) technology based on the local knowledge base. The local knowledge base stores the security vulnerability rule base and test template base of the target operating system. Log unit 705 is used to sequentially execute test cases in the test case set in the isolated test environment and generate an execution log of the test case set. The isolated test environment is a test environment built based on container and hardware simulation technology and matched with the target operating system and its hardware architecture. The execution log is obtained after real-time monitoring of the resource status of the isolated test environment and the integrity of the test object during the execution process. The optimization unit 707 is used to generate the execution log, fine-tune the large language model LLM locally using low-rank adaptation (LoRA) technology, and optimize the local knowledge base and system test strategy configuration, thereby forming a self-learning closed-loop test system.

[0117] Optionally, the testing unit is further configured to: perform format validation on the input test information; if the format validation passes, guide the local large language model LLM to parse the validated test information and extract test requirements through a preset first prompt word template; based on the test requirements, retrieve relevant security vulnerability rules and test templates from the local knowledge base using RAG technology; inject the retrieved security vulnerability rules and test templates into a preset second prompt word template to guide the local large language model LLM to generate structured test cases, wherein the generated test cases include at least a test description, operation steps, expected results, and confidence assessment; add test cases with confidence scores higher than a set threshold to the test case set; if the number of test cases in the test case set does not meet the requirements, optimize the second prompt word template and guide the local large language model LLM to regenerate structured test cases.

[0118] Optionally, the testing unit is further configured to: after adding test cases with confidence levels higher than a set threshold to the test case set, store the test cases with confidence levels higher than the set threshold, the corresponding test requirements, and the security vulnerability rules generated this time as new knowledge entries in the local knowledge base; and dynamically adjust the weight or relevance of the new knowledge entries in the local knowledge base based on feedback from subsequent test audit reports regarding the effectiveness of test cases in the new knowledge entries.

[0119] Optionally, the log unit is further configured to: create an offline test environment image corresponding to the system version of the target operating system using container technology, and simulate the hardware architecture environment of the target operating system within the container using a hardware simulator; sequentially inject test cases from the test case set into the isolated test environment for execution, simulating security vulnerability attack scenarios; during test case execution, collect test environment and target software resource usage indicators at fixed time intervals through a system resource monitoring library, and compare them with dynamically adjusted resource thresholds; before and after test case execution, calculate the digest value of the tested file using a hash algorithm for integrity verification; determine in real time whether the resource usage indicators exceed the resource threshold or whether the file integrity has been compromised: if so, interrupt the execution of the current test case and record the abnormal event to the execution log; if not, successfully execute and record the execution log after normal execution; after all test cases have been executed, summarize and generate a structured execution log containing resource monitoring records, integrity verification results, and details of abnormal events.

[0120] Optionally, the testing unit is further configured to: after interrupting the execution of the current test case, record the exception type that caused the interruption, the instantaneous value of the resource consumption index, and the integrity verification result to the execution log; according to a preset retry strategy, after adjusting the test environment parameters or clearing the test state, re-execute the interrupted test case, wherein the preset retry strategy is used to indicate the number of retries or the retry duration; if the retry still fails according to the preset retry strategy, record the final failure status, and skip the test case and continue executing subsequent test cases.

[0121] Optionally, the optimization unit is further configured to: use the execution log as training data, fine-tune the local large language model LLM using low-rank adaptation (LoRA) technology to optimize its test case generation capability; generate a test audit report containing test coverage and vulnerability analysis results based on the parsing of the execution log; generate test strategy optimization suggestions based on the analysis conclusions in the test audit report; update the system test strategy configuration based on the test strategy optimization suggestions; and store the test audit report, model fine-tuning information, and updated system test strategy configuration related to this test in the local knowledge base for optimizing subsequent test processes.

[0122] The above-mentioned test strategy optimization suggestions include two types of suggestions. One type of suggestion is used to update the execution parameters of the system test strategy configuration, and the other type of suggestion is used to fine-tune the prompt word optimization instructions or training data labels of the local large language model LLM.

[0123] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run in a corresponding hardware environment, and can be implemented in software or hardware, wherein the hardware environment includes a network environment.

[0124] According to another aspect of the embodiments of this application, a server or terminal is also provided for implementing the above-described software offline automated testing method based on a local large language model and RAG enhancement.

[0125] Figure 8 This is a structural block diagram of a terminal according to an embodiment of this application, such as... Figure 8 As shown, the terminal may include: one or more (only one is shown in the figure) processors 801, memory 803, and transmission devices 805, such as... Figure 8 As shown, the terminal may also include input / output devices 807.

[0126] The memory 803 can be used to store software programs and modules, such as the program instructions / modules corresponding to the software offline automated testing method and apparatus based on a local large language model and RAG enhancement in this embodiment. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 803, thereby realizing the aforementioned software offline automated testing method based on a local large language model and RAG enhancement. The memory 803 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 803 may further include memory remotely located relative to the processor 801, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0127] The aforementioned transmission device 805 is used to receive or send data via a network, and can also be used for data transfer between the processor and memory. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 805 includes a Network Interface Controller (NIC), which can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 805 is a radio frequency (RF) module used for wireless communication with the Internet.

[0128] Specifically, memory 803 is used to store application programs.

[0129] The processor 801 can invoke the application program stored in the memory 803 via the transmission device 805 to perform the following steps: Obtain test information, wherein the test information is used to indicate the software code and / or requirements description of the target software, and the target software is security software that needs to run on the target operating system; Using a locally deployed Large Language Model (LLM) and combined with Retrieval Enhancement Generation (RAG) technology based on a local knowledge base, a set of test cases matching the test information and the security environment of the target operating system is generated. The local knowledge base stores the security vulnerability rule base and test template base of the target operating system. Test cases in the test case set are executed sequentially in the isolated test environment, and an execution log of the test case set is generated. The isolated test environment is a test environment built based on container and hardware simulation technology and matched with the target operating system and its hardware architecture. The execution log is obtained after real-time monitoring of the resource status of the isolated test environment and the integrity of the test object during the execution process. Based on the execution log generation, the large language model (LLM) is fine-tuned locally using low-rank adaptation (LoRA) technology, and the local knowledge base and system testing strategy configuration are optimized, thereby forming a self-learning closed-loop testing system.

[0130] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0131] Those skilled in the art will understand that Figure 8 The structure shown is for illustrative purposes only. The terminal can be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal devices. Figure 8 This does not limit the structure of the aforementioned electronic device. For example, the terminal may also include components that are more... Figure 8 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 8 The different configurations shown.

[0132] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0133] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to execute program code for a software offline automated testing method based on a local large language model and RAG enhancement.

[0134] Optionally, in this embodiment, the storage medium may be located on at least one of the network devices in the network shown in the above embodiment.

[0135] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: Obtain test information, wherein the test information is used to indicate the software code and / or requirements description of the target software, and the target software is security software that needs to run on the target operating system; Using a locally deployed Large Language Model (LLM) and combined with Retrieval Enhancement Generation (RAG) technology based on a local knowledge base, a set of test cases matching the test information and the security environment of the target operating system is generated. The local knowledge base stores the security vulnerability rule base and test template base of the target operating system. Test cases in the test case set are executed sequentially in the isolated test environment, and an execution log of the test case set is generated. The isolated test environment is a test environment built based on container and hardware simulation technology and matched with the target operating system and its hardware architecture. The execution log is obtained after real-time monitoring of the resource status of the isolated test environment and the integrity of the test object during the execution process. Based on the execution log generation, the large language model (LLM) is fine-tuned locally using low-rank adaptation (LoRA) technology, and the local knowledge base and system testing strategy configuration are optimized, thereby forming a self-learning closed-loop testing system.

[0136] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0137] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0138] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0139] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0140] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0142] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0144] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / string>

Claims

1. A software offline automated testing method based on a local large language model and RAG enhancement, characterized in that, include: Obtain test information, wherein the test information is used to indicate the software code and / or requirements description of the target software, and the target software is security software that needs to run on the target operating system; Using a locally deployed Large Language Model (LLM) and combined with Retrieval Enhancement Generation (RAG) technology based on a local knowledge base, a set of test cases matching the test information and the security environment of the target operating system is generated. The local knowledge base stores the security vulnerability rule base and test template base of the target operating system. Test cases in the test case set are executed sequentially in the isolated test environment, and an execution log of the test case set is generated. The isolated test environment is a test environment built based on container and hardware simulation technology and matched with the target operating system and its hardware architecture. The execution log is obtained after real-time monitoring of the resource status of the isolated test environment and the integrity of the test object during the execution process. Based on the execution log generation, the large language model (LLM) is fine-tuned locally using low-rank adaptation (LoRA) technology, and the local knowledge base and system testing strategy configuration are optimized, thereby forming a self-learning closed-loop testing system.

2. The method according to claim 1, characterized in that, Using a locally deployed Large Language Model (LLM) and combined with Retrieval Enhancement Generation (RAG) technology based on a local knowledge base, a set of test cases matching the test information and the security environment of the target operating system is generated, including: Perform format validation on the input test information; If the format verification passes, the local large language model LLM is guided by the preset first prompt word template to parse the verified test information and extract the test requirements; Based on the aforementioned testing requirements, relevant security vulnerability rules and test templates are retrieved from the local knowledge base using RAG technology; The retrieved security vulnerability rules and test templates are injected into a preset second prompt word template to guide the local large language model LLM to generate structured test cases. The generated test cases include at least a test description, operation steps, expected results, and confidence assessment. Test cases with confidence scores higher than a set threshold are added to the test case set. If the number of test cases in the test case set does not meet the requirements, the local large language model LLM is guided to regenerate structured test cases after optimizing the second prompt word template.

3. The method according to claim 2, characterized in that, After adding test cases with confidence levels higher than a set threshold to the test case set, the method further includes: The test cases with confidence levels higher than the set threshold, along with their corresponding test requirements and security vulnerability rules, generated this time will be stored as new knowledge entries in the local knowledge base. Based on feedback from subsequent test audit reports regarding the effectiveness of test cases in new knowledge entries, the weights or relevance of new knowledge entries in the local knowledge base are dynamically adjusted.

4. The method according to claim 1, characterized in that, The test cases in the test case set are executed sequentially in the isolated test environment, and an execution log of the test case set is generated, including: An offline test environment image corresponding to the system version of the target operating system is created using container technology, and the hardware architecture environment of the target operating system is simulated within the container using a hardware simulator, thereby obtaining the isolated test environment; The test cases in the test case set are sequentially injected into the isolated test environment for execution to simulate security vulnerability attack scenarios. During test case execution, resource usage metrics of the test environment and target software are collected at fixed time intervals through the system resource monitoring library and compared with dynamically adjusted resource thresholds. Before and after the test cases are executed, the digest value of the file under test is calculated using a hash algorithm to perform integrity verification. Real-time determination of whether resource usage exceeds resource thresholds or file integrity is compromised: if so, interrupt the execution of the current test case and record the exception event in the execution log; if not, record successful execution in the execution log after normal completion. After all test cases have been executed, a structured execution log is generated, which includes resource monitoring records, integrity verification results, and details of abnormal events.

5. The method according to claim 4, characterized in that, After interrupting the execution of the current test case, the method further includes: Record the type of exception that caused the interruption, the instantaneous value of the resource usage indicator, and the integrity verification result to the execution log; According to the preset retry strategy, after adjusting the test environment parameters or clearing the test state, the interrupted test cases are re-executed. The preset retry strategy is used to indicate the number of retries or the retry duration. If the test fails after retrying according to the preset retry strategy, the final failure status is recorded, and the test case is skipped before continuing to execute subsequent test cases.

6. The method according to any one of claims 1 to 5, characterized in that, Based on the execution logs, the large language model (LLM) is fine-tuned locally using low-rank adaptation (LoRA) technology, and the local knowledge base and system testing strategy configuration are optimized, including: Using the execution logs as training data, the local large language model LLM is fine-tuned using low-rank adaptation (LoRA) technology to optimize its test case generation capability. Based on the parsing of the execution logs, a test audit report containing test coverage and vulnerability analysis results is generated. According to the analysis conclusions in the test audit report, test strategy optimization suggestions are generated. The system test strategy configuration is updated according to the test strategy optimization suggestions. The test audit report, model fine-tuning information, and updated system test strategy configuration related to this test are fed back and stored in the local knowledge base for optimizing subsequent test processes.

7. The method according to claim 6, characterized in that, The test strategy optimization suggestions include two types of suggestions. One type of suggestion is used to update the execution parameters of the system test strategy configuration, and the other type of suggestion is used to fine-tune the prompt word optimization instructions or training data labels of the local large language model LLM.

8. A software offline automated testing device based on a local large language model and RAG enhancement, characterized in that, include: An acquisition unit is used to acquire test information, wherein the test information is used to indicate the software code and / or requirements description of the target software, and the target software is security software that needs to run on a target operating system; The testing unit is used to generate a set of test cases that match the test information and the security environment of the target operating system by utilizing the locally deployed Large Language Model (LLM) and combining it with the retrieval enhancement generation (RAG) technology based on the local knowledge base. The local knowledge base stores the security vulnerability rule base and test template base of the target operating system. The log unit is used to sequentially execute test cases in the test case set in the isolated test environment and generate an execution log of the test case set. The isolated test environment is a test environment built based on container and hardware simulation technology and matched with the target operating system and its hardware architecture. The execution log is obtained after real-time monitoring of the resource status of the isolated test environment and the integrity of the test object during the execution process. An optimization unit is used to generate, based on the execution log, fine-tune the large language model LLM locally using low-rank adaptation (LoRA) technology, and optimize the local knowledge base and system test strategy configuration, thereby forming a self-learning closed-loop test system.

9. A computer-readable storage medium, characterized in that, The storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 7 when it is run.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the method described in any one of claims 1 to 7 through the computer program.

Images