Method for generating automatic test script of time sequence system of Internet of Things based on large model
Through the automated test script generation method based on large models, the problem of low efficiency and lack of timing feature generation of test scripts in the Internet of Things timing system is solved, and efficient and comprehensive test script generation and test coverage are achieved.
Patent Information
- Application Number
- CN202510275298.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-10
- Publication Date
- 2025-06-10
AI Technical Summary
Existing IoT timing system test script generation methods are inefficient, lack timing feature support, and difficult to model data flow characteristics.
Using a large model-based automated test script generation method, combining natural language generation capabilities, LoRA fine-tuning technology and RAG retrieval enhancement, we extract key characteristics of the device data flow and generate real-time, timing dependence and exception testing scripts.
It significantly improves the efficiency of testing script writing, fully supports IoT timing feature testing, enhances script adaptability and scalability, and improves test quality and result analysis capabilities.
Smart Images
Figure CN120123246A_ABST
Abstract
Description
Technical Field
[0001] The present invention discloses an automated test script generation method for an Internet of Things (IoT) time series system based on a large model, which relates to the technical field of testing IoT systems. Background Art
[0002] The IoT time series system has complex data flow characteristics and strict real-time requirements. The existing test script generation methods have the following deficiencies:
[0003] 1. Low script generation efficiency: Test scripts usually need to be written manually, which is time-consuming and laborious, and it is difficult to cover complex scenarios.
[0004] 2. Lack of support for time series characteristics: Existing methods are difficult to fully simulate the time series dependence and real-time requirements between devices.
[0005] 3. Difficult to model data flow characteristics: The time series data of IoT devices has diversity and dynamics, and existing tools are difficult to automatically extract and utilize these characteristics. Summary of the Invention
[0006] Aiming at the problems of the existing technology, the present invention provides an automated test script generation method for an IoT time series system based on a large model, which combines the natural language generation ability of the large model, as well as the LoRA (Low-Rank Adaptation) efficient fine-tuning technology and RAG (Retrieval-Augmented Generation) to enhance the generation quality, and can automatically generate high-quality test scripts and adapt to the complex requirements of the IoT time series system.
[0007] The specific solution proposed by the present invention is:
[0008] The present invention provides an automated test script generation method for an IoT time series system based on a large model, including:
[0009] Step 1: Use a time series analysis tool to extract the key characteristics of the device data flow. The key characteristics include time series dependence and data flow pattern characteristics. The time series dependence is used to analyze the trigger relationship between devices, and the data flow pattern characteristics are used to identify periodic, trend, and sudden characteristics.
[0010] Step 2: Select a large model according to the test requirements, fine-tune the large model for the Internet of Things time series system scenario, use RAG retrieval enhancement to dock with an external knowledge base, dynamically retrieve relevant information and combine it with the large model, and generate test scripts based on the key characteristics of the device data stream. The test scripts include real-time test scripts, time series dependency test scripts, and exception test scripts. The real-time test scripts are used to verify the system response time, latency, and throughput. The time series dependency test scripts are used to simulate complex causal chains. The exception test scripts are used to insert abnormal data streams to verify the fault tolerance of the system.
[0011] Furthermore, in the method for generating an automated test script for an Internet of Things time series system based on a large model, it further includes Step 3: Simulate device behavior, generate test data streams that conform to time series characteristics, execute the test scripts based on the test data streams, record the system behavior, generate a test report containing test metrics, and perform a quality assessment on the test scripts.
[0012] Furthermore, in Step 3 of the method for generating an automated test script for an Internet of Things time series system based on a large model, generating test data streams that conform to time series characteristics includes:
[0013] Generate test data streams in the real mode, which are used to reproduce real device behavior based on historical data.
[0014] Generate test data streams in the random mode, which are used to test the robustness of the system.
[0015] Generate test data streams in the abnormal mode, which are used to verify the fault tolerance performance of the system.
[0016] Furthermore, in Step 3 of the method for generating an automated test script for an Internet of Things time series system based on a large model, generating a test report containing test metrics, where the test metrics include:
[0017] Real-time metrics: Response time, latency, throughput;
[0018] Data integrity metrics: Packet loss rate, data consistency;
[0019] Time series dependency verification metrics: Correctness of the causal chain.
[0020] Furthermore, in Step 3 of the method for generating an automated test script for an Internet of Things time series system based on a large model, performing a quality assessment on the test scripts includes:
[0021] Establish quality assessment metrics and perform a quality assessment through the quality assessment metrics. The quality assessment metrics include: Function coverage rate, script execution success rate, and manual scoring metrics.
[0022] When conducting automated evaluation, the function coverage rate is calculated based on the number of target function points covered by the generated test script. By automatically running the generated test script, the success rate and failed test cases are recorded, and the executability of the script is quantified to calculate the script execution success rate.
[0023] When conducting manual scoring, scores are calculated according to the script readability, function coverage, and boundary condition support metrics in the manual scoring metrics, and the scoring results are recorded in the database for subsequent training and optimization of the generated model.
[0024] The present invention also provides an automated test script generation device for an Internet of Things timing system based on a large model, including a data stream parsing module and a large model generation module.
[0025] The data stream parsing module uses time series analysis tools to extract the key characteristics of the device data stream. The key characteristics include timing dependence and data stream pattern characteristics. Timing dependence is used to analyze the trigger relationship between devices, and data stream pattern characteristics are used to identify periodic, trend, and burst characteristics.
[0026] The large model generation module selects a large model according to the test requirements, fine-tunes the large model for the Internet of Things timing system scenario, uses RAG retrieval enhancement to dock with an external knowledge base, dynamically retrieves relevant information and combines it with the large model, and generates test scripts based on the key characteristics of the device data stream. The test scripts include real-time test scripts, timing dependence test scripts, and exception test scripts. The real-time test scripts are used to verify the system response time, latency, and throughput. The timing dependence test scripts are used to simulate complex causal chains. The exception test scripts are used to insert abnormal data streams to verify the fault tolerance of the system.
[0027] Furthermore, the automated test script generation device for an Internet of Things timing system based on a large model further includes a simulation module, a verification module, and a quality evaluation module. The simulation module simulates device behavior and generates test data streams that conform to timing characteristics. The verification module executes the test scripts based on the test data streams, records the system behavior, and generates a test report containing test metrics. The quality evaluation module evaluates the quality of the test scripts.
[0028] Furthermore, the simulation module of the automated test script generation device for an Internet of Things timing system based on a large model generates test data streams that conform to timing characteristics, including:
[0029] Generating test data streams in a real mode for reproducing real device behavior based on historical data.
[0030] Generating test data streams in a random mode for testing the robustness of the system.
[0031] Generating test data streams in an abnormal mode for verifying the fault tolerance performance of the system.
[0032] Further, the verification module of the described automated test script generation device for the Internet of Things timing system based on a large model generates a test report containing test metrics, where the test metrics include:
[0033] Real-time metrics: response time, latency, throughput;
[0034] Data integrity metrics: packet loss rate, data consistency;
[0035] Timing dependency verification metrics: correctness of the causal chain.
[0036] Further, the quality assessment module of the described automated test script generation device for the Internet of Things timing system based on a large model conducts a quality assessment of the test script, including:
[0037] Establish quality assessment metrics, and conduct quality assessment through the quality assessment metrics. The quality assessment metrics include: function coverage rate, script execution success rate, and manual scoring metrics.
[0038] When conducting automated assessment, calculate the function coverage rate based on the number of target function points covered by the generated test script, record the success rate and failed test cases according to the automatically run test script, and calculate the script execution success rate by quantifying the executability of the script.
[0039] When conducting manual scoring, calculate the score based on the script readability, function coverage, and boundary condition support metrics in the manual scoring metrics, and record the scoring results in the database for subsequent training and optimization of the generation model.
[0040] The beneficial effects of the present invention are:
[0041] The writing efficiency of the test script is significantly improved: By utilizing the natural language generation ability of the large model, the present invention realizes the full-automatic generation of test scripts. Compared with the method of relying on manual writing of scripts, the present invention can quickly generate high-quality test scripts, significantly shorten the script development cycle, reduce labor costs, and greatly improve the test efficiency.
[0042] Fully support the testing of Internet of Things timing characteristics: The present invention can generate dedicated test scripts for the characteristics of the Internet of Things timing system, such as timing dependency, real-time performance, and data flow characteristics, covering complex scenarios that are difficult to handle in the prior art. Through timing dependency analysis and real-time verification, the present invention can more comprehensively detect the performance and reliability of the Internet of Things system, effectively improving the test coverage rate and accuracy.
[0043] Enhanced adaptability and scalability of test scripts: The present invention introduces LoRA fine-tuning technology and RAG retrieval-enhanced generation technology, enabling large models to adapt to the diverse requirements of the Internet of Things (IoT) time-series system. The LoRA fine-tuning technology enables the generation model to be efficiently optimized for IoT scenarios, while the RAG technology enhances the accuracy and context relevance of script generation by dynamically retrieving relevant knowledge bases, ensuring the applicability and dynamic expansion ability of scripts in different IoT scenarios.
[0044] Improved device behavior simulation ability: The present invention can further generate multi-mode device data streams through the designed simulation module, providing diverse inputs for test scripts. Compared with the limitations of existing methods that are difficult to effectively simulate complex device behaviors, the present invention can more accurately simulate the real operating environment of IoT devices, thereby improving the authenticity and reliability of testing.
[0045] Significantly improved test quality and test result analysis ability: The present invention can further generate high-quality test scripts to verify key performance indicators such as system real-time performance, packet loss rate, and throughput by combining time-series analysis tools such as ARIMA and LSTM with large models, improving the accuracy and scientific nature of testing. In addition, through the test reports generated by the verification module, the present invention can comprehensively record the behavior of the system in complex scenarios, providing a reliable basis for performance optimization and problem location. Brief Description of the Drawings
[0046] Figure 1 It is a schematic diagram of the device framework of the present invention.
[0047] Figure 2 It is a schematic diagram of the application process of the method of the present invention.
[0048] Figure 3 It is a schematic diagram of the process for extracting key characteristics of device data streams. Detailed Embodiments
[0049] The following further describes the present invention in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the specific embodiments cited are not intended to limit the present invention.
[0050] Embodiment 1
[0051] The present invention provides a method for generating automated test scripts for IoT time-series systems based on large models. The process can be referred to as follows:
[0052] Step 1: Use time series analysis tools such as ARIMA models, LSTM, Transformer, etc. to extract the key features of the device data stream. The key features include temporal dependence and data stream pattern features. Temporal dependence is used to analyze the trigger relationship between devices, and data stream pattern features are used to identify periodic, trend, and burst features.
[0053] Step 2: Select a large model according to the test requirements, such as GPT-4, Llama large model, and fine-tune the large model for the Internet of Things temporal system scenario. Use RAG retrieval enhancement to dock with an external knowledge base, dynamically retrieve relevant information and combine it with the large model. Based on the key features of the device data stream, generate test scripts. The test scripts include real-time test scripts, temporal dependence test scripts, and exception test scripts. The real-time test scripts are used to verify the system response time, latency, and throughput. The temporal dependence test scripts are used to simulate complex causal chains. The exception test scripts are used to insert abnormal data streams to verify the fault tolerance of the system.
[0054] Step 3: Simulate device behavior to generate test data streams that conform to temporal characteristics. Based on the test data streams, execute the test scripts, record the system behavior, generate a test report containing test metrics, and conduct a quality assessment of the test scripts.
[0055] Among them, generating test data streams that conform to temporal characteristics in Step 3 may include:
[0056] Generate test data streams in real mode for reproducing real device behavior based on historical data.
[0057] Generate test data streams in random mode for testing the robustness of the system.
[0058] Generate test data streams in abnormal mode for verifying the fault tolerance performance of the system.
[0059] Generate a test report containing test metrics, where the test metrics include:
[0060] Real-time metrics: response time, latency, throughput;
[0061] Data integrity metrics: packet loss rate, data consistency;
[0062] Temporal dependence verification metrics: correctness of the causal chain.
[0063] Conduct a quality assessment of the test scripts, including:
[0064] Establish quality assessment metrics and conduct a quality assessment through the quality assessment metrics. The quality assessment metrics include: function coverage Recall rate, script execution success rate AC rate, and manual scoring metrics.
[0065] When conducting automated evaluation, the function coverage rate is calculated based on the number of target function points covered by the generated test script. By automatically running the generated test script, the success rate and failed test cases are recorded, and the executability of the script is quantified to calculate the script execution success rate.
[0066] When conducting manual scoring, according to the manual scoring metrics:
[0067] Script readability, scored from 1 to 5, to determine whether it is easy to understand and maintain for scoring;
[0068] Function coverage, scored from 1 to 5, to determine whether all input requirements are covered for scoring;
[0069] Boundary condition support index, scored from 1 to 5, to determine whether outliers and boundary values are covered for scoring.
[0070] Calculate the score, record the scoring results in the database for subsequent training and optimization of the generated model.
[0071] The comparative advantage of the method of the present invention in test results. In the experimental environment, when using the test script generated by the present invention to test the Internet of Things platform, compared with the existing test of manually written scripts:
[0072] The test script writing time of the present invention is reduced by 75%;
[0073] The test coverage rate is increased by 15%, especially in the aspect of timing-dependent testing and abnormal scenario testing;
[0074] Compared with directly calling the LLM to write test scripts: the Recall rate is increased by 20%, and the AC rate is increased by 15%; the test accuracy is increased by 5%, effectively reducing the misjudgment rate caused by script defects.
[0075] It can be seen that the method of the present invention reduces the development cost and threshold: significantly reducing the participation requirements of professionals and lowering the technical threshold for test script development. Even in an environment with complex and variable test requirements, high-quality test scripts can be quickly generated and executed at a low cost.
[0076] Through the above analysis, it can be seen that the present invention surpasses the existing technology in terms of test efficiency, coverage rate, adaptability, scalability, and test accuracy, and can effectively solve the difficulties and pain points in the testing process of the Internet of Things timing system, providing significant technical advantages and economic value for practical applications.
[0077] Embodiment 2
[0078] The present invention also provides an automated test script generation device for the Internet of Things timing system based on a large model, including a data flow parsing module and a large model generation module.
[0079] The data stream analysis module uses time series analysis tools to extract the key characteristics of the device data stream. The key characteristics include temporal dependence and data stream pattern characteristics. Temporal dependence is used to analyze the trigger relationship between devices, and the data stream pattern characteristics are used to identify periodic, trend, and burst characteristics.
[0080] The large model generation module selects a large model according to the test requirements, fine-tunes the large model for the Internet of Things time series system scenario, uses RAG retrieval enhancement to dock with the external knowledge base, dynamically retrieves relevant information and combines it with the large model, and generates test scripts based on the key characteristics of the device data stream. The test scripts include real-time test scripts, temporal dependence test scripts, and exception test scripts. The real-time test scripts are used to verify the system response time, latency, and throughput. The temporal dependence test scripts are used to simulate complex causal chains. The exception test scripts are used to insert abnormal data streams to verify the fault tolerance of the system.
[0081] Regarding the information interaction and execution process among the modules in the above device, since they are based on the same concept as the method embodiment of the present invention, the specific content can be referred to the description in the method embodiment of the present invention and will not be elaborated here.
[0082] Similarly, the advantages of the device of the present invention are:
[0083] The test script writing efficiency is significantly improved: The present invention realizes the fully automated generation of test scripts by utilizing the natural language generation ability of the large model. Compared with the method of relying on manual script writing, the present invention can quickly generate high-quality test scripts, significantly shorten the script development cycle, reduce labor costs, and greatly improve the test efficiency.
[0084] Fully support the testing of Internet of Things time series characteristics: The present invention can generate dedicated test scripts for the characteristics of the Internet of Things time series system, such as temporal dependence, real-time, and data stream characteristics, covering complex scenarios that are difficult to handle in the prior art. Through temporal dependence analysis and real-time verification, the present invention can more comprehensively detect the performance and reliability of the Internet of Things system, effectively improving the test coverage rate and accuracy.
[0085] The adaptability and expandability of the test scripts are enhanced: The present invention introduces the LoRA fine-tuning technology and the RAG retrieval enhancement generation technology, enabling the large model to adapt to the diverse needs of the Internet of Things time series system. The LoRA fine-tuning technology enables the generation model to be efficiently optimized for the Internet of Things scenario, while the RAG technology enhances the accuracy and context relevance of script generation by dynamically retrieving relevant knowledge bases, ensuring the applicability and dynamic expansion ability of the scripts in different Internet of Things scenarios.
[0086] Enhancement of Device Behavior Simulation Capability: The present invention can further generate multi-mode device data streams through the designed simulation module, providing diverse inputs for test scripts. Compared with the limitations of existing methods that are difficult to effectively simulate complex device behaviors, the present invention can more accurately simulate the real operating environment of Internet of Things devices, thereby improving the authenticity and reliability of testing.
[0087] Significantly Improve Test Quality and Test Result Analysis Capability: The present invention can further generate high-quality test scripts to verify key performance indicators such as the real-time performance, packet loss rate, and throughput of the system by combining time series analysis tools such as ARIMA and LSTM with large models, improving the accuracy and scientific nature of testing. In addition, through the test reports generated by the verification module, the present invention can comprehensively record the behaviors of the system in complex scenarios, providing a reliable basis for performance optimization and problem location.
[0088] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted according to needs. The system structures described in the above-mentioned embodiments can be physical structures or logical structures. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be jointly implemented.
[0089] The above-described embodiments are merely preferred embodiments cited to fully illustrate the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the present invention is subject to the claims.
Claims
1. A method for generating automated test scripts for IoT timing systems based on a large model, characterized by: include: Step 1: Use time series analysis tools to extract key features of device data streams. Key features include timing dependency and data stream pattern features. Timing dependency is used to analyze the trigger relationship between devices, and data stream pattern features are used to identify periodicity, trend, and burst characteristics. Step 2: Select a large model based on the test requirements, and fine-tune the large model for the IoT timing system scenario. Use RAG retrieval to enhance the connection with the external knowledge base, dynamically retrieve relevant information and combine it with the large model, and generate test scripts based on the key characteristics of the device data flow. The test scripts include real-time test scripts, timing dependency test scripts, and exception test scripts. The real-time test script is used to verify the system response time, delay, and throughput. The timing dependency test script is used to simulate complex causal chains. The exception test script is used to insert abnormal data flows to verify the system's fault tolerance.
2. According to claim 1, a method for generating automated test scripts for an Internet of Things timing system based on a large model is characterized by: It also includes step 3: simulating device behavior, generating a test data stream that meets the timing characteristics, executing the test script based on the test data stream, recording the system behavior, generating a test report containing test indicators, and performing quality assessment on the test script.
3. According to claim 2, a method for generating automated test scripts for an Internet of Things timing system based on a large model is characterized by: In step 3, a test data stream that meets the timing characteristics is generated, including: Generate realistic test data streams to reproduce real device behavior based on historical data, Generate a random pattern of test data streams to test the robustness of the system, Generate test data streams with abnormal patterns to verify the fault-tolerance performance of the system.
4. According to claim 2, a method for generating automated test scripts for an Internet of Things timing system based on a large model is characterized by: In step 3, a test report containing test indicators is generated, where the test indicators include: Real-time indicators: response time, latency, and throughput; Data integrity indicators: packet loss rate, data consistency; Timing dependency verification indicator: Correctness of causal chain.
5. According to claim 2, a method for generating automated test scripts for an Internet of Things timing system based on a large model is characterized by: In step 3, the test script is evaluated for quality, including: Establish quality assessment indicators and conduct quality assessment through quality assessment indicators, including: function coverage, script execution success rate and manual scoring indicators, When conducting automated evaluation, the functional coverage is calculated based on the number of target functional points covered by the generated test scripts. The success rate and failure cases are recorded based on the test scripts automatically run, and the script execution success rate is calculated based on the executability of the scripts. When performing manual scoring, the score is calculated based on the script readability, functional coverage, and boundary condition support indicators in the manual scoring indicators, and the scoring results are recorded in the database for subsequent training and optimization of the generated model.
6. A device for generating automatic test scripts for an Internet of Things timing system based on a large model, characterized in that Including data flow analysis module and large model generation module, The data flow parsing module uses time series analysis tools to extract key features of device data flows, including timing dependencies and data flow pattern features. Timing dependencies are used to analyze the triggering relationship between devices, and data flow pattern features are used to identify periodicity, trend, and burst characteristics. The large model generation module selects a large model according to the test requirements, and fine-tunes the large model for the IoT timing system scenario. It uses RAG retrieval to enhance the connection with the external knowledge base, dynamically retrieves relevant information and combines it with the large model, and generates test scripts based on the key characteristics of the device data flow. The test scripts include real-time test scripts, timing dependency test scripts and exception test scripts. The real-time test script is used to verify the system response time, delay and throughput. The timing dependency test script is used to simulate complex causal chains. The exception test script is used to insert abnormal data flows to verify the system's fault tolerance.
7. The device for generating automatic test scripts for an Internet of Things timing system based on a large model according to claim 6 is characterized in that It also includes a simulation module, a verification module and a quality assessment module. The simulation module simulates device behavior and generates a test data stream that conforms to timing characteristics. The verification module executes the test script based on the test data stream, records the system behavior, and generates a test report containing test indicators. The quality assessment module performs quality assessment on the test script.
8. The device for generating automatic test scripts for timing systems of the Internet of Things based on a large model according to claim 7 is characterized in that The simulation module generates test data streams that meet timing characteristics, including: Generate realistic test data streams to reproduce real device behavior based on historical data, Generate a random pattern of test data streams to test the robustness of the system, Generate test data streams with abnormal patterns to verify the fault-tolerance performance of the system.
9. The device for generating automatic test scripts for an Internet of Things timing system based on a large model according to claim 7 is characterized in that The verification module generates a test report containing test indicators, including: Real-time indicators: response time, latency, and throughput; Data integrity indicators: packet loss rate, data consistency; Timing dependency verification indicator: Correctness of causal chain.
10. The device for generating automatic test scripts for an Internet of Things timing system based on a large model according to claim 7, characterized in that The quality assessment module performs quality assessment on test scripts, including: Establish quality assessment indicators and conduct quality assessment through quality assessment indicators, including: function coverage, script execution success rate and manual scoring indicators, When conducting automated evaluation, the functional coverage is calculated based on the number of target functional points covered by the generated test scripts. The success rate and failure cases are recorded based on the test scripts automatically run, and the script execution success rate is calculated based on the executability of the scripts. When performing manual scoring, the score is calculated based on the script readability, functional coverage, and boundary condition support indicators in the manual scoring indicators, and the scoring results are recorded in the database for subsequent training and optimization of the generated model.