Automatic testing system and method based on AI large model

Abstract

The invention relates to the technical field of intelligent terminals and network testing, in particular to an automatic testing system and method based on an AI large model.According to the system, parameters are configured and manually rechecked according to the testing type and the network environment, knowledge base data are retrieved through the RAG technology, an optimal testing case combination is dynamically generated in combination with the testing stage, the equipment model and the like, and the testing efficiency is improved. Testing of reliability, stability, performance and the like is automatically executed, key indexes are collected and re-checked by professionals after AI intelligent merging analysis, finally, a standardized report is output and fed back to a data optimization system and a knowledge base, a computing power-model-agent collaborative system is constructed, and automation of a whole test link is achieved; according to the method, the test period is shortened by more than 50%, the test precision and coverage are remarkably improved, the method is adaptive to multiple types of terminals and multiple scenes, test resources are optimized, it is ensured that results are objective and repeatable, and the method has sustainable evolution ability.

Description

Technical Field

[0001] This invention relates to the field of smart terminal and network testing technology, specifically an automated testing system and method based on a large AI model. Background Technology

[0002] With the rapid development of artificial intelligence technology, the functions of smart terminal devices are becoming increasingly complex, and the testing requirements for their software systems and AI capabilities are also rising. Traditional automated testing systems mostly rely on pre-written scripts and fixed rules. The generation, execution, and result judgment of test cases usually require a large amount of manual intervention, making it difficult to adapt to the testing needs of rapid iteration, multimodal interaction, and high-concurrency scenarios. Especially when dealing with multiple types of terminals such as mobile phones, computers, cameras, and all-in-one machines, traditional testing systems often need to build test environments and write adaptation scripts for different device types, resulting in low testing efficiency, incomplete coverage, and high maintenance costs, and making it difficult to objectively and quantitatively evaluate AI-related functions.

[0003] While some existing automated testing tools incorporate simple machine learning methods to assist testing, their AI capabilities are often limited to single stages, such as test data generation or defect prediction, failing to permeate the entire testing chain and lacking a systematic "computing power-model-agent" collaborative testing framework. Furthermore, traditional testing systems still exhibit significant shortcomings in multimodal data fusion processing, dynamic test path planning, and cross-terminal unified scheduling, resulting in limited testing intelligence and an inability to truly automate the entire process from test case design to result evaluation. This hinders the overcoming of problems such as strong reliance on manual intervention, poor adaptability, low efficiency, and subjective evaluation. Therefore, there is an urgent need to develop an automated testing system and method based on large AI models to overcome the deficiencies in current practical applications. Summary of the Invention

[0004] The purpose of this invention is to provide an automated testing system and method based on large AI models to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: An automated testing system based on an AI large model includes a management service layer, an AI intelligent decision service layer, a test execution server cluster, the device under test, and external devices under test. The management service layer, as the core control system, communicates with the AI ​​intelligent decision service layer and the test execution server cluster, and is responsible for test material management, test case management, test task distribution and management, test result statistics and manual review. The AI ​​intelligent decision service layer communicates with the test execution server cluster and serves as the core control decision layer for test steps and the judgment layer for the test process. It is used to dynamically generate test cases based on RAG technology and to perform intelligent analysis of test data. The test execution server cluster is used to schedule and execute specific test tasks and control the device under test and external devices under test to complete the test. The device under test includes multiple types of physical test terminals and virtual test terminals, which are used to simulate user operations in a real network environment for performance testing. The external device under test is connected to the device under test, and the test is completed using the signals provided by the device under test. The components work together to build a collaborative testing system of computing power, model, and intelligent agent, realizing intelligent and automated testing across the entire process, and the test data can be fed back to the system for continuous optimization.

[0006] As a further aspect of the present invention: the management service layer is also used to receive test environment parameter data, which includes test environment parameter data, network status data, and scenario-related parameters obtained directly from the device under test, obtained from the device under test management platform, manually entered, or obtained from an external data platform through an interface.

[0007] As a further aspect of the present invention: the AI ​​intelligent decision service layer has a built-in knowledge base, which stores test cases, test scenario data, historical test cases and multi-scenario test-related data, to provide data support for dynamic generation of test cases and AI intelligent analysis.

[0008] As a further aspect of the present invention: the management service layer provides a visualization interface, which is used to display the changing trends of key indicators, the location of anomalies and the analysis of possible causes, and supports manual review.

[0009] An automated testing method based on large AI models includes the following steps: Step 1: Test environment configuration and parameter setting. Based on the test type, the device under test and the network environment, relevant parameter data are obtained and key test parameters are automatically set, which are then manually reviewed and confirmed. Step 2: Dynamic generation of test cases. Based on RAG technology, knowledge base data is retrieved, and the optimal combination of test cases is intelligently matched and generated by combining the test stage, the model of the device under test, and historical test results. Step 3: Automated test execution. According to the test cases and configuration parameters, automatically execute the tests and collect multi-dimensional key performance indicators in real time; Step 4: AI intelligent merging analysis. Using AI models, the collected test data is correlated, noisy data is filtered, outliers are identified, and the causes of anomalies are preliminarily determined to generate preliminary test conclusions. Step 5: Manual review and confirmation. Professional technicians confirm or correct the test results based on the preliminary test conclusions and the actual situation. Step 6: Test results output and optimization. A standardized test report is automatically generated, and relevant test data is fed back to the knowledge base to optimize the system and AI model for continuous evolution.

[0010] As a further aspect of the present invention: in step 1, the parameter data includes the parameters of the device under test, network status data, and scenario-related parameters, and the key test parameters include test materials, test duration, and test locations.

[0011] As a further aspect of the present invention: in step 2, the testing phase includes selection, delivery, installation, scheduled maintenance and flight phases, with different test case matching logics emphasizing different aspects for each testing phase.

[0012] As a further aspect of the present invention: in step 3, the test includes reliability testing, stability testing and bandwidth testing, and the key performance indicators include network connection success rate, average latency, packet loss rate, throughput, application response time, disconnection recovery time and latency jitter.

[0013] As a further aspect of the present invention: in step 4, the noise data includes data of transient anomalies caused by temporary equipment failures and signal interference that recover quickly, and the correlation analysis is to perform correlation processing on the index data of different test points and different test stages.

[0014] As a further aspect of the present invention: In step 6, the test-related data fed back to the knowledge base includes test parameters, test cases, collected indicator data, and manual review opinions. The optimization includes updating the test data in the knowledge base and optimizing the association algorithm and anomaly judgment logic of the AI ​​analysis model.

[0015] Compared with the prior art, the beneficial effects of the present invention are: Testing efficiency has been significantly improved, with automated processes shortening the testing cycle by more than 50% and greatly improving the utilization of testing resources. The accuracy of testing has been significantly improved. Through AI-powered intelligent judgment, the matching degree between test results and actual network performance has been enhanced, effectively solving the problem of large discrepancies between traditional test results and user experience. The system's adaptive capabilities have been enhanced, enabling it to automatically adapt to testing requirements in different terminal categories and scenarios, achieving accurate testing across multiple scenarios and product categories; Test resources have been optimized. The dynamic test case generation mechanism based on the iteratively updated RAG knowledge base not only improves test case coverage but also increases test case generation efficiency and reduces redundant tests. The test results are objective and repeatable, eliminating interference from human factors, making the test results highly comparable and repeatable, and providing an objective basis for network quality assessment. The system has the ability to continuously evolve. Through continuous learning, it constantly optimizes testing strategies and judgment models, thereby achieving self-improvement in testing capabilities. Attached Figure Description

[0016] Figure 1 This is a deployment architecture diagram of an automated testing system based on a large AI model, as described in an embodiment of the present invention.

[0017] Figure 2 This is an overall flowchart of the automated testing method based on a large AI model in an embodiment of the present invention.

[0018] Figure 3 This is a flowchart illustrating the operation of an automated testing task based on a large AI model in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0021] Please see Figures 1-3 This invention provides an automated testing system and method based on a large AI model, applicable to automated testing of various types of smart terminals and external network devices. Its core is to achieve intelligent and automated testing across the entire testing chain by constructing a collaborative testing system of "computing power-model-intelligent agent". The technical solution of this invention will be described in detail below with reference to specific implementation methods.

[0022] The automated testing system of this invention includes a management service layer, an AI intelligent decision service layer, a test execution server cluster, the device under test, and external devices under test. The management service layer, as the core control system, is responsible for test material management, test case management, test task distribution and management, test result statistics, and manual review. The AI ​​intelligent decision-making service layer communicates with the test execution server cluster, serving as the core control decision-making layer and judgment layer for the test steps. The test execution server cluster schedules and executes specific test tasks, communicating with the management service layer, the AI ​​intelligent decision-making service layer, and the airborne broadband network control system in the rear cabin of civil aviation passenger aircraft. It controls the test terminals and virtual test terminals in the test equipment cluster. The device under test includes various types of device terminals (mobile phones, computers, cameras, and all-in-one machines, etc.) and virtual test terminals. By simulating user operations in a real network environment, the stability, reliability, bandwidth performance, and related performance tests of the device under test are conducted. External devices under test (APs, smart optical modems, and smart routers, etc.) are directly connected to the device under test, using the signals provided by the device under test to complete the test.

[0023] The testing method based on the above system specifically includes the following steps: Step 1: Test environment configuration and parameter settings.

[0024] Depending on the test type, aircraft model, and current network equipment environment, the system can directly obtain data such as parameters and current network status of the tested equipment, obtain data from the tested equipment management platform, or manually input data. It can also automatically obtain aircraft model parameters (such as rear cabin space dimensions, number of AP deployment points, etc.) and real-time airborne network status data (such as the number of currently connected terminals, signal coverage strength, etc.) from the civil aviation flight data platform via an interface. Combined with historical test data, the system automatically sets key parameters such as test materials, test duration, and test points. The test duration can be set according to the test type (such as 2 hours for scheduled maintenance tests, or the duration of a flight test following a complete flight). Test points can cover the front, middle, and rear of the rear cabin, as well as typical user positions such as windows and aisles. The system also performs manual verification and confirmation to provide an accurate environmental foundation for subsequent tests. This method avoids the subjectivity and inaccuracy of environmental configuration in traditional tests and improves the basic accuracy of the tests.

[0025] Step 2: Dynamically generate test cases.

[0026] Based on RAG (Retrieval Enhanced Generation) technology and the set test environment parameters, the system retrieves the latest test cases, test scenario data, and historical test cases from the knowledge base. The knowledge base also stores multi-scenario test data under different flight altitudes and weather conditions. According to the current test stage (selection, delivery, installation, scheduled maintenance, and flight), the model of the equipment under test, and historical test results, the selection stage focuses on equipment compatibility testing, and the flight stage focuses on stability testing under dynamic environments. The system intelligently matches and generates the optimal combination of test cases to ensure the comprehensiveness and relevance of the test cases. Compared with traditional fixed test cases, this dynamic generation mechanism significantly improves the test case coverage and generation efficiency, and reduces redundant testing.

[0027] Step 3: Automated test execution.

[0028] The system automatically executes reliability tests, stability tests, and bandwidth tests according to the generated test cases and test configuration parameters. Reliability tests can simulate the continuity of network connectivity throughout the entire process of flight takeoff, cruise, and landing. Stability tests can perform long-term (e.g., 4 hours) continuous data transmission tests. Bandwidth tests can simulate concurrent download and upload data tests from multiple terminals. During the test, network performance data is collected in real time, including key indicators such as network connection success rate, average latency, packet loss rate, throughput, application response time, disconnection recovery time, and latency jitter. Throughput is distinguished between uplink and downlink throughput, and application response time includes webpage loading time and video buffering time. This achieves unmanned operation of test execution, reduces the cost of manual intervention, and improves test efficiency.

[0029] Step 4: AI-powered intelligent merging analysis.

[0030] The system utilizes AI models to perform correlation analysis on the collected multi-dimensional test data, automatically identify anomalies, generate preliminary test conclusions, and automatically filter noise data, including anomalies caused by temporary equipment failures or signal interference that result in a sudden increase in packet loss rate but recover within 1 second. Focusing on key performance indicators, and based on a historical anomaly case library, the system makes preliminary judgments on possible causes of anomalies (such as signal obstruction at the location or insufficient AP power), avoiding the subjectivity and limitations of manual analysis and improving the objectivity and accuracy of test results.

[0031] Step 5: Manual review and confirmation.

[0032] Based on the preliminary analysis report generated by AI, professional technicians, combined with actual conditions and usage experience, confirm or correct the test results. During manual review, key indicator trends, anomaly location, and possible cause analysis can be viewed through the system's visualization interface. The visualization interface adopts conventional data chart display formats in the field (such as line charts, bar charts, heat maps, etc.). This two-stage analysis mechanism ensures both the efficiency of the test and the accuracy and reliability of the results.

[0033] Step Six: Test Result Output and Optimization.

[0034] The system automatically generates standardized test reports, including the test environment, test cases, key indicator data and industry standard value comparisons, AI analysis conclusions, manual review comments, and improvement suggestions. Simultaneously, it feeds back the test data to the knowledge base, including test parameters, test cases, collected indicator data, and manual review comments. This feedback is used to optimize the RAG knowledge base and AI model. Specific optimizations include updating the test data in the RAG knowledge base and optimizing the correlation algorithm and anomaly detection logic of the AI ​​analysis model, enabling continuous evolution and optimization of the testing methods. The test reports can be directly used for network optimization decisions, equipment selection, and subsequent test strategy adjustments, ensuring the system can adapt to the testing needs of future 5G-ATG and satellite converged networks and new device models, providing data support for the continuous improvement of related networks.

[0035] Example 1: Testing of Multiple Types of Smart Terminals This embodiment performs automated testing on various types of smart terminals, including mobile phones, computers, cameras, and all-in-one PCs. The specific implementation process is as follows: First, the test environment was configured and parameters were set. The system obtained parameters such as the mobile phone model, computer configuration, camera resolution, and all-in-one system version of the device under test through the device management platform. Combined with the current bandwidth, latency, and other status data of the laboratory network, as well as the test data of similar terminals in the past, the test materials were automatically set to high-definition video, text and image files, real-time data streams, etc. The test duration was set to 2 hours, and the test points covered three areas with strong, medium, and weak network signals. After the technical personnel reviewed and confirmed the environment configuration, the consistency between the test environment and the actual network usage scenario was completed.

[0036] Subsequently, the system retrieves the latest smart terminal reliability test cases, multimodal interaction test scenario data, and historical fault cases from the knowledge base based on RAG technology. According to the current testing stage of "incoming inspection", it intelligently matches and generates the optimal combination of test cases, including continuous power-on stability test, multi-task concurrent test, network switching test, and AI function response test, to ensure coverage of the terminal's core functions and potential risk points.

[0037] During the test execution phase, the test execution server cluster schedules the corresponding test terminals and virtual test terminals, automatically executes the generated test cases, and collects key indicators such as network connection success rate, average data transmission latency, video stuttering and packet loss rate, and AI image recognition response time of each terminal in real time.

[0038] In the AI ​​intelligent merging analysis phase, the AI ​​model performs correlation analysis on the multi-dimensional indicators of each terminal, automatically filters out noise data caused by instantaneous network fluctuations, identifies problems such as abnormal video packet loss rate of a certain mobile phone in areas with weak network signals, and decreased throughput of a certain computer when multiple tasks are running concurrently, and generates a preliminary analysis report.

[0039] During manual review, technicians use the system's visual interface to view the change curves of abnormal indicators and locate the test scenarios. By combining the terminal hardware configuration and network environment characteristics, they confirm the existence of the abnormal issues and correct some of the analysis conclusions.

[0040] Finally, the system generates a standardized test report, clarifying whether the test results of each terminal are qualified. At the same time, the test data is fed back to the knowledge base to optimize the test case generation logic and AI analysis model for subsequent similar terminals.

[0041] This embodiment achieves one-stop automated testing of multiple types of smart terminals through the above solution. The testing cycle is shortened by more than 50% compared with traditional methods, the test case coverage is improved, the matching degree between test results and actual terminal performance is significantly improved, and there is no need to build separate test environments and write adaptation scripts for different terminals, which greatly reduces maintenance costs.

[0042] Example 2: External Network Device Testing This embodiment focuses on automated testing of external devices under test, such as access points (APs), smart optical modems, and smart routers. The specific implementation process is as follows: During the test environment configuration phase, the system manually inputs parameters such as the model, rated bandwidth, and supported network protocols of the external device under test. It obtains the connection status between the device under test and the test terminal, and automatically sets the test materials to large file upload and download packages and real-time voice data streams, based on the current test network topology data. The test duration is set to 4 hours, and the test points are the edge and center areas covered by the signal of the external device. The configuration is completed after verification.

[0043] During the dynamic generation phase of test cases, the system uses RAG technology to retrieve external device bandwidth test cases, long-term operational stability test scenarios, and historical compatibility issue cases. Based on the test phase being "scheduled inspection and maintenance," the system intelligently matches and generates test case combinations that include full-load bandwidth testing, continuous 72-hour operation testing, multi-terminal simultaneous access testing, and different network protocol switching testing.

[0044] During test execution, the test execution server cluster controls the device under test to establish a connection with the external device under test, simulating 20 terminals simultaneously accessing the external device, automatically executing test cases, and collecting indicators such as the maximum throughput of the external device, long-term fault-free operation time, average latency when multiple terminals are accessing, and protocol switching success rate in real time.

[0045] In the AI-powered intelligent merging and analysis phase, the AI ​​model compares and analyzes the collected metrics with the rated parameters of external devices and historical test data. It automatically identifies problems such as a smart router failing to reach its rated throughput under full load or an AP exhibiting abnormal signal attenuation in the edge area, and generates a preliminary report.

[0046] During manual review, technicians consider factors such as the deployment location and transmission distance of external devices to confirm the accuracy of the analysis conclusions and supplement explanations of some environmental influencing factors. Finally, the system generates a test report, providing a basis for quality assessment of external devices and network optimization decisions. Simultaneously, the test data is fed back to the knowledge base to optimize the RAG knowledge base and AI model.

[0047] This embodiment achieves fully automated testing of external network devices through the above-described solution, effectively solving the problems of low testing efficiency and subjective evaluation in traditional testing of such devices. The test results are highly comparable and repeatable, providing an objective basis for network device selection and maintenance. Furthermore, the system has continuous evolution capabilities; as test data accumulates, the test strategies and judgment models are continuously optimized, and the test accuracy is continuously improved.

[0048] Example 3: Performance Test of Airborne Broadband Network Communication in the Rear Cabin of Civil Aviation This embodiment focuses on the automated testing of the communication performance of the in-flight broadband network in the rear cabin of civil aircraft. The testing system, based on the existing architecture, explicitly includes the in-flight broadband network under test. This network is centered around APs (Access Points) distributed across multiple locations in the rear cabin of the civil aircraft. These APs provide the in-flight broadband network access environment and upload their own operational status data (such as current load rate and signal transmission power). The specific implementation process is as follows: (a) Test environment configuration and parameter settings Based on the test type being "flight test," the aircraft type being tested (civilian passenger aircraft), and the current deployment environment of the onboard network equipment, the parameter configuration process is initiated. The system automatically obtains parameters such as the rear cabin space dimensions and the number of AP deployment points for the current aircraft type, as well as real-time status data such as the number of connected terminals and signal coverage strength of the current onboard network, from the civil aviation flight data platform via an interface. For special data that cannot be obtained from the flight data platform, it is supplemented through manual entry.

[0049] Subsequently, the system calls the historical test database, compares and analyzes the real-time data with the test data of the same aircraft model and network environment in the past, and automatically sets the test materials to simulate user access to web page data, high-definition video streams, real-time voice data, etc. The test duration follows the full flight duration, and the test points cover the front, middle, and rear of the rear cabin, as well as typical user positions such as window seats and aisle seats. After being reviewed and confirmed by technical personnel with experience in civil aviation network testing, the environment configuration is completed to ensure that the test environment is highly consistent with the actual use scenario of the flight.

[0050] (ii) Dynamic generation of test cases Based on RAG technology, the system accesses a built-in test knowledge base, which stores test case templates, multi-scenario test data under different flight altitudes and weather conditions, and historical test cases (such as successful installation test cases for similar aircraft models and high-altitude signal anomaly handling cases). The system retrieves, filters, and reorganizes the knowledge base data based on the current testing phase ("flight test") (emphasizing stability testing under dynamic environments), the specific model of the network device under test (such as the manufacturer and model of the rear cabin AP), and historical test results (such as the high-altitude signal switching scenario in historical flight tests where anomalies are prone to occur). It dynamically generates targeted test case combinations, including high-altitude signal switching scenario test cases, multi-terminal concurrent access test cases, and long-term continuous transmission test cases, ensuring coverage of the core functions and potential risk points of civil aviation airborne networks throughout the entire flight process.

[0051] (III) Automated Test Execution The system automatically sends test tasks to the test execution server cluster based on the generated test cases and confirmed test parameters, initiating the automated testing process. The test content covers three core dimensions: First, reliability testing was conducted to simulate the continuity of network connectivity throughout the entire flight's takeoff, cruise, and landing processes, and to statistically analyze the network connection success rate and disconnection recovery time. Second, stability testing involves continuous data transmission for the entire flight duration to monitor the trends in average latency, latency jitter, and packet loss rate. Thirdly, bandwidth testing was conducted, simulating 50 terminals simultaneously downloading and uploading data to test the network's uplink throughput, downlink throughput, and application response time (such as webpage loading time and video buffering time).

[0052] During the test, the test execution server cluster controls the test equipment cluster (including various types of physical test terminals such as mobile phones, tablets and laptops of different brands and systems, as well as virtual test terminals generated through virtualization technology), simulating user operations in a real network environment, collecting the above-mentioned multi-dimensional network performance data and AP operating status data (such as load rate and signal transmission power) in real time, and uploading the data to the data storage module in real time, without the need for manual intervention throughout the process.

[0053] (iv) AI-powered intelligent merging analysis The system invokes a pre-trained AI analysis model to intelligently merge and analyze the collected multi-dimensional test data. The AI ​​model first preprocesses the data, filtering out noise data caused by temporary equipment malfunctions and signal interference (such as abnormal data like a sudden spike in packet loss rate followed by recovery within 1 second), focusing on key performance indicators such as network connection success rate, average latency, packet loss rate, throughput, application response time, disconnection recovery time, and latency jitter. Then, through correlation analysis algorithms, it associates indicator data from different test locations and different test phases (takeoff, cruise, and landing), automatically identifying anomalies (such as a packet loss rate consistently exceeding 5% at a certain point during cruise, or a sudden increase in latency exceeding 500ms during high-altitude signal switching). Finally, based on a historical anomaly case library, it preliminarily determines possible causes of the anomalies (such as signal obstruction at the location or insufficient AP power), generating a preliminary test conclusion report, thus avoiding the subjectivity and limitations of manual analysis.

[0054] (v) Manual review and confirmation After the initial test conclusions are generated, they are manually reviewed by professional technicians with experience in civil aviation network testing. The system provides a data visualization interface, displaying the changing trends of key indicators (such as the overall packet loss rate curve), the specific locations of anomalies (such as the window position at the rear of the cabin), and the possible causes identified by the AI ​​model in the form of line graphs, heat maps, etc. The technicians combine their own experience in civil aviation network testing (such as whether there have been historical signal weakness issues at this location) and the actual testing environment (such as whether there are metal obstacles obstructing the signal in the rear cabin during the test) to confirm or revise the initial conclusions. For example, if the AI ​​determines that packet loss at a certain location is due to insufficient AP power, and the technicians find on-site that temporarily placed luggage is obstructing the signal, then the cause of the anomaly is corrected and the conclusion is updated.

[0055] (vi) Test result output and optimization Upon completion of the review, the system automatically generates a standardized test report. The report includes the test environment (aircraft type, network equipment model, test flight information), a list of test cases (test objectives and execution steps for each case), key performance indicator (KPI) data and industry standard value comparisons, AI analysis conclusions (anomaly list and preliminary causes), manual review comments (corrected conclusions), and improvement suggestions (such as adjusting AP deployment locations and optimizing signal transmission power). This report can be directly used for airborne network optimization decisions, equipment selection, and subsequent test strategy adjustments. Simultaneously, the system feeds all test data (including test parameters, test cases, collected KPI data, AP operating status data, and manual review comments) back to the test knowledge base. This updates the RAG knowledge base's test data, optimizes the AI ​​analysis model's correlation algorithm and anomaly detection logic, and enables continuous evolution and optimization of the test methods. This ensures the system can adapt to the testing needs of future 5G-ATG and satellite converged networks and new aircraft models.

[0056] This embodiment achieves fully automated testing of the communication performance of airborne broadband networks in the rear cabin of civil aviation through the above-mentioned scheme. The testing cycle is shortened by more than 50% compared with traditional manual testing. The matching degree between the test results and the network performance under actual flight scenarios is significantly improved. It effectively solves the problems of traditional testing relying on manual monitoring, single parameter detection, and large result deviation. The test results have high comparability and repeatability, providing objective and accurate data support for the high-quality construction and operation and maintenance of airborne broadband networks in the rear cabin of civil aviation.

[0057] It should be noted that, in this invention, although the specification describes the embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An automated testing system based on a large AI model, characterized in that, It includes a management service layer, an AI intelligent decision service layer, a test execution server cluster, the device under test, and external devices under test; The management service layer, as the core control system, communicates with the AI ​​intelligent decision service layer and the test execution server cluster, and is responsible for test material management, test case management, test task distribution and management, test result statistics and manual review. The AI ​​intelligent decision service layer communicates with the test execution server cluster and serves as the core control decision layer for test steps and the judgment layer for the test process. It is used to dynamically generate test cases based on RAG technology and to perform intelligent analysis of test data. The test execution server cluster is used to schedule and execute specific test tasks and control the device under test and external devices under test to complete the test. The device under test includes multiple types of physical test terminals and virtual test terminals, which are used to simulate user operations in a real network environment for performance testing. The external device under test is connected to the device under test, and the test is completed using the signals provided by the device under test. The components work together to build a collaborative testing system of computing power, model, and intelligent agent, realizing intelligent and automated testing across the entire process, and the test data can be fed back to the system for continuous optimization.

2. The automated testing system based on AI large-scale models according to claim 1, characterized in that, The management service layer is also used to receive test environment parameter data, which includes test environment parameter data, network status data, and scenario-related parameters obtained directly from the device under test, obtained from the device under test management platform, manually entered, or obtained from an external data platform through an interface.

3. The automated testing system based on a large AI model according to claim 1, characterized in that, The AI ​​intelligent decision service layer has a built-in knowledge base that stores test cases, test scenario data, historical test cases, and multi-scenario test-related data, which provides data support for dynamic generation of test cases and AI intelligent analysis.

4. The automated testing system based on a large AI model according to claim 1, characterized in that, The management service layer provides a visual interface, which is used to display the changing trends of key indicators, locate anomalies and analyze possible causes, and supports manual review.

5. An automated testing method based on a large AI model, applied to the automated testing system based on a large AI model as described in any one of claims 1-4, characterized in that, Includes the following steps: Step 1: Test environment configuration and parameter setting. Based on the test type, the device under test and the network environment, relevant parameter data are obtained and key test parameters are automatically set, which are then manually reviewed and confirmed. Step 2: Dynamic generation of test cases. Based on RAG technology, knowledge base data is retrieved, and the optimal combination of test cases is intelligently matched and generated by combining the test stage, the model of the device under test, and historical test results. Step 3: Automated test execution. According to the test cases and configuration parameters, automatically execute the tests and collect multi-dimensional key performance indicators in real time; Step 4: AI intelligent merging analysis. Using AI models, the collected test data is correlated, noisy data is filtered, outliers are identified, and the causes of anomalies are preliminarily determined to generate preliminary test conclusions. Step 5: Manual review and confirmation. Professional technicians confirm or correct the test results based on the preliminary test conclusions and the actual situation. Step 6: Test results output and optimization. A standardized test report is automatically generated, and relevant test data is fed back to the knowledge base to optimize the system and AI model for continuous evolution.

6. The automated testing method based on a large AI model according to claim 5, characterized in that, In step 1, the parameter data includes the parameters of the device under test, network status data, and scenario-related parameters. The key test parameters include test materials, test duration, and test locations.

7. The automated testing method based on a large AI model according to claim 5, characterized in that, In step 2, the testing phase includes selection, delivery, installation, scheduled maintenance and flight phases, with different test case matching logics emphasizing different aspects for each testing phase.

8. The automated testing method based on a large AI model according to claim 5, characterized in that, In step 3, the tests include reliability testing, stability testing, and bandwidth testing. The key performance indicators include network connection success rate, average latency, packet loss rate, throughput, application response time, disconnection recovery time, and latency jitter.

9. The automated testing method based on a large AI model according to claim 5, characterized in that, In step 4, the noise data includes data of transient anomalies caused by temporary equipment failures and signal interference that recover quickly, and the correlation analysis is to perform correlation processing on the indicator data of different test points and different test stages.

10. The automated testing method based on a large AI model according to claim 5, characterized in that, In step 6, the test-related data fed back to the knowledge base includes test parameters, test cases, collected indicator data, and manual review opinions. The optimization includes updating the test data in the knowledge base and optimizing the association algorithm and anomaly judgment logic of the AI ​​analysis model.

Images