An automated testing method, system, storage medium, and program product

Abstract

Embodiments of the present application provide an automatic testing method, system, storage medium and program product, the method comprising: obtaining an upgrade package of a to-be-tested device; controlling the to-be-tested device to upgrade based on the upgrade package; inputting a test case into a generation model to obtain a corresponding test script; sending the test script to a test execution device to control the test execution device to execute an automatic testing task and generate a test log; inputting the test log into an analysis model to obtain analysis information including a test result; and performing a corresponding management operation based on the analysis information. Through the cloud platform, the whole process of device upgrading, intelligent script generation, test execution and result analysis is integrated, and a complete closed loop of automatic testing is realized.

Description

Technical Field

[0001] This application relates to the field of vehicle testing technology, and more specifically, to an automated testing method, system, storage medium, and program product. Background Technology

[0002] Automated testing of smart cockpits often focuses on the test execution phase itself, such as triggering test processes through pre-defined controls or performing UI testing by parsing page structures. However, these methods fail to automate the entire process from device preparation and script generation to result analysis. The development and maintenance of test scripts, the upgrading of test equipment, and the analysis and defect management of massive test logs still heavily rely on manual processing, resulting in low testing efficiency. Summary of the Invention

[0003] The purpose of this application is to provide an automated testing method, system, storage medium, and program product to achieve the technical effect of improving testing efficiency.

[0004] The first aspect of this application provides an automated testing method, which is applied to a cloud platform; the method includes: Obtain the upgrade package for the device under test; Control the device under test to upgrade based on the upgrade package; Input the test cases into the generation model to obtain the corresponding test scripts; The test script is sent to the test execution device, which is then controlled to execute automated test tasks and generate test logs. The test logs are input into the analysis model to obtain analysis information including the test results; Based on the analyzed information, corresponding management operations are performed.

[0005] In the aforementioned implementation process, a complete closed loop of automated testing was achieved by integrating the entire process of equipment upgrades, intelligent script generation, test execution, and result analysis through a cloud platform. Specifically, test scripts are automatically created using generation models, replacing traditional manual development and maintenance; logs are automatically parsed and decisions are made using analysis models, replacing manual result analysis and defect management; this significantly reduces reliance on manual labor at each stage, improving testing efficiency while lowering labor costs and the risk of human error.

[0006] Furthermore, the step of controlling the device under test to upgrade based on the upgrade package includes: Send the upgrade package to the device under test; Obtain the current software version of the device under test; If the current software version is inconsistent with the target version of the upgrade package, the device under test is controlled to perform an upgrade operation.

[0007] In the above implementation process, by automating the sending-comparison-triggering process, on-demand upgrades and status management of the software version of the device under test are achieved. This avoids the efficiency bottlenecks and operational errors of traditional manual offline upgrades via USB flash drives and other methods, ensuring that the device can automatically and accurately reach the target test version before testing, thus laying a reliable foundation for subsequent fully automated testing.

[0008] Furthermore, the generated model is a large-scale language model trained based on preset data associated with the test script; the preset data includes at least one of the following: test tool operation manual, script set, or functional specification.

[0009] In the above implementation process, the generated model is trained based on domain-specific pre-set materials (operation manuals, script sets, functional specifications), thus ensuring that the generated automated test scripts are not only syntactically correct but also conform to internal tool specifications and business logic, improving the accuracy of the scripts. This fundamentally solves the problem of over-reliance on the individual abilities of engineers for manually written scripts.

[0010] Furthermore, the step of performing corresponding management operations based on the analyzed information includes: If the test result is determined to be a failure based on the analysis information, and the reason for the failure is a valid issue, then an issue record associated with the test case is created. If the test result is determined to be a failure based on the analysis information, and if the cause of failure is not an effective problem, then if the cause of failure is determined to originate from the test script, the test script shall be adjusted.

[0011] In the above implementation process, by distinguishing between "valid problems" and "script defects", the fuzzy judgment process that traditionally relied entirely on manual analysis is transformed into a precise and automated process based on rules and models.

[0012] Furthermore, the adjustment of the test script includes: If the number of adjustments to the test script does not exceed a preset threshold, then the test script is adjusted through the generation model; If the number of adjustments to the test script exceeds the preset threshold, a prompt message indicating that manual intervention is required will be generated.

[0013] In the above implementation process, when the automatic repair cannot solve the problem within a limited number of attempts, the problem is stopped in time and handed over to manual processing, thus avoiding infinite loops and waste of resources caused by the limitations of the model's capabilities or the complexity of the problem.

[0014] Furthermore, the test cases are generated through the following steps: Obtain the test requirements document; The test requirement document is input into the generation model to generate the test cases corresponding to the test requirement document.

[0015] In the above implementation process, by reusing the ability to generate models, the transformation from requirements to scripts was realized, which changed the slow process of test cases relying on manual writing and review, improved the efficiency of test design, and reduced the omission or error of test cases caused by human misunderstanding.

[0016] A second aspect of this application provides an automated testing system, the system comprising: The cloud platform is configured as follows: Obtain the upgrade package for the device under test; Control the device under test to upgrade based on the upgrade package; Input the test cases into the generation model to obtain the corresponding test scripts; Send the test script to the test execution device; Input the test logs into the analysis model to obtain analysis information including the test results; Perform corresponding management operations based on the analyzed information; The host computer, acting as the test execution device, is communicatively connected to the cloud platform and is configured as follows: Receive test scripts from the cloud platform; Execute the automated test tasks indicated by the test script and generate the test logs; The device under test, which is communicatively connected to the cloud platform and the host computer, is configured as follows: Upgrades are performed in response to controls from the cloud platform. The test operation is executed in response to the instructions from the host computer.

[0017] In the above implementation process, by utilizing the scheduling and decision-making capabilities of the cloud platform, the execution and control capabilities of the host computer, and the response capabilities of the device under test, an end-to-end unmanned testing system was constructed, enabling the complete closed loop from device upgrade, script generation, test execution to result analysis to run autonomously, reliably, and efficiently in an unattended environment.

[0018] Furthermore, the cloud platform is also configured as follows: Send the upgrade package to the device under test; Obtain the current software version of the device under test, and if it is determined that the current software version is inconsistent with the target version of the upgrade package, control the device under test to perform an upgrade operation.

[0019] In the above implementation process, the cloud platform ensures that the upgrade operation is only triggered when the device is truly necessary (i.e., the versions are inconsistent), thereby eliminating invalid duplicate upgrades and resource waste.

[0020] A third aspect of this application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described in the first aspect.

[0021] A fourth aspect of this application provides a computer program product, the computer program product including a computer program, which, when executed by a processor, implements any of the methods described in the first aspect. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating an automated testing method provided in an embodiment of this application; Figure 2 An overall flowchart provided for an embodiment of this application; Figure 3 A flowchart illustrating OTA package deployment and device upgrade is provided in this application embodiment; Figure 4 A flowchart for test case distribution and automated test script generation provided in this application embodiment; Figure 5 A flowchart of an automated test task execution provided in this application embodiment; Figure 6 A flowchart for test log report analysis and bug management provided in this application embodiment; Figure 7 This is a schematic diagram of an overall device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0025] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] In related technologies, the commonly used methods include: developing in-vehicle test applications within the vehicle's infotainment system and configuring pre-set functional controls, automatically executing the corresponding vehicle control test process by triggering the controls, and generating test logs and test reports by the log module and report module respectively; or, after receiving UI automation test requests, performing automated testing by parsing the page XML tree of the test object and outputting assertion results.

[0027] However, the aforementioned technologies primarily focus on automating the test execution phase, failing to develop an automated solution covering the entire testing process. In actual testing, aspects such as the preparation and upgrading of the equipment under test, the generation of automated test scripts, and the analysis and defect (bug) management after test case execution failures still rely heavily on manual intervention, which is time-consuming and labor-intensive, making it difficult to achieve a truly efficient closed loop.

[0028] Specifically, the relevant technology has the following drawbacks: The development and maintenance of automated test scripts rely on manual labor, which is inefficient and costly. The analysis of test results and defect management processes involve too much manual intervention and lack sufficient automation. Equipment upgrades typically require offline methods such as USB drives, making seamless integration with automated testing processes impossible.

[0029] The causes of the above defects include: There is a lack of intelligent generation mechanisms for automated test scripts, relying instead on the personal experience of engineers; The test log data is massive and lacks semantic processing, making it difficult to achieve automated and accurate analysis. The upgrade process for the equipment under test was not integrated into the automated testing process, resulting in process interruption and decreased efficiency.

[0030] To address any of the aforementioned shortcomings, this application aims to achieve a complete closed loop from OTA deployment and test case upload to test report generation, thereby improving testing efficiency and shortening the testing cycle. Specifically, this application provides an integrated, unmanned, automated laboratory management testing system. This system, centered on a cloud platform, collaborates with a host computer, the device under test, and an unmanned management room to construct a highly efficient testing environment. Based on this, this application designs and develops several tools: first, an automated test script generation tool that intelligently converts test cases into executable test scripts; second, a test log analysis and bug management tool that performs automated semantic analysis of test results and generates structured reports; and third, an OTA deployment and device upgrade tool that enables automatic detection and upgrade of the software version of the device under test, ensuring that it meets the testing prerequisites.

[0031] Through the above design, this application replaces the traditional model of relying on manual development and maintenance of test scripts, reducing labor costs and improving script processing efficiency. At the same time, this application changes the status quo of manual log analysis and bug management, improves the automation and accuracy of problem handling, and effectively reduces human error and omissions. In addition, this application further improves the continuity and efficiency of the overall testing process by seamlessly integrating the equipment upgrade process into the automated testing process.

[0032] Based on this, the embodiments of this application provide an automated testing method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an automated testing method provided in an embodiment of this application.

[0033] In this embodiment, the method is applied to a cloud platform; the method includes: Step S10: Obtain the upgrade package for the device under test; It should be noted that the equipment to be tested refers to the automotive intelligent cockpit hardware that needs to be automated tested, such as cockpit test benches, body test benches, or complete vehicles.

[0034] Upgrade package: also known as OTA (Over-the-Air) upgrade package, refers to a software update package pushed through the cloud, used to upgrade the device to the target test version.

[0035] Specifically, when a test version is available, the test engineer deploys the OTA package for the test version on the cloud platform and specifies the device to be tested and the push strategy (such as the maximum number of retries and the retry interval). The cloud platform automatically obtains the upgrade package and prepares to enter the device upgrade process.

[0036] Step S20: Control the device under test to upgrade based on the upgrade package; Step S30: Input the test cases into the generation model to obtain the corresponding test scripts; It should be noted that the test cases are described in a structured format (such as .excel) as the test scenarios, steps, and expected results; the test cases have been reviewed and approved.

[0037] The generative model is an automated script generation tool based on a large language model (LLM) trained on.

[0038] Test script: A sequence of test instructions that can be executed automatically, based on self-developed testing tools.

[0039] Specifically, test engineers upload the reviewed and approved test cases to the cloud platform. The cloud platform automatically selects the device to be tested based on the test case type (e.g., functional, flashing, response) and the status of the host computer (offline, idle, busy). The generated model generates corresponding automated test scripts based on the uploaded test cases and training data, and associates them with the test cases.

[0040] Step S40: Send the test script to the test execution device, and control the test execution device to execute automated test tasks and generate test logs; It should be noted that the test execution device, also known as the host computer, refers to the device that has self-developed test tool client, experiment management software, etc. installed, and is used to execute test scripts and control the device under test.

[0041] Automated test tasks: A series of test operations defined by test scripts.

[0042] Test log: A log file generated during the execution of the test, including operation records, result output, error messages, etc.

[0043] Specifically, test engineers select test case sets and create test tasks on the cloud platform. The cloud platform automatically distributes the associated automated test scripts to the designated host computer. After receiving the scripts, the host computer automatically deploys them to the test environment. The test execution engine executes the test tasks according to preset scheduling strategies (such as maximum number of retries, retry interval, execution mode, etc.), collects logs and execution results in real time, and generates a preliminary test report.

[0044] Step S50: Input the test log into the analysis model to obtain analysis information including the test results; It should be noted that the analysis model refers to a test log analysis tool trained based on LLM technology, which is capable of semantic understanding and failure cause analysis of logs.

[0045] Analysis information includes not only pass / fail results, but also structured judgment information such as failure reason classification, whether it is a valid problem, and whether it is due to a script error.

[0046] Specifically, after the test task is completed, the system automatically retrieves the test log report and inputs it into the analysis model. The analysis model performs semantic analysis on the logs to identify the root cause of the test failure and determine whether the failure represents a valid bug (i.e., a software defect) or is caused by a problem with the test script itself or an environmental issue.

[0047] Step S60: Execute the corresponding management operation based on the analysis information.

[0048] It should be noted that management operations refer to subsequent process actions triggered based on the analysis results, aiming to achieve closed-loop management of the testing process.

[0049] The type of management operation is related to the analysis information; for example, different failure reasons will trigger different operations.

[0050] As an example, the system automatically performs corresponding management operations based on the analysis information output by the analysis model: If it is a valid bug: the system automatically creates a bug report (proposal) on the bug management platform based on preset proposal principles (such as bug level, description, expected result, etc.) and associates the bug link with the corresponding test case; If it is not a valid bug and originates from a script error: The system checks if the preset maximum number of times the automated script has been adjusted has been exceeded. If not, it uses LLM technology to automatically adjust erroneous automated test scripts. If the maximum number of times has been exceeded, the system notifies the test engineer to manually adjust the scripts. After adjusting the scripts, the system re-executes the tests and repeats the analysis process.

[0051] If the error is neither a valid bug nor a script error: the system will automatically add the reason for the failure to the test report and update the test result status.

[0052] In this embodiment, a complete closed loop of automated testing is achieved by integrating the entire process of equipment upgrades, intelligent script generation, test execution, and result analysis through a cloud platform. Specifically, test scripts are automatically created using a generation model, replacing traditional manual development and maintenance; logs are automatically parsed and decisions are made using an analysis model, replacing manual result analysis and defect management; this significantly reduces reliance on manual labor at each stage, improving testing efficiency while lowering labor costs and the risk of human error.

[0053] Based on any of the above embodiments, controlling the device under test to upgrade based on the upgrade package includes: Send the upgrade package to the device under test; Obtain the current software version of the device under test; If the current software version is inconsistent with the target version of the upgrade package, the device under test is controlled to perform an upgrade operation.

[0054] The current software version refers to the version number of the internal software system (such as the in-vehicle infotainment system, domain controller software, etc.) that the device under test was actually running before the upgrade operation.

[0055] Target version: This refers to the software version that the device is expected to reach during this test, i.e., the version number embedded in the upgrade package. If the current software version differs from the target version number, it indicates that the device needs to be updated.

[0056] Specifically, after the test engineer deploys the OTA package for the test version on the cloud platform, the cloud platform automatically initiates a push command according to the set push strategy (such as network priority and batching strategy), streaming the upgrade package file to the target device under test. The OTA client on the device under test is responsible for receiving and temporarily storing the upgrade package. After the upgrade process is triggered, the cloud platform sends a version query command to the device under test via a device management protocol (such as MQTT or HTTP). The agent program on the device under test responds to this command, sending its current software version number (e.g., "V2.1.0") back to the cloud platform. The cloud platform compares the obtained current software version with the target version preset in the upgrade package. If they match, it means the device is already at the target version and no upgrade is needed. The cloud platform records that the version is ready and triggers the subsequent test process. If they do not match, it means the device needs to be upgraded. The cloud platform then sends an upgrade command to the device under test. After receiving the command, the device starts its internal security verification and installation program to flash the received upgrade package. The system monitors the upgrade result. If the upgrade fails (e.g., verification fails, flashing is interrupted), and the number of retries has not exceeded the maximum number of retries, the cloud platform will take control of the device to perform the upgrade operation again. If the upgrade still fails after exceeding the maximum number of retries, the process will be terminated and the engineer will be notified.

[0057] In this embodiment, by automating the send-compare-trigger process, on-demand upgrades and status management of the software version of the device under test are achieved. This avoids the efficiency bottlenecks and operational errors of traditional manual offline upgrades via USB flash drives, ensuring that the device can automatically and accurately reach the target test version before testing, thus laying a reliable foundation for subsequent fully automated testing.

[0058] Based on any of the above embodiments, the generated model is a large language model trained on preset data associated with the test script; the preset data includes at least one of test tool operation manuals, script sets, or functional specifications.

[0059] It should be noted that the generative model is a large language model used to transform test cases described in natural language into executable test scripts. It is a pre-trained model based on the Transformer architecture, which, through domain-specific fine-tuning, has the ability to understand test requirements and generate structured code.

[0060] Pre-set data refers to a collection of domain-specific data used for fine-tuning or training the generative model. This data is logically and content-wise strongly correlated with the final test script to be generated, ensuring that the model learns the correct domain knowledge and coding standards.

[0061] Specifically, the preset data includes: Test Tool Operation Manual: This refers to the operation manual for the self-developed test tools. It is a document guiding users on how to use internal test tools (such as instruction sets, APIs, and clients), and forms the basis for the model to learn how to operate them. Script set: This refers to the corresponding automated test script set. It's a historically accumulated, engineer-written, and validated library of correct test scripts. It provides high-quality code examples for the model; Functional specifications: Documents that describe the expected behavior, interface definitions, and business logic of the software to be tested (such as the functional modules of a smart cockpit). This helps the model understand what to test, thereby generating logically correct verification points.

[0062] As an example, when a test engineer uploads a test case (e.g., "Verify that the voice assistant's response time to navigation commands in driving mode is less than 2 seconds") to the cloud platform, the cloud platform preprocesses the test case, describing it in natural language, and formats it into prompts agreed upon by the model. The cloud platform then calls the deployed generative model and inputs the formatted test case. Based on knowledge learned from pre-set materials (according to the "Test Tool Operation Manual," it knows which API to call to simulate voice input and timing; according to the "Functional Specification," it knows how to set the system state corresponding to "driving mode"; and according to the historical script set, it learns the structure and assertion syntax of similar response time test scripts), the generative model outputs a complete, syntactically correct, and directly executable automated test script code that can be directly executed by the self-developed testing tool, and automatically associates it with the test case.

[0063] In this embodiment, the generative model is trained based on domain-specific pre-defined materials (operation manuals, script sets, functional specifications), thus ensuring that the generated automated test scripts are not only syntactically correct but also conform to internal tool specifications and business logic, improving script accuracy. This fundamentally solves the problem of excessive reliance on individual engineer skills for manually written scripts.

[0064] Based on any of the above embodiments, the step of performing the corresponding management operation based on the analysis information includes: If the test result is determined to be a failure based on the analysis information, and the reason for the failure is a valid issue, then an issue record associated with the test case is created. If the test result is determined to be a failure based on the analysis information, and if the cause of failure is not an effective problem, then if the cause of failure is determined to originate from the test script, the test script shall be adjusted.

[0065] It should be noted that the analysis information refers to the structured conclusions output by the analysis model after semantic processing of the test logs. Based on the conclusion fields in the analysis information, if it is determined that the test execution did not achieve the expected results, then the test result is considered a failure.

[0066] A valid problem refers to a failure determined by the analysis model to be caused by a defect inherent in the software under test (such as a smart cockpit system), rather than by the testing environment, test scripts, or random factors. A valid problem is a software issue that requires tracking and remediation.

[0067] Create an issue record: Automatically create a new work order in the issue management or defect tracking system. This issue record links back to the original test case that triggered the issue, establishing a traceability relationship.

[0068] Invalid problem: refers to a failure of the analysis model not caused by a defect in the software under test.

[0069] The failure was attributed to issues such as logical errors, inaccurate instructions, or incompatibility with the current software version, as diagnosed by the analysis model.

[0070] As an example, once the test task is completed, the analysis model outputs analysis information, and the cloud platform initiates closed-loop management based on this information: Scenario A: Discovering a valid software defect: Analysis of the information shows that the test failed because "the map rendering module exhibited screen flickering in night mode" and was classified as a "valid bug". The cloud platform executes automatically: it calls the API of the issue management platform to create an issue record according to a preset template. The title is automatically generated as "[Automatic Order Submission] Map Rendering - Night Mode Screen Flickering". The content includes a summary of the failure log, the test case ID (for association), and a comparison of the expected and actual results. Result: A work order for an issue to be addressed by R&D was created and assigned, and the test engineer did not need to manually copy and paste any information.

[0071] Scenario B: Discovering a defect in the test script itself: Analysis of the information shows that the test failed because "the ID of the control clicked in the script does not exist in the current version", and it was classified as "invalid issue - script error". The cloud platform automatically executes the "script adjustment" process. The system sends the failed analysis information and the original script to the generated model, requesting correction. The generated model outputs the corrected script (e.g., updating the control ID to the correct ID of the new version). Result: The defective test script was automatically corrected, preparing for subsequent re-execution of the test.

[0072] In this embodiment, by distinguishing between "valid issues" and "script defects," the traditionally fuzzy judgment process that relies entirely on manual analysis is transformed into a precise and automated process based on rules and models.

[0073] Based on any of the above embodiments, adjusting the test script includes: If the number of adjustments to the test script does not exceed a preset threshold, then the test script is adjusted through the generation model; If the number of adjustments to the test script exceeds the preset threshold, a prompt message indicating that manual intervention is required will be generated.

[0074] It should be noted that the number of adjustments refers to the cumulative number of times the same test script is automatically adjusted within a single test task cycle.

[0075] The preset threshold is an upper limit on the number of times set to prevent getting stuck in an infinite adjustment loop; for example, the preset threshold is 3 times.

[0076] In practice, the generation model is invoked, and the original script is corrected, optimized, or rewritten based on the failure cause analysis results.

[0077] The number of adjustments exceeds the preset threshold, meaning that the script has been adjusted multiple times but still fails to solve the problem.

[0078] The prompt information includes any one or more of the following: the identifier of the script to be processed, the summary of the failure analysis, the historical adjustment record, and the recommended manual operation suggestions.

[0079] As an example, when entering the script adjustment phase, the following adaptive repair process is initiated: (1) Scenarios where the number of adjustments does not exceed the threshold: A test script failed to execute on its first run, and the analysis model determined that the script was faulty. The script was checked and found to have 0 adjustments in this round of testing, with a preset threshold of 3 (0 < 3), which meets the automatic adjustment conditions. The cloud platform sends the failure logs, original scripts, and analysis information to the generated model, requesting corrections. The generated model returns the corrected script V1; Re-execute script V1. If it succeeds, the process ends. If it fails again and is still judged as a script error, the number of attempts is increased by 1 (becoming 1), and the process proceeds to the next round of judgment.

[0080] (2) Scenarios where the number of adjustments exceeds the threshold: A script failed to execute after three consecutive automatic adjustments (adjustment count = 3). When the adjustment is required for the fourth time, the inspection finds that the number of adjustments (3) has reached the preset threshold (3), triggering the manual transfer mechanism; The cloud platform automatically generated and sent a notification message to the designated test engineer. The message read: "[Manual intervention required] Script [TC_2024_Speech_001] has reached its automatic adjustment limit (3 attempts). Failure summary: Control positioning failed, suspected UI structure change. Historical adjustment record: Attempts have been made to update control ID, use XPath positioning, and add a wait delay. Please manually check and update the script." After receiving the notification, the engineer manually analyzed and repaired the script, and then resubmitted the repaired script to the system for execution.

[0081] In this embodiment, when the automatic repair cannot solve the problem within a limited number of attempts, the problem is stopped in time and handed over to manual processing, thus avoiding infinite loops and waste of resources caused by the limitations of the model's capabilities or the complexity of the problem.

[0082] Based on any of the above embodiments, the test cases are generated through the following steps: Obtain the test requirements document; The test requirement document is input into the generation model to generate the test cases corresponding to the test requirement document.

[0083] It should be noted that a test requirements document is a technical document that describes the expected behavior, functionalities, performance metrics, and business logic of the software under test (such as new features in a smart cockpit). Test requirements documents can take the form of a Product Requirements Document (PRD), Functional Specification Document (FSD), or design document, etc.

[0084] As an example, the aforementioned documents are entered into the cloud platform system through methods such as uploading and interface synchronization, making them data objects that can be processed by the model.

[0085] It should be understood that the generative model is also used to understand the requirements of natural language descriptions and output them in accordance with the specification format of test cases.

[0086] The generated model outputs structured, executable test cases based on the input document content. A complete test case includes fields such as test case number, title, test function, preconditions, test steps, and expected results.

[0087] As an example, a functional requirements document (PDF / Word format) regarding "multi-area voice interaction in an intelligent cockpit" is uploaded to a specific module on the cloud platform. The cloud platform parses the document (e.g., using OCR and text extraction) and converts it into plain text. Then, the system constructs a prompt, such as: "Please generate detailed functional test cases based on the following product requirements document. Each test case must include: test case title, test function point, preconditions, detailed execution steps, and expected results." The parsed requirements text is then attached and input into the generation model. Based on its knowledge learned from massive amounts of technical documents and test cases, the generation model performs the following tasks: Understanding the needs: Identify the key functionalities in the needs, such as "driver's voice wake-up", "passenger's voice command rejection", and "independent volume adjustment for rear seats"; Structured Generation: For each identified functional point, one or more test cases conforming to standard templates are generated. For example, for "passenger voice command rejection", the model generates a test case titled "Verify that non-driver passengers cannot execute core vehicle control commands via voice", and lists the steps in detail: "1. The vehicle is in motion; 2. The passenger in the front seat says the command 'Open the window'; 3. Observe the system response...", and the expected result: "The system should provide a voice prompt 'This function is not currently supported' or not respond."

[0088] In addition, the generated test cases enter the "online review" stage. After the review is passed, the generated model further converts them into test scripts.

[0089] In this embodiment, by reusing the ability to generate models, the transformation from requirements to scripts is realized, which changes the slow process of test cases relying on manual writing and review, improves the efficiency of test design, and reduces test case omissions or errors caused by human misunderstanding.

[0090] Furthermore, embodiments of this application also provide an automated testing system, the system comprising: The cloud platform is configured as follows: Obtain the upgrade package for the device under test; Control the device under test to upgrade based on the upgrade package; Input the test cases into the generation model to obtain the corresponding test scripts; Send the test script to the test execution device; Input the test logs into the analysis model to obtain analysis information including the test results; Perform corresponding management operations based on the analyzed information; The host computer, acting as the test execution device, is communicatively connected to the cloud platform and is configured as follows: Receive test scripts from the cloud platform; Execute the automated test tasks indicated by the test script and generate the test logs; The device under test, which is communicatively connected to the cloud platform and the host computer, is configured as follows: Upgrades are performed in response to controls from the cloud platform. The test operation is executed in response to the instructions from the host computer.

[0091] It should be noted that the cloud platform, acting as a "command center," is responsible for the scheduling, decision-making, and data management of the entire process.

[0092] Specifically, the cloud platform provides an interface that allows test engineers to configure OTA packages for the test versions. The cloud platform automatically pushes the OTA packages to the designated devices under test for upgrades. The cloud platform integrates or calls generation models to convert uploaded test cases into scripts. The cloud platform schedules tasks and distributes scripts based on device status. Simultaneously, the cloud platform receives logs from the host computer, calls analysis models for analysis, and automatically executes operations such as creating issue tickets or triggering script adjustments based on the analysis results.

[0093] The host computer is responsible for connecting to and controlling the device under test at the physical layer and executing specific test actions.

[0094] Specifically, the host computer communicates with the cloud platform via a local area network or dedicated network, receiving instructions and reporting status. The host computer pulls or receives automated test script files pushed by the cloud platform. The host computer is equipped with self-developed test tool clients, experiment management software, etc., parses and runs the scripts, controls the device under test via hardwired connections (such as CAN, LIN, Ethernet), and simultaneously collects all operation and feedback information to form a log.

[0095] The device under test (DUT) is the physical hardware of the intelligent cockpit. The DUT has a network interface to connect to the cloud platform to receive upgrade packages, and a hardwired interface to connect to a host computer to receive test commands.

[0096] Specifically, the OTA client of the device under test receives and verifies the upgrade package issued by the cloud platform and performs the flashing operation. At the same time, the device under test receives simulated user commands (such as touch screen clicks, voice injection, CAN signal simulation, etc.) sent by the host computer via hardwire and responds to them, providing the host computer with the results.

[0097] In its implementation, the automated testing system performs the following operations: Initialization: A device under test (vehicle unit 1) and its corresponding host computer (host computer 1) are connected via hardwire and placed in an unmanned room managed by category. Both are connected to the network and have completed registration with the cloud platform; Upgrade Phase Operation: Version 1 is submitted for testing. The test engineer deploys the OTA package on the cloud platform. The cloud platform pushes the package to Vehicle Infotainment System 1 and controls its upgrade. After completing the upgrade, Vehicle Infotainment System 1 reports a successful status. The test preparation and execution phase involves the engineer uploading the test case list to the cloud platform. The cloud platform generates a model to obtain the script, which is then sent to host computer 1. Host computer 1 executes the script, controlling the vehicle's infotainment system 1 via hardwired connections to complete a series of tests (such as clicking the screen and playing sound effects), and simultaneously collects logs. Analysis and closed-loop operation: The host computer 1 sends the logs back to the cloud platform. The cloud platform calls the analysis model to obtain the analysis report. If a valid bug is found, the cloud platform automatically submits a request for help in systems such as Jira; if a script error is found, an automatic adjustment process is triggered. Finally, the analyzed test report is displayed on the cloud platform interface.

[0098] In this embodiment, by utilizing the scheduling and decision-making capabilities of the cloud platform, the execution and control capabilities of the host computer, and the response capabilities of the device under test, an end-to-end unmanned testing system is constructed, enabling the complete closed loop from device upgrade, script generation, test execution to result analysis to operate autonomously, reliably, and efficiently in an unattended environment.

[0099] Based on any of the above embodiments, the cloud platform is further configured as follows: Send the upgrade package to the device under test; Obtain the current software version of the device under test, and if it is determined that the current software version is inconsistent with the target version of the upgrade package, control the device under test to perform an upgrade operation.

[0100] Specifically, the cloud platform reads the deployed upgrade package (i.e., OTA package) from its storage service, establishes a point-to-point or broadcast channel with the target device under test through a secure network protocol (such as HTTPS, MQTT over TLS), and pushes the upgrade package data stream to the device.

[0101] It should be understood that the current software version refers to the identifier of the device's operating system or application software (e.g., HW_OS_V2.3.1). The cloud platform sends a version query command to the connected target device by calling the device management service. The agent program on the device responds to this command and sends its current, running software version number back to the cloud platform.

[0102] The target version is embedded in the metadata of the upgrade package.

[0103] Specifically, the cloud platform compares the current software version string it obtains with the target version string parsed from the upgrade package. If they differ, an inconsistency is determined. After determining that an upgrade is needed, the cloud platform does not simply resend the upgrade package; instead, it sends an upgrade command to the device. Only after receiving this command will the device start its local secure bootloader to flash and install the received upgrade package image. In other words, in this embodiment, the upgrade operation is conditional and precisely triggered, rather than indiscriminate.

[0104] Optionally, the generated model is a large language model trained based on preset data associated with the test script; the preset data includes at least one of test tool operation manuals, script sets, or functional specifications.

[0105] It should be noted that the pre-set materials refer to domain-specific knowledge bases used for targeted training of generative models. These materials are closely related to the final test scripts in terms of technical details and context. The content of these materials directly supports the writing of the test scripts. For example, the user manual guides "how to call the API," the script set provides "code examples," and the functional specifications define "what logic to test."

[0106] Specifically, the pre-set materials include a self-developed testing tool operation manual, an automated test script set, and functional specifications. (This manual enables the generation model to master the "syntax" and "vocabulary" for interacting with the device under test. For example, the model learns all valid instructions, function signatures, parameter formats, and device control protocols of the self-developed testing tool. When generating scripts, the model can accurately use the correct tool APIs. The automated test script set is a historical, proven, and successful script codebase, providing the generation model with high-quality "writing templates" and best practice patterns. By learning these scripts, the model masters the mapping rules from test cases to code, code structure, exception handling logic, and the team's coding style, thereby generating scripts that are not only functionally correct but also stylistically consistent and highly maintainable.) Functional specifications define the expected behavior of the software under test. By learning functional specifications, the generation model understands the business logic and requirement boundaries, thus ensuring that the generated test scripts are reasonable at the business level and that the verification points are meaningful, rather than just syntactically correct code. This corresponds to a leap from "how to write" to "what to write." Supervised fine-tuning of general large language model bases (such as GPT series, LLaMA series, etc.) is performed using preset data to ultimately produce a generative model specifically for this automated testing field.

[0107] Optionally, the cloud platform is further configured as follows: If the test result is determined to be a failure based on the analysis information, and the reason for the failure is a valid issue, then an issue record associated with the test case is created. If the failure reason is not a valid issue and the failure originates from the test script, then adjust the test script.

[0108] It should be noted that the analytical information is received and processed directly by the cloud platform.

[0109] The cloud platform's decision engine automatically determines whether the test passes or fails based on the conclusion fields (such as "verdict": "FAIL") in the analysis information.

[0110] Valid issues: Further categorized and marked in the analysis information (e.g., “root_cause”: “SOFTWARE_BUG”), the failure is caused by functional defects, substandard performance, or logical errors in the software under test (e.g., intelligent cockpit system).

[0111] The cloud platform is configured to automatically call the application programming interface of an external problem management platform (such as Jira or ZenTao) to create a new defect work order, i.e., the problem record, according to predefined templates and rules.

[0112] During the creation of a work order, the cloud platform will fill in the unique identifier (such as the test case ID) of the original test case that triggered the failure into a specific field of the work order, or establish a two-way hyperlink to form a traceable link.

[0113] Invalid issues: These are issues where the analysis results indicate that the failure is not due to a software defect. They may be categorized as "ENVIRONMENT_ISSUE" or "TEST_SCRIPT_ERROR".

[0114] The test script specifically refers to information analyzed within the category of "non-valid issues," which is further diagnosed as an error in the script itself (such as "sub_category": "SCRIPT_LOGIC_ERROR" or "SCRIPT_ELEMENT_NOT_FOUND").

[0115] For example, the test script can be adjusted as follows: The cloud platform is configured to trigger an internal workflow. This workflow includes: retrieving the problematic original script and detailed failure analysis, calling the generation model for correction, receiving and verifying the corrected script, and finally updating it to the script library to prepare for the next execution.

[0116] In this embodiment, the cloud platform ensures that the upgrade operation is only triggered when the device is truly necessary (i.e., the versions are inconsistent), thereby eliminating invalid duplicate upgrades and resource waste.

[0117] Furthermore, based on the same concept as the aforementioned automated testing method and system, this application also provides a testing method and apparatus, with reference to... Figures 2-6 The test method includes the following steps: 1. OTA package deployment and upgrade of the device under test: All devices to be tested have been uploaded to the cloud (i.e., the devices can communicate and operate with the cloud platform via the network), and are placed in designated unmanned management rooms according to their categories (such as cockpit test bench, body test bench, and complete vehicle). When there is a test version, the test engineer only needs to deploy the test version OTA package, specify the device to be tested, and push strategy (such as maximum number of failure retries, retry interval, etc.) on the cloud platform. Then the cloud platform will automatically push the OTA package to the designated device to be tested for upgrade. During OTA upgrade, the software version of the device to be tested is automatically obtained and compared with the test version to determine whether they are consistent: (1) If they are inconsistent, it is determined whether the number of OTA upgrades exceeds the maximum number of failure retries set: (1.1) If it does not exceed the limit, the designated device to be tested will be upgraded again; (1.2) If it exceeds the limit, the test engineer will be notified to update the test version OTA package, redeploy, and repeat the above overall process. (2) If they are consistent, it means that the device to be tested has been successfully upgraded and the test prerequisites have been met, and the subsequent test process can be entered. 2. Test case distribution and automated test script generation: Test engineers upload reviewed and approved test cases (e.g., in .excel format) to the cloud platform. Then, based on the test case type (e.g., functional, blinking, response, etc.) and the host computer status (offline, idle, busy, etc.), the platform automatically selects the device to be tested and proceeds with the automated script generation process. Using the self-developed testing tool's operation manual, corresponding automated test script sets, functional specifications, and other materials as datasets, a large model is trained to automatically generate automated test scripts based on the self-developed testing tool and associate them with corresponding test cases. Finally, test engineers select a test case set and create a test task, which automatically distributes the corresponding automated test scripts to the host computer corresponding to the specified device to be tested for automated testing. Optionally, test requirement documents can be uploaded to the cloud platform, and LLM technology can be used to automatically generate test cases and associate them with corresponding requirement points for online review. This process can replace the step of "uploading reviewed test cases to the cloud platform," improving the efficiency of automated testing and completing the entire process from requirement to test result without manual intervention. 3. Automated test task execution: The host computer is equipped with a self-developed testing tool client and experiment management software. After receiving the automated test script issued in step 2, it will automatically deploy it to the test environment to meet the execution conditions of the automated test script. Then, the test execution engine will execute the test task according to the scheduling strategy (such as the maximum number of failure retries, retry interval, execution mode, etc.), and collect logs and execution results in real time to generate a test report. 4. Test log report analysis and bug management: After step 3 is completed, the test log report information is automatically obtained, and the failure reason is analyzed using LLM technology to determine whether it is a valid bug: (1) If it is a valid bug, the bug report is automatically submitted based on the principles of the submission platform (such as problem level, problem description, expected result, etc.), and the bug link is associated with the test case; (2) If not, it is determined whether the failure is caused by the incorrect automated test script itself: (2.1) If not, the failure reason is supplemented and the test result is updated to the test report; (2.2) If yes, it is determined whether the maximum number of times the automated test script is automatically adjusted is exceeded: (2.2.1) If not, the automated test script is automatically adjusted using LLM technology; (2.2.2) If it is exceeded, the test engineer is notified to manually adjust the automated test script, and then the adjusted automated test script is re-executed, and step 4 is repeated.

[0118] Reference Figure 7 The testing apparatus includes: 1. Cloud Platform: The cloud platform is used for uploading and downloading test cases, OTA deployment and push, remote distribution of test cases and test tasks, and display of test results. It communicates with the device under test and the host computer via the network. 2. Host computer: The host computer is used for automated test script generation and adjustment, test command issuance, test log and execution result collection, test result analysis, and bug management. It communicates with the cloud platform via network and with the device under test, microphone, robotic arm, simulated mouth, camera, etc. via hardwired connections. 3. Equipment to be tested: The device under test is used to receive OTA upgrade packages and test commands, and to perform OTA upgrades and actual tests. It communicates with the cloud platform via a network and with the host computer, programmable power supply, VN5620, etc., via hardwired connections. 4. Room: The room is used to store the equipment to be tested and the host computer, and is classified according to the equipment category.

[0119] For example, suppose there is only one device under test, the vehicle infotainment system 1, and its corresponding host computer 1, connected via a hardwired connection. When version 1 is submitted for testing, the test engineer deploys an OTA upgrade package on the cloud platform and pushes it to the designated device under test, the vehicle infotainment system 1. After receiving the software upgrade package, the vehicle infotainment system 1 automatically upgrades and returns the upgrade result. Upon successful upgrade, the test engineer uploads the reviewed and approved test case Excel spreadsheet to the cloud platform. The cloud platform then automatically parses the test cases, automatically generates automated test scripts, and associates them with the corresponding tests. The scripts include the test case number, test case name, test function point, test case type, preconditions, execution steps, expected results, automated test script link, and are displayed on the host computer 1. The test engineer only needs to select the test case set and create a test task to send the automated test script to the host computer 1 for automated testing. Finally, the system automatically analyzes the test results and sends the analyzed test report back to the cloud platform.

[0120] In this embodiment, the existing method of relying on test engineers to manually develop and maintain automated test scripts is replaced, which effectively reduces labor costs and improves the efficiency of automated test script development and maintenance. The existing method of relying on test engineers to manually analyze test log reports and manage bugs is replaced, which effectively improves the efficiency of bug analysis and management and reduces the disadvantages of manual bug analysis, which is prone to errors and omissions. The addition of OTA deployment and upgrade of the device under test further improves the efficiency of automated testing.

[0121] Based on the methods described in any of the above embodiments, this application also provides a computer storage medium storing a computer program, which, when executed by a processor, can be used to perform the methods described in any of the above embodiments.

[0122] Based on the methods described in any of the above embodiments, this application also provides a computer program product, which includes one or more computer programs or instructions. The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. When executed by a processor, the computer program implements the methods described in any of the above embodiments.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0124] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0125] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0127] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0128] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. An automated testing method, characterized by, The method is applied to a cloud platform; the method includes: Obtain the upgrade package for the device under test; Control the device under test to upgrade based on the upgrade package; Input the test cases into the generation model to obtain the corresponding test scripts; The test script is sent to the test execution device, which is then controlled to execute automated test tasks and generate test logs. The test logs are input into the analysis model to obtain analysis information including the test results; Based on the analyzed information, corresponding management operations are performed.

2. The method of claim 1, wherein, The process of controlling the device under test to upgrade based on the upgrade package includes: Send the upgrade package to the device under test; Obtain the current software version of the device under test; If the current software version is inconsistent with the target version of the upgrade package, the device under test is controlled to perform an upgrade operation.

3. The method of claim 1, wherein, The generated model is a large language model trained based on preset data associated with the test script; the preset data includes at least one of the following: test tool operation manual, script set, or functional specification.

4. The method according to claim 1, characterized in that, The execution of corresponding management operations based on the analyzed information includes: If the test result is determined to be a failure based on the analysis information, and the reason for the failure is a valid issue, then an issue record associated with the test case is created. If the test result is determined to be a failure based on the analysis information, and if the cause of failure is not an effective problem, then if the cause of failure is determined to originate from the test script, the test script shall be adjusted.

5. The method according to claim 4, characterized in that, The adjustment of the test script includes: If the number of adjustments to the test script does not exceed a preset threshold, then the test script is adjusted through the generation model; If the number of adjustments to the test script exceeds the preset threshold, a prompt message indicating that manual intervention is required will be generated.

6. The method according to claim 1, characterized in that, The test cases are generated through the following steps: Obtain the test requirements document; The test requirement document is input into the generation model to generate the test cases corresponding to the test requirement document.

7. An automated testing system, characterized in that, The system includes: The cloud platform is configured as follows: Obtain the upgrade package for the device under test; Control the device under test to upgrade based on the upgrade package; Input the test cases into the generation model to obtain the corresponding test scripts; Send the test script to the test execution device; Input the test logs into the analysis model to obtain analysis information including the test results; Perform corresponding management operations based on the analyzed information; The host computer, acting as the test execution device, is communicatively connected to the cloud platform and is configured as follows: Receive test scripts from the cloud platform; Execute the automated test tasks indicated by the test script and generate the test logs; The device under test, which is communicatively connected to the cloud platform and the host computer, is configured as follows: Upgrades are performed in response to controls from the cloud platform. The test operation is executed in response to the instructions from the host computer.

8. The system according to claim 7, characterized in that, The cloud platform is also configured as follows: Send the upgrade package to the device under test; Obtain the current software version of the device under test, and if it is determined that the current software version is inconsistent with the target version of the upgrade package, control the device under test to perform an upgrade operation.

9. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of any of the methods described in claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images