Method and system for automated testing of embedded software

Abstract

This application discloses an automated testing method and system for embedded software, relating to the field of software testing. The method includes responding to remote repository code commit events, generating a unique tracking identifier and performing consistency verification on the committed content. Upon successful verification, static code analysis and unit testing are triggered. After successful unit testing, an environment snapshot is generated. Based on the test environment, automated write operations are performed, and the status is monitored to generate logs. Finally, various data are correlated to generate a structured test report and pushed to a preset terminal. The application also includes a corresponding automated testing system. This application achieves the technical effect of automatically testing embedded software, realizing full tracking and control of the testing process, generating a structured test report containing correlation analysis results, facilitating test result backtracking and root cause localization, and improving testing efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of automotive electronic software testing, and in particular to an automated testing method and system for embedded software. Background Technology

[0002] The current embedded software flashing and testing process in the automotive software testing field has mainly gone through three development stages: the traditional flashing stage, the semi-automated stage, and the early CI / CD stage. The emergence of the traditional flashing stage provided a preliminary operational process for automotive software testing, the semi-automated stage improved efficiency to a certain extent, and the early CI / CD stage took a further step towards automation. These development stages have gradually driven the progress of the automotive software testing field and laid the foundation for subsequent technological developments.

[0003] In existing technologies, different methods are used at different stages to solve problems in the automotive software testing process. In the traditional flashing stage, it is entirely manual, relying on engineers to manually execute the CANoe project, and requires specialized equipment to complete the software flashing and testing. The semi-automated stage mainly uses CANoe for scripted flashing, which reduces manual operation to some extent, but version management depends on local folder naming, and multiple branches of code need to be switched manually. In the early stages of CI / CD, by applying Jenkins to in-vehicle testing, some of the workflow operations from code submission to compilation, and then to flashing and testing have been realized.

[0004] However, existing technologies have significant drawbacks. They are highly reliant on manual operation, with traditional flashing processes relying entirely on manual intervention. Each flash is time-consuming, inefficient, and prone to errors. Version management is chaotic; in the semi-automated stage, there are no dedicated version management tools, and manual switching between multiple code branches easily leads to version incompatibility issues with hardware. The automation chain is incomplete; in the early stages of CI / CD, cross-platform operations lack closed-loop automation, resulting in the separation of code and flashing configuration, a lack of remote control, and the need for physical contact with production line equipment, making unattended flashing impossible and failing to meet the demands for efficient, standardized, and automated testing. Summary of the Invention

[0005] This invention relates to the field of automotive electronic software testing technology, specifically providing an automated testing method and system for embedded software integrating multiple components, aiming to solve the core pain points of low testing efficiency, process disruption, and difficulty in problem localization in current automotive software testing under continuous integration environments.

[0006] Firstly, this application discloses an automated testing method for embedded software, employing the following technical means: An automated testing method for embedded software includes the following steps: Respond to code commit events in the remote repository, generate a unique tracking identifier for the current test process, and perform consistency verification on the executable files and source code in the committed content; Once the consistency verification is passed, static code analysis and unit testing are automatically triggered to obtain preliminary test results, which include a static code analysis report, the pass rate of unit test cases, and code coverage. Once the unit test passes, an environment snapshot of the current test environment and a corresponding snapshot identifier are generated. Based on the test environment, an automated flashing operation of the embedded software is performed, the flashing status of the flashing operation is monitored, and a flashing log is generated. Based on the unique tracking identifier, the submitted content, the initial test results, the snapshot identifier, and the write log are associated to generate a structured test report containing the association analysis results and push it to a preset terminal.

[0007] By adopting the above technical solutions, generating a unique tracking identifier enables full tracking and control of the testing process; consistency verification avoids invalid tests caused by issues with submitted content; automatically triggering static code analysis and unit testing can promptly identify code problems; generating environment snapshots allows for locating the corresponding environment during test result backtracking; automated flushing operations and status monitoring, along with log generation, improve flushing efficiency and accuracy; and associating various data points to generate structured test reports and push them to preset terminals facilitates testers' viewing and analysis of test results.

[0008] Preferably, the step of responding to the code commit event of the remote repository, generating a unique tracking identifier for the current test process, and performing consistency verification on the executable files and source code in the committed content specifically includes the following steps: When a code commit event is detected in the remote repository, a unique tracking identifier for the current test process is generated. The code commit event is an event that pushes the commit content to a preset branch of the remote repository. The commit content includes local code changes. The commit content is associated with the unique tracking identifier, which includes at least the timestamp of the code commit event and the unique identifier of the remote repository; The consistency verification process is triggered based on the code submission event and a verification result is obtained. The consistency verification process includes calculating the hash values ​​of the executable file and the source code in the submission content respectively. If the hash value of the executable file and the hash value of the source code are the same, the verification result is passed; otherwise, the verification result is failed. The verification result is associated with the unique tracking identifier.

[0009] By adopting the above technical solutions, generating a unique tracking identifier enables full tracking and control of the testing process. Associating submitted content with the unique tracking identifier facilitates subsequent traceability. Injecting the unique tracking identifier into the test environment configuration file before the test process starts ensures the effective application of the tracking identifier. Performing consistency verification on the executable file and source code can avoid testing problems caused by inconsistencies between the two, thereby improving the accuracy and reliability of the test.

[0010] Preferably, after the consistency verification passes, static code analysis and unit testing are automatically triggered to obtain preliminary test results, specifically including the following steps: Once the consistency verification is passed, static code analysis and unit testing will be automatically triggered by the continuous integration tool. The static code analysis includes a comprehensive scan of the code according to preset code style check rules, obtaining a static analysis report, and parsing the number of fatal errors and high-level warnings from the static analysis report as the first primary test data. The unit test includes calling a preset command-line interface to execute preset unit test cases, obtaining a unit test report, and parsing the test case pass rate and code coverage of the unit test cases from the unit test report as second primary test data; The first and second primary test data are compared with preset quality access control thresholds respectively; The unit test is considered to have passed when the number of fatal errors is zero, the test case pass rate reaches a first threshold, and the code coverage is not lower than a second threshold; otherwise, it fails.

[0011] By adopting the above technical solutions, once the consistency verification is passed, the continuous integration tools can automatically trigger static code analysis and unit testing, which can improve testing efficiency and accuracy. A comprehensive scan of the submitted content and execution of unit test cases can discover fatal errors, high-level warnings, unit test pass rates, and code coverage issues in the code. By comparing the initial test data with the preset quality threshold, it is possible to accurately determine whether the unit tests have passed, ensuring the standardization and reliability of the testing process.

[0012] Preferably, the step of generating an environment snapshot of the current test environment and a corresponding snapshot identifier after the unit test passes specifically includes the following steps: Once the unit test passes, the environmental elements of the current test environment are captured. These environmental elements include system configuration information, dependent component status, and running application data. Containerization technology is used to generate an immutable image file containing these environmental elements as an environment snapshot. A unique hash value is calculated based on the hash algorithm to obtain the immutable image file, and the unique hash value is used as the snapshot identifier; Using the unique tracking identifier as an index, the snapshot identifier, image creation time, and image dependency list are stored to obtain an association relationship, which is used to locate the corresponding environment snapshot when backtracking test results.

[0013] By adopting the above technical solution, after the unit test passes, the system configuration information, dependent component status, and running application data of the test environment are captured. Containerization technology is used to generate an immutable image file as an environment snapshot, which avoids environment dependency conflicts and improves the system's portability and stability. A unique hash value is calculated based on a hash algorithm as the environment snapshot identifier, which can uniquely identify the environment snapshot. The environment snapshot identifier, image creation time, and image dependency list are stored as an index using a unique tracking identifier to obtain the association, which facilitates locating the corresponding environment snapshot when backtracking test results.

[0014] Preferably, the step of performing automated flashing operations on the embedded software based on the test environment, monitoring the flashing status of the flashing operation, and generating flashing logs specifically includes the following steps: The flashing tool is driven by automated control commands through a graphical user interface to perform the flashing operation. The flashing status is monitored in real time and the log file output by the flashing tool is read. The key status codes and progress information in the log file are extracted and associated with the unique tracking identifier. The progress control position and status prompt text on the flashing tool interface are captured by the graphical interface automation control tool, and the progress value corresponding to the progress control position and the status prompt text are associated and recorded with the unique tracking identifier. The key status codes and progress information in the log file are compared with the status prompt text and progress value obtained by the graphical interface automation control tool. If the two are consistent and the flashing status meets the preset flashing completion conditions, the flashing is determined to be successful; otherwise, the flashing fails.

[0015] By adopting the above technical solution, and using graphical user interface automated control commands and tool-driven flashing tools to perform flashing operations, the flashing of embedded software can be automated, improving flashing efficiency. Real-time monitoring of the flashing status and reading of log files, along with the association and recording of key status codes, progress information, progress values, and status prompt text with unique tracking identifiers, allows for full tracking and control of the flashing operation. Comparison between the log file and the information obtained by the graphical interface automated control tool accurately determines whether the flashing was successful, improving the accuracy and reliability of the flashing process.

[0016] Preferably, if the flashing is determined to be successful, the following steps are also included: If the flashing is confirmed to be successful through the unique tracking identifier, the system test is automatically triggered, and the communication message data of the vehicle bus is collected synchronously, and the communication message data is used as part of the system test results. The system test results are associated with the unique tracking identifier.

[0017] By adopting the above technical solution, system testing is automatically triggered after successful flashing, and vehicle bus communication message data is collected synchronously and used as part of the system test results. The system test results are associated with a unique tracking identifier, realizing automated advancement of the test process, facilitating traceability and management of test results, and improving the efficiency and accuracy of automated testing of embedded software.

[0018] Preferably, the correlation analysis results include root cause localization prompts. When the system test fails, generating the root cause localization prompts specifically includes the following steps: Analyze failed system test cases and extract abnormal bus communication messages related to the failed system test cases. The abnormal bus communication messages include expected CAN bus signals and actual CAN bus signals. Based on the pre-configured signal and software component mapping table, locate the source code file that processes the abnormal bus communication message; Automatically retrieve and highlight all warnings that appear in the static analysis report and the code coverage in the unit tests for the source code file.

[0019] By adopting the above technical solution, when system testing fails, the failed system test cases can be parsed, relevant abnormal bus communication messages can be extracted, and the source code file that handles the abnormal bus communication messages can be located according to the signal-to-software component mapping table. The source code file can be automatically retrieved and highlighted in all warnings in the static analysis report and code coverage in unit tests, thus realizing root cause localization prompts and helping to quickly find the cause of system test failure.

[0020] Preferably, pushing the structured test report to a preset terminal includes the following steps: The structured test report is sent to a predetermined group of terminals in a preset format. The structured test report includes at least the unique tracking identifier of the current test process, the test conclusion, and key quantitative indicator data. The test conclusion includes success or failure. When the test conclusion is failure, the unique tracking identifier in the structured test report is set as an interactive hyperlink, and the hyperlinked page displays all related data related to the failure of this test process.

[0021] By adopting the above technical solution, structured test reports are sent to designated groups of terminals, making it convenient for relevant personnel to obtain test information. The reports include unique tracking identifiers, test conclusions, and key quantitative indicator data, making the information complete and easy to view. When a test fails, the unique tracking identifier is set as an interactive hyperlink, which can quickly display all related data related to the test failure, facilitating the location and analysis of the problem.

[0022] Secondly, this application discloses an automated testing system for embedded software, employing the following technical means: An automated testing system for embedded software includes the following modules: The trigger verification module is used to respond to code commit events in the remote repository, generate a unique tracking identifier for the current test process, and perform consistency verification on the executable files and source code in the committed content; The test execution module is used to automatically trigger static code analysis and unit testing after the consistency verification is passed, and obtain preliminary test results, including static code analysis report, pass rate of unit test cases and code coverage. The flashing monitoring module is used to generate an environment snapshot and corresponding snapshot identifier of the current test environment after the unit test passes, execute an automated flashing operation of the embedded software based on the test environment, monitor the flashing status of the flashing operation and generate a flashing log. The associated push module is used to associate the submitted content, the initial test results, the snapshot identifier, and the write log based on the unique tracking identifier, generate a structured test report containing the association analysis results, and push it to a preset terminal.

[0023] By adopting the above technical solutions, consistency verification of the submitted executable files and source code can be performed, avoiding test problems caused by inconsistencies between code and executable files; static code analysis and unit testing are automatically triggered, yielding preliminary test results such as static code analysis reports, unit test case pass rates, and code coverage, which helps to identify potential code problems; environment snapshots and identifiers of the test environment are generated, facilitating the location of the corresponding environment during test result backtracking; automated flashing operations of embedded software are performed and their status is monitored, improving flashing efficiency and accuracy; a unique tracking identifier is associated with all previous test data to generate structured test reports, enabling automation of the entire testing process and effective data management and traceability, improving testing efficiency and accuracy, reducing reliance on manual intervention, and enhancing the standardization and traceability of testing.

[0024] Preferably, the trigger verification module, the test execution module, the flush monitoring module, and the associated push module are implemented by integrating a code management platform, a continuous integration server, static analysis tools, unit testing tools, flush tools, and system testing tools; The integration of the flashing monitoring module with the flashing tool is achieved through a graphical interface automation adapter. The graphical interface automation adapter is used to simulate human-computer interaction by executing scripts when the flashing tool does not provide a command-line interface, thereby driving the graphical interface to complete automated flashing.

[0025] By adopting the above technical solutions, integrating code management platform, continuous integration server, static analysis tools, unit testing tools, flashing tools, and system testing tools, the functions of each module of the embedded software automated testing system can be realized, improving testing efficiency and accuracy. When the flashing tool does not provide a command-line interface, the graphical interface automation adapter drives the graphical interface to complete automated flashing by executing scripts to simulate human-computer interaction, solving some of the automation operation problems of flashing tools and improving the applicability and operability of the system.

[0026] In summary, this application includes at least one of the following beneficial technical effects: (1) This application verifies the consistency of the executable files and source code in the submitted content, and combined with standardized rule settings, can avoid the execution of invalid test tasks caused by accidental triggering, thereby improving test efficiency; (2) This application has automated the static code analysis, unit testing, software flashing, system testing and other processes, achieving full automation from code submission to testing, reducing the dependence on manual labor and improving the accuracy of testing; (3) This application generates a structured test report and pushes it by generating a unique tracking identifier to associate the data of each stage of the test process, which facilitates the backtracking of test results and root cause localization, and improves the traceability and standardization of the test. Attached Figure Description

[0027] Figure 1 This is a structural diagram of an automated testing system for embedded software; Figure 2 This is a flowchart of an automated testing method for embedded software; Figure 3 It is a global data flow and trace identifier association diagram of automated testing methods for embedded software. Detailed Implementation

[0028] The technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments are only possible technical implementations of the present invention, but are not limited thereto. Other embodiments obtained by those skilled in the art in conjunction with the embodiments of the present invention without creative effort are also within the protection scope of the present invention.

[0029] This application mainly adopts the integration of multiple tools to build a fully closed-loop automated test chain, which achieves the effect of efficient, standardized and unmanned automotive software testing. The following is a further detailed description of this application.

[0030] Example 1 This embodiment provides an automated testing system for embedded software, such as... Figure 1 As shown, the system includes a trigger verification module, a test execution module, a flush monitoring module, and an association push module. These modules work collaboratively to achieve automated testing of embedded software, improve testing efficiency and accuracy, and facilitate problem tracing and management by responding to code submission events, executing tests, monitoring flush operations, and associating and pushing test results. Each module has a clear division of labor and cooperates with the others to form a complete automated testing chain. Specifically, it includes the following modules: The trigger verification module is used to respond to code commit events in the remote repository, generate a unique tracking identifier for the current test process, and perform consistency verification on the executable files and source code in the committed content; The test execution module is used to automatically trigger static code analysis and unit testing after the consistency verification is passed, and obtain preliminary test results. The preliminary test results include static code analysis report, pass rate of unit test cases and code coverage. The flash monitoring module is used to generate an environment snapshot and corresponding snapshot identifier of the current test environment after the unit test passes, execute automated flashing operations of the embedded software based on the test environment, monitor the flashing status of the flashing operation and generate flashing logs. The associated push module is used to associate submitted content, preliminary test results, snapshot identifiers, and write logs based on a unique tracking identifier, generate a structured test report containing the associated analysis results, and push it to a preset terminal.

[0031] In this embodiment, the trigger verification module, test execution module, flush monitoring module, and associated push module are implemented by integrating a code management platform, a continuous integration server, static analysis tools, unit testing tools, flush tools, and system testing tools.

[0032] Specifically, the system in this embodiment is implemented by integrating the following tools and developing custom logic modules. The core environment configuration is as follows: Code Management Platform: Completes the local deployment and configuration of GitLab remote code repositories, used to host C / C++ source code, MATLAB / Simulink models and compiled VBF (Vector BinaryFormat) files for the AUTOSAR architecture. It provides source code storage, branch management and remote access support, and sets strict "merge request" protection rules for the main branch to prohibit direct push.

[0033] Continuous Integration / Continuous Deployment (CI / CD) Server: Jenkins version 2.470 is used, running in a JDK 17 environment, to complete the local deployment of the Jenkins server. Jenkins has GitLab, QAC, Tessy, and other plugins installed, and a Python script execution environment for controlling CANoe and DSA tools is configured, including network environment configuration and user authentication information settings.

[0034] The testing toolchain includes static code analysis tools, unit testing tools, system testing tools, and flashing tools, as detailed below.

[0035] Static code analysis tools: PRQA QAC 2022 is used to check code style and potential defects; Unit testing tool: Tessy is used to execute unit tests and generate coverage reports.

[0036] System testing tools: CANoe is used, with its built-in vTESTstudio for designing test cases and the DVA component for automated execution.

[0037] Flashing tool: Vector's DSA (Deployable Software Archive) tool is used.

[0038] The core custom components in this embodiment include a graphical interface automation adapter and a central data association, as detailed below.

[0039] Graphical User Interface (GUI) Automation Adapter: Developed using the Python libraries pyautogui and pyguiauto, this graphical interface is used to automate the operation of DSA tools.

[0040] Central Data Correlator: A lightweight web service developed based on the Python Flask framework, with a built-in SQLite database for storing and correlating data throughout the testing process. It generates and maintains a unique tracking identifier for each testing process instance.

[0041] In this embodiment, the integration of the flashing monitoring module and the flashing tool is achieved through a graphical interface automation adapter. The graphical interface automation adapter is used to simulate human-computer interaction by executing scripts when the flashing tool does not provide a command-line interface, thereby driving the graphical interface to complete automated flashing.

[0042] This embodiment of the automated testing system integrates multiple tools and modules to achieve full-process automation of embedded software testing. The modules collaborate closely, using unique tracking identifiers to link data from each testing stage, facilitating the management and analysis of test results. The graphical interface automation adapter solves the problem of the lack of a command-line interface for the flashing tool, improving the system's compatibility and applicability. Compared to existing technologies, this system reduces reliance on manual intervention, improves testing accuracy and standardization, and meets the requirements for efficient, standardized, and automated testing.

[0043] Example 2 This embodiment provides an automated testing method for embedded software. The method is based on the testing system of Embodiment 1, such as... Figure 2 As shown, it includes the following steps: S1. Respond to code commit events in the remote repository, generate a unique trace identifier for the current test process, and perform consistency verification on the executable files and source code in the committed content. This includes the following steps: S11. When a code commit event is detected in the remote repository, a unique tracking identifier for the current test process is generated. The code commit event is the event that pushes the commit content to the preset branch of the remote repository. The commit content includes local code changes.

[0044] In this embodiment, the GitLab remote code repository is configured with a webhook. A webhook is a mechanism that allows external systems to receive GitLab event notifications. By setting up a webhook in the GitLab repository, a communication bridge can be established between GitLab and other systems. In this embodiment, a communication bridge is established between GitLab and Jenkins.

[0045] In this embodiment, the event is defined as a code commit event, specifically a code push to the protected main branch (master / main). This means that subsequent Jenkins build tasks will only be triggered when the developer pushes the code to these two specific main branches. This ensures that changes to core code branches can be processed automatically and promptly.

[0046] Regular developers typically lack the authority to directly push code to the protected main branch. After completing feature development or bug fixes locally, developers push the code to a temporary branch and then create a merge request on the code hosting platform, requesting that the code from the temporary branch be merged into the protected main branch. Only after the merge request passes review, automated checks, and other processes will the code be actually merged into the main branch. This is equivalent to completing the code push to the protected main branch and can trigger automated tasks such as builds and tests within the CI / CD workflow.

[0047] When GitLab detects a specific code commit event—a push to the protected master branch—it sends a notification to Jenkins via a previously configured webhook.

[0048] Once Jenkins receives the notification, it will automatically start the pre-defined build task. This build task can include a series of operations such as code compilation, testing, and deployment, thereby automating the process of handling code changes.

[0049] S12, such as Figure 3 As shown, the commit content is associated with a unique tracking identifier, which includes at least the timestamp of the code commit event and the unique identifier of the remote repository.

[0050] This unique tracking identifier is used to track and control the entire testing process. Before the testing process starts, the unique tracking identifier is injected into the configuration file of the testing environment.

[0051] S13. Trigger the consistency verification process based on the code commit event and obtain the verification result. The consistency verification process includes calculating the hash values ​​of the executable file and source code in the committed content using a hash algorithm.

[0052] In this embodiment, the calculation involves the pre-compiled VBF (VectorBinaryFormat) file in the submission content and the SHA256 hash value of the source code from the local repository.

[0053] S14. If the hash value of the executable file is the same as the hash value of the source code, the verification result is passed; otherwise, the verification result is failed. Figure 3 As shown, the verification result is associated with a unique tracking identifier.

[0054] If the verification result is successful, the successful verification result and hash value will be associated with and stored with the unique tracking identifier; if the verification result is unsuccessful, the verification failure result, hash value comparison details and unique tracking identifier will be associated with and stored, the current test process will be terminated, and a verification failure notification containing the unique tracking identifier will be pushed to the designated terminal so that staff can query the reason for the failure through the unique tracking identifier.

[0055] S2. Once the consistency verification passes, static code analysis and unit testing are automatically triggered to obtain preliminary test results. These results include a static code analysis report, the pass rate of unit test cases, and code coverage. Specifically, the following steps are included: S21. Once the consistency verification is passed, static code analysis and unit testing will be automatically triggered based on the continuous integration tool.

[0056] S211. Static code analysis includes a comprehensive scan of the code based on preset code style check rules, obtaining a static analysis report, and extracting the number of fatal errors and high-level warnings from the static analysis report as the first primary test data.

[0057] In this embodiment, the Jenkins task scheduling system identifies the current test process through a unique tracking identifier and calls the integrated QAC plugin to perform static code analysis. In this embodiment, the number of fatal errors must be 0, and the number of high-level warnings must not exceed 10.

[0058] S212. Unit testing includes calling a preset command-line interface to execute preset unit test cases, obtaining a unit test report, and parsing the test case pass rate and code coverage of the unit test cases from the unit test report as secondary primary test data.

[0059] In this embodiment, Tessy's command-line interface (CLI) is invoked to execute preset unit test cases. If an exception occurs during Tessy execution, the abnormal scenario is automatically recorded, and the recording file is stored in the SVN knowledge base, achieving precise binding between abnormal data and unique tracking identifiers.

[0060] S22. Compare the first primary test data and the second primary test data with the preset quality access control thresholds respectively.

[0061] S23. When the number of fatal errors is zero, the test case pass rate reaches the first threshold, and the code coverage is not lower than the second threshold, the unit test is considered passed; otherwise, it fails. The access control thresholds set in this embodiment are: a test case pass rate of 98% and a code coverage rate of not less than 80%.

[0062] The central data correlator associates and stores primary test results such as the number of fatal errors from static analysis, the number of high-level warnings, and the pass and coverage rates of unit tests with the current unique trace identifier.

[0063] S3. After the unit test passes, generate an environment snapshot of the current test environment and the corresponding snapshot identifier, which includes the following steps: S31. After the unit test passes, capture the environment elements of the current test environment. The environment elements include system configuration information, dependent component status and running application data. Use containerization technology to generate an immutable image file containing the environment elements as an environment snapshot.

[0064] S32. Calculate a unique hash value for the immutable image file based on a hash algorithm, and use the unique hash value as a snapshot identifier.

[0065] In this embodiment, specifically, Docker containerization technology is used to package the operating system version, toolchain (Jenkins, GitLab, QAC, Tessy, CANoe) version, and library files that the current test environment depends on into an immutable image, and the unique hash value of the generated image is used as a snapshot identifier.

[0066] S33. In the environmental snapshot management system, a unique tracking identifier is used as the index, such as... Figure 3 As shown, the storage snapshot identifier, image creation time, and image dependency list are correlated, and the correlation is used to locate the corresponding environment snapshot when backtracking test results.

[0067] When it is necessary to reproduce the test environment later, the corresponding snapshot identifier can be queried through the unique tracking identifier to quickly pull the matching Docker image. The immutable image file contains complete environment elements, which can ensure that the test environment can be accurately reproduced. By storing the association relationship in the database, it can also support bidirectional positioning of test results and environment.

[0068] S4. After the unit test passes, the target flashing tool is driven to perform automated flashing operations on the target embedded software through graphical user interface (GUI) automated control commands. The flashing status of the flashing operation is monitored in real time and a flashing log is generated. The specific steps include the following: S41. The flashing tool is driven by the graphical user interface to perform the flashing operation through automated control commands. In this embodiment, the flashing tool is a DSA flashing tool.

[0069] The DSA (Deployable Software Archive) flashing automation in this embodiment is a Python script control module based on the pyguiauto library. The script automatically identifies the DSA version and matches the corresponding graphical user interface operation logic, and sequentially and automatically executes the operations of loading the SDDB file, loading the APP and CAL packages, and triggering the flashing process. This module realizes the automated operation of the DSA graphical user interface by simulating human-computer interaction.

[0070] S42. During the flashing process, monitor the flashing status in real time and read the log file output by the flashing tool. Extract the key status codes and progress information from the log file and associate the extracted key status codes and progress information with a unique tracking identifier.

[0071] For example, key status codes: 0x00 indicates successful initialization, 0xFF indicates failed flashing; progress information such as 30%, 75%, 100%.

[0072] S43. Call the image recognition interface of the pyguiauto library through a Python script, and capture the position of the progress control and status prompt text on the DSA flashing tool interface through a graphical interface automation control tool. The status prompt text is such as flashing in progress, flashing completed, flashing failed. Associate the progress value and status prompt text corresponding to the position of the progress control with a unique tracking identifier and record them.

[0073] S44. Compare the key status codes in the log file with the status prompt text obtained by the graphical interface automation control tool, and compare the progress information in the log file with the progress value obtained by the graphical interface automation control tool.

[0074] S45. If the above comparisons of the two are consistent and the flashing status meets the preset flashing completion conditions, the flashing is determined to be successful; otherwise, the flashing fails.

[0075] The flashing completion conditions are as follows: the key status code is the preset success code, the progress information is 100%, and the status prompt text is flashing complete. If any of the above comparisons are inconsistent or the flashing completion conditions are not met, the flashing is determined to have failed. The cross-validation differences, failure status codes, and corresponding interface screenshots are associated and stored with the unique tracking identifier, and a flashing failure report containing the unique tracking identifier is generated.

[0076] If the flashing process fails, the error code and failure node information in the log file are extracted and stored in association with the unique tracking identifier. The test process is then terminated and a notification containing the unique tracking identifier and failure node information is sent. Staff can directly locate the specific step where the flashing failed through the unique tracking identifier.

[0077] S46. If the flashing is successful, the following steps are also included: S461. If Jenkins confirms successful flashing through a unique tracking identifier, the integrated CANoe will automatically trigger system testing based on the DVA project script.

[0078] S462. Synchronously collect vehicle bus communication message data, and include the system test report and bus communication message data as part of the system test results; associate the system test results with the unique tracking identifier.

[0079] If the system test fails, the failed test case number, failure time, bus exception message and unique trace identifier will be bound to provide a data index for subsequent root cause analysis.

[0080] The system test failure is a conclusion automatically determined by the automated testing system of this embodiment based on a predefined set of rules. The determination criteria include, but are not limited to, one or more of the following: Test case execution failures include assertion failures, execution timeouts, or execution errors. Abnormal system-level monitoring indicators: including bus communication errors, critical signal value range or logic abnormalities, and ECU state machine entering unexpected states; Non-functional metrics exceed limits: including response time and CPU / memory usage exceeding preset thresholds; Test process execution errors include test environment connection failures and test toolchain integration interface call failures.

[0081] S463. The process of root cause analysis is as follows: Analyze failed system test cases and extract abnormal bus communication messages related to the failed system test cases. The abnormal bus communication messages include expected CAN bus signals and actual CAN bus signals. Based on the pre-configured signal and software component mapping table, locate the source code file that handles abnormal bus communication messages; Automatically retrieves and highlights all warnings that appear in the static analysis report and code coverage in the unit tests of the source code files.

[0082] S5. Based on the unique tracking identifier, associate the submitted content, preliminary test results, snapshot identifier, and write logs to generate a structured test report containing the association analysis results and push it to the preset terminal. The specific steps include the following: S51. Based on the unique tracking identifier, aggregate the associated submission content, preliminary test results, snapshot identifier, write logs, and system test results to generate a structured test report containing the correlation analysis results.

[0083] The correlation analysis results include root cause localization tips, which are displayed in the structured test report, indicating the suspected root cause modules identified by the correlation analysis and their related evidence.

[0084] S52. Send the structured test report in a preset format to the designated group on the terminal. The structured test report shall include at least a unique traceability identifier for the current test process, test conclusions, key quantitative indicator data, and a link to view the report. Key quantitative indicator data includes data such as total test time and unit test coverage.

[0085] The test results can be either success or failure. When the test result is failure, the unique tracking identifier in the structured test report is set as an interactive hyperlink. The hyperlink redirects to a page that displays all relevant data related to the failure of this test process.

[0086] Specifically, when a user clicks the hyperlink on the terminal interface, they will be redirected to the details page of the test data management platform. This page displays all related data related to the failure of this test, including the flashing process log, automated operation screen recording files, and abnormal bus message data, enabling quick location and troubleshooting of problems.

[0087] In a specific feasible implementation, this method integrates the messaging robot service of the Lark collaboration platform, automatically calls the Lark Webhook API address to push structured test reports, and automatically sends notification messages to pre-defined Lark groups or individuals.

[0088] This embodiment of the automated testing method automates the entire process from code verification, test execution, flushing operations to result reporting by responding to and processing code submission events. Consistency verification ensures the correctness of the submitted code, automatic test triggering improves testing efficiency, environment snapshots facilitate problem tracing, flushing monitoring ensures the reliability of flushing, and correlation analysis and structured reports facilitate the management and analysis of test results. Compared with existing technologies, it greatly reduces reliance on manual intervention, improves the accuracy and standardization of testing, and meets the requirements for efficient, standardized, and automated testing.

[0089] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. An automated testing method for embedded software, characterized in that, Includes the following steps: Respond to code commit events in the remote repository, generate a unique tracking identifier for the current test process, and perform consistency verification on the executable files and source code in the committed content; Once the consistency verification is passed, static code analysis and unit testing are automatically triggered to obtain preliminary test results, which include a static code analysis report, the pass rate of unit test cases, and code coverage. Once the unit test passes, an environment snapshot of the current test environment and a corresponding snapshot identifier are generated. Based on the test environment, an automated flashing operation of the embedded software is performed, the flashing status of the flashing operation is monitored, and a flashing log is generated. Based on the unique tracking identifier, the submitted content, the initial test results, the snapshot identifier, and the write log are associated to generate a structured test report containing the association analysis results and push it to a preset terminal.

2. The automated testing method for embedded software according to claim 1, characterized in that, The response to the remote repository's code commit event generates a unique tracking identifier for the current test process, and performs consistency verification on the executable files and source code in the committed content, specifically including the following steps: When a code commit event is detected in the remote repository, a unique tracking identifier for the current test process is generated. The code commit event is an event that pushes the commit content to a preset branch of the remote repository. The commit content includes local code changes. The commit content is associated with the unique tracking identifier, which includes at least the timestamp of the code commit event and the unique identifier of the remote repository; The consistency verification process is triggered based on the code submission event and a verification result is obtained. The consistency verification process includes calculating the hash values ​​of the executable file and the source code in the submission content respectively. If the hash value of the executable file and the hash value of the source code are the same, the verification result is passed; otherwise, the verification result is failed. The verification result is associated with the unique tracking identifier.

3. The automated testing method for embedded software according to claim 1, characterized in that, Once the consistency verification passes, static code analysis and unit testing are automatically triggered to obtain preliminary test results, specifically including the following steps: Once the consistency verification is passed, static code analysis and unit testing will be automatically triggered by the continuous integration tool. The static code analysis includes a comprehensive scan of the code according to preset code style check rules, obtaining a static analysis report, and parsing the number of fatal errors and high-level warnings from the static analysis report as the first primary test data. The unit test includes calling a preset command-line interface to execute preset unit test cases, obtaining a unit test report, and parsing the test case pass rate and code coverage of the unit test cases from the unit test report as second primary test data; The first and second primary test data are compared with preset quality access control thresholds respectively; The unit test is considered to have passed when the number of fatal errors is zero, the test case pass rate reaches a first threshold, and the code coverage is not lower than a second threshold; otherwise, it fails.

4. The automated testing method for embedded software according to claim 2, characterized in that, Once the unit test passes, an environment snapshot of the current test environment and a corresponding snapshot identifier are generated, which specifically includes the following steps: Once the unit test passes, the environmental elements of the current test environment are captured. These environmental elements include system configuration information, dependent component status, and running application data. Containerization technology is used to generate an immutable image file containing these environmental elements as an environment snapshot. A unique hash value is calculated based on the hash algorithm to obtain the immutable image file, and the unique hash value is used as the snapshot identifier; Using the unique tracking identifier as an index, the snapshot identifier, image creation time, and image dependency list are stored to obtain an association relationship, which is used to locate the corresponding environment snapshot when backtracking test results.

5. The automated testing method for embedded software according to claim 3, characterized in that, The automated flashing operation of the embedded software based on the test environment, monitoring the flashing status of the flashing operation and generating flashing logs, specifically includes the following steps: The flashing tool is driven by automated control commands through a graphical user interface to perform the flashing operation. The flashing status is monitored in real time and the log file output by the flashing tool is read. The key status codes and progress information in the log file are extracted and associated with the unique tracking identifier. The progress control position and status prompt text on the flashing tool interface are captured by the graphical interface automation control tool, and the progress value corresponding to the progress control position and the status prompt text are associated and recorded with the unique tracking identifier. The key status codes and progress information in the log file are compared with the status prompt text and progress value obtained by the graphical interface automation control tool. If the two are consistent and the flashing status meets the preset flashing completion conditions, the flashing is determined to be successful; otherwise, the flashing fails.

6. The automated testing method for embedded software according to claim 5, characterized in that, If the flashing process is deemed successful, the following steps are also included: If the flashing is confirmed to be successful through the unique tracking identifier, the system test is automatically triggered, and the communication message data of the vehicle bus is collected synchronously, and the communication message data is used as part of the system test results. The system test results are associated with the unique tracking identifier.

7. The automated testing method for embedded software according to claim 6, characterized in that, The correlation analysis results include root cause localization hints. When the system test fails, generating the root cause localization hints specifically includes the following steps: Analyze failed system test cases and extract abnormal bus communication messages related to the failed system test cases. The abnormal bus communication messages include expected CAN bus signals and actual CAN bus signals. Based on the pre-configured signal and software component mapping table, locate the source code file that processes the abnormal bus communication message; Automatically retrieve and highlight all warnings that appear in the static analysis report and the code coverage in the unit tests for the source code file.

8. The automated testing method for embedded software according to claim 1, characterized in that, Pushing the structured test report to the preset terminal includes the following steps: The structured test report is sent to a predetermined group of terminals in a preset format. The structured test report includes at least the unique tracking identifier of the current test process, the test conclusion, and key quantitative indicator data. The test conclusion includes success or failure. When the test conclusion is failure, the unique tracking identifier in the structured test report is set as an interactive hyperlink, and the hyperlinked page displays all related data related to the failure of this test process.

9. An automated testing system for embedded software, characterized in that, Includes the following modules: The trigger verification module is used to respond to code commit events in the remote repository, generate a unique tracking identifier for the current test process, and perform consistency verification on the executable files and source code in the committed content; The test execution module is used to automatically trigger static code analysis and unit testing after the consistency verification is passed, and obtain preliminary test results, including static code analysis report, pass rate of unit test cases and code coverage. The flashing monitoring module is used to generate an environment snapshot and corresponding snapshot identifier of the current test environment after the unit test passes, execute an automated flashing operation of the embedded software based on the test environment, monitor the flashing status of the flashing operation and generate a flashing log. The associated push module is used to associate the submitted content, the initial test results, the snapshot identifier, and the write log based on the unique tracking identifier, generate a structured test report containing the association analysis results, and push it to a preset terminal.

10. The automated testing system for embedded software according to claim 9, characterized in that: The trigger verification module, the test execution module, the write monitoring module, and the associated push module are implemented by integrating a code management platform, a continuous integration server, static analysis tools, unit testing tools, write tools, and system testing tools. The integration of the flashing monitoring module with the flashing tool is achieved through a graphical interface automation adapter. The graphical interface automation adapter is used to simulate human-computer interaction by executing scripts when the flashing tool does not provide a command-line interface, thereby driving the graphical interface to complete automated flashing.

Images