OTA automatic test system and method

Abstract

The invention relates to the technical field of vehicle remote upgrade, and discloses an OTA automatic test system and method, the system comprises an OTA holder, a test holder and a tested device, the tested device is provided with a test application and an FOTA application, and is deployed with an OTA channel; the OTA holder is used for deploying upgrading information used by an actual OTA upgrading task; the test holder is used for configuring a test task for testing the OTA upgrading process; the test application is used for receiving a test task and performing test operation on the FOTA application according to the test task; the FOTA application is used for acquiring upgrading information from the OTA holder according to the test operation of the test application, and upgrading the OTA channel by using the upgrading information. After the test task is issued, the test holder does not need to participate in the upgrading process any more, decoupling of the test holder and the tested equipment is achieved, a large amount of communication interaction is reduced, comprehensive evaluation is conducted on the complexity of software deployment, hardware cost and site cost, and the cost of an automatic test scheme is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle remote upgrade, and in particular to an OTA automatic testing system and method. Background Art

[0002] As the level of automobile intelligence continues to improve, the iteration cycle of automobile software development is gradually shortened, the product repair and function iteration are getting faster and faster, and the importance of OTA (Over The Air technology) upgrades is gradually increasing. At present, one OTA upgrade system is that the OTA gimbal (cloud platform for OTA upgrade) sends the newly developed upgrade package to the vehicle side, and the vehicle side receives the upgrade package sent by the OTA gimbal through the FOTA (Firmware Over-The-Air) application, and can display the operation page. The user needs to perform a series of point-and-click operations on the operation page displayed by the FOTA application to determine the time and scope of the upgrade, and then the FOTA application controls the OTA channel of the vehicle side according to the user's operation to obtain some signal states through the OTA channel, make some judgments, and then copy the upgrade package to the corresponding vehicle-side firmware to complete the upgrade. When testing the OTA upgrade system, it is often necessary to run the above-mentioned real upgrade process, and the OTA gimbal is also required to send the upgrade data to the vehicle side, and then the tester manually operates the FOTA application for testing. The complexity of OTA testing and the long manual testing cycle need to be solved urgently, and automated testing is essential. At present, the related technology has an automated OTA testing system that provides a host computer that acts as a robot, establishes communication connections with the OTA gimbal and the vehicle, and controls the OTA gimbal and the vehicle step by step to exchange data and simulate manual operation of the vehicle upgrade interface until the test is completed. First of all, the host computer deployed in this solution requires a high degree of interaction between the ends. Each test step requires the host computer to send commands to the OTA gimbal or the vehicle, and the deployment is relatively complicated. In addition, because the host computer needs to control the vehicle to be tested, the distance between the host computer and the vehicle to be tested cannot be very far, resulting in high requirements for the space of the test scene. In general, the current automated testing solution has a high implementation cost. Summary of the invention

[0003] In view of this, the present invention provides an OTA automated testing system and method to solve the problem of high implementation cost of current OTA automated testing solutions.

[0004] In the first aspect, the present invention provides an OTA automatic testing system, including: an OTA gimbal, a test gimbal and a device under test, the device under test including at least one of a whole vehicle device and a bench device; the device under test is installed with a test application and a FOTA application, and an OTA channel is deployed; the OTA gimbal is used to deploy upgrade information used for actual OTA upgrade tasks; the test gimbal is used to configure test tasks for testing the OTA upgrade process; the test application is used to receive the test tasks, and perform test operations on the FOTA application according to the test tasks; the FOTA application is used to obtain upgrade information from the OTA gimbal according to the test operations of the test application, and use the upgrade information to perform upgrade operations on the OTA channel.

[0005] According to the above technical means, the test gimbal provided by the present invention is only used to configure the test task. In addition, the test application is deployed on the actual vehicle or the test bench under test. The test application is used to receive the test task sent by the test gimbal. The test task has been associated with the actual OTA upgrade task of the OTA gimbal, thereby ensuring the consistency of the test and the actual upgrade. After the test application receives the test task, the test application then controls the FOTA application on the actual vehicle or the test bench under test according to the instructions of the test task. The FOTA application actively requests the relevant upgrade information for the upgrade task from the OTA gimbal to perform the upgrade operation of the test. After the test task is issued, the test gimbal no longer needs to participate in the upgrade process, which realizes the decoupling of the test gimbal and the device under test, reduces a large amount of communication interaction, and significantly reduces the cost of the automated test solution from the comprehensive evaluation of the complexity of software deployment, hardware cost and site cost.

[0006] In some optional implementations, the system further includes a test bench engine, which is used to simulate vehicle sensors during the upgrade and provide the test bench device with message information required for the upgrade.

[0007] According to the above-mentioned technical means, when the device under test is a bench device, since the bench device does not have a complete vehicle structure like a whole vehicle, the present invention also provides a bench engine for simulating vehicle sensors, so that the bench device does not need to be installed with various complete vehicle sensors like a real vehicle. It is only necessary to simulate various sensor messages through the bench engine, thereby providing the bench device with the message information required for the upgrade, which significantly improves the flexibility of OTA automated testing.

[0008] In some optional implementations, the test gimbal includes: a configuration management module, a test case management module, a test scenario management module, a test task management module, a result analysis center module, a log management module and an abnormal result processing module; the configuration management module is used to configure the first test case of the test application and the second test case of the test bench engine; the test case management module is used to combine the first test case and the second test case to generate a number of complete standard test cases; the test scenario management module is used to generate a number of test scenarios based on one or more standard test cases; the test task management module is used to create a number of test tasks according to different test scenarios, and associate each test task with the actual OTA upgrade task in the OTA gimbal; the result analysis center module is used to analyze the use case execution results uploaded by the device under test; the log management module is used to store the test log files uploaded by the device under test; the abnormal result processing module is used to perform corresponding processing according to the analysis results of the result analysis center module.

[0009] According to the above technical means, a complete software architecture of the test gimbal is provided, including the test case configuration module of the test application and the test bench engine, and the function of how to use the test case to generate the test task. At the same time, the result analysis center module and the abnormal result processing module can analyze the data feedback from the test application, so as to detect the success and failure of the test task, and give warning information for the problems existing in the OTA upgrade system according to the detection results, and timely troubleshoot the problems to ensure the reliability of the OTA upgrade system.

[0010] In some optional implementations, the test application includes: a first network monitoring module, a first task parsing module, a first task execution module, a first data sorting module, a first log management module and an abstract interface module; the first network monitoring module is used to monitor the received test tasks; the first task parsing module is used to parse the test tasks into first test case instructions that can be executed by the test application; the first task execution module is used to call the abstract interface module by executing the first test case instructions; the abstract interface module is used to be called by the first task execution module to send instructions to the FOTA application for test operations, or to receive the FOTA application log, FOTA application screenshot data and the first use case execution result fed back by the FOTA application and the OTA channel; the first log management module is used to manage the first log file during the test process, and the first log file includes the log generated by the test application and the FOTA application log; the first data sorting module is used to sort the first log file, the first use case execution result and the FOTA application screenshot data, and upload the sorted data to the test gimbal.

[0011] According to the above technical means, a software architecture of the test application is provided, so that the test application can automatically generate program code for software execution according to the test steps described in the test task, thereby calling the abstract interface module to control the FOTA application and simulating the interactive operation between the user and the FOTA application to realize the upgrade process. During the test process, the FOTA application screenshot data can also be collected by calling the interface, and the execution result data returned by the FOTA application can be received, providing reliable data for the analysis of subsequent test results.

[0012] In some optional embodiments, the test bench engine includes: a second network monitoring module, a second task parsing module, a second task execution module, a second data sorting module, a second log management module and a message management module; the second network monitoring module is used to monitor the actual OTA upgrade task of the OTA channel, and the actual OTA upgrade task is the task that the OTA channel must execute according to the acquired upgrade information; the second task parsing module is used to parse out the message program instructions to ensure the smooth progress of the actual OTA upgrade task according to the second test case; the second task execution module is used to execute the message program instructions; the message management module is used to be called by the second task execution module, generate message information according to the message program instructions, and send it to the OTA channel; the second log management module is used to manage the second log file generated by the test bench engine during the upgrade process; the second data sorting module is used to sort the second log file and the second use case execution results generated by the test bench engine, and upload the sorted data to the test gimbal.

[0013] Based on the above-mentioned technical means, a software architecture of a test bench engine is provided. The software architecture of the test bench engine is basically the same as the test application. The difference is that the test bench engine calls the message management module according to the program code used for software execution, thereby simulating the vehicle sensor messages required by the test bench equipment, and can provide a complete test environment for accurate testing of the test bench equipment.

[0014] In the second aspect, the present invention provides an OTA automated testing method, which is applied to an OTA automated testing system, and the method includes: configuring a test task on a test gimbal, and issuing the test task to a test application of a device under test; the device under test receives the test task through the test application, and performs a test operation on a FOTA application according to the test task; the FOTA application obtains upgrade information from the OTA gimbal in response to the test operation, and the upgrade information is information used to deploy actual OTA upgrade tasks; the FOTA application performs an upgrade operation on an OTA channel using the upgrade information in response to the test operation; and the OTA channel executes a corresponding upgrade process in response to the upgrade operation.

[0015] According to the above-mentioned technical means, the test gimbal provided in the present invention is only used to configure test tasks. In addition, a test application is deployed on the actual vehicle under test or the test bench. The test application is used to receive the test task sent by the test gimbal. The test task has been associated with the actual OTA upgrade task of the OTA gimbal, thereby ensuring the consistency of the test and the actual upgrade. After receiving the test task, the test application actively requests relevant upgrade information for the upgrade task from the OTA gimbal, and then the test application controls the FOTA application to perform the test upgrade operation on the actual vehicle under test or the test bench according to the instructions of the test task. After the test task is issued, the test gimbal no longer needs to participate in the upgrade process, thereby realizing the decoupling of the test gimbal and the device under test, reducing a large amount of communication interactions, and comprehensively evaluating the complexity of software deployment, hardware costs and site costs, significantly reducing the cost of the automated testing solution.

[0016] In some optional implementations, when the device under test is a bench device, during the OTA channel's response to the upgrade operation to perform a corresponding upgrade process, the bench engine also includes: monitoring the status of the OTA channel, and providing the OTA channel with the message information required for the upgrade according to the status of the OTA channel during the upgrade process.

[0017] According to the above-mentioned technical means, when the device under test is a bench device, since the bench device does not have a complete vehicle structure like a whole vehicle, the present invention also provides a bench engine for simulating vehicle sensors, so that the bench device does not need to be installed with various complete vehicle sensors like a real vehicle. It is only necessary to simulate various sensor messages through the bench engine, thereby providing the bench device with the message information required for the upgrade, which significantly improves the flexibility of OTA automated testing.

[0018] In some optional implementations, a test task for testing the OTA upgrade process is configured on the test gimbal, and the test task is issued to the test application of the device under test, including: configuring a first test case of the test application and a second test case of the test bench engine through a configuration management module; combining the first test case and the second test case through a test case management module to generate several complete standard test cases; generating several test scenarios according to one or more standard test cases through a test scenario management module; creating several test tasks according to different test scenarios through a test task management module, and associating each test task with an actual OTA upgrade task in the OTA gimbal; detecting whether the device under test has other ongoing Other test tasks conflict with the target test tasks to be issued; if a conflict occurs, return to the step of detecting whether the device under test has other ongoing test tasks that conflict with the target test tasks to be issued; if no conflict occurs, detect whether the information of the device under test is consistent with the actual object under test, and the actual object under test is the object under test recorded in the actual OTA upgrade task in the OTA gimbal; if the detection is consistent with the actual object under test, issue the target test task to the test application; if the detection is inconsistent with the actual object under test, report the information of the device under test to the OTA gimbal, so that the OTA gimbal modifies the object under test recorded in the actual OTA upgrade task, and return to the step of detecting whether the information of the device under test is consistent with the actual object under test.

[0019] According to the above technical means, the test case configuration is realized based on the functional module of the test gimbal. In addition, before the test task is issued, it is also detected whether the device under test has other ongoing test tasks that conflict with the target test task to be issued. The test task is issued only if there is no conflict. Moreover, if the test gimbal detects that the actual object under test is inconsistent with the object under test recorded in the actual OTA upgrade task in the OTA gimbal, the object under test recorded by the OTA gimbal is also modified, further improving the accuracy and reliability of the test.

[0020] In some optional implementations, sending a target test task to a test application includes: determining whether the target test task is a test task for a bench device; if it is not a test task for a bench device, directly sending the target test task to the test application; if it is a test task for a bench device, sending a startup notification to the bench engine, so that the bench engine performs initialization startup in response to the startup notification, and feeds back a ready status signal to the test gimbal when initialization is successful; when a ready status signal is received, sending the target test task to the test application; when an initialization failure signal is received, stopping sending the target test task, the initialization failure signal being a signal sent after the bench engine has failed to initialize a preset number of times.

[0021] According to the above technical means, before the test task is issued, if the test task is a test task for a test bench device, the test bench engine must be initialized in advance to ensure that the test bench engine is in a usable ready state, thereby ensuring the reliability of the message simulation and further improving the reliability of the overall test.

[0022] In some optional implementations, a test operation is performed on a FOTA application according to a test task, including: parsing the target test task into a first test case instruction through a first task parsing module; executing the first test case instruction through a first task execution module, and calling an abstract interface module according to the first test case instruction to send an instruction for performing the test operation to the FOTA application.

[0023] In some optional implementations, before the FOTA application performs an upgrade operation on the OTA channel using the upgrade information in response to a test operation, it also includes: the FOTA application verifies whether the actual OTA upgrade task corresponding to the upgrade information is consistent with the target test task; if the verification is inconsistent, the test is ended; if the verification is consistent, the step of the FOTA application performing an upgrade operation on the OTA channel using the upgrade information in response to the test operation is executed.

[0024] According to the above technical means, before the FOTA application performs the upgrade operation, the consistency of the test task with the actual upgrade task in the OTA gimbal is further confirmed to ensure the effectiveness of the test, avoid the scenario where the test content is not actually used for the upgrade, and improve the reliability of the test.

[0025] In some optional embodiments, after the OTA channel completes the corresponding upgrade process in response to the upgrade operation, the method further includes: the OTA channel sends the first use case execution result to the FOTA application and the test application respectively; the FOTA application sends the first use case execution result and the FOTA application log generated by the FOTA application to the OTA gimbal; the test application sends the first log file, the collected FOTA application screenshot data and the first use case execution result to the test gimbal, and notifies the test gimbal of the end of the test; when the test gimbal receives the test end status, it sends a stop signal to the test bench engine; when the test bench engine receives the stop signal, it turns off monitoring of the OTA channel, and sends the second log file and the second use case execution result to the test gimbal; the test gimbal performs test result analysis on the received first log file, second log file, first use case execution result, second use case execution result and FOTA application screenshot data through the result analysis center module.

[0026] According to the above technical means, a technical logic is provided to control the test application to upload the test results to the OTA gimbal and the test gimbal, while the control rack engine automatically stops and uploads the corresponding test results to the test gimbal. While ensuring the link for uploading the original upgrade process data, the test data is also fully collected to the test gimbal, providing a reliability basis for subsequent test result analysis.

[0027] In some optional implementations, the test gimbal performs test result analysis on the received first log file, second log file, first use case execution result, second use case execution result, and FOTA application screenshot data through a result analysis center module, including: when the log files are normal and the use case execution results are successful, determining that the test is successful; when there are abnormal problems in the log files and the use case execution results are successful, assisting in analyzing whether the test is successful based on the FOTA application screenshot data; when the log files are normal and there are unsuccessful problems in the use case execution results, assisting in analyzing whether the test is successful based on the FOTA application screenshot data.

[0028] According to the above technical means, the present invention provides an algorithm for reliability analysis of test result data, which provides an accurate basis for judging the success or failure of OTA automatic testing and further improves the test reliability.

[0029] In some optional embodiments, when the log files are normal and the use case execution results are successful, the test is determined to be successful, including: when the log files are normal and the use case execution results are also successful, detecting whether there is a standard interface image in the image library; if there is no standard interface image, using the FOTA application screenshot data as the standard interface image, and determining that the test is successful; if there is a standard interface image, comparing the FOTA application screenshot data and the standard interface image; if the comparison difference value is less than or equal to a first preset threshold, the test is determined to be successful; if the comparison difference value is greater than the first preset threshold and less than or equal to the second preset threshold, the test is determined to be successful, and the user is notified to manually confirm whether to use the FOTA application screenshot data as the new standard interface image; if the comparison difference value is greater than the second preset threshold, the test is determined to be successful, and the user is notified to manually confirm whether to use the FOTA application screenshot data as the new standard interface image.

[0030] In some optional implementations, when there are abnormal problems in the log file and the use case execution results are all successful, the FOTA application screenshot data is used to assist in analyzing whether the test is successful, including: when there are abnormal problems in the log file and the use case execution results are all successful, detecting whether there is a standard interface image in the image library; if there is no standard interface image in the image library, notifying the user to troubleshoot the problem and determining that the test has failed; if there is a standard interface image in the image library, comparing the FOTA application screenshot data and the standard interface image; if the comparison difference value is less than or equal to the first preset threshold, notifying the user to troubleshoot the problem and determining that the test is successful; if the comparison difference value is greater than the first preset threshold and less than or equal to the second preset threshold, notifying the user to manually confirm whether to use the FOTA application screenshot data as the new standard interface image, notifying the user to troubleshoot the problem, and determining that the test is successful; if the comparison difference value is greater than the second preset threshold, determining that the test has failed, and notifying the user to troubleshoot the problem.

[0031] In some optional implementations, when the log files are normal and the use case execution results are unsuccessful, the FOTA application screenshot data is used to assist in analyzing whether the test is successful, including: when the log files are normal and the use case execution results are unsuccessful, detecting whether there is a standard interface image in the image library; if there is no standard interface image in the image library, notifying the user to troubleshoot the problem and determining that the test has failed; if there is a standard interface image in the image library, comparing the FOTA application screenshot data and the standard interface image; if the comparison difference value is less than or equal to a first preset threshold, notifying the user to troubleshoot the problem and determining that the test is successful; if the comparison difference value is greater than the first preset threshold and less than or equal to the second preset threshold, notifying the user to manually confirm whether to use the FOTA application screenshot data as a new standard interface image, notifying the user to troubleshoot the problem, and determining that the test is successful; if the comparison difference value is greater than the second preset threshold, determining that the test has failed, and notifying the user to troubleshoot the problem.

[0032] The technical solution provided by the present invention has the following advantages:

[0033] (1) According to the above technical means, the test gimbal provided by the present invention is only used to configure the test task. In addition, the test application is deployed on the actual vehicle or the test bench under test. The test application is used to receive the test task sent by the test gimbal. The test task has been associated with the actual OTA upgrade task of the OTA gimbal, thereby ensuring the consistency of the test and the actual upgrade. After the test application receives the test task, the test application then controls the FOTA application on the actual vehicle or the test bench under test according to the instructions of the test task. The FOTA application actively requests the relevant upgrade information for the upgrade task from the OTA gimbal to perform the upgrade operation of the test. After the test task is issued, the test gimbal no longer needs to participate in the upgrade process, which realizes the decoupling of the test gimbal and the device under test, reduces a large amount of communication interaction, and significantly reduces the cost of the automated test solution from the comprehensive evaluation of the complexity of software deployment, hardware cost and site cost.

[0034] (2) According to the above technical means, when the device under test is a bench device, since the bench device does not have a complete vehicle structure like a whole vehicle, the present invention also provides a bench engine for simulating vehicle sensors. Therefore, there is no need to install various complete vehicle sensors on the bench device like a real vehicle. It is only necessary to simulate various sensor messages through the bench engine to provide the bench device with the message information required for the upgrade, which significantly improves the flexibility of OTA automated testing.

[0035] (3) Based on the above technical means, a complete software architecture of the test gimbal is provided, including a test case configuration module for the test application and the test bench engine, and a function for generating test tasks using test cases. At the same time, the result analysis center module and the abnormal result processing module can analyze the data feedback from the test application, thereby detecting the success and failure of the test task, and giving warning information for problems existing in the OTA upgrade system based on the detection results, timely troubleshooting the problems, and ensuring the reliability of the OTA upgrade system.

[0036] (4) Based on the above technical means, a software architecture of the test application is provided, so that the test application can automatically generate program code for software execution according to the test steps described in the test task, thereby calling the abstract interface module to control the FOTA application and simulating the interactive operation between the user and the FOTA application to realize the upgrade process. During the test process, the FOTA application screenshot data can also be collected by calling the interface, and the execution result data returned by the FOTA application can be received, providing reliable data for the analysis of subsequent test results.

[0037] (5) Based on the above technical means, a software architecture of a test bench engine is provided. The software architecture of the test bench engine is basically the same as the test application. The difference is that the test bench engine calls the message management module according to the program code used for software execution, thereby simulating the vehicle sensor messages required by the test bench equipment, and can provide a complete test environment for accurate testing of the test bench equipment.

[0038] (6) According to the above technical means, the test case configuration is realized based on the functional module of the test gimbal. In addition, before the test task is issued, it is also detected whether the device under test has other ongoing test tasks that conflict with the target test task to be issued. The test task is issued only if there is no conflict. Moreover, if the test gimbal detects that the actual object under test is inconsistent with the object under test recorded in the actual OTA upgrade task in the OTA gimbal, the object under test recorded by the OTA gimbal is also modified, further improving the accuracy and reliability of the test.

[0039] (7) According to the above technical means, before the test task is issued, if the test task is a test task of a test bench device, the test bench engine must be initialized in advance to ensure that the test bench engine is in a usable ready state, thereby ensuring the reliability of the message simulation and further improving the reliability of the overall test.

[0040] (8) According to the above technical means, before the FOTA application executes the upgrade operation, the consistency of the test task and the actual upgrade task in the OTA gimbal is further confirmed to ensure the effectiveness of the test, avoid the test content is not the scene used for the actual upgrade, and improve the reliability of the test.

[0041] (9) Based on the above technical means, a technical logic is provided to control the test application to upload the test results to the OTA gimbal and the test gimbal, and at the same time control the rack engine to automatically stop and upload the corresponding test results to the test gimbal. While ensuring the link for uploading the original upgrade process data, the test data is also fully collected to the test gimbal, providing a reliability basis for subsequent test result analysis.

[0042] (10) Based on the above technical means, the present invention provides an algorithm for reliability analysis of test result data, which provides an accurate basis for judging the success or failure of OTA automatic testing and further improves the test reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 is a structural schematic diagram of an OTA automated testing system according to an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of the structure of a test pan / tilt according to an embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of the effect of the first test case according to an embodiment of the present invention;

[0047] Figure 4 is a schematic diagram of the effect of the second test case according to an embodiment of the present invention;

[0048] Figure 5 is a schematic diagram of the structure of a test application according to an embodiment of the present invention;

[0049] Figure 6 is a schematic diagram of the structure of a test bench engine according to an embodiment of the present invention;

[0050] Figure 7 2 is a schematic diagram of the effect of a state machine of a FOTA application according to an embodiment of the present invention;

[0051] Figure 8 is a flow chart of an OTA automated testing method according to an embodiment of the present invention;

[0052] Fig. 9 is another flow chart of an OTA automated testing method according to an embodiment of the present invention;

[0053] Fig.10 is another flow chart of an OTA automated testing method according to an embodiment of the present invention;

[0054] Fig.11 is another flow chart of an OTA automated testing method according to an embodiment of the present invention;

[0055] Fig.12 is another flow chart of an OTA automated testing method according to an embodiment of the present invention;

[0056] Fig.13 is another flow chart of an OTA automated testing method according to an embodiment of the present invention;

[0057] Fig.14 is another flow chart of an OTA automated testing method according to an embodiment of the present invention;

[0058] Fig.15 1 is another flow chart of an OTA automated testing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0060] According to an embodiment of the present invention, an OTA automated testing system is provided. Figure 1 As shown, the system includes: an OTA gimbal 20, a test gimbal 10 and a device under test. The device under test includes at least one of a complete vehicle device and a bench device. The complete vehicle device refers to a complete assembled vehicle, and the bench device refers to a simple component that only contains necessary test hardware but is not installed on the vehicle. In this embodiment, the device under test is installed with a test application 30 and a FOTA application 40, and an OTA channel 50 is deployed. In the embodiment of the present invention, the OTA platform 20 refers to a cloud platform used to publish various upgrade information during conventional OTA upgrades. The OTA platform 20 is used to deploy the upgrade information used for the actual OTA upgrade task; and the test platform 10 is a cloud platform specifically set up for testing the OTA upgrade process. In this embodiment, the test platform 10 is used to configure a test task for testing the OTA upgrade process. The test task includes a description of the test process and a description of the actual OTA upgrade task. The test task is used to inform the test application 30 of the device under test of the test steps to be performed during the upgrade test, which installation packages and status information in the OTA platform are required, etc. The test task does not include all the upgrade information of the actual OTA upgrade task, for example, the installation package entity for the upgrade is still stored in the OTA platform 20 without being transmitted to the test platform. The test application 30 is a software module deployed in the device under test, which is used to receive the test task and perform test operations on the FOTA application 40 according to the test task. The FOTA application 40 is an indispensable software module in conventional OTA upgrades. In the related art, the FOTA application 40 controls the OTA channel 50 on the vehicle side according to the user's interactive operation in the operation interface, thereby completing the test. The test operation of the test application 30 provided in the embodiment of the present invention acts on the FOTA application 40, and the FOTA application 40 will receive instructions corresponding to the test operation, so that the FOTA application 40 is also used to obtain upgrade information from the OTA gimbal 20 according to the test operation of the test application 30, and use the upgrade information to perform upgrade operations on the OTA channel 50.

[0061] According to the above technical means, the test gimbal 10 provided by the present invention is only used to configure the test task. In addition, the test application 30 is deployed on the actual vehicle under test or the test bench. The test application 30 is used to receive the test task sent by the test gimbal 10. The test task has been associated with the actual OTA upgrade task of the OTA gimbal 20, thereby ensuring the consistency of the test and the actual upgrade. After the test application 30 receives the test task, the test application 30 then controls the FOTA application 40 on the actual vehicle under test or the test bench according to the instructions of the test task. The FOTA application 40 actively requests the relevant upgrade information for the upgrade task from the OTA gimbal 20 to perform the upgrade operation of the test. After the test task is issued, the test gimbal 10 no longer needs to participate in the upgrade process, which realizes the decoupling of the test gimbal 10 and the device under test, reduces a large amount of communication interaction, and significantly reduces the cost of the automated test solution from the comprehensive evaluation of the complexity of software deployment, hardware cost and site cost.

[0062] In some optional embodiments, Figure 1 As shown, the system further includes a test bench engine 60, which is used to simulate vehicle sensors during upgrading and provide the test bench device with message information required for upgrading.

[0063] Specifically, when the device under test is a bench device, since the bench device does not have a complete vehicle structure like a complete vehicle, the present invention also provides a bench engine 60 for simulating vehicle sensors, so that the bench device does not need to be installed with various complete vehicle sensors like a real vehicle. The bench engine 60 can be a software engine module installed on a vehicle, a specific server or a specific computer. The bench engine 60 performs corresponding environmental simulation by monitoring the OTA channel 50 and the FOTA application 40 state machine state of the bench device. It is only necessary to simulate various sensor messages through the bench engine 60 to provide the bench device with the sensor message information required for the upgrade, thereby significantly improving the flexibility of OTA automated testing.

[0064] In some optional embodiments, Figure 2As shown, the test gimbal 10 includes: a configuration management module 101, a test case management module 102, a test scenario management module 103, a test task management module 104, a result analysis center module 105, a log management module 106 and an abnormal result processing module 107; wherein the configuration management module 101 is used to configure the first test case of the test application 30 and the second test case of the test bench engine 60; wherein the test case management module 102 is used to combine the first test case and the second test case to generate a number of complete standard test cases; wherein the test scenario management module 103 is used to generate a number of test scenarios according to one or more standard test cases; wherein the test task management module 104 is used to create a number of test tasks according to different test scenarios, and associate each test task with the actual OTA upgrade task in the OTA gimbal 20; wherein the result analysis center module 105 is used to analyze the execution results of the use case uploaded by the device under test; wherein the log management module 106 is used to store the test log file uploaded by the device under test; wherein the abnormal result processing module 107 is used to perform corresponding processing according to the analysis results of the result analysis center module 105.

[0065] Specifically, the configuration management module 101 of the embodiment of the present invention is a module for configuring test cases. The configuration management mainly manages the mapping relationship between text cases in text form and script cases in code form, specifically including the first test case of the test application 30 and the second test case of the test bench engine 60. The configuration management module 101 mainly includes four configuration capabilities: preconditions, test steps, expected results, and dictionary management. For example Figure 3 As shown, it is a first test case configuration diagram for the test application 30, which mainly includes preconditions, test steps, expected results, corresponding Command IDs (command IDs) and corresponding text case semantics. When the user creates a case, the text case is displayed, and when the task case data is issued, the command ID is used. The issued command is defined, executed, and judged in the test application 30. Among them, the precondition refers to the state condition that must be met to execute the current command, such as Figure 3 The configuration phase test to be performed in the test must be normal on the vehicle network and the FOTA application must be idle. The test steps refer to the order in which the test commands are executed. The expected result refers to the feedback information expected to be received after the command is executed, which can be used as a basis for judging whether the test is successful.

[0066] like Figure 4As shown, it is a schematic diagram of the second test case configuration for the test bench engine 60. For example, it includes a monitoring script and an execution script. The monitoring script corresponds to the precondition of the use case dimension of this article. When a certain state or a certain signal is monitored, the corresponding execution script will be run. The execution script corresponds to the operation steps and expected results of the text use case dimension. The script executes certain signal sending, relay control and other operations to achieve test environment simulation and generate the required message information. Dictionary management refers to Figure 3 , Figure 4 The mapping relationship of this type of table is the mapping relationship between text use cases and script use cases.

[0067] For example, in one specific embodiment, Figure 4 The immediate upgrade and vehicle lock upgrade in the monitoring script refer to the content that needs to be monitored during the immediate upgrade and vehicle lock upgrade tests. For example, the data structure is {"monitorContent":{"otaType":4.0,"value":17}}, where 17 refers to a state of the FOTA application state machine, specifically "VEHICLE_INSTALLING_STATE". Different OTA architecture states are also compatible here. For a detailed explanation of the concept of the FOTA application state machine, see below. Figure 7 Related description. The "otaType" field refers to the architecture platform of OTA channel 50, such as OTA channel 50 of version 4.0. The monitoring script also has another data structure of monitoring message {"monitorContent":{"channel":2,"diagnosisId":1989,"respId":1997,"value":"67 02"}}. This data structure indicates the corresponding data item channel number 2 for monitoring. The dictionary table of "channel" is the data item "channel", 2 corresponds to "PTCAN500K", "diagnosisId" is the diagnosis Id, which is used to indicate the name of the controller, corresponding to the decimal name of the controller, converting 1989 to hexadecimal as 7C5. "respId" is the response Id of the control, which is hexadecimal. Value refers to the data replied by the monitoring response Id. If the data is detected, the corresponding execution script is run. For example, if the immediate upgrade stage is detected, the execution script for immediate upgrade is started. With Figure 4For example, the data structure of the execution script in the immediate upgrade phase is {"type":"message","data":[{"id":"0x5FD","channel":"BCAN500K","data":"00 10 00 00 00 00 00 00","type":"can","times":"-1","interval":"10","name":"BatteryAccuracy","delay":"0"},{......}]}. This data structure is the specific content executed when the execution script is running. "type" is mainly divided into message and power. Message is information control data, which is mainly executed by message collection and sending equipment. Power is power control data, indicating power-off and power-on operations. Data is specific data. In the specific data: "id" is the diagnostic id, "data" is the message data, "type" is the message type (such as CAN, CANFD, LIN, ETH, etc.), "times" indicates the number of times sent (for example, -1 means infinite times), "name" is the description of the message data, and the signal filled in here indicates the battery progress signal. "delay" is the time required to delay sending before executing the script (this scenario is suitable for some situations where you enter a certain state but need to wait for a certain time before starting the operation, such as power failure during downloading, entering the download stage and needing to wait for the download address to obtain the operation before actually entering the download, etc., at this time a certain time delay is required). The second type of data structure for executing the script is {"type":"power","data":[{"com":"com16","baudrate":9600,"channel":"channel1","state":"off"},{"com":"com16","baudrate":9600,"channel":"channel1","state":"on"}],"time":5}. This data structure represents the power control class. The data content is the COM port, baud rate and channel information. "state" represents the power control state (off means off, on means on). "time" represents the interval between two executions in seconds.

[0068] For example, in a specific embodiment, Figure 3The first test case shown is mainly divided into 7 stages, namely the configuration stage, software information collection stage, version detection stage, download stage, deployment stage, installation stage and end stage. The commands of the corresponding stage (preconditions, test steps, expected results) will be executed in each specific stage. For example, in the download stage, "the car is in the mobile network state" can be set as a precondition, and the use case can choose to execute the T_EXP_403 command "automatically download in the mobile network state". One test step can correspond to the execution of multiple expected results, such as T_CMD_202 "Get Submaster version collection status", which can correspond to the execution of T_EXP_202, T_EXP_203, and T_EXP_204. The above use cases are executed based on the information callback by FOTA application 40, and can be further expanded according to business needs.

[0069] The test case management module 102 combines the first test case of the test application 30 and the second test case of the test bench engine 60 according to user requirements, configures them into standard test cases, and manages them in a unified manner.

[0070] The test scenario management module 103 is used to combine one or more standard test cases into a test scenario. The test scenario can be repeatedly referenced and used by different tasks. The test scenario management module 103 performs operations such as creating, editing, and deleting a scenario.

[0071] The test task management module 104 is used to create test tasks. When creating a test task, it is necessary to select information such as the test scenario, vehicle model, upgrade object, upgrade version, test object, and stress test times. The test task management can provide editing capabilities for the created test tasks. When creating a test task, the test gimbal 10 needs to synchronize the upgrade information of the actual OTA upgrade task of the OTA gimbal 20 (including but not limited to OTA task ID, major version number, upgrade object, version number, etc.) and associate the OTA upgrade task ID with the test task ID to ensure the consistency of the test task and the actual OTA upgrade task during the test process.

[0072] The result analysis center module 105 is used to analyze the FOTA application logs, use case execution results, FOTA application image data, test bench engine task result data and logs uploaded by the test application 30, including the success or failure of the tasks and use cases, success rate, failure rate, upgrade rate, upgrade time, image comparison results, log analysis results, etc., and finally display them visually through the front-end page.

[0073] The log management module 106 stores the execution logs of the test application 30 and the test engine 60 to facilitate troubleshooting and log recording.

[0074] The abnormal result processing module 107 will store the error code and error cause information of the test application 30 and the test bench engine 60. If an abnormal result occurs and a defect is found, an abnormal analysis will be performed (through error codes, message data, log data, etc.), and the classification and attribution of the defects will be automatically identified and analyzed. Automatic bill of lading will be performed, corresponding information will be filled in, and corresponding log data will be uploaded.

[0075] According to the above technical means, the embodiment of the present invention provides a complete software architecture of the test gimbal 10, including the test case configuration module of the test application 30 and the test bench engine 60, and provides a function of how to use the test case to generate a test task. At the same time, the result analysis center module 105 and the abnormal result processing module 107 can analyze the data feedback from the test application 30, thereby detecting the success and failure of the test task, and giving warning information for the problems existing in the OTA upgrade system according to the detection results, timely troubleshooting the problems, and ensuring the reliability of the OTA upgrade system.

[0076] In some optional implementations, the test application 30 includes: a first network monitoring module 301 , a first task parsing module 302 , a first task execution module 304 , a first data sorting module 305 , a first log management module 303 and an abstract interface module 306 . The first network monitoring module 301 is used to monitor the received test tasks; the first task parsing module 302 is used to parse the test tasks into the first test case instructions that can be executed by the test application 30; the first task execution module 304 is used to call the abstract interface module 306 by executing the first test case instructions; the abstract interface module 306 is used to be called by the first task execution module 304 to send instructions to the FOTA application 40 for test operations, or to receive the FOTA application log, FOTA application screenshot data and the first use case execution result fed back by the FOTA application 40 and the OTA channel 50; the first log management module 303 is used to manage the first log file during the test process, and the first log file includes the log generated by the test application 30 and the FOTA application log; the first data sorting module 305 is used to sort the first log file, the first use case execution result and the FOTA application screenshot data, and upload the sorted data to the test gimbal 10.

[0077] Specifically, the software framework of the test application 30 is as follows: Figure 5As shown, the first network monitoring module 301 will monitor the task, mainly monitoring the task issuance, start and end, and the monitored data and request will be given to the first task parsing module 302. Because the test case is a text description in the test cloud platform 10, in order to facilitate data transmission, the command ID is issued, but the command ID does not belong to the program code directly executable by the test application 30, so the first task parsing module 302 is used to parse the monitored task data and convert it into a program code form executable by the test application 30, that is, the first test case instruction, which mainly includes the preconditions, operation steps, expected results and stress measurement information of the test case. The first task execution module 304 executes the parsed instructions and calls the abstract interface to obtain data for execution assertion. The abstract interface module 306 is an interface defined together with the FOTA application 40, which can receive the callback data of the FOTA application 40, directly call the open interface of the FOTA application, and call the system screenshot function to save the FOTA application screenshot data during the task execution process, thereby recording the interactive operation process of the test application 30 to the FOTA application 40. The first log management module 303 records the entire test process log and the FOTA application execution log, which is collectively referred to as the first log file in this embodiment. The first data sorting module 305 sorts the test case execution data in the test task, including the first test case execution result obtained after the first test case is executed. The first data sorting module 305 also uploads the execution log of the FOTA application 40, the test case execution result, and the FOTA application screenshot data to the test gimbal 10 at the same time.

[0078] According to the above technical means, the present invention provides a software architecture of a test application 30, so that the test application 30 can realize the test process such as automatically generating program code for software execution and calling the abstract interface module 306 to control the FOTA application 40 according to the description of the test task, and automatically simulate the interactive operation between the user and the FOTA application 40 at the end of the tested device to realize the upgrade process. During the test process, the FOTA application screenshot data can also be collected by calling the abstract interface module 306, and the execution result data returned by the FOTA application 40 is received, so as to provide reliable data for the analysis of subsequent test results.

[0079] In some optional implementations, the test bench engine 60 includes: a second network monitoring module 601, a second task parsing module 602, a second task execution module 604, a second data sorting module 605, a second log management module 603 and a message management module 606; the second network monitoring module 601 is used to monitor the actual OTA upgrade task of the OTA channel 50, and the actual OTA upgrade task is the task that the OTA channel 50 is to execute according to the acquired upgrade information; the second task parsing module 602 is used to parse out the message program instructions to ensure the smooth progress of the actual OTA upgrade task according to the second test case; the second task execution module 604 is used to execute the message program instructions; the message management module 606 is used to be called by the second task execution module 604, generate message information according to the message program instructions, and send it to the OTA channel 50; the second log management module 603 is used to manage the second log file generated by the test bench engine 60 during the upgrade process; the second data sorting module 605 is used to sort the second log file and the second use case execution result generated by the test bench engine 60, and upload the sorted data to the test gimbal 10.

[0080] Specifically, Figure 6 As shown, it is the software framework of the test bench engine 60. In the embodiment of the present invention, the overall structure of the test bench engine 60 is substantially the same as that of the test application 30. The difference lies in the monitored task data structure, the script to be executed, the specific driver execution module, the message management module 606 and the abstract interface module 306 of the test application 30. The functional structure of other modules is consistent with the test application 30 and will not be repeated here. In this embodiment, the test bench engine 60 mainly performs corresponding environmental simulation by monitoring the status signal of the OTA channel 50 and the status signal of the FOTA application state machine. The FOTA application state machine refers to the test phases of the FOTA application 40 divided according to the operation order, and each phase corresponds to a status signal, for example Figure 7 As shown, the FOTA application state machine is generally divided into 19 states, which will be synchronized to the test platform 10 and then to the test bench engine 60 by the test application 30. The test bench engine 60 can simulate the stage environment by monitoring these states, and the test application 30 will also execute specific commands according to these states. It should be noted that in the embodiment of the present invention, the test bench engine 60 obtains the state signal of the FOTA application state machine, not directly interacting with the FOTA application 40, but the FOTA application 40 sends the state signal to the test application 30, and the test application 30 sends it to the test platform 10. The test bench engine 60 obtains the state signal of the FOTA application state machine from the test platform 10. The purpose of this is to reduce the coupling between the FOTA application 40 and other software. When the degree of software coupling is higher, problems are more likely to occur during the test process. The setting means of the embodiment of the present invention are conducive to improving the reliability of the test process.

[0081] The message management module 606 is mainly responsible for message tool management and message sending and receiving management. It is connected to the test bench bus through tools such as ZLG, CANoe, and LIN, and can simulate CAN bus messages, LIN bus messages, Ethernet messages, etc. The test bench engine 60 provided by the embodiment of the present invention, when monitoring the OTA channel 50 to start the OTA upgrade task, realizes the simulation of the vehicle sensor messages required by the test bench equipment, and can provide a complete test environment for the accurate testing of the test bench equipment, further improves the reliability of the test, reduces the installation of real sensors, and reduces the test cost.

[0082] According to an embodiment of the present invention, an OTA automated testing method embodiment is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0083] In this embodiment, an OTA automated testing method is provided, which can be used in the above-mentioned OTA automated testing system. Figure 8 is a flow chart of an OTA automated testing method according to an embodiment of the present invention, the process comprising the following steps:

[0084] Step S801, configuring a test task on the test platform, and sending the test task to the test application of the device under test;

[0085] Step S802, the device under test receives the test task through the test application, and performs a test operation on the FOTA application through the test application according to the test task;

[0086] Step S803, the FOTA application obtains upgrade information from the OTA gimbal in response to the test operation, where the upgrade information is information used to deploy the actual OTA upgrade task;

[0087] Step S804, the FOTA application performs an upgrade operation on the OTA channel using the upgrade information in response to the test operation;

[0088] Step S805: The OTA channel executes a corresponding upgrade process in response to the upgrade operation.

[0089] Specifically, the principle of the automated OTA test in steps S801 to S805 of this embodiment may refer to the relevant description of the above system embodiment, which will not be repeated here.

[0090] In some optional implementations, when the device under test is a bench device, when executing step S805, the method further includes:

[0091] Step a1: the test bench engine monitors the status of the OTA channel and provides the OTA channel with the message information required for the upgrade according to the status of the OTA channel during the upgrade process.

[0092] Specifically, when the device under test is a bench device, since the bench device does not have a complete vehicle structure like a whole vehicle, it may lack various vehicle sensors, which affects the test. The present invention simulates various sensor messages through the bench engine, thereby providing the bench device with the message information required for the upgrade, which significantly improves the flexibility of OTA automated testing. For the technical principles of the bench engine, please refer to the relevant description of the above system embodiment, which will not be repeated here.

[0093] In some optional implementations, the above step S801 includes:

[0094] Step b1, configuring a first test case of the test application and a second test case of the test bench engine through a configuration management module;

[0095] Step b2, combining the first test case and the second test case through the test case management module to generate several complete standard test cases;

[0096] Step b3, generating a number of test scenarios according to one or more standard test cases through a test scenario management module;

[0097] Step b4, creating several test tasks according to different test scenarios through the test task management module, and associating each test task with the actual OTA upgrade task in the OTA gimbal;

[0098] Step b5, detecting whether the device under test has other ongoing test tasks that conflict with the target test task to be issued;

[0099] Step b6: if a conflict occurs, return to the step of detecting whether the device under test has other ongoing test tasks that conflict with the target test task to be issued;

[0100] Step b7: if no conflict occurs, detecting whether the information of the device under test is consistent with the actual object under test, where the actual object under test is the object under test recorded in the actual OTA upgrade task in the OTA gimbal;

[0101] Step b8: If the detection is consistent with the actual object under test, the target test task is sent to the test application;

[0102] Step b9, if the detected device is inconsistent with the actual object under test, the information of the device under test is reported to the OTA gimbal, so that the OTA gimbal modifies the object under test recorded in the actual OTA upgrade task, and returns to the step of detecting whether the information of the device under test is consistent with the actual object under test.

[0103] Specifically, regarding the part of creating the test task described in steps b1 to b4, its principle explanation can refer to the relevant description of the above-mentioned system embodiment, which will not be repeated here. After the test task is created, the embodiment of the present invention also performs task conflict detection and object detection before issuing the test task. Among them, it is detected whether the device under test has other ongoing test tasks that conflict with the target test task to be issued. The current test task is issued only if the condition of non-conflict is met, otherwise the detection is performed again until there is no conflict. In addition, if the test gimbal detects that the actual object under test (such as the actual vehicle model or chip model under test) is inconsistent with the object under test recorded in the actual OTA upgrade task in the OTA gimbal (such as the vehicle model or chip model under test recorded in the OTA gimbal), the embodiment of the present invention also modifies the object under test recorded by the OTA gimbal to ensure the consistency of the upgrade task and the test task of the OTA gimbal, further improving the accuracy and reliability of the test.

[0104] In some optional implementations, the above step b8 includes:

[0105] Step c1, determining whether the target test task is a test task for a bench device;

[0106] Step c2: If it is not a test task for the bench device, directly send the target test task to the test application;

[0107] Step c3, if it is a test task of a bench device, a start notification is sent to the bench engine, so that the bench engine performs initialization startup in response to the start notification, and feeds back a ready state signal to the test pan-tilt when the initialization is successful;

[0108] Step c4: when a ready state signal is received, the target test task is sent to the test application;

[0109] Step c5, when an initialization failure signal is received, stop sending the target test task. The initialization failure signal is a signal sent after the test bench engine has failed to initialize a preset number of times.

[0110] Specifically, before the test task is issued, the embodiment of the present invention first determines whether the test task to be executed is executed on the test bench device or on the actual vehicle. If the test task is a test task for the test bench device, considering that the test bench test requires the use of a test bench engine, if the test bench engine is restarted during the execution of the test task, the engine may fail to start or start incorrectly, resulting in the message information not being sent correctly, thereby causing the test task to crash. Therefore, this embodiment also needs to initialize the test bench engine in advance to ensure that the test bench engine is in a usable ready state. If the test bench engine is initialized successfully, the test task is issued again. If the initialization fails once, the initialization is repeated. When the preset number of initializations (for example, 3 times) are all unsuccessful, the task is terminated. The solution provided by this embodiment ensures the reliability of the message simulation and further improves the reliability of the overall test.

[0111] In some optional implementations, the above step S802 includes:

[0112] Step d1, parsing the target test task into a first test case instruction through a first task parsing module;

[0113] Step d2, executing the first test case instruction through the first task execution module, and calling the abstract interface module according to the first test case instruction, sending an instruction for performing a test operation to the FOTA application.

[0114] Specifically, regarding the principle of the test application performing the test operation on the FOTA application, reference may be made to the relevant description of the above system embodiment, which will not be repeated here.

[0115] In some optional implementation manners, before step S804, the method further includes:

[0116] Step e1, the FOTA application verifies whether the actual OTA upgrade task corresponding to the upgrade information is consistent with the target test task;

[0117] Step e2, if the verification is inconsistent, end the test;

[0118] Step e3, if the verification is consistent, execute the step of the FOTA application performing an upgrade operation on the OTA channel using the upgrade information in response to the test operation.

[0119] Specifically, before the FOTA application executes the upgrade operation, the embodiment of the present invention further confirms the consistency of the test task with the actual upgrade task in the OTA gimbal to further ensure the effectiveness of the test and avoid scenarios where the test content is not actually used for the upgrade, thereby further improving the reliability of the test.

[0120] In some optional implementations, after step S805, the method further includes:

[0121] Step f1, the OTA channel sends the first use case execution result to the FOTA application and the test application respectively;

[0122] Step f2, the FOTA application sends the first use case execution result and the FOTA application log generated by the FOTA application to the OTA gimbal;

[0123] Step f3, the test application sends the first log file, the collected FOTA application screenshot data and the first use case execution result to the test gimbal, and notifies the test gimbal that the test is finished;

[0124] Step f4, when the test gimbal receives the test end status, it sends a stop signal to the test bench engine;

[0125] Step f5, when the test bench engine receives the stop signal, it turns off the monitoring of the OTA channel, and sends the second log file and the second use case execution result to the test gimbal;

[0126] Step f6, the test gimbal performs test result analysis on the received first log file, second log file, first use case execution result, second use case execution result and FOTA application screenshot data through the result analysis center module.

[0127] Specifically, after the test is completed, the embodiment of the present invention also uploads the use case execution results and log files in the test process to the test platform, so as to facilitate data analysis and determine the success or failure of this test. Among them, the first use case execution result after the OTA channel test is sent in two ways, one way to the FOTA application and the other way to the test application. Among them, the FOTA application will send the first use case execution result and the FOTA application log generated by itself to the OTA platform, that is, step f2. Step f2 is the data feedback process of the original process of OTA upgrade, and this test scheme is still retained to ensure the integrity of the test. In addition, from f3 to step f5, the test application collects the first log file composed of the log generated by the test application and the FOTA application log, the FOTA application screenshot data obtained by taking a screenshot of the FOTA application during the test, and the aforementioned first use case execution result, and sends the collected content to the test platform. After receiving the above content, the test platform identifies the end of the test task, thereby notifying the bench engine to stop running. In addition, after the bench engine receives the stop notification, it will organize the second log file and the second use case execution result in the test process, and feedback to the test platform before the bench engine stops. Based on the above steps, a technical logic is provided to control the test application to upload the test results to the OTA gimbal and the test gimbal, and at the same time control the rack engine to automatically stop and upload the corresponding test results to the test gimbal. While ensuring the link for uploading the original upgrade process data, the test data is also fully collected to the test gimbal, which provides a reliability basis for the test result analysis step in the subsequent step f6 and improves the reliability of the test.

[0128] In some optional implementations, the above step f6 includes:

[0129] Step g1: When all log files are normal and the test case execution results are successful, the test is determined to be successful;

[0130] Step g2: when the log file has abnormal problems and the test case execution results are all successful, the FOTA application screenshot data is used to assist in analyzing whether the test is successful;

[0131] Step g3: When the log files are normal but the test case execution results are unsuccessful, the FOTA application screenshot data is used to assist in analyzing whether the test is successful.

[0132] According to the above technical means, the present invention provides an algorithm for reliability analysis of test result data, which is divided into three main situations for analysis. The first is that the log and the use case execution results are normal. At this time, the impact of the FOTA application screenshot data does not need to be considered, and the test can be determined to be a successful result. The second is that the log is abnormal but the use case is normal, and the third is that the use case is abnormal but the log is normal. Both of these situations cannot directly determine whether the test is successful. At this time, the interactive operations of the test application and the FOTA application should be analyzed with the help of the FOTA application screenshot data to determine whether they are all executed correctly, so as to assist in analyzing whether the test is successful. The algorithm for reliability analysis of test result data provided by the embodiment of the present invention provides an accurate basis for judging the success or failure of OTA automatic testing, and further improves the reliability of the test.

[0133] In some optional implementations, the above step g1 includes:

[0134] Step h1, when the log file is normal and the use case execution result is successful, check whether there is a standard interface image in the image library;

[0135] Step h2: if there is no standard interface image, the FOTA application screenshot data is used as the standard interface image, and the test is determined to be successful;

[0136] Step h3: if there is a standard interface image, compare the FOTA application screenshot data with the standard interface image;

[0137] Step h4: if the comparison difference value is less than or equal to the first preset threshold, the test is determined to be successful;

[0138] Step h5: if the comparison difference value is greater than the first preset threshold value and less than or equal to the second preset threshold value, the test is determined to be successful, and the user is notified to manually confirm whether to use the FOTA application screenshot data as the new standard interface image;

[0139] Step h6: If the comparison difference value is greater than the second preset threshold, the test is determined to be successful, and the user is notified to manually confirm whether to use the FOTA application screenshot data as the new standard interface image.

[0140] Specifically, when the log file is normal and the use case execution result is also successful, the embodiment of the present invention can directly determine that the test is successful, in order to further improve the reliability of the next round of testing. The embodiment of the present invention also detects whether there is a standard interface diagram for auxiliary analysis in the image library of the test gimbal. If there is no standard interface diagram, the FOTA application screenshot data of this time can be used as a standard interface diagram, which is convenient for the next round of testing and the FOTA application screenshot data collected in the next round to compare, thereby realizing auxiliary analysis. If there is a standard interface diagram, the FOTA application screenshot data of this round is compared with the standard interface diagram. If the comparison difference value is less than or equal to the first preset threshold, it is directly determined that the test is successful, and there is no need to adjust the standard interface diagram, wherein the first preset threshold is the agreed difference error value, which can generally be determined by the change values ​​of each interface part of different models, such as time, status bar, progress bar ratio, etc., and the initial experience initial value is determined and then adjusted continuously according to the continuous test personnel's manual confirmation of whether it is covered.

[0141] If the comparison difference value is greater than the first preset threshold and less than or equal to the second preset threshold, the test is determined to be successful, and the user is notified to manually confirm whether to use the FOTA application screenshot data as the new standard interface image. This is because some of the FOTA application operation pages may be modified through subsequent development by the user, resulting in a slightly larger difference between the application screenshot data collected this time and the standard interface image. Therefore, it is more reasonable to notify the user to manually confirm whether to use this round of FOTA application screenshot data as the new standard interface image. If the current round of FOTA application screenshot data is used as the new standard interface image, the reliability of image comparison can be improved during the next test. The second preset threshold is the maximum value of the manual confirmation interval value. This interval is theoretically passed, but the development may have changed some interfaces. The second preset threshold will also be adjusted continuously through each manual confirmation. The more times the execution is performed, the more stable and reasonable the first preset threshold and the second preset threshold will become.

[0142] If the comparison difference value is greater than the second preset threshold, the test is determined to be successful, and the user is notified to manually confirm whether to use the FOTA application screenshot data as the new standard interface image. This is because there are no problems with the log file and the use case execution results, so it cannot be completely denied that the FOTA application screenshot data does not meet expectations. This situation may be caused by major changes to the interface or the addition of a pop-up interface. At this time, the tester can choose to save this time or abandon the FOTA application screenshot data.

[0143] Through the technical solution provided by the embodiment of the present invention, the standard interface diagram is updated according to different situations, thereby further improving the reliability of the next round of testing and improving the accuracy of test analysis.

[0144] In some optional implementation manners, the above step g2 includes:

[0145] Step I1, when there is an abnormal problem in the log file and the execution results of the use case are all successful, check whether there is a standard interface image in the image library;

[0146] Step I2, if there is no standard interface image in the image library, the user is notified to troubleshoot the problem and the test is determined to have failed;

[0147] Step I3: if there is a standard interface image in the image library, compare the FOTA application screenshot data with the standard interface image;

[0148] Step I4: if the comparison difference value is less than or equal to the first preset threshold, the user is notified to troubleshoot the problem and the test is determined to be successful;

[0149] Step I5: if the comparison difference value is greater than the first preset threshold value and less than or equal to the second preset threshold value, notify the user to manually confirm whether to use the FOTA application screenshot data as the new standard interface image, and notify the user to troubleshoot the problem and confirm that the test is successful;

[0150] Step I6: If the comparison difference value is greater than the second preset threshold, the test is determined to have failed, and the user is notified to troubleshoot the problem.

[0151] Specifically, if there is an abnormal problem in the log but the use case is successfully executed, it is possible that an internal error occurs in the log execution, but it does not affect the business. Therefore, the embodiment of the present invention performs auxiliary analysis based on the FOTA application screenshot data to determine whether this round of testing is successful. First, detect whether there is a standard interface diagram in the image library. If there is no standard interface diagram in the image library, and because there is a problem with the log, there is no standard for auxiliary analysis. Therefore, it is inappropriate to directly use this round of FOTA application screenshot data as a standard interface diagram. This embodiment will notify the user to troubleshoot the problem and determine that the test failed. If there is a standard interface diagram in the image library, the FOTA application screenshot data and the standard interface diagram are compared to start auxiliary analysis. If the comparison difference value is less than or equal to the aforementioned first preset threshold, it means that the image comparison of the FOTA application screenshot data is passed, and the interactive operation of the test application operation FOTA application is accurate, and the test can be determined to be successful. For abnormal logs, it is only necessary to notify the user to troubleshoot the log problem. If the comparison difference value is greater than the first preset threshold and less than or equal to the second preset threshold, it means that the authentication of the image can basically be regarded as passed. The difference may be slightly larger due to the fine-tuning of the operation page of the FOTA application. It can be determined that the test is successful. At this time, the user is notified to manually confirm whether to use the FOTA application screenshot data as the new standard interface image, and the user is notified to check the log problem. If the comparison difference value is greater than the second preset threshold, under the premise that there is a problem with the log, the comparison between the FOTA application screenshot data and the standard interface image cannot pass, so it is more reasonable to directly determine that the test has failed, and notify the user to check the problems during the test.

[0152] Through the technical solution provided in the embodiment of the present invention, the probability of test success is further analyzed based on the comparison between the FOTA application screenshot data and the standard interface diagram, and when the difference in the comparison results is small, the reliability of the next round of testing is further improved, thereby improving the accuracy of the test analysis.

[0153] In a complete implementation, the steps of the above method embodiments are integrated into one embodiment, referring to Figures 9 to 14 The complete process of the OTA automatic testing method provided by the present invention is as follows.

[0154] S401: Configure the use case items and dictionary table through the test platform to prepare for creating the use case, including but not limited to: Figure 3 , 4 The test cases shown can also be expanded according to test requirements.

[0155] S402: Create a test case through the test PTZ, combine preconditions, operation steps, expected results, monitoring scripts, execution scripts, etc. to form a completed test case.

[0156] S403: Select corresponding test cases according to the actual application scenario through the test PTZ to create a test scenario.

[0157] S404: Configure the test task information through the test gimbal, select the test scenario, and perform task ID mapping according to the OTA upgrade information sent from the OTA gimbal in step S405 to establish an association between the test task and the actual OTA upgrade task, thereby determining the upgrade object, version information, etc.

[0158] S405: The OTA gimbal sends OTA upgrade information to the test gimbal.

[0159] S406: Start the test task through the test gimbal.

[0160] S407: Determine whether the current test task conflicts by testing the PTZ, for example, whether the designated vehicle is already executing other test tasks.

[0161] S408: If a conflict is detected, the startup will fail. After the startup fails, a prompt will be given and the conflicting tasks will be displayed on the test gimbal, asking whether to choose to close them.

[0162] S409: When the test gimbal finds that there is no conflict in the tasks, it will continue to determine whether the test object information (upgrade object, version information, etc.) in the test task is consistent with the OTA gimbal mounting data.

[0163] S410: If they are inconsistent, the OTA gimbal will mount the corresponding information according to the test object information selected by the test gimbal.

[0164] S411: When the information of the test gimbal and the OTA gimbal is consistent, the test gimbal can start the process of sending the test task data to the corresponding test application.

[0165] S412: Before actually sending the data, the test gimbal also determines whether the current task to be executed is a bench task or a real vehicle task. If it is a bench task, S413 is executed, so that the bench engine is ready before sending the task data to the test application to ensure that the bench engine is in a normal state.

[0166] S413: The test bench engine performs data analysis on the test task and obtains a startup notification for initializing itself.

[0167] S414: Initialize the test bench engine status, start task monitoring and signal sending and receiving, and synchronize the status to the test gimbal.

[0168] S415: After the test gimbal receives the synchronization notification, it determines whether the state of the gimbal engine is ready. If the state is not ready, it will retry the initialization state of the gimbal engine three times. If the gimbal engine is still not ready after three times, the process ends abnormally.

[0169] S416: The test bench engine is ready, and the test gimbal sends the test task to the test application.

[0170] S417: If it is a bench task, the test application receives the test task from step S416; if it is a real vehicle task, it receives the test task from step S412; and then the test application performs test task data analysis through the first task analysis module.

[0171] S418: The test application executes the task script through the first task execution module, triggering the FOTA application to start the OTA upgrade process.

[0172] S419: The FOTA application starts executing the test process.

[0173] S420: The FOTA application obtains task information of the OTA gimbal, including but not limited to the upgrade object, upgrade version, upgrade ID and other related information.

[0174] S421: The OTA gimbal sends OTA task data to the FOTA application.

[0175] S422: The FOTA application determines whether the ID of the actual OTA task is the ID associated with the ID of this test task. If it is correct, S423 is executed; otherwise, S430 is executed incorrectly.

[0176] S423: The FOTA application executes the OTA upgrade task.

[0177] S424: Actual vehicle OTA / bench OTA (including OTA channels composed of OTA Master, OTASubmaster, and controller) executes the upgrade task.

[0178] S425: The test bench engine monitors that the OTA channel is executing at a specific stage (such as immediate upgrade requires conditional check and needs to send a conditional signal that meets the upgrade requirements) and sends a signal for simulation.

[0179] S426: After the OTA channel task is executed, the result is reported to the FOTA application and the test application.

[0180] S427: The FOTA application organizes the logs and uploads them to the OTA gimbal (the logic of the original OTA upgrade system executing the upgrade process).

[0181] S428: The OTA gimbal records logs.

[0182] S429: The test application processes the test results, uploads the log results to the test gimbal, and synchronizes the task execution status.

[0183] S430: When it is determined in step S422 that the task ID of the OTA gimbal and the ID of the test task are incorrect, an error is reported to the test gimbal.

[0184] S431: Normal or abnormal termination will change the test task status to end and synchronize it to the test bench engine.

[0185] S432: The test bench engine monitors the stopped state, the equipment is shut down, and the task ends.

[0186] S433: The execution results and logs of the test bench engine are uploaded to the test gimbal for unified analysis and processing.

[0187] S434: The test PTZ performs task result analysis, log storage, and report presentation.

[0188] S435: Determine whether there is any abnormality during the test process.

[0189] S436: Automatically carry out bill of lading if there is any abnormality.

[0190] S437: After the entire test is completed, the corresponding test person in charge will be notified by email.

[0191] In the above step S435, the complete process of determining whether there is an abnormality in the test can be referred to Fig.15 , specifically including the following steps:

[0192] S501: The test gimbal receives the use case execution results, FOTA application execution logs, and UI image data (FOTA application screenshot data) uploaded by the test application.

[0193] S502: Parse the execution logs of the FOTA application and the test application.

[0194] S503: Determine whether the FOTA application log result is normal. If normal, proceed to S504; otherwise, proceed to S515.

[0195] S504: Determine whether the use case execution result is successful. If successful, go to S505; if failed, go to S528.

[0196] S505: Determine whether the UI image already has a standard gallery. If so, go to S508; otherwise, go to S506.

[0197] S506: Update the UI image data of this execution to the standard library for comparison when the same task is executed next time.

[0198] S507: After comprehensive judgment, it is determined that the result of this task is a successful test.

[0199] S508: Compare the UI image data of this time with the standard image library of this task to obtain the difference value.

[0200] S509: Determine whether the difference is less than X% (X% is the first preset threshold, i.e., the agreed difference error value, which can generally be determined through the change values ​​of various interface parts of different car models, such as time, status bar, progress bar ratio, etc., and the initial value is initially determined based on experience and then continuously adjusted based on manual confirmation of coverage by testers).

[0201] S510: Image comparison passed.

[0202] S511: Determine whether the difference value is less than Y% and greater than X% (the second preset threshold value of Y% is the maximum value of the manual confirmation interval. This interval is theoretically passed, but the developer may change some UI / UE. Y% will also be adjusted through each manual confirmation. The more times it is executed, the more stable and reasonable the X% and Y% values ​​are).

[0203] S512: The image comparison result is to be confirmed, and the tester needs to confirm whether to save the image data of this time as the standard library.

[0204] S513: Determine whether the difference value is greater than Y%.

[0205] S514: This image comparison fails, but this does not completely deny that it does not meet expectations, because there may be major changes to the UI / UE or a new pop-up interface. At this time, the tester can choose to save this or give up (no operation will be given up by default).

[0206] S515: Determine whether the use case execution result is successful. If successful, go to S516; if failed, go to S526.

[0207] S516: Determine whether the UI image data already has a standard library. If yes, go to S517; otherwise, go to S526.

[0208] S517: Compare the UI image data with the standard image library to obtain difference values.

[0209] S518: Determine whether the difference value is less than X%.

[0210] S519: Image comparison passed.

[0211] S520: Automatically notify FOTA developers and testers to troubleshoot the problem.

[0212] S521: The task result is successful.

[0213] S522: Determine whether the difference value is less than Y% and greater than X%.

[0214] S523: Image comparison is to be confirmed, and the tester needs to confirm to keep the image data or standard library data. This is because internal errors may occur in log execution, but it does not affect the business.

[0215] S524: Determine whether the difference value is greater than Y%.

[0216] S525: Image comparison fails.

[0217] S526: Automatic bill of lading notification automation tool owner and FOTA developer troubleshoot issues.

[0218] S527: Task result failed.

[0219] S528: Whether the UI image data already has a standard library. If so, go to S529; otherwise, go to S538.

[0220] S529: Compare the UI image data with the standard library and obtain the difference value.

[0221] S530: Determine whether the difference value is less than X%.

[0222] S531: Image comparison passed.

[0223] S532: Automatic bill of lading notification automation tool owner to troubleshoot the issue.

[0224] S533: The task result is successful.

[0225] S534: Determine whether the difference value is less than Y% and greater than X%.

[0226] S535: The image comparison is to be confirmed. The tester needs to confirm to keep the current image data or the standard library data.

[0227] S536: Determine whether the difference value is greater than Y%.

[0228] S537: Image comparison failed.

[0229] S538: The person in charge of the automated bill of lading notification automation tool should troubleshoot the issue.

[0230] S539: Task result failed.

[0231] Through the above steps of the embodiment of the present invention, a new OTA automatic testing method is provided, which significantly reduces the testing cost and improves the testing efficiency.

[0232] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0233] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0234] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An OTA automated testing system, characterized in that: include: OTA gimbal, test gimbal and device under test, wherein the device under test includes at least one of vehicle equipment and bench equipment; the device under test is installed with a test application and a FOTA application, and is deployed with an OTA channel; The OTA gimbal is used to deploy the upgrade information used for the actual OTA upgrade task; the test gimbal is used to configure the test task for testing the OTA upgrade process; The test application is used to receive the test task and perform a test operation on the FOTA application according to the test task; The FOTA application is used to obtain the upgrade information from the OTA gimbal according to the test operation of the test application, and use the upgrade information to perform an upgrade operation on the OTA channel.

2. The system according to claim 1, characterized in that The system further comprises a test bench engine, which is used to simulate vehicle sensors during upgrading and provide the test bench device with message information required for upgrading.

3. The system according to claim 2, characterized in that The test platform includes: a configuration management module, a test case management module, a test scenario management module, a test task management module, a result analysis center module, a log management module and an abnormal result processing module; The configuration management module is used to configure a first test case of the test application and a second test case of the test bench engine; The test case management module is used to combine the first test case and the second test case to generate a number of complete standard test cases; The test scenario management module is used to generate a number of test scenarios according to one or more standard test cases; The test task management module is used to create a number of the test tasks according to different test scenarios, and associate each of the test tasks with the actual OTA upgrade task in the OTA gimbal; The result analysis center module is used to analyze the use case execution results uploaded by the device under test; The log management module is used to store the test log files uploaded by the device under test; The abnormal result processing module is used to perform corresponding processing according to the analysis result of the result analysis center module.

4. The system according to claim 3, characterized in that The test application includes: a first network monitoring module, a first task parsing module, a first task execution module, a first data sorting module, a first log management module and an abstract interface module; The first network monitoring module is used to monitor the received test task; The first task parsing module is used to parse the test task into a first test case instruction that can be executed by the test application; The first task execution module is used to call the abstract interface module by executing the first test case instruction; The abstract interface module is used to be called by the first task execution module to send instructions to the FOTA application to perform the test operation, or to receive the first use case execution result fed back by the FOTA application, the FOTA application log, the FOTA application screenshot data, and the OTA channel; The first log management module is used to manage a first log file during the test process, wherein the first log file includes a log generated by the test application and the FOTA application log; The first data sorting module is used to sort the first log file, the first use case execution result and the FOTA application screenshot data, and upload the sorted data to the test gimbal.

5. The system according to claim 3, characterized in that The platform engine includes: a second network monitoring module, a second task parsing module, a second task execution module, a second data sorting module, a second log management module and a message management module; The second network monitoring module is used to monitor the actual OTA upgrade task of the OTA channel, where the actual OTA upgrade task is a task that the OTA channel is to execute correspondingly according to the acquired upgrade information; The second task parsing module is used to parse out a message program instruction for ensuring the smooth progress of the actual OTA upgrade task according to the second test case; The second task execution module is used to execute the message program instruction; The message management module is used to be called by the second task execution module, generate the message information according to the message program instruction, and send it to the OTA channel; The second log management module is used to manage the second log file generated by the platform engine during the upgrade process; The second data sorting module is used to sort the second log file and the second use case execution result generated by the test bench engine, and upload the sorted data to the test gimbal.

6. An OTA automated testing method, characterized in that: Applied to the OTA automated test system, the method includes: Configure the test task on the test platform and send the test task to the test application of the device under test; The device under test receives the test task through the test application, and performs a test operation on the FOTA application according to the test task; The FOTA application obtains upgrade information from the OTA gimbal in response to the test operation, where the upgrade information is information used to deploy an actual OTA upgrade task; The FOTA application performs an upgrade operation on the OTA channel using the upgrade information in response to the test operation; The OTA channel executes a corresponding upgrade process in response to the upgrade operation.

7. The method according to claim 6, characterized in that When the device under test is a bench device, in the process in which the OTA channel performs a corresponding upgrade in response to the upgrade operation, the process further includes: The test bench engine monitors the state of the OTA channel and provides the OTA channel with message information required for upgrading according to the state of the OTA channel during the upgrading process.

8. The method according to claim 7, characterized in that The step of configuring the test task on the test platform and sending the test task to the test application of the device under test includes: Configure a first test case of a test application and a second test case of the test bench engine through a configuration management module; Combining the first test case and the second test case through a test case management module to generate several complete standard test cases; Generate several test scenarios according to one or more standard test cases through the test scenario management module; Create several test tasks according to different test scenarios through the test task management module, and associate each of the test tasks with the actual OTA upgrade task in the OTA gimbal; Detecting whether the device under test has other ongoing test tasks that conflict with the target test task to be issued; If a conflict occurs, return to the step of detecting whether the device under test has other ongoing test tasks that conflict with the target test task to be issued; If no conflict occurs, detecting whether the information of the device under test is consistent with the actual object under test, where the actual object under test is the object under test recorded in the actual OTA upgrade task in the OTA gimbal; If the detection is consistent with the actual object under test, sending the target test task to the test application; If the detection is inconsistent with the actual object under test, the information of the device under test is reported to the OTA gimbal, so that the OTA gimbal modifies the object under test recorded in the actual OTA upgrade task, and returns to the step of detecting whether the information of the device under test is consistent with the actual object under test.

9. The method according to claim 8, characterized in that The sending the target test task to the test application includes: Determining whether the target test task is a test task for the bench equipment; If it is not a test task for the bench device, directly sending the target test task to the test application; If it is a test task of the bench device, a start notification is sent to the bench engine, so that the bench engine performs initialization startup in response to the start notification, and feeds back a ready state signal to the test pan-tilt when the initialization is successful; When the ready state signal is received, sending the target test task to the test application; When an initialization failure signal is received, the target test task is stopped from being issued. The initialization failure signal is a signal sent after the test bench engine has failed to initialize for a preset number of times.

10. The method according to claim 8, characterized in that The testing operation on the FOTA application according to the test task includes: Parsing the target test task into a first test case instruction by a first task parsing module; The first test case instruction is executed by a first task execution module, and an abstract interface module is called according to the first test case instruction to send an instruction for performing the test operation to the FOTA application.

11. The method according to claim 8, characterized in that Before the FOTA application performs an upgrade operation on the OTA channel using the upgrade information in response to the test operation, the method further includes: The FOTA application verifies whether the actual OTA upgrade task corresponding to the upgrade information is consistent with the target test task; If the verification is inconsistent, the test ends; If the verification is consistent, the step of performing an upgrade operation on the OTA channel using the upgrade information in response to the test operation by the FOTA application is executed.

12. The method according to claim 8, characterized in that After the OTA channel completes the corresponding upgrade process in response to the upgrade operation, the method further includes: The OTA channel sends the first use case execution result to the FOTA application and the test application respectively; The FOTA application sends the first use case execution result and the FOTA application log generated by the FOTA application to the OTA gimbal; The test application sends the first log file, the collected FOTA application screenshot data and the first use case execution result to the test gimbal, and notifies the test gimbal that the test is finished; When the test gimbal receives a test end status, a stop signal is sent to the test bench engine; When the test bench engine receives the stop signal, it turns off monitoring of the OTA channel and sends a second log file and a second use case execution result to the test gimbal; The test gimbal performs test result analysis on the received first log file, the second log file, the first use case execution result, the second use case execution result and the FOTA application screenshot data through the result analysis center module.

13. The method according to claim 12, characterized in that The test platform performs test result analysis on the received first log file, the second log file, the first use case execution result, the second use case execution result, and the FOTA application screenshot data through a result analysis center module, including: When all log files are normal and the test case execution results are successful, the test is considered successful; When there are abnormal problems in the log file and the test case execution results are all successful, the FOTA application screenshot data is used to assist in analyzing whether the test is successful; When the log files are normal but the test case execution results are unsuccessful, the FOTA application screenshot data is used to assist in analyzing whether the test is successful.

14. The method according to claim 13, characterized in that When the log files are normal and the test case execution results are successful, the test is determined to be successful, including: When the log file is normal and the test case execution result is successful, check whether there is a standard interface image in the image library; If the standard interface image does not exist, the FOTA application screenshot data is used as the standard interface image, and the test is determined to be successful; If the standard interface image is available, comparing the FOTA application screenshot data with the standard interface image; If the comparison difference value is less than or equal to the first preset threshold, the test is determined to be successful; If the comparison difference value is greater than the first preset threshold and less than or equal to the second preset threshold, the test is determined to be successful, and the user is notified to manually confirm whether to use the FOTA application screenshot data as a new standard interface image; If the comparison difference value is greater than the second preset threshold, the test is determined to be successful, and the user is notified to manually confirm whether to use the FOTA application screenshot data as the new standard interface image.

15. The method according to claim 13, characterized in that When the log file has abnormal problems and the test case execution results are all successful, the FOTA application screenshot data is used to assist in analyzing whether the test is successful, including: When there are abnormal problems in the log file and the test case execution results are all successful, check whether there is a standard interface image in the image library; If the standard interface image does not exist in the image library, the user is notified to troubleshoot the problem and the test is determined to have failed; If the image library has the standard interface image, the FOTA application screenshot data is compared with the standard interface image; If the comparison difference value is less than or equal to the first preset threshold, the user is notified to troubleshoot the problem and the test is confirmed to be successful; If the comparison difference value is greater than the first preset threshold and less than or equal to the second preset threshold, the user is notified to manually confirm whether to use the FOTA application screenshot data as a new standard interface image, and the user is notified to troubleshoot the problem and determine that the test is successful; If the comparison difference value is greater than the second preset threshold, the test is determined to have failed, and the user is notified to troubleshoot the problem.

16. The method according to claim 13, characterized in that When the log files are normal and the test case execution result is unsuccessful, the FOTA application screenshot data is used to assist in analyzing whether the test is successful, including: When the log files are normal and the use case execution results are unsuccessful, check whether there is a standard interface image in the image library; If the standard interface image does not exist in the image library, the user is notified to troubleshoot the problem and the test is determined to have failed; If the image library has the standard interface image, the FOTA application screenshot data is compared with the standard interface image; If the comparison difference value is less than or equal to the first preset threshold, the user is notified to troubleshoot the problem and the test is confirmed to be successful; If the comparison difference value is greater than the first preset threshold and less than or equal to the second preset threshold, the user is notified to manually confirm whether to use the FOTA application screenshot data as a new standard interface image, and the user is notified to troubleshoot the problem and determine that the test is successful; If the comparison difference value is greater than the second preset threshold, the test is determined to have failed, and the user is notified to troubleshoot the problem.

Images