YOLO-based interface identification and automatic testing method in multi-system environment

Abstract

The invention relates to the technical field of software testing, and provides a YOLO-based interface recognition and automatic testing method in a multi-system environment, which comprises the following steps of: performing recognition detection on a test software interface screenshot obtained through desktop screenshots by utilizing a pre-trained YOLO model; an RPA or ADB tool is adopted to realize automatic operation; performing a black box test, and verifying whether output results of the YOLO model under various inputs conform to expectation from the perspective of a user without paying attention to internal implementation details of a tool; carrying out white-box testing, and carrying out fine-grained testing on the YOLO model, the RPA operation script and the ADB tool command; and compiling a test script, and automatically executing identification and operation test. Through efficient and accurate recognition, powerful automatic operation, excellent real-time performance and a reliable exception handling mechanism, the software testing efficiency and accuracy are remarkably improved, and the method has wide application prospects and important technical value.

Description

Technical Field

[0001] This invention relates to the field of software testing technology, and more specifically, to a YOLO-based interface recognition and automated testing method in a multi-system environment. Background Technology

[0002] Automated software testing is a crucial means to improve testing efficiency and reduce human error. However, in the current software testing field, especially in interface testing under multi-system environments, there are many problems and shortcomings that urgently need to be addressed. First, existing automated identification tools are often limited to specific operating systems or application environments and cannot be used across platforms. This leads to frequent tool switching during testing, increasing the complexity and cost of testing. Furthermore, minor changes to interface elements can cause identification tools to malfunction, failing to accurately identify target elements and thus affecting the continuity and stability of testing. Additionally, some automated tools lack precision when performing interface operations, easily resulting in misoperations or operation failures. Moreover, the interface operation methods differ significantly across operating systems and devices, making existing automated tools unusable across multiple system environments. This necessitates the separate development and maintenance of operation scripts for each platform, increasing development and maintenance costs. Summary of the Invention

[0003] In view of this, the present invention proposes a YOLO-based interface recognition and automated testing method in a multi-system environment to solve the problems existing in the prior art.

[0004] To achieve the above objectives, this invention proposes a YOLO-based interface recognition and automated testing method for multi-system environments, including: The pre-trained YOLO model is used to identify and detect test software interface screenshots obtained through desktop screenshots; Automated operations can be achieved using RPA or ADB tools, including process recording, custom logic, simulating mouse and keyboard operations, and calling external scripts and commands. Conduct black-box testing, from the user's perspective, without focusing on the internal implementation details of the tool, to verify whether the output results of the YOLO model under various inputs meet expectations; Perform white-box testing to conduct fine-grained testing of the YOLO model, RPA operation scripts, and ADB tool commands; Write test scripts to automate the execution of identification and operation tests.

[0005] Furthermore, the testing method also includes the following steps: Verify the recognition accuracy of the YOLO model in different operating systems and application environments; Verify whether the RPA or ADB tool can perform interface operations based on the recognition results; Verify whether the exception handling mechanism of the RPA or ADB tool is effective.

[0006] Further, the test method further comprises the following steps: Test the recognition accuracy of the YOLO tool for pop-up windows, buttons, prompt information, and icons in various scenarios. Test the automation operation capability of the RPA or ADB tool to ensure that operations are performed based on the coordinates and content of the recognition results. Test the flow control capability of the tool to verify whether the RPA or ADB tool can correctly select and execute the corresponding operation process under different recognition results. Test the tool's processing capability in the case of recognition failure, operation failure, and interface changes.

[0007] Further, the test method further comprises the following steps: Test the total response time of the tool from screenshot acquisition, target recognition to automation operation in different operating environments. Test the processing delay in different scenarios, calculate the average delay and maximum delay, and evaluate the impact on actual test work. Run multiple recognition and operation tasks simultaneously on the same device to test the concurrent processing capability of the tool and verify whether the RPA or ADB tool can maintain stable operation and respond in time under high load.

[0008] Further, the test method further comprises the following steps: Simulate different abnormal situations to test the exception handling capability of the YOLO recognition module and verify whether the tool can execute remedial measures after recognition failure. Simulate abnormal situations during the automation operation process to test the exception handling capability of the RPA or ADB tool and ensure that the tool can perform log recording, exception feedback, and attempt recovery after operation failure. Introduce artificial fault points in the test process to verify whether the recovery mechanism of the tool can recover to the last checkpoint and continue execution after an exception occurs.

[0009] Further, the test method further comprises the following steps: Configure the test environment, install relevant software and tools, ensure that the test device is in normal working condition, import the YOLO model and dataset, and configure the RPA tool and ADB. According to the function test items in the test plan, test the recognition module, automation operation module, and flow control module one by one, and record the test input, output, actual result, and expected result in each test item. Measure the response time of the tool in different scenarios, record the average delay, maximum delay and concurrent processing performance, and verify the performance and stability of the tool under high load conditions; Simulate abnormal scenarios to verify the abnormal handling and recovery ability of the tool, and record the types, frequency, processing time and recovery of the abnormality; Analyze the test results to evaluate the functional integrity, identification accuracy, operation accuracy, real-time performance and abnormal handling ability of the tool, and generate a test report according to the test results.

[0010] Compared with the prior art, the beneficial effects of the present application are: The pre-trained YOLO model can quickly and accurately identify interface elements in different operating systems (Windows, Linux, Android) and various application environments. In the test, the identification accuracy is more than 95%. By combining RPA or ADB tools, this method can realize various automatic operations such as mouse clicking, keyboard input and mobile interface operation. In actual testing, the operation success rate is more than 90%, which ensures that the tool can accurately perform various interface operations according to the identification results of the YOLO model, greatly improving the test efficiency and accuracy.

[0011] In the test results of the present application, the total response time from screenshot acquisition, target identification to automatic operation is less than 1 second on average, meeting the real-time requirements, so that the tool can complete complex test tasks in a short time, adapt to various real-time application scenarios, and improve the efficiency and practicality of the test.

[0012] The test results of the present application show that the abnormal handling success rate is more than 85%, which ensures that the tool can still run stably in complex and unstable environments, reducing test interruptions and errors caused by abnormalities.

[0013] This method is not only suitable for various operating systems and application environments, but also can be flexibly configured and extended according to different test requirements. By writing test scripts, a large number of identification and operation tests can be automatically executed, which is suitable for various complex test scenarios and has strong universality and scalability. BRIEF DESCRIPTION OF DRAWINGS

[0014] Various other advantages and benefits will become clear to those of ordinary skill in the art from the following detailed description of the preferred embodiments. The drawings are only for the purpose of illustrating the preferred embodiments and are not considered to be limiting on the present application. In the drawings: Figure 1 The principle diagram of the interface identification and automatic test method flow under the multi-system environment based on YOLO proposed by the present application. DETAILED DESCRIPTION

[0015] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood, and so that the scope of the present disclosure can be completely conveyed to those skilled in the art. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0016] In the YOLO and RPA / ADB-based automated terminal testing system, a pre-trained YOLO model is used to recognize and detect the test software interface screenshots obtained through desktop screenshots, and RPA or ADB tools are used to realize automated operations, including process recording, custom logic, simulation of mouse and keyboard operations, calling of external scripts and commands.

[0017] The present embodiment further proposes a YOLO-based interface recognition and automated testing method in a multi-system environment for the above system, as shown in Figure 1 , which includes: Black box testing: From the user's perspective, without focusing on the internal implementation details of the tool, verify whether the output results of the tool under various inputs meet the expectations.

[0018] White box testing: Understand the internal structure and logic of the tool, and perform fine-grained testing on the recognition algorithm, RPA operation script, ADB command, etc., to ensure that each module works correctly according to the design.

[0019] Automated testing: Write test scripts to automate the execution of a large number of recognition and operation tests, improve testing efficiency, and ensure the consistency and stability of the tool in multiple operations.

[0020] In order to ensure the stability, accuracy and efficiency of the intelligent recognition general tool in various scenarios, the present embodiment formulates a detailed test plan. The test plan will focus on the core functions of the tool, including interface recognition (YOLO), automated operation (RPA / ADB), process control and exception handling. The goal of the test plan is to verify whether the functions of the tool meet the design requirements, whether the performance meets the real-time application requirements, and whether it remains stable under various abnormal conditions.

[0021] The test objectives include the following aspects: (1) Verify the recognition accuracy of the tool in different operating systems (Windows, Linux, Android) and different application environments.

[0022] (2) Verify if the automation operation module (RPA / ADB) can accurately perform interface operations based on the recognition results.

[0023] (3) Verify the real-time performance of the entire tool to ensure that recognition and operation are completed within a reasonable time.

[0024] (4) Verify the effectiveness of the tool's exception handling mechanism to ensure that the process can recover and run stably under abnormal conditions.

[0025] Test scope Recognition module test (YOLO): Test the accuracy of YOLO tool in identifying pop-up windows, buttons, prompt information, and icons in various scenarios.

[0026] Automation operation test (RPA / ADB): Test the automation operation capabilities of RPA and ADB, including simulating mouse clicks, keyboard inputs, and mobile interface operations. Ensure accurate operation based on the coordinates and content of the recognition results.

[0027] Process control test: Test the process control capabilities of the tool to verify whether the tool can correctly select and execute the corresponding operation process under different recognition results.

[0028] Exception handling test: Test the tool's handling capabilities in the event of recognition failure, operation failure, and interface changes, and verify the robustness of the exception handling mechanism.

[0029] Test environment Hardware environment: PC (Windows, Linux system), smartphone (Android), and instrument test equipment.

[0030] Software environment: Windows test software, Android application, YOLO model (Python implementation), RPA tool (UiPath), and ADB tool.

[0031] Dataset: Contains screenshot datasets of various scenarios, including instrument software interfaces, mobile prompt information, and interface screenshots of different resolutions and languages.

[0032] The logical structure of the test platform in this embodiment is shown in Figure 1 The test content includes functional testing, performance testing, and exception handling testing. The functional testing includes: Recognition accuracy test: Test the recognition accuracy of the YOLO model in different scenarios, including pop-up windows, buttons, prompt information, and icons.

[0033] Test in different resolutions, different system interfaces, and different language environments to verify the model's generalization ability.

[0034] Automation operation test: Test the accuracy of RPA in performing simulated operations (such as clicking, inputting, window switching) in a Windows environment. Test the accuracy of ADB in performing screen operations (such as clicking, swiping, inputting) on an Android phone. Verify that the automation operations are accurately performed according to the coordinates and content identified by YOLO, ensuring the accuracy of the operations.

[0035] Flow control test: Test whether the tool can correctly select the corresponding operation flow under the input of different recognition results. If an "error prompt" is detected, the tool should execute the "error handling" branch; if a "confirmation button" is recognized, the "confirmation handling" branch should be executed. Introduce various scenarios (such as continuous operations, multi-step tests) in flow control to verify the stability of the tool in complex operation flows.

[0036] Performance testing includes: Real-time testing: Test the total response time of the tool from screenshot acquisition, target recognition to automation operation in different operation environments. Ensure that the tool completes the entire operation within hundreds of milliseconds to several seconds, meeting the real-time requirements.

[0037] Test the processing delay under different scenarios, calculate the average delay and maximum delay, and evaluate their impact on actual test work.

[0038] Multi-task processing test: Run multiple recognition and operation tasks simultaneously on the same device to test the concurrent processing capability of the tool. Verify whether the tool can maintain stable operation and respond in time under high load.

[0039] Exception handling test includes: Recognition exception test: Simulate different abnormal situations such as interface element changes, image blurring, partial occlusion, etc., to test the exception handling capability of YOLO recognition module. Verify whether the tool can perform remedial measures (such as re-taking screenshots, retrying recognition) after recognition failure.

[0040] Operation exception test: Simulate abnormal situations during automation operations, such as operation targets not being in the expected position, controls being unable to click, etc., to test the exception handling capability of RPA and ADB. Ensure that the tool can perform log recording, exception feedback and attempt recovery after operation failure.

[0041] Recovery mechanism test: Add artificial fault points in the test process to verify whether the tool's recovery mechanism can recover to the last checkpoint and continue execution after an exception occurs, ensuring the continuity of the process.

[0042] As a preferred embodiment, the test process of the present embodiment is as follows: 1) Test preparation: Configure the test environment, install relevant software and tools, and ensure that the test equipment is in normal working condition. Import the YOLO model and dataset, and configure the RPA tool and ADB.

[0043] 2) Perform functional testing: According to the functional test items in the test plan, test the recognition module, automation operation module and process control module one by one. In each test item, record the test input, output, actual result and expected result.

[0044] 3) Perform performance testing: Measure the response time of the tool under different scenarios, record the average latency, maximum latency and concurrent processing performance. In high load conditions, verify the performance and stability of the tool.

[0045] 4) Perform exception testing: Simulate abnormal scenarios to verify the exception handling and recovery ability of the tool. Record the types, frequency, processing time and recovery of exceptions.

[0046] 5) Result analysis: Analyze the test results, evaluate the functional integrity, recognition accuracy, operation accuracy, real-time performance and exception handling ability of the tool. According to the test results, generate a test report, point out the advantages and disadvantages of the tool, and provide a basis for subsequent optimization.

[0047] Test indicators include: Recognition accuracy: The number of correctly recognized interface elements / the total number of recognized targets, ensuring that the recognition accuracy is above 95%.

[0048] Operation success rate: The number of successful automation operations / the total number of operations, ensuring that the operation success rate is not less than 90%.

[0049] Average response time: The total time from screenshot to operation completion, ensuring that the average response time is less than 1 second.

[0050] Exception handling success rate: The number of successfully handled and recovered exceptions / the total number of exceptions, ensuring that the exception handling success rate is not less than 85%.

[0051] This embodiment takes the TC-11551 scenario as an example to perform detailed testing and observe the test results.

[0052] Test case TC-11551 is a functional test under the WEA detection scenario.

[0053] WEA testing mainly verifies whether the mobile phone can receive and process wireless emergency alerts, which is a mandatory requirement for mobile terminals in some countries and regions. The detection process may include simulating emergency situations to check whether the mobile phone can receive the corresponding alert signal and correctly display the alert information.

[0054] The test description of TC-11551 is as follows: By using the TC-11551 case, the complete process of automatically identifying pop-up windows, processing prompt information and performing related operations when a specific event is triggered in the instrument test software is simulated. This scenario is used to verify the overall performance of the intelligent identification tool, including the accuracy of the YOLO identification module, the precision of the RPA automation operation module, and the process control and exception handling capabilities.

[0055] Scenario setting: During the running of the instrument test software, a test step will pop up a prompt box containing test status (such as "test passed" or "test failed") and related information. The tool needs to identify the content of the prompt box and take corresponding automatic operations.

[0056] Test steps and observation results Purpose: To verify whether the terminal device (TU) can specify different message identifiers for different categories of alert messages.

[0057] Prerequisites and settings: The terminal device (TU) supports WEA function; 5G device configuration is complete, supporting CMAS message identifier in 5G SA network.

[0058] Test steps 1. Before the test starts, make sure all alert functions on the TU are enabled, and turn off the device.

[0059] 2. Configure the 5G device to support 5G SA network and activate the NR cell.

[0060] 3. Turn on the TU device, make sure it starts normally, and prepare to receive CMAS messages.

[0061] 4. Make sure the UE (user equipment) successfully attaches to the 5G SA NR network, and the test environment is ready.

[0062] 5. Send the national alert message "National alert_A_A" to the TU, with message ID 4370.

[0063] 6. Verify whether the TU receives the national alert message "National alert_A_A" and confirms that its message identifier is specified as MsgID1.

[0064] A pop-up box appears on the TU interface, showing the alert information, indicating that the TU successfully received the alert message.

[0065] 7. Confirm the "National alert_A_A" alert message on the TU and observe the system response.

[0066] 8. Send the emergency alert message "CMAS_Imminent_A" to the TU with a message ID of 4371.

[0067] 9. Verify that the TU received the emergency alert message "CMAS_Imminent_A" and confirm that its message identifier is assigned as MsgID2.

[0068] A pop-up box appears on the TU interface, showing the detailed information of the emergency alert, indicating that the TU successfully received and processed the alert.

[0069] 10. Confirm the "CMAS_Imminent_A" alert on the TU and observe the device's subsequent processing behavior.

[0070] 11. Send the emergency alert message "CMAS_Imminent_B" to the TU with a message ID of 4372.

[0071] 12. Verify that the TU received the emergency alert message "CMAS_Imminent_B" and confirm that its message identifier is assigned as MsgID3.

[0072] If the TU interface displays information about the new alert, it indicates that the device correctly received and processed different types of alerts.

[0073] 13. Confirm the "CMAS_Imminent_B" alert on the TU.

[0074] 14. Deactivate the 5G cell and turn off the TU device, completing the test.

[0075] Decision criteria If the TU successfully performs all the above steps and correctly assigns the corresponding message identifiers (MsgID1, MsgID2, MsgID3) for each alert message, the test is passed.

[0076] Observation: As shown in the screenshots in each step, the TU device can correctly receive, recognize, and display alert information, indicating that the device meets the requirements of the test case.

[0077] Test results After each alarm is sent, the TU device interface pops up a corresponding prompt message, matching the content of the sent alarm message, indicating that the TU successfully received and processed the alarm.

[0078] The TU device can correctly specify the message identifier and perform the confirmation operation.

[0079] Test Conclusion Based on the observed screenshots and descriptions, the TU device performed well in this test case, meeting the requirements of each test point, so it can be determined that this test case passes.

[0080] The following is the process of executing this test case using RPA / ADB automation tools and the output test results.

[0081] Test Preparation: Start the intelligent recognition general tool, ensure that the YOLO model is configured for interface recognition, and use RPA for automation operation.

[0082] Connect the terminal device (TU) in the test environment to the system, ensuring that YOLO and RPA / ADB tools can take screenshots and operate the interface of the TU device.

[0083] Test Steps: 1. Use RPA / ADB automation tools to start the TU device and perform interface checks to ensure the device is ready.

[0084] 2. The relevant scripts in the test instrument execution script are executed, simulating the activation of a 5G SA NR cell.

[0085] 3. Simulate sending an alarm message in the test instrument system, sending "National alert_A_A", with message ID 4370.

[0086] 4. Get the instrument interface and identify the pop-up window in the instrument, use the YOLO model to identify the screenshot, find and determine the alarm pop-up window containing "National Level Alerts".

[0087] Use RPA / ADB automation tools to compare the recognized text content to confirm whether "National Level Alert_A_A" is displayed on the interface.

[0088] Record the results: save the recognition results to the log, marked as successful or failed recognition.

[0089] 5. Use RPA / ADB automation tools to simulate clicking the "Submit" button in the test instrument pop-up window to complete the confirmation of the alarm. Record the results of the confirmation operation.

[0090] 6. Simulate sending alert message "CMAS_Imminent_A", message ID 4371.

[0091] 7. Verify that the test meter pop-up is recognized through the same screenshot and recognition process, and verify that the TU received the "CMAS_Imminent_A" alert message "Extreme Alerts with Severity of Extreme, Urgency of Immediate and Certainty of Observed".

[0092] Record the verification results.

[0093] 8. Confirm the emergency alert message "CMAS_Imminent_A" and use the RPA / ADB automation tool to automatically click the "OK" button in the pop-up, and record the operation results.

[0094] Use the RPA / ADB automation tool to automatically click the "Submit" button in the test meter pop-up.

[0095] 9. Simulate sending alert message "CMAS_Imminent_B", message ID 4372.

[0096] 10. The automation tool intercepts the screen and recognizes the pop-up content to verify whether it is "CMAS_Imminent_B".

[0097] Record the verification results.

[0098] 11. Use RPA to automatically click the "OK" button in the pop-up, and use RPA to automatically click the "Submit" button in the test meter pop-up to confirm the emergency alert message "CMAS_Imminent_B".

[0099] 12. The automation tool deactivates the 5G cell, turns off the TU device, ends the test, and outputs the test results.

[0100] If the recognition result is the identification of a national level alert message (National Level Alert_A_A), it indicates that the identification state is successful, and the YOLO model successfully identifies that the pop-up content is "National Level Alerts" and the text content matches the sent message.

[0101] If the system records the receipt and confirmation of the "National Level Alert_A_A" message in the log, it indicates that the state is successful.

[0102] If the imminent alert message is identified as (CMAS_Imminent_A), it indicates that the identification state is successful.

[0103] If the smart identification tool successfully identifies the alert pop-up containing "Extreme Alerts with Severity of Extreme" and records the corresponding message ID, and the identification and confirmation operations are recorded in the log, it indicates that the state is successful.

[0104] If the imminent alert message is identified as (CMAS_Imminent_B), it indicates that the identification state is successful, and the YOLO model identifies the alert message in the pop-up and confirms that the content of "Extreme Alerts with Severity of Extreme, Urgency of Immediate" matches, indicating that the operation result is recorded as successful.

[0105] In this embodiment, the automated tool can accurately click the "OK" button after each identification step to complete the confirmation of the alert. The system records detailed information in the log during each operation, including screenshots, identified content, operation status, and operation delay.

[0106] In addition, during the test, the system did not have identification errors or operation failures, so the exception handling mechanism was not triggered.

[0107] In the performance indicators, the average identification time is about 0.9 seconds, which meets the real-time requirements. The average operation time is about 0.8 seconds, and the operation is accurate and error-free.

[0108] Test results summary The YOLO model successfully identified all the alert messages sent in the TC-11551 test and confirmed that the pop-up content matched the sent alerts, with an accuracy of 100%.

[0109] In addition, the RPA tool can accurately simulate the click operation to complete the confirmation of the alert message, and the operation success rate is also 100%.

[0110] No abnormal situations occurred in this test, and the tool performed stably.

[0111] In summary, the intelligent identification general tool performed well in the TC-11551 use case test, accurately identified the alert messages on the TU device interface, and completed the automated operation, meeting the requirements of the test case.

[0112] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit it. Although the present application has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the specific embodiments of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A method for interface recognition and automatic testing based on YOLO in a multi-system environment, characterized in that, The method comprises the following steps: Using a pre-trained YOLO model to recognize and detect the test software interface screenshot obtained through desktop screenshot; Using RPA or ADB tools to realize automatic operation, including process recording, custom logic, simulating mouse and keyboard operation, calling external scripts and commands; Performing black-box testing from the user's perspective, without focusing on the internal implementation details of the tools, to verify whether the output results of the YOLO model under various inputs meet the expectations; Performing white-box testing on the YOLO model, RPA operation scripts, and ADB tool commands for fine-grained testing; Writing test scripts to automate the recognition and operation testing.

2. The interface identification and automated testing method of claim 1, wherein, The test method further comprises the following steps: Verifying the recognition accuracy of the YOLO model in different operating systems and different application environments; Verifying whether the RPA or ADB tools can perform interface operations based on the recognition results; Verifying whether the exception handling mechanism of the RPA or ADB tools is effective.

3. The interface identification and automated testing method of claim 1, wherein, The test method further comprises the following steps: Testing the recognition accuracy of the YOLO tool for pop-ups, buttons, prompt information, and icons in various scenarios; Testing the automation operation capability of the RPA or ADB tools to ensure that operations are performed based on the coordinates and content of the recognition results; Testing the process control capability of the tools to verify whether the RPA or ADB tools can correctly select and execute the corresponding operation processes under different recognition results; Testing the processing capability of the tools in the case of recognition failure, operation failure, and interface changes.

4. The interface identification and automated testing method of claim 1, wherein, The test method further comprises the following steps: Testing the total response time of the tools from screenshot acquisition to automatic operation in different operating environments; Testing the processing delay in different scenarios, calculating the average delay and maximum delay, and evaluating the impact on actual testing work; Running multiple recognition and operation tasks simultaneously on the same device to test the concurrent processing capability of the tools and verify whether the RPA or ADB tools can maintain stable operation and respond in a timely manner under high load.

5. The interface identification and automated testing method of claim 1, wherein, The test method further comprises the following steps: Simulating different abnormal situations to test the exception handling capability of the YOLO recognition module and verify whether the tool can perform remedial measures after recognition failure; Simulating abnormal situations during the automation operation process to test the exception handling capability of the RPA or ADB tools, ensuring that the tools can perform log recording, exception feedback, and attempt to recover after operation failure; Introducing artificial fault points into the test process to verify whether the recovery mechanism of the tools can recover to the last checkpoint and continue execution after an exception occurs.

6. The interface recognition and automated testing method of claim 1, wherein, The test method further comprises the following steps: Configuring the test environment, installing relevant software and tools, ensuring that the test device is in normal working condition, importing the YOLO model and dataset, and configuring the RPA tool and ADB; Testing the recognition module, automation operation module, and process control module one by one according to the functional test items in the test plan, and recording the test input, output, actual result, and expected result in each test item; Measuring the response time of the tools in different scenarios, recording the average delay, maximum delay, and concurrent processing performance, and verifying the performance and stability of the tools under high load. Simulate abnormal scenarios, verify the tool's exception handling and recovery capabilities, and record the types, frequency, processing time, and recovery of exceptions. Analyze test results, evaluate the tool's functional integrity, identification accuracy, operational precision, real-time performance, and exception handling capabilities, and generate a test report based on the test results.

Images