Mobile phone interface automatic testing method based on ADB and YOLO

Abstract

The invention relates to the technical field of mobile phone testing, and provides a mobile phone interface automatic testing method based on ADB and YOLO, which comprises the following steps: acquiring a mobile phone screenshot by using an ADB tool; carrying out image preprocessing on the obtained mobile phone screenshot; a pre-trained YOLO model is loaded, the preprocessed mobile phone screenshot is input into the YOLO model for target detection, and a detection result is obtained; executing OCR character recognition in a detection result of the YOLO model, and extracting prompt text content on a mobile phone interface; the text content recognized by the OCR is matched with predefined keywords, rules or regular expressions, whether prompt information on a mobile phone interface meets the expectation or not is judged, and the current operation state of the mobile phone is judged by recognizing the state of a specific icon or button on the mobile phone interface; and recording prompt information of a mobile phone interface into a log according to an identification result, and when further operation is needed, simulating key operation by using an ADB command to complete a corresponding automatic task.

Description

Technical Field

[0001] This invention relates to the field of mobile phone testing technology, and more specifically, to an automated testing method for mobile phone interfaces based on ADB and YOLO. Background Technology

[0002] Automated testing is an important means to improve testing efficiency and reduce human error. Traditional automated testing tools rely heavily on script writing, which makes it difficult to adapt to complex and dynamically changing interface environments. Although automated testing methods have been widely used, there are still some problems that need to be solved, which to some extent limit testing efficiency and accuracy, and affect the development and optimization process of mobile applications.

[0003] First, traditional mobile interface testing methods mostly rely on manual operation or simple script automation. Manual testing is not only time-consuming and labor-intensive, but also prone to inconsistencies in test results due to human factors. Simple script automation often fails to accurately identify and handle various interface states when faced with complex interface elements and dynamically changing prompts. Second, existing technologies may experience delays in the testing process when acquiring and processing mobile screen screenshots due to limitations in screenshot frequency and transmission speed, affecting the real-time performance and smoothness of the test. Furthermore, existing testing technologies have poor adaptability to different mobile phone brands, system versions, and applications. Due to the diversity and complexity of mobile interface elements, including different resolutions, fonts, background colors, and prompt types, existing testing methods often struggle to fully cover these differences, impacting the accuracy and reliability of test results. Finally, existing testing technologies have shortcomings in automated operation feedback. After recognizing prompts requiring further operation, they cannot automatically execute the corresponding button operations, still requiring manual intervention. This not only increases testing costs but also reduces testing efficiency. Summary of the Invention

[0004] In view of this, the present invention proposes an automated testing method for mobile phone interfaces based on ADB and YOLO, so as to improve the efficiency and accuracy of automated testing of mobile phone interfaces and solve the problems existing in the prior art.

[0005] To achieve the above objectives, this invention proposes an automated testing method for mobile phone interfaces based on ADB and YOLO, characterized by the following steps: Use ADB tools to capture screenshots from your phone and then transfer them to your local computer for processing. The acquired screenshots are preprocessed, including resizing and color space conversion, to meet the input requirements of the YOLO model. Load the pre-trained YOLO model, input the pre-processed mobile phone screenshot into the YOLO model for object detection, and obtain the detection results including bounding box position, class label and confidence score; Perform OCR text recognition on the detection results of the YOLO model to extract the prompt text content on the mobile phone interface; The text content recognized by OCR is matched with predefined keywords, rules or regular expressions to determine whether the prompts on the mobile phone interface meet expectations, and the current operation status of the mobile phone is determined by recognizing the state of specific icons or buttons on the mobile phone interface. Based on the recognition results, the prompts on the mobile phone interface are recorded in the log, and when further operation is required, ADB commands are used to simulate key presses to complete the corresponding automated tasks.

[0006] Furthermore, the step of using ADB tools to obtain screenshots of the mobile phone screen also includes adjusting the screenshot frequency according to needs to balance real-time performance and performance, and using a compression format to speed up the transmission.

[0007] Furthermore, in the image preprocessing step, the obtained mobile phone screenshot is adjusted to the standard size of the YOLO model input, and there is no need to convert the image to grayscale; it is directly input into the YOLO model for recognition.

[0008] Furthermore, in the step of loading the pre-trained YOLO model, the model can be trained to recognize various prompts, pop-ups, icons, and buttons on the mobile phone interface. For specific recognition tasks, the YOLO model is fine-tuned using transfer learning to adapt to specific application scenarios.

[0009] Furthermore, in the step of performing OCR text recognition on the region detected by YOLO, the detection results provided by the YOLO model are used to perform OCR recognition on a specific region in the detection results to extract the text content.

[0010] Furthermore, in the process of determining whether the prompts on the mobile phone interface meet expectations, the YOLO model is trained by adding categories of icons in different states. The YOLO model is then used to identify the state of specific icons or buttons on the mobile phone interface, determine the current operating state of the mobile phone, and perform classification.

[0011] Furthermore, when the recognition result indicates that further operation is required, ADB commands are used to simulate key presses to complete the corresponding automated task.

[0012] Furthermore, the training data for the YOLO model during the training process includes different mobile phone brand system versions and prompts, icons, and buttons in the application.

[0013] Furthermore, the process of using ADB commands to simulate key presses to complete corresponding automated tasks includes: When a mobile device is connected to a PC, the ADB server automatically recognizes the connected device and starts the ADB daemon process on the device. Start the ADB server and establish communication with the Android device to obtain the device's serial number and connection status; Obtain the corresponding ADB command based on the prompts on the mobile phone screen, and execute the ADB command on the mobile device through the ADB daemon process; The execution result is transmitted back to the ADB server via the TCP protocol, and the ADB server then forwards the result to the ADB client, which outputs it to the PC's command line.

[0014] Furthermore, the ADB commands include, but are not limited to, installing APKs, uninstalling applications, transferring files between PC and mobile phone, viewing mobile device logs, entering the Linux shell of mobile device, and restarting mobile device.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention combines ADB tools with the YOLO model. ADB is used to capture screenshots of the mobile phone screen and transmit them to a local computer. The YOLO model then performs object detection and OCR text recognition, achieving automated testing of the mobile phone interface. This not only improves the accuracy and efficiency of testing but also reduces the need for manual intervention. Furthermore, by adjusting the screenshot frequency and using compressed formats for screenshot transmission, this invention effectively balances real-time performance and performance. This allows the testing process to maintain high real-time performance while reducing transmission latency and resource consumption, ensuring smooth and stable testing. In this invention's technical solution, the YOLO model is fine-tuned through transfer learning to adapt to interface elements of different mobile phone brands, system versions, and applications. Simultaneously, OCR recognition has strong adaptability to different fonts, background colors, and brightness, improving the generalization ability and accuracy of the test. Finally, based on the recognition results, this invention automatically records prompts and uses ADB commands to simulate key presses, completing the automated task. This not only improves the degree of automation in testing but also reduces the error rate of manual operation, thus increasing testing efficiency. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a schematic diagram of the overall process of the automated testing method for mobile phone interfaces based on ADB and YOLO proposed in this invention. Detailed Implementation

[0017] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] This embodiment proposes an automated testing method for mobile phone interfaces based on ADB and YOLO, such as... Figure 1 As shown, it includes: Step 1: Take a screenshot Using ADB to capture phone screenshots: This tool allows you to remotely control an Android phone and capture screenshots. ADB provides screenshot commands that can be used to capture the current screen of the phone in real time and transfer the screenshots to your local computer for processing.

[0019] Optimize screenshot transmission: Since transmitting screenshots via ADB may introduce some latency, the screenshot frequency can be adjusted according to needs to balance real-time performance and efficiency. Using a compression format (such as JPEG) can speed up transmission.

[0020] Step 2: Image Preprocessing Adjust size: Adjust the obtained mobile phone screenshot to the standard size of the YOLO model input (such as 416x416 or 640x640) to meet the requirements of the YOLO model.

[0021] Color space conversion: YOLO typically uses RGB images for detection, so there is no need to convert the image to grayscale; it can be directly input into the model for recognition.

[0022] Step 3: Image recognition and analysis based on the YOLO model YOLO Model Loading: This section describes object detection using a pre-trained YOLO model. The model can be trained to recognize various prompts, pop-ups, icons, buttons, etc., on a mobile interface. For specific recognition tasks, such as specific text prompts or icon state changes, transfer learning can be used to fine-tune the YOLO model to adapt it to specific application scenarios.

[0023] Object detection: Input the screenshot into the YOLO model to obtain the detection results, including bounding box positions, class labels, and confidence scores. For mobile interface recognition, class labels can include "tooltip", "error message", "button", "icon", etc.

[0024] Step 4: Use OCR for text recognition Perform OCR on the area detected by YOLO: Use OCR (such as Tesseract) to recognize text in the cropped prompt information area and extract the prompt text content on the mobile phone interface. This process utilizes the precise positioning provided by the YOLO model to focus on OCR recognition of specific areas, avoiding interference from irrelevant areas and improving recognition accuracy.

[0025] Step 5: Content Analysis and Judgment Content matching: Matches the text content recognized by OCR with predefined keywords, rules, or regular expressions to determine whether the prompts on the mobile interface meet expectations. For example, it detects whether the prompt box contains keywords such as "test passed" or "error".

[0026] State determination: By recognizing the state (such as color or shape changes) of specific icons or buttons on the phone's interface, the current operating state of the phone can be determined. This can be achieved by adding icon categories for different states during YOLO model training, and then using the model for classification.

[0027] Step Six: Operation Feedback Results Recording: Based on the recognition results, the prompts on the mobile phone interface are recorded in the log for subsequent analysis and reference.

[0028] Automated operation: If the recognition result indicates that further operation is required (such as clicking the confirmation button or entering information), ADB commands can be used to simulate key presses to complete the corresponding automated task.

[0029] As a preferred embodiment, since different mobile phone brands, system versions, and application interface elements vary greatly, including prompts, icons, buttons, etc., which differ significantly in shape, color, and size, the YOLO model needs to cover a sufficiently diverse range of samples during the training phase to have the ability to generalize and recognize these elements.

[0030] Furthermore, the prompts on the mobile interface are dynamically changing; the text prompts and icon states may change depending on the application's running status. Therefore, OCR recognition needs to be highly adaptable, accurately extracting text information under different conditions.

[0031] As a preferred embodiment, in order to accurately identify the prompts on the mobile phone interface, the YOLO model needs to be finely labeled and trained. The training data needs to cover various types of prompts, icons, and button states to improve the model's detection accuracy.

[0032] Steps such as ADB screenshot acquisition, YOLO detection, and OCR recognition need to be completed within a short time to meet the requirements of real-time automated testing. Therefore, the selection and use of the YOLO model requires a trade-off between model size, input size, and detection speed to ensure that the entire process can be completed within a few hundred milliseconds.

[0033] In addition, some icons on mobile phone interfaces have multiple states (such as color and shape changes). The YOLO model needs to be able to distinguish these states. By labeling the icon categories of different states in the training dataset, the model can be made capable of recognizing these changes.

[0034] ADB Implementation Principles and Technical Analysis ADB works by establishing a communication bridge between a PC and an Android device, allowing the PC to remotely control the Android device via command line. Its core principles include the following components: (1) Client-server architecture ADB uses a client-server architecture and consists of three parts: ADB client: Runs on the developer's PC and sends commands to the ADB server.

[0035] ADB Server: Runs on a PC and is responsible for managing communication with Android devices. After starting, the ADB server listens on a specific port (usually 5037), receives commands from clients, and forwards them to the corresponding devices.

[0036] The ADB daemon (adbd) runs on the Android device and is responsible for executing received commands. It establishes communication with the ADB server, receives commands, and executes them on the device.

[0037] (2) Communication Protocol Communication between the ADB client and server uses the TCP protocol. The ADB server communicates with the ADB daemon on the Android device via USB or network connection. Through this communication method, ADB can transmit commands from the PC to the device for execution and return the execution results to the PC.

[0038] (3) Command execution The ADB client sends commands to the ADB server, which forwards the commands to the ADB daemon on the specified device. Upon receiving the command, the daemon executes the corresponding operation on the device and returns the result to the server, which then sends the result back to the client.

[0039] The process of using ADB mainly involves the following steps: Step 1: Establish a connection USB connection: Connect your Android device to your PC via USB cable. The ADB server will automatically recognize the connected device and start the ADB daemon on the device.

[0040] Wi-Fi connectivity: Establishes a connection with the Android device via a wireless network. Developers can use ADB commands to put the device into wireless debugging mode and then connect via the device's IP address.

[0041] In this embodiment, in order to ensure the stability of the test environment and the compatibility of the application scenarios, a USB connection is used for testing and verification.

[0042] Step 2: Start the ADB server Type `adbstart-server` in the command line to start the ADB server. The ADB server will run in the background and listen on port 5037 on your PC to receive commands from ADB clients.

[0043] Step 3: Device Identification Use the command `adbdevices` to list currently connected Android devices. The ADB server communicates with Android devices via USB or Wi-Fi to obtain information such as the device's serial number and connection status.

[0044] Step 4: Command Execution ADB provides a variety of commands that can perform various operations on the device. Commonly used ADB commands include: Application management: adbinstall (install APK), adbuninstall (uninstall application).

[0045] File transfer: adbpush (transfer files from PC to mobile phone), adbpull (transfer files from mobile phone to PC).

[0046] Log output: adblogcat (view device logs).

[0047] Device control: adbshell (enter the device's Linux shell), adbreboot (reboot the device).

[0048] When the adbshell command is executed, the ADB client will enter the Android device's Linux command-line environment, allowing the execution of various Linux commands to manipulate the device's file system, view process status, and so on.

[0049] Step 5: Command results are returned. After the ADB daemon executes a command on the mobile device, it sends the execution result back to the ADB server via the TCP protocol. The server then forwards the result to the ADB client, which outputs it to the command line on the PC.

[0050] As a preferred embodiment, the key technical aspects of ADB include the following: Client-Server-Daemon Architecture: ADB employs a three-tier architecture. The client sends commands, the server manages devices and forwards commands, and the daemon executes commands on the device. This design allows ADB to flexibly manage multiple devices and provides powerful command control capabilities.

[0051] Multiple device connections: ADB supports connecting Android devices via both USB and Wi-Fi, facilitating device debugging for developers in different environments. Wi-Fi connectivity allows developers to remotely debug devices, which is particularly suitable for scenarios requiring simultaneous testing of multiple devices.

[0052] A comprehensive command set: ADB provides a rich set of commands covering various aspects such as device management, application management, file operations, log output, and network debugging. Through the adbshell command, developers can directly access the device's Linux command-line environment for system-level operations.

[0053] Data transfer and file system operations: ADB supports the push and pull commands to transfer files from a PC to a device or vice versa. This allows developers to easily perform file operations between devices and PCs, manage the file system, and retrieve test data.

[0054] Log System: Using the `adblogcat` command, ADB can capture system logs on the device in real time, allowing developers to perform debugging and analysis. This log system records detailed information such as the device's operating status, application behavior, and system errors, making it an important tool for debugging and problem localization.

[0055] AndroidShell Interaction: ADB allows direct access to the device's command-line environment via shell commands. Developers can use various Linux commands to perform file operations, launch applications, view system processes, and more. This provides great flexibility for more advanced device operations and custom testing.

[0056] Verification method (1) Screenshot recognition accuracy verification Test Dataset: A large number of screenshots were collected from different types of mobile phones and application interfaces to construct a diverse test dataset. The test dataset includes screenshots of different resolutions, languages, and tooltip types to comprehensively evaluate the recognition capabilities of the YOLO model.

[0057] Detection accuracy: Run the YOLO model on the test dataset and calculate the detection accuracy (number of correctly detected elements / total number of elements). Adjust model parameters to improve the model's recognition performance on different mobile phone interfaces.

[0058] (2) Verification of OCR recognition accuracy Text recognition test: Perform OCR recognition on the prompt area detected by YOLO, and calculate the OCR accuracy (number of correctly recognized characters / total number of characters). Consider various conditions during the test, such as different fonts, background colors, and brightness, to evaluate the adaptability of the OCR.

[0059] (3) Real-time performance testing Processing latency: The total latency from acquiring the screenshot to completing image detection and text recognition was calculated to ensure the entire process could be completed within hundreds of milliseconds. Real-time performance was optimized by adjusting the input size of the YOLO model and reducing the number of detection layers.

[0060] (4) Scalability verification Multi-device testing: Test the system on various Android devices and application scenarios to verify its recognition accuracy under different resolutions, interface element changes, and language environments. Based on the test results, optimize the model and dataset to improve the system's cross-device adaptability.

[0061] Unless otherwise stated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. While only preferred methods and materials have been described herein, any methods and materials similar or equivalent to those described herein may be used in the implementation or testing of this invention. All references to this specification are incorporated by way of citation to disclose and describe methods and / or materials associated with those references. In the event of any conflict with any incorporated reference, the content of this specification shall prevail.

[0062] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A mobile interface automated testing method based on ADB and YOLO, characterized in that, Includes the following steps: Use ADB tools to capture screenshots on your phone and then transfer them to your local computer for processing. The acquired screenshots are preprocessed, including resizing and color space conversion, to meet the input requirements of the YOLO model. Load the pre-trained YOLO model, input the pre-processed mobile phone screenshot into the YOLO model for object detection, and obtain the detection results including bounding box position, class label and confidence score; Perform OCR text recognition on the detection results of the YOLO model to extract the prompt text content on the mobile phone interface; The text content recognized by OCR is matched with predefined keywords, rules or regular expressions to determine whether the prompts on the mobile phone interface meet expectations, and the current operation status of the mobile phone is determined by recognizing the state of specific icons or buttons on the mobile phone interface. Based on the recognition results, the prompts on the mobile phone interface are recorded in the log, and when further operation is required, ADB commands are used to simulate key presses to complete the corresponding automated tasks.

2. The automated testing method for mobile phone interfaces based on ADB and YOLO according to claim 1, characterized in that, The steps of using ADB tools to obtain screenshots of mobile phone screens also include adjusting the screenshot frequency according to needs to balance real-time performance and using compression formats to speed up transmission.

3. The automated testing method for mobile phone interfaces based on ADB and YOLO according to claim 1, characterized in that, In the image preprocessing step, the obtained mobile phone screenshot is adjusted to the standard size of the YOLO model input, and the image is directly input into the YOLO model for recognition without converting it to grayscale.

4. The automated testing method for mobile phone interfaces based on ADB and YOLO according to claim 1, characterized in that, In the step of loading the pre-trained YOLO model, the model can be trained to recognize various prompts, pop-ups, icons, and buttons on the mobile phone interface. For specific recognition tasks, the YOLO model is fine-tuned using transfer learning to adapt to specific application scenarios.

5. The automated testing method for mobile phone interfaces based on ADB and YOLO according to claim 1, characterized in that, In the step of performing OCR text recognition on the area detected by YOLO, the detection results provided by the YOLO model are used to perform OCR recognition on a specific area in the detection results to extract the text content.

6. The automated testing method for mobile phone interfaces based on ADB and YOLO according to claim 1, characterized in that, In the process of determining whether the prompts on the mobile phone interface meet expectations, the YOLO model is trained by adding categories of icons in different states. The YOLO model is then used to identify the state of specific icons or buttons on the mobile phone interface, determine the current operating state of the mobile phone, and perform classification.

7. The automated testing method for mobile phone interfaces based on ADB and YOLO according to claim 1, characterized in that, When the recognition result indicates that further operation is required, ADB commands are used to simulate key presses to complete the corresponding automated task.

8. The automated testing method for mobile phone interfaces based on ADB and YOLO according to claim 1, characterized in that, The training data for the YOLO model during training includes different mobile phone brand system versions and application prompts, icons, and buttons.

9. The automated testing method for mobile phone interfaces based on ADB and YOLO according to claim 1, characterized in that, The process of using ADB commands to simulate key presses to complete corresponding automated tasks includes: When a mobile device is connected to a PC, the ADB server automatically recognizes the connected device and starts the ADB daemon process on the device. Start the ADB server and establish communication with the Android device to obtain the device's serial number and connection status; Obtain the corresponding ADB command based on the prompts on the mobile phone screen, and execute the ADB command on the mobile device through the ADB daemon process; The execution result is transmitted back to the ADB server via the TCP protocol, and the ADB server then forwards the result to the ADB client, which outputs it to the PC's command line.

10. The automated testing method for mobile phone interfaces based on ADB and YOLO according to claim 1, characterized in that, The ADB commands include, but are not limited to, installing APKs, uninstalling applications, transferring files between PC and mobile phone, viewing mobile device logs, entering the Linux shell of mobile device, and restarting mobile device.

Images