Mobile application automatic testing method based on multi-axis mechanical arm physical simulation and reinforcement learning

By combining robotic arm control technology and reinforcement learning model, non-invasive automated testing exploration for mobile applications is realized, solving the shortcomings of existing automated testing methods in unknown or closed systems, and improving the comprehensiveness and perfection of the test.

CN118918056APending Publication Date: 2024-11-08NANJING UNIV
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202310512064.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing automated testing exploration methods tend to be software testing and rely on invasive means. It is difficult to find touch problems in actual operation, and it is impossible to deal with unknown or closed systems. The test is not perfect and comprehensive.

Method used

Combining multi-axis robotic arm control technology, image processing technology and automated testing methods based on reinforcement learning, we take screen photos of mobile applications in real time through robotic arm cameras, use traditional computer vision and deep learning technologies to understand images, and generate action commands to control machinery. The arm is subject to automated testing exploration.

Benefits of technology

It realizes automated testing exploration based on robotic arms, can analyze mobile application interfaces in real time, accurately identify controls and page layouts, and perform touch operations through reinforcement learning models, improving the perfection and comprehensiveness of tests, solving the limitations of invasive testing. sex.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918056A_ABST
    Figure CN118918056A_ABST
Patent Text Reader

Abstract

The invention discloses a mobile application automatic testing method based on multi-axis mechanical arm physical simulation and reinforcement learning. A photo processing module, an image understanding module, a reinforcement learning module and a mechanical arm control module are logically included. And the photo processing module is used for controlling the definition of a photo shot by the camera and carrying out image enhancement, contour detection and screen capture by using a series of traditional computer vision technologies. The image understanding module comprises control recognition and page layout recognition, UI controls in a screen are recognized by comprehensively adopting multiple technologies through control recognition, the UI controls are structurally organized through page layout, and overall page information is provided. The reinforcement learning module comprises deep reinforcement learning components such as a Q-Network neural network model, self-training is carried out according to image understanding information, and an instruction is generated to guide the mechanical arm to carry out automatic test exploration. And the mechanical arm control module receives operation scheduling from the server through an RPC protocol, analyzes an instruction through an inverse kinematics algorithm, and converts and completes simulation test operation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of software testing, and is particularly used for application scenarios combining non-invasive testing with automated testing exploration. The images of the application being tested taken by the robot arm camera in real time are continuously processed and transmitted to the server. The server deeply understands the images, learns the application status in the test space with the help of a reinforcement learning model constructed by a neural network, and generates operation instructions in real time to drive the robot arm to simulate human interaction and implement automated testing. Background Art

[0002] In recent years, mobile Internet and mobile devices have developed rapidly, and mobile applications have been related to all aspects of social life. Therefore, the testing technology that ensures the quality of mobile applications has become increasingly important. In order to meet the characteristics of diverse mobile application versions and rapid iterations, automated testing technology for mobile applications has become a hot topic in academia and industry. Automated testing is a process that converts human-driven testing behavior into machine execution or even machine drive. Usually, after designing test cases and passing the review, testers perform the test step by step according to the procedures described in the test cases to compare the actual results with the expected results. In this process, the use of automated mobile application testing can save manpower, time or hardware resources, greatly improve testing efficiency and reduce testing costs.

[0003] One method of automated testing is the automated testing exploration technology for mobile applications, which has developed rapidly in recent years. This technology aims to use the mobile application itself as the only source of information, automatically build and execute the test process according to a certain method and strategy, and minimize the human resource investment in testing. The mainstream methods are to build exploration strategies based on randomness, model-based, and reinforcement learning. The generalization ability of existing automated testing exploration technologies is getting stronger and stronger, but almost all of them contain invasive components, requiring plug-ins or relying on system-provided interfaces (such as ADB in Android systems), and cannot truly achieve complete platform isolation, and cannot meet the software testing requirements of touch-screen electronic products or embedded devices on unknown systems and closed systems. At the same time, some practical problems in the testing process, such as the UI control being unclickable due to the occlusion of the special-shaped screen, cannot be perceived due to the poor simulation of invasive testing. In addition, some software products with strict operating environment requirements may be affected by invasive operations such as plug-ins, and cannot be tested accurately and effectively.

[0004] Introducing robotic arms into the automated testing process can achieve non-invasive testing and effectively solve the above problems. In terms of combining robotic arms with automated testing technology, some researchers have done some work, such as using cameras and physical robotic arms to assist in recording and replaying test scripts, verifying the feasibility of robotic arm testing tools for quantitative testing of industrial application scenarios and their potential generalization capabilities.

[0005] Based on the above work, the present invention utilizes the flexibility and operability of a multi-axis physical robotic arm, and deeply mines the rich information in the image. In the process of automated test exploration, performing test operations in an invasive manner is fast, stable and easy to implement, but for the robotic arm, further technical processing is required to correctly and efficiently connect to the cutting-edge automated test exploration method for testing. Therefore, based on these existing research results, the present invention summarizes and modifies the robotic arm control technology, image processing technology and automated testing methods based on reinforcement learning, and applies them to screen and control recognition in mobile application images captured by cameras, as well as test operation selection strategies, thereby using robotic arms to perform intelligent non-invasive mobile application automated testing exploration. Summary of the invention

[0006] The problem to be solved by the present invention is that the existing automated test exploration methods tend to be software testing and rely on intrusive means to varying degrees, making it difficult to discover problems such as touch in actual operations, and difficult to deal with unknown or closed systems, and the completeness and comprehensiveness of the test are poor. Our invention can realize automated test exploration based on a robotic arm, capture and analyze the interface of mobile applications in real time, perform touch operations according to the guidance of a reinforcement learning model, and replace testers to perform tests, thereby solving the limitations of existing automated testing.

[0007] The technical solution of the present invention is: a method for automated testing of mobile applications based on multi-axis robotic arm physical simulation and reinforcement learning, which is characterized by being able to parse the screen and deeply understand the screen content based on the real-time application photos taken by the robotic arm camera, and generate action instructions to control the robotic arm for automated testing exploration through real-time learning and decision-making through the reinforcement learning model. The method logically includes the following four modules:

[0008] 1) Photo processing module: The monitoring camera captures the screen shooting process, captures continuous shooting frames and performs blur detection, retaining relatively clear shooting results. Based on the photos of the screen taken, a series of traditional computer vision technologies are used based on the OpenCV library to enhance the image, using the Canny edge detection and edge enhancement methods to obtain closed contours, using the minimum rectangle approach to identify the screen contour and perform perspective transformation and deflection correction, from which the effective screen part is captured to complete the screenshot.

[0009] 2) Image understanding module: Perform deep recognition of controls and page layouts respectively. In control recognition, OCR technology is used to recognize text content, and a combination of traditional computer vision technology based on the OpenCV library and deep learning technology is used to recognize graphic controls. Controls are initially extracted through methods such as Canny edge detection, and then the control is identified through the target detection model of GUI controls based on the Yolov7 algorithm retrained on the Rico dataset. The preliminary recognition results are screened and supplemented. The recognition results of text content and graphic controls are integrated and embedded using the pre-trained VGG-16 model to obtain the corresponding vectors; in page layout recognition, page sections are cut and grouped according to the outer edges, and UI controls are organized into a tree structure and serialized, and embedded using the pre-trained LSTM model to obtain the corresponding vectors.

[0010] 3) Reinforcement learning module: Based on the results of image understanding, the vector representations of controls and screen layouts are connected to represent the application state of the test space, and the controls and corresponding operation methods are vector-embedded to represent the test operation. Both are used as the input of the Q-Network neural network model, and the corresponding Q value is output and the test operation with a large Q value is selected as the next operation to be executed. At the same time, the model will continuously update the Exploration Memory based on the historical state, historical execution actions and state transfer, and perform real-time iterative training to provide better decisions.

[0011] 4) Robotic arm control module: The server completes the conversion of the coordinates of the control to be clicked from the image space to the physical operation space, and uses RPC technology to control the robotic arm to click the target control according to the agreed interface and rules. The robotic arm opens the port to listen to the method call from the server, obtains the coordinates of the target space and calculates the moving distance, uses the inverse kinematics algorithm to convert the moving distance from the physical operation space to the robotic arm joint space, obtains the angle that each servo needs to rotate, and calls the underlying API pre-packaged by the robotic arm system to control the robotic arm.

[0012] The present invention is characterized in that:

[0013] 1. For the first time in the field of mobile application automated testing, robotic arm simulation is combined with automated testing exploration methods based on reinforcement learning.

[0014] 2. For the first time, a non-invasive automated screenshot method was established to capture, identify and capture screen images in real time.

[0015] 3. Optimize the deep image understanding method of the mobile application interface and connect it with the deep reinforcement learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1Flow chart for implementing the present invention.

[0017] Figure 2 This is a detailed flowchart of the photo processing module.

[0018] Figure 3 Detailed flowchart of the image understanding module.

[0019] Figure 4 Detailed flowchart of the reinforcement learning module.

[0020] Figure 5 This is a detailed flow chart of the robotic arm control module. DETAILED DESCRIPTION

[0021] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification.

[0022] This patent implements non-intrusive mobile application automated testing through the following methods, mainly using traditional computer vision technology and reinforcement learning technology. The key specific technologies involved include RPC technology, OCR technology, Canny technology, image enhancement technology, deep reinforcement learning technology, inverse kinematics technology, etc.

[0023] 1. Encapsulate interactive instructions

[0024] In the present invention, we use RPC technology to encapsulate the interaction between the robotic arm and the server. The server remotely calls the interface provided by the robotic arm system, informs the robotic arm of the operation type and the location of the operation target control, controls the robotic arm to perform the test operation, and calls the robotic arm camera to take pictures after the execution is completed and returns the shooting results to the server.

[0025] 2. Processing shooting process

[0026] In the present invention, in order to control the clarity of the shooting results, we adopt the producer-consumer model and establish a thread-safe queue, continuously pushing the camera frames to the end of the queue, and continuously popping up the frames at the head of the queue, converting their grayscale and performing convolution operations with the Laplace kernel, performing blur detection according to the variance size, and retaining the clearest frame until its clarity reaches a threshold or the maximum shooting time is set.

[0027] 3. Capture the effective screen

[0028] In the present invention, we use grayscale conversion, Gaussian blur noise reduction, and Boolean index shadow removal to enhance the image according to the shooting results, use Canny edge detection, probabilistic Hough line transform and contour dilation to obtain closed contours, use minimum rectangle approximation to identify the screen contour and perform perspective transformation and deflection correction, so as to capture the effective screen part from the captured photo containing the screen.

[0029] 4. Identify UI controls

[0030] In the present invention, we use three recognition methods to optimize control recognition. First, OCR technology is used to recognize text information. Second, traditional computer vision technology represented by Canny edge detection is used to recognize graphic controls. Third, Yolov7 target detection model retrained on Rico dataset is used to recognize UI controls. The three methods are mutually screened and supplemented in results, and embedded into vectors through pre-trained VGG-16 model to connect to reinforcement learning model. Among them, OCR refers to the process of electronic devices checking printed characters, determining their shapes by detecting dark and light patterns, and then translating the shapes into computer text using character recognition methods; Canny edge detection algorithm first applies Gaussian filtering to smooth the image to remove noise, then finds the intensity gradient of the image, then applies non-maximum suppression technology to eliminate edge false detection, then applies double threshold method to determine possible boundaries, and finally uses hysteresis technology to track boundaries; Yolov7 target detection model is an advanced open source target detection model based on Yolo algorithm and constructed by neural network, and Rico dataset is an open source dataset specifically for mobile applications containing a large number of UI control graphics, text and structural information.

[0031] 5. Identify page layout

[0032] In the present invention, we divide the page sections into groups according to the outer edges, further refine the layers within the groups, organize the UI controls into a tree structure and serialize them, and embed them into vectors through a pre-trained LSTM model for docking with the reinforcement learning model.

[0033] 6. Reinforcement Learning Decision Making

[0034] The entire automated testing process is abstracted into a Markov process, and the embedded vectors of controls and layout information are defined as application states. The Q-Network neural network model is used to autonomously conduct test exploration, continuously learn and predict the optimal transfer of the current state in the state space, and output a set of instructions, including the operation method and target control coordinates for the next operation.

[0035] 7. Perform test operations

[0036] In the present invention, during the process of calling the corresponding server-side remote method, we use the inverse kinematics algorithm to convert the moving distance of the click position from the operation space to the joint space, and after obtaining the solution required for each servo to move, we call the pre-packaged underlying API to control the robotic arm to perform the test operation.

[0037] The present invention is a method for automated testing of mobile applications based on multi-axis robotic arm physical simulation and reinforcement learning, which mainly includes:

[0038] Photo processing module: ensures clear photos through blur detection, and identifies and captures the screen part of the photo through technical means such as grayscale conversion, image enhancement and edge detection.

[0039] Image understanding module: Uses OCR technology, traditional computer vision technology, and deep learning technology to deeply understand screen images, extract controls and layout information, embed them through a pre-trained neural network model, and obtain the corresponding vector representation.

[0040] Reinforcement learning module: Combine controls and page layout to build mobile application states, and use Q-Network to train and make decisions in real time based on state transitions.

[0041] Robotic arm control module: The server remotely calls the interface provided by the robotic arm through the RPC framework to control the robotic arm; the robotic arm obtains the rotation parameters of each axis through inverse kinematics calculation, uses the underlying API encapsulated by the system to implement the interface, and controls the robotic arm to perform test operations.

Claims

1. A method for automated testing of mobile applications based on multi-axis robotic arm physical simulation and reinforcement learning, characterized in that The test process is executed in a loop for automated test exploration. In each loop, the robot arm camera is controlled to take clear screen photos and send them to the server. The server deeply analyzes and understands the screen content and uses the deep reinforcement learning model to generate instructions for test exploration. The robot arm receives the instructions and performs test operations. The steps of this method are as follows: 1) The robot arm and the device where the application to be tested is located move to the initial position, and the robot arm remote call service is enabled 2) Start the server, which remotely calls the control robot to click the default initial position to activate the test process 3) The robot arm camera takes a clear screen photo and returns it to the server 4) Screen recognition: the server recognizes and captures the effective screen part from the captured photo 5) Control and layout identification: the server deeply analyzes the control and page layout information from the screenshot 6) The server-side reinforcement learning model receives the current page information input and makes decisions, while automatically completing real-time training 7) The server generates operation instructions based on the decision of the deep reinforcement learning model and uses the remote call service to control the robot arm to perform the operation 8) The robotic arm receives the command, parses and calculates the required rotation angle of each servo, and simulates the touch screen test operation 9) Jump to step c and execute the test process in a loop until the test stop condition is met.

2. The method for automated testing of mobile applications based on multi-axis robotic arm physical simulation and reinforcement learning according to claim 1, characterized in that: The clarity control of camera photos continuously captures the frames taken by the camera, converts their grayscale and performs convolution operation with the Laplace kernel, performs blur detection according to the variance size, and retains the clearest frame until its clarity reaches the threshold or the maximum shooting time is set, so as to compensate for the influence of information variation and loss caused by the blur of the shooting results on subsequent processing to the greatest extent.

3. The method for automated testing of mobile applications based on multi-axis robotic arm physical simulation and reinforcement learning according to claim 1, characterized in that: Screen recognition uses grayscale conversion, Gaussian blur noise reduction, and Boolean indexing to remove shadows for image enhancement. It uses Canny edge detection, probabilistic Hough line transform, and contour dilation to obtain closed contours. It uses the minimum rectangle approximation to identify the screen contour and perform perspective transformation and deflection correction, thereby capturing the effective screen portion from the captured photo containing the screen and achieving non-invasive screenshot acquisition.

4. The method for automated testing of mobile applications based on multi-axis robotic arm physical simulation and reinforcement learning according to claim 1, characterized in that: Control and layout recognition uses a combination of three recognition methods for control recognition and adds analysis and understanding of page layout information. The three control recognition methods are OCR technology used to recognize text information, traditional computer vision technology represented by Canny edge detection used to recognize graphic controls, and deep learning target detection models used to comprehensively recognize UI controls. The three methods filter and complement each other in terms of results, thereby improving the recognition accuracy.

5. The method for automated testing of mobile applications based on multi-axis robotic arm physical simulation and reinforcement learning according to claim 1, characterized in that: Deep reinforcement learning introduces deep reinforcement learning technology into the exploration of robotic arm automated testing, so that the reinforcement learning model plays the role of the robotic arm scheduler, guiding the robotic arm to conduct automated testing exploration, and realizing intelligent robotic arm simulation non-invasive mobile application automated testing.

6. The method for automated testing of mobile applications based on multi-axis robotic arm physical simulation and reinforcement learning according to claim 1, characterized in that: The operation mode of the robotic arm is that the robotic arm and the server interact and transmit information through the RPC protocol with the help of the RPC framework, making the interaction between the two stateless, with higher performance and efficiency.

Citation Information

Patent Citations

  • Automatic test device and test method for mobile terminal application program

    CN104679017A

  • Black box automatic test system and method combined with deep learning technology

    CN111737141A

  • Cross-platform script recording and playback method based on image recognition

    CN112579431A

  • Three-dimensional space touch screen equipment visualization testing method based on multi-axis mechanical arm

    CN113504063A

  • Mobile application cross-platform reinforcement learning traversal test technology based on depth image understanding

    CN114138653A