Automatic testing method of mobile terminal and corresponding mobile terminal
By recording and analyzing test videos from the mobile user interface, generating and automatically executing test cases, the problem of inefficiency of regression testing is solved and efficient automated testing is achieved.
Patent Information
- Application Number
- CN202411681273.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-03-28
AI Technical Summary
In the prior art, regression testing requires repeated execution of the same test case, resulting in excessive time consumption of testers and inefficient efficiency.
By recording test videos in the mobile user interface, extracting video frames, identifying touch points using the target detection model, calculating the duration and type of touch operations, generating test cases, and automatically performing them.
It realizes rapid automatic generation and execution of test cases, significantly improving the efficiency of regression tests, and greatly improving the work efficiency of testers.
Smart Images
Figure CN119718911A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to an automated testing method for a mobile terminal (eg, a mobile phone) user interface. Background Art
[0002] Automated testing refers to the use of software or scripting tools to automatically execute test cases after generating test cases, and determine whether the test cases are executed successfully.
[0003] One type of software testing is regression testing. Regression testing is to re-execute previous test cases after modifying the old code to confirm that the modification has not introduced new errors or caused errors in other codes. Because regression testing requires repeatedly executing the same test case, it consumes a lot of testers' time. Summary of the invention
[0004] The technical problem to be solved by this application is how to automatically generate and execute test cases to improve the efficiency of software testing.
[0005] In order to solve the above technical problems, the present application proposes an automated testing method for a mobile terminal, comprising the following steps. Step S1: Set the operating system of the mobile terminal and enable the visualization of touch operations of the mobile terminal user interface. Step S2: Record an operation video of a complete test work performed manually in the user interface of the mobile terminal, hereinafter referred to as the test video. Step S3: Extract all image frames in the test video. Step S4: Send all image frames of the test video to a trained target detection model, and the target detection model selects the image frames with visualized touch points and gives the coordinates of the visualized touch points in the image frames. Step S5: According to the frame rate of the test video and the position of the image frames with visualized touch points in all image frames, calculate the duration of each touch operation and the waiting time between every two adjacent touch operations, determine the type of each touch operation, and record the operation trajectory and expected results of each touch operation. Step S6: Generate an operation step according to the type, operation track, duration, and expected result of each touch operation, and generate a test case according to the sequence of the touch operations in the test video and the waiting time between each two adjacent touch operations. Step S7: Read the test case and automatically execute the test case in the user interface of the mobile terminal.
[0006] Furthermore, in step S2, a test operation recorded in the test video is a combination of a series of touch operations performed in the user interface of the mobile terminal.
[0007] Furthermore, in step S3, the header information of the test video file is first parsed to obtain the frame rate of the test video; and then, according to the frame rate of the test video, image frames are extracted frame by frame in chronological order.
[0008] Preferably, in step S4, single target detection is performed on the visualized touch points in the image frame to identify the coordinates of the visualized touch points in the image frame.
[0009] Preferably, in step S4, the VOLOv5 deep learning framework is used as the target detection model. The training method of the VOLOv5 deep learning framework as the target detection model is as follows: first, prepare a large number of pictures with visual touch points, and mark the coordinates of the visual touch points to form a training data set; then, based on the VOLOv5 deep learning framework, set the pre-training model, training parameters, learning rate, data set path, and model output position parameters, and then use the training data set for training; after the accuracy converges and reaches the expectation, determine a set of parameters with the highest accuracy as the parameters of the trained target detection model, and set the prediction rate threshold according to the training accuracy. The method of using the VOLOv5 deep learning framework as a target detection model is as follows: send all image frames of the test video to the trained target detection model for prediction, filter out image frames according to the prediction rate threshold, determine the filtered image frames as having visual touch points, and record the coordinates of the visual touch points in the filtered image frames.
[0010] Further, in step S5, among all the image frames sorted by time, if there is a group of continuous image frames that all have visual touch points, at least one image frame before this group of continuous image frames does not have a visual touch point, and at least one image frame after this group of continuous image frames does not have a visual touch point, then it is determined that this group of continuous image frames represents a touch operation on the mobile user interface; the duration represented by each image frame is calculated according to the frame rate of the video, and the duration of the touch operation represented is calculated according to the number of frames contained in the group of continuous image frames representing a touch operation. Among all the image frames sorted by time, if there is a first group of continuous image frames representing a first touch operation, and a second group of continuous image frames representing an adjacent second touch operation, and a third group of continuous image frames between these two groups of image frames do not have visual touch points, then it is determined that the third group of continuous image frames represents the portion between two adjacent touch operations; the waiting time between the two adjacent touch operations represented is calculated according to the number of frames contained in a group of continuous image frames representing the portion between two adjacent touch operations.
[0011] Furthermore, in step S5, the touch operations in the mobile user interface are divided into the following types: click, long press, slide, and multi-finger operation.
[0012] Furthermore, in step S5, in a set of continuous image frames representing a touch operation, the change in the coordinates of the visualized touch points in each image frame represents the operation trajectory of the touch operation.
[0013] Furthermore, in step S5, for the image frame after a group of continuous image frames representing a touch operation, the text content in the image frame is recognized using OCR technology, and the text with the highest recognition rate is taken as the expected result of the previous touch operation. In step S7, during the execution of the test case, an image of the user interface of the mobile terminal is captured, and the text content in the image after each operation step is executed is recognized using OCR technology, and the text with the highest recognition rate is taken as the actual result of the previous operation step; if the actual results after all the operation steps are executed are equal to the corresponding expected results, the entire test case is executed successfully; if the actual results after any one or more operation steps are executed are not equal to the corresponding expected results, the entire test case fails.
[0014] The present application also proposes a mobile terminal for implementing automated testing, including a visualization setting unit, a video recording unit, an image frame extraction unit, an image frame detection and recognition unit, a touch operation calculation unit, a test case generation unit, and an automatic test unit. The visualization setting unit is used to set the operating system of the mobile terminal and enable the touch operation visualization of the mobile terminal user interface. The video recording unit is used to record an operation video of a complete test work performed manually in the user interface of the mobile terminal, hereinafter referred to as the test video. The image frame extraction unit is used to extract all image frames in the test video. The image frame detection and recognition unit is used to send all image frames of the test video to a trained target detection model, and the target detection model selects the image frames with visualized touch points and gives the coordinates of the visualized touch points in the image frames. The touch operation calculation unit is used to calculate the duration of each touch operation and the waiting time between each two adjacent touch operations according to the frame rate of the test video and the position of the image frame with the visualized touch point in all the image frames, determine the type of each touch operation, and record the operation track and expected results of each touch operation. The test case generation unit is used to generate an operation step according to the type, operation track, duration, and expected result of each touch operation, and to generate a test case according to the sequence of the touch operations in the test video and the waiting time between every two adjacent touch operations. The automatic test unit is used to read the test case and automatically execute the test case in the user interface of the mobile terminal.
[0015] The technical effects achieved by this application are as follows. First, by recording videos, and then using video frame parsing, target detection, text recognition and other means to automatically generate test cases for the mobile terminal. Second, by automatically executing the generated test cases, the effect of UI playback can be achieved. This application can quickly and automatically generate test cases and automatically execute them, which can improve the efficiency of regression testing and greatly improve the work efficiency of testers. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 It is a flowchart of the automated testing method for a mobile terminal proposed in this application.
[0017] Figure 2 This is a schematic diagram of touch points after the Android system enables touch operation visualization.
[0018] Figure 3 This is a schematic diagram of touch points after the iOS system enables touch operation visualization.
[0019] Figure 4 It is a schematic diagram of obtaining an expected result of a previous touch operation in an image frame following a group of continuous image frames representing a touch operation.
[0020] Figure 5 It is a structural diagram of the mobile terminal for realizing automated testing proposed in this application.
[0021] Explanation of the reference numerals in the figure: visualization setting unit 1, video recording unit 2, image frame extraction unit 3, image frame detection and recognition unit 4, touch operation calculation unit 5, test case generation unit 6, automatic testing unit 7. DETAILED DESCRIPTION
[0022] See also Figure 1 , the mobile terminal automated testing method proposed in this application includes the following steps.
[0023] Step S1: Setting the operating system of the mobile terminal to enable visualization of touch operations of the mobile terminal user interface, thereby providing visual prompts for touch operations of the mobile terminal user interface.
[0024] If the mobile terminal uses the Android operating system, this step is to turn on "Show touch operations" in the Android system settings. An exemplary turning on path in a certain version of the Android system is: Settings → System and Updates → Developer Options → Show touch operations. Figure 2 ,After the Android system enables touch operation visualization (display touch operations), the user interface displays the ,visualized touch points in a circle shape, as shown in the rounded rectangle.
[0025] If the mobile terminal uses the iOS operating system, this step is to turn on "Assistive Touch" in the iOS system settings. An exemplary turning-on path in a certain version of the iOS system is: Settings → Accessibility → Touch → Assistive Touch → Assistive Touch. Please refer to Figure 3 After the iOS system enables touch operation visualization (Assistive Touch), the user interface displays visual touch points in a circle, as shown in the rounded rectangle.
[0026] Step S2: Record a video of a complete test operation performed manually in the user interface of the mobile terminal, hereinafter referred to as a test video. The test operation recorded in the test video is a combination of a series of touch operations performed in the user interface of the mobile terminal.
[0027] Step S3: Extract all image frames (also called video frames) in the test video. Specifically, first parse the header information of the test video file to obtain the basic parameters of the test video, such as resolution, duration, frame rate, etc. Then, according to the frame rate of the test video, extract the image frames frame by frame in chronological order. For each image frame, operations such as picture format conversion, image enhancement, target detection, and image segmentation can be performed optionally. Since the touch operation visualization is enabled in step S1, the part of the image frame where the touch operation exists has a visualized touch point displayed, and the remaining image frames where there is no touch operation do not display the visualized touch point. Therefore, it is possible to judge whether there is a touch operation on the mobile user interface at the corresponding moment of the image frame based on whether there is a visualized touch point displayed in the image frame.
[0028] Step S4: All the image frames of the test video are sent to a trained object detection model, and the object detection model selects the image frames with the visualized touch points and gives the coordinates of the visualized touch points in the image frames. The presence of a visualized touch point in the image frame indicates that there is a touch operation on the mobile user interface at the corresponding moment, and the coordinates of the visualized touch point represent the coordinates of the touch operation on the mobile user interface.
[0029] Object detection refers to separating the object of interest (region of interest, ROI, also known as the region of interest) from the background in the image, identifying and extracting the object of interest from the image. Object detection technology mainly includes two stages: object localization and object classification. Object localization is used to determine the position of the object (here is the visual touch point) in the image. Common object localization methods include region proposal-based methods and anchor-based methods. The region proposal-based method first generates a series of candidate boxes that may contain the object, and uses a convolutional neural network (CNN) to score these candidate boxes, and selects the candidate box with a high score as the box containing the object. Object classification is used to identify the category of the object. Object classification can be performed using traditional machine learning methods (such as support vector machines and decision trees) or deep learning algorithms (such as convolutional neural networks).
[0030] Preferably, in this step, single object detection is performed on the visualized touch points in the image frame to identify the coordinates of the visualized touch points in the image frame.
[0031] The user interfaces of various APPs (applications) may appear in the user interface of the mobile terminal. If the traditional image feature recognition is used to identify and detect the visual touch points, there are several problems. First, there are a large number of various APPs and the user interfaces are complex, so a huge number of training samples need to be produced. Second, once the training samples are determined, they cannot cover the user interfaces of new APPs, and they need to be retrained for the user interfaces of new APPs. Third, the detection speed is slow and time-consuming, and it consumes a lot of computing and storage resources, and the implementation cost is high.
[0032] Single target detection refers to the use of computer vision technology to identify and locate only one specific target in an image. In this case, it only detects visual touch points and does not detect other targets. This can omit the step of identifying image features, which has the following advantages. First, there is no need to collect and organize the user interfaces of various apps. Second, it can cover the user interfaces of various apps, including new apps that have not appeared before. Third, the detection speed is fast, the time consumption is short, the consumption of computing and storage resources is small, and the implementation cost is low.
[0033] As an example, this step uses the VOLO (You Only Look Once) v5 deep learning framework as the object detection model. VOLOv5 is an end-to-end deep learning model that can detect and localize objects from images. It uses convolutional neural networks to learn the features of objects in images and uses multi-scale prediction and grid segmentation to detect and localize objects.
[0034] The training method of the VOLOv5 deep learning framework as a target detection model is as follows: first, prepare a large number of pictures with visualized touch points, and mark the coordinates of the visualized touch points to form a training data set. Then, based on the VOLOv5 deep learning framework, set the pre-trained model, training parameters, learning rate, data set path, model output position and other parameter information, and use the training data set for training. After the accuracy converges and reaches the expectation, determine the set of parameters with the highest accuracy as the parameters of the trained target detection model, and set the prediction rate threshold according to the training accuracy.
[0035] The method of using the VOLOv5 deep learning framework as a target detection model is as follows: all image frames of the test video are sent to the trained target detection model for prediction, image frames are filtered out according to the prediction rate threshold, the filtered image frames are judged as having visualized touch points, and the coordinates of the visualized touch points in the filtered image frames are recorded.
[0036] Step S5: According to the frame rate of the test video and the position of the image frame with the visualized touch point in all image frames, calculate the duration of each touch operation and the waiting time between every two adjacent touch operations, determine the type of each touch operation, and record the operation trajectory and expected results of each touch operation.
[0037] Among all the image frames sorted by time, if there is a group of continuous image frames that all have visual touch points, at least one image frame before this group of continuous image frames does not have a visual touch point, and at least one image frame after this group of continuous image frames does not have a visual touch point, then it is determined that this group of continuous image frames represents a touch operation on the mobile user interface. The frame rate of a video indicates the number of frames displayed per second. The duration represented by each image frame is calculated based on the frame rate of the video. For example, if the frame rate is 30FPS (frame per second), the duration represented by each image frame is one-thirtieth of a second. The duration of the touch operation represented is calculated based on the number of frames contained in a group of continuous image frames representing a touch operation.
[0038] Among all the image frames sorted in time, if there is a first group of continuous image frames representing a first touch operation, and a second group of continuous image frames representing an adjacent second touch operation, and all image frames between these two groups of image frames (for example, called a third group of continuous image frames) do not have a visible touch point, then it is determined that the third group of continuous image frames represents the portion between two adjacent touch operations. The waiting time between the two adjacent touch operations represented is calculated based on the number of frames included in a group of continuous image frames representing the portion between the two adjacent touch operations.
[0039] Touch operations in mobile user interfaces are divided into the following types: click, long press, slide, and multi-finger operation.
[0040] In a set of continuous image frames representing a touch operation, if the coordinates of the visualized touch point in the first image frame and the coordinates of the visualized touch point in the last image frame are within a very small range, for example, less than or equal to 50 pixels, then the touch operation is determined to be a click or a long press. The duration of the touch operation is then used to distinguish whether it is a click or a long press, for example, less than or equal to 500ms is a click, and greater than 500ms is a long press.
[0041] In a set of continuous image frames representing a touch operation, if there is a certain difference in the coordinates of the visualized touch points in the continuous multiple image frames, for example, greater than 50 pixels, then the touch operation is determined to be a slide. As long as it is a slide operation, there must be a coordinate difference in the visualized touch points in the image frame. The reason why multiple continuous image frames are required for judgment is that the sliding process may be slow and the coordinate difference of the visualized touch points in two consecutive image frames is not obvious. Judging by multiple continuous image frames can avoid this situation and make a more accurate judgment. For example, the duration of a certain slide operation is 2 seconds, and the frame rate of the video is 30FPS. There are 60 continuous image frames in 2 seconds. One image frame is selected from every 5 image frames to determine whether it is a slide operation based on whether there is a certain difference in the coordinates of the visualized touch points in the selected 12 image frames.
[0042] In a group of continuous image frames representing a touch operation, if the number of visible touch points in most (eg, more than 90%) of the image frames is greater than one, the touch operation is determined to be a multi-finger operation.
[0043] In a group of continuous image frames representing a touch operation, changes in coordinates of visualized touch points in each image frame represent the operation trajectory of the touch operation.
[0044] For the image frame following a group of continuous image frames representing a touch operation, the OCR (Optical Character Recognition) technology is used to recognize the text content in the image frame, and the text with the highest recognition rate is taken out as the expected result of the previous touch operation, so as to determine whether the previous touch operation is successful.
[0045] See also Figure 4, the recognized texts in the image frame are "ishing" and "with birds". Assuming that the recognition rate of the "ishing" text is 0.997 and the recognition rate of the "with birds" text is 0.867, the "ishing" text with the highest recognition rate is taken as the expected result of the previous touch operation.
[0046] From step S2 to step S5, the present application can completely capture all touch operations of a test work by detecting and identifying the coordinates of the visualized touch points in the image frame, accurate to the duration of each touch operation, the waiting time between every two adjacent touch operations, the type of each touch operation, the operation trajectory of each touch operation, and the expected result of each touch operation. Through these data, the present application can restore all the touch operation content in the test video one-to-one, that is, restore a complete test work. Compared with other detection and restoration methods, the present application can also accurately capture, parse and restore complex operating behaviors, such as multi-finger operations.
[0047] Step S6: Generate an operation step according to the type, operation track, duration, and expected result of each touch operation. There is a corresponding relationship between the operation step and the expected result, and one operation step corresponds to one expected result. Then, generate a test case according to the sequence of the touch operations in the test video and the waiting time between each two adjacent touch operations. For example, the test case is json data, which is convenient for subsequent automated execution.
[0048] Step S7: read the test case, and automatically execute the test case in the user interface of the mobile terminal to achieve the automated test of UI (user interface) playback type. As an example, this step is implemented using Appium, a framework for auxiliary automated testing. Appium supports automated execution of both Android and iOS systems, and can achieve the automated test effect of UI playback.
[0049] A test case consists of multiple operation steps and the same number of expected results. Each operation step corresponds to an expected result after execution. In the process of executing the test case, take an image of the user interface of the mobile terminal (or take a video of the user interface of the mobile terminal and then extract the image frame), use OCR technology to recognize the text content in the image after each operation step is executed, and take out the text with the highest recognition rate as the actual result of the previous operation step. If the actual results after all the operation steps are executed are equal to the corresponding expected results, the entire test case is executed successfully. If the actual results after any one or more operation steps are executed are not equal to the corresponding expected results, the entire test case fails.
[0050] See also Figure 5 The mobile terminal for realizing automated testing proposed in this application includes a visualization setting unit 1, a video recording unit 2, an image frame extraction unit 3, an image frame detection and recognition unit 4, a touch operation calculation unit 5, a test case generation unit 6, and an automatic testing unit 7. Figure 5 The device shown corresponds to Figure 1 The method shown.
[0051] The visualization setting unit 1 is used to set the operating system of the mobile terminal and enable the touch operation visualization of the mobile terminal user interface.
[0052] The video recording unit 2 is used to record an operation video of a complete test work performed manually in the user interface of the mobile terminal, hereinafter referred to as the test video.
[0053] The image frame extraction unit 3 is used to extract all image frames in the test video.
[0054] The image frame detection and recognition unit 4 is used to send all the image frames of the test video to a trained target detection model, and the target detection model selects the image frames with visualized touch points and gives the coordinates of the visualized touch points in the image frames.
[0055] The touch operation calculation unit 5 is used to calculate the duration of each touch operation and the waiting time between every two adjacent touch operations according to the frame rate of the test video and the position of the image frame with the visualized touch point in all image frames, determine the type of each touch operation, and record the operation trajectory and expected results of each touch operation.
[0056] The test case generation unit 6 is used to generate an operation step according to the type, operation trajectory, duration, and expected result of each touch operation; and then generate a test case according to the sequence of the touch operations in the test video and the waiting time between every two adjacent touch operations.
[0057] The automatic testing unit 7 is used to read the test cases and automatically execute the test cases in the user interface of the mobile terminal.
[0058] This application can automatically generate a test case by recording a test video of the mobile user interface, identifying the touch operation tracks in the test video, and identifying the coordinates of the touch operation in the mobile user interface. It can also perform multiple automated executions based on the generated test case to achieve the test effect of UI playback. This is particularly suitable for regression testing in software testing, which can greatly reduce the test time of testers and greatly improve test efficiency.
[0059] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.
Claims
1. A mobile terminal automated testing method, characterized in that: The method comprises the following steps: Step S1: Setting the operating system of the mobile terminal to enable touch operation visualization of the mobile terminal user interface; Step S2: Recording a video of a complete test operation performed manually in the user interface of the mobile terminal, hereinafter referred to as the test video; Step S3: extracting all image frames in the test video; Step S4: sending all the image frames of the test video to a trained target detection model, and the target detection model selects the image frames with the visualized touch points and gives the coordinates of the visualized touch points in the image frames; Step S5: according to the frame rate of the test video and the position of the image frame with the visualized touch point in all the image frames, calculate the duration of each touch operation and the waiting time between every two adjacent touch operations, determine the type of each touch operation, and record the operation track and expected result of each touch operation; Step S6: generating an operation step according to the type, operation track, duration, and expected result of each touch operation, and generating a test case according to the sequence of the touch operations in the test video and the waiting time between every two adjacent touch operations; Step S7: Read the test case, and automatically execute the test case in the user interface of the mobile terminal.
2. The automated testing method for a mobile terminal according to claim 1, characterized in that: In step S2, a test operation recorded in the test video is a combination of a series of touch operations performed in the user interface of the mobile terminal.
3. The automated testing method for a mobile terminal according to claim 1, characterized in that: In the step S3, the header information of the test video file is first parsed to obtain the frame rate of the test video; then, according to the frame rate of the test video, image frames are extracted frame by frame in chronological order.
4. The automated testing method for a mobile terminal according to claim 1, characterized in that: In the step S4, single target detection is performed on the visualized touch points in the image frame to identify the coordinates of the visualized touch points in the image frame.
5. The automated testing method for a mobile terminal according to claim 1, characterized in that: In step S4, the VOLOv5 deep learning framework is used as the target detection model; The training method of the VOLOv5 deep learning framework as a target detection model is as follows: first, a large number of pictures with visualized touch points are prepared, and the coordinates of the visualized touch points are annotated to form a training data set; then, based on the VOLOv5 deep learning framework, a pre-trained model, training parameters, learning rate, data set path, and model output position parameters are set, and then the training data set is used for training; after the accuracy converges and reaches expectations, a set of parameters with the highest accuracy is determined as the parameters of the trained target detection model, and a prediction rate threshold is set according to the training accuracy; The method of using the VOLOv5 deep learning framework as a target detection model is as follows: all image frames of the test video are sent to the trained target detection model for prediction, image frames are filtered out according to the prediction rate threshold, the filtered image frames are judged as having visualized touch points, and the coordinates of the visualized touch points in the filtered image frames are recorded.
6. The automated testing method for a mobile terminal according to claim 1, characterized in that: In the step S5, among all the image frames sorted in time, if there is a group of continuous image frames that all have visualized touch points, at least one image frame before the group of continuous image frames does not have a visualized touch point, and at least one image frame after the group of continuous image frames does not have a visualized touch point, then it is determined that the group of continuous image frames represents a touch operation on the mobile user interface; the duration represented by each image frame is calculated according to the frame rate of the video, and the duration of the touch operation represented is calculated according to the number of frames contained in the group of continuous image frames representing the touch operation; If, among all the image frames sorted in time, a first group of continuous image frames represents a first touch operation, and a second group of continuous image frames represents an adjacent second touch operation, and a third group of continuous image frames between the two groups of image frames do not have a visible touch point, then it is determined that the third group of continuous image frames represents a portion between two adjacent touch operations; The waiting time between the two adjacent touch operations is calculated according to the number of frames included in a group of continuous image frames representing the portion between the two adjacent touch operations.
7. The automated testing method for a mobile terminal according to claim 6, wherein: In step S5, the touch operations in the mobile user interface are divided into the following types: click, long press, slide, and multi-finger operation.
8. The automated testing method for a mobile terminal according to claim 6, characterized in that: In step S5, in a group of continuous image frames representing a touch operation, the change of the coordinates of the visualized touch points in each image frame represents the operation track of the touch operation.
9. The automated testing method for a mobile terminal according to claim 6, characterized in that: In step S5, for an image frame following a group of continuous image frames representing a touch operation, the text content in the image frame is recognized using OCR technology, and the text with the highest recognition rate is taken out as the expected result of the previous touch operation; In step S7, during the execution of the test case, an image of the user interface of the mobile terminal is captured, and the text content in the image after each operation step is executed is recognized using OCR technology, and the text with the highest recognition rate is taken out as the actual result of the previous operation step; if the actual results after all the operation steps are executed are equal to the corresponding expected results, the entire test case is executed successfully; if the actual results after any one or more operation steps are executed are not equal to the corresponding expected results, the entire test case fails.
10. A mobile terminal for implementing automated testing, characterized in that: It includes a visualization setting unit, a video recording unit, an image frame extraction unit, an image frame detection and recognition unit, a touch operation calculation unit, a test case generation unit, and an automatic testing unit; The visualization setting unit is used to set the operating system of the mobile terminal and enable the touch operation visualization of the mobile terminal user interface; The video recording unit is used to record an operation video of a complete test work performed manually in the user interface of the mobile terminal, hereinafter referred to as the test video; The image frame extraction unit is used to extract all image frames in the test video; The image frame detection and recognition unit is used to send all the image frames of the test video to a trained target detection model, and the target detection model selects the image frames with the visualized touch points and gives the coordinates of the visualized touch points in the image frames; The touch operation calculation unit is used to calculate the duration of each touch operation and the waiting time between every two adjacent touch operations according to the frame rate of the test video and the position of the image frame with the visualized touch point in all the image frames, determine the type of each touch operation, and record the operation track and expected result of each touch operation; The test case generation unit is used to generate an operation step according to the type, operation track, duration, and expected result of each touch operation, and to generate a test case according to the sequence of the touch operations in the test video and the waiting time between every two adjacent touch operations; The automatic testing unit is used to read the test case and automatically execute the test case in the user interface of the mobile terminal.