A cross-platform script recording and migration method based on image understanding

Abstract

This invention discloses a cross-platform script recording and transfer method based on image understanding, consisting of two stages: recording and playback. The recording stage comprises exporting screenshot scripts, while the playback stage involves training an object detection model, extracting components from the replay application, training a contrastive learning model, and matching script components based on image understanding. The specific steps are as follows: For a mobile application, during a click process, screenshots of its boundary recording components are obtained to generate a platform-independent script; a YOLOv3 object detection model capable of detecting all component boundaries from the app interface is trained based on the open-source Rico dataset; a MOCOV2 contrastive learning model for understanding image semantics is trained based on component images labeled in the Rico dataset; YOLOv3 is used to extract all component images from the target platform app's GUI, and MOCOV2 is used to match the component images recorded in the script to obtain the script transferred to the target platform.

Description

Technical Field

[0001] This invention belongs to the fields of software engineering and deep learning. Specifically, it relates to control information in mobile application interfaces, convolutional neural network technology, end-to-end object detection technology, and contrastive learning technology, and more specifically, a method for cross-platform migration of mobile application scripts based on object detection and contrastive learning. Background Technology

[0002] Most mobile applications adopt a client / server (C / S) architecture, with a unified server and users from different operating systems. Therefore, an app typically has versions adapted for different mobile operating systems, leading to fragmentation issues on the Android platform. Fragmentation refers to the phenomenon where the same app, despite having identical functionality, exhibits differences in the location and images of functional components on different systems such as Android, iOS, and mobile web platforms. This is a common phenomenon in cross-platform apps. For example, widely used applications like WeChat, Taobao, and novel readers usually have Android and iOS versions. Shopping apps like Taobao and novel readers often also have web versions.

[0003] For software testing engineers, new applications or different versions of applications require multiple thorough tests. Given rapid version iterations and the complex and diverse states of app scenarios, these multiple thorough tests can lead to a significant workload. Test recording and replay technology is a crucial part of app development testing, used for stability testing and bug reproduction, and for highly automating the testing process. However, fragmentation issues require reproducing the same app execution process on different platforms, which involves platform-independent recording and replay and script migration techniques.

[0004] For this platform-independent script migration requirement, using computer vision (CV) technology is a natural and common approach. Prior to this work, there had been work on cross-platform migration using traditional CV methods. However, traditional CV methods often cannot solve the fragmentation problem because they can only identify identical components or components with size-invariant features, and cannot identify and locate semantically identical components that differ in the actual image. Deep learning technology, on the other hand, can solve such semantic understanding problems of images.

[0005] YOLOv3 is an end-to-end object detection model that integrates object region prediction and object category prediction into a single neural network model, achieving fast object detection and recognition with high accuracy. Mocov2 is a contrastive learning model within unsupervised learning. This model can learn image features from large amounts of unlabeled data, and the resulting model can outperform supervised learning in downstream classification tasks. This method uses YOLOv3 for component extraction and a model trained with Mocov2 for semantic matching of component images, thus completing the replay and script transfer stages. It is a novel transfer and replay method. Summary of the Invention

[0006] The purpose of this invention is to achieve cross-platform script recording and replay using deep learning and image understanding methods. It proposes a method based on object detection and contrastive learning to understand the semantics of component images. The scripts recording the app's runtime process are recorded using an Appium custom format. The existing open-source UI dataset Rico is used to train an object detection model YOLOv3 and a contrastive learning model Mocov2, respectively. These models are used for component extraction and semantic matching of target components on the target app during replay, achieving the goal of cross-platform script replay.

[0007] To achieve the aforementioned objectives, the present invention employs the following technical solution: a cross-platform script recording and migration method based on image understanding, characterized in that the method comprises the following steps:

[0008] 1) Design the recording script format to record the click event flow and image information in the source app. During recording, record the coordinates of the clicked component, the component screenshot, ID, etc., and generate a platform-independent script for subsequent replay.

[0009] 2) Based on the UI screenshots and view structures provided by the open-source Rico dataset, the sematic annotations of the dataset are converted into a format usable by the YOLOv3 model. The YOLOv3 model is then trained using this dataset. This YOLOv3 model will be used for extracting candidate UI components during subsequent replays.

[0010] 3) Using the boundary coordinates of UI components labeled in the Rico dataset, a large number of component images were extracted. These images were then compared with a pre-trained Mocov2 model to learn the features of the UI components. This model was used for component matching and selection during subsequent replays.

[0011] 4) Replay stage: Before each click according to the source app component images (referred to as query) recorded in the script, for the UI interface of the target app: Obtain the current UI interface screenshot, use the YOLOv3 model in 2) to locate all components in this UI interface and extract the images of each component. The set of these component images is called keySet. Use the Mocov2 model obtained in 3) to query the image in keySet that is semantically closest to query, and perform coordinate positioning clicks.

[0012] 5) Repeat step 4), and record the new script in the manner defined in 1).

[0013] The above cross-platform script recording and migration method based on image understanding is further characterized in that:

[0014] The script format defined in step 1) and the script recording the app click event stream are the inputs in the migration and replay stage. The script recording the app event stream includes the following sub-steps:

[0015] A. Define the required script format. This method is a device-independent script recording and replay method, so the script needs to record platform-independent information such as the images of the clicked components. According to the information required for replay and migration, an operation O is defined here as <operation ID, screenshot of the current UI, bounds of the operation component, screenshot of the operation component, time offset relative to the start of this script recording>. The entire script is composed of a sequence of Os corresponding to each operation, and the script ID is composed of the DSN and the timestamp of the script recording to form a unique script ID. For example, an O can be <0, UI screenshot, [1018, 1857, 1250, 2025], component screenshot, 168s>, and EMULATOR30X8X4X0-2022-07-25-23:46:59.811822 is the ID value of the recorded script.

[0016] B. Write a Python program to obtain the information and files required during the click process to generate a platform-independent script. Install the source app on an Android emulator or a real device. Use the open-source test framework uiautomator2 to obtain the clickable components on the interface. Randomly select a clickable component, record its bounds, and then use the cv2 library to capture the screenshot of the component at the specified coordinates. Obtain other required information in the format of O defined in A. After recording O, use the device.click() method provided by uiautomator2 to click the component to make the app enter the next state. Repeat this step until the set number of steps to collect the recorded sequence of Os until the required complete script is obtained, and name it in the ID format defined in A. This script will be used for subsequent replay.

[0017] The aforementioned image understanding-based cross-platform script recording and migration method is further characterized by:

[0018] Step 2) Using the YOLOv3 model to extract components is not unique to this method, but the dataset used is different from existing methods.

[0019] Step 2) Train the YOLOv3 model based on the UI screenshots and view structures provided by the open-source Rico dataset. This includes the following sub-steps:

[0020] A. The open-source Rico dataset provides several types of files: screenshots of the entire UI, XML files corresponding to the UI screenshots, and semantic annotation files (JSON format) for the UI. The semantic annotation files provide the bounds and labels of each component in the UI. By extracting the bounds and component labels of each independent component without child nodes, the information extracted from all annotation files is combined into a JSON file format that meets the requirements of the YOLOv3 model for YOLOv3 training.

[0021] B. Train the YOLOv3 model based on the processed dataset. YOLOv3, proposed by Joseph Redmond in 2015, has several subsequent optimized versions. While other advanced object detection models of the same period required multiple network passes for candidate regions of an image, YOLOv3 only needs to be viewed once in the network, i.e., only one forward pass is required to make the final prediction. The model has 24 convolutional layers, 4 max pooling layers, and 2 fully connected layers. The YOLOv3 model is trained using the data processed in step A, where the coordinates and labels are the bounds coordinates and component_label, respectively. The YOLOv3 model uses the binary cross-entropy loss function, the SGD optimizer, a learning rate of 0.004, a momentum of 0.9, and a training epoch of 100. After forward propagation, backpropagation, and gradient descent algorithm to update the parameters, the object detection model used for component extraction is obtained.

[0022] The aforementioned image understanding-based cross-platform script recording and migration method is further characterized by:

[0023] Step 3) Extract a large number of component images from the Rico dataset and use a pre-trained contrastive learning Mocov2 model to learn the features of the UI components. This includes the following sub-steps:

[0024] A. Extract component images from the Rico dataset. Similar to step 2)A, based on the component bounds annotation in the semantic_annotation, use dict.has_key('children') to determine if a component has no child nodes in the XML structure. If it is a leaf node, i.e., a single existing component, extract it as a component image. After extracting images of all individual components in the UI, this will be used as the dataset for subsequent Mocov2 use. The image dataset here is unlabeled data.

[0025] B. The ResNet-50 model was trained using the Mocov2 method. The Mocov2 method is one of the most state-of-the-art unsupervised contrastive learning methods, bridging the gap between unsupervised and supervised learning. It improves the accuracy of unsupervised learning in classification tasks to a level higher than the supervised learning baseline, and several subsequent optimized versions have been developed. In this work, the task requires matching two semantically identical component images that are disparate. However, datasets with pairwise matching require extensive manual acquisition and labeling, and semantically similar components are often paired or only a few in number, making it difficult to find enough supervised learning models. Therefore, a more suitable unsupervised learning method was used for this task.

[0026] The Mocov2 method defines two encoders: a query encoder (Q) and a momentum encoder (M). The Q encoder encodes the original image. M is typically initialized to be the same as Q. During model training, M is updated with a large momentum at an extremely slow pace. The purpose of this is to ensure that M encodes all other samples to maintain a large and consistent dictionary. In each epoch, only one mini-batch of data from this dictionary is encoded and updated with the new M. This work uses the ResNet50 encoder architecture, and this encoder serves as the final learning target.

[0027] In Mocov2, the original image is first cropped using `transforms.RandomResizedCrop`, then grayscaled using `transforms.RandomGrayscale`, and finally flipped using Gaussian blur and `transforms.RandomHorizontalFlip`. This resulting image is used as a positive sample from the original image, while the remaining images in the dataset are used as negative samples. The model is trained using these positive and negative samples, with SGD as the optimizer, a learning rate of 3e-2 cosine constants, 200 training epochs, and a momentum of 0.999. After forward propagation and backward gradient descent, a converged model is obtained. The resulting ResNet network serves as the encoder for subsequent component images.

[0028] The aforementioned image understanding-based cross-platform script recording and migration method is further characterized by:

[0029] Step 4) Using the models obtained in Steps 2) and 3), for an action O in the recorded script, replay and migrate, the specific sub-steps are as follows:

[0030] A. Obtain the corresponding UI interface file of the target program. Write a Python program to connect to the virtual machine or physical device using uiautomator2.connect(), and open the target app based on the app's package name. For the target interface that has reached a certain state U, assuming the current replay operation is Ou, first obtain a screenshot of the target app using uiautomator2, download it to the current folder, and then perform subsequent component matching and location work.

[0031] B. Extract each component and its bounds from the extracted target screenshot. This requires using the YOLOv3 model trained in step 2). After converting the screenshot to the size required by the YOLOv3 model, the model is used to predict boxes. Since the YOLO model predicts a large number of boxes, errors such as duplicates and non-exclusive components may occur, reducing prediction accuracy. Therefore, we define a confidence threshold. In this work, since recall is more important than precision, a confidence threshold of 0.2 was chosen after multiple experiments. Therefore, all components in the screenshot predicted by YOLOv3 with a confidence level higher than 0.1 are extracted and used as the candidate set keySet for matching.

[0032] C. Using the component screenshot Sq recorded in Ou, the encoder model learned in 3) is used to match semantically similar components in the keySet. The ResNet50 network learned in 3) can encode the features of component images for semantic understanding. In this task, it is not necessary to predict the category of the component, but only to determine whether the icons of two components are similar in image semantics. Therefore, ResNet50 is only used as a feature encoder for component images. First, Sq is encoded into Vq through ResNet50. All component images in the keySet are also encoded into features through ResNet50. Then, Vq is compared with all feature vectors in the keySet using Euclidean distance. The feature vector Vt with the closest distance is selected. Here, the component Wt corresponding to Vt is considered to be the small component that O needs to operate on in the current interface of the target app. Its absolute coordinates C in the UI interface are obtained.

[0033] D. Based on the coordinates C of the target component, the target app is moved to the next state, and the corresponding script is recorded. After obtaining the bounds of the target component, the migration action can be completed by clicking the center point of the bounds. Here, uiautomator2 is still used to implement the click event, and the operation Ot defined in 1) is recorded at the same time. This completes one migration of O from the source platform app to the target app.

[0034] The aforementioned image understanding-based cross-platform script recording and migration method is further characterized by:

[0035] Step 5) Repeat steps 4) until all operations of the recorded script have been migrated and a new script has been formed. The specific steps are as follows:

[0036] During each execution of step 4), a new Ot is generated. These Ots are assigned ids in sequence. After the entire power-up migration is completed, the ID of the migrated script is the same as that defined in 1), which is a combination of the device serial number and the timestamp.

[0037] Beneficial effects: During the development of mobile applications, the recording, generation, and migration of scripts have always been important components of testing. However, there has been no good solution to the fragmentation problem existing in mobile applications. This method uses image understanding and an automated testing framework. Based on the open-source Rico dataset, it performs object detection on components in the UI interface and uses the contrastive learning method in the field of unsupervised learning to understand the semantic features of components, enabling the cross-platform migration of platform-independent scripts recorded. This method can save the cost of writing cross-platform scripts for the same app, improve the efficiency of software developers and testers, and also help with the semantic understanding of component images. Brief Description of the Drawings

[0038] Figure 1 It is a structural flowchart of the cross-platform script recording and migration method based on image understanding of the present invention.

[0039] Figure 2 It is a UI screenshot of the source app (left) and a screenshot of a small component in the operation (right).

[0040] Specific Embodiments

[0041] The following combines the drawings and embodiments to further elaborate on the technical solutions of the present invention.

[0042] The process of the cross-platform script recording and migration method based on image understanding in this embodiment is as Figure 1 follows, and the specific implementation method is:

[0043] 1) Input a mobile application that needs script recording and cross-platform operation. The user needs to generate a platform-independent script. The specific steps are as follows:

[0044] 1.1) Define the required script format. The script format needs to be a platform-independent script. In this work, to ensure the platform independence of the script and the uniqueness of the ID, the operation format for each step is defined as <operation ID, screenshot of the current UI, bounds of the operation component, screenshot of the operation component, time offset relative to the start of this script recording>. Specifically, for example, for the Figure 2 (left) interface, after clicking on the dialog box in it, the recorded O for this time is <0, Figure 2 (left), [1018, 1857, 1250, 2025], Figure 2 (right), 168s>.

[0045] 1.2) Write a Python program to obtain the information needed during the click process and generate a platform-independent script. This method focuses on the migration and replay stage; the user only needs to provide a script in the format defined in 1.1). This paper uses Monkey's algorithm to generate the required script. Install the source app on an Android emulator or real device, use the open-source testing framework uiautomator2 to obtain clickable components on the interface, randomly select a clickable component, and record its bounds. Repeat the process to obtain the script S.

[0046] 2) Train the YOLOv3 model based on the UI screenshots and view structures provided by the open-source Rico dataset. The specific steps are as follows:

[0047] 1.1) Rico provides semantic annotations, while YOLOv3 requires data including the coordinates of the component's bounds and the component's label. Since components themselves don't have labels, the component's control category is used here, such as "button," "text," "Icon," "Text Button," "Input," etc., as the category for each component during training. Furthermore, components without children are filtered out as independent nodes; for example, the toolbar always contains other nodes and cannot be used as detection targets.

[0048] 1.2) Train the YOLOv3 model based on the processed dataset. The YOLOv3 training parameters set in this work are as follows: SGD optimizer is selected, learning rate is set to 0.004, momentum is 0.9, training epoch is 100, weight_decay is 1e-4, and box_threshold = 0.2.

[0049] 3) Extract component images from the Rico dataset and use a pre-trained contrastive learning Mocov2 model to learn the features of the UI components. This includes the following sub-steps:

[0050] 3.1) Extract component images from the Rico dataset. Similarly, use the annotations in `sematic_annotation` to extract components. After reading the images using the `cv2.imread()` function, select the specified range of `img` and save it. For example, `cropped = img[1018:1857, 1250:2025]`. Use all component images as the training dataset.

[0051] 3.2) The ResNet-50 model was trained using the Mocov2 method. The ResNet-50 model used here was pre-trained on the ImageNet dataset. Since the component images themselves have many classes and multiple semantics, training on other datasets is helpful for understanding the image semantics of this task. The training parameters set in this work are as follows: SGD as the optimizer, learning rate as an initial cosine-type learning rate of 3e-2, batch size of 32, training epochs of 200, and momentum of 0.999.

[0052] 4) Using the models obtained in steps 2) and 3), replay and migrate an action O model in the recorded script.

[0053] 4.1) Obtain the corresponding UI interface file of the target program. Write a Python program to connect to the virtual machine or physical device via uiautomator2.connect(machine DSN), and open the target app according to the app's package name, for example, device.app_start('com.android.browser'). Obtain a screenshot of the target app through uiautomator2, download it to the current folder, and then perform subsequent component matching and location work.

[0054] 4.2) Obtain boundaries using the YOLOv3 model. Predict all component boundaries using `boxes = yolo.detect_image(image)`, where the model's `boxthreshold` is set to 0.2 as defined previously. Similarly, use the clipping function provided by cv2 to clip the coordinates corresponding to each box in `boxes`, resulting in a set of candidate components.

[0055] 4.3) Encode the matching components using the ResNet50 trained with the Moco method. Resize the query component images in the script and all component sets from 4.2) to a size of 224*224 for ResNet50 feature encoding. Use Euclidean distance to measure the semantic similarity between the script and the candidate components. Record the script on this target app using the format defined in 1.1), thus completing the transfer of one operation.

[0056] 4.4) Repeat the process in 4.3) until the script completes the migration.

[0057] The above are merely preferred embodiments of the present invention. For those skilled in the art, any modifications or equivalent substitutions made to the present invention without departing from the spirit and principles thereof should be included within the scope of protection of the claims of the present invention.

Claims

1. A cross-platform script recording and migration method based on image understanding, characterized in that, Includes the following steps: Step 1: Design the format of the recording script to record the click event flow and image information in the source app; during the recording process, record the coordinates of the clicked component, the screenshot of the component, the ID, etc., and generate a platform-independent script for subsequent replay. Step 2: Based on the UI screenshots and view structures provided by the open-source Rico dataset, convert the sematic_annotations of the dataset into a format that the YOLOv3 model can use, and train the YOLOv3 model with this format dataset; this YOLOv3 model will be used for the extraction of candidate UI components during subsequent replays. Step 3: Using the boundary coordinates of components in the UI interface labeled in the Rico dataset, a large number of component images are extracted. The images are then compared with the trained Mocov2 model to learn the features of the UI components. This model is used for component matching and selection during subsequent replays. Step three specifically includes the following steps: A. Extract component images from the Rico dataset: Based on the component bounds annotation in semantic_annotation, use dict.has_key to check if it contains the children attribute to determine if a component has no child nodes in the XML structure. If it is a leaf node, i.e. a single existing component, then extract it as a component image; after extracting the images of all individual components in the UI, use them as the dataset for subsequent Mocov2 use. The image dataset here is unlabeled data without annotations. B. Training the ResNet-50 model using the Mocov2 method: The Mocov2 method defines two encoders, a query encoder and a momentum encoder, abbreviated as Q and M respectively. The Q encoder is used to encode the original image. M is generally initialized to be the same as Q. During the model training process, M will be set to a large momentum and extremely slow update, so that M encodes all other samples to maintain a large and consistent dictionary. In each epoch, only one mini_batch of data in the dictionary is encoded and updated with the new M. Step 4, Replay Phase: Before each click is performed based on the source app component images recorded in the script—that is, the image query—the target app's UI is: captured as a screenshot of the current UI, and the YOLOv3 model from Step 2 is used to locate all components in this UI and extract the image of each component. This set of component images is called keySet. The Mocov2 model obtained in Step 3 is used to query the keySet for the image that is semantically closest to the query, and the coordinates are used for the click. Step four specifically includes the following steps: A. Obtain the corresponding UI interface file of the target program: Write a Python program to connect to the virtual machine or physical machine through uiautomator2.connect(), and open the target app according to the app's package name; for the target interface that has reached a certain state U, let the current replay operation be Ou, then first obtain the screenshot of the target app through uiautomator2, download it to the current folder, and then do the subsequent component matching and positioning work. B. Extract each component and its bounds from the extracted target Screenshot: After converting the Screenshot to the size required by the YOLOv3 model, use the model to predict the boxes; extract all components in the Screenshot that are predicted by YOLOv3 with a confidence level higher than 0.1, and use them as the candidate set for matching, keySet; C. Using the component screenshot Sq recorded in Ou, the encoder model learned in step three is used to match semantically similar components in the keySet; the resnet50 network learned in step three is used to encode the component images for semantic understanding; Sq is encoded using resnet50 to obtain Vq, and all component images in the keySet are also encoded using resnet50. Then, Vq is compared with all feature vectors in the keySet using Euclidean distance, and the closest feature vector Vt is selected. The component Wt corresponding to Vt is the component that O needs to operate on in the current interface of the target app, and its absolute coordinates C in the UI interface are obtained. D. Based on the absolute coordinates C of the target component, the target app is operated to enter the next state, and the corresponding script is recorded: After obtaining the bounds of the target component, the action after migration is completed by clicking the center point of the bounds. The click event is implemented using uiautomator2, and the operation Ot defined in step one is recorded at the same time; thus, the migration of O from the source platform app to the target app is completed. Step 5: Repeat Step 4, recording the new script in the manner defined in Step 1, until the entire script has been migrated across platforms.

2. The cross-platform script recording and migration method based on image understanding according to claim 1, characterized in that: The script format defined in Step 1 and the script for recording the app click event stream are inputs during the migration replay phase. The script for recording the app click event stream includes the following sub-steps: A. Define the required script format: This method is a device-independent script recording and replay method. Therefore, platform-independent information such as the images of clicked components needs to be recorded in the script. According to the information required for replay and migration, an operation O is defined here as <operation ID, screenshot of the current UI, bounds of the operation component, screenshot of the operation component, time offset relative to the start of this script recording>. The entire script is composed of a sequence of Os corresponding to each operation, and the script ID is jointly composed of the DSN and the timestamp of the script recording to form a unique script ID; B. Write a Python program to obtain the information required during the click process and generate a platform-independent script: Install the source app on an Android emulator or real device. Use the open-source testing framework uiautomator2 to obtain the clickable components on the interface. Randomly select a clickable component, record its bounds, and then use the cv2 library to capture the screenshot of the component at the specified coordinates. Obtain other required information according to the format of O defined in A. After recording O, use the device.click() method provided by uiautomator2 to click the component to make the app enter the next state. Repeat this step until the set number of steps to collect the recorded sequence of Os until the complete script required is obtained, and name it in the ID format defined in A; this script will be used for subsequent replay.

3. The cross-platform script recording and migration method based on image understanding according to claim 1, characterized in that, Step 2 specifically includes the following steps: A. The open-source Rico dataset provides several types of files: screenshot of the entire UI, xml file corresponding to the UI screenshot, semantic annotation semantic_annotation file for the UI, and this semantic annotation file is in json format; the semantic annotation file gives the bounds and component labels of the small components in each UI interface. By extracting the bounds and component_label information of each independent component without child nodes, the information extracted from all annotation files forms a json file format that meets the requirements of the YOLOv3 model for training; B. Train the YOLOv3 model based on the processed dataset: Use the data processed in step A to train the YOLOv3 model, where the coordinates and labels use the coordinates of the bounds and component_label respectively; the loss function used by the YOLOv3 model is binary cross-entropy, the optimizer is selected as SGD, the learning rate is set to 0.004, momentum is 0.9, and the training epoch is 100. Through forward propagation, backpropagation, and gradient descent algorithms for parameter update, the object detection model for component extraction is obtained.

4. The cross-platform script recording and migration method based on image understanding according to claim 1, characterized in that: In Mocov2, the original image is first cropped using `transforms.RandomResizedCrop`, then grayscaled using `transforms.RandomGrayscale`, and finally flipped using Gaussian blur and `transforms.RandomHorizontalFlip`. This resulting image is used as a positive sample of the original image, while the rest of the images in the dataset are used as negative samples. The model is trained using these positive and negative samples, with SGD as the optimizer, a learning rate of 3e-2 cosine-type variation, 200 training epochs, and a momentum of 0.

999. After forward propagation and backward gradient descent, a converged model is obtained. The resulting ResNet network serves as the encoder for subsequent component images.

5. The cross-platform script recording and migration method based on image understanding according to claim 1, characterized in that, In step five: During each execution of step four, a new Ot is generated. These Ots are assigned IDs in sequence. After the entire power-up migration is completed, the ID of the migrated script is the same as that defined in step one, which is a combination of the device serial number and the timestamp.

Images