Page information collection method and electronic equipment

Abstract

The invention provides a page information collection method and electronic equipment. The method is applied to first equipment. Comprising the steps that a first program is preset in a first device, and when the first program is operated, the following operations are executed: a second device is controlled to display a page in sequence based on an automatic command, and page information of a current page is obtained after the second device is controlled to display a page every time. In addition, if the user starts the automatic label mode, the first equipment also adopts the model to obtain a label corresponding to the current page based on the page information; and if the user starts the manual label mode, the first device also receives a label corresponding to the current page input by the user. And finally, the first equipment associates and stores the page information and the label of the current page, further can obtain an identifier of an application to which the current page belongs, an identifier of an activity and the like, and associates and stores one or more of the identifiers with the page information and the label. Therefore, the labor cost is reduced, and the efficiency and accuracy of data collection and label labeling are improved.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to a method for collecting page information and an electronic device. Background Art

[0002] At present, more and more businesses rely on collecting large amounts of high-quality data. Take page information data as an example. By collecting a large amount of page information and labeling different categories of page information with corresponding labels, more services can be provided for pages with different labels in the future, such as optimizing the operating parameters corresponding to different pages, and placing targeted advertisements on target pages. However, current page information usually relies on manual operations, and how to collect large amounts of high-quality layout information is an urgent problem to be solved. Summary of the invention

[0003] The present application provides a method for collecting page information and an electronic device, and the method is applied to a first device. The method includes: the first device can use an automated command to control the second device to display a page in sequence, and obtain the page information of the current page after controlling the second device to display a page each time. In addition, if the user turns on the automatic labeling mode, the first device also uses a model to obtain the label corresponding to the current page based on the page information; if the user turns on the manual labeling mode, the first device also receives the label corresponding to the current page input by the user. Finally, the first device stores the page information and label of the current page in association with each other. Further, it can also obtain the identifier of the application to which the current page belongs, the identifier of the activity, etc., and store one or more of them in association with the page information and the label. Thereby reducing labor costs and improving the efficiency and accuracy of data collection and labeling.

[0004] In a first aspect, the present application provides a method for collecting page information, which is applied to a first device, and the method includes: the first device executes a first program, and performs the following operations based on the first program: the first device controls the second device to display each page in a first group of pages in sequence; wherein, in the process of controlling the second device to display the first page, the first device also performs the following operations: obtaining layout information of the first page, the first page being any page in the first group of pages; determining a corresponding first tag based on the layout information of the first page, the first tag indicating the scene of the first page.

[0005] After implementing the method provided in the first aspect, the use of automated commands can quickly and batch-wise collect page information, quickly label pages, and significantly improve work efficiency.

[0006] In combination with the method described in the first aspect, after controlling the second device to display each page in the first group of pages in sequence, the method also includes: obtaining a screenshot of the first page; and associating the storage path of the layout information of the first page and the storage path of the screenshot of the first page with the first tag for storage.

[0007] In this way, associating the storage tags with the layout information of the page and the storage path of the screenshots can help users further query and associate the collected data by categories.

[0008] In combination with the method described in the first aspect, after obtaining the screenshot of the first page, the method also includes: associating and storing the storage path of the layout information of the first page and the storage path of the screenshot of the first page with one or more of the following: an identifier of the application to which the first page belongs and an identifier of the first page.

[0009] In this way, multiple dimensions can be used to classify and store the collected page information, which is convenient for subsequent query of the target page information.

[0010] In combination with the method described in the first aspect, before controlling the second device to display each page in the first group of pages in sequence, the method also includes: receiving a first operation; determining a corresponding first tag based on the layout information of the first page, specifically including: responding to the first operation, determining the corresponding first tag based on the layout information of the first page.

[0011] In this way, the first device can start the automatic labeling working mode based on the user operation.

[0012] In combination with the method described in the first aspect, the method also includes: the first device controls the second device to display each page in the second group of pages in sequence; wherein, in the process of controlling the second device to display the second page, the first device also performs the following operations: obtaining the layout information of the second page, the second page being any page in the second group of pages; receiving a second operation of inputting a second tag, determining the second tag as a tag corresponding to the second page, and the second tag indicating the scene of the second page.

[0013] In this way, on the basis of providing automatic collection of page information, in addition to providing an automatic labeling method, this application also provides a manual labeling method. This not only improves the accuracy and efficiency of classified collection of page information, but also can avoid the model from being unable to accurately output untrained labels when facing new page scenarios. Manual labeling can be used to classify and collect page information, and the manually processed page information and corresponding labels can also be used as training data for subsequent stage models to further update the model and ensure that the model learns the data in the new scenario in real time.

[0014] In combination with the method described in the first aspect, before controlling the second device to display each page in the second group of pages in sequence, the method also includes: receiving a third operation; receiving a second operation of inputting a second tag, and determining the second tag as the tag corresponding to the second page, specifically including: in response to the third operation, displaying information prompting the user to input a tag; receiving a second operation of inputting a second tag, and determining the second tag as the tag corresponding to the second page.

[0015] In this way, the first device can start the manual labeling working mode based on the user operation.

[0016] In combination with the method described in the first aspect, the first device controls the second device to sequentially display each page in the first group of pages, specifically including: the first device controls the second device to sequentially display each page in the first group of pages based on a command of a simulated operation.

[0017] Thus, in order to realize the rapid collection of page information, in addition to efficient automatic tagging, it is also necessary to automatically switch the page scene. The use of simulation operations provides a feasible way to quickly switch pages and realize the rapid collection of page information.

[0018] In combination with the method described in the first aspect, the first device controls the second device to display each page in the first group of pages in sequence, specifically including: the first device sends a plurality of simulated operation commands to the second device in sequence, so that the second device responds to each simulated operation command in sequence and displays each page in the first group of pages in sequence; or, the first device sends a command to start an automation application to the second device, so that the second device starts the automation application, and generates a plurality of simulated operation commands through the automation application in sequence, responds to each simulated operation command in sequence, and displays a plurality of pages of each application in the first group of applications in sequence.

[0019] In this way, the first device can automatically send a simulated operation command to the second device at regular intervals, or the first device only needs to control the second device to start a related automation application, and the automation application in the second device automatically generates a simulated operation command at regular intervals, etc., providing a variety of implementation methods.

[0020] In combination with the method described in the first aspect, before obtaining the layout information of the first page, the method further includes: the first device obtaining an identifier of an application to which the first page belongs from a dump file; and the first device obtaining an activity identifier of the first page from the dump file.

[0021] In this way, relevant information about the focus page being displayed in the second device can be obtained from the dump file.

[0022] In combination with the method described in the first aspect, the first device obtains the identifier of the application to which the first page belongs from the dump file, specifically including: the first device obtains the identifier of the application to which the first page matches the first regular expression from the dump file; the first device obtains the activity identifier of the first page from the dump file, specifically including: the first device obtains the activity identifier of the first page that matches the second regular expression from the dump file.

[0023] In this way, when the dump file stores complex information, regular expressions can be used to accurately obtain the target information and improve the quality of the information.

[0024] In combination with the method described in the first aspect, determining the corresponding first label based on the layout information of the first page specifically includes: inputting the characteristics of the layout information of the first page into a label prediction model to obtain the first label corresponding to the first page, wherein the label prediction model is trained by multiple groups of data, and one group of the data includes: the characteristics of the layout information of the page, and the label corresponding to the page.

[0025] In this way, the label prediction model is used to automatically label pages, which reduces dependence on human resources and reduces the cost of manual labeling. It can easily cope with the growth of data volume and meet the needs of large-scale data processing.

[0026] In combination with the method described in the first aspect, before determining the corresponding first label based on the layout information of the first page, the method also includes: obtaining a screenshot of the first page; inputting the features of the layout information of the first page into a label prediction model, supervising the label prediction model through the screenshot of the first page, and obtaining a first label corresponding to the first page.

[0027] In this way, the supervised approach can improve the model prediction results.

[0028] In a second aspect, the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method as described in any one of the first aspects.

[0029] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any method described in the first aspect.

[0030] In a fourth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the method described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1A schematic diagram of a scenario for collecting page information by category provided in an embodiment of the present application; Figure 2 A flowchart of a method for manually classifying and collecting page information provided in an embodiment of the present application; Figure 3 A flowchart of a method for automatically combining manual classification and collecting page information provided in an embodiment of the present application; Figure 4 A schematic diagram of an interactive process for automatically classifying and collecting page information provided in an embodiment of the present application; Figure 5 A schematic diagram of an interactive process for manually classifying and collecting page information provided in an embodiment of the present application; Figure 6 A schematic diagram of the architecture of a decision tree model provided in an embodiment of the present application; Figure 7 A schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. Mentioning "embodiment" in the present application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application may be combined with other embodiments.

[0033] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form that the user can receive. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0034] refer to Figure 1 , Figure 1A schematic diagram of a scenario for collecting page information by category is exemplified.

[0035] like Figure 1 As shown, the scenario includes a first device and a second device, and the first device establishes a connection with the second device. Based on the connection, the first device can obtain page information data from the second device, and then classify and annotate each obtained page based on the obtained page information data, that is, annotate each page with a corresponding label. Subsequently, data support can be further provided for various businesses based on a large number of tagged pages, and the embodiment of the present application does not specifically limit the further business.

[0036] Among them, the first device is a larger terminal device with strong data storage and data analysis capabilities, such as a desktop computer, a laptop computer, a notebook computer, etc.

[0037] The second device is a smaller terminal device with a display screen, such as a mobile phone, a watch, a tablet, etc.

[0038] The connection between the first device and the second device may be a wired connection or a wireless connection. The wired connection may be, for example, a connection established via a USB cable, and the wireless connection may be, for example, a connection established via WiFi.

[0039] Among them, the tags can be specifically divided according to the application scenario to which the page belongs. For example, the tags can include the following categories: horizontal screen live broadcast, vertical screen live broadcast, horizontal screen short video, vertical screen short video, shopping, consultation browsing, chat, etc. The embodiment of the present application does not impose specific restrictions on the category and number of tags.

[0040] refer to Figure 2 , Figure 2 A method flow of manually classifying and collecting page information is exemplified.

[0041] like Figure 2 As shown, the method comprises the following steps: Before the user manually collects page information, some preparations need to be done to make the first device and the second device ready before the first device can be used to collect page information from the second device. For example, these preparations include the following: First, the user needs to connect the first device and the second device via a USB cable. Secondly, open the developer options in the second device and enable the USB debugging function in the developer options to ensure that the subsequent second device can respond to the commands sent by the first device through the USB connection. Finally, open the command line tool in the first device, such as opening ADB (Android Debug Bridge), and use the tool to send the commands subsequently entered by the user to the second device so that the second device responds to the command.

[0042] S21, manually inputting a command to the first device or directly operating the second device to switch pages in the second device.

[0043] Specifically, after manually opening the command line tool in the first device, inputting a specified command, the first device can be used to control the second device to open a specified page. Alternatively, the user can directly operate on the screen of the second device to open the specified page.

[0044] When using the command input method to open the specified page in the second device, every time you need to switch to a new page, you need to manually enter the command "adb shell am start-n package name activity name" into the command line tool to open the specified page of the specified application. Using this method requires knowing the package name and activity name corresponding to each application in the first device in advance. Since the same application is installed in a second device with a different new system, its package name and activity name are named differently. If you need to obtain page information in multiple second devices, you also need to obtain the package name and activity name of the application in multiple second devices in advance, and finally enter the command in sequence to control the second device to switch pages. It can be seen that this method not only faces a large labor cost, but also brings additional data statistics difficulties.

[0045] When directly operating the second device to open a specified page in the second device, every time you need to switch to a new page, you need to manually input click and / or slide operations into the second device to open the page of the specified application. In this way, not only do you need to manually operate the two devices synchronously, first operate the second device to open the specified page, and then input subsequent commands to the first device to obtain the information of the current page of the second device, but you also need to manually count which applications and pages in the second device have been opened, and which applications and pages have not been opened, which is prone to problems such as missed opening or repeated opening.

[0046] S22, manually inputting a command to the first device to obtain page information of the current page in the second device.

[0047] Specifically, after the second device displays the specified page, a person can manually input a specified command into the command line tool of the first device to obtain the page information of the currently displayed page in the second device. The page information includes layout information, or also includes a screenshot. The layout information of the page includes the type, position, and size of each control constituting the page.

[0048] When obtaining the layout information of the current page, use the adb shell am start-n command to open the specified page, use the adb shell uiautomator dump command to dump the layout information of the current page into an XML file, and copy the XML file to the computer through the adb pull command. For example, enter the command "adb shell uiautomatordump path 1 in the second device" to grab the layout information of the current page in the second device and store it in the corresponding path 1 in the second device, and then enter the command "adb pull path 1 in the second device path 1 in the first device" to dump the layout information of the current page stored in the second device to the first device.

[0049] When getting a screenshot of the current page, you need to enter the command "adb shell screencap path 2 in the second device" to capture the image of the page in the current second device and store it in the corresponding path 2 in the second device. Then enter the command "adb pull path 2 in the second device path 2 in the first device" to dump the screenshot of the current page stored in the second device to the first device.

[0050] It can be seen that every time the second device switches to a new page, it is necessary to manually input 2 or 4 commands to the first device in order to obtain the page information of the current new page.

[0051] S23, manually inputting tags into the first device to label the current page.

[0052] Specifically, after the first device obtains the page information of the current page in the second device, it determines which tag the page corresponds to by manually observing the page information, and inputs the corresponding tag into the first device, thereby marking the current page with the corresponding tag.

[0053] S24: The first device stores the tagged page information.

[0054] Specifically, after the first device receives the manually input tag, the tag can be associated with the page information of the current page for storage. In an optional storage method, when the tag is manually input into a file storing the page information of the current page in the first device, the tag and the page information are associated and stored in the file; in another optional storage method, when the tag is manually input into the file name storing the page information of the current page in the first device, the tag and the page information are associated and stored in the file name and file.

[0055] S25, manually determine whether the pages to be switched in the second device have been traversed.

[0056] Specifically, each time the page in the second device is switched, the information of the page is obtained and the label is marked, it is necessary to determine whether all the pages to be switched have been traversed. If so, the current work is terminated, otherwise it is necessary to switch to the next page, obtain the information of the next page and mark the label until all the pages to be switched are traversed.

[0057] Based on the manual classification method of collecting page information introduced in the previous article, it can be seen that this method of page collection and labeling relies heavily on manual operations, such as manual page switching, manual command input, manual labeling, manual sorting of statistical data, etc. These manual operations are not only inefficient, but also prone to errors, which seriously affects the efficiency and accuracy of data processing.

[0058] In order to solve the above-mentioned technical problems, the present application provides a method for collecting page information, which is applied to a first device. The method includes: presetting a first program in the first device, and performing the following operations when the first program is run: controlling the second device to display a page in sequence based on an automated command, and obtaining the page information of the current page after each control of the second device to display a page. In addition, if the user turns on the automatic labeling mode, the first device also uses a model to obtain the label corresponding to the current page based on the page information; if the user turns on the manual labeling mode, the first device also receives the label corresponding to the current page input by the user. Finally, the first device stores the page information and label of the current page in association, and further, it can also obtain the identifier of the application to which the current page belongs, the identifier of the activity, etc., and store one or more of them in association with the page information and the label. Thereby reducing labor costs and improving the efficiency and accuracy of data collection and labeling.

[0059] The page information includes the layout information of the page, and the layout information of the page includes the type, size, and position of the controls constituting the page, etc. Optionally, the page information may also include a screenshot of the page, etc.

[0060] Among them, the automation commands include but are not limited to: commands for switching pages, commands for obtaining the current page layout information, commands for obtaining a screenshot of the current page, commands for obtaining the identifier of the application to which the current page belongs, commands for obtaining the activity identifier of the current page, etc.

[0061] Among them, the label of the page specifically indicates the application scenario of the page. The labels of the page include but are not limited to the following: horizontal screen live broadcast, vertical screen live broadcast, horizontal screen short video, vertical screen short video, shopping, consultation and browsing, and chat.

[0062] After implementing the method provided in this application, the following beneficial effects can be brought about: (1) Improve efficiency: The use of automated commands can quickly and in batches collect page information, and the use of models can quickly label pages, significantly improving work efficiency.

[0063] (2) Improve accuracy: The use of automated commands can reduce errors caused by manual operations and ensure the accuracy and consistency of collected page information.

[0064] (3) Reduce costs: Reduce dependence on human resources and reduce the cost of collecting page information.

[0065] (4) Enhanced scalability: Manual methods are difficult to cope with the rapid growth of data volume and cannot meet the needs of large-scale data processing. Automated systems can easily cope with the growth of data volume and meet the needs of large-scale data processing.

[0066] (5) Provide high-quality and efficient data support for market research and scientific research, and promote the development and progress of related fields.

[0067] refer to Figure 3 , Figure 3 A method flow of collecting page information by automatic and manual classification is exemplified.

[0068] like Figure 3 As shown, after the first program is preinstalled in the first device, when the first device runs the first program, a method for automatically combining manual classification and collecting page information can be provided, and the method includes the following steps. Among them, the first program can be a script programmed in a specific language.

[0069] S3, receiving an operation of selecting a tag mode.

[0070] Specifically, after the first device runs the first program, the first device can output prompt information for manual selection of labeling mode, such as automatic labeling options and manual labeling options. When the operation of manually selecting automatic labeling is received, subsequent S311-S314 is executed. When the operation of manually selecting automatic labeling is received, subsequent S321-S323 is executed.

[0071] Among them, automatic labeling refers to the use of a model to predict the label corresponding to the page based on the collected page information, so as to achieve classified collection without manual calculation.

[0072] Among them, manual labeling refers to the subsequent manual calculation method based on the collected page information to determine the label corresponding to the page in order to achieve classified collection, which requires manual calculation.

[0073] In one optional manner, S3 can be executed after the first program is run, or, in another optional manner, the automatic labeling mode is adopted by default after the first program is run. If an operation to reselect the labeling mode is subsequently received, the manually selected mode will be adopted for operation.

[0074] S311-S314: Automatically collect page information, automatically mark tags, and store them in association. In addition, during the execution of S311-S314, the tag mode can be switched manually at any time. If a manual tag mode change operation is received, the subsequent S321-S323 can be switched to be executed.

[0075] S311, obtaining page information based on the automation command.

[0076] Specifically, the first program running on the first device includes a series of automated commands, which are automatically sent to the second device at regular intervals. Some automated commands can simulate user operations to control the second device to automatically switch pages, and some automated commands can also automatically obtain information about each current page in the second device.

[0077] The page information is page layout information, including the type, size, position, etc. of the controls that make up the page. Optionally, the page information may also include screenshots, which are more intuitive and can assist staff in viewing.

[0078] For the specific implementation of obtaining page information, please refer to the following Figure 4 A detailed description of the method shown will not be repeated here.

[0079] S312, extracting features based on the page information.

[0080] Specifically, after obtaining the page information, the first device needs to extract features from the page information, such as the type of controls, the number of controls, the relative position of controls, the absolute position of controls, and other information in the page, and convert this information into numerical features for subsequent models to predict corresponding labels.

[0081] S313, using the model to extract output labels based on features.

[0082] Specifically, a label prediction model is pre-installed in the first device, and the features extracted from the page information are input into the label prediction model to output the label corresponding to the page.

[0083] The label prediction model preset in the first device is a random forest tree (randomforest) model, which is trained based on the following method: 1. Pre-set label categories, such as classification by scene, including but not limited to: horizontal screen live broadcast, vertical screen live broadcast, horizontal screen short video, vertical screen short video, shopping, consultation and browsing, chatting, etc.

[0084] 2. Taking tags as the dimension, multiple samples are sampled from a large number of application pages for each tag (for example, layout information of 100 pages).

[0085] 3. Feature extraction: extract the control type, control quantity, control relative position, control absolute position, etc. from the page layout information and convert them into numerical features.

[0086] 4. Model optimization: input the extracted features into the pre-established initial model, optimize the parameters in the model, ensure that the labels output by the optimized model are consistent with the labels corresponding to the samples, or if the accuracy reaches a certain index, the model will be regarded as a trained model.

[0087] S314, associate and store page information and tags.

[0088] Specifically, after the tag corresponding to the page is obtained, the page information of the page is associated with the tag and stored.

[0089] In an optional manner, the tag can be used as part of the file name, and the corresponding page information can be stored in a file under the file name. Optionally, the file type is an Extensive Markup Language (XML) file. Furthermore, the identification of the application to which the page belongs and the page identification can also be used as part of the file name.

[0090] In another optional manner, the storage path of the tag and the corresponding page information can also be stored in a file, and optionally, the file type is a comma separated value (CSV) file. Furthermore, the identification of the application to which the page belongs, the identification of the page, etc. can also be associated and stored in the file.

[0091] S321-S323: Automatically collect page information, manually mark tags, and store them in association. In addition, during the execution of S321-S323, the tag mode can be switched manually at any time. If a manual tag mode change operation is received, the subsequent S311-S314 can be switched to be executed.

[0092] S321, obtaining page information based on the automation command.

[0093] Specifically, this step is similar to the aforementioned S311, except that the acquired page information includes screenshots, and optionally may also include page layout information, because manual labeling requires reliance on analysis of screenshots.

[0094] S322, receiving manually input labels.

[0095] Specifically, after obtaining the page information, information prompting manual input of the tag corresponding to the page can be output. In an optional manner, the prompt information includes options corresponding to multiple tags for manual selection. In another optional manner, the prompt information instructs manual input of a specific tag, which is not limited in the present embodiment.

[0096] S323, associate and store page information and tags.

[0097] Specifically, this step is similar to the aforementioned S314 and will not be described in detail here.

[0098] It can be seen that the present application provides a method for collecting page information for manual free selection of label modes. In this method, an automated command is included, and the automated command mainly includes simulated user operations of switching pages, obtaining layout information of pages, obtaining screenshots of pages, or also includes obtaining the identifier of the application to which the page belongs and the activity identifier of the page. In this method, the obtained page can also be annotated by automatic tags or manual tags. For example, a label prediction model is included, and the label prediction model can predict the label corresponding to the page based on the layout information of the page. For example, it also includes a plurality of pre-set labels for manual selection of corresponding labels based on the screenshots of the page. In addition, in order to facilitate the further use of the collected data in the later stage, the file is named by using a label, an application identifier to which the page belongs, an activity identifier of the page, etc., and the collected source data is stored under the file name, for example, the associated layout information is stored in the xml file format and the associated screenshots are stored in the jpg file format. In addition, in order to further integrate the collected data, the file path for storing the layout information and the file path for storing the screenshots and the corresponding labels, application identifiers, activity identifiers, etc. are also associated and stored in a CSV file.

[0099] Next, the specific implementation of automatically classifying and collecting page information after selecting automatic tags and the specific implementation of manually classifying and collecting page information after selecting manual tags are introduced in detail.

[0100] refer to Figure 4 , Figure 4 A schematic diagram of an interactive process for automatically classifying and collecting page information provided in an embodiment of the present application.

[0101] like Figure 4As shown, the interactive process involves the automated testing tool, package name acquisition module, activity name acquisition module, label prediction model and file management module in the first device, and also involves the automated testing tool and display screen in the second device. The automated testing tool in the first device is the first program or script referred to above, for example, uiautomator or uiautomator2. The automated testing tool in the second device is compatible with the automated testing tool in the first device, and the two can establish a connection for communication.

[0102] S411: The automated testing tool in the first device receives an operation of selecting an automatic label.

[0103] Specifically, the first device is pre-installed with an automated testing tool. After starting the automated testing tool, a prompt message prompting a manual selection of a labeling mode can be output. When a manual user wants to use the automatic labeling mode, an operation for selecting an automatic label can be input. The operation for selecting an automatic label can also be referred to as a first operation, and the automatic label can also be referred to as others. This application does not limit the names, and it only represents the use of a model to automatically label labels.

[0104] The automated testing tool in the first device can provide the following functions: automatically and regularly generate commands to control the automated testing tool in the second device to simulate user operations to perform corresponding tasks. For example, commands simulating sliding, clicking and other operations can be generated at regular intervals to control the second device to automatically switch applications, switch pages in applications, etc. at regular intervals. For details, please refer to the subsequent S412-S414.

[0105] The automated testing tool in the first device can also provide the following functions: the automated testing tool periodically and automatically generates commands to control the second device, and obtains page information of the current page, including layout information, or also screenshots, after controlling the second device to display the switched page each time. For details, please refer to the subsequent S419-S420.

[0106] S412: The automated testing tool of the first device sends a command to the automated testing tool of the second device to simulate a user operation.

[0107] Specifically, the second device is pre-installed with an automated testing tool. After starting the automated testing tool, it can receive commands sent by the automated testing tool of the first device. The automated testing tool of the first device can automatically send commands to the second device at regular intervals, so that the automated testing tool of the second device simulates user operations based on the commands. Among them, the automated testing tool can use the run function of the subprocess module of Python to automatically execute instructions to control the second device to switch pages, obtain the identifier of the application to which the page belongs, the identifier of the page, etc., and prepare for the subsequent storage and naming of data. In this way, automatic and continuous switching of pages can be achieved by executing simulated operation instructions, thereby realizing rapid collection of page information.

[0108] In addition, during the interval time when the automated testing tool of the first device sends commands, the automated testing tool of the first device can also obtain the identifier of the application to which the current page in the second device belongs, the identifier of the page, page information, tags, etc., see S415-S422 below for details.

[0109] S413, the second device simulates user operation through an automated testing tool to control the display screen to switch pages.

[0110] Specifically, after the second device receives the command sent by the automated testing tool of the first device through the automated testing tool, it can simulate user operations based on the command, such as clicking, sliding the screen, etc., thereby controlling the display screen of the second device to switch to a new page.

[0111] S414, the second device displays the switched page through the display screen.

[0112] Specifically, the second device displays that the page after switching is different from the page before switching, and the application to which the page after switching belongs may be the same as or different from the application to which the page before switching belongs.

[0113] In an embodiment of the present application, in automatic tab mode, the page switched by the second device each time based on a simulated operation can be called a first page. Within a period of time, multiple first pages switched based on multiple simulated operations belong to a first group of pages. The first group of pages can be multiple pages of the same application in the second device or multiple pages of different applications.

[0114] S415: The automated testing tool of the first device requests to obtain the identifier of the application to which the current page of the second device belongs from the package name acquisition module through the first regular expression.

[0115] Specifically, the automated testing tool of the first device includes a first regular expression, which describes a piece of data in the dump file containing the package name of the focus application. Therefore, the first regular expression can be used to request the package name acquisition module to query the dump file to obtain the identifier of the application to which the matching current page belongs, such as the package name.

[0116] Among them, the package name acquisition module is, for example, "get_current_package_name".

[0117] The dump file is obtained by the first device from the second device, and the dump file stores information about the focus application currently running on the first device and information about the focus activity of the focus application. The focus refers to the application running in the foreground, and a human can directly interact with the application and the application's page through the display screen.

[0118] S416: The package name acquisition module of the first device returns the identifier of the application to which the current page of the second device belongs to the automated testing tool.

[0119] Specifically, the package name acquisition module (get_current_package_name) can obtain the application package name matching the first regular expression from the dump file and send it to the automated testing tool. The application package name is the package name of the focus application currently running in the second device.

[0120] S417: The automated testing tool of the first device requests to obtain the identifier of the current page of the second device from the activity name acquisition module through the second regular expression.

[0121] Specifically, the automated testing tool of the first device includes a second regular expression, which describes a piece of data containing the activity name of the focus page of the focus application in the dump file. Therefore, the second regular expression can be used to request the package name acquisition module to query the dump file to obtain the matching identifier of the current page, such as the activity name.

[0122] The activity name acquisition module is, for example, "get_activity_name".

[0123] S418: The activity name acquisition module of the first device returns the identifier of the current page of the second device to the automated testing tool.

[0124] Specifically, the activity name acquisition module (get_activity_name) can obtain the activity name matching the second regular expression from the dump file and send it to the automated testing tool. The activity package name is the identifier of the focus page currently running in the second device.

[0125] S419, the automated testing tool of the first device sends a command to the automated testing tool of the second device to obtain page information (layout information, screenshots) of the current page of the second device.

[0126] Specifically, when the automatic labeling mode is selected, the command sent by the first device may be used only to request the layout information of the current page in the second device, and optionally, a screenshot of the current page in the second device may also be obtained.

[0127] The command for obtaining the layout information of the current page in the second device may be, for example, “adb shelluiautomator dump”.

[0128] The command for obtaining a screenshot of the current page in the second device may be, for example, “adb shellscreencap”.

[0129] S420: The automated testing tool of the second device obtains page information of the current page.

[0130] Specifically, after the automated testing tool of the second device receives the command sent by the automated testing tool of the first device, it obtains the information of the current page through the "auto_get_current_ScreenShot_and_UI" module based on the command, including obtaining the layout information of the current page and taking a screenshot from the display screen.

[0131] S421: The automated testing tool of the second device sends page information of the current page to the automated testing tool of the first device.

[0132] It can be understood that the aforementioned S415-S421 is executed after the first device sends a command to control the second device to switch pages and before sending the next command to switch the next page. The embodiment of the present application does not limit the relative execution order between S415-S421.

[0133] S422: The automated testing tool of the first device obtains a tag corresponding to the current page through a tag prediction model.

[0134] Specifically, a trained label prediction model "mark_label" is pre-installed in the first device, and the acquired layout information of the page is input into the label prediction model, which is a decision tree model, and the corresponding label can be marked for the current page based on the decision tree model. The label predicted by the model can also be called the first label.

[0135] The decision tree model can be found in Figure 6 The architecture shown, Figure 6The decision tree model shown is composed of nodes and edges. Each internal node represents a judgment on a feature, each edge represents the result of the judgment, and each leaf node represents a category or decision result (i.e. the predicted label). The implementation of the decision tree model can handle classification problems with multi-dimensional outputs and is highly efficient.

[0136] Since the cost of building and training an image recognition model is high and difficult, this application avoids using additional screenshots of the page to identify the corresponding labels of the image, and adopts a decision tree model to identify the corresponding labels based on the layout information (xml file) of the page. In addition, a similar semi-supervised learning method can also be used to improve the accuracy of the model and the accuracy of data annotation.

[0137] S423, the first device associates and stores the label of the current page, the identifier of the application, the identifier of the page, and the page information.

[0138] Specifically, the first device can call the file management module to associate and store the obtained previous page label, application identifier, page identifier and page information, for example, first store the obtained current page layout information as an XML file, and use the corresponding label, application package name, activity name and random number to name the XML file, where the random number is a unique and non-repeating number randomly generated for each XML file. Then, the storage path of the current page layout information, the storage path of the screenshot and the corresponding label, application package name, and activity name are associated and stored as a csv file for manual query.

[0139] refer to Figure 5 , Figure 5 A schematic diagram of an interactive process for manually classifying and collecting page information provided in an embodiment of the present application.

[0140] like Figure 5 As shown, the interactive process involves the automated testing tool, package name acquisition module, activity name acquisition module, label prediction model and file management module in the first device, and also involves the automated testing tool and display screen in the second device. The automated testing tool in the first device is the first program or script referred to above, for example, uiautomator or uiautomator2. The automated testing tool in the second device is compatible with the automated testing tool in the first device, and the two can establish a connection for communication.

[0141] S511: The automated testing tool in the first device receives an operation of selecting a manual tag.

[0142] Specifically, the first device is pre-installed with an automated testing tool. After the automated testing tool is started, a prompt message for manually selecting a labeling mode can be output. When the manual labeling mode is used, the operation of selecting a manual label can be input. The operation of selecting a manual label can also be referred to as a third operation, and the manual label can also be referred to as others. This application does not limit the name, and it only represents the manual labeling of labels by manual input.

[0143] The automated testing tool in the first device can provide the following functions: automatically and regularly generate commands to control the automated testing tool in the second device to simulate user operations to perform corresponding tasks. For example, commands simulating sliding, clicking and other operations can be generated at regular intervals to control the second device to automatically switch applications, switch pages in applications, etc. at regular intervals. For details, please refer to the subsequent S512-S514.

[0144] The automated testing tool in the first device can also provide the following functions: the automated testing tool periodically and automatically generates commands to control the second device, and obtains page information of the current page, including layout information, or also screenshots, after controlling the second device to display the switched page each time. For details, please refer to the subsequent S519-S520.

[0145] S512: The automated testing tool of the first device sends a command to the automated testing tool of the second device to simulate a user operation.

[0146] Specifically, the second device is pre-installed with an automated testing tool. After starting the automated testing tool, it can receive commands sent by the automated testing tool of the first device. The automated testing tool of the first device can automatically send commands to the second device at regular intervals, so that the automated testing tool of the second device simulates user operations based on the commands. Among them, the automated testing tool can use the run function of the subprocess module of Python to automatically execute instructions to control the second device to switch pages, obtain the identifier of the application to which the page belongs, the identifier of the page, etc., and prepare for the subsequent storage and naming of data. In this way, automatic and continuous switching of pages can be achieved by executing simulated operation instructions, thereby realizing rapid collection of page information.

[0147] In addition, during the interval time when the automated testing tool of the first device sends commands, the automated testing tool of the first device can also obtain the identifier of the application to which the current page in the second device belongs, the identifier of the page, page information, tags, etc., see S515-S522 below for details.

[0148] S513, the second device simulates user operation through an automated testing tool to control the display screen to switch pages.

[0149] Specifically, after the second device receives the command sent by the automated testing tool of the first device through the automated testing tool, it can simulate user operations based on the command, such as clicking, sliding the screen, etc., thereby controlling the display screen of the second device to switch to a new page.

[0150] S514, the second device displays the switched page through the display screen.

[0151] Specifically, the second device displays that the page after switching is different from the page before switching, and the application to which the page after switching belongs may be the same as or different from the application to which the page before switching belongs.

[0152] In an embodiment of the present application, in manual tab mode, the page switched by the second device based on a simulated operation each time can be called a second page. Within a period of time, multiple second pages switched based on multiple simulated operations belong to a second group of pages. The second group of pages can be multiple pages of the same application in the second device or multiple pages of different applications.

[0153] S515: The automated testing tool of the first device requests to obtain the identifier of the application to which the current page of the second device belongs from the package name acquisition module through the first regular expression.

[0154] Specifically, the automated testing tool of the first device includes a first regular expression, which describes a piece of data in the dump file containing the package name of the focus application. Therefore, the first regular expression can be used to request the package name acquisition module to query the dump file to obtain the identifier of the application to which the matching current page belongs, such as the package name.

[0155] Among them, the package name acquisition module is, for example, "get_current_package_name".

[0156] The dump file is obtained by the first device from the second device, and the dump file stores information about the focus application currently running on the first device and information about the focus activity of the focus application. The focus refers to the application running in the foreground, and a human can directly interact with the application and the application's page through the display screen.

[0157] S516: The package name acquisition module of the first device returns the identifier of the application to which the current page of the second device belongs to the automated testing tool.

[0158] Specifically, the package name acquisition module (get_current_package_name) can obtain the application package name matching the first regular expression from the dump file and send it to the automated testing tool. The application package name is the package name of the focus application currently running in the second device.

[0159] S517: The automated testing tool of the first device requests the activity name acquisition module to acquire the identifier of the current page of the second device through the second regular expression.

[0160] Specifically, the automated testing tool of the first device includes a second regular expression, which describes a piece of data containing the activity name of the focus page of the focus application in the dump file. Therefore, the second regular expression can be used to request the package name acquisition module to query the dump file to obtain the matching identifier of the current page, such as the activity name.

[0161] The activity name acquisition module is, for example, "get_activity_name".

[0162] S518: The activity name acquisition module of the first device returns the identifier of the current page of the second device to the automated testing tool.

[0163] Specifically, the activity name acquisition module (get_activity_name) can obtain the activity name matching the second regular expression from the dump file and send it to the automated testing tool. The activity package name is the identifier of the focus page currently running in the second device.

[0164] S519: The automated testing tool of the first device sends a command to the automated testing tool of the second device to obtain page information (layout information, screenshots) of the current page of the second device.

[0165] Specifically, when the manual labeling mode is selected, the command sent by the first device may only be used to request a screenshot of the current page in the second device, which can facilitate manual and intuitive calculation of the corresponding label. Optionally, the layout information of the current page in the second device may also be obtained.

[0166] The command for obtaining the layout information of the current page in the second device may be, for example, “adb shelluiautomator dump”.

[0167] The command for obtaining a screenshot of the current page in the second device may be, for example, “adb shellscreencap”.

[0168] S520: The automated testing tool of the second device obtains information of the current page.

[0169] Specifically, after the automated testing tool of the second device receives the command sent by the automated testing tool of the first device, it obtains the information of the current page through the "auto_get_current_ScreenShot_and_UI" module based on the command, including obtaining the layout information of the current page and taking a screenshot from the display screen.

[0170] S521: The automated testing tool of the second device sends information about the current page to the automated testing tool of the first device.

[0171] It can be understood that the aforementioned S515-S521 is executed after the first device sends a command to control the second device to switch pages and before sending the next command to switch the next page. The embodiment of the present application does not limit the relative execution order between S515-S521.

[0172] S522: The automated testing tool of the first device receives an operation of inputting a tag.

[0173] Specifically, the automated testing tool of the first device is preset with multiple tags. After the first device obtains a screenshot of the current page of the second device, it can output information prompting a human to select a corresponding tag, so that the human can select a matching tag from the multiple tags based on the screenshot. The operation of manually inputting a tag can also be referred to as a second operation, and the tag manually input can also be referred to as a second tag.

[0174] Optionally, when the current page obtained is a new scene, and there is no label that matches the new page in the preset labels in the automated testing tool of the first device, it can also support manual input of new labels. In addition, the page information and labels in the new scene are used as training data for subsequent models to ensure that the real-time updated model can output correct new labels in the case of automatic labeling.

[0175] S523: The first device associates and stores the label of the current page, the identifier of the application, the identifier of the page, and the page information.

[0176] Specifically, the first device can call the file management module to associate and store the obtained previous page label, application identifier, page identifier and page information, for example, first store the obtained current page layout information as an XML file, and use the corresponding label, application package name, activity name and random number to name the XML file, where the random number is a unique and non-repeating number randomly generated for each XML file. Then, the storage path of the current page layout information, the storage path of the screenshot and the corresponding label, application package name, and activity name are associated and stored as a csv file for manual query.

[0177] It can be seen that on the basis of providing automatic collection of page information, in addition to providing an automatic labeling method, the present application also provides a manual labeling method. This not only improves the accuracy and efficiency of classified collection of page information, but also can avoid the model from being unable to accurately output untrained labels when facing new page scenarios. Manual labeling can be used to classify and collect page information, and the manually processed page information and corresponding labels can also be used as training data for the subsequent stage model to further update the model and ensure that the model learns the data in the new scenario in real time.

[0178] Next, the product form and software and hardware architecture of the electronic device involved in this application are introduced.

[0179] The first device and the second device involved in this application can be devices equipped with iOS®, Android®, Microsoft® or other operating systems. When the operating systems are different, the specific automated testing tools used are different, but they can all achieve the above Figure 3-Figure 5 The method introduced.

[0180] The first device is, for example, a mobile phone, a tablet computer, an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device and / or a smart city device, etc.

[0181] The second device is, for example, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), and the like.

[0182] For the hardware architecture of the first device and the second device, reference may be made to the following introduction to the electronic device 100 .

[0183] Figure 7 The schematic diagram of the hardware architecture of the electronic device 100 is exemplarily shown.

[0184] like Figure 7 As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a sensor module 180, and a display screen 194, etc. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180K, etc.

[0185] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0186] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. Among them, the controller can be the nerve center and command center of the electronic device 100. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0187] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0188] In the embodiment of the present application, the processor 110 of the first device and the processor 110 of the second device can call the wireless communication module 160 to establish a connection with each other, so that the automated test tool in the first device can communicate with the automated test tool in the second device to achieve the aforementioned Figure 3-Figure 5 Introduced method.

[0189] In some embodiments, the processor 110 may include one or more interfaces, such as a universal serial bus (USB) interface.

[0190] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0191] In the embodiment of the present application, the USB interface 130 of the first device and the USB interface 130 of the second device can be connected to each other through a USB cable, so that the automated test tool in the first device can communicate with the automated test tool in the second device to achieve the aforementioned Figure 3-Figure 5 Introduced method.

[0192] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0193] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0194] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0195] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0196] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device, or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0197] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0198] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0199] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0200] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microled, micro-oled, quantum dot light emitting diodes (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0201] In an embodiment of the present application, the display screen 194 of the first device can display the acquired page information of the second device, etc., as well as display stored xml files, csv files, etc.

[0202] In the embodiment of the present application, the display screen 194 of the second device can display the switched page based on the simulation operation.

[0203] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0204] The random access memory may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc. The random access memory may be directly read and written by the processor 110, and may be used to store executable programs (such as machine instructions) of an operating system or other running programs, and may also be used to store user and application data, etc.

[0205] Non-volatile memory may include disk storage devices and flash memory. Flash memory may include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle, single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the potential level of the storage cell, and universal flash storage (UFS), embedded multi media card (eMMC), etc. according to the storage specification. Non-volatile memory can also store executable programs and store user and application data, etc., which can be loaded into random access memory in advance for direct reading and writing by processor 110.

[0206] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0207] In the embodiment of the present application, the memory of the first device may store the aforementioned automated testing tool (also called the first program or script), and the memory of the second device may also store the aforementioned automated testing tool. The memory of the first device may also store associated information such as the acquired page information, tags, application package names, and activity names.

[0208] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0209] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0210] In the embodiment of the present application, the first device can receive manually input operations through the pressure sensor 180A and the touch sensor 180K, such as the operation of selecting automatic labels or manual labels, and the operation of outputting labels, etc.

[0211] It should be understood that each step in the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0212] The present application also provides an electronic device, which may include a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the method executed by the electronic device in any one of the above embodiments.

[0213] The present application also provides a chip system, including a processing circuit and an interface circuit, the interface circuit is used to receive computer instructions and transmit them to the processing circuit, and the processing circuit is used to run the computer instructions to implement a method executed by an electronic device as in any of the above embodiments.

[0214] The present application also provides a chip system, which includes at least one processor, for implementing the method executed by the electronic device in any of the above embodiments. In a possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0215] A chip system can be composed of chips or include chips and other discrete devices.

[0216] Optionally, there may be one or more processors in the chip system. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0217] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0218] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0219] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method executed by the electronic device in any one of the above embodiments is implemented.

[0220] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the method executed by the electronic device in any of the above embodiments.

[0221] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0222] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0223] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0224] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0225] The terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0226] In short, the above is only an embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of this application shall be included in the protection scope of this application.

Claims

1. A method for collecting page information, characterized in that: The method is applied to a first device, and the method includes: The first device executes a first program, and performs the following operations based on the first program: The first device controls the second device to display each page in the first group of pages in sequence; In the process of controlling the second device to display the first page, the first device further performs the following operations: Acquire layout information of the first page, where the first page is any page in the first group of pages; A corresponding first tag is determined based on the layout information of the first page, where the first tag indicates a scene of the first page.

2. The method according to claim 1, characterized in that After controlling the second device to sequentially display each page in the first group of pages, the method further includes: Obtain a screenshot of the first page; The storage path of the layout information of the first page and the storage path of the screenshot of the first page are associated with the first tag and stored.

3. The method according to claim 2, characterized in that After obtaining the screenshot of the first page, the method further includes: The storage path of the layout information of the first page and the storage path of the screenshot of the first page are stored in association with one or more of the following: an identifier of the application to which the first page belongs and an identifier of the first page.

4. The method according to claim 1, characterized in that Before controlling the second device to sequentially display each page in the first group of pages, the method further includes: A first operation is received; Determining the corresponding first tag based on the layout information of the first page specifically includes: responding to the first operation, determining the corresponding first tag based on the layout information of the first page.

5. The method according to claim 1, characterized in that The method further comprises: The first device controls the second device to display each page in the second group of pages in sequence; In the process of controlling the second device to display the second page, the first device further performs the following operations: Acquire layout information of the second page, where the second page is any page in the second group of pages; A second operation of inputting a second tag is received, and the second tag is determined as a tag corresponding to the second page, where the second tag indicates a scene of the second page.

6. The method according to claim 5, characterized in that Before the first device controls the second device to sequentially display each page in the second group of pages, the method further includes: A third operation is received; Receiving a second operation of inputting a second tag, and determining the second tag as a tag corresponding to the second page, specifically includes: responding to the third operation, displaying information prompting the user to input a tag, receiving a second operation of inputting a second tag, and determining the second tag as a tag corresponding to the second page.

7. The method according to claim 1, characterized in that The first device controls the second device to sequentially display each page in the first group of pages, specifically including: the first device controls the second device to sequentially display each page in the first group of pages based on a command of a simulated operation.

8. The method according to claim 1, characterized in that The first device controls the second device to sequentially display each page in the first group of pages, specifically including: The first device sequentially sends a plurality of simulation operation commands to the second device, so that the second device sequentially responds to each simulation operation command and sequentially displays each page in the first group of pages; Alternatively, the first device sends a command to start an automation application to the second device, so that the second device starts the automation application, and generates multiple simulation operation commands in sequence through the automation application, responds to each simulation operation command in sequence, and displays each page in the first group of pages in sequence.

9. The method according to claim 1, characterized in that: Before acquiring the layout information of the first page, the method further includes: The first device obtains an identifier of an application to which the first page belongs from the dumped file; The first device obtains the activity identifier of the first page from the dump file.

10. The method according to claim 9, characterized in that The first device obtains the identifier of the application to which the first page belongs from the dump file, specifically including: the first device obtains the identifier of the application to which the first page belongs that matches the first regular expression from the dump file; The first device obtains the activity identifier of the first page from the dump file, specifically including: the first device obtains the activity identifier of the first page matching the second regular expression from the dump file.

11. The method according to claim 1, characterized in that: Determining a corresponding first tag based on the layout information of the first page specifically includes: Input the features of the layout information of the first page into a label prediction model to obtain a first label corresponding to the first page, wherein the label prediction model is trained by multiple groups of data, and one group of the data includes: the features of the layout information of the page, and the label corresponding to the page.

12. The method according to claim 11, characterized in that Before determining the corresponding first tag based on the layout information of the first page, the method further includes: acquiring a screenshot of the first page; The features of the layout information of the first page are input into a label prediction model, and the label prediction model is supervised by a screenshot of the first page to obtain a first label corresponding to the first page.

13. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images