User interface element identification method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a user interface element recognition method and device, equipment and a storage medium. The method comprises the steps that a resource file, a code file and resource request information are analyzed to obtain at least one first target element identifier and corresponding first target element attribute information; performing picture interception on the loaded user interface to obtain a user interface image, and identifying the user interface image to obtain at least one second target element identifier and corresponding second target element position information; and based on the at least one first target element identifier, the corresponding first target element attribute information, the at least one second target element identifier and the corresponding second target element position information, performing de-duplication combination to obtain an element identification result, and feeding back the element identification result to the background server. According to the scheme, user interface element recognition and integration can be performed on data sources of different dimensions, the accuracy and integrity of element recognition are guaranteed, and the recognition efficiency is remarkably improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for identifying user interface elements. Background Art

[0002] With the rapid development of computer technology, applications are becoming increasingly integrated into people's lives, work, entertainment, and other areas, providing a convenient digital experience. The user interface (UI) serves as the direct interface between software and users. Through UI elements and interaction logic, users can complete operations without having to understand complex code. Its stability, ease of use, and functional integrity directly impact users' overall evaluation of the software. To ensure application quality, automated UI testing is required to comprehensively and efficiently examine various UI functions and interactions, promptly identifying and remediating potential issues.

[0003] Identifying and locating UI elements is a core step in developing user interface test scripts. Related technologies rely on test engineers manually processing and annotating UI elements to identify and locate them. However, with the increasing functionality of applications and the increasing complexity of user interfaces, this approach has led to low efficiency in identifying and locating UI elements, high recognition error rates, and the inability to provide accurate and reliable element recognition results for subsequent test script generation. Summary of the Invention

[0004] The embodiments of the present application provide a user interface element recognition method, apparatus, device, and storage medium to solve the technical problems of low efficiency and high recognition error rate in identifying and locating user interface elements in related technologies. It can identify and integrate user interface elements for data sources of different dimensions separately, ensure the accuracy and completeness of element recognition results, significantly improve recognition efficiency, and provide reliable element recognition results for subsequent generation of test scripts.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying a user interface element, the method comprising:

[0006] Obtain the installation package of the target application and extract the resource files and code files in the preset directory of the installation package;

[0007] Loading a user interface corresponding to the target application, and obtaining resource request information associated with the user interface during the loading process;

[0008] Parsing the resource file, the code file, and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information;

[0009] Taking a screenshot of the loaded user interface to obtain a user interface image, and identifying the user interface image to obtain at least one second target element identifier and corresponding second target element position information;

[0010] Based on the at least one first target element identifier and the corresponding first target element attribute information, and the at least one second target element identifier and the corresponding second target element position information, decomposition and combination are performed to obtain an element recognition result, and the element recognition result is fed back to the background server.

[0011] In a second aspect, an embodiment of the present application further provides a user interface element recognition device, comprising:

[0012] A file extraction module configured to obtain an installation package of a target application and extract resource files and code files in a preset directory in the installation package;

[0013] a request information acquisition module configured to load the user interface corresponding to the target application and acquire resource request information associated with the user interface during the loading process;

[0014] A first element determination module is configured to parse the resource file, the code file and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information;

[0015] A second element determination module is configured to capture the loaded user interface to obtain a user interface image, and identify the user interface image to obtain at least one second target element identifier and corresponding second target element position information;

[0016] The element integration feedback module is configured to obtain an element recognition result based on the decomposition and combination of the at least one first target element identifier and the corresponding first target element attribute information, and the at least one second target element identifier and the corresponding second target element position information, and to feed back the element recognition result to the background server.

[0017] In a third aspect, an embodiment of the present application further provides a user interface element recognition device, the device comprising:

[0018] one or more processors;

[0019] a storage device configured to store one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the user interface element recognition method described in the embodiment of the present application.

[0021] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are configured to execute the user interface element recognition method described in the embodiment of the present application.

[0022] In an embodiment of the present application, by obtaining the installation package of the target application, the resource files and code files in the preset directory in the installation package are extracted; the user interface corresponding to the target application is loaded, and the resource request information associated with the user interface during the loading process is obtained; the resource files, code files and resource request information are parsed to obtain at least one first target element identifier and corresponding first target element attribute information; the loaded user interface is captured to obtain a user interface image, and the user interface image is identified to obtain at least one second target element identifier and corresponding second target element position information; based on the at least one first target element identifier and the corresponding first target element attribute information, and the at least one second target element identifier and the corresponding second target element position information, decomposition and combination are performed to obtain an element recognition result, and the element recognition result is fed back to the background server. In the above scheme, by extracting the resource files and code files in the preset directory in the installation package, and obtaining the resource request information associated with the user interface during the loading process, diverse reference information can be provided for comprehensive and accurate identification of user interface elements; by parsing the resource files, the code files and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information, and identifying the user interface image to obtain at least one second target element identifier and corresponding second target element position information, user interface elements can be identified separately for data sources of different dimensions to ensure the accuracy and completeness of the element identification results; finally, by de-duplicating and combining the at least one first target element identifier and the corresponding first target element attribute information, and the at least one second target element identifier and the corresponding second target element position information to obtain the element identification result, the identification efficiency is significantly improved, and reliable element identification results are provided for the subsequent generation of test scripts. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a method for identifying user interface elements provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a data processing process of a method for identifying user interface elements provided in an embodiment of the present application;

[0025] Figure 3 A flowchart of a user interface element recognition method including a process of determining a first target element identifier and corresponding first target element attribute information provided in an embodiment of the present application;

[0026] Figure 4 A flowchart of a user interface element recognition method including a process of determining a first element identifier and corresponding first element attribute information provided in an embodiment of the present application;

[0027] Figure 5 A flowchart of a user interface element recognition method including a process of determining a second target element identifier and corresponding second target element position information provided in an embodiment of the present application;

[0028] Figure 6 A flowchart of a user interface element recognition method including a process of determining an element recognition result provided in an embodiment of the present application;

[0029] Figure 7 A flowchart of a user interface element identification method including a process of extracting resource files and code files provided in an embodiment of the present application;

[0030] Figure 8 A flowchart of a method for identifying user interface elements including a process for generating a test script provided in an embodiment of the present application;

[0031] Figure 9 A structural block diagram of a user interface element recognition device provided in an embodiment of the present application;

[0032] Figure 10 A schematic diagram of the structure of a user interface element recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following is a further detailed description of the embodiments of the present application in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the embodiments of the present application, and are not intended to limit the embodiments of the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions of the embodiments of the present application, rather than all structures.

[0034] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0035] In the user interface element recognition method provided in the embodiment of the present application, the execution entity of each step can be a computer device, which refers to any electronic device with data calculation, processing and storage capabilities, such as mobile phones, PCs (Personal Computers), tablet computers and other terminal devices, or servers and other devices. The embodiment of the present application does not limit this.

[0036] Figure 1 A flowchart of a method for identifying user interface elements provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the user interface element recognition method specifically includes the following steps:

[0037] Step S101: Obtain the installation package of the target application, and extract the resource files and code files in the preset directory in the installation package.

[0038] Among them, the installation package can be a file collection of various resources and codes required to run the target application. The installation packages specifically adapted to different operating systems may be APK (Android Package Kit, Android application package), EXE (executable file), etc. The preset target can be a specific directory determined according to the specifications and conventions of application development, which is used to specifically locate the resource files and code files in the installation package. The resource files may include layout files, image resource files, static resource files, etc. The code files may include configuration files, bytecode files, etc. It should be noted that the resource files may include information such as the type, position, size, and hierarchical relationship of the user interface elements, and the code files may include information such as the initial attribute settings and creation process of the user interface elements. Therefore, the resource files and code files can be used for subsequent identification of user interface elements.

[0039] Step S102: Load the user interface corresponding to the target application, and obtain resource request information associated with the user interface during the loading process.

[0040] The process of loading the user interface corresponding to the target application triggers a request to load dynamic resources, which may include information about the user interface elements that need to be dynamically loaded. Therefore, the resource request information associated with the user interface during the loading process can be obtained for subsequent identification of the user interface elements.

[0041] Step S103: parse the resource file, the code file, and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information.

[0042] Among them, resource files, code files and resource request information can be used as different data sources to parse and determine user interface elements. Specifically, preset regular expressions or preset grammar rules can be used to extract user interface elements from resource files or code files. Network models based on machine learning or deep learning can also be used to comprehensively analyze resource files and code files and extract user interface elements. User interface elements can also be extracted from resource request information using element name or code matching. By comprehensively analyzing the results of resource files, code files and resource request information, at least one first target element identifier and corresponding first target element attribute information can be obtained. The first target element identifier can be a unique identifier of a specific user interface element predefined according to development rules. The first target element attribute information can include path, class name, coordinate position, etc.

[0043] Step S104: Capture the loaded user interface to obtain a user interface image, and identify the user interface image to obtain at least one second target element identifier and corresponding second target element position information.

[0044] Among them, since the user interface with a higher degree of complexity may rely on parsing resource files, code files and resource request information, it is impossible to fully cover all user interface elements of the user interface. Therefore, the loaded user interface can be captured to obtain a user interface image, and the user interface image can be identified to extract user interface elements. Specifically, the user interface image can be used to identify and extract user interface elements based on a network model of machine learning or deep learning, or it can be used to identify and extract user interface elements based on preset key characters or preset shape outlines, which is not limited in this application. By identifying the user interface image, at least one second target element identifier and corresponding second target element position information can be obtained. The second target element identifier can be a unique identifier of the image content corresponding to a specific user interface element pre-defined according to the development rules, and the second target element information can be the coordinate position of the user interface element corresponding to the second target element identifier in the user interface.

[0045] Step S105: De-recombining and obtaining an element recognition result based on at least one first target element identifier and corresponding first target element attribute information, and at least one second target element identifier and corresponding second target element position information, and feeding back the element recognition result to the background server.

[0046] Among them, since at least one first target element identifier and the corresponding first target element attribute information are the parsing results for resource files, code files and resource request information, and at least one second target element identifier and the corresponding second target element position information are the parsing results for a specific user interface image, the two may contain the same user interface elements, constituting repeated recognition results. Therefore, based on at least one first target element identifier and the corresponding first target element attribute information, and at least one second target element identifier and the corresponding second target element position information, deduplication and combination can be performed to eliminate repeated user interface elements, and combine user interface elements identified from different data sources to obtain element recognition results, which can be buttons, text boxes, menus, check boxes, etc. By feeding back the element recognition results to the background server, reliable reference information can be provided for the development of automated test scripts. Figure 2 A schematic diagram of a data processing process of a method for identifying user interface elements provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the resource file 101 and code file 102 derived from the installation package, and the resource request information 103 derived from the user interface loading process can be parsed to obtain at least one first target element identifier 104 and corresponding first target element attribute information 105, and the user interface image 106 obtained by capturing the user interface screen can be identified to obtain at least one second target element identifier 107 and corresponding second target element position information 108. Finally, the at least one first target element identifier 104 and the corresponding first target element attribute information 105, as well as the at least one second target element identifier 107 and the corresponding second target element position information 108 are de-recombined to obtain an element recognition result 109.

[0047] In the above, by obtaining the installation package of the target application, the resource files and code files in the preset directory in the installation package are extracted; the user interface corresponding to the target application is loaded, and the resource request information associated with the user interface during the loading process is obtained; the resource files, code files and resource request information are parsed to obtain at least one first target element identifier and corresponding first target element attribute information; the loaded user interface is captured to obtain a user interface image, and the user interface image is identified to obtain at least one second target element identifier and corresponding second target element position information; based on the at least one first target element identifier and the corresponding first target element attribute information, and the at least one second target element identifier and the corresponding second target element position information, decomposition and combination are performed to obtain an element recognition result, and the element recognition result is fed back to the background server. In the above scheme, by extracting the resource files and code files in the preset directory in the installation package, and obtaining the resource request information associated with the user interface during the loading process, diversified reference information can be provided for comprehensive and accurate identification of user interface elements; by parsing the resource files, code files and resource request information to obtain at least one first target element identifier and corresponding first target element attribute information, and identifying the user interface image to obtain at least one second target element identifier and corresponding second target element position information, user interface elements can be identified separately for data sources of different dimensions to ensure the accuracy and completeness of the element identification results; finally, by de-duplicating and combining the element identification results based on at least one first target element identifier and corresponding first target element attribute information, and at least one second target element identifier and corresponding second target element position information, the identification efficiency is significantly improved, and reliable element identification results are provided for subsequent generation of test scripts.

[0048] Figure 3 A flowchart of a user interface element recognition method including a process of determining a first target element identifier and corresponding first target element attribute information is provided in an embodiment of the present application. Figure 3 As shown, the user interface element recognition method specifically includes the following steps:

[0049] Step S201: Obtain the installation package of the target application, and extract the resource files and code files in the preset directory in the installation package.

[0050] Step S202: Load the user interface corresponding to the target application, and obtain resource request information associated with the user interface during the loading process.

[0051] Step S203: Identify and integrate the resource file and the code file to obtain at least one first element identifier and corresponding first element attribute information.

[0052] Among them, for resource files and code files that also come from the installation package, user interface elements can be extracted from the resource files and code files separately based on preset regular expressions or preset grammar rules, or the resource files and code files can be input together into a network model based on machine learning or deep learning to realize the extraction of user interface elements, that is, to obtain at least one first element identifier and corresponding first element attribute information, which is not limited in this application.

[0053] Step S204: performing an associated search on the resource request information based on the preset element name to obtain at least one second element identifier and corresponding second element attribute information.

[0054] For resource request information generated during the user interface loading process, the preset element name can be predefined by the developer for the user interface element to be dynamically loaded, such as a "login button," a "drop-down menu," etc. Therefore, an associated search can be performed based on the preset element name to extract the dynamically loaded user interface element in the resource request information, thereby obtaining at least one second element identifier and corresponding second element attribute information.

[0055] Step S205: De-recombining at least one first element identifier and corresponding first element attribute information, and at least one second element identifier and corresponding second element attribute information to obtain at least one first target element identifier and corresponding first target element attribute information.

[0056] Among them, since the element extraction results from different data sources may be repeated, it is necessary to deduplicate and combine at least one first element identifier and the corresponding first element attribute information, as well as at least one second element identifier and the corresponding second element attribute information, eliminate duplicate user interface elements, and obtain at least one first target element identifier and the corresponding first target element attribute information.

[0057] Step S206: Capture the loaded user interface to obtain a user interface image, and identify the user interface image to obtain at least one second target element identifier and corresponding second target element position information.

[0058] Step S207: De-recombining and obtaining an element recognition result based on at least one first target element identifier and corresponding first target element attribute information, and at least one second target element identifier and corresponding second target element position information, and feeding back the element recognition result to the background server.

[0059] As described above, by identifying user interface elements for internal files from the installation package and resource requests that occur during page loading, different data sources can be effectively combined to capture user interface elements, which is conducive to providing comprehensive and accurate element extraction results.

[0060] Figure 4 A flowchart of a user interface element recognition method including a process of determining a first element identifier and corresponding first element attribute information is provided in an embodiment of the present application. Figure 4 As shown, the user interface element recognition method specifically includes the following steps:

[0061] Step S301: Obtain the installation package of the target application, and extract the resource files and code files in the preset directory in the installation package.

[0062] Step S302: Load the user interface corresponding to the target application, and obtain resource request information associated with the user interface during the loading process.

[0063] Step S303: extract at least one first candidate element identifier and corresponding first candidate element attribute information from the resource file based on a preset regular expression.

[0064] Among them, since the element identifier and element attribute information corresponding to the user interface element follow specific grammatical rules in the resource file, a preset regular expression can be used for matching and extraction to obtain at least one first candidate element identifier and corresponding first candidate element attribute information.

[0065] Step S304: Input the resource file and the code file into the trained first network model to obtain at least one second candidate element identifier and corresponding second candidate element attribute information.

[0066] Among them, since the preset regular expression is suitable for quickly extracting simple patterned content, it may not be possible to completely extract all user interface elements for resource files involving nested structures and complex grammar. Therefore, you can choose to input the resource files and code files into the trained first network model for parsing to make up for the shortcomings of extracting user interface elements based solely on the preset regular expression. Optionally, the first network model can be an NLP (Natural Language Processing) model, a GNN (Graph Neural Network) model, etc. The specific training data can be the resource files and code files extracted from the open source installation package, as well as manually annotated user interface elements, including element identifiers and element attribute information, which are not limited in this application.

[0067] Step S305: De-recombining at least one first candidate element identifier and corresponding first candidate element attribute information, and at least one second candidate element identifier and corresponding second candidate element attribute information to obtain at least one first element identifier and corresponding first element attribute information.

[0068] Among them, since the element extraction results corresponding to the preset regular expression and the first network model may be repeated, it is necessary to deduplicate and combine at least one first candidate element identifier and the corresponding first candidate element attribute information, as well as at least one second candidate element identifier and the corresponding second candidate element attribute information, eliminate repeated user interface elements, and obtain at least one first element identifier and the corresponding first element attribute information.

[0069] Step S306: performing an associated search on the resource request information based on the preset element name to obtain at least one second element identifier and corresponding second element attribute information.

[0070] Step S307: De-recombining at least one first element identifier and corresponding first element attribute information, and at least one second element identifier and corresponding second element attribute information to obtain at least one first target element identifier and corresponding first target element attribute information.

[0071] Step S308: Capture the loaded user interface to obtain a user interface image, and identify the user interface image to obtain at least one second target element identifier and corresponding second target element position information.

[0072] Step S309: De-recombining and obtaining an element recognition result based on at least one first target element identifier and corresponding first target element attribute information, and at least one second target element identifier and corresponding second target element position information, and feeding back the element recognition result to the background server.

[0073] As described above, by processing resource files and code files using different extraction methods, the recognition pattern of user interface elements can be optimized. By combining the element extraction results corresponding to the preset regular expression and the first network model, it is beneficial to capture user interface elements more comprehensively and accurately.

[0074] Figure 5 A flowchart of a user interface element recognition method including a process of determining a second target element identifier and corresponding second target element position information is provided in an embodiment of the present application, such as Figure 5 As shown, the user interface element recognition method specifically includes the following steps:

[0075] Step S401: Obtain the installation package of the target application, and extract the resource files and code files in the preset directory in the installation package.

[0076] Step S402: Load the user interface corresponding to the target application, and obtain resource request information associated with the user interface during the loading process.

[0077] Step S403: parse the resource file, the code file, and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information.

[0078] Step S404: Capture the loaded user interface to obtain a user interface image.

[0079] Step S405: Input the user interface image into the trained second network model to obtain at least one third candidate element identifier and corresponding first candidate element position information.

[0080] Among them, since user interface elements may be dynamically generated when the application is running, there are limitations in identifying user interface elements based solely on resource files, code files, and resource request information, and it is impossible to capture user interface elements that are dynamically rendered during operation. Therefore, you can choose to use a trained second network model to identify the captured user interface image. The second network model can be a YOLOv8 model, a Faster R-CNN model, etc. The specific training data can be the user interface image captured when the open source application is running and the manually annotated user interface elements, including element identification and element location information, which is not limited in this application.

[0081] Step S406: performing local image recognition matching preset key characters on the user interface image to determine at least one fourth candidate element identifier and corresponding second candidate element position information.

[0082] Among them, there are user interface elements containing specific characters in the user interface, such as titles, function buttons, etc. Therefore, local image recognition can be performed by matching preset key characters. The preset key characters can be a pre-built character template, which is used to determine the user interface element corresponding to the local image, and then obtain the fourth candidate element identifier and the corresponding second candidate element position information.

[0083] Step S407: De-recombining at least one third candidate element identifier and corresponding first candidate element position information, and at least one fourth candidate element identifier and corresponding second candidate element position information to obtain at least one second target element identifier and corresponding second target element position information.

[0084] Among them, since the element extraction results corresponding to the second network model and the preset key characters may be repeated, it is necessary to deduplicate and combine at least one third candidate element identifier and the corresponding first candidate element position information, as well as at least one fourth candidate element identifier and the corresponding second candidate element position information, eliminate repeated user interface elements, and obtain at least one second target element identifier and the corresponding second target element position information.

[0085] Step S408: De-recombining and obtaining an element recognition result based on at least one first target element identifier and corresponding first target element attribute information, and at least one second target element identifier and corresponding second target element position information, and feeding back the element recognition result to the background server.

[0086] As described above, by using different recognition methods to extract user interface elements from the captured user interface image, the user interface elements displayed in the user interface image can be more comprehensively located and captured, thereby ensuring the integrity of the element extraction result.

[0087] Figure 6 A flowchart of a user interface element recognition method including a process of determining an element recognition result is provided in an embodiment of the present application, such as Figure 6 As shown, the user interface element recognition method specifically includes the following steps:

[0088] Step S501: Obtain the installation package of the target application, and extract the resource files and code files in the preset directory in the installation package.

[0089] Step S502: Load the user interface corresponding to the target application, and obtain resource request information associated with the user interface during the loading process.

[0090] Step S503: parse the resource file, the code file, and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information.

[0091] Step S504: Capture the loaded user interface to obtain a user interface image, and identify the user interface image to obtain at least one second target element identifier and corresponding second target element position information.

[0092] Step S505: extract the first target element position information from the first target element attribute information, eliminate the second target element identifiers in at least one second target element identifier that satisfy the corresponding second target element position information that is consistent with the first target element position information to obtain a third target element identifier, combine at least one first target element identifier and the corresponding first target element attribute information, and the third target element identifier and the corresponding second target element position information to obtain an element recognition result, and feed back the element recognition result to the background server.

[0093] Among them, the element extraction results obtained by parsing based on resource files, code files and resource request information, and the element extraction results obtained by identifying based on user interface images may contain the same user interface elements with repeated positions. Therefore, the first target element position information can be extracted from the first target element attribute information, and the first target element position information and the second target element position information can be compared to determine whether duplicate results appear. Furthermore, the second target element identifiers that meet the corresponding second target element position information consistent with the first target element position information in at least one second target element identifier are eliminated to obtain a third target element identifier, and the additional user interface elements obtained by identifying the user interface image are retained. Finally, at least one first target element identifier and the corresponding first target element attribute information, as well as the third target element identifier and the corresponding second target element position information are combined to obtain a complete element recognition result.

[0094] As mentioned above, by eliminating the second target element identifiers in at least one second target element identifier that satisfy the corresponding second target element position information that is consistent with the first target element position information to obtain the third target element identifier, duplicate user interface elements can be removed, which is beneficial to ensure the integrity of the element recognition results while reducing the amount of data and facilitating subsequent data transmission.

[0095] Figure 7 A flowchart of a user interface element identification method including a process of extracting resource files and code files is provided in an embodiment of the present application, such as Figure 7 As shown, the user interface element recognition method specifically includes the following steps:

[0096] Step S601: Obtain the installation package of the target application, determine the corresponding preset resource directory and preset code directory according to the current format of the installation package, extract resource files from the installation package based on the preset resource directory, and extract code files from the installation package based on the preset code directory.

[0097] For example, if the installation package is APK, the preset resource directories can be "res / " and "assets / ", and the preset code directory can be "apk / ". Correspondingly, resource files can include image files and layout files in the "res / " directory, as well as static resource files in the "assets / " directory. Code files can include the configuration file AndroidManifest.xml and the bytecode file classes.dex. Of course, the specific preset resource directories and preset code directories can be configured in other formats, which is not detailed in this application.

[0098] Step S602: Load the user interface corresponding to the target application, and obtain resource request information associated with the user interface during the loading process.

[0099] Step S603: parse the resource file, the code file, and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information.

[0100] Step S604: Capture the loaded user interface to obtain a user interface image, and identify the user interface image to obtain at least one second target element identifier and corresponding second target element position information.

[0101] Step S605: De-recombining and obtaining an element recognition result based on at least one first target element identifier and corresponding first target element attribute information, and at least one second target element identifier and corresponding second target element position information, and feeding back the element recognition result to the background server.

[0102] As described above, by determining the corresponding preset resource directory and preset code directory according to the current format of the installation package, it is possible to effectively be compatible with installation packages of different formats for file extraction and meet the user interface element recognition requirements of applications on different platforms.

[0103] Figure 8 A flowchart of a user interface element identification method including a process of generating a test script is provided in an embodiment of the present application, such as Figure 8 As shown, the user interface element recognition method specifically includes the following steps:

[0104] Step S701: Obtain the installation package of the target application, and extract the resource files and code files in the preset directory in the installation package.

[0105] Step S702: Load the user interface corresponding to the target application, and obtain resource request information associated with the user interface during the loading process.

[0106] Step S703: parse the resource file, the code file, and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information.

[0107] Step S704: Capture the loaded user interface to obtain a user interface image, and identify the user interface image to obtain at least one second target element identifier and corresponding second target element position information.

[0108] Step S705: De-recombining and obtaining an element recognition result based on at least one first target element identifier and corresponding first target element attribute information, and at least one second target element identifier and corresponding second target element position information, and feeding back the element recognition result to the background server.

[0109] Step S706: Obtain the target test element type and the test target description text, filter out at least one third target element identifier and corresponding element association information that matches the target test element type from the element recognition results, input the at least one third target element identifier and corresponding element association information, and the test target description text into the trained third network model, obtain the test script, and feed the test script back to the background server.

[0110] The target test element type can be a specific type of user interface element that needs to be tested in the user interface pre-set by the developer, such as a button, input box, etc. The test target description text can be a test step described in natural language, for example, "enter the user name on the login page and click the login button." The element association information can be a path, location, class name, etc. The third network model can be a GPT model, a UniXcoder model, etc., which is not limited in this application.

[0111] As described above, by inputting at least one third target element identifier and corresponding element association information, as well as the test target description text into the trained third network model, a test script can be quickly generated to provide developers with a reference test script that can meet the testing requirements of different element combinations.

[0112] Figure 9 This is a structural block diagram of a user interface element recognition device provided in an embodiment of the present application. The device is configured to execute the user interface element recognition method provided in the above embodiment and has the corresponding functional modules and beneficial effects of the execution method. Figure 9 As shown, the device specifically includes:

[0113] The file extraction module 201 is configured to obtain the installation package of the target application and extract the resource files and code files in the preset directory in the installation package;

[0114] The request information acquisition module 202 is configured to load the user interface corresponding to the target application and obtain resource request information associated with the user interface during the loading process;

[0115] A first element determination module 203 is configured to parse the resource file, the code file, and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information;

[0116] The second element determination module 204 is configured to capture the loaded user interface to obtain a user interface image, and identify the user interface image to obtain at least one second target element identifier and corresponding second target element position information;

[0117] The element integration feedback module 205 is configured to obtain an element recognition result based on at least one first target element identifier and corresponding first target element attribute information, and at least one second target element identifier and corresponding second target element position information, and feed back the element recognition result to the background server.

[0118] In the above, by obtaining the installation package of the target application, the resource files and code files in the preset directory in the installation package are extracted; the user interface corresponding to the target application is loaded, and the resource request information associated with the user interface during the loading process is obtained; the resource files, code files and resource request information are parsed to obtain at least one first target element identifier and corresponding first target element attribute information; the loaded user interface is captured to obtain a user interface image, and the user interface image is identified to obtain at least one second target element identifier and corresponding second target element position information; based on the at least one first target element identifier and the corresponding first target element attribute information, and the at least one second target element identifier and the corresponding second target element position information, decomposition and combination are performed to obtain an element recognition result, and the element recognition result is fed back to the background server. In the above scheme, by extracting the resource files and code files in the preset directory in the installation package, and obtaining the resource request information associated with the user interface during the loading process, diversified reference information can be provided for comprehensive and accurate identification of user interface elements; by parsing the resource files, code files and resource request information to obtain at least one first target element identifier and corresponding first target element attribute information, and identifying the user interface image to obtain at least one second target element identifier and corresponding second target element position information, user interface elements can be identified separately for data sources of different dimensions to ensure the accuracy and completeness of the element identification results; finally, by de-duplicating and combining the element identification results based on at least one first target element identifier and corresponding first target element attribute information, and at least one second target element identifier and corresponding second target element position information, the identification efficiency is significantly improved, and reliable element identification results are provided for subsequent generation of test scripts.

[0119] In a possible embodiment, the first element determination module 203 is further configured to:

[0120] Identify and integrate the resource file and the code file to obtain at least one first element identifier and corresponding first element attribute information;

[0121] Performing an associated search on the resource request information based on the preset element name to obtain at least one second element identifier and corresponding second element attribute information;

[0122] At least one first element identifier and corresponding first element attribute information, and at least one second element identifier and corresponding second element attribute information are demultiplexed and combined to obtain at least one first target element identifier and corresponding first target element attribute information.

[0123] In a possible embodiment, the first element determination module 203 is further configured to:

[0124] Extracting at least one first candidate element identifier and corresponding first candidate element attribute information from the resource file based on a preset regular expression;

[0125] Inputting the resource file and the code file into the trained first network model to obtain at least one second candidate element identifier and corresponding second candidate element attribute information;

[0126] At least one first candidate element identifier and corresponding first candidate element attribute information, and at least one second candidate element identifier and corresponding second candidate element attribute information are demultiplexed and combined to obtain at least one first element identifier and corresponding first element attribute information.

[0127] In a possible embodiment, the second element determination module 204 is further configured to:

[0128] Inputting the user interface image into the trained second network model to obtain at least one third candidate element identifier and corresponding first candidate element position information;

[0129] Performing local image recognition matching preset key characters on the user interface image to determine at least one fourth candidate element identifier and corresponding second candidate element position information;

[0130] At least one third candidate element identifier and corresponding first candidate element position information, and at least one fourth candidate element identifier and corresponding second candidate element position information are demultiplexed and combined to obtain at least one second target element identifier and corresponding second target element position information.

[0131] In a possible embodiment, the element integration feedback module 205 is further configured to:

[0132] Extracting first target element position information from first target element attribute information;

[0133] Eliminate the second target element identifiers whose corresponding second target element position information is consistent with the first target element position information from at least one second target element identifier to obtain a third target element identifier;

[0134] An element recognition result is obtained by combining at least one first target element identifier and corresponding first target element attribute information, and a third target element identifier and corresponding second target element position information.

[0135] In a possible embodiment, the file extraction module 201 is further configured to:

[0136] Determine the corresponding preset resource directory and preset code directory according to the current format of the installation package;

[0137] Extract resource files from the installation package based on the preset resource directory, and extract code files from the installation package based on the preset code directory.

[0138] In a possible embodiment, a script generation module is further included, configured to:

[0139] Get the target test element type and test target description text;

[0140] Filtering at least one third target element identifier and corresponding element association information that matches the target test element type from the element recognition results;

[0141] At least one third target element identifier and corresponding element association information, as well as a test target description text, are input into the trained third network model to obtain a test script, and the test script is fed back to the backend server.

[0142] Figure 10 A schematic diagram of a user interface element recognition device provided in an embodiment of the present application is shown in FIG. Figure 10 As shown, the device includes a processor 301, a memory 302, an input device 303 and an output device 304; the number of processors 301 in the device can be one or more. Figure 10 In the embodiment, a processor 301 is used as an example; the processor 301, the memory 302, the input device 303 and the output device 304 in the device can be connected by a bus or other means. Figure 10 The example of the connection via bus is taken. The memory 302, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the user interface element recognition method in the embodiment of the present application. The processor 301 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 302, that is, implements the above-mentioned user interface element recognition method. The input device 303 can be configured to receive input digital or character information, and generate key signal input related to the user settings and function control of the device. The output device 304 may include a display device such as a display screen.

[0143] The user interface element recognition device provided above can be used to execute the user interface element recognition method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0144] An embodiment of the present application also provides a non-volatile storage medium containing computer-executable instructions, which, when executed by a computer processor, is configured to execute a user interface element identification method described in the above embodiment, which includes: obtaining an installation package of a target application, extracting resource files and code files in a preset directory in the installation package; loading a user interface corresponding to the target application, and obtaining resource request information associated with the user interface during the loading process; parsing the resource files, code files, and resource request information to obtain at least one first target element identifier and corresponding first target element attribute information; taking a screenshot of the loaded user interface to obtain a user interface image, identifying the user interface image to obtain at least one second target element identifier and corresponding second target element position information; decomposing and combining based on at least one first target element identifier and corresponding first target element attribute information, and at least one second target element identifier and corresponding second target element position information to obtain an element identification result, and feeding back the element identification result to the background server.

[0145] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROMs, floppy disks, or tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media, optical storage; registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system that is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) that can be executed by one or more processors.

[0146] Of course, the storage medium containing computer-executable instructions provided in an embodiment of the present application is not limited to the above user interface element recognition method, and its computer-executable instructions can also execute related operations in the user interface element recognition method provided in any embodiment of the present application.

[0147] It should be noted that the numbering of each step in this solution is only used to describe the overall design framework of this solution, and does not represent the necessary order relationship between the steps. On the basis that the overall implementation process conforms to the overall design framework of this solution, it belongs to the protection scope of this solution, and the order of precedence in the form of text when describing is not an exclusive limitation on the specific implementation process of this solution. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (fl ash RAM). The memory is an example of a computer-readable medium.

[0148] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0149] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for identifying user interface elements, characterized in that: include: Obtain the installation package of the target application and extract the resource files and code files in the preset directory of the installation package; Loading a user interface corresponding to the target application, and obtaining resource request information associated with the user interface during the loading process; Parsing the resource file, the code file, and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information; Taking a screenshot of the loaded user interface to obtain a user interface image, and identifying the user interface image to obtain at least one second target element identifier and corresponding second target element position information; Based on the at least one first target element identifier and the corresponding first target element attribute information, and the at least one second target element identifier and the corresponding second target element position information, decomposition and combination are performed to obtain an element recognition result, and the element recognition result is fed back to the background server.

2. The user interface element recognition method according to claim 1, characterized in that: The parsing of the resource file, the code file, and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information includes: Identifying and integrating the resource file and the code file to obtain at least one first element identifier and corresponding first element attribute information; Performing an associated search on the resource request information based on the preset element name to obtain at least one second element identifier and corresponding second element attribute information; The at least one first element identifier and the corresponding first element attribute information, and the at least one second element identifier and the corresponding second element attribute information are demultiplexed and combined to obtain at least one first target element identifier and the corresponding first target element attribute information.

3. The user interface element recognition method according to claim 2, characterized in that: The identifying and integrating the resource file and the code file to obtain at least one first element identifier and corresponding first element attribute information includes: Extracting at least one first candidate element identifier and corresponding first candidate element attribute information from the resource file based on a preset regular expression; Inputting the resource file and the code file into the trained first network model to obtain at least one second candidate element identifier and corresponding second candidate element attribute information; The at least one first candidate element identifier and the corresponding first candidate element attribute information, and the at least one second candidate element identifier and the corresponding second candidate element attribute information are demultiplexed and combined to obtain at least one first element identifier and the corresponding first element attribute information.

4. The user interface element recognition method according to claim 1, characterized in that: The identifying the user interface image to obtain at least one second target element identifier and corresponding second target element position information includes: Inputting the user interface image into the trained second network model to obtain at least one third candidate element identifier and corresponding first candidate element position information; Performing local image recognition matching preset key characters on the user interface image to determine at least one fourth candidate element identifier and corresponding second candidate element position information; The at least one third candidate element identifier and the corresponding first candidate element position information, and the at least one fourth candidate element identifier and the corresponding second candidate element position information are demultiplexed and combined to obtain at least one second target element identifier and the corresponding second target element position information.

5. The user interface element recognition method according to claim 1, characterized in that: The de-duplication and combination based on the at least one first target element identifier and the corresponding first target element attribute information, and the at least one second target element identifier and the corresponding second target element position information to obtain the element recognition result includes: Extracting first target element position information from the first target element attribute information; Eliminate the second target element identifiers whose corresponding second target element position information is consistent with the first target element position information from the at least one second target element identifier to obtain a third target element identifier; The at least one first target element identifier and the corresponding first target element attribute information, and the third target element identifier and the corresponding second target element position information are combined to obtain an element recognition result.

6. The user interface element recognition method according to claim 1, characterized in that: The preset directory includes a preset resource directory and a preset code directory, and extracting the resource files and code files under the preset directory in the installation package includes: Determine the corresponding preset resource directory and preset code directory according to the current format of the installation package; Resource files are extracted from the installation package based on the preset resource directory, and code files are extracted from the installation package based on the preset code directory.

7. The user interface element recognition method according to claim 1, characterized in that: After feeding back the element recognition result to the backend server, the method further includes: Get the target test element type and test target description text; Filtering out at least one third target element identifier and corresponding element association information that matches the target test element type from the element recognition results; The at least one third target element identifier and the corresponding element association information, as well as the test target description text are input into the trained third network model to obtain a test script, and the test script is fed back to the backend server.

8. A user interface element recognition device, characterized in that: include: A file extraction module configured to obtain an installation package of a target application and extract resource files and code files in a preset directory in the installation package; a request information acquisition module configured to load the user interface corresponding to the target application and acquire resource request information associated with the user interface during the loading process; A first element determination module is configured to parse the resource file, the code file and the resource request information to obtain at least one first target element identifier and corresponding first target element attribute information; A second element determination module is configured to capture the loaded user interface to obtain a user interface image, and identify the user interface image to obtain at least one second target element identifier and corresponding second target element position information; The element integration feedback module is configured to obtain an element recognition result based on the decomposition and combination of the at least one first target element identifier and the corresponding first target element attribute information, and the at least one second target element identifier and the corresponding second target element position information, and to feed back the element recognition result to the background server.

9. A user interface element recognition device, comprising: one or more processors; The storage device is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the user interface element recognition method according to any one of claims 1 to 7.

10. A non-volatile storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are configured to perform the user interface element recognition method according to any one of claims 1 to 7.

Images