User interface processing method and device, medium and program product

Abstract

The embodiment of the invention discloses a user interface processing method and device, a medium and a program product, and the method comprises the steps: obtaining a test case of a first operating system; testing the target user interface by using the test case to obtain first element information of the target user interface; based on a framework corresponding to the second operating system, determining an element information table of the target user interface under the second operating system; performing element marking processing on a display page under the second operating system based on the target user interface to obtain a first element marking graph; and performing element positioning processing based on the first element information, the element information table and the element marking graph by utilizing an element positioning model to obtain an element positioning result. According to the embodiment of the invention, the test efficiency of different operating systems for the same user interface can be improved, and the manual writing cost of the test case can be reduced.

Description

Technical Field

[0001] The present disclosure relates to testing technology, and in particular to a method, device, medium and program product for processing a user interface. Background Art

[0002] User Interface (UI) automated testing is a means of operating the UI interface by simulating human behavior through code or tools, thereby verifying whether the system functions meet expectations.

[0003] At present, UI test cases are usually written to operate and locate user interface elements. Since different user interfaces use different operating systems, the operation and positioning methods of user interface elements are different. If UI test cases are written for the same function of different operating systems, multiple UI test cases need to be written to adapt to different operating systems. The writing efficiency of UI test cases is low and the cost is high.

[0004] Appium is an automation tool for mobile applications. It allows testers to use the same set of application programming interfaces (APIs) to write test scripts on different mobile operating systems (such as IOS, Android, or Hongmeng), which can greatly improve the reusability of code between different operating system test suites. Airtest is a test framework that uses image recognition technology to automate UI and supports multi-platform application testing. Through its rich cross-platform APIs, it simplifies a series of operations such as installing applications and simulating user input, greatly improving test efficiency and convenience.

[0005] In the related art, although the operating specifications of multiple operating systems can be unified using the Appium framework or the Airtest framework, the user interface elements between different operating systems cannot be converted, resulting in the need to write different UI test cases when locating elements for the user interface in different operating systems. Summary of the invention

[0006] The embodiments of the present disclosure provide a method, device, medium and program product for processing a user interface to solve the above problems.

[0007] According to a first aspect of an embodiment of the present disclosure, a method for positioning a user interface element is provided, comprising:

[0008] Acquire a test case of a first operating system, wherein the test case is generated based on a unified operation specification for a user interface of multiple preset operating systems, and the first operating system is one of the multiple preset operating systems;

[0009] Use the test case to test the target user interface to obtain the first element information of the target user interface;

[0010] Based on the framework corresponding to the second operating system, determine the element information table of the target user interface under the second operating system, where the second operating system is one of the multiple preset operating systems and is different from the first operating system;

[0011] Perform element marking processing on the display page of the target user interface under the second operating system to obtain a first element marking diagram including various element markings in the display page;

[0012] Use the element positioning model to perform element positioning processing based on the first element information, the element information table, and the element marking diagram to obtain the element positioning result of the target user interface under the second operating system, so as to perform user interface automation testing on the target user interface under the second operating system based on the element positioning result.

[0013] In some embodiments of the present disclosure, the page operations corresponding to the unified operation specification include at least one of page jump operation, page click operation, information input operation, page sliding operation, and element positioning method.

[0014] In some embodiments of the present disclosure, the using the test case to test the target user interface to obtain the first element information of the target user interface includes:

[0015] Use the element positioning method in the test case to perform element positioning processing on the target user interface, and obtain the element position information of the target user interface under the first operating system based on the element positioning result;

[0016] Use the screenshot function in the test case to obtain a screenshot of the target user interface under the first operating system;

[0017] Mark the screenshot based on the element position information to obtain a second element marking diagram, where the first element information includes the second element marking diagram.

[0018] In some embodiments of the present disclosure, the based on the framework corresponding to the second operating system, determining the element information table of the target user interface under the second operating system includes:

[0019] Use the preset function provided by the framework corresponding to the second operating system to obtain the second element information of the target user interface;

[0020] The element information table is generated based on the element attribute of each element in the second element information.

[0021] In some embodiments of the present disclosure, the using of the element positioning model to locate the element of the target user interface based on the first element information, the element information table and the element marking diagram to obtain the element positioning result of the target user interface under the second operating system includes:

[0022] Acquire a prompt word that is adapted to the element positioning model and corresponds to the target user interface;

[0023] The element positioning model is used to locate the element of the target user interface based on the first element information, the element information table, the element marking diagram and the prompt word, and the element positioning result of the target user interface under the second operating system is obtained.

[0024] According to a second aspect of an embodiment of the present disclosure, a user interface processing device is provided, including:

[0025] A test case acquisition module, configured to acquire a test case for a first operating system, wherein the test case is generated based on a unified operation specification for a user interface of a plurality of preset operating systems, and the first operating system is one of the plurality of preset operating systems;

[0026] A testing module, used to test the target user interface using the test case to obtain the first element information of the target user interface;

[0027] an element information table determining module, configured to determine an element information table of the target user interface under a second operating system based on a framework corresponding to the second operating system, wherein the second operating system is one of the plurality of preset operating systems and is different from the first operating system;

[0028] An element marking map acquisition module, used to perform element marking processing based on a display page of the target user interface under the second operating system, and obtain a first element marking map including the element markings in the display page;

[0029] An element positioning module is used to utilize an element positioning model to perform element positioning processing based on the first element information, the element information table and the element marking diagram, and obtain the element positioning result of the target user interface under the second operating system, so as to perform user interface automation testing on the target user interface under the second operating system based on the element positioning result.

[0030] In some embodiments of the present disclosure, the page operation corresponding to the unified operation specification includes at least one of a page jump operation, a page click operation, an information input operation, a page sliding operation, and an element positioning method.

[0031] In some embodiments of the present disclosure, the test module is used to perform element positioning processing on the target user interface using the element positioning method in the test case, and obtain element position information of the target user interface under the first operating system based on the element positioning result;

[0032] The test module is further configured to obtain a screenshot of the target user interface under the first operating system by using a screenshot function in the test case;

[0033] The testing module is further used to mark the screenshot based on the element position information to obtain a second element marking image, wherein the first element information includes the second element marking image.

[0034] In some embodiments of the present disclosure, the element information table determination module is used to obtain the second element information of the target user interface by using a preset function provided by the corresponding framework of the second operating system; the element information table determination module is also used to generate the element information table based on the element attributes of each element in the second element information.

[0035] In some embodiments of the present disclosure, the element positioning module is used to obtain a prompt word that is adapted to the element positioning model and corresponds to the target user interface; the element positioning module is also used to utilize the element positioning model to locate the element of the target user interface based on the first element information, the element information table, the element marking diagram and the prompt word, and obtain the element positioning result of the target user interface under the second operating system.

[0036] In some embodiments of the present disclosure, performing element marking processing on a display page based on the target user interface under the second operating system to obtain a first element marking diagram including element marks in the display page includes:

[0037] Acquire an image including each element to be located in the display page;

[0038] Perform element marking processing on each of the elements to be located in the image to obtain the first element marking map.

[0039] In some embodiments of the present disclosure, the target user interface includes a plurality of associated user interfaces for element positioning.

[0040] According to a third aspect of the present disclosure, an electronic device is provided, including:

[0041] A memory for storing a computer program product;

[0042] The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in the first aspect above is implemented.

[0043] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0044] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to execute the method described in the first aspect.

[0045] The user interface processing method, device, medium and program product of the disclosed embodiment can test the target user interface using the test case of the first operating system in multiple different operating systems, and obtain the first element information of the target user interface under the first operating system. The element information table of the target user interface under the second operating system is obtained by using the framework of the second operating system, and the display page of the target user interface under the second operating system is marked to obtain the first element marking diagram. Since the element structure of the same user interface under different operating systems is different, the display page is slightly different, but the element correspondence is the same, the element marking model can be used, based on the first element information of the target user interface under the first operating system, combined with the element information table and element marking diagram of the target user interface under the second operating system, so that the element positioning of the target user interface under the second operating system can be achieved without writing test cases for the second operating system, thereby not only improving the testing efficiency of different operating systems for the same user interface, but also reducing the cost of manual writing of test cases.

[0046] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0048] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0049] Figure 1 A flowchart of a method for processing a user interface in some embodiments of the present disclosure;

[0050] Figure 2is an element labeling diagram in the first element labeling diagram in an example of the present disclosure;

[0051] Figure 3 is a flowchart of step S2 in some embodiments of the present disclosure;

[0052] Figure 4 is a flowchart of step S3 in some embodiments of the present disclosure;

[0053] Figure 5 is a structural block diagram of a user interface processing device in some embodiments of the present disclosure;

[0054] Figure 6 It is a structural block diagram of an electronic device in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0055] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0056] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0057] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0058] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0059] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0060] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0061] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0062] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0063] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0064] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0065] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0066] Figure 1 FIG. 1 is a flow chart of a method for processing a user interface in some embodiments of the present disclosure. Figure 1 As shown, the method for processing the user interface includes the following steps:

[0067] S1: Obtain a test case of a first operating system.

[0068] Among them, the test cases are generated based on the unified operation specifications of multiple operating systems for the user interface (UI). Multiple operating systems are multiple preset operating systems that generate user interfaces, interact with users through user interfaces, and maintain user interfaces, for example, they may include Android operating system, Hongmeng operating system, Linux operating system, Unix operating system, etc. Unix operating systems may include MacOS and iOS operating systems.

[0069] The unified operation specification for the user interface is pre-set and is used to standardize the same operation under multiple preset operating systems. The same operation may include a click operation, a slide operation, and a shake operation.

[0070] Select a first operating system from multiple operating systems, for example, the first operating system can be an Android operating system, a Hongmeng operating system, a Linux operating system, or a Unix operating system.

[0071] The staff may write test cases applicable to the first operating system according to the unified operating specification, and then send the written test cases to the UI automated testing program so that the UI automated testing program performs UI testing.

[0072] It is also possible to call a pre-set test case template that is applicable to the first operating system, generate a test case applicable to the first operating system according to the interface parameters of the target user interface, and send the written test case to the UI automation test program so that the UI automation test program can perform UI testing. The interface parameters of the target user interface may include: the number of elements, element attributes, and element positions of the target user interface. Element attributes may include the type of each element in the target user interface, and the type of user operation that each element can accept. The element position may include the coordinate position of each element in the target user interface.

[0073] S2: Testing the target user interface using the test case to obtain first element information of the target user interface.

[0074] In some optional embodiments of the present disclosure, the target user interface is a plurality of associated user interfaces for which element positioning is to be performed. The plurality of user interfaces include at least two user interfaces with a page link relationship. For example, the plurality of user interfaces at least include a main interface after a user logs into a preset application, and a message interface with a page connection relationship with the main interface (i.e., the main interface has a link to the message interface, and the user can enter the message interface after clicking the link).

[0075] In an example of the present disclosure, the target user interface may include multiple user interfaces in a real estate agency service application for users to browse information related to a certain house. The user can enter the interface of the room support information by clicking a page link of "Room Support Information" in the house homepage in the multiple user interfaces.

[0076] Since the test case is written and generated using the programming language provided by the first operating system, the test case can be executed to perform automated testing on the target user interface. Based on the automated testing results, element information of the target user interface under the first operating system can be obtained, which is recorded as first element information.

[0077] The first element information may include the position information of each element of the target user interface in the first operating system, for example, it may include the pixel coordinates of the preset position (such as the center point, the upper left corner point or the lower left corner point) of each element of the target user interface in the first operating system, and the number of pixels corresponding to the width / height of each element in the target user interface. The first element information may also include text information corresponding to each element and icons of each element.

[0078] S3: Determine an element information table of the target user interface in the second operating system based on a framework corresponding to the second operating system, wherein the second operating system is one of a plurality of preset operating systems, and the second operating system is different from the first operating system.

[0079] A second operating system different from the first operating system is selected from multiple preset operating systems, and the framework corresponding to the second operating system is used to obtain the element information table of the target user interface for the second operating system. For example, when the first operating system is an Android operating system, the second operating system can be selected from a Hongmeng operating system, a Linux operating system, or a Unix operating system.

[0080] In an example of the present disclosure, when the second operating system is an Android operating system, the dump_hierarchy() function provided by the uiautomator2 framework can be used to obtain the element information table of the user interface under Android operation. Among them, the uiautomator2 framework is an open source Android automation testing framework, which provides a rich application programming interface (API) to facilitate testers to write automated test scripts, thereby obtaining the element information table of the target user interface for the Android operating system according to the written automated test scripts. The dump_hierarchy() function can be used to output the structure and hierarchical relationship of the program when the program is running. In this example, the element information of the target application program interface under Android operation can be output.

[0081] In another example of the present disclosure, when the second operating system is an iOS operating system, the source(format='json') function provided by the wda_client framework can be used to obtain the element information table of the user interface for the Android operating system. The wda_client framework is a test framework for the iOS operating system, which facilitates testers to write automated test scripts, thereby obtaining the element information table of the target user interface under the iOS operating system according to the source(format='json') function in the written automated test script.

[0082] In another example of the present disclosure, when the second operating system is Hongmeng operating system, the UiTree() function provided by the Hypium framework can be used to obtain the element information table of the user interface under Hongmeng operating system. Among them, the hypium framework is an automated testing framework based on Python for Hongmeng operating system, which is convenient for testers to write automated testing scripts, so as to obtain the element information table of the target user interface under Hongmeng operating system according to the UiTree() function in the written automated testing script.

[0083] S4: performing element marking processing on a display page of the target user interface under the second operating system to obtain a first element marking diagram including element marks in the display page.

[0084] In some optional embodiments of the present disclosure, step S4 may include: acquiring an image including each element to be located in the display page; marking each element to be located in the image to obtain a first element marking map.

[0085] For a client of the second operating system that logs in to the target user interface, a screenshot of the logged-in target user interface can be taken using the screenshot function, screenshot software, or a plug-in with a screenshot function provided by the second operating system to obtain an image including the elements to be located in the display page. Element marking processing is performed on the image to obtain an element marking map of the target user interface under the second operating system, which is recorded as the first element marking map.

[0086] The element marking process can use the cv2.rectangle() function in OpenCV to mark each element in the display page. OpenCV is a cross-platform computer vision and machine learning software library released under the Apache2.0 license (open source), which can run on Linux, Windows, Android and MacOS operating systems. The cv2.rectangle() function can be used to draw a rectangle on each element of the image, providing the coordinates of the upper left corner vertex of the rectangle, the coordinates of the lower right corner vertex of the rectangle, and the border color and width of the rectangle.

[0087] Figure 2 is an element labeling diagram in the first element labeling diagram in an example of the present disclosure. Figure 2 As shown, in the first element marking diagram, there are element markings of each element in the display page of the target user interface for the second operating system, including, for example: element markings of avatar elements, element markings of "My Focus" elements, element markings of "Browsing History" elements, ... , element markings of "My" elements.

[0088] S5: Utilize the element positioning model to perform element positioning processing based on the first element information, the element information table and the element marking diagram to obtain the element positioning result of the target user interface under the second operating system, so as to perform user interface automation testing on the target user interface under the second operating system based on the element positioning result.

[0089] Using the pre-trained element positioning model, the first element information, the element information table and the element marking diagram are processed for element positioning, and the element positioning result of the target user interface under the second operating system is output, wherein the element positioning result includes the position information of each element, such as the element coordinate value range.

[0090] The element positioning model can be trained based on element positioning samples and sample labels, wherein the element positioning samples include sample images with element labels of the sample user interface under the second operating system, a sample element information table of the sample user interface under the second operating system, and element information of the sample user interface under the first operating system.

[0091] After obtaining the element positioning result, the element position information of the target user interface that needs to be tested under the second operating system can be determined according to the element positioning result, and then the target user interface can be subjected to UI automation testing under the second operating system according to the element information that needs to be tested, so as to test whether the target user interface has achieved the expected function under the second operating system.

[0092] In this embodiment, the target user interface is tested using the test case of the first operating system among multiple different operating systems, and the first element information of the target user interface under the first operating system can be obtained. The element information table of the target user interface under the second operating system is obtained using the framework of the second operating system, and the display page of the target user interface under the second operating system is marked to obtain the first element marking diagram. Since the element structure of the same user interface under different operating systems is different, the display page is slightly different, but the element correspondence is the same, the element marking model can be used, based on the first element information of the target user interface under the first operating system, combined with the element information table and element marking diagram of the target user interface under the second operating system, so that the element positioning of the target user interface under the second operating system can be achieved without writing test cases for the second operating system, thereby not only improving the testing efficiency of different operating systems for the same user interface, but also reducing the cost of manual writing of test cases.

[0093] In some embodiments of the present disclosure, the page operation corresponding to the unified operation specification includes at least one of a page jump operation, a page click operation, an information input operation, a page sliding operation, and an element positioning method.

[0094] In an example of the present disclosure, unified specifications may be made for the following operations: go_to_schema (jump to a specified page), click, send_keys, swipe, find_element_by_xpath (locate an element by xpath), and find_element_by_text (locate an element by text).

[0095] In this embodiment, since page jump operations, page click operations, information input operations, page sliding operations and element positioning methods include commonly used user interface operations, setting unified operation specifications for page jump operations, page click operations, information input operations, page sliding operations and element positioning methods helps to improve the element positioning efficiency and automated testing efficiency of multiple preset operating systems for the same user interface.

[0096] Figure 3 FIG. 1 is a flow chart of step S2 in some embodiments of the present disclosure. Figure 3 As shown, step S2 may include the following steps:

[0097] S2-1: Perform element positioning processing on the target user interface using the element positioning method in the test case, and obtain element position information of the target user interface in the first operating system based on the element positioning result.

[0098] After executing the test case for the target user interface, the element positioning method set in the test case is used to perform element positioning processing on the target user interface, and the element position information of the target user interface under the first operating system can be obtained. Among them, the preset element positioning method can include an xpath positioning method and a text positioning method.

[0099] In an example of the present disclosure, when the first operating system is an Android operating system, the XPath positioning function device.xpath(xpath) provided by the Android operating system can be used to perform XPath positioning on the elements of the target user interface under the Android operating system, or the text positioning method of the Android operating system can be used to perform text positioning on the elements of the target user interface under the Android operating system. For example, the following programming language can be used for text positioning: device.xpath(f" / / *[@text='{text}'or@content-desc='{text}'or@resource-id='{text}']").

[0100] In another example of the present disclosure, when the first operating system is an iOS operating system, the xpath positioning function device.wda_client.xpath(xpath) provided by the iOS operating system can be used to perform xpath positioning on the elements of the target user interface under the iOS operating system, or the text positioning method of the iOS operating system can be used to perform text positioning on the elements of the target user interface under the iOS operating system. For example, text positioning can be performed using the following programming language: device.wda_client.xpath(f" / / *[@label='{text}'or@value='{text}']").

[0101] In another example of the present disclosure, when the first operating system is the HarmonyOS, the xpath positioning function UiTree() provided by the Hypium framework of the HarmonyOS can be used to perform xpath positioning on the elements of the target user interface under the HarmonyOS, or the text positioning function find_component(BY.text(text)) of the HarmonyOS can be used to perform text positioning on the elements of the target user interface under the HarmonyOS.

[0102] S2-2: Using the screenshot function in the test case, obtain a screenshot of the target user interface in the first operating system.

[0103] In an example of the present disclosure, when the first operating system is an Android operating system, a screenshot function of the uiautomator2 framework may be used to obtain a screenshot of the target user interface under the Android operating system.

[0104] In another example of the present disclosure, when the first operating system is an iOS operating system, a screenshot function of a wda framework may be used to obtain a screenshot of a target user interface under the iOS operating system.

[0105] In another example of the present disclosure, when the first operating system is the Hongmeng operating system, the screenshot function of the Hypium framework can be used to obtain a screenshot of the target user interface under the Hongmeng operating system.

[0106] S2-3: Marking the screenshot based on the element position information to obtain a second element marking diagram, wherein the first element information includes the second element marking diagram, wherein the first element information includes the second element marking diagram.

[0107] According to the element position information of the target user interface under the first operating system, the enclosing box and the number of each element can be generated in the screenshot of the target user interface under the first operating system to obtain the second element marking diagram (such as Figure 2 as shown

[0108] In this embodiment, by using the element positioning method and screenshot function in the test case, the element position information and screenshot of the target user interface under the first operating system can be obtained correspondingly. Furthermore, a second element marking map representing the element positions of the target user interface under the first operating system and capable of distinguishing different elements can be generated. Since the element objects of the target user interface are the same under the first operating system and the second operating system but the element positions may be different, the second element marking map helps to improve the element positioning accuracy of the target user interface under the second operating system.

[0109] Figure 4 is a flowchart of step S3 in some embodiments of the present disclosure. As Figure 4 shown, step S3 may include the following steps:

[0110] S3-1: Use the preset function provided by the corresponding framework of the second operating system to obtain the second element information of the target user interface.

[0111] The method of extracting element information from the user interface is encapsulated into the device.hierarchy_to_element() function. Under the hood, it actually calls each operating system to obtain the element information of the target user interface under the second operating system, denoted as the second element information. For example, when the second operating system is the Android operating system, the second element information can be obtained based on the dump_hierarchy() function provided by the uiautomator2 framework; when the second operating system is the iOS operating system, the second element information can be obtained based on the source(format='json') function provided by the wda_client framework; when the second operating system is the HarmonyOS operating system, the second element information can be obtained based on the UiTree() function provided by the hypium framework.

[0112] After the device.hierarchy_to_element() function is encapsulated, the encapsulated device.hierarchy_to_element() function can be used to extract the second element information.

[0113] S3-2: Generate an element information table based on the element attributes of each element in the second element information.

[0114] Since the page elements of the first operating system and the second operating system are inconsistent, the element attributes of all elements in the target user interface can be obtained by traversing the first element information and the second element information, and then an element information table can be generated.

[0115] In one example of the present disclosure, the structure of the element information table is as follows:

[0116] 1:{"text":"","xpath":"","clickable":false,"inputable":false,"scrollable":false,"x":0,"y":0,"width":0,"height":0};

[0117] 2:{"text":"","xpath":"","clickable":false,"inputable":false,"scrollable":false,"x":0,"y":0,"width":0,"height":0};

[0118] 3:{"text":"","xpath":"","clickable":false,"inputable":false,"scrollable":false,"x":0,"y":0,"width":0,"height":0}.

[0119] Among them, "1", "2" and "3" represent the element numbers 1, 2 and 3 respectively; text represents the text of the element (in the Android operating system, the text attribute and the context-desc attribute are unified into the text attribute); xpath represents the positioning method of the element; clickable indicates whether the element is clickable; inputable indicates whether the element is an input box; scrollable represents whether the element is scrollable; x, y, width, height represent the x, y coordinates and width and height of the element respectively.

[0120] In a specific example of the present disclosure, Figure 2The avatar element in the Hongmeng operating system is: 8:{"text":"","xpath":"","clickable":true,"inputable":false,"scrollable":false,"x":1014,"y":311,"width":163,"height":163}, which means that the element number of the avatar element is 8, the text of the avatar element is empty, the xpath positioning method of the avatar element is empty, the avatar element is clickable, the avatar element is not an input box, the avatar element cannot be slid, the x,y coordinates of the avatar element (upper left corner) are (1014, 311), and the width and height of the avatar element are (163, 163) respectively. The avatar element is 8 under the Android operating system: {"text":"avatar","xpath":"","clickable":true,"inputable":false,"scrollable":false,"x":1014,"y":311,"width":160,"height":160}, which means that the element number of the avatar element is 8, the text of the avatar element is "avatar", the xpath positioning method of the avatar element is empty, the avatar element is clickable, the avatar element is not an input box, the avatar element cannot be slid, the x,y coordinates of the avatar element (upper left corner) are (1014, 311), and the width and height of the avatar element are (160, 160) respectively.

[0121] In this embodiment, the preset function provided by the corresponding framework of the second operating system can be used to quickly and accurately obtain the second element information of the target user interface, and then generate an element information table of the target user interface under the second operating system. Since the element information table can contain element information such as the location information and attribute information of each element, it helps to improve the element positioning accuracy of the target user interface under the second operating system.

[0122] In some embodiments of the present disclosure, step S5 may include: obtaining a prompt word that is adapted to the element positioning model and corresponds to the target user interface; using the element positioning model, locating the element of the target user interface based on the first element information, the element information table, the element marking diagram and the prompt word, and obtaining the element positioning result of the target user interface under the second operating system.

[0123] You can use an element location model that uses prompts to mark elements. For example, you can choose GPT-4o as an element location model. GPT-4o is a language model released by OpenAI for the chatbot ChatGPT, which can perform real-time reasoning on audio, vision, and text. The prompt is the prompt information input to the element location model, which can specifically include the number and type of each element that the element location model needs to locate. For example, for Figure 2 The prompt words of the page shown may include: (34, picture) and (45, picture). The element positioning model performs element positioning on the image numbered 34 and the image numbered 45 based on the above prompt words, and correspondingly outputs the position information of the image numbered 34 and the position information of the image numbered 45. Providing prompt words to the element positioning model helps to distinguish elements of the same type, thereby helping to improve the accuracy of element positioning.

[0124] In addition, to avoid exceeding the token limit of the element location model, the width of the element marker map can be adjusted to a preset value, for example, the width resolution of the element marker map can be adjusted to 320. Token limit means that the element location model can only process a limited number of tokens when processing text due to limitations in computing resources and model design. A token is the basic unit in natural language processing and can be a letter, a word, a number, etc.

[0125] By using the GPT-4o model to process the first element information, the element information table, the element marking diagram and the prompt word, a high-precision element positioning result of the target user interface under the second operating system can be obtained.

[0126] In this embodiment, since the prompt word can indicate the type of each element of the target user interface under the second operating system, the first element information represents the element information of the target user interface under the first operating system, the element information table has the information of each element of the target user interface under the second operating system, and the element marking diagram helps to distinguish the elements of the target user interface under the second operating system, the first element information, the element information table, the element marking diagram and the prompt word are processed using the element positioning model to obtain a high-precision element positioning result of the target user interface under the second operating system.

[0127] Figure 5 FIG. 1 is a structural block diagram of a user interface processing device in some embodiments of the present disclosure. Figure 5 As shown, the positioning device of the user interface element includes:

[0128] A test case acquisition module 100 is used to acquire a test case of a first operating system, wherein the test case is based on a unified operation specification of multiple operating systems for a user interface, and the first operating system is one of multiple preset operating systems;

[0129] The testing module 200 is used to test the target user interface using the test case to obtain the first element information of the target user interface;

[0130] An element information table determination module 300 is used to determine an element information table of a target user interface under a second operating system based on a framework corresponding to the second operating system, wherein the second operating system is one of a plurality of preset operating systems and the second operating system is different from the first operating system;

[0131] An element marking map acquisition module 400 is used to perform element marking processing on a display page of a target user interface under a second operating system to obtain a first element marking map including element markings on the display page;

[0132] The element positioning module 500 is used to utilize the element positioning model to perform element positioning processing based on the first element information, the element information table and the element marking diagram, and obtain the element positioning result of the target user interface under the second operating system, so as to perform user interface automation testing on the target user interface under the second operating system based on the element positioning result.

[0133] In some embodiments of the present disclosure, the page operation corresponding to the unified operation specification includes at least one of a page jump operation, a page click operation, an information input operation, a page sliding operation, and an element positioning method.

[0134] In some embodiments of the present disclosure, the test module 100 is used to perform element positioning processing on the target user interface using the element positioning method in the test case, and obtain element position information of the target user interface in the first operating system based on the element positioning result;

[0135] The test module 100 is further used to obtain a screenshot of the target user interface in the first operating system using a screenshot function in the test case;

[0136] The testing module 100 is further used to mark the screenshot based on the element position information to obtain a second element marking diagram, wherein the first element information includes the second element marking diagram.

[0137] In some embodiments of the present disclosure, the element information table determination module 300 is used to obtain the second element information of the target user interface by using a preset function provided by the corresponding framework of the second operating system; the element information table determination module 300 is also used to generate an element information table based on the element attributes of each element in the second element information.

[0138] In some embodiments of the present disclosure, an element positioning module 500 is configured to obtain a prompt word adapted to an element positioning model and corresponding to a target user interface; the element positioning module 500 is further configured to use the element positioning model to perform element positioning on the target user interface based on first element information, an element information table, an element marking map, and the prompt word, so as to obtain an element positioning result of the target user interface under a second operating system.

[0139] It should be noted that the specific implementation manner of the user interface processing device in the embodiments of the present disclosure is similar to the specific implementation manner of the user interface processing method in the embodiments of the present disclosure, and the technical effects of the user interface processing device in the embodiments of the present disclosure are similar to the technical effects of the user interface processing method in the embodiments of the present disclosure. For details, refer to the description in the part of the user interface processing method. To avoid redundancy, no further elaboration is provided.

[0140] In addition, embodiments of the present disclosure further provide an electronic device, including:

[0141] A memory for storing a computer program;

[0142] A processor for executing the computer program stored in the memory, and when the computer program is executed, implementing the user interface processing method described in any one of the above embodiments of the present disclosure.

[0143] Next, refer to Figure 6 to describe the electronic device according to the embodiments of the present disclosure. As Figure 6 shown, the electronic device includes one or more processors and a memory.

[0144] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0145] The memory may store one or more computer program products. The memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products may be stored on the computer-readable storage media, and the processor may run the computer program products to implement the user interface processing methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0146] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0147] In addition, the input device may further include, for example, a keyboard, a mouse, and so on.

[0148] The output device may output various information to the outside, including the determined distance information, direction information, and so on. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, and so on.

[0149] Of course, for simplicity, Figure 6 only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, and so on are omitted. In addition, according to specific application scenarios, the electronic device may further include any other appropriate components.

[0150] In addition to the above methods and devices, embodiments of the present disclosure may also be a computer program product, which includes computer program instructions. When the computer program instructions are run by a processor, the processor is caused to execute the steps in the processing method of the user interface according to various embodiments of the present disclosure described in the above part of this specification.

[0151] The computer program product may be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, executed as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0152] In addition, embodiments of the present disclosure may also be a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are run by a processor, the processor is caused to execute the steps in the processing method of the user interface according to various embodiments of the present disclosure described in the above part of this specification.

[0153] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0154] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0155] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0156] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0157] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0158] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0159] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0160] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for processing a user interface, characterized in that: include: Acquire a test case of a first operating system, wherein the test case is generated based on a unified operation specification for a user interface of multiple operating systems, and the first operating system is one of the multiple preset operating systems; Testing the target user interface using the test case to obtain first element information of the target user interface; Determining, based on a framework corresponding to a second operating system, an element information table of the target user interface under the second operating system, wherein the second operating system is one of the plurality of preset operating systems, and the second operating system is different from the first operating system; Performing element marking processing on a display page of the target user interface under the second operating system to obtain a first element marking diagram including element markings in the display page; Utilizing an element positioning model, element positioning processing is performed based on the first element information, the element information table and the element marking diagram to obtain an element positioning result of the target user interface under the second operating system, so as to perform automated user interface testing on the target user interface under the second operating system based on the element positioning result.

2. The method according to claim 1, characterized in that: The page operation corresponding to the unified operation specification includes at least one of a page jump operation, a page click operation, an information input operation, a page sliding operation and an element positioning method.

3. The method according to claim 1, characterized in that The step of testing the target user interface by using the test case to obtain the first element information of the target user interface includes: Performing element positioning processing on the target user interface by using the element positioning method in the test case, and acquiring element position information of the target user interface under the first operating system based on the element positioning result; Using the screenshot function in the test case, obtaining a screenshot of the target user interface under the first operating system; The screenshot is marked based on the element position information to obtain a second element marking image, wherein the first element information includes the second element marking image.

4. The method according to claim 1, characterized in that: The determining, based on the framework corresponding to the second operating system, the element information table of the target user interface under the second operating system includes: Using a preset function provided by the corresponding framework of the second operating system, obtaining second element information of the target user interface; The element information table is generated based on the element attribute of each element in the second element information.

5. The method according to any one of claims 1 to 4, characterized in that: The using the element positioning model to locate the element of the target user interface based on the first element information, the element information table and the element marking diagram to obtain the element positioning result of the target user interface under the second operating system includes: Acquire a prompt word that is adapted to the element positioning model and corresponds to the target user interface; The element positioning model is used to locate the element of the target user interface based on the first element information, the element information table, the element marking diagram and the prompt word, and the element positioning result of the target user interface under the second operating system is obtained.

6. The method according to any one of claims 1 to 4, characterized in that: The performing element marking processing on the display page based on the target user interface under the second operating system to obtain a first element marking diagram including element marks in the display page includes: Acquire an image including each element to be located in the display page; Perform element marking processing on each of the elements to be located in the image to obtain the first element marking map.

7. The method according to any one of claims 1 to 4, characterized in that: The target user interface includes a plurality of associated user interfaces for element positioning.

8. An electronic device, characterized in that: include: A memory for storing a computer program product; A processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements the method described in any one of claims 1 to 5.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 5 is implemented.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to execute the method according to any one of claims 1 to 5.

Images