Collaborative testing method and device for graphical user interface, electronic equipment and product

Abstract

The invention discloses a collaborative testing method and device for a graphical user interface, electronic equipment and a product, and the method comprises the steps: obtaining a control tree structure and an image recognition model of the graphical user interface of an application program, and transmitting the control tree structure and the image recognition model to all agents of collaborative testing; decomposing a test task into a plurality of sub-tasks, and sending all the sub-tasks to an intelligent agent for collaborative test; and according to the received subtasks, the intelligent agent performs test operation on the graphical user interface by utilizing a control tree structure or an image recognition model, returns test results and summarizes to generate collaborative test results of the graphical user interface. Through cooperation of the control tree structure and the image recognition model, the accuracy of graphical user interface control recognition is improved, the test task can be decomposed into a plurality of sub-tasks, multi-agent distributed collaborative testing is achieved, and the method has the advantages of being high in test efficiency and good in test quality and is particularly suitable for large-scale test scenes.

Description

Technical Field

[0001] The present invention belongs to the field of automatic testing of graphical user interfaces, and in particular relates to a collaborative testing method, device, electronic equipment and product of graphical user interfaces. Background Art

[0002] With the rapid development of domestic Linux operating systems and the widespread popularity of their applications, the need for automated testing of applications running on them has become increasingly urgent. Under existing technical conditions, traditional automated testing methods for graphical user interfaces have limited testing effects when dealing with complex and changeable graphical user interfaces (GUIs), especially in terms of accurately identifying and locating graphical user interface (GUI) graphic elements. Especially in distributed testing environments, there are problems such as low efficiency of automated testing of graphical user interfaces and inaccurate testing results. Summary of the invention

[0003] In view of this, the present invention aims to overcome the defects in the prior art and proposes a collaborative testing method, device, electronic device and product of a graphical user interface.

[0004] To achieve the above object, the technical solution of the present invention is achieved as follows: In a first aspect, the present invention discloses a collaborative testing method for a graphical user interface, comprising: Get the control tree structure of the application's graphical user interface and send it to all agents in the collaborative test; Based on the element graphics of the graphical user interface, an image recognition model is established and sent to all agents in the collaborative test; Get the test task, decompose it into several subtasks, and send all subtasks to the collaborative testing agents respectively; The agent performs test operations on the graphical user interface based on the received subtasks and uses the control tree structure or image recognition model, and returns the test results; Generate collaborative test results for the graphical user interface based on the test results returned by all intelligent agents in the collaborative test.

[0005] In another embodiment of the present invention, obtaining a control tree structure of a graphical user interface of an application includes: obtaining the control tree structure using an API interface provided by an operating system or a third-party library; the control tree structure includes: graphical user interface elements, element types, hierarchical relationships, and attribute information.

[0006] In another embodiment of the present invention, an image recognition model is established based on element graphics of a graphical user interface, including: obtaining the element graphics as training data, training a neural network model, and obtaining an image recognition model that can recognize the element graphics.

[0007] In another embodiment of the present invention, a test task is obtained, decomposed into several subtasks, and all subtasks are sent to intelligent agents of collaborative testing respectively, including: according to a load balancing algorithm, several subtasks are sent to several intelligent agents of collaborative testing in a balanced manner.

[0008] In another embodiment of the present invention, obtaining an element graph as training data includes: extracting a feature vector of the element graph as training data.

[0009] In another embodiment of the present invention, the control tree structure of the graphical user interface of the application is obtained and sent to all agents of the collaborative test, including: the control tree structure is sent to all agents of the collaborative test using XML format or JSON format.

[0010] In another embodiment of the present invention, extracting a feature vector of the element graph as training data includes: extracting the feature vector of the element graph using an OpenCV computer vision library.

[0011] In a second aspect, the present invention discloses a collaborative testing device for a graphical user interface, the device comprising: A control tree structure sending module obtains the control tree structure of the graphical user interface of the application and sends it to all agents in the collaborative test; An image recognition model sending module, which builds an image recognition model based on the element graphics of the graphical user interface and sends it to all intelligent agents in the collaborative test; The subtask sending module obtains the test task, decomposes the test task into several subtasks, and sends all subtasks to the collaborative test agents respectively; The first result generation module, the agent performs the test operation on the graphical user interface according to the received subtask, using the control tree structure or the image recognition model, and returns the test result; The second result generating module generates the collaborative test result for the graphical user interface according to the test results returned by all the intelligent agents of the collaborative test.

[0012] In a third aspect, the present invention discloses an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0013] In a fourth aspect, the present invention discloses a computer program product, including a computer program, which implements the above method when executed by a processor.

[0014] Compared with the prior art, the present invention has the following advantages: The present invention discloses a collaborative testing method, device, electronic device and product of a graphical user interface, including: obtaining a control tree structure of a graphical user interface of an application, and sending it to all agents of the collaborative test; establishing an image recognition model based on element graphics of the graphical user interface, and sending it to all agents of the collaborative test; obtaining a test task, decomposing the test task into a plurality of subtasks, and sending all subtasks to the agents of the collaborative test respectively; the agent performs a test operation on the graphical user interface using a control tree structure or an image recognition model according to the received subtasks, and returns a test result; and generating a collaborative test result of the graphical user interface according to the test results returned by all agents of the collaborative test. The present invention discloses a collaborative testing method, device, electronic device and product of a graphical user interface, which improves the accuracy of identifying controls of the graphical user interface, improves the quality and efficiency of testing the graphical user interface, and can decompose the test task into a plurality of subtasks to realize distributed collaborative testing of multiple agents, has the characteristics of high test efficiency and good test quality, and is particularly suitable for large-scale test scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0016] In the attached picture: Figure 1 A schematic diagram of an application scenario of a collaborative testing method for a graphical user interface according to an embodiment of the present invention; Figure 2 A schematic diagram of a collaborative testing method for a graphical user interface according to an embodiment of the present invention; Figure 3 A task decomposition diagram of a collaborative testing method for a graphical user interface according to an embodiment of the present invention; Figure 4 A schematic diagram of a collaborative testing device of a graphical user interface according to an embodiment of the present invention; Figure 5 A schematic diagram of an electronic device for collaborative testing of a graphical user interface according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0018] In the description of the present invention, it should be further explained that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0019] The present invention discloses a collaborative testing method, device, electronic device and product application scenarios of a graphical user interface. Figure 1 As shown, when testing a graphical user interface, in order to improve the test efficiency and test effect, multiple agents are used to collaboratively test. The collaborative testing method, device, electronic device and product of a graphical user interface disclosed in the present invention improve the accuracy of identifying graphical user interface controls and the quality and efficiency of graphical user interface testing through the coordination of a control tree structure and an image recognition model, and can decompose the test task into several subtasks to realize collaborative testing of multiple agents, which has the characteristics of high test efficiency and good test quality, and is particularly suitable for large-scale test scenarios.

[0020] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0021] In one embodiment of the present invention, Figure 2 As shown, a collaborative testing method for a graphical user interface includes: Step 201, obtaining the control tree structure of the graphical user interface of the application and sending it to all agents in the collaborative test; In this embodiment, it includes: a main control agent and several execution agents. The main control agent is used to obtain the control tree structure of the graphical user interface of the application and send it to the several execution agents.

[0022] The execution agents are deployed on different physical or virtual machines and receive subtasks sent by the master agent. The execution agents are independent of each other and can execute test tasks in parallel, which significantly improves the test efficiency. Even if an execution agent fails, it will not affect the normal operation of other agents, ensuring the stability and reliability of the test process. This distributed agent testing method can fully utilize the advantages of distributed computing and improve test efficiency.

[0023] In this embodiment, the control tree structure is obtained by using an API interface provided by the operating system or a third-party library; the control tree structure includes: graphical user interface elements, element types, hierarchical relationships, and attribute information.

[0024] Exemplarily, the graphical user interface elements include, for example, buttons, text boxes, drop-down lists, and the like.

[0025] Exemplarily, the pyatspi tool is used in the Linux system to obtain the control tree structure, for example, information such as the ID, type, and location of elements such as a menu bar, a toolbar, a text editing area, and a dialog box.

[0026] The control tree structure is sent to all agents in the collaborative test using XML or JSON format.

[0027] Step 202, based on the element graphics of the graphical user interface, an image recognition model is established and sent to all agents in the collaborative test; In this embodiment, the master agent is used to build an image recognition model and send it to several execution agents.

[0028] The master agent communicates with several execution agents through the cross-platform gRPC framework as the underlying communication protocol, and uses TLS / SSL technology to encrypt communication data.

[0029] In this embodiment, the master intelligent agent communicates with several execution intelligent agents, supports real-time data transmission and remote control instruction issuance, and has good fault tolerance and resistance to network fluctuations.

[0030] In this embodiment, element graphics are obtained as training data, and a neural network model is trained to obtain an image recognition model that can recognize element graphics.

[0031] Exemplarily, the image recognition model can be a convolutional neural network (CNN) model.

[0032] In this embodiment, obtaining the element graph as training data includes: extracting a feature vector of the element graph as training data.

[0033] For some GUI elements, such as custom controls, dynamic loading, etc., the execution agent identifies them through the image recognition model. The execution agent can find the GUI elements that need to be tested in the subtask through the image recognition model.

[0034] In this embodiment, the control tree structure is used to crop the area where the target element graphic is located from the entire screenshot, thereby narrowing the image recognition range and improving the recognition speed and accuracy of the image recognition model.

[0035] Step 203, obtaining a test task, decomposing the test task into several subtasks, and sending all subtasks to the collaborative test agents respectively; In this embodiment, according to the load balancing algorithm, Figure 3As shown, several subtasks are evenly sent to several execution agents of collaborative testing.

[0036] In this embodiment, in the prior art, a centralized test task is decomposed into multiple subtasks that can be executed in parallel, and the entire test process is completed through the collaborative work of multiple intelligent agents, which effectively improves the test speed.

[0037] Step 204, the agent performs a test operation on the graphical user interface based on the received subtask using the control tree structure or the image recognition model, and returns the test result; In this embodiment, exemplarily, the executing agent can quickly locate the target element in the control tree structure according to the logical description of the received subtask, and use the API provided by the operating system or simulate user operations to achieve precise control of the target element, such as clicking a button, entering text, selecting a list item, etc.

[0038] Since the operation is based on the control tree structure rather than coordinates, it is not affected by environmental changes and has better stability and reliability.

[0039] In this embodiment, the comprehensive application of the control tree structure and the image recognition model improves the accuracy and robustness of element recognition in the graphical user interface, and especially solves the recognition difficulties in complex scenarios such as dynamic interfaces and custom controls.

[0040] Exemplarily, the test operation may be clicking a button, inputting text, selecting a list item, and the like.

[0041] Exemplarily, the test result may be whether the subtask is executed successfully.

[0042] Step 205: Generate collaborative test results for the graphical user interface based on the test results returned by all agents in the collaborative test.

[0043] Exemplarily, the collaborative test result may be a summary of the test results.

[0044] Based on the previous embodiment, in another embodiment of the present invention, extracting a feature vector of an element graph as training data includes: extracting the feature vector of the element graph using an OpenCV computer vision library.

[0045] In another embodiment of the present invention, as follows: Remote collaborative testing of the graphical user interface of WPS Office in the Linux system desktop environment: The master agent obtains the following test tasks: 1. Open an existing document ("Test Document.docx").

[0046] 2. Insert a text at the end of the document: "This is the text inserted by the automated test.".

[0047] 3. Set the inserted text to bold, red, and size 14 font.

[0048] 4. Save the document.

[0049] The master agent decomposes the test task into several subtasks and sends them to the execution agent 1, denoted as Worker Agent 1; the execution agent 2, denoted as Worker Agent 1; 1. Worker Agent 1: Open "test document.docx" (according to the file path).

[0050] 2. Worker Agent 1: Position to the end of the document (may require the use of key combinations or specific APIs).

[0051] 3. Worker Agent 1: Enter the text "This is the text inserted by the automated test.".

[0052] 4. Worker Agent 2: Click the “Start” menu.

[0053] 5. Worker Agent 2: In the toolbar of the Start menu, locate and click the "Bold" button (use the control tree first, and use image recognition if it cannot be located).

[0054] 6. Worker Agent 2: Locate and click the "Font Color" selector (use the control tree first, if it cannot be located, use image recognition), and select "Red" in the color selector.

[0055] 7. Worker Agent 2: Locate and open the "Font Size" drop-down list (use the control tree first, if it cannot be located, use image recognition), and select "14".

[0056] 8. Worker Agent 1: Save the document (using shortcut keys or menu operations).

[0057] 9. Worker Agent 1: Close the document.

[0058] In this embodiment, tasks related to document content editing are assigned to Worker Agent 1, and tasks related to formatting are assigned to Worker Agent 2, simulating a multi-person collaboration scenario.

[0059] Further, the execution agent performs the assigned subtasks and returns the test results to the master agent; The master agent summarizes the execution results of all subtasks and determines whether the test has passed based on preset standards.

[0060] In this embodiment, if a certain execution agent fails, the master control agent identifies the failure through a heartbeat detection mechanism and reallocates the corresponding subtasks to other available execution agents.

[0061] In this embodiment, the master agent can dynamically adjust task allocation according to the preset scheduling strategy and the real-time status information of the execution agent, make full use of system resources, and avoid resource idleness or load imbalance.

[0062] like Figure 4 As shown, the present invention also discloses a collaborative testing device of a graphical user interface, comprising: A control tree structure sending module 401 obtains the control tree structure of the graphical user interface of the application and sends it to all agents in the collaborative test; An image recognition model sending module 402 is used to establish an image recognition model based on the element graphics of the graphical user interface and send it to all agents in the collaborative test; The subtask sending module 403 obtains the test task, decomposes the test task into several subtasks, and sends all the subtasks to the intelligent agents of the collaborative test respectively; The first result generation module 404, the agent performs a test operation on the graphical user interface based on the received subtask using the control tree structure or the image recognition model, and returns the test result; The second result generating module 405 generates the collaborative test result for the graphical user interface according to the test results returned by all the intelligent agents of the collaborative test.

[0063] The present invention also discloses an electronic device, such as Figure 5 As shown, an embodiment is disclosed, which is a block diagram of an electronic device applicable to the method of collaborative testing of the above-mentioned graphical user interface.

[0064] The electronic device 50 of this embodiment includes a processor 501, which can perform various appropriate actions and processes according to the program stored in the ROM 502 or the program loaded from the storage part 508 to the RAM 503. The processor 501 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a dedicated microprocessor, etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present invention.

[0065] In RAM503, various programs and data required for the operation of electronic device 50 are stored. Processor 501, ROM502 and RAM503 are connected to each other via bus 504, and processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM502 and / or RAM503. It should be noted that the program can also be stored in one or more memories other than ROM502 and RAM503, and processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in one or more memories.

[0066] According to an embodiment of the present invention, the electronic device 50 may further include an I / O interface 505, which is also connected to the bus 504. The electronic device 50 may further include one or more of the following components connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube, a liquid crystal display, and a speaker; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 5010 is also connected to the I / O interface 505 as needed. A removable medium 5011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 5010 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0067] The present invention also provides a computer-readable storage medium.

[0068] The computer-readable storage medium may be included in the electronic device / device system described in the above embodiment; or it may exist independently without being assembled into the electronic device / device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0069] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory RAM, a read-only memory ROM, an erasable programmable read-only memory EPROM or a flash memory, a portable compact disk read-only memory CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0070] Embodiments of the present invention also include a computer program product.

[0071] The computer program product includes a computer program, which contains program codes for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program codes are used to enable the electronic device to implement the method provided by the embodiment of the present invention.

[0072] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium. The program code included in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0073] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written by any combination of one or more programming languages, and specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages. Programming languages ​​include but are not limited to programming languages ​​such as Java, C++, python, C language or similar. The program code can be executed completely on the user computing device, partially on the user device, partially on the remote computing device, or completely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network or a wide area network, or can be connected to an external computing device.

[0074] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teaching of the present invention, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0075] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination. The scope of the present invention is limited by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A collaborative testing method for a graphical user interface, characterized in that: include: Get the control tree structure of the application's graphical user interface and send it to all agents in the collaborative test; Building an image recognition model based on the element graphics of the graphical user interface and sending it to all the intelligent agents in the collaborative test; Obtaining a test task, decomposing the test task into a plurality of subtasks, and sending all the subtasks to the intelligent agents of the collaborative test respectively; The agent performs a test operation on the graphical user interface according to the received subtask and uses the control tree structure or the image recognition model, and returns a test result; A collaborative test result for the graphical user interface is generated based on the test results returned by all the agents in the collaborative test.

2. A collaborative testing method for a graphical user interface according to claim 1, characterized in that: The obtaining of the control tree structure of the graphical user interface of the application includes: obtaining the control tree structure using an API interface provided by the operating system or a third-party library; the control tree structure includes: the graphical user interface elements, element types, hierarchical relationships and attribute information.

3. A collaborative testing method for a graphical user interface according to claim 1, characterized in that: The image recognition model is established based on the element graphics of the graphical user interface, including: obtaining the element graphics as training data, training a neural network model, and obtaining the image recognition model that can recognize the element graphics.

4. A collaborative testing method for a graphical user interface according to claim 1, characterized in that: The obtaining of the test task, decomposing the test task into a plurality of subtasks, and sending all the subtasks to the intelligent agents of the collaborative test respectively, includes: sending the plurality of subtasks to the plurality of intelligent agents of the collaborative test in a balanced manner according to a load balancing algorithm.

5. A collaborative testing method for a graphical user interface according to claim 3, characterized in that: The acquiring the element graph as training data includes: extracting a feature vector of the element graph as the training data.

6. A collaborative testing method for a graphical user interface according to claim 1, characterized in that: The obtaining of the control tree structure of the graphical user interface of the application and sending it to all the agents in the collaborative test includes: sending the control tree structure to all the agents in the collaborative test in XML format or JSON format.

7. A collaborative testing method for a graphical user interface according to claim 5, characterized in that: The extracting the feature vector of the element graph as the training data includes: extracting the feature vector of the element graph using an OpenCV computer vision library.

8. A collaborative testing device for a graphical user interface, characterized in that: The device comprises: A control tree structure sending module obtains the control tree structure of the graphical user interface of the application and sends it to all agents in the collaborative test; An image recognition model sending module, which establishes an image recognition model based on the element graphics of the graphical user interface and sends it to all the intelligent agents in the collaborative test; A subtask sending module obtains a test task, decomposes the test task into a plurality of subtasks, and sends all the subtasks to the intelligent agents of the collaborative test respectively; A first result generating module, wherein the agent performs a test operation on the graphical user interface according to the received subtask and uses the control tree structure or the image recognition model, and returns a test result; The second result generating module generates the collaborative test result of the graphical user interface according to the test results returned by all the intelligent agents of the collaborative test.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to perform the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images