Intelligent client UI automatic testing method and device and storage medium

Abstract

The invention relates to an intelligent client UI automatic testing method and device and a storage medium, and is applied to the technical field of UI automatic testing, and the method comprises the steps that a testing access node is established, an execution machine resource pool supporting a video and a driving data path is established in the testing access node, and the execution machine resource pool comprises access addresses of all executed machines; a user uses a browser to access the execution machine to be tested in the execution machine resource pool through a certain node A of the Internet or a local area network, video and drive dual channels with the execution machine to be tested are established, and a control drive mode or a picture drive mode is selected to record a test case. Visual use case recording is carried out, and the regional use limitation is broken through.

Description

Technical Field

[0001] The present invention relates to the technical field of UI automated testing, and in particular to an intelligent client UI automated testing method, device and storage medium. Background Art

[0002] Automated testing technology is a technology that simulates the manual testing process and converts it into a computer program for automatic execution in order to save testing manpower costs. Automated testing technology is classified into UI automated testing and interface automated testing according to whether the application under test contains a GUI. According to the different carriers of the application under test, for example, the testing technology for applications on mobile phones is called mobile UI automated testing. The testing technology for applications on Windows and Linux operating systems is called client UI automated testing technology, and the carrier is a browser, which is called web UI automated testing technology.

[0003] In the prior art, testers must be on the same computer as the application system to enter use cases. For example, when the system under test is restricted from remote access, testers are subject to obvious geographical restrictions. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an intelligent client UI automation testing method, device and storage medium to solve the problem in the prior art that testers must enter use cases on the same computer as the application system, and when the system under test is restricted from remote access, testers are subject to obvious geographical usage restrictions.

[0005] According to a first aspect of an embodiment of the present invention, a smart client UI automation testing method is provided, the method comprising:

[0006] Establishing a test access node, creating an execution machine resource pool supporting video and drive data paths in the test access node, wherein the execution machine resource pool includes access addresses of all executed machines;

[0007] The user uses a browser to access the execution machine to be tested in the execution machine resource pool through the Internet or a node A of the local area network, and establishes a video and drive dual channel with the execution machine to be tested;

[0008] The user selects a control driving mode or an image driving mode. The control driving mode is to call Windows or Linux API through JNA to poll all window processes on the desktop of the execution machine to be tested, obtain window information of all window processes on the desktop, obtain execution information based on the window information and return it to node A through video and driver dual channels;

[0009] The image driving mode is to obtain the screenshot information on the execution machine to be tested, send the screenshot information to the service platform, and the service platform uses the image analysis engine in the form of an operator to analyze the elements contained in the screenshot information, obtain the execution information, and return the execution information to node A through the video and drive dual channels;

[0010] The user operates the execution machine to be tested in the browser video area of ​​node A according to the execution information, and the execution machine to be tested executes the corresponding operation instructions to complete the use case test.

[0011] Preferably, it also includes:

[0012] The operation instructions of each test case are converted into digital use case steps, and a test case set is established based on the digital use case steps of multiple test cases.

[0013] Preferably, it also includes:

[0014] The user uses a browser to access the test case set through the Internet or a node A in the local area network, selects one or more test cases therein, and converts them into a test plan;

[0015] The user selects the test plan to be executed asynchronously or synchronously on node A. The asynchronous execution is to select any idle execution machine from the execution machine resource pool, convert the digital test case steps of the test case into mouse and keyboard instructions, and issue them to the selected idle execution machine through the video and drive dual channels to start remote unattended test case execution;

[0016] The synchronous execution is to select a designated execution machine from the execution machine resource pool, convert the digital test case steps of the test case into mouse and keyboard instructions, and issue them to the selected designated execution machine through the video and drive dual channels. During the test case execution process, the execution result is synchronously returned to node A.

[0017] Preferably,

[0018] The acquiring execution information based on the window information comprises:

[0019] The window information includes a window handle set, a title, and area coordinate information. The window handle set is filtered according to the title and area coordinate information to obtain a unique window handle. Execution information is obtained according to the unique window handle, and the execution information includes coordinate information and handle information of the unique window handle.

[0020] Preferably,

[0021] The obtaining of the screenshot information on the execution machine to be tested, sending the screenshot information to the service platform, the service platform using an image analysis engine in the form of an operator to analyze the elements contained in the image on the screenshot information, and obtaining the execution information includes:

[0022] The user takes a screenshot of the desktop of the tested execution machine projected by the browser of node A, obtains the object image by means of a mouse selection operation, and sends the object image and the desktop screenshot to the service platform. The service platform uses an image analysis engine to calculate the real-time coordinate information of the object image on the desktop screenshot. The real-time coordinate information of the object image is the execution information.

[0023] Preferably, it also includes:

[0024] The object image is stored in the image server in the form of url or base64.

[0025] Preferably,

[0026] In the control drive mode, the digitization use case steps include: window handle collection and filter condition information;

[0027] In the picture driven mode, the digitization use case step includes: object pictures stored in the form of url or base64.

[0028] Preferably, it also includes:

[0029] If the digitization use case steps in the picture-driven mode are run across execution machines;

[0030] The spanned execution machine to be tested receives the object image stored in the form of url or base64 in the digital use case step, performs pyramid image transformation on the object image, scales it into multiple object images of different scales, performs template matching on the multiple object images of different scales and the desktop screenshot of the spanned execution machine to be tested, selects the object image with the highest confidence and greater than a preset threshold, calculates the ROI area according to the scaling ratio corresponding to the selected object image, and obtains the real-time coordinate information of the object image on the desktop of the spanned execution machine to be tested;

[0031] If the highest confidence level of the matching between the multiple object images of different proportions and the desktop screenshot of the spanned execution machine to be tested is less than a preset threshold, the object image is input into a pre-built visual deep learning model, and the pre-built visual deep learning model extracts features of the object image through different feature layers, and then fuses the features extracted from different feature layers to output a feature vector matrix of the object image;

[0032] Obtaining a feature vector matrix of the desktop screenshot of the spanned execution machine to be tested;

[0033] Performing softmax calculation on the feature vector matrix of the object image and the desktop screenshot of the spanned execution machine to be tested, to obtain the real-time coordinate information of the object image on the desktop of the spanned execution machine to be tested;

[0034] The real-time coordinate information of the object image on the desktop of the spanned execution machine to be tested is returned to node A.

[0035] According to a second aspect of an embodiment of the present invention, a smart client UI automation testing device is provided, the device comprising:

[0036] Resource pool construction module: used to establish a test access node, create an execution machine resource pool supporting video and drive data paths in the test access node, and the execution machine resource pool contains access addresses of all executed machines;

[0037] Path construction module: used for a user to use a browser to access the execution machine to be tested in the execution machine resource pool through the Internet or a certain node A of the local area network, and to establish a video and drive dual path with the execution machine to be tested;

[0038] Control driving module: used for users to select control driving mode or image driving mode. The control driving mode is to call Windows or Linux API through JNA to poll all window processes on the desktop of the execution machine to be tested, obtain window information of all window processes on the desktop, obtain execution information based on the window information and return it to node A through video and driver dual channels;

[0039] Image driving module: used for the image driving mode to obtain screenshot information on the execution machine to be tested, send the screenshot information to the service platform, the service platform uses the image analysis engine in the form of an operator to analyze the elements contained in the screenshot information, obtain the execution information, and return the execution information to node A through the video and driver dual channels;

[0040] Execution module: used for the user to operate the execution machine to be tested in the browser video area of ​​node A according to the execution information, and the execution machine to be tested executes the corresponding operation instructions to complete the use case test.

[0041] According to a third aspect of an embodiment of the present invention, there is provided a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a main controller, each step in the above method is implemented.

[0042] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0043] The present application establishes a test access node, and creates an execution machine resource pool that supports video and drive data paths in the test access node. The execution machine resource pool contains the access addresses of all executed machines. The user uses a browser to access the execution machine to be tested in the execution machine resource pool through the Internet or a node A on a local area network, establishes video and drive dual paths with the execution machine to be tested, and selects a control drive mode or an image drive mode to record test cases. Based on the present application, users can remotely connect to any node to perform visual use case recording, breaking geographical usage restrictions.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0046] Figure 1 It is a flowchart of a smart client UI automation testing method according to an exemplary embodiment;

[0047] Figure 2 It is a schematic diagram of the overall framework of a smart client UI automation testing method according to an exemplary embodiment;

[0048] Figure 3 is a schematic diagram of executing coordinate information conversion across execution machines according to another exemplary embodiment;

[0049] Figure 4 is a system schematic diagram of a smart client UI automation testing device according to another exemplary embodiment;

[0050] In the attached figure: 1-resource pool construction module, 2-pathway construction module, 3-control driving module, 4-image driving module, 5-execution module. DETAILED DESCRIPTION

[0051] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0052] Embodiment 1

[0053] Figure 1FIG. 1 is a flow chart of a method for automatically testing a smart client UI according to an exemplary embodiment. Figure 1 As shown, the method includes:

[0054] S1, establishing a test access node, creating an execution machine resource pool supporting video and drive data paths in the test access node, the execution machine resource pool including access addresses of all executed machines;

[0055] S2, the user uses a browser to access the execution machine to be tested in the execution machine resource pool through the Internet or a node A of the local area network, and establishes a video and drive dual channel with the execution machine to be tested;

[0056] S3, the user selects a control driving mode or an image driving mode, wherein the control driving mode is to call Windows or Linux API through JNA to poll all window processes on the desktop of the execution machine to be tested, obtain window information of all window processes on the desktop, obtain execution information based on the window information and return it to node A through video and driver dual channels;

[0057] S4, the image driving mode is to obtain screenshot information on the execution machine to be tested, send the screenshot information to the service platform, and the service platform uses the image analysis engine in the form of an operator to analyze the elements contained in the screenshot information, obtain the execution information, and return the execution information to node A through the video and drive dual channels;

[0058] S5, the user performs an operation on the execution machine to be tested in the browser video area of ​​node A according to the execution information, and the execution machine to be tested executes the corresponding operation instruction to complete the use case test;

[0059] It is understandable that in order to facilitate understanding of this embodiment, the existing technical framework is briefly described: the industry UI automation technology mainly includes the following typical representatives:

[0060] Framework 1: Microsoft UI Automation framework, which is a new accessibility framework for Microsoft Windows and can be used in all operating systems that support Windows Presentation Foundation (WPF). UIAutomation provides programmatic access to most user interface (UI) elements on the desktop, enabling assistive technology products (such as screen readers) to provide information about the UI to end users and to operate the UI in ways other than standard input. UI Automation also allows automated test scripts to interact with the UI.

[0061] Framework 2: Win App Driver framework, a UI automation testing tool developed and maintained by Microsoft for testing Windows desktop applications and Universal Windows Platform (UWP) applications.

[0062] Framework 1 and Framework 2 are bottom-level general frameworks. Some other frameworks are developed and expanded based on 1 and 2, such as AutoIt, Coded UI Test, TestComplete, etc. The use cases used in these automated testing technologies are mainly organized in script code, which maintains the basic existence of Framework 1 and Framework 2. In addition, use case entry needs to be performed on the same machine as the application under test, which limits the flexibility of use case writers in using UI automation technology.

[0063] Sikuli automation technology is a novel graphical scripting language. Users only need to have the most basic programming skills and mainly use screenshots to complete the expression of use cases. However, this method also has obvious disadvantages because the technology relies on image search technology, which is pixel-sensitive. Once the use case is transplanted to another computer with a different display resolution, the use case will fail to execute, resulting in an increase in the failure rate of the automated use case.

[0064] In addition, the automation execution technology of some RPA (automated operation and maintenance software) is basically the same as the driving technology of UI automation, but it focuses on execution and maintenance, lacks test recall and inspection. Its driving technology is also extended by using framework 1 and framework 2, or using method 4, which also has pixel sensitivity problems and low execution efficiency problems;

[0065] In short, the problems with existing frameworks are:

[0066] Personnel are required to enter UI automation use cases in front of the computer of the application under test;

[0067] With existing automation technologies, the application under test can only be deployed on a fixed device. When no test is being carried out, the machine is idle.

[0068] There are existing UI automation use cases. When the use cases are migrated from a machine with resolution A to a machine with resolution B, the test case execution fails due to insufficient existing image recognition capabilities, which introduces a large use case maintenance cost.

[0069] In order to solve the above problems, this embodiment provides the following technical solutions, as shown in the attached Figure 2 As shown:

[0070] Users can remotely connect to any node to record visual use cases. In addition, users can configure test sets and asynchronously trigger remote executors to complete the execution of test plans and return test result reports. By creating an executor resource pool that supports video and drive data channels, online recording of use cases is achieved based on the video data channel. With the support of cross-resolution algorithms, shared execution of use cases is achieved without considering the difference in executor resolution. The specific solutions are as follows:

[0071] The test executor uses any node browser in the network to access the system platform, establishes a video channel in the browser page, visualizes the remote desktop, and operates the application under test in the browser window. At the same time, the relevant operation process is converted into the steps of the automated use case that supports controls and images in real time, which mainly includes the following sub-processes:

[0072] Establish a channel. At a node A in a local area network or the Internet that is reachable by the network channel, use a browser to access the test access node, select an execution machine B in the resource pool of the test access node, establish a video and drive dual channel, and realize remote control. Specifically, establish a video transmission channel through no-vnc and vnc server, and establish an instruction drive channel through jna technology;

[0073] Start recording. Users can choose two recording modes through the platform, one is control-driven mode and the other is image-driven mode. In control-driven mode, all window processes on the desktop of machine B are trained in rotation through JNA calls Windows or Linux API, and window information is analyzed, including handles, titles, area coordinates and other information. When the use case is executed, the operating system desktop window handle set is analyzed, filtered according to information such as title and area coordinates, and a unique window handle is obtained, and then the execution basis of the use case steps, i.e., coordinate information and handle information, is obtained, that is, the execution information in control-driven mode;

[0074] In the picture-driven mode, during the recording and playback process on the execution machine B, the user uses the desktop screen information of B projected on A to select the operation object with the help of the mouse, and performs real-time calculations on the desktop background picture and the object picture on B to obtain the coordinate information of the object picture in the desktop background picture, that is, the execution information in the picture-driven mode;

[0075] Complete recording: The user performs operations in the video area of ​​the browser A based on the feedback execution information on A. The operation instructions act on the execution machine B on the one hand, and on the other hand, the auxiliary information of the operation is converted into digital steps of the automation use case. For example, in the control-driven mode, the steps of the UI automation use case include the window handle information and filtering condition information when the operating system is running, such as the class name and title name to filter by; in the image-driven mode, the steps are composed of the screenshot information of the control selected by the mouse during the use case recording process, such as the URL and base64 information stored in the image. The above are stored in the database, which is convenient for real-time calculation of the handle and coordinate information required for the use case playback and execution when the use case is played back and executed;

[0076] It is worth emphasizing that the object images obtained in the image-driven mode are stored in the image server in the form of url and base64. The above recording, playback and execution process can be seen in real time through the projection on A;

[0077] The digital steps of each test case are stored to obtain a test case set. When a test case needs to be executed, the user logs in to node A from the local area network or the Internet, accesses the test access node through a web browser, selects one or more test cases in the test case set, and converts them into a test plan. The test plan can be executed asynchronously and in conjunction with the xxl-job scheduled task to achieve scheduled execution. It can also be triggered by other third parties, such as events in the devops and jenkins pipelines.

[0078] Asynchronous execution: If you choose asynchronous execution, an idle machine C will be selected from the resource pool containing multiple test execution machines, and then each test step of the test case will be converted into mouse and keyboard commands, which will be sent to machine C through the driver channel to start remote unattended test case execution.

[0079] Synchronous execution: The process of synchronous execution is basically the same as that of asynchronous execution. The difference is that in synchronous execution, the user can see the execution result synchronously on A. At the same time, before execution, the specified machine must be selected;

[0080] Finally, relevant test results, including use case execution results, process desktop video recording, process execution log and other information, are presented in the browser of node A;

[0081] It is worth emphasizing that if the test case is recorded on the execution machine B, and the machine that needs to execute the test case later is the execution machine C, due to the different screen resolutions of the execution machines B and C, the coordinate information of the object image of the execution machine B will be different on the execution machine C. In order to solve this problem, this embodiment proposes:

[0082] As attached Figure 3 As shown:

[0083] S101, if the digital use case steps in the image-driven mode are run across execution machines;

[0084] S201, the spanned execution machine to be tested receives the object image stored in the form of url or base64 in the digital use case step, performs pyramid image transformation on the object image, scales it into multiple object images of different scales, performs template matching on the multiple object images of different scales and the desktop screenshot of the spanned execution machine to be tested, selects the object image with the highest confidence and greater than a preset threshold, calculates the ROI area according to the scaling ratio corresponding to the selected object image, and obtains the real-time coordinate information of the object image on the desktop of the spanned execution machine to be tested;

[0085] S301, if the highest confidence level of the matching between the plurality of object images of different proportions and the desktop screenshot of the spanned execution machine to be tested is less than a preset threshold, the object image is input into a pre-built visual deep learning model, the pre-built visual deep learning model extracts features of the object image through different feature layers respectively, and then fuses the features extracted from the different feature layers to output a feature vector matrix of the object image;

[0086] S401, obtaining a feature vector matrix of the desktop screenshot of the spanned execution machine to be tested;

[0087] S501, performing softmax calculation on the feature vector matrix of the object image and the desktop screenshot of the spanned execution machine to be tested, to obtain the real-time coordinate information of the object image on the desktop of the spanned execution machine to be tested;

[0088] S601, returning the real-time coordinate information of the object image on the desktop of the spanned execution machine to be tested to node A;

[0089] In simple terms, assuming that the executor that needs to execute the test case is C, the executor C receives the data, the service receives the API request and responds to the analysis result. After receiving the request, there are two types of image information included in the request parameter, one is the URL method, and the other is the base64 method. The former uses object storage middleware, such as minio and other tools to complete the image transfer and temporary storage, and the latter is to transmit the image by base64 encoding. This solution can be selected according to the user's call situation;

[0090] The received image is image template positioned. In order to solve the differences in machine resources in the resource pool, for example, different machines have different display resolutions, and the size of the corresponding screen and control screenshots will change, which leads to recognition failure in the traditional visual matching algorithm. The processing flow of this embodiment is to first perform pyramid image changes on the template image (the template image refers to the object image stored in the digitization step of the test case converted after each case test is completed, and is assumed to be the object image during the test case execution process on the execution machine B), scale the image to 0.7, 0.8, 0.9, 1.1, 1.2, 1.3 times multiple image backups, and perform template matching with the base map (the base map here refers to the spanned execution machine, that is, the screen desktop screenshot on the execution machine C), select the image with the highest confidence and greater than 95%, and calculate the ROI area according to the corresponding zoom multiple, and return the coordinates to node A, and node A performs automatic operations according to the coordinate information;

[0091] If the highest confidence of the matching result in the above steps is lower than 95%, the deep feature extraction technology will be further used. The vgg19 model is used in this embodiment. The effects of other basic models are similar. The template image is input into the vgg19 model. The vgg19 model contains different feature layers. Each feature layer extracts the features of the template image. Then, the features of different layers are fused to output the feature vector matrix. For example, the second and fourth layers are feature fused. Then, the feature vector matrices of the template and the base map are softmax calculated to obtain the coordinate information and return it to node A.

[0092] This embodiment provides a remote access method for online writing and recording of use cases, and removes the time and location restrictions for writing use cases, which will greatly release the efficiency of automated work; by establishing a resource pool technology that supports dual channels of video and drive, the overall shared utilization rate of the equipment is improved; through the pyramid and deep feature fusion technology, the technology of cross-resolution image recognition is solved, the robustness of automated use cases is improved, and the use case maintenance cost is reduced.

[0093] Embodiment 2

[0094] Figure 4 : is a system schematic diagram of a smart client UI automation testing device according to another exemplary embodiment, comprising:

[0095] Resource pool construction module 1: used to establish a test access node, in which an execution machine resource pool supporting video and drive data paths is created, wherein the execution machine resource pool contains access addresses of all executed machines;

[0096] Path construction module 2: used for a user to use a browser to access the execution machine to be tested in the execution machine resource pool through the Internet or a certain node A of the local area network, and establish a video and drive dual path with the execution machine to be tested;

[0097] Control driving module 3: used for the user to select control driving mode or image driving mode. The control driving mode is to call Windows or Linux API through JNA to poll all window processes on the desktop of the execution machine to be tested, obtain window information of all window processes on the desktop, obtain execution information based on the window information and return it to node A through video and driver dual channels;

[0098] Image driving module 4: used for the image driving mode, which is to obtain the screenshot information on the execution machine to be tested, and send the screenshot information to the service platform. The service platform uses the image analysis engine in the form of an operator to analyze the elements contained in the screenshot information, obtain the execution information, and return the execution information to node A through the video and drive dual channels;

[0099] Execution module 5: used for the user to operate the execution machine to be tested in the browser video area of ​​node A according to the execution information, and the execution machine to be tested executes the corresponding operation instructions to complete the use case test.

[0100] Embodiment three:

[0101] This embodiment provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a main controller, each step in the above method is implemented;

[0102] It is understandable that the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0103] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0104] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0105] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0106] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0107] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0108] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0109] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0110] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0111] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A smart client UI automation testing method, characterized in that: The method comprises: Establishing a test access node, creating an execution machine resource pool supporting video and drive data paths in the test access node, wherein the execution machine resource pool includes access addresses of all executed machines; The user uses a browser to access the execution machine to be tested in the execution machine resource pool through the Internet or a node A of the local area network, and establishes a video and drive dual channel with the execution machine to be tested; The user selects a control driving mode or an image driving mode. The control driving mode is to call Windows or Linux API through JNA to poll all window processes on the desktop of the execution machine to be tested, obtain window information of all window processes on the desktop, obtain execution information based on the window information and return it to node A through video and driver dual channels; The image driving mode is to obtain the screenshot information on the execution machine to be tested, send the screenshot information to the service platform, and the service platform uses the image analysis engine in the form of an operator to analyze the elements contained in the screenshot information, obtain the execution information, and return the execution information to node A through the video and drive dual channels; The user operates the execution machine to be tested in the browser video area of ​​node A according to the execution information, and the execution machine to be tested executes the corresponding operation instructions to complete the use case test.

2. The method according to claim 1, characterized in that Also includes: The operation instructions of each test case are converted into digital use case steps, and a test case set is established based on the digital use case steps of multiple test cases.

3. The method according to claim 2, characterized in that Also includes: The user uses a browser to access the test case set through the Internet or a node A in the local area network, selects one or more test cases therein, and converts them into a test plan; The user selects the test plan to be executed asynchronously or synchronously on node A. The asynchronous execution is to select any idle execution machine from the execution machine resource pool, convert the digital test case steps of the test case into mouse and keyboard instructions, and issue them to the selected idle execution machine through the video and drive dual channels to start remote unattended test case execution; The synchronous execution is to select a designated execution machine from the execution machine resource pool, convert the digital test case steps of the test case into mouse and keyboard instructions, and issue them to the selected designated execution machine through the video and drive dual channels. During the test case execution process, the execution result is synchronously returned to node A.

4. The method according to claim 2, characterized in that: The acquiring execution information based on the window information comprises: The window information includes a window handle set, a title, and area coordinate information. The window handle set is filtered according to the title and area coordinate information to obtain a unique window handle. Execution information is obtained according to the unique window handle, and the execution information includes coordinate information and handle information of the unique window handle.

5. The method according to claim 2, characterized in that: The obtaining of the screenshot information on the execution machine to be tested, sending the screenshot information to the service platform, the service platform using an image analysis engine in the form of an operator to analyze the elements contained in the image on the screenshot information, and obtaining the execution information includes: The user takes a screenshot of the desktop of the tested execution machine projected by the browser of node A, obtains the object image by means of a mouse selection operation, and sends the object image and the desktop screenshot to the service platform. The service platform uses an image analysis engine to calculate the real-time coordinate information of the object image on the desktop screenshot. The real-time coordinate information of the object image is the execution information.

6. The method according to claim 5, characterized in that Also includes: The object image is stored in the image server in the form of url or base64.

7. The method according to any one of claims 6 or 4, characterized in that: In the control drive mode, the digitization use case steps include: window handle collection and filter condition information; In the picture driven mode, the digitization use case step includes: object pictures stored in the form of url or base64.

8. The method according to claim 7, characterized in that Also includes: If the digitization use case steps in the picture-driven mode are run across execution machines; The spanned execution machine to be tested receives the object image stored in the form of url or base64 in the digital use case step, performs pyramid image transformation on the object image, scales it into multiple object images of different scales, performs template matching on the multiple object images of different scales and the desktop screenshot of the spanned execution machine to be tested, selects the object image with the highest confidence and greater than a preset threshold, calculates the ROI area according to the scaling ratio corresponding to the selected object image, and obtains the real-time coordinate information of the object image on the desktop of the spanned execution machine to be tested; If the highest confidence level of the matching between the multiple object images of different proportions and the desktop screenshot of the spanned execution machine to be tested is less than a preset threshold, the object image is input into a pre-built visual deep learning model, and the pre-built visual deep learning model extracts features of the object image through different feature layers, and then fuses the features extracted from different feature layers to output a feature vector matrix of the object image; Obtaining a feature vector matrix of the desktop screenshot of the spanned execution machine to be tested; Performing softmax calculation on the feature vector matrix of the object image and the desktop screenshot of the spanned execution machine to be tested, to obtain the real-time coordinate information of the object image on the desktop of the spanned execution machine to be tested; The real-time coordinate information of the object image on the desktop of the spanned execution machine to be tested is returned to node A.

9. An intelligent client UI automation testing device, characterized in that: The device comprises: Resource pool construction module: used to establish a test access node, create an execution machine resource pool supporting video and drive data paths in the test access node, and the execution machine resource pool contains access addresses of all executed machines; Path construction module: used for a user to use a browser to access the execution machine to be tested in the execution machine resource pool through the Internet or a certain node A of the local area network, and to establish a video and drive dual path with the execution machine to be tested; Control driving module: used for users to select control driving mode or image driving mode. The control driving mode is to call Windows or Linux API through JNA to poll all window processes on the desktop of the execution machine to be tested, obtain window information of all window processes on the desktop, obtain execution information based on the window information and return it to node A through video and driver dual channels; Image driving module: used for the image driving mode to obtain screenshot information on the execution machine to be tested, send the screenshot information to the service platform, the service platform uses the image analysis engine in the form of an operator to analyze the elements contained in the screenshot information, obtain the execution information, and return the execution information to node A through the video and driver dual channels; Execution module: used for the user to operate the execution machine to be tested in the browser video area of ​​node A according to the execution information, and the execution machine to be tested executes the corresponding operation instructions to complete the use case test.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the main controller, each step in the intelligent client UI automatic testing method as described in any one of claims 1-8 is implemented.

Images