Cross-system user interface automatic test method and test platform

Abstract

The embodiment of the invention provides a cross-system user interface automatic test method and a test platform, and the method comprises the steps: obtaining a target test script and a target client selected by a user, and converting the target test script into a general control operation instruction irrelevant to a system; dynamically selecting a control path analysis mode or an image recognition mode as a target analysis mode according to the current operation environment of the target client; the general control operation instruction is mapped into a native control operation instruction suitable for the target client in a target analysis mode, and the native control operation instruction comprises a control path of the target control on the target client; and the native control operation instruction is sent to the target client, so that the target client locates the target control according to the control path and executes the native control operation instruction on the target control, and automatic testing is realized. The cross-system user interface automatic test is realized, the cross-system compatibility of the test script is improved, and the script development and maintenance cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software testing, in particular to a cross-system user interface automatic testing method and a testing platform. BACKGROUND

[0002] With the explosive growth of mobile applications, how to efficiently and correctly perform user interface (UI) automatic testing has become an important problem. Current mobile UI automatic testing faces multiple challenges: traditional scripted testing relies on manual coding, with high development and maintenance costs; and the recording and playback technology with higher automation has problems such as poor system compatibility (such as IOS / Android / Harmony differences), insufficient dynamic control adaptation, and the like, resulting in low test case reuse rate and cross-end execution failure. SUMMARY

[0003] The embodiments of the present application provide a cross-system user interface automatic testing method and a testing platform, which realize cross-system user interface automatic testing, improve the cross-system compatibility of test scripts, and reduce the cost of script development and maintenance.

[0004] In a first aspect, the embodiments provide a cross-system user interface automatic testing method, which comprises:

[0005] obtaining a target test script selected by a user and a target client, and converting the target test script into general control operation instructions independent of a system, the target test script being written in a unified service description language;

[0006] dynamically selecting a control path analysis method or an image recognition method as a target analysis method according to a current running environment of the target client;

[0007] mapping the general control operation instructions into native control operation instructions applicable to the target client by using the target analysis method, the native control operation instructions containing a control path of a target control in the target client;

[0008] sending the native control operation instructions to the target client, so that the target client locates the target control according to the control path and executes the native control operation instructions on the target control, thereby realizing automatic testing.

[0009] In a second aspect, the embodiments provide a cross-system user interface automatic testing platform, which comprises:

[0010] The script selection module is configured to obtain a target test script and a target client selected by a user, and convert the target test script into a system-independent universal control operation instruction, wherein the target test script is written in a unified service description language.

[0011] The double-engine module is configured to dynamically select a control path analysis method or an image recognition method as a target analysis method according to a current running environment of the target client.

[0012] The instruction generation module is configured to map the universal control operation instruction into a native control operation instruction applicable to the target client by using the target analysis method, wherein the native control operation instruction comprises a control path of a target control in the target client.

[0013] The test module is configured to send the native control operation instruction to the target client, so that the target client locates the target control according to the control path and executes the native control operation instruction on the target control, thereby realizing automated testing.

[0014] The embodiment of the present application provides a cross-system user interface automated testing method and a testing platform, which comprises the following steps: firstly, obtaining a target test script and a target client selected by a user, and converting the target test script into a system-independent universal control operation instruction, wherein the target test script is written in a unified service description language; secondly, dynamically selecting a control path analysis method or an image recognition method as a target analysis method according to a current running environment of the target client; thirdly, mapping the universal control operation instruction into a native control operation instruction applicable to the target client by using the target analysis method, wherein the native control operation instruction comprises a control path of a target control in the target client; and finally, sending the native control operation instruction to the target client, so that the target client locates the target control according to the control path and executes the native control operation instruction on the target control, thereby realizing automated testing. The above technical solution uses the unified service description language to write the test script, generates the system-independent universal control operation instruction by analyzing the test script, intelligently selects the control path analysis method or the image recognition method to convert the standardized control expression into the native control operation instruction applicable to the target client according to the current running environment, and can realize the automated testing on the client in different systems. The test script development is realized once, and the effect that the test script can be used in multiple systems is realized, the cross-system compatibility is improved, and the cost of script development and maintenance is reduced.

[0015] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a cross-system user interface automated testing method provided in Embodiment 1 of the present invention;

[0018] Figure 2 This is an example diagram of the client and server architecture in a cross-system user interface automated testing method provided in Embodiment 1 of the present invention;

[0019] Figure 3 This is a flowchart illustrating another cross-system user interface automated testing method provided in Embodiment 2 of the present invention;

[0020] Figure 4 This is a flowchart illustrating another cross-system user interface automated testing method provided in Embodiment 3 of the present invention;

[0021] Figure 5 This is an example flowchart of a cross-system user interface automated testing method provided in Embodiment 3 of the present invention;

[0022] Figure 6 This is a schematic diagram of the structure of a cross-system user interface automated testing platform provided in Embodiment 3 of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It is to be understood that the terms "first", "second", and the like, used in the description and the claims of the application, as well as the above-described drawings, are used to distinguish between similar objects and not necessarily describe a particular sequential or chronological order. It should be understood that the use of such terms is interchangeable under appropriate circumstances such that the embodiment of the application described herein are capable of operation in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprise", "comprising", and "including" and any variations thereof, are intended to cover a non-exclusive inclusion, such that processes, methods, systems, products, or devices that comprise, include, or are including a list of steps or units not only consist of those steps or units but can include other not expressly listed steps or units.

[0025] With the rapid development of mobile internet technology, mobile applications have widely covered multiple operating systems such as Android, IOS, and Harmony. In order to meet the needs of different user groups, developers usually need to adapt multiple system versions for the same application. However, with the rapid iteration of application functions, testing requirements are increasing, and traditional manual testing methods are inefficient and costly, making it difficult to meet the needs of modern software development. Automated testing has gradually become an industry standard.

[0026] Currently, mobile UI automated testing mainly relies on the following two types of technologies:

[0027] 1) Scripted testing: Testers locate control attributes (such as ID, Xpath) and simulate user operations by writing code (such as Appium scripts). Although this method has high flexibility, it has the problems of long development cycle and high maintenance cost. In addition, due to the significant differences in control attributes between Android, IOS and Harmony systems, the same script cannot be reused across systems and needs to be developed separately for different systems, resulting in low testing efficiency. Appium is an open-source automated testing framework that supports multiple platforms and multiple programming languages, including iOS and Android. Xpath (XML Path Language) is an XML path language that is used to determine the location of a part of an XML document.

[0028] 2) Recording playback test: generate test scripts by recording user operations, reduce the coding threshold. However, such tools usually rely on native control structures, and when faced with the underlying differences between Android, IOS and Harmony systems, the recorded scripts are prone to playback failure due to control recognition failure. In addition, for dynamically rendered interfaces or custom controls, due to the lack of stable attribute identification, traditional recording and playback technology is difficult to accurately locate, resulting in insufficient test coverage. Rendering refers to the process of converting 3D models or scenes into 2D images, adding color, texture, lighting, and other effects to the model through a series of calculations and processing, making it appear realistic or have a specific visual style.

[0029] In recent years, with the popularity of Harmony systems, the complexity of cross-end testing has further increased. Harmony systems use the ArkUI framework, which has a significantly different control system than Android and IOS. Existing automated testing tools lack native support for Harmony systems, resulting in the need for repeated development of test scripts, further increasing testing costs. Although image recognition technology (such as optical character recognition, OCR) can compensate for the lack of control positioning to some extent, when used alone, it has low execution efficiency and is easily disturbed by screen resolution or light changes, making it difficult to run stably in complex scenarios.

[0030] In the prior art, the patent with application number 202310412996.6 and the invention name of a cross-system automated testing method, device, equipment and medium, discloses a cross-system automated testing method, device, equipment and medium. First, the Appium Client obtains the test start instruction of the client, and connects the client running on the device according to the test start instruction; the Appium Server reads the YAML configuration file corresponding to the client running platform according to the test start instruction, wherein the YAML configuration file includes test case name, element positioning information, operation type name, operation variable type and value; the YAML configuration file is parsed, and the client is subjected to UI automated testing.

[0031] YAML (YAML Ain't Markup Language) is a lightweight markup language based on data serialization, supporting complex data structures and widely used in configuration files, data exchange and other fields. The existing technology realizes cross-system automated testing based on the Appium framework and YMAL file configuration, which improves the test efficiency to some extent, but still has significant technical defects:

[0032] 1) Strong dependency on control properties leads to insufficient cross-system compatibility: existing technologies rely on Appium to locate elements by control properties and generate YAML configuration files, however, the naming rules and structure of control properties differ significantly across different systems (such as IOS, Android, and Harmony), the same control may not be located by the same property in different systems, especially under the ArkUI framework of Harmony system, existing technologies are difficult to directly adapt, and configuration files need to be written for different systems, reducing script reuse rate.

[0033] 2) Lack of support for dynamic interfaces and custom controls: existing technologies rely on control properties to locate elements, but dynamic rendering interfaces and custom controls often lack stable property identifiers, limiting test coverage.

[0034] 3) High maintenance cost of YAML configuration files: with application function iteration, interface elements change frequently, especially in the context of multiple systems coexisting, testers need to constantly update and verify configuration files for different systems, increasing manpower investment.

[0035] 4) Existing technologies lack native support for Harmony system: unable to directly adapt to ArkUI framework of Harmony, additional development of adaptation logic is required when running on Harmony system, further increasing development cost and test complexity.

[0036] In summary, existing technologies have obvious limitations in cross-system compatibility, dynamic adaptability, and test script reuse rate. How to balance the efficiency of control positioning and the flexibility of image recognition, and build a UI automation testing solution that can automatically adapt to Android, IOS, and Harmony, achieving "one development, multiple runs", has become a key problem to improve the efficiency of mobile application testing.

[0037] Embodiment one

[0038] Figure 1 A flowchart of a cross-system user interface automation testing method provided by the first embodiment of the application is provided. The method can be used for cross-system user interface automation testing. The method can be executed by a cross-system user interface automation testing platform, which can be realized in the form of hardware and / or software.

[0039] As shown in Figure 1 The cross-system user interface automation testing method provided by the first embodiment can include the following steps:

[0040] S101, obtaining a target test script and a target client selected by a user, and converting the target test script into a system-independent universal control operation instruction, the target test script being written in a unified service description language.

[0041] In the embodiment, a client+server architecture is adopted to implement cross-system UI automatic testing of Android, IOS and Harmony systems. The client is an application to be tested running on the Android, IOS and Harmony systems. The Android client refers to an application to be tested running on the Android system, the IOS client refers to an application to be tested running on the IOS system, and the Harmony client refers to an application to be tested running on the Harmony system. The server is an automatic testing platform based on HTML5 technology, namely a web platform tool. The mobile end UI automatic testing refers to writing an automatic script through a specific tool and framework to simulate user interaction with the application interface on the mobile device, such as clicking a button, sliding a screen, inputting text, selecting a menu, etc., to automatically verify whether the function, interface layout and interaction effect of the mobile application UI meet the expected design and functional requirements.

[0042] In the embodiment, the automatic testing platform provides a web interface for a tester to write a test script, configure a test task and view a test result. The user can write a test script through the web interface to define operation logic and expected results. In addition to writing a test script by the user, a recording method can also be used, that is, the user records operation data when operating the client, and converts the recorded operation data into a test script based on the operation data. In the embodiment, the test script is written in a unified service description language (USDL). The USDL describes various aspects of a service in a standardized and unified manner, so that different systems, platforms and applications can better understand and interact with the service.

[0043] In the embodiment, the test script selected by the user to be used is recorded as a target test script, and the client to be tested is recorded as a target client. The target test script and the target client selected by the user are obtained, and the target test script is converted into a general control operation instruction independent of the system, that is, the target test script is parsed, and the USDL script is converted into an operation instruction independent of the system.

[0044] S102, dynamically selecting a control path parsing method or an image recognition method as a target parsing method according to a current running environment of the target client.

[0045] In this embodiment, the control path analysis method can be understood as a method of locating the control through the path analysis engine, the image recognition method can be understood as a method of locating the control through the image recognition engine, and the target analysis method can be understood as finally determining the analysis method to be used in practice. The analysis method includes two analysis methods of control path analysis method and image recognition method, and the optimal positioning strategy is dynamically selected according to the running environment. This step can be understood as being executed by the intelligent adaptation decision layer.

[0046] Specifically, the running environment of the target client is detected in real time, denoted as the current running environment, and the running environment includes system type, control state, rendering mode, etc. According to the environment perception result, the control path analysis method or the image recognition method is dynamically selected as the target analysis method. According to the system (such as Android, IOS and Hongmeng) running on the target client, as well as the control state, the rendering model of the control, the control path analysis or image recognition engine is selected to execute the operation, that is, the control path analysis method or the image recognition method is selected as the target analysis method.

[0047] For example, the current device information is obtained, such as the system is Hongmeng 3.0, and the control is in a dynamic rendering state. At this time, the control path analysis method is preferentially called to execute the operation, and if it is detected that the control attribute is unstable or the system difference causes the positioning to fail, it is seamlessly switched to the image recognition method. At the same time, the switching time can also be recorded and the multi-terminal mapping rule library can be optimized to improve the decision efficiency of subsequent tests. The multi-terminal mapping rule library is used to record the mapping relationship between the unified control path and the control path of the control on the client.

[0048] S103, using the target analysis method, mapping the universal control operation instruction into a native control operation instruction suitable for the target client, the native control operation instruction containing the control path of the target control on the target client.

[0049] Multi-terminal mapping interface: a mechanism used in software development to realize data interaction and function docking between different terminal devices (such as mobile terminal, desktop terminal, Web terminal, etc.) and servers or other systems. It is mainly responsible for converting, adapting and mapping the requests and data of different terminals, so that each terminal can communicate with the backend system in a unified way, and the consistency and synchronization of data between different terminals can be maintained.

[0050] When the target analysis method is the control path analysis method, the step of mapping the general control operation instruction to the native control operation instruction suitable for the target client by using the target analysis method can be described as: analyzing the general control operation instruction, extracting the control hierarchy relationship and semantic label of the target control; determining the uniform control path corresponding to the target control according to the control hierarchy relationship and semantic label of the target control; mapping the uniform control path corresponding to the target control to the control path of the target control in the target client through the multi-terminal mapping interface; and generating the native control operation instruction suitable for the target client based on the control path of the target control in the target client.

[0051] When the target analysis method is the image recognition method, the step of mapping the general control operation instruction to the native control operation instruction suitable for the target client by using the target analysis method can be described as: intercepting a current interface image of the target client; determining whether the current interface image is successfully matched with a template image according to the similarity between the current interface image and each template image in an image feature library; if the matching is successful, obtaining the control path bound by the successfully matched template image as the control path of the target control in the target client, and generating the native control operation instruction containing the control path; and if the matching fails, triggering a retry or prompting manual intervention.

[0052] S104, sending the native control operation instruction to the target client, so that the target client locates the target control according to the control path and executes the native control operation instruction on the target control, to realize the automatic test.

[0053] In the embodiment, the server establishes a connection with the client through an http or WebSocket protocol, sends the native control operation instruction, the client executes corresponding operations after receiving the instruction, such as clicking a button and inputting text, and returns an execution result. The http and WebSocket are two communication protocols. Specifically, the server sends the native control operation instruction to the target client, the native control operation instruction includes the control path corresponding to the control to be operated, the target control can be located based on the control path, and then the native control operation instruction is executed on the target control, the client returns a test execution result, so as to realize the automatic test of the target client. The test execution result, such as a test report, is returned to the server, and can be presented to the user on the interface.

[0054] To cope with the mobile ecology of multiple systems (IOS / Android / Harmony) coexisting, in the embodiment, a three-end integrated UI automatic testing method based on HTML5 cross-platform technology is adopted, a triple-engine cooperative mechanism of "control path + image recognition + multi-end mapping" is adopted, cross-end adaptation capability and testing intelligent level are improved, repeated development cost is reduced, and application efficient iteration and quality assurance are enabled. Through the bimodal cooperative mechanism of fusing the control path analysis engine and the image recognition engine, combined with the intelligent adaptation decision layer, seamless compatible testing of Android, IOS and Harmony systems is realized.

[0055] For a clearer description of how to perform automatic testing on the client based on the server, Figure 2 An example diagram of a client and a server architecture in a cross-system user interface automatic testing method provided for the first embodiment of the application is shown in Figure 2 As shown, the server is an automatic testing platform, including: a test management module, providing a Web interface for test personnel to write test scripts, configure test tasks and view test results; a script analysis engine, analyzing test scripts (based on USDL unified script description language) to generate system-independent operation instructions; a dual-engine system module, including a control path analysis engine and an image recognition engine, dynamically selecting the optimal positioning strategy according to the running environment; a recording and playback module, based on JavaScript and native application interaction, recording user operations and generating reusable test scripts. The client is the application under test, the Android client refers to the application under test running on the Android system; the IOS client refers to the application under test running on the IOS system; and the Harmony client refers to the application under test running on the Harmony system.

[0056] The above technical solution uses a unified service description language independent of the system to write test scripts, analyzes the test scripts to generate system-independent general control operation instructions, and then intelligently selects a control path analysis method or an image recognition method to convert the standardized control expression into native control operation instructions applicable to the target client, thereby enabling automatic testing of clients on different systems. The test script is developed once, and multiple system testing can be used, which improves cross-system compatibility and reduces the cost of script development and maintenance.

[0057] As an optional embodiment of the application, on the basis of the above embodiment, when the control path analysis method is used to analyze the target control, the method further includes:

[0058] If the control path analysis method fails to locate the target control, the image recognition method is switched as the target analysis method.

[0059] In the embodiment, when the control path analysis fails, the image recognition mode is switched as the target analysis mode. For example, the entire playback test process is equivalent to a test script based on recording, and the operation is performed on the target system through the double-engine cooperative mechanism and the result is verified. The playback process can be described as follows: (1) the user selects the recorded test script on the server and specifies the target system; (2) the server sends the script to the target client and starts the playback task; (3) the client attempts to locate the target control and perform the operation according to the control path expression in the script; (4) if the target control path fails to locate, the image recognition mode is switched to, and the target control is located and the operation is performed by comparing the pre-stored image template with the current interface; (5) the operation result is verified and the test report is generated.

[0060] Specifically, when the control path analysis fails, the image recognition module is triggered, the current interface is intercepted, and the key area features are extracted. Through the multi-scale template matching algorithm, the real-time screenshot is compared with the pre-stored template image for similarity calculation to locate the target control. If the matching is successful, the simulation operation instruction is generated, such as coordinate-based clicking; if the matching fails, the feedback is fed back to the intelligent adaptive decision layer for strategy optimization.

[0061] The above technical solution increases the image recognition mode as the target analysis mode when the control path analysis mode fails, uses the image recognition mode as a substitute, and realizes the accurate positioning of the target control.

[0062] Embodiment two

[0063] Figure 3 The flowchart of another cross-system user interface automatic test method provided by the embodiment two of the application is shown. The embodiment is a further optimization of the above-mentioned embodiment. In the embodiment, the "dynamically selecting the control path analysis mode or the image recognition mode as the target analysis mode according to the current running environment of the target client" is further optimized, and the "mapping the universal control operation instruction into the native control operation instruction suitable for the target client by using the target analysis mode" is further optimized.

[0064] As shown in the embodiment two, the cross-system user interface automatic test method comprises the following steps: Figure 3

[0065] S201, obtaining the target test script selected by the user and the target client, and converting the target test script into a universal control operation instruction independent of the system, wherein the target test script is written in a unified service description language.

[0066] S202, obtaining the first historical index score of the control path analysis mode on each evaluation index, and the second historical index score of the image recognition mode on each evaluation index.​

[0067] In this embodiment, the optimal method is selected by scoring the historical performance of the control path analysis method and the image recognition method, and dynamically adjusting the weights according to the scoring results. The scoring indicators include historical success rate, execution efficiency, and resource consumption, etc. The historical success rate refers to the positioning success rate of the engine in similar scenarios. The execution efficiency refers to the positioning time consumption of the engine (such as the control path is usually faster than image recognition). The resource consumption refers to the computing resource occupation of the engine (such as image recognition requires higher CPU / GPU). In this embodiment, the dynamic switching logic and decision algorithm of the control path and image recognition mainly adopt a decision algorithm based on weighted scoring, and dynamically optimize it combined with the reinforcement learning idea in machine learning. The following describes the dynamic switching logic and decision algorithm of the control path and image recognition.

[0068] First, for each evaluation indicator, the indicator score of the control path analysis method in the historical time period is obtained, denoted as the first historical indicator score. At the same time, the indicator score of the image recognition method in the historical time period is obtained, denoted as the second historical indicator score.

[0069] S203, according to the current running environment of the target client, the first weight corresponding to each evaluation indicator of the control path analysis method is allocated, and the second weight corresponding to each evaluation indicator of the image recognition method is allocated.

[0070] This step is used for weight allocation, and the weights are dynamically adjusted according to the current running environment of the target client. The current running environment includes system type, control state, rendering mode, etc. For example, in a static interface, the weight of the control path engine is higher (efficiency first). In a dynamic interface, the weight of the image recognition engine is higher (stability first).

[0071] Specifically, according to the current running environment of the target client, the weights of each evaluation indicator of the control path analysis method are allocated, and the weights of each evaluation indicator corresponding to the control path analysis method are denoted as the first weight. The weights of each evaluation indicator of the image recognition method are allocated, and the weights of each evaluation indicator corresponding to the image recognition method are denoted as the second weight.

[0072] To further improve the accuracy of decision-making, the reinforcement learning idea is introduced in this embodiment, and the weight distribution strategy is optimized through continuous interaction with the environment. The core logic is as follows: (1) State: the current running environment (system type, control state, rendering mode, resolution, etc.); (2) Action: select the control path engine or the image recognition engine; (3) Reward: successful positioning: +1, failed positioning: -1, positioning time is shorter: +0.5, resource consumption is lower: +0.5; (4) Strategy optimization: through model-free reinforcement learning algorithm (Q-learning) or deep Q network (DQN) and other reinforcement learning algorithms, the weight distribution strategy is continuously optimized to maximize the long-term reward.

[0073] The application process and dynamic switching logic are as follows: initialization stage: the environment perception module detects the running environment, and initializes the scoring mechanism and weight distribution. Positioning stage: select the optimal engine (i.e. control path or image recognition) according to the weighted score; if the control path engine fails, trigger the image recognition engine. Result feedback and optimization: record the positioning result (i.e. success / failure, time consumption, resource consumption). Through reinforcement learning, the weight distribution strategy is optimized to improve the accuracy of subsequent decision-making.

[0074] S204, according to the first historical index score and the first weight, determine the first comprehensive score of the control path analysis method.

[0075] In this embodiment, the comprehensive scores of the control path analysis method and the image recognition method are calculated respectively. The comprehensive score of each method is calculated: Score = w1 x success rate + w2 x efficiency + w3 x resource consumption, where w1, w2, and w3 are weight coefficients, which are dynamically adjusted according to the scene.

[0076] Specifically, the first historical index score and the first weight of the control path analysis method are substituted into the comprehensive score calculation formula, and the result is recorded as the first comprehensive score.

[0077] S205, according to the second historical index score and the second weight, determine the second comprehensive score of the image recognition method.

[0078] Specifically, the second historical index score and the second weight of the control path analysis method are substituted into the comprehensive score calculation formula, and the result is recorded as the second comprehensive score.

[0079] S206, the method corresponding to the higher value of the first comprehensive score and the second comprehensive score is used as the target analysis method.

[0080] Specifically, the higher value in the first comprehensive score and the second comprehensive score is taken as the target analysis mode. For example, if the first comprehensive score is greater than the second comprehensive score, the control path analysis mode is taken as the target analysis mode. If the second comprehensive score is greater than the first comprehensive score, the image recognition mode is taken as the target analysis mode.

[0081] S207, analyze the general control operation instruction, and extract the control level relationship and semantic label of the target control.

[0082] In this embodiment, how to realize the positioning of the target control and the generation of the native operation instruction when the target analysis mode is the control path analysis mode is described.

[0083] The Document Object Model (DOM) structure is a programming structure used to represent and manipulate the structure of HTML, XML and other documents. It parses the document into a tree structure composed of nodes and objects, and developers can access and manipulate these nodes through JavaScript and other scripting languages, thereby dynamically modifying the content, structure and style of the document. HTML5 is a language description method for building web content.

[0084] In this embodiment, the general control operation instruction is analyzed, that is, the DOM structure of the HTML5 application is analyzed, and the control level relationship and semantic label are extracted. The control level relationship can be understood as the layer of the control in the DOM structure. The semantic label refers to an HTML element with a clear meaning, which can clearly describe the structure and function of the content it contains.

[0085] For example, ArkUI is the UI framework of the Harmony system, and its control system is significantly different from traditional Android / iOS. In order to realize the automatic testing of the Harmony system, the application adopts the ArkUI control mapping method to associate the HTML5 control path with the Harmony native control. The core principle is as follows: (1) Control identification analysis: the Harmony ArkUI control is identified by a unique identifier (such as ohos:id), similar to the resource-id of Android and the accessibility-id of iOS. The DOM structure of the HTML5 application is analyzed, and the level relationship and semantic label (such as div, button) of the control are extracted.

[0086] S208, according to the control level relationship and semantic label of the target control, determine the uniform control path corresponding to the target control.

[0087] In this embodiment, according to the control hierarchy relationship and semantic label of the target control, a standardized control path expression is generated, denoted as a uniform control path corresponding to the target control, such as / Page / Container[1] / Button[@text=‘Submit’], which supports cross-platform reuse.

[0088] S209, through the multi-end mapping interface, mapping the uniform control path corresponding to the target control to the control path of the target control in the target client.

[0089] In this embodiment, through the multi-end mapping interface, the multi-end mapping interface includes the interfaces of Android, IOS and Harmony, which are AndroidAccessibility, IOS XCUITest and Harmony ArkUI respectively, the native attributes of the target control are obtained, and the uniform control path corresponding to the target control is mapped to the control path of the target control in the client.

[0090] This step is equivalent to constructing a mapping rule library: a mapping rule library of HTML5 control path and Harmony ArkUI control is established. For example:

[0091] HTML5 path: / Page / Container[1] / Button[@text=′Submit′]

[0092] Harmony ArkUI path: / / ArkUIView[@ohos:id=′submit_button′]

[0093] Through the multi-end mapping interface (AndroidAccessibility, iOS XCUITest, and Harmony ArkUI), the HTML5 control path is mapped to the Harmony native control operation instruction.

[0094] Further, according to the version of the Harmony system (such as HarmonyOS2.0, 3.0) and the characteristics of the ArkUI framework, the mapping rules can be dynamically adjusted and dynamically adapted to ensure compatibility.

[0095] S210, based on the control path of the target control in the target client, generating a native control operation instruction suitable for the target client.

[0096] In this embodiment, the control path of the target control on the target client is obtained. Knowing the control path of the target control on the target client is equivalent to knowing the location of the target control, and then native control operation instructions are generated based on the location of the target control. That is, DOM elements are mapped to native control operation instructions through multi-platform mapping interfaces (AndroidAccessibility, iOS XCUITest, HarmonyOS ArkUI).

[0097] For example, the HarmonyOS system's operation command generation method, based on the distributed capabilities of the ArkUI framework and the features of the Ark Compiler, designs a lightweight instruction set to simulate user operations (such as clicking, swiping, and inputting). Its core principle is as follows:

[0098] (1) Event Injection Interface: HarmonyOS provides a rich set of event injection interfaces (such as UiDriver) for simulating user operations. By interacting with native applications through JavaScript, HarmonyOS's event injection interfaces are called to generate operation commands.

[0099] (2) Instruction Set Design: Click Instruction: Simulates a user click operation, calling the UiDriver.click() method. Swipe Instruction: Simulates a user swipe operation, calling the UiDriver.swipe() method. Input Instruction: Simulates a user input operation, calling the UiDriver.setText() method.

[0100] (3) Instruction optimization: Based on the performance characteristics of HarmonyOS (such as distributed capabilities and Ark Compiler optimization), the operation instructions are designed to be lightweight to improve execution efficiency.

[0101] It achieves native support for HarmonyOS, enabling control mapping and operation command generation within the HarmonyOS Ark framework.

[0102] S211. The native control operation instruction is sent to the target client, so that the target client locates the target control according to the control path and executes the native control operation instruction on the target control to achieve automated testing.

[0103] The technical solution above specifically realizes the steps of dynamically selecting the control path analysis mode or the image recognition mode, and how to realize the positioning of the target control and the generation of the native control operation instruction when the target analysis mode is the control path analysis mode. The limitations in cross-system compatibility, dynamic adaptability, and test script reuse rate in mobile application testing are considered, and a new UI automation testing scheme for automatically adapting Android, IOS, and Harmony three terminals is provided. By analyzing the DOM structure of the HTML5 application, a standardized control expression independent of the system is generated, the control path analysis mode or the image recognition mode is dynamically selected according to the environmental perception result, and the optimal mode is intelligently adapted. On this basis, a multi-scale template matching algorithm is fused to locate the dynamic interface or the custom control, and based on JavaScript and the native application, interaction is carried out, USDL script abstraction encapsulation is introduced, data transmission and function calling with the tested application are realized, seamless compatible testing of Android, IOS, and Harmony is supported, and “one development, multi-terminal running” is achieved. The problems in the prior art are solved, the cross-system compatibility is significantly improved, the dynamic scene support capability is enhanced, the script maintenance cost is reduced, and the boundary of the Harmony native support capability is expanded.

[0104] Embodiment three

[0105] Figure 4 The flowchart of another cross-system user interface automation testing method provided by Embodiment Three of the application is provided, and this embodiment is a further optimization of the above-mentioned embodiment. In this embodiment, the step of “mapping the universal control operation instruction to a native control operation instruction suitable for the target client by using the target analysis mode” is further optimized.

[0106] As shown in Figure 4 , Embodiment Three provides a cross-system user interface automation testing method, which specifically includes the following steps:

[0107] S301, obtaining a target test script selected by a user and a target client, and converting the target test script into a universal control operation instruction independent of the system, wherein the target test script is written in a unified service description language.

[0108] S302, dynamically selecting a control path analysis mode or an image recognition mode as a target analysis mode according to a current running environment of the target client.

[0109] S303, intercepting a current interface image of the target client.

[0110] The embodiment describes how to realize the positioning of the target control and the generation of the native operation instruction when the target analysis mode is an image recognition mode. Specifically, an image recognition engine is triggered, and the interface of the target client at the current time is intercepted, denoted as a current interface image.

[0111] S304, according to the similarity of the current interface image and each template image in the image feature library, it is determined whether the current interface image is matched successfully with the template image.

[0112] In order to adapt to dynamic interface control positioning and cross-system compatibility enhancement, a multi-scale template matching algorithm is used in the embodiment, that is, the current interface image is scaled in multiple scales to generate images of different proportions, denoted as to-be-matched images. All pre-stored template images and their multi-scale versions are traversed, and the normalized cross-correlation algorithm is used to calculate the similarity of each template image and each to-be-matched image. Then, based on the similarity of the current interface image and the template image, the template image with the highest similarity is selected, and the similarity is compared with the set similarity threshold value. Based on the comparison result, it is determined whether there is a template image matched successfully with the current interface image. The pre-stored template images come from the standardized image feature library or the Hongmeng Ark feature library.

[0113] As a specific implementation, the step of determining whether the current interface image is matched successfully with the template image according to the similarity of the current interface image and each template image in the image feature library can be optimized, comprising:

[0114] a1) scaling the current interface image in multiple scales to generate to-be-matched images of different proportions.

[0115] In the embodiment, a multi-scale matching algorithm is used. First, the current interface image is scaled in multiple scales, such as 0.8x, 1.0x, and 1.2x, to generate to-be-matched images of different proportions, and template image matching is performed under different scales.

[0116] b1) using the normalized cross-correlation matching algorithm to calculate the similarity of each to-be-matched image and each template image in the image feature library.

[0117] Among them, the normalized cross-correlation matching (Normalized Cross-Correlation, NCC): by calculating the normalized cross-correlation value of the template image and the target image, the similarity of the two is measured. It is robust to illumination changes and contrast differences. The formula for calculating the similarity using the normalized cross-correlation matching algorithm is:

[0118] R(x,y) =∑x',y'(T(x',y') - μT)(I(x+x',y+y') - μI) ∑x',y'(T(x',y') - μT)2∑x',y'(I(x+x',y+y') - μI)2R(x,y) =∑x',y'(T(x',y') - μT)2∑x',y'(I(x+x',y+y') - μI)2∑x',y'(T(x',y') - μT)(I(x+x',y+y') - μI)

[0119] Wherein, T is a template image, I is a target image, μT and μI are the mean of the template and target respectively, (x, y) is a pixel point in the target image, (x', y') is a pixel point in the template image.

[0120] Specifically, a multi-scale template image matching algorithm is adopted, and the similarity of each to-be-matched image and each template image in the image feature library is calculated based on a normalized cross-correlation matching algorithm, and template matching is performed at different scales.

[0121] c1) determining whether the highest similarity is greater than a set similarity threshold.

[0122] The set similarity threshold can be understood as a threshold for determining whether two images are similar, and can be set according to actual conditions. For example, the set similarity threshold can be a value between 0 and 1, and is usually set to be greater than or equal to 0.85. Specifically, the highest similarity is selected from the similarity of each to-be-matched image and each template image, and it is determined whether the highest similarity is greater than the set similarity threshold.

[0123] d1) If greater, it is determined that the template image corresponding to the highest similarity is successfully matched with the current interface image.

[0124] Specifically, if the highest similarity is greater than the set similarity threshold, it is determined that the template image corresponding to the highest similarity is successfully matched with the current interface image, which is equivalent to realizing the positioning of the target control, and selecting the region with the highest similarity score as the position of the target control.

[0125] e1) Otherwise, the matching fails.

[0126] If the highest similarity is less than or equal to the set similarity threshold, it is determined that the template and the current interface image fail to match, indicating that the similarity of each to-be-matched image and each template image is low.

[0127] The above technical solution specifically realizes the step of determining whether the current interface image and each template image are matched by calculating the similarity of the current interface image and each template image. By scaling the current interface image at different scales and using different scale matching algorithms, different resolutions or interface scaling ratios are adapted, and cross-system compatibility is improved.

[0128] S305, if the matching is successful, obtaining the control path bound by the template image of the successful matching as the control path of the target control on the target client, and generating a native control operation instruction containing the control path.

[0129] Specifically, if the matching is successful, the control path bound by the template image of the successful matching is obtained as the control path of the target control on the target client. After the control path of the target control on the target client is known, the coordinate position of the target control is known, and then a coordinate-based operation instruction, such as clicking and sliding, is generated as the native control operation instruction.

[0130] S306, if the matching fails, triggering retry or manual intervention.

[0131] Specifically, if the matching fails, feedback to the intelligent adaptive decision layer, triggering retry or manual intervention.

[0132] S307, sending the native control operation instruction to the target client, so that the target client locates the target control according to the control path and executes the native control operation instruction on the target control, realizing automatic testing.

[0133] The above technical solution specifically realizes how to realize the positioning of the target control and the generation of the native control operation instruction when the target analysis method is the image recognition method, provides another control positioning analysis method as a substitute for the control path analysis method, and realizes accurate positioning of the target control when the current running environment is suitable for using the image recognition method or the control positioning analysis method fails.

[0134] As an optional embodiment of the embodiment of the application, on the basis of the above embodiment, the construction step of the image feature library can be optimized, including:

[0135] a2) When the user operates the control of the client, the interface screenshot of the control is intercepted as a template image.

[0136] In this embodiment, the automation recording and playback of user operations are realized through the interaction of the programming language of Web (such as JavaScript) and the native application (i.e. the client). This function combines the control path analysis and image recognition technology, and can run seamlessly on Android, IOS and Hongmeng.

[0137] In the recording stage, when the user operates the target control, the user operation is captured, a system application programming interface (API) is called to intercept an interface screenshot of the control region as a template image. For example, the control region can be a button icon, and the interface screenshot of the button icon is intercepted as the template image. When the user tests the client, the user will operate multiple controls of the client in turn, and the interface screenshots of the controls are intercepted as the template images. This step is equivalent to intercepting the current interface and saving it as an image template when the user records the operation, for comparison in the image recognition mode.

[0138] b2) pre-processing the template image to obtain a processed template image.

[0139] This step is used to pre-process each image template, such as cropping, grayscale, noise reduction, edge detection, and extraction of key features of the target region, such as shape, edge contour, etc., to obtain a processed template image. Grayscale is a process of converting a color image into a grayscale image. Each pixel in a color image is composed of three color channels: red, green, and blue. A single numerical value representing the grayscale value is obtained by performing weighted summation or other operations on the values of the three channels through a specific algorithm, so that the image appears in black and white, which is helpful for subsequent image analysis and processing.

[0140] c2) multi-scale scaling of the processed template image to generate template images of different scales.

[0141] In this embodiment, the processed template image is multi-scale scaled to generate multi-scale versions, i.e., template images of different scales. For example, different scales can be represented as 0.8x, 1.0x, and 1.2x. 0.8x means that the original template image is reduced to 0.8 times, 1.0x means the original template image, and 1.2x means that the original template image is enlarged to 1.2 times. This ensures coverage of different resolution scenarios and realizes multi-scale template storage.

[0142] d2) binding each template image to the control path of the control in the client and storing it in the image feature library.

[0143] In this embodiment, each template image is bound to the control path of the corresponding control in the client and the control attribute, and is stored in the image feature library, e.g., marked as No. 15. Further, a dynamic optimization step can be set, i.e., template updating: if the matching fails but the control exists as confirmed by a human, the template in the image feature library is updated to optimize the subsequent matching accuracy. Further, parameter adjustment can be performed, i.e., dynamic adjustment of the scaling range or the similarity threshold according to the historical matching results to improve the algorithm efficiency.

[0144] The technical solution above specifically realizes the construction steps of the image feature library. When subsequent control positioning is performed based on the image recognition method, the template image in the image feature library can be matched to provide a basis for the use of the subsequent image recognition method.

[0145] Optionally, the control path determination step of the client includes:

[0146] a3) extracting the control identification number of the control from the recorded operation data of the user on the control in the client.

[0147] In this embodiment, the operation of the user on the application interface is captured in real time through the interaction of the programming language (such as JavaScript) of the Web with the native application (i.e., the client). The user starts the recording function on the client (Android / IOS / Harmony). The server establishes a connection with the client through WebSocket. The user operates on the application interface. The operation capture module captures the operation type and control information through the JavaScript event listener. The captured operation data, such as the operation type, control ID, operation timestamp, etc., is sent to the server.

[0148] In this embodiment, the attribute information of the control, such as ID, Xpath, and text content, is extracted to generate the control path expression. The control identification number of the control is extracted from the recorded operation data of the user on the control in the client.

[0149] b3) extracting the control hierarchical relationship and semantic label of the control based on the control identification number of the control.

[0150] In this embodiment, based on the control identification number (ID) of the provided control, the DOM structure of the HTML5 application is parsed to extract the hierarchical relationship and semantic label of the control. The DOM structure is a tree structure, and each control is different, with a parent and a child. For example, the child corresponds to a control ID of 1. Through the control ID, it is known that it belongs to the nth layer of the DOM structure, and it is known that the control hierarchical relationship and semantic label of the control are extracted.

[0151] c3) determining the uniform control path corresponding to the control according to the control hierarchical relationship and semantic label of the control.

[0152] In this embodiment, according to the control hierarchical relationship and semantic label of the control, a standardized control path expression is generated, which is denoted as the uniform control path corresponding to the control. The uniform control path is independent of the system and can be stored in the control path library.

[0153] d3) mapping, by the multi-terminal mapping interface, the uniform control path corresponding to the target control to the control path of the control in the client.

[0154] In this embodiment, through the multi-terminal mapping interface, the multi-terminal mapping interface includes the interfaces of Android, IOS and Harmony, that is, AndroidAccessibility, IOS XCUITest and Harmony ArkUI, the native attributes of the control are obtained, and the uniform control path corresponding to the target control is mapped to the control path of the control in the client.

[0155] The technical solution described above specifically determines the control path of the control in the client, and provides a basis for subsequent image recognition control path based on the control path.

[0156] As an optional embodiment of the embodiment, on the basis of the above-mentioned embodiment, the test script can be optimized to be written by a user or generated based on recorded operation data of the user on the control in the client.

[0157] In this embodiment, the constructed automatic test platform provides a Web interface for a test personnel to write a test script, configure a test task and view a test result. The user can write the test script, define operation logic and expected results through the Web interface. In addition to the user writing the test script, a recording method can also be used, that is, when the user operates the client, the operation data is recorded, and the recorded operation data of the user on the control in the client is converted into a test script based on the operation data.

[0158] In this embodiment, the generated test script can be further optimized to improve its cross-system compatibility and execution efficiency. Specifically, the control path expression in the script is analyzed to check whether there is a cross-system compatibility problem, such as inconsistent control attributes; for the script segment with the compatibility problem, a bottom logic of image recognition is added; the script is compressed and de-duplicated to reduce redundant operations and improve execution efficiency.

[0159] The technical solution described above specifically realizes two ways of generating a test script.

[0160] Further, based on the recorded operation data of the user on the control in the client, the test script is generated, including:

[0161] a4) extracting the operation type of the control from the recorded operation data of the user on the control in the client.

[0162] In this embodiment, the user operation is recorded and a reusable test script is generated through the interaction of JavaScript and a native application. According to the provided recorded operation data of the user on the control in the client, the operation type is obtained therefrom. For example, the operation type is a single click.

[0163] b4) generating a script fragment of the control according to the operation type of the control and the control path of the control in the client.

[0164] In the embodiment, the control path of the control in the client also needs to be acquired. After the control path of the control in the client is known, the position of the control can be determined, and the control can be located. After the operation type of the control is known, the operation to be performed on the control can be determined. Based on the operation type and the control path, the script fragment of the control can be generated. For example, the script fragment of clicking the control can be represented as (click(Button@Submit)).

[0165] c4) combining the script fragments according to the operation sequence of the user to generate a complete test script.

[0166] Specifically, the script fragments are combined according to the operation sequence to generate a complete test script.

[0167] The technical solution above specifically implements the step of converting the recorded operation data into a test script based on USDL, and realizes the generation of the test script in the recording mode.

[0168] For example, in order to make the cross-system user interface automatic test method provided by the embodiment of the application clearer, Figure 5 The example flowchart of the cross-system user interface automatic test method provided by the third embodiment of the application is shown in FIG. 3. Figure 5 As shown in FIG. 3, the server provides a test management function, provides a Web interface for a tester to write a test script, configure a test task, and view a test result, and also provides a function of analyzing a test script (based on USDL, a unified script description language) to generate operation instructions irrelevant to a system. An intelligent adaptive decision layer dynamically selects a control path analysis engine or an image recognition engine, and a double-engine cooperation module provides the control path analysis engine or the image recognition engine. The double-engine cooperation module processes and sends native control operation instructions to a client, such as an Android client, an IOS client, or a HongMeng client, to realize automatic test and generate a test report. When the tester performs a test operation on the client, the operation data can also be captured through a recording function, and a script can be generated according to the operation data. An image template can also be generated according to a screenshot in the captured data, and the image template is bound with a control attribute and stored in an image feature library. The test script recorded can also be played back to perform automatic test.

[0169] Embodiment three

[0170] Figure 6A structural schematic diagram of a cross-system user interface automatic test platform is provided for the third embodiment of the present application. The platform can be suitable for cross-system user interface automatic test. The cross-system user interface automatic test platform can be realized in the form of hardware and / or software. As shown in the figure, the platform comprises a script selection module 31, a double-engine module 32, an instruction generation module 33 and a test module 34, wherein, Figure 6

[0171] The script selection module 31 is configured to obtain a target test script and a target client selected by a user, and convert the target test script into universal control operation instructions independent of a system, wherein the target test script is written in a unified service description language.

[0172] The double-engine module 32 is configured to dynamically select a control path analysis method or an image recognition method as a target analysis method according to a current running environment of the target client.

[0173] The instruction generation module 33 is configured to map the universal control operation instructions into native control operation instructions suitable for the target client by using the target analysis method, wherein the native control operation instructions contain a control path of a target control in the target client.

[0174] The test module 34 is configured to send the native control operation instructions to the target client, so that the target client locates the target control according to the control path and executes the native control operation instructions on the target control, thereby realizing automatic test.

[0175] The above technical solution uses the unified service description language independent of the system to write the test script. By analyzing the test script, the universal control operation instructions independent of the system are generated. Then, according to the current running environment, the control path analysis method or the image recognition method is intelligently selected to convert the standardized control expression into the native control operation instructions suitable for the target client, thereby realizing the automatic test on the client of different systems. The test script development is realized once, and the effect of multi-system test can be used. The cross-system compatibility is improved, and the cost of script development and maintenance is reduced.

[0176] Optionally, the double-engine module 32 is specifically configured to:

[0177] Obtain a first historical index score of the control path analysis method with respect to each evaluation index, and a second historical index score of the image recognition method with respect to each evaluation index.

[0178] According to the current running environment of the target client, a first weight corresponding to each evaluation index of the control path analysis method and a second weight corresponding to each evaluation index of the image recognition method are allocated.​

[0179] determine a first comprehensive score of the control path resolution manner according to the first historical index score and the first weight;

[0180] determine a second comprehensive score of the image recognition manner according to the second historical index score and the second weight;

[0181] take the manner corresponding to the higher value between the first comprehensive score and the second comprehensive score as the target resolution manner.

[0182] Optionally, the apparatus further comprises a switching module configured to, when the target resolution manner is the control path resolution manner:

[0183] if the target control fails to be located by using the control path resolution manner, switch the image recognition manner as the target resolution manner.

[0184] Optionally, the instruction generation module 33 is configured to, when the target resolution manner is the control path resolution manner:

[0185] resolve the universal control operation instruction, and extract the control hierarchical relationship and semantic label of the target control;

[0186] determine the uniform control path corresponding to the target control according to the control hierarchical relationship and semantic label of the target control;

[0187] map the uniform control path corresponding to the target control to the control path of the target control in the target client through a multi-end mapping interface;

[0188] generate the native control operation instruction applicable to the target client based on the control path of the target control in the target client.

[0189] Optionally, the instruction generation module 33 is configured to, when the target resolution manner is the image recognition manner:

[0190] specifically configured to:

[0191] an image intercepting unit configured to intercept a current interface image of the target client;

[0192] a matching unit configured to determine whether the current interface image is matched successfully with a template image in the image feature library according to the similarity between the current interface image and the template image;

[0193] an instruction generation unit configured to, if matched successfully, acquire the control path bound by the template image matched successfully as the control path of the target control in the target client, and generate a native control operation instruction containing the control path.

[0194] a retry unit, configured to trigger a retry or manual intervention if the matching fails.

[0195] Optionally, the matching unit is specifically configured to:

[0196] perform multi-scale scaling on the current interface image to generate different scale images to be matched;

[0197] adopt a normalized cross-correlation matching algorithm to respectively calculate similarities of each of the images to be matched and each of the template images in the image feature library;

[0198] determine whether the highest similarity is greater than a set similarity threshold;

[0199] if yes, it is determined that the template image corresponding to the highest similarity is successfully matched with the current interface image;

[0200] otherwise, the matching fails.

[0201] Optionally, the device further comprises a feature library construction module, which is specifically configured to:

[0202] when a user operates a control of a client, intercept an interface screenshot of the control as a template image;

[0203] perform preprocessing on the template image to obtain a processed template image;

[0204] perform multi-scale scaling on the processed template image to generate different scale template images;

[0205] bind each of the template images and a control path of the control in the client, and store them in the image feature library.

[0206] Optionally, the device further comprises a path determination module, which is configured to:

[0207] extract a control identification number of the control from recorded operation data of the user on the control in the client;

[0208] extract a control hierarchical relationship and a semantic label of the control based on the control identification number of the control;

[0209] determine a uniform control path corresponding to the control according to the control hierarchical relationship and the semantic label of the control;

[0210] map the uniform control path corresponding to the target control to a control path of the control in the client through a multi-end mapping interface.

[0211] Optionally, the test script is written by a user or generated based on recorded operation data of the user on the control in the client.

[0212] Further, the device comprises a script generation module, configured to:

[0213] extract the operation type of the control from the recorded operation data of the user on the control in the client;

[0214] generate a script segment of the control according to the operation type of the control and the control path of the control in the client;

[0215] combine each script segment according to the operation sequence of the user to generate a complete test script.

[0216] The cross-system user interface automatic test platform provided by the embodiments of the present application can execute the cross-system user interface automatic test method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0217] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present application can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0218] The specific embodiments described above do not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A cross-system automated user interface testing method, characterized in that, include: The system obtains the target test script and target client selected by the user, and converts the target test script into system-independent general control operation instructions. The target test script is written in a unified service description language. Based on the current operating environment of the target client, dynamically select either the control path parsing method or the image recognition method as the target parsing method; Using the target parsing method, the general control operation instructions are mapped to native control operation instructions suitable for the target client, and the native control operation instructions include the control path of the target control on the target client; The native control operation instructions are sent to the target client, so that the target client can locate the target control according to the control path and execute the native control operation instructions on the target control to achieve automated testing.

2. The method according to claim 1, characterized in that, The step of dynamically selecting either the control path parsing method or the image recognition method as the target parsing method based on the current operating environment of the target client includes: Obtain the first historical index score of the control path parsing method for each evaluation indicator, and the second historical index score of the image recognition method for each evaluation indicator; Based on the current operating environment of the target client, assign a first weight to the control path parsing method and a second weight to the image recognition method for each evaluation indicator; Based on the first historical indicator score and the first weight, a first comprehensive score for the control path parsing method is determined; A second comprehensive score for the image recognition method is determined based on the second historical index score and the second weight. The method that corresponds to the higher value between the first comprehensive score and the second comprehensive score is used as the target analysis method.

3. The method according to claim 1, characterized in that, When the target parsing method is the control path parsing method, it also includes: If the target control cannot be located using the control path parsing method, then the image recognition method is switched to be the target parsing method.

4. The method according to claim 1, characterized in that, When the target parsing method is the control path parsing method; The step of using the target parsing method to map the general control operation instructions to native control operation instructions suitable for the target client includes: Parse the general control operation instructions and extract the control hierarchy and semantic tags of the target control; Based on the control hierarchy and semantic tags of the target control, determine the unified control path corresponding to the target control; Through the multi-terminal mapping interface, the unified control path corresponding to the target control is mapped to the control path of the target control on the target client; Based on the control path of the target control in the target client, native control operation instructions suitable for the target client are generated.

5. The method according to claim 1, characterized in that, When the target parsing method is image recognition; The step of using the target parsing method to map the general control operation instructions to native control operation instructions suitable for the target client includes: Capture the current interface image of the target client; Based on the similarity between the current interface image and each template image in the image feature library, determine whether the current interface image is successfully matched with the template image; If a match is successful, the control path bound to the successfully matched template image is obtained as the control path of the target control on the target client, and native control operation instructions containing the control path are generated. If a match fails, a retry will be triggered or a manual intervention will be prompted.

6. The method according to claim 5, characterized in that, The step of determining whether the current interface image matches the template image successfully based on the similarity between the current interface image and each template image in the image feature library includes: The current interface image is scaled at multiple scales to generate images to be matched at different ratios; A normalized cross-correlation matching algorithm is used to calculate the similarity between each of the images to be matched and each of the template images in the image feature library. Determine if the highest similarity score is greater than a set similarity threshold; If the similarity is greater than 1, then the template image corresponding to the highest similarity is determined to be a successful match with the current interface image. Otherwise, the match will fail.

7. The method according to claim 5, characterized in that, The steps for constructing the image feature library include: When a user interacts with a control on the client, a screenshot of the control's interface is captured and used as a template image. The template image is preprocessed to obtain the processed template image; The processed template image is scaled at multiple scales to generate template images of different proportions; Each template image is bound to the control path on the client and stored in the image feature library.

8. The method according to claim 7, characterized in that, The steps for determining the control's path on the client side include: Extract the control identifier from the recorded user interaction data of the controls in the client; Based on the control identifier of the control, extract the control hierarchy and semantic tags of the control; Based on the control hierarchy and semantic tags of the controls, determine the unified control path corresponding to the control; Through the multi-terminal mapping interface, the unified control path corresponding to the target control is mapped to the control path of the control on the client.

9. The method according to claim 1, characterized in that, The test script is written by the user or generated based on recorded user interaction data with controls in the client. Based on recorded user interaction data with controls in the client, the test script is generated, including: Extract the operation type of the control from the recorded user operation data of the control in the client; Based on the operation type of the control and the control path of the control on the client, generate a script fragment for the control; The script fragments are combined according to the user's operation sequence to generate a complete test script.

10. A cross-system user interface automated testing platform, characterized in that, include: The script selection module is used to obtain the target test script and target client selected by the user, and convert the target test script into system-independent general control operation instructions. The target test script is written in a unified service description language. The dual-engine module is used to dynamically select either the control path parsing method or the image recognition method as the target parsing method based on the current running environment of the target client. The instruction generation module is used to map the general control operation instructions into native control operation instructions suitable for the target client using the target parsing method. The native control operation instructions include the control path of the target control on the target client. The testing module is used to send the native control operation instructions to the target client, so that the target client can locate the target control according to the control path and execute the native control operation instructions on the target control, thereby realizing automated testing.

Images