Hybrid architecture application testing method and computer program product

Abstract

The present disclosure relates to the technical field of application program testing, in particular to a hybrid architecture application program testing method and computer program product, comprising: obtaining a hot area coordinate of a target application program; using an operation control hook to collect a screen coordinate of a test operation; judging whether the screen coordinate falls into a hot area corresponding to the hot area coordinate, to obtain a first judgment result; when the first judgment result includes that the screen coordinate falls into the hot area, calling a top window interface of the screen coordinate to obtain a current top window handle at the screen coordinate; judging whether the current top window handle conforms to an embedded browser window handle at the hit hot area, to obtain a second judgment result; when the second judgment result is conforming, selecting a browser debugging protocol channel, otherwise selecting a native accessibility function channel. The scheme can improve the test automation level of the hybrid architecture application program.

Description

Technical Field

[0001] This disclosure relates to the field of application testing technology, and in particular to a hybrid architecture application testing method and computer program product. Background Technology

[0002] This section is intended to provide background or context for the embodiments disclosed herein. The description herein is not intended to imply that it is prior art simply because it is included in this section.

[0003] As desktop applications become increasingly complex, many applications adopt hybrid architectures. For example, the main framework of the application might be developed using Qt, Electron, or WPF (Qt, Electron, and WPF are all names for existing application frameworks), responsible for native windows, menus, dialog boxes, etc.; while the application's content display area, settings panel, and rich interactive areas would be rendered using CEF (Chromium Embedded Framework) or Qt WebEngine (Qt WebEngine is a browser component included with the Qt framework). Typical scenarios include instant messaging clients, enterprise cloud storage / collaboration tools, and security management clients.

[0004] However, automated testing of such applications faces a fundamental contradiction: two completely different control systems exist within the same application window. The native control layer is exposed through the operating system's accessibility interface. These controls have stable attributes such as AutomationId, Name, and ControlType, which can be accurately located through the accessibility tree. On the other hand, the native accessibility interface has extremely weak ability to recognize DOM (Document Object Model) elements of CEF-embedded web pages. It can usually only recognize outer browser controls and cannot penetrate to DOM nodes such as buttons and input boxes within the web page. The web page content needs to access its DOM tree through the browser debugging protocol (Chrome DevTools Protocol, CDP).

[0005] Therefore, when recording test cases, the system cannot automatically determine whether the target clicked or entered by the tester is on a native control or an embedded webpage. This forces the system to manually select a test channel or blindly select one first and then switch to another when it is found that the test cannot be performed. This approach limits the improvement of the automation level of application testing. Summary of the Invention

[0006] Therefore, it is necessary to provide a hybrid architecture application testing method that can accurately select test channels to address the aforementioned technical problems.

[0007] Firstly, this disclosure provides a method for testing hybrid architecture applications. The method includes:

[0008] Obtain the hot zone coordinates of the target application, which includes a native control layer and a web page content layer; the hot zone coordinates are the set of region coordinates of all embedded browser windows in the currently visible top-level window.

[0009] Use the operation control hook to collect the screen coordinates of the test operation;

[0010] Determine whether the screen coordinates fall within the hot zone corresponding to the hot zone coordinates to obtain a first determination result;

[0011] When the first judgment result includes screen coordinates falling into the hot zone, the operating system's interface for obtaining the top-level window from the screen coordinates is called to obtain the current top-level window handle at the screen coordinates; it is determined whether the current top-level window handle matches the embedded browser window handle at the hot zone to obtain the second judgment result; when the second judgment result matches, the browser debugging protocol channel is selected.

[0012] When the first judgment result includes screen coordinates not falling into the hot zone, or when the second judgment result is not compliant, the native accessibility channel is selected.

[0013] Optionally, the method for obtaining the coordinates of the hot zone includes:

[0014] Enumerate the currently visible top-level windows of the target application, and identify the currently visible top-level windows whose window titles contain a preset identifier or whose window class names contain a preset class name as embedded browser windows; obtain the region coordinates of the embedded browser windows, and summarize them to obtain the hot zone coordinates;

[0015] Alternatively, the target application can be prompted to actively report hot zone coordinates by injecting test stubs.

[0016] Optionally, before obtaining the hotspot coordinates of the target application, the method further includes:

[0017] The system detects whether the target application simultaneously possesses a native control tree and a browser debugging protocol, and obtains the first detection result.

[0018] When the first detection result is yes, the target application is determined to be a hybrid architecture application;

[0019] When the first detection result includes a target application that does not have both a native control tree and a browser debugging protocol, a test channel is selected through image recognition.

[0020] Optionally, after selecting the browser debugging protocol channel, the method further includes:

[0021] When a webpage element cannot be located through the browser's debug protocol channel, the native accessibility channel is used after going back; the situations in which the webpage element cannot be located include no response from the debug protocol, viewport coordinates going out of bounds after coordinate transformation, or no DOM interactive element at the screen coordinates.

[0022] Optionally, the coordinate transformation method is as follows:

[0023] Get the coordinates of the window origin and the window DPI scaling factor;

[0024] The screen coordinates are converted into viewport coordinates based on the window origin coordinates and the window DPI scaling factor.

[0025] Optionally, after locating the native control via the native accessibility channel, the method further includes:

[0026] The test requirements are encapsulated into command closures using the main thread;

[0027] The main thread uses a command channel to send the command closure to a dedicated worker thread.

[0028] After initializing the component object model single-threaded unit within the dedicated worker thread, an auxiliary function interface object is created and held, and the UIA operation corresponding to the command closure is executed to obtain a value type result.

[0029] Optionally, after completing the test case recording, the method further includes:

[0030] When the test operation includes mouse operation, delete the mouse movement trajectory in the test case, delete the mouse hover event in the test case that is less than the first preset duration and has no tooltip, delete the mouse focus switching event in the test case that does not trigger a state change, and delete the mouse operation event in the test case that only includes scrolling.

[0031] When the test operation includes keyboard operation, determine whether adjacent keyboard inputs in the same input box are single characters and the interval is less than the second preset time, and obtain a third judgment result; when the third judgment result is yes, determine that the adjacent keyboard inputs correspond to continuous typing, and concatenate the text of each continuous typing; when the third judgment result is no, determine that the adjacent keyboard inputs correspond to re-entry, and only retain the final input text of the corresponding adjacent keyboard inputs.

[0032] Optionally, after selecting the test channel, the method further includes:

[0033] During test case recording, when the keyboard hook callback receives a key press, the handle of the current focused control is queried, and the handle of the current focused control and the current keyboard event are packaged and stored. When recording stops, the current focused control handle is converted into a snapshot of the user interface element through the auxiliary function interface, which is used as the target input box for the corresponding keyboard input operation.

[0034] Optionally, the method further includes:

[0035] Use the operation control hook to collect the monotonic clock timestamp of the test operation;

[0036] The coordinates of the hot zone are dynamically refreshed.

[0037] Secondly, this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0038] Obtain the hot zone coordinates of the target application, which includes a native control layer and a web page content layer; the hot zone coordinates are the set of region coordinates of all embedded browser windows in the currently visible top-level window.

[0039] Use the operation control hook to collect the screen coordinates of the test operation;

[0040] Determine whether the screen coordinates fall within the hot zone corresponding to the hot zone coordinates to obtain a first determination result;

[0041] When the first judgment result includes screen coordinates falling into the hot zone, the operating system's interface for obtaining the top-level window from the screen coordinates is called to obtain the current top-level window handle at the screen coordinates; it is determined whether the current top-level window handle matches the embedded browser window handle at the hot zone to obtain the second judgment result; when the second judgment result matches, the browser debugging protocol channel is selected.

[0042] When the first judgment result includes screen coordinates not falling into the hot zone, or when the second judgment result is not compliant, the native accessibility channel is selected.

[0043] The aforementioned hybrid architecture application testing method and computer program product provide a novel solution for automatically selecting test channels in hybrid architecture application testing. First, a hotspot is established using all embedded browser windows in the currently visible top-level window. Then, it is determined whether the screen position of the test operation is within the hotspot. If not, the native accessibility channel is directly selected. If it is within the hotspot, a secondary determination is made based on the window handles obtained through both methods. If the handles match, the browser debugging protocol channel is selected; otherwise, the native accessibility channel is selected. The entire solution is simple and clear, significantly improving the accuracy of test channel selection, thereby enhancing the level of test automation for hybrid architecture applications. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0045] Figure 1 This is a flowchart illustrating a hybrid architecture application testing method in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description, in conjunction with the accompanying drawings and embodiments, provides a more comprehensive understanding of this disclosure. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this disclosure. In the following detailed description of this application, certain specific details are described in detail. Those skilled in the art can fully understand this application even without these detailed descriptions. To avoid obscuring the essence of this application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0047] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims. Furthermore, since the methods for automated web page testing are based on software code in practical applications, some of which lack a unified Chinese name in the field, or even have no corresponding Chinese name, this application strives to explain the solutions clearly using Chinese. When Chinese cannot accurately express the meaning, the meaning referred to by the corresponding English name shall prevail.

[0048] The hybrid architecture application testing method provided in this disclosure can be applied to a testing environment for hybrid architecture applications. The hybrid architecture can be a software program architecture that simultaneously includes a native UI layer and a web content layer. The native UI layer refers to the collection of all native UI controls drawn by the system's native API, relying on the operating system's native window container. The web content layer refers to the web UI (user interface) area rendered and output by the Web (Web represents a webpage) rendering engine within the native window container, containing DOM elements and webpage layers. The testing environment generally includes a server and display terminals, operating terminals, and a data storage system, all communicating with the server. The data storage system can store the data that the server needs to process. The data storage system can be located locally or in the cloud. The server can run test programs to test the target application, including recording test cases. During test case recording, if it is necessary to determine whether a test item should use the browser debugging protocol channel or the native accessibility channel, the following method can be used: the server obtains the hot zone coordinates of the target application, where the target application is a hybrid architecture application including a native UI layer and a web content layer; the hot zone coordinates are the area coordinates of the embedded browser window in the currently visible top-level window. After testers perform test operations using the operating terminal, the server uses an operation control hook to collect the screen coordinates of the test operations. The results of the test operations can be displayed on the display terminal. The server determines whether the screen coordinates fall within a hot zone, obtaining a first determination result; the hot zone is the rectangular display area corresponding to the hot zone coordinates. When the first determination result includes screen coordinates falling within the hot zone, the server calls the operating system's "get top-level window from screen coordinates" interface, determines whether the returned top-level window handle matches the window handle of the hot zone, obtaining a second determination result. When the second determination result is correct, the server locates the web page element through the browser debugging protocol channel, thereby completing the selection of the test channel. When the first determination result includes screen coordinates not falling within the hot zone, or when the second determination result is incorrect, the server locates the native control through the native accessibility channel, thereby completing the selection of the test channel. The display terminal can be, but is not limited to, the monitor of various personal computers, laptops, smartphones, tablets, etc. The operating terminal can be a mouse and / or keyboard. The server performance needs to meet the various requirements of software testing, and can be implemented using a dedicated server or a server cluster composed of multiple servers.

[0049] In one embodiment, such as Figure 1As shown, a hybrid architecture application testing method is provided. Taking the application of this method in the aforementioned application environment as an example, the method includes the following steps:

[0050] Step 102: Obtain the hot zone coordinates of the target application, which includes a native control layer and a web page content layer; the hot zone coordinates are the set of region coordinates of all embedded browser windows in the currently visible top-level window.

[0051] Here, "currently visible top-level window" can refer to the top-level embedded webpage window currently visible on the display terminal during the test. "Region coordinates" can refer to screen coordinate information that describes the boundaries of the display window.

[0052] Specifically, if the target application contains both a native control layer and a web page content layer, it indicates that the target application is a hybrid architecture application and falls within the scope of this application. During the testing of the target application, such as when recording test cases for automated software testing, testers use devices like a mouse and keyboard to control the testing process, and the display terminal shows the execution status of the test program. This may include one or more display windows. At the current node, the display window that is not obscured at any position on the display terminal screen is the currently visible top-level window. That is, at any given moment, the display terminal can display one or more currently visible top-level windows, and a currently visible top-level window can be partially covered by other currently visible top-level windows. A currently visible top-level window may be an embedded browser window or a native layer window. The set of areas covered by all embedded browser windows within the currently visible top-level window is called a hot zone. In other words, on the display screen, as long as a display area is contained within the range of any embedded browser window in the currently visible top-level window, that display area belongs to the hot zone. The area coordinates of an embedded browser window generally include the coordinates of the four vertices of the rectangular display area. The area coordinates of all embedded browser windows within the currently visible top-level window belong to the hot zone coordinates. The boundaries of the hot zone can be clearly defined through the hot zone coordinates. In other words, the hot zone coordinates may include the screen positioning parameters and window origin coordinates of each embedded browser window.

[0053] Step 104: Use the operation control hook to collect the screen coordinates of the test operation.

[0054] In this context, an operation control hook refers to a hook (in English, "hook") that can capture and process test operation information. A test operation can refer to the actions performed by testers or intelligent test programs during software testing.

[0055] Specifically, operation control hooks can include at least one of low-level keyboard hooks (WH_KEYBOARD_LL) and low-level mouse hooks (WH_MOUSE_LL). Low-level keyboard hooks can monitor keyboard key events, including monitoring key information and the screen coordinates of the cursor in the current input box. Low-level mouse hooks can monitor cursor events, including monitoring the screen coordinates of mouse click operations, for example, down to the physical pixels corresponding to the mouse click operation. The low-level keyboard hook callback parameters themselves do not contain focus information, but the handle of the focus control (hwndFocus) can be obtained by calling GetForegroundWindow() + GetGUIThreadInfo() within the callback function. This is a pure User32 call (User32 refers to the user32.dll file in Windows), a microsecond-level call that does not involve COM, satisfying the hook's 2ms time constraint. Generally, the focus control handle is checked at the moment of key press; if the focus is retrieved afterward, it will lead to a misalignment between the focus and the control. Operation control hooks can be used to obtain the screen coordinates corresponding to test operations performed on the display terminal using controls such as the mouse and keyboard.

[0056] Step 106: Determine whether the screen coordinates fall within the hot zone corresponding to the hot zone coordinates to obtain the first determination result.

[0057] Specifically, given the screen coordinates and hotspot coordinates for the test operation, it can be determined whether the screen coordinates fall within the corresponding hotspot. For example, by traversing the area coordinates of each embedded browser window within the hotspot coordinates, it can be determined whether the screen coordinates fall within the rectangular area of ​​any embedded browser window. Of course, other determination methods can also be used. For instance, a specific program can be used to first define the boundaries of the entire hotspot based on the hotspot coordinates, and then determine whether the screen coordinates are within the hotspot boundary range. This embodiment does not limit the determination method; technicians can choose a suitable solution based on actual needs.

[0058] Step 108: When the first judgment result includes screen coordinates falling into the hot zone, the operating system's interface for obtaining the top-level window from the screen coordinates is called to obtain the current top-level window handle at the screen coordinates; it is determined whether the current top-level window handle matches the embedded browser window handle at the hot zone to obtain the second judgment result; when the second judgment result matches, the browser debugging protocol channel is selected.

[0059] The current top-level window handle can refer to the handle of the window that is at the top of the Z-order.

[0060] Specifically, operating systems used for software testing all have an "interface for retrieving the top-level window from screen coordinates." For example, in Windows, this includes `WindowFromPoint` and `ChildWindowFromPoint`. This interface allows you to obtain the handle of the current top-level window at the screen coordinates location. After the first judgment result (screen coordinates falling within a hot zone) is obtained, the system can determine which embedded browser window the screen coordinates fall into based on the hot zone coordinates, thus identifying the handle of the embedded browser window at the hit hot zone. Then, it checks if the current top-level window handle matches the handle of that embedded browser window to obtain the second judgment result. If both match, the Chrome DevTools Protocol (CDP) channel is selected to obtain the target element. It should be noted that because windows may have embedding relationships, both the current top-level window handle and the embedded browser window handle can include the handle of the parent window and the handles of its descendants.

[0061] Step 110: When the first judgment result includes screen coordinates not falling into the hot zone, or when the second judgment result is not compliant, select the native accessibility channel.

[0062] Among them, native accessibility channels can refer to mechanisms built into the operating system that are used to achieve barrier-free interaction or specific system-level communication.

[0063] Specifically, the names of the native accessibility channels differ across operating systems. For example, in Windows it's UIAutomation (UIA), in macOS it's AXUIElement, and in Linux it's AssistiveTechnology Service Provider Interface (AT-SPI). When the first judgment result includes screen coordinates not falling within the hot zone, or when the second judgment result is not met, the native accessibility channel is directly selected to locate the native control. After locating the target element through the selected browser debugging protocol channel or the native accessibility channel, the located target element can be associated with the test operation to generate test cases.

[0064] This embodiment provides a novel solution for automatically selecting test channels in hybrid architecture application testing. First, a hotspot is established using all embedded browser windows in the currently visible top-level window. Then, it's determined whether the screen position of the test operation falls within this hotspot. If not, the native accessibility channel is directly selected. If within the hotspot, a second check is performed based on the window handles obtained through two methods. If the handles match, the browser debugging protocol channel is selected; otherwise, the native accessibility channel is selected. This solution is concise and clear, significantly improving the accuracy of test channel selection and thus enhancing the level of test automation for hybrid architecture applications. Furthermore, since this solution is not tied to the Windows system, it only requires rewriting the native UI access backend for each operating system, while the web side continues to reuse CDP. This allows the solution to be ported to macOS and Linux hybrid desktop application automation scenarios, meaning it has a wide range of applicability.

[0065] In one embodiment, the method for obtaining the coordinates of the hot zone includes:

[0066] Enumerate the currently visible top-level windows of the target application, and identify the currently visible top-level windows whose window titles contain a preset identifier or whose window class names contain a preset class name as embedded browser windows; obtain the region coordinates of the embedded browser windows, and summarize them to obtain the hot zone coordinates.

[0067] Here, the default identifier can refer to a pre-defined identifier that can be used as an embedded browser window marker in the window title. The default class name can refer to a pre-defined class name that can be used as an embedded browser window marker in the window class name.

[0068] Specifically, the window title of an embedded browser window typically contains a specific identifier indicating its window type, such as Chrome. The window class name of an embedded browser window typically contains a feature class name indicating its window type, such as Chrome_WidgetWin. We can first list all currently visible top-level windows of the target application, and then filter them based on their window titles and class names. If a window title contains a preset identifier or a window class name contains a preset class name, then the corresponding visible top-level window is identified as an embedded browser window. Then, we summarize the region coordinates of all identified embedded browser windows and record them as hotspot coordinates. The solution in this embodiment can quickly determine hotspot coordinates in a simple way and has universal applicability to testing hybrid architecture applications.

[0069] In one embodiment, the method for obtaining the hot zone coordinates includes: inducing the target application to actively report the hot zone coordinates by injecting test stubs.

[0070] In this context, a test stub can refer to a small piece of proxy code / module specifically used for testing. Test stub injection refers to loading a test stub into the process of the application under test, so that it runs in the same process space as the application under test.

[0071] Specifically, for target applications that can use the test stub injection method, the target application can be made to actively report the hot zone coordinates through test stub injection. This method can quickly and accurately identify the embedded browser window in the currently visible top-level window.

[0072] In one embodiment, before obtaining the hot zone coordinates of the target application, the method further includes: detecting whether the target application simultaneously possesses a native control tree and a browser debugging protocol, and obtaining a first detection result; when the first detection result is yes, determining that the target application is a hybrid architecture application.

[0073] Among them, the native control tree can refer to the accessible and automated logic tree provided by the operating system.

[0074] Specifically, one can try initializing operating system accessibility interfaces, such as Windows' UIAutomation COM (COM stands for Component Object Model). Success indicates that the native control tree is available, meaning the target application possesses a native control tree. Embedded browser debugging ports can be scanned. If a port is found, a two-step security check is performed using loopback address binding and process identity verification. If both checks pass, the target application can be determined to have a browser debugging protocol. Before obtaining the target application's hotspot coordinates, detecting the native control tree and browser debugging protocol determines whether the target application is a suitable hybrid architecture application for this application's solution. This allows for rapid identification of the target application's architectural characteristics, avoiding situations where the test object and test method are incompatible.

[0075] In one embodiment, after selecting the browser debugging protocol channel, the method further includes:

[0076] When a webpage element cannot be located through the browser's debug protocol channel, the native accessibility channel is used after going back; the situations in which the webpage element cannot be located include no response from the debug protocol, viewport coordinates going out of bounds after coordinate transformation, or no DOM interactive element at the screen coordinates.

[0077] Specifically, selecting the browser debugging protocol channel does not guarantee successful testing. Uncommon situations may arise that prevent the browser debugging protocol channel from locating web page elements. These include situations such as the debugging protocol becoming unresponsive, viewport coordinates exceeding limits after coordinate transformation, or the absence of interactive DOM elements at the screen coordinates. Unresponsiveness can include CDP WebSocket connection freezes, the page still loading, CEF kernel freezes, WebSocket disconnections, and timeouts in CDP command calls without response. In the case of viewport coordinates exceeding limits after coordinate transformation, after the screen coordinates are converted by subtracting the window origin and DPI scaling to obtain the viewport coordinates required by CDP, if the x or y coordinates in the viewport are negative, or if the viewport coordinates are outside the visible area of ​​the web page, DOM queries will fail. If the elementFromPoint (elementFromPoint is a native DOM API provided by the browser, API stands for Application Programming Interface) returns null, it indicates that there are no interactive DOM elements at the screen coordinates. If the browser debugging protocol channel fails to locate web page elements, fall back and use the native accessibility channel for testing. In the native accessibility channel, screen coordinates are used, not viewport coordinates. If the test still cannot be completed after switching to the native accessibility channel, manual intervention by the tester is required.

[0078] This embodiment provides a solution for when testing cannot be completed after selecting the browser debugging protocol channel, which can improve test robustness and avoid "the routing decision logic is correct, but a partial anomaly in the webpage directly causes the recording of the entire test case to be interrupted".

[0079] In one embodiment, the coordinate transformation is performed as follows:

[0080] Get the coordinates of the window origin and the window DPI scaling factor;

[0081] The screen coordinates are converted to viewport coordinates based on the window origin coordinates and the window DPI scaling factor. For example, the conversion can be achieved using the following formulas: x = (a2 - a1) / c, y = (b2 - b1) / c, where x represents the x-coordinate of the viewport coordinates, y represents the y-coordinate of the viewport coordinates, a1 represents the x-coordinate of the window origin coordinates, b1 represents the y-coordinate of the window origin coordinates, a2 represents the x-coordinate of the screen coordinates, b2 represents the y-coordinate of the screen coordinates, and c represents the window DPI scaling factor. The window origin coordinates can refer to the coordinates of the origin of the embedded browser window. The window DPI scaling factor can refer to the ratio of the DPI of the embedded browser window corresponding to the screen coordinates to the system baseline DPI. DPI represents the page resolution of a webpage, i.e., dots per inch.

[0082] In this embodiment, a coordinate transformation method is provided, which can switch between screen coordinates and viewport coordinates to adapt to the data usage requirements of different test channels.

[0083] In one embodiment, after locating the native control via the native accessibility channel, the method further includes:

[0084] The test requirements are encapsulated into command closures using the main thread;

[0085] The main thread uses a command channel to send the command closure to a dedicated worker thread.

[0086] After initializing the component object model single-threaded unit within the dedicated worker thread, an auxiliary function interface object is created and held, and the UIA operation corresponding to the command closure is executed to obtain a value type result.

[0087] Specifically, operating system accessibility interfaces, such as Windows' UIAutomation, are single-threaded apartment (STA) objects based on the COM component object model, bound to the thread that created them. Direct cross-thread calls violate the COM thread model, potentially leading to deadlocks or access conflicts. To address this, this embodiment encapsulates the test requirements (or operations to be performed) into a command closure using the main thread. This command closure only captures ordinary numerical values, strings, and other data that can be copied across threads, excluding COM object handles and interface pointers. The main thread then sends this command closure to a dedicated worker thread (or dedicated COM thread) via a command channel. The dedicated worker thread continuously reads the channel commands, initializes internally, creates and holds the accessibility interface object, executes the UIA operation corresponding to the command closure, obtains the value type result, and returns it to the main thread. This ensures that the main thread only receives ordinary data such as coordinates, text, and boolean values, never obtaining the native UIA COM pointer, thus preventing cross-thread calls at the source. It is important to note that a safe destructing order must be used in the aforementioned process to avoid deadlock and segmentation fault: first, release the command sending end so that the command receiving loop of the dedicated worker thread exits due to the channel closure; then, the main thread waits for the dedicated worker thread to execute the cleanup logic and cannot directly kill the thread; finally, before the dedicated worker thread exits, it releases the auxiliary function interface object and then calls COM deinitialization.

[0088] In this embodiment, a dedicated STA thread is created to hold the UIA object. The main thread sends operation commands without COM pointers through the message channel, and the dedicated thread performs UI lookup and operations. Furthermore, the destruction sequence of "closing the channel → waiting for the thread to exit → releasing the UIA object and then unloading the COM object within the thread" is strictly followed, which can avoid deadlock and program crashes.

[0089] In one embodiment, after the test case recording is completed, the method further includes:

[0090] When the test operation includes mouse operation, delete the mouse movement trajectory in the test case, delete the mouse hover event in the test case that is less than the first preset duration and has no tooltip, delete the mouse focus switching event in the test case that does not trigger a state change, and delete the mouse operation event in the test case that only includes scrolling.

[0091] When the test operation includes keyboard operation, determine whether adjacent keyboard inputs in the same input box are single characters and the interval is less than the second preset time, and obtain a third judgment result; when the third judgment result is yes, determine that the adjacent keyboard inputs correspond to continuous typing, and concatenate the text of each continuous typing; when the third judgment result is no, determine that the adjacent keyboard inputs correspond to re-entry, and only retain the final input text of the corresponding adjacent keyboard inputs.

[0092] The first preset duration can refer to a pre-set threshold for mouse hover duration. Tooltips are non-obstructive UI elements that temporarily pop up as a small speech bubble when a user hovers the mouse over icons, buttons, links, or other controls (desktop) or clicks / long-presses (mobile). The second preset duration can refer to a pre-set threshold for the duration of consecutive keyboard inputs, or the time interval threshold between two keystrokes in the same input field. Adjacent keyboard inputs refer to keyboard inputs that are adjacent in time, with no other keyboard input events between them.

[0093] Specifically, the first preset duration can range from 600 to 1000 ms, such as 600 ms, 800 ms, or 1000 ms. All mouse movement is discarded; only mouse clicks, drag start points, and drag end points are retained. Pure sliding movements are discarded entirely. For example, hundreds of consecutive movement events from point A to point B are all deleted, and only the final operation point is recorded. If the mouse hovers over a control for a short time, but not long enough to exceed the first preset duration, and no hover tooltip appears, it is considered unintentional and not a valid operation. If the hover exceeds the first preset duration, or triggers a tooltip, the hover event is retained. If the cursor briefly jumps to another control and then back, it's merely a focus blink or temporary switch without any change in interface data or control state; this is considered system focus jitter noise and is deleted. If simply scrolling through content without clicking, typing, or selecting anything, the scrolling action is not recorded. If a click is performed immediately after scrolling, the scrolling action is retained.

[0094] The second preset duration can range from 400 to 600 ms, for example, 400 ms, 500 ms, or 600 ms. A single character can refer to one event corresponding to one character in the original keyboard events; for example, "hello" has five characters, corresponding to five events. If the second preset duration is 500 ms, then for the same input box, if the interval between characters is less than 500 ms, it is considered normal continuous typing, and all characters are concatenated into a complete text operation. The recording result is "type hello," showing only one step instead of five. If the interval between characters is ≥ 500 ms, it is considered an interruption and re-entry, discarding the previous content and saving the final input text. For example, if "he" is typed, pauses for 0.6 seconds, deletes, and "hi" is typed again, the script only records the final input "hi," not retaining invalid intermediate segments.

[0095] In this embodiment, the solution addresses the issue of significant redundant noise in the raw operation event stream recorded by the GUI. It employs an offline rule engine for event noise reduction, which is network-independent and allows for traceable behavior. Five types of rules are used to eliminate noisy events such as invalid mouse movements, brief hovering without prompts, focus jitter without state changes, and simple scrolling. For continuous keyboard events in the same input box, a second preset duration is used as a typing rhythm threshold to distinguish between continuous input and re-input behavior, automatically merging consecutive characters into complete text. This achieves a lightweight recording process that conforms to the semantics of human operation.

[0096] In one embodiment, recording the test cases includes:

[0097] When the keyboard hook callback receives a key press, it queries the handle of the currently focused control, packages and stores the handle of the currently focused control and the current keyboard event; when recording stops, it converts the current focused control handle into a snapshot of the user interface element through the auxiliary function interface, which serves as the target input box for the corresponding keyboard input operation.

[0098] Specifically, the underlying keyboard hook can only capture "which key was pressed," but cannot directly know which window or input box the character was input into. Therefore, when the keyboard hook callback receives a key press, it can query the handle of the currently focused control. For example, Windows' GetForegroundWindow retrieves the current foreground window, and GetGUIThreadInfo queries the window's thread information, retrieving hwndFocus, which is the handle of the control window currently with keyboard focus, and recording it in the event data. When recording stops, this handle is converted into a snapshot of the user interface element via the accessibility interface (ElementFromHandle), serving as the target input box for the keyboard input operation, thus achieving the "event-focus-control" triple association.

[0099] The solution in this embodiment can accurately attribute input actions and precisely identify which editing box each piece of text belongs to, thus avoiding "text attribution confusion".

[0100] In one embodiment, the method further includes:

[0101] Use the operation control hook to collect the monotonic clock timestamp of the test operation;

[0102] The coordinates of the hot zone are dynamically refreshed.

[0103] Specifically, all events are accompanied by a monotonic clock timestamp for causal ordering. A process-level monotonic clock can be used, unaffected by system time rollback / NTP adjustments. Since the embedded browser window may be moved or resized by the user during test case recording, the acquired hotspot coordinates may be outdated. Before the first judgment is executed, the currently visible top-level window can be re-enumerated, and the hotspot coordinates refreshed to ensure that the coordinate determination uses the latest embedded browser window position.

[0104] In one embodiment, the method further includes: when the first detection result includes a target application that does not have a native control tree and a browser debugging protocol, selecting a test channel through image recognition.

[0105] Specifically, when the first detection result includes the target application having neither a native control tree nor a browser debugging protocol, the target application is equivalent to a black box. In this case, artificial intelligence image recognition can be used to identify screen content, and the test channel can be intelligently selected based on the recognition result.

[0106] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0107] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0108] Obtain the hot zone coordinates of the target application, which includes a native control layer and a web page content layer; the hot zone coordinates are the set of region coordinates of all embedded browser windows in the currently visible top-level window.

[0109] Use the operation control hook to collect the screen coordinates of the test operation;

[0110] Determine whether the screen coordinates fall within the hot zone corresponding to the hot zone coordinates to obtain a first determination result;

[0111] When the first judgment result includes screen coordinates falling into the hot zone, the operating system's interface for obtaining the top-level window from the screen coordinates is called to obtain the current top-level window handle at the screen coordinates; it is determined whether the current top-level window handle matches the embedded browser window handle at the hot zone to obtain the second judgment result; when the second judgment result matches, the browser debugging protocol channel is selected.

[0112] When the first judgment result includes screen coordinates not falling into the hot zone, or when the second judgment result is not compliant, the native accessibility channel is selected.

[0113] In one embodiment, when the computer program is executed by a processor, it also implements the steps in any of the above method embodiments.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0115] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (RRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0116] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the appended claims.

Claims

1. A method for testing hybrid architecture applications, characterized in that, The method includes: Obtain the hot zone coordinates of the target application, which includes a native control layer and a web page content layer; the hot zone coordinates are the set of region coordinates of all embedded browser windows in the currently visible top-level window. Use the operation control hook to collect the screen coordinates of the test operation; Determine whether the screen coordinates fall within the hot zone corresponding to the hot zone coordinates to obtain a first determination result; When the first judgment result includes screen coordinates falling into the hot zone, the operating system's interface for obtaining the top-level window from the screen coordinates is called to obtain the current top-level window handle at the screen coordinates; it is determined whether the current top-level window handle matches the embedded browser window handle at the hot zone to obtain the second judgment result; when the second judgment result matches, the browser debugging protocol channel is selected. When the first judgment result includes screen coordinates not falling into the hot zone, or when the second judgment result is not compliant, the native accessibility channel is selected.

2. The method according to claim 1, characterized in that, The method for obtaining the coordinates of the hot zone includes: Enumerate the currently visible top-level windows of the target application, and identify the currently visible top-level windows whose window titles contain a preset identifier or whose window class names contain a preset class name as embedded browser windows; obtain the region coordinates of the embedded browser windows, and summarize them to obtain the hot zone coordinates; Alternatively, the target application can be prompted to actively report hot zone coordinates by injecting test stubs.

3. The method according to claim 1, characterized in that, Before obtaining the hot zone coordinates of the target application, the method further includes: The system detects whether the target application simultaneously possesses a native control tree and a browser debugging protocol, and obtains the first detection result. When the first detection result is yes, the target application is determined to be a hybrid architecture application; When the first detection result includes a target application that does not have both a native control tree and a browser debugging protocol, a test channel is selected through image recognition.

4. The method according to claim 1, characterized in that, After selecting the browser debugging protocol channel, the method further includes: When a webpage element cannot be located through the browser's debug protocol channel, the native accessibility channel is used after going back; the situations in which the webpage element cannot be located include no response from the debug protocol, viewport coordinates going out of bounds after coordinate transformation, or no DOM interactive element at the screen coordinates.

5. The method according to claim 4, characterized in that, The coordinate transformation method is as follows: Get the coordinates of the window origin and the window DPI scaling factor; The screen coordinates are converted into viewport coordinates based on the window origin coordinates and the window DPI scaling factor.

6. The method according to claim 1, characterized in that, After locating native controls via the native accessibility channel, the method further includes: The test requirements are encapsulated into command closures using the main thread; The main thread uses a command channel to send the command closure to a dedicated worker thread. After initializing the component object model single-threaded unit within the dedicated worker thread, an auxiliary function interface object is created and held, and the UIA operation corresponding to the command closure is executed to obtain a value type result.

7. The method according to any one of claims 1 to 6, characterized in that, After completing the test case recording, the method further includes: When the test operation includes mouse operation, delete the mouse movement trajectory in the test case, delete the mouse hover event in the test case that is less than the first preset duration and has no tooltip, delete the mouse focus switching event in the test case that does not trigger a state change, and delete the mouse operation event in the test case that only includes scrolling. When the test operation includes keyboard operation, determine whether adjacent keyboard inputs in the same input box are single characters and the interval is less than the second preset time, and obtain a third judgment result; when the third judgment result is yes, determine that the adjacent keyboard inputs correspond to continuous typing, and concatenate the text of each continuous typing; when the third judgment result is no, determine that the adjacent keyboard inputs correspond to re-entry, and only retain the final input text of the corresponding adjacent keyboard inputs.

8. The method according to claim 1, characterized in that, After selecting the test channel, the method further includes: During test case recording, when the keyboard hook callback receives a key press, the handle of the current focused control is queried, and the handle of the current focused control and the current keyboard event are packaged and stored. When recording stops, the current focused control handle is converted into a snapshot of the user interface element through the auxiliary function interface, which is used as the target input box for the corresponding keyboard input operation.

9. The method according to claim 2, characterized in that, The method further includes: Use the operation control hook to collect the monotonic clock timestamp of the test operation; The coordinates of the hot zone are dynamically refreshed.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images