Automatic testing method, device, equipment, medium and program product

Abstract

The embodiment of the invention provides an automatic testing method and device, equipment, a medium and a program product, and relates to the field of financial science and technology or other related fields. The method comprises the steps that in the automatic testing process, a client side process is started or closed through a process control module, and the process control module is a component used for controlling the process of a local operating system; window handles of the client side and the webpage side are obtained through a window handle locking module, and window switching is carried out; and executing the element operation of the client based on the window handle through the simulation operation module. Based on the method, through the synergistic effect of three aspects of process control, window handle locking and simulation operation, the interaction process of the webpage side and the client side is seamlessly connected, the coverage range of automatic testing is expanded, and the test control problem of cross-architecture interaction in a complex service scene can be solved.

Description

Technical Field

[0001] This application relates to the field of financial technology or other related fields, and in particular to an automated testing method, apparatus, equipment, medium and program product. Background Technology

[0002] In the fintech sector and other related fields, with the development of technology, mobile applications (Apps) and web applications have become important tools in scenarios such as financial and lifestyle services. For more complex business scenarios such as finance and shopping, due to the increased complexity of human-computer interaction and the extension of processing flows, automated testing of multi-platform applications is required to improve the response quality and stability of applications.

[0003] However, existing automated testing methods lack interaction mechanisms across architectural patterns. Current testing methods either implement client-server (C / S) interaction based on a client / server architecture or browser-server (B / S) interaction based on a webpage-server architecture. For complex business scenarios involving automated testing across architectural patterns, such interaction becomes difficult to achieve.

[0004] Therefore, there is an urgent need for an automated testing method to solve the test control problem of difficulty in achieving cross-architecture interaction in complex business scenarios. Summary of the Invention

[0005] This application provides an automated testing method, apparatus, device, medium, and program product for implementing test control across architectures in complex business scenarios, covering automated testing of cross-architecture interactions.

[0006] In a first aspect, embodiments of this application provide an automated testing method, the method comprising:

[0007] During automated testing, the client process is started or stopped through the process control module, which is a component used to control the local operating system processes.

[0008] The window handle locking module obtains the window handles of the client and the web page, and switches between windows.

[0009] The simulation operation module performs element operations on the client based on the window handle.

[0010] Secondly, embodiments of this application provide an automated testing apparatus, the apparatus comprising:

[0011] The process control unit is used to start or stop client processes during automated testing. The process control module is a component used to control local operating system processes.

[0012] The window unit is used to obtain the window handles of the client and the web page through the window handle locking module, and to switch between windows.

[0013] The simulation operation unit is used to perform client-side element operations based on the window handle through the simulation operation module.

[0014] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the implementation method described in the first aspect above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the embodiments described in the first aspect above.

[0016] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements the implementation methods described in the first aspect above.

[0017] The automated testing method, apparatus, device, medium, and program products provided in this application embodiment allow for direct startup or shutdown of the client process during automated testing via a process control module, ensuring client controllability. A window handle locking module enables the acquisition of window handles, preventing interaction failures due to changes in interface layout. A simulated operation module allows for the execution of client-side element operations based on window handles, covering the interaction needs of non-standard client controls. The synergistic effect of these three components seamlessly connects the interaction flow between the web page and the client, expanding the scope of automated testing and resolving test control issues related to cross-architecture interactions in complex business scenarios. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] Figure 1 A flowchart illustrating the automated testing method provided in this application embodiment;

[0020] Figure 2 This is a schematic diagram of the structure of the automated testing system provided in the embodiments of this application;

[0021] Figure 3A schematic block diagram illustrating the execution process of the automated testing method provided in this application embodiment;

[0022] Figure 4 A schematic diagram of the intelligent waiting execution process for elements provided in the embodiments of this application;

[0023] Figure 5 A schematic diagram of the intelligent pop-up processing execution flow provided in the embodiments of this application;

[0024] Figure 6 This is a schematic diagram of the structure of the automated testing device provided in the embodiments of this application;

[0025] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0026] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0029] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0030] The following is a description of some terms and concepts used in the embodiments of this application:

[0031] Java Native Access (JNA): JNA is an open-source Java framework based on Java Native Interface (JNI) technology. It provides utility classes to dynamically call system native libraries, allowing developers to perform cross-platform native method calls without writing native JNI code. By creating an interface that inherits from a class and loading a dynamic link library, methods within the library can be declared and called. Dynamic link libraries include, for example, Dynamic Link Libraries (DLLs) and Shared Objects (SOs).

[0032] Robot: Robot is a class belonging to the Java Abstract Window Toolkit (java.awt). This Robot class is used to generate native system input events for test automation, self-running demos, and other applications that require control of the mouse and keyboard.

[0033] The Process class: The Process class is part of the Java Language package (java.lang) and is primarily used to control processes in the local operating system. Through the Process class, you can start new processes, interact with them (e.g., input data into or read data from a process), wait for them to terminate, and check their exit status.

[0034] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0035] In the embodiments of this application, the use of terms such as "first" and "second" is to distinguish between identical or similar items that have essentially the same function and effect. For example, "first electronic device" and "second electronic device" are merely used to distinguish different electronic devices and do not limit their order of execution. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0036] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0037] With the rapid development of internet technology, mobile apps and web applications have become deeply integrated into complex business scenarios such as finance, e-commerce, and logistics. For example, in the fintech field, users may need to complete operations such as opening an account, transferring funds, and applying for loans through a web page, which often requires data interaction with the client (C-end) (such as identity verification and SMS verification code input). In e-commerce scenarios, users may need to browse products and add them to their shopping cart on the web, and complete payments or check logistics information through the client.

[0038] Such cross-architecture (hybrid C / S and B / S) interaction processes place higher demands on automated testing. For example, testers need to ensure the consistency of functional logic between the web and client sides, the stability of interface interactions, and the ability to tolerate abnormal scenarios. However, existing automated testing technologies struggle to cover such complex scenarios, leading to low testing efficiency and high maintenance costs.

[0039] In addition, pop-ups (such as advertising pop-ups, permission requests, system prompts, etc.) may frequently appear during the operation of web applications. The randomness and diversity of these pop-ups can interrupt the automated testing process, causing element location failure or test case execution interruption.

[0040] Meanwhile, due to factors such as network fluctuations and differences in device performance, page loading time is difficult to predict accurately. Fixed waiting times or simple element detection methods can easily lead to resource waste or operation failures. Therefore, there is an urgent need for an automated testing method that can interact across architectures, intelligently handle pop-ups, and dynamically determine page loading status to improve testing stability, efficiency, and adaptability, and meet the high-quality testing requirements of complex business scenarios.

[0041] Existing automated testing methods are mainly designed based on a single architectural pattern (C / S or B / S), making it difficult to achieve bidirectional interactive control between the web and client sides. For example, web-side testing typically relies on the WebDriver framework to manipulate browser elements, while client-side testing requires tools such as Appium to simulate user operations. The two cannot collaboratively handle cross-architecture processes.

[0042] Furthermore, existing technologies for handling pop-ups rely on static blacklist strategies, which can only handle known pop-up types and cannot adapt to the dynamic appearance of new pop-ups. Moreover, the full-scale detection mechanism leads to resource waste. Determining page loading status often involves either fixed-time waiting or simple element detection. The former causes unnecessary waiting time, while the latter may result in misjudgments due to elements not fully loading, affecting test progress.

[0043] Meanwhile, existing technologies heavily rely on hard-coded logic for handling pop-ups and page elements. When the application interface is updated, test scripts need frequent maintenance, significantly increasing development costs. Furthermore, traditional methods lack adaptability to testing environments (such as network conditions and system load), making it difficult to balance testing efficiency and stability.

[0044] For example, existing technical solutions rely on a single architectural pattern (such as supporting only the web or client side) in cross-architecture interactions, failing to achieve bidirectional control and resulting in incomplete test coverage for complex business processes. For instance, in financial transaction scenarios, the web-based account opening operation requires interaction with client-side authentication, but traditional automation tools can only manipulate web-based elements and cannot simulate client-side behavior, leading to test interruptions. Furthermore, the randomness and diversity of pop-ups (such as advertising pop-ups and permission requests) can disrupt the testing process. Existing static blacklist strategies cannot dynamically adapt to new pop-up types, and full-scale detection mechanisms consume significant resources. Page loading status judgments rely on fixed waiting times or simple element detection, which are susceptible to network fluctuations, leading to resource waste or operation failures.

[0045] Starting with problems identified in existing technologies, the inventors first addressed the lack of cross-architecture interaction by proposing a system-level control approach (such as Process, JNA, and Robot classes) to achieve bidirectional interaction between the web client and the client-side. This approach solves client-side control challenges through window handle locking and coordinate simulation. Secondly, to address pop-up interference, a pop-up specification library and a self-learning mechanism were designed, reducing maintenance costs through tagging pop-up types and dynamic update strategies. Finally, to address inaccurate page loading judgments, a large model was introduced to predict the maximum probability of waiting time from historical logs, combined with a dynamic polling mechanism to achieve adaptive waiting. Through iterative optimization of these technical approaches, a closed-loop optimized automated testing framework was ultimately formed, resolving issues of testing stability, efficiency, and adaptability in complex business scenarios.

[0046] In view of this, embodiments of this application provide an automated testing method that can construct an automated testing framework for cross-architecture interaction and intelligent element processing. It addresses technical issues such as insufficient test coverage, pop-up interference, and inaccurate page loading judgments in complex business scenarios through a dynamic adaptive mechanism. This framework is based on cross-modal interaction, combined with intelligent pop-up handling strategies and dynamic element waiting technology, forming a closed-loop optimized testing process: reducing maintenance costs through standardized pop-up handling specifications and self-learning mechanisms, balancing testing efficiency and stability through large-model prediction and dynamic waiting strategies, ultimately achieving high efficiency, adaptability, and scalability in automated testing.

[0047] It should be noted that the automated testing methods, equipment, media, and program products provided in this application can be used in the fintech field, or in any field other than fintech. The application fields of the automated testing methods, equipment, media, and program products in this application are not limited.

[0048] The methods provided in this application can be applied to mixed web and client interaction testing in complex business scenarios such as finance, e-commerce, and logistics. For example, in a financial transaction scenario, users need to complete account opening through a web page and identity verification through a client; in an e-commerce scenario, users need to browse products on the web and switch to the client to complete payment. These scenarios involve bidirectional interaction between C / S and B / S architectures, and page loading time is greatly affected by network fluctuations and device performance, while frequently encountering interfering elements such as pop-up ads and permission requests. Existing technologies struggle to simultaneously meet the requirements of cross-architecture control, adaptive pop-up handling, and dynamic page loading judgment, resulting in low testing efficiency and high maintenance costs.

[0049] The method provided in this application can be applied to applications, websites, or mini-programs with automated test task processing capabilities. Automated test task processing capabilities are implemented on applications, websites, or mini-programs. For example, a computer with an automated test application deployed can implement automated test task processing capabilities by running the automated test application. Another example is a terminal electronic device, such as a mobile phone, with an automated test mini-program deployed, which can implement automated test task processing capabilities by running the automated test mini-program.

[0050] The technical solutions of this application will be described in detail below with reference to specific embodiments. The specific embodiments described below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0051] Figure 1This is a flowchart illustrating the automated testing method provided in an embodiment of this application. The executing entity of this method can be an electronic device with corresponding data storage and computing capabilities, such as a computer, server, or server cluster. Figure 1 As shown, the method includes:

[0052] S101, during automated testing, starts or stops client processes through the process control module, which is a component used to control local operating system processes.

[0053] For example, a process control module can be understood as a component used to control local operating system processes, which starts or stops the client by calling the system application programming interface (API). A process control module can, for example, be built based on the Process class.

[0054] The Process class in Java is used to control processes on the local operating system. It can start and stop external programs (such as clients). During automated testing, when starting or stopping client processes through a process control module, for example, the exec method of the Process class in Java can be called to start the client process.

[0055] S102, obtain the window handles of the client and the web page through the window handle locking module, and switch windows.

[0056] For example, a window handle can be understood as a unique numerical value that identifies a window in the operating system. The window handle can be used to locate and manipulate the window. The window handle locking module can be understood as a component used to obtain and lock the window handles on the client and web sides. It can achieve cross-platform window management by calling local libraries (such as the core dynamic link library of the operating system).

[0057] The JNA framework enables cross-platform native method calls by dynamically invoking system native libraries, eliminating the need to write JNI code. For example, Java's JNA framework can obtain window handles by dynamically calling API functions in the operating system's core dynamic link libraries. After obtaining the window handles on both the client and web page sides, it can also switch between windows.

[0058] S103, through the simulation operation module, performs element operations on the client based on the window handle.

[0059] For example, a simulation operation module can be understood as a component used to simulate mouse and / or keyboard operations, which can perform client-side element operations by generating system-level input events (such as coordinate clicks, text input). The simulation operation module can, for example, be built based on the Robot class.

[0060] The Robot class is a Java class used to simulate mouse and keyboard input, generating system-level input events. For example, Java's Robot class uses the mouseMove and keyPress methods to simulate user interaction.

[0061] Based on the above steps, the process control module can start the client process, ensuring the client is in an interactive state. The window handle locking module dynamically obtains the window handles of the client and web client by calling a local library, and can achieve precise window switching based on the window handles. After locking the window, the simulation operation module can generate mouse and keyboard element operation events based on the window handle, window coordinates, and element attributes to complete the simulation operation of client elements.

[0062] Based on the synergistic effect of the above steps, the process control module ensures the controllability of the client, the window handle locking module solves the problem of cross-platform window switching, and the simulation operation module covers the interaction needs of non-standard controls on the client. The overall process of this method does not rely on the client's user interface (UI) framework or API, and realizes bidirectional interaction between the web client and the client through system-level control.

[0063] For example, to address the interaction control needs between the web client and the client-side (C-end), the Process class can be used to start and stop the client process; by using the JNA framework to lock the window handles of the web client and the client, window switching can be achieved. The Robot class can simulate mouse and keyboard operations and locate client elements based on screen coordinates.

[0064] For example, in financial transaction scenarios, after completing the account opening process on the web client, the system needs to switch to the client for authentication. The system starts the client process via Process, JNA locks the client window handle, and Robot simulates clicking the verification code input box and entering information. Based on this, bidirectional interactive control between the web client and the client is achieved through system-level process control and window handle locking. This method, based on the operating system's process management and graphical interface control capabilities, establishes a cross-mode interaction mechanism, which can overcome the limitation of existing automation tools that can only operate on a single architecture mode, thus covering the full-process testing needs under complex business scenarios.

[0065] The automated testing method provided in this application allows for direct startup or shutdown of the client process via a process control module, ensuring client controllability. A window handle locking module acquires the window handle, preventing interaction failures due to changes in interface layout. A simulated operation module executes client-side element operations based on the window handle, covering the interaction requirements of non-standard client controls. The synergistic effect of these three modules seamlessly connects the interaction flow between the web page and the client, expanding the scope of automated testing and resolving test control issues related to cross-architecture interactions in complex business scenarios.

[0066] For example, existing technologies struggle to achieve bidirectional interactive control between the web client and the client application, leading to incomplete test coverage for complex business scenarios (such as financial transactions and e-commerce payments). The method provided in this application can build a cross-modal interaction mechanism based on the Process class, JNA, and Robot class to improve test coverage.

[0067] In one possible implementation, the window handle locking module obtains the window handles of the client and the web page, and performs window switching, including: dynamically calling the local library to obtain the window handle of the client and dynamically calling the local library to obtain the window handle of the web page; and performing window switching based on the window handles of the client and the web page.

[0068] For example, dynamically calling native libraries can be understood as dynamically loading and calling methods of operating system native libraries through a pre-selected framework (such as JNA). For instance, the JNA framework can be used to dynamically call system native libraries, lock window handles for the web client and the client, and implement window switching.

[0069] Window switching can be understood as the operation of switching focus to a specified window by manipulating the window handle. For example, JNA switches the operating system's input focus to the client window by calling system functions, making it the currently active window that can receive input elements such as keyboard and mouse input.

[0070] By dynamically calling local libraries to obtain the window handles of both the client and web applications, accurate window positioning can be ensured. Performing window switching operations based on these window handles enables precise switching between the web and client windows. For example, in a financial trading scenario, after obtaining the client's window handle using a window lookup method, the system calls the method for specifying the active window to switch focus to the client and complete the authentication process.

[0071] In this embodiment, dynamically calling a local library to obtain the window handle overcomes the limitations of relying on static window identifiers. Dynamically calling the local library adapts to different operating systems and interface layout changes, ensuring real-time acquisition of the window handle. Handle-based window switching avoids interaction failures caused by changes in interface design, thereby improving the stability and compatibility of cross-modal interactions.

[0072] For example, during the operation of a web application, pop-ups such as advertisements and permission requests may appear randomly, interrupting the automated testing process. Relying on a static blacklist strategy to handle pop-ups makes it difficult to dynamically adapt to new pop-up types, and the efficiency of a full-scale detection mechanism will decrease. Therefore, pre-built intelligent pop-up processing modules (such as pop-up detection modules and pop-up processing modules) can be used to dynamically manage pop-ups.

[0073] In one possible implementation, after the simulation operation module performs element operations on the client based on the window handle, the method further includes: detecting whether a pop-up exists in the client window through a pop-up detection module; and if a pop-up is detected, performing pop-up handling operations through a pop-up handling module according to a preset pop-up handling strategy.

[0074] For example, a pop-up detection module can be understood as a component used to detect the presence of pop-ups in a client window. For instance, a pop-up detection module might be a functional component that detects pop-ups through image recognition or text matching. A pop-up processing module can be understood as a component used to perform pop-up handling operations. For instance, a pop-up processing module might be a functional component that simulates clicking a close button or executes a self-recovery strategy. The self-recovery strategy could be automatic processing logic for a specific pop-up (such as a network timeout prompt), such as retrying the operation or refreshing the page.

[0075] For example, in an e-commerce testing scenario, after performing element operations on the client side, if an advertising pop-up is detected by the preset pop-up detection module, the pop-up processing module will automatically perform a closing operation to close the advertising pop-up according to the automatic closing policy for advertising pop-ups, thus preventing the pop-up from blocking the subsequent testing process.

[0076] For example, the pop-up handling module can load a pop-up specification library. This library can predefine pop-up type enumerations, tagged pop-up content, and pop-up handling strategies for each type or tagged pop-up. Pop-up handling strategies may include actions such as closing the pop-up, self-recovery, and ending the transaction. The pop-up specification library ensures the scalability of pop-up handling strategies through enumerated design and tagged content.

[0077] Furthermore, a dynamic matching and self-learning mechanism can be used to retrieve pop-up content during test script execution, match it against the pop-up specification library, and execute the processing strategy for successfully matched pop-ups. If the match fails, the test script can be interrupted, and the pop-up content can be added to the pop-up specification library for automatic upgrades and maintenance. Alternatively, upgrades and maintenance can be performed by prompting maintenance personnel to manually review and update the library. The dynamic matching mechanism can combine text content with interface features (such as button position and shape) to improve matching accuracy. Moreover, the self-learning mechanism can automatically record and categorize undefined pop-ups, enabling continuous optimization of the standardized pop-up specification library.

[0078] For example, in a bank's testing scenario, once an ad pop-up successfully matches the pop-up handling strategy of automatically closing it, the pop-up can be automatically closed and the test can continue. If an undefined permission request pop-up appears, the test script can be interrupted, and maintenance personnel can be prompted to add a new pop-up label. By using a pop-up specification library, dynamic matching, and a self-learning mechanism, the limitations of static blacklist strategies can be overcome, reducing the test interruption rate caused by pop-up interference, while also reducing manual maintenance costs and improving the stability of the testing process.

[0079] In this embodiment, the defect of test interruption caused by pop-up interference can be overcome by pop-up detection and processing. The pop-up detection module can also combine image recognition and text matching technologies to ensure accurate identification of pop-ups. The pop-up processing module can dynamically adjust the pop-up handling operation through preset pop-up processing strategies, improving the fault tolerance and stability of the testing process.

[0080] For example, if the pop-up detection process relies on the text or interface features within the pop-up, and the content or layout of the pop-up changes, then pop-up recognition is based on a preset content or layout (hard-coded strategy), which can easily lead to misjudgments or missed judgments. Therefore, the method provided in this application introduces image recognition algorithms and Optical Character Recognition (OCR) algorithms. It extracts image features and text content from the pop-up through screenshots, and can combine a pop-up specification library for multimodal feature matching (text labels, image templates), thereby improving the comprehensiveness and accuracy of pop-up detection.

[0081] In one possible implementation, detecting the presence of a pop-up in the client window using a pop-up detection module includes: extracting image features of the client window using an image recognition algorithm; extracting text content of the client window using an optical character recognition algorithm; and jointly matching the image features and text content using the pop-up detection module to detect the presence of a pop-up in the client window.

[0082] For example, an image recognition algorithm can be understood as an algorithm that extracts and recognizes image features (such as shape and color) from a window image. These image features include, for example, image features characterizing the image's position, shape, outline, size, color, color histogram, and / or texture.

[0083] Image recognition algorithms can be based on open-source computer vision libraries (OpenCV). For example, OpenCV can be used to perform grayscale conversion, edge detection, and region segmentation on screenshots of screen windows, extracting image features from the bounding box region of pop-up windows.

[0084] Optical Character Recognition (OCR) algorithms can be understood as algorithms that extract text content from a pop-up window by recognizing optical characters. Examples of OCR algorithms include the open-source Tesseract OCR. By extracting text content through OCR, text recognition can be performed on pop-up areas to extract key text, such as "Permission Request" or "Close Ad." For instance, Tesseract OCR can recognize buttons containing text, such as "Close" and "Cancel," in pop-ups.

[0085] For example, image recognition algorithms and OCR algorithms can be used to extract the image features and text content of the pop-up window, and joint matching can be performed based on these two features to detect whether a pop-up window exists in the client window. For instance, in ad pop-up detection, using OpenCV to extract the pop-up window boundary and using OCR to recognize the "click to close" text content can ensure the accuracy of pop-up window detection.

[0086] For example, when detecting whether a pop-up exists in a client window by jointly matching image features and text content through the pop-up detection module, a multimodal matching strategy can be used to improve the comprehensiveness and accuracy of pop-up detection.

[0087] Multimodal matching can be understood as a strategy that jointly matches image features and text content against a pop-up specification library. Furthermore, multimodal matching can dynamically adjust the weighting of the image and text dimensions during matching. For example, the priority of text matching can be adjusted to be higher than that of image features. Based on dynamic weight adjustment, matching efficiency can be further optimized, the false positive rate reduced, and the test framework's adaptability to interface optimizations and changes in pop-up types enhanced.

[0088] The pop-up specification library can pre-store and maintain sets of historical image features and historical text content, which can be obtained from the images and text of historical pop-ups. A matching score is calculated by comparing the similarity between image features and the similarity between text content, and combining their respective weights. The matching score is then judged using a preset matching threshold to detect the existence of a pop-up.

[0089] Image recognition algorithms can extract image features such as the bounding box and color histogram of the pop-up window, while OCR algorithms can extract the text content. By jointly matching these features with the labeled pop-up window types (such as semantic tags) in the pop-up window specification library, the pop-up window detection results can be obtained quickly.

[0090] This embodiment combines image recognition and OCR algorithms. The pop-up detection can adapt to changes in text and interface image layout, reducing false positives caused by interface optimization. During joint matching, for example, when the pop-up text changes from "Click to close" to "Click here to close," OCR can still recognize key operation instructions, and image template matching can cover changes in interface design. This allows for accurate multimodal pop-up detection. Furthermore, the multimodal matching strategy reduces reliance on hard-coded element location strategies, further improving the adaptability of the test script.

[0091] In this embodiment, multimodal feature fusion matching of image features and text content can overcome the limitations of single feature matching of text or images. Joint matching of image features and text content can improve the comprehensiveness and accuracy of pop-up detection and reduce the false positive rate of pop-ups.

[0092] In one possible implementation, the method further includes: performing semantic recognition on the text content using a natural language processing model to obtain the semantic recognition result of the text content; generating semantic tags based on the semantic recognition result, and using the semantic tags for joint matching of pop-up detection.

[0093] For example, a natural language processing model can be understood as an artificial intelligence model used for semantic understanding and intent recognition of natural language; it can be any large language model. The semantic recognition result can be the specific semantics of the text content, and semantic tags can be generated by summarizing the specific semantics.

[0094] For example, in the process of building and maintaining the pop-up specification library, a natural language processing model can be introduced to perform semantic analysis on the extracted text content through the natural language processing model, and automatically extract the key intent of the pop-up text content, such as "requesting permission", "advertisement closed" and "system failure".

[0095] For example, pre-trained natural language processing models, such as transformer-based bidirectional encoder representation models, can be used to extract semantic vectors and identify intent from text content. For instance, classification models can be used to classify and identify the specific semantics corresponding to the text content. Based on the specific semantics, language tags can be automatically generated and synchronously updated to the pop-up specification library.

[0096] For example, when the text content of the pop-up changes from "Please allow access to location" to "Location permission request", the natural language processing model can still recognize its core intent and generate consistent semantic labels, thus ensuring the stability of the pop-up processing strategy.

[0097] In this embodiment, semantic recognition of text content and generation of semantic tags through a natural language processing model can achieve the goal of dynamically generating semantic tags, reducing the time cost and difficulty of manually maintaining the pop-up specification library. Furthermore, automatically generated semantic tags can quickly adapt to the emergence of new pop-up types, improving the autonomous maintenance capability of the testing process.

[0098] For example, if the pop-up handling strategy relies on manual definition and maintenance, it is difficult to adapt to the dynamic needs of complex scenarios. For instance, a network timeout pop-up should prioritize retrying rather than closing. To address this, the method provided in this application introduces a reinforcement learning algorithm to dynamically optimize the pop-up handling strategy based on historical pop-up handling results, thereby improving the intelligence level of strategy decision-making.

[0099] In one possible implementation, the pop-up handling module performs pop-up handling operations according to a preset pop-up handling strategy, including: updating the pop-up handling strategy in the pop-up specification library based on historical pop-up handling results using a reinforcement learning model; and performing pop-up handling operations according to the preset pop-up handling strategy based on the updated pop-up specification library.

[0100] For example, a reinforcement learning model can be understood as a machine learning model that drives policy iteration through a reward function. For instance, a reinforcement learning model could be a Q-learning model, which, through reinforcement learning and training based on historical pop-up handling results, can summarize experience in handling pop-ups, thereby updating the preset pop-up handling strategy and updating and maintaining the pop-up specification library.

[0101] For example, by analyzing the historical pop-up handling results such as "transaction successful after closing the pop-up" or "page blank after self-recovery" through reinforcement learning models, the pop-up handling strategies in the pop-up specification library can be dynamically updated. Then, the handling operations for various types of pop-ups can be executed based on the updated strategy specification library.

[0102] For example, when dealing with "network timeout" pop-ups, reinforcement learning models can continuously optimize pop-up handling strategies based on historical feedback to adapt to complex and dynamic needs, such as prioritizing retries over closing the pop-up. In testing scenarios such as financial transactions, if repeatedly closing the "network timeout" pop-up leads to transaction failures, the reinforcement learning model will learn from these historical pop-up handling results, generate a better pop-up handling strategy such as "prioritizing page refresh," and update it to the pop-up specification library. This enables the upgrade and maintenance of the pop-up specification library, which improves the fault tolerance and intelligent level of strategy decision-making in subsequent tests while reducing the workload of manual maintenance.

[0103] When learning and updating pop-up handling strategies in a reinforcement learning model, the pop-up type, page context (such as the current action step), and processing result (e.g., transaction interruption or success) can be defined as the state space of the reinforcement learning model. A reward function is used to assign reward values ​​based on the processing result, driving strategy iteration. For example, a successful transaction earns a reward of +1, a blank page earns a reward of -0.5, and an open pop-up earns a reward of -1. During updates, pop-up handling strategies with higher reward values ​​can be prioritized.

[0104] In this embodiment, a reinforcement learning model is used to dynamically optimize the pop-up handling strategy, overcoming the limitations of static strategies in the pop-up specification library. Dynamically updating the pop-up specification library can adapt to changing needs in various complex scenarios, reducing the workload of manual maintenance of pop-up handling strategies, improving maintenance efficiency, and enhancing the intelligence level of strategy decision-making.

[0105] For example, during automated testing, it is necessary to wait for the test elements to load or respond. If a strategy with a fixed waiting time (hard-coded strategy) is adopted, it is difficult to adapt to dynamic factors such as network fluctuations and differences in device performance, which may lead to inaccurate page loading judgment, resulting in wasted resources or operation failure. To address this, the method provided in this application introduces a large model prediction and dynamic polling mechanism to overcome the inaccurate page loading judgment caused by factors such as network fluctuations and differences in device performance, thereby improving the efficiency of test resource utilization and the success rate of operation.

[0106] In one possible implementation, after the pop-up handling module performs the pop-up handling operation according to the preset pop-up handling strategy, the method further includes: predicting the loading time of the target element based on the historical execution log by the element waiting module; and dynamically adjusting the timeout threshold for element search based on the loading time.

[0107] For example, historical execution logs can be understood as log data recording the historical execution process of automated test scripts. For instance, historical execution logs may include data such as element loading time and the frequency of pop-up appearances. Dynamically adjusting the timeout threshold can be understood as adjusting the timeout duration for element lookup based on prediction results. For example, when network latency is high, the timeout threshold can be increased to achieve dynamic adjustment of the timeout threshold.

[0108] For example, the large model can be a pre-trained artificial intelligence model used to predict the loading time of a target element based on historical execution logs. This large AI model can be a model built on neural networks. The large model can analyze success logs in the historical execution logs to calculate the maximum probability wait time for a single step. This maximum probability wait time can be used as a benchmark, with a preset percentage (e.g., 20%) added upwards as a dynamically adjusted timeout threshold.

[0109] By analyzing historical execution logs to predict the loading time of target elements and dynamically adjusting timeout thresholds, test time consumption and resource waste caused by fixed waiting times can be avoided. For example, in financial transaction testing scenarios, based on historical execution logs, a large model can predict the loading time of the transfer result page. Dynamically adjusting the timeout threshold based on the predicted loading time ensures the timeliness and accuracy of element operations.

[0110] In some possible implementations, network status monitoring can be introduced into the element waiting module. By combining the prediction results of the large model, the timeout threshold of individual steps can be dynamically adjusted to make the waiting strategy more consistent with the actual network environment. Network status monitoring includes, for example, monitoring Hypertext Transfer Protocol (HTTP) request response time or Domain Name System (DNS) resolution latency.

[0111] For example, network status metrics such as current network latency (e.g., ping value), bandwidth utilization, and DNS resolution time can be obtained in real time through system APIs, enabling real-time network status monitoring. By weighting and fusing historically predicted waiting times with real-time monitored network status metrics, dynamic timeout durations can be generated (e.g., increasing the weight to 30% when network latency is high), and the timeout threshold for element lookup can also be dynamically adjusted.

[0112] Furthermore, during automated testing, operations on elements may require multiple rounds of waiting or repeated operations after failure. For example, in a bank's testing scenario, after operating on a target element, the system defaults to polling multiple times within 5 seconds to find the target element. If it is not found, the system checks the pop-up window and refreshes and retryes according to the preset pop-up handling strategy; if it is still not found, an exception is thrown and the transaction is terminated.

[0113] To address this issue, an adaptive polling strategy can be employed to reduce testing resource consumption. This strategy involves adjusting the polling frequency based on a dynamic timeout duration. For example, when the network is stable, the polling frequency or total number of polls can be dynamically increased; when the network fluctuates, the polling frequency or total number of polls can be dynamically decreased, thus reducing testing resource consumption.

[0114] Based on the methods provided in the above embodiments, the large model, leveraging its statistical learning and analysis capabilities on historical execution logs, can accurately predict reasonable waiting times under different scenarios. The adaptive dynamic polling strategy combined with the pop-up handling strategy can avoid test misjudgments caused by pop-up blocking. The dynamic adjustment of the timeout threshold can balance test resource consumption and response speed.

[0115] When network fluctuations cause page loading delays, using a large-scale model to predict and extend the timeout threshold can avoid false alarms caused by premature timeouts. If a pop-up appears, it can be handled first before retrying, ensuring the integrity of the operational logic. For example, in a financial transaction scenario, after a user submits a transfer request, they need to wait for the result page to load. A large-scale model can predict the maximum probable waiting time (e.g., 5 seconds), and the system can poll for a "transfer successful" button. If it is not found within 5 seconds, the system checks for pop-ups (e.g., network timeout pop-ups), refreshes the page according to the preset pop-up handling strategy, and retryes. Based on this, the efficiency and stability of testing can be improved, balancing waiting time and resource utilization efficiency.

[0116] As can be seen from the above embodiments, by using real-time network monitoring and dynamic weight calculation, the element waiting strategy can more accurately match changes in the actual environment. For example, in network congestion scenarios, the system automatically extends the waiting time and reduces the polling frequency to avoid misjudgments caused by premature timeouts. In stable network scenarios, high-frequency polling can quickly respond to element loading and reduce invalid waiting time, thus improving the fault tolerance of the automated testing process to network fluctuations.

[0117] In this embodiment, the problem of low environmental adaptability of fixed waiting times is solved by dynamically adjusting the timeout threshold using data driven by historical execution logs. Large-scale model prediction based on historical execution logs can accurately match reasonable waiting times for different scenarios. This mechanism of dynamically adjusting the timeout threshold balances resource consumption and response speed, improving testing efficiency and stability.

[0118] For example, based on any of the above embodiments, a cross-architecture interactive parallel execution mechanism based on distributed computing can also be introduced. Through a cross-mode interaction module, a distributed task scheduling framework can be introduced to break down the interaction operations between the web client and the client into independent tasks for parallel execution, improving the testing efficiency of complex business processes.

[0119] The cross-modal interaction module can decompose tasks, breaking down cross-architecture interaction processes into independent subtasks, such as "submitting a form on the web" and "verifying identity on the client side." Then, the execution of each subtask can be scheduled in parallel. A message queue distributes each subtask to different worker nodes for parallel execution. After execution, the results of each subtask can be synchronized. Through shared memory or a database, the execution status and results of each subtask are synchronized to ensure the logical consistency of the final operation.

[0120] Therefore, the distributed parallel execution mechanism can shorten the testing time for complex business processes. For example, in e-commerce payment scenarios, the web-based product checkout and client-side payment verification can be executed in parallel, avoiding the wasted waiting time caused by serial execution. Furthermore, task decomposition and synchronization mechanisms reduce system resource contention, improving the scalability of the testing framework and adapting to high-concurrency testing needs.

[0121] This application also provides an automated testing method. The method involves: starting or stopping the client process via a process control module; obtaining and switching window handles between the client and web page via a window handle locking module; performing element operations on the client based on the window handles via a simulated operation module; detecting the presence of pop-ups in the client window via a pop-up detection module; if a pop-up is detected, performing pop-up handling operations according to a preset pop-up handling strategy via a pop-up handling module; and predicting the loading time of the target element based on historical execution logs via an element waiting module, and dynamically adjusting the timeout threshold for element searching based on the predicted loading time.

[0122] The method in this application integrates the execution steps of multiple functional modules. Through process control, window switching, simulated operation, pop-up detection and processing, and dynamic waiting, it achieves a complete automated testing process for cross-mode interaction. By integrating the above functional modules, a closed-loop optimized testing framework is formed. The synergistic effect of cross-mode interaction, intelligent pop-up processing, and dynamic waiting significantly improves test coverage, test stability, and efficiency, meeting the high-quality testing requirements in complex business scenarios.

[0123] The method provided in this application constructs a closed-loop optimized automated testing process through the collaborative effects of cross-mode interaction, intelligent pop-up handling, and dynamic element waiting. First, the cross-mode interaction mechanism utilizes system-level control technologies (such as the Process class, JNA, and Robot class) to enable bidirectional operation between the web client and the client, solving the problem of insufficient test coverage in complex business scenarios. This makes bidirectional operation between the web client and the client possible, significantly improving the testing scope and covering the full-process testing needs in complex business scenarios.

[0124] Secondly, the intelligent pop-up processing module dynamically matches and processes various pop-ups through a standardized pop-up specification library and a self-learning mechanism, adapting to changes in pop-up types, reducing test interruptions caused by pop-up interference, and simultaneously reducing maintenance costs and improving test stability.

[0125] Finally, the element intelligent waiting module predicts the waiting time in historical logs based on a large model, and dynamically adjusts the timeout time in combination with the pop-up handling strategy. This reduces invalid waiting, balances test efficiency, optimizes resource consumption, and improves test stability, avoiding resource waste or operation failures caused by traditional fixed waiting.

[0126] The three parts mentioned above work together to form a complete closed loop from interactive control to exception handling, achieving high efficiency, stability and adaptability of automated testing, and meeting the high-quality testing needs of complex business scenarios such as finance and e-commerce.

[0127] The following is combined with Figures 2 to 5 The automated testing method provided in the embodiments of this application will be further described. The embodiments of this application provide an automated testing method for complex web applications based on cross-modal interaction and intelligent element processing. This method can, for example... Figure 2 The automated testing system shown is implemented.

[0128] Figure 2 This is a schematic diagram of the structure of the automated testing system provided in the embodiments of this application, such as... Figure 2 As shown, this automated testing system includes a cross-mode interaction module and an element intelligent processing module. The cross-mode interaction module can record window handles using JNA, switch between web and client windows, and simulate keyboard and mouse operations on the client. The element intelligent processing module can implement an intelligent element waiting mechanism and intelligent pop-up handling.

[0129] The cross-modal interaction module enables cross-modal interaction mechanisms. In complex trading scenarios (such as in the fintech field), transactions involve interaction between the web client and the client application. This interaction between two different architectures can be achieved through the Process class, JNA, and Robot class, allowing switching between the client and the web client and control of the client application. This solves the problem of the testing framework being unable to operate the client, and enables scripts to execute the client's business processes.

[0130] The element intelligent waiting mechanism can be achieved by analyzing the logs of historically successfully executed scripts through a large model to obtain the maximum probability waiting time for a single step. For example, it can be increased by 20% as the final timeout time (timeout threshold) for that step. Within the timeout period, elements are polled and searched. Once a search is successful, the polling stops. Combined with explicit waiting and intelligent pop-up handling, this greatly improves test execution efficiency and reduces the false timeout rate, while achieving timely response and script execution stability.

[0131] Intelligent handling of pop-up windows can be achieved by defining pop-up window types and handling strategies during the program design phase and establishing a pop-up window specification library. When a pop-up window appears during test script execution, it is matched against the pop-up window specification library and automatically processed. At the same time, the pop-up window specification library is continuously enriched based on the pop-ups that appear during test script execution, thereby enabling more stable, efficient, and self-maintainable automated testing during script execution.

[0132] Existing automated testing only operates and controls based on web pages or app pages, and cannot achieve simultaneous control and two-way interaction between the web end and the client (C end). Cross-mode interaction establishes a two-way interactive control mechanism between the web end and the client (C end), thus completing the application scope of automated testing.

[0133] Figure 3 A schematic block diagram illustrating the execution process of the automated testing method provided in this application embodiment is shown below. Figure 3 As shown, after the script starts executing, it records the web window handle via JNA, activates the client via Process, and records the client window handle via JNA. After the client logs in with its account, it waits for the client to complete the login process before switching back to the teller's client (web client). It waits for the web client to complete its business process before switching back to the client client to execute the business. JNA and the Robot class simulate keyboard and mouse operations on the client; after the client operations are completed, it switches back to the teller's client.

[0134] When the client is opened or closed, the `Process` class is used to implement script framework processes that open and close processes on the local operating system to achieve client opening and closing. When locking and switching windows, the Java open-source framework using JNA is used to lock the window handles of both the client and web interfaces for precise switching. During client operations, the Java open-source framework using JNA and the `Robot` class are used to control the local operating system to simulate mouse and keyboard output, locate the monitor coordinates of client elements, and perform mouse cursor movement and clicks.

[0135] The execution of automated scripts may be affected by network conditions, system load, and other upstream and downstream environmental factors, resulting in a delay after automated operations on elements, with the page not responding immediately and the delay being unpredictable. Existing automation frameworks typically have static waits and explicit waits. Implicit waits are global settings that affect the entire WebDriver lifecycle, resulting in a lot of invalid polling and only checking the existence of elements. Explicit waits are local settings that require repeated declarations, leading to a lot of code redundancy. Furthermore, both wait methods are hard-coded with fixed wait intervals, making it impossible to adapt to environmental differences (development / testing / production), network fluctuations, and changes in system load, severely impacting execution efficiency.

[0136] Figure 4 This is a schematic diagram of the intelligent waiting execution process for elements provided in the embodiments of this application, such as... Figure 4 As shown, the system provided in this application embodiment parses historical successful execution logs using a large model, outputs the maximum probability waiting time for each step, and sets the timeout for that step by 20%. After an element operation, it checks for the specified element within a default 5-second loop. If the element appears, the loop immediately stops and the script continues execution. If the element does not appear within 5 seconds, it checks for a pop-up prompt. If a pop-up appears, it follows an intelligent pop-up handling strategy, such as refreshing and retrying or skipping and continuing execution. If no pop-up appears, it uses the maximum probability waiting time for that step from the large model, sets it by 20% as the timeout, and continues looping until the target element appears and the loop stops. If the target element does not appear within the timeout, an element search exception is thrown, and the transaction ends.

[0137] By using the intelligent waiting mechanism described above, a timeout is set for each step, and the polling stops once the search is successful. This improves test execution efficiency, reduces the false alarm rate of timeout, and balances immediate response with resource consumption, turning the waiting process from passive to proactive prediction.

[0138] Figure 5 This is a schematic diagram of the intelligent pop-up processing execution flow provided in the embodiments of this application, such as... Figure 5 As shown, regarding the mechanism for intelligent handling of pop-ups, when initializing the pop-up specification library, pop-up types can be defined during the program design phase by forcibly enumerating pop-up types, standardizing pop-up data attributes, and ensuring consistent structural design and documentation specifications. Based on business characteristics, pop-ups are tagged, and different pop-up handling strategies are set for pop-ups with different tags. These strategies include closing the pop-up and continuing execution, performing self-recovery, and ending the transaction, thereby establishing the pop-up specification library.

[0139] During the execution of the process, when a pop-up appears on the page during the execution of the test script, the pop-up content can be retrieved and matched against the pop-up specification library. If the pop-up content matches successfully with the specification library, it can be processed according to the specification library strategy. If the match fails, the transaction can be terminated in time, the pop-up content can be retrieved, added to the pop-up specification library, and the pop-up content can be highlighted with color in the automation report for having an undefined tag. This will remind maintenance personnel to maintain the pop-up specification library, find the record, tag it, and set the corresponding processing strategy, thereby further enriching the pop-up specification library.

[0140] After determining the type of the pop-up, it's possible to decide whether the transaction can continue. If so, close the pop-up and continue execution. This can be done if the pop-up's meaning doesn't affect subsequent transactions. If not, further assessment is needed to determine if the fault is self-healing. When initiating a self-healing pop-up handling strategy, different strategies can be set based on business characteristics. For example, for network timeouts, close the pop-up and retry; for blank pages, refresh the system and re-execute the script; for accounts not existing, trigger the account opening interface to open an account; for insufficient funds, automatically trigger a recharge operation, etc. The strategy for ending a transaction pop-up can be to interrupt the current script and execute the next script when the pop-up indicates an environmental problem, such as a node failure, preventing further transactions.

[0141] By intelligently handling each process through the aforementioned pop-up windows, the test script execution process can be automated more stably, efficiently, and self-maintainingly, ultimately improving software quality and development testing efficiency.

[0142] For example, in pop-up recognition, in addition to pre-screening methods based on page source, image recognition technology can be combined to assist pop-up detection, which can improve the recognition rate. In page loading judgment, performance-based criteria, such as CPU utilization and memory usage, can be added as auxiliary judgment criteria. In anomaly handling, more complex decision trees or machine learning models can be introduced to improve the accuracy of anomaly classification and handling.

[0143] Based on the automated testing system and methods described in the above embodiments, the problem of non-interactive automated testing across architectural patterns in complex business scenarios is solved, as well as the problem of the automated framework being unable to operate the client. Test stability is significantly improved, effectively handling various interfering pop-ups and drastically reducing test case interruption rates. Test efficiency is greatly enhanced; through intelligent pop-up handling and intelligent element waiting, test execution time is reduced by about half, and script execution success rate is improved.

[0144] Furthermore, maintenance costs are effectively reduced. System anomalies and timeouts causing script execution failures are the two most common errors. Intelligent pop-up handling and intelligent element waiting significantly reduce the workload of script maintenance. Resource consumption is reduced, avoiding unnecessary full pop-up checks and fixed-time waits, thus lowering CPU and memory usage. It has a wide range of applications and can be used in various web page and mobile application automated testing scenarios, including functional testing, compatibility testing, and cross-mode automated testing.

[0145] This application also provides a cross-mode interaction testing method based on a modular architecture. The method includes: starting a client process by calling the operating system application interface through a process control module; obtaining window handles of the client and web page through a local library by calling a window handle locking module, and switching windows based on the handles; and performing element operations on the client based on the window handles through a simulation operation module, wherein the simulation operation module and the window handle locking module interact with each other through shared memory.

[0146] For example, this method achieves cross-mode interaction through a modular architecture. The process control module calls the operating system API to start the client process; the window handle locking module calls the local library to obtain the window handle and passes it to the simulation operation module through shared memory; the simulation operation module performs element operations (such as clicking and input) based on the handle, avoiding cross-module communication delays.

[0147] Building upon the aforementioned embodiments, this method further addresses the issue of low communication efficiency between test framework modules through a modular architecture and shared memory communication. The decoupling design of the process control module, window handle locking module, and simulation operation module significantly improves system scalability and compatibility; shared memory communication reduces data transmission latency, ensuring real-time cross-mode interaction.

[0148] This application embodiment also provides a cross-modal interaction testing system, the system comprising:

[0149] The system comprises the following modules: a process control module for starting or stopping the client process by calling the operating system's application programming interface; a window handle locking module for obtaining window handles for the client and web page from a local library and switching windows based on these handles; a simulation operation module for performing element operations on the client based on the window handles, where the simulation operation module and the window handle locking module interact via shared memory; a pop-up detection module for detecting the presence of pop-ups in the client window; a pop-up handling module for performing pop-up handling operations according to a preset pop-up handling strategy; and an element waiting module for predicting the loading time of the target element based on historical execution logs and dynamically adjusting the timeout threshold for element lookup.

[0150] For example, shared memory can be understood as a communication method where multiple functional modules directly share data through memory. For instance, the window handle locking module writes the acquired handle to shared memory, and the simulation operation module reads it directly. Native libraries can be understood as dynamic link libraries provided by the operating system, used to implement cross-platform function calls.

[0151] This cross-mode interactive testing system achieves functional decoupling through modular design. The process control module and the window handle locking module cooperate to ensure client control; the simulation operation module and the window handle locking module interact in real time through shared memory to improve data transmission efficiency; the pop-up detection and processing module dynamically optimizes the test process; and the element waiting module combines historical logs to predict loading time, forming a closed-loop testing mechanism.

[0152] Based on the above embodiments, this cross-mode interaction testing system forms a complete cross-mode testing framework through system-level modular design. Each module achieves efficient collaboration through shared memory and local library calls, covering the entire process of client startup, window switching, element operations, pop-up handling, and dynamic waiting, significantly improving test coverage and execution stability, and meeting the automated testing needs of complex business scenarios.

[0153] Figure 6 This is a schematic diagram of the structure of the automated testing device provided in the embodiments of this application, such as... Figure 6 As shown in the figure, this application provides an automated testing device, which includes:

[0154] The process control unit 601 is used to start or stop the client process through the process control module during automated testing. The process control module is a component used to control the local operating system process.

[0155] The window unit 602 is used to obtain the window handles of the client and the web page through the window handle locking module and switch between windows;

[0156] The simulation operation unit 603 is used to perform client-side element operations based on the window handle through the simulation operation module.

[0157] In one possible implementation, the device further includes a spring-loaded unit for:

[0158] The pop-up detection module detects whether a pop-up exists in the client window.

[0159] If a pop-up window is detected, the pop-up window handling module will perform pop-up window handling operations according to the preset pop-up window handling strategy.

[0160] In one possible implementation, the pop-up unit is specifically used for:

[0161] Image features of the client window are extracted using image recognition algorithms;

[0162] The text content of the client window is extracted using an optical character recognition algorithm;

[0163] The pop-up detection module performs joint matching of image features and text content to detect whether a pop-up exists in the client window.

[0164] In one possible implementation, the device further includes a semantic recognition unit for:

[0165] The semantic recognition results of the text content are obtained by performing semantic recognition on the text content using a natural language processing model.

[0166] Semantic labels are generated based on the semantic recognition results, and these semantic labels are used for joint matching of pop-up detection.

[0167] In one possible implementation, the pop-up unit is specifically used for:

[0168] The reinforcement learning model updates the pop-up handling strategies in the pop-up specification library based on the historical pop-up handling results.

[0169] Based on the updated pop-up specification library, the pop-up processing module performs pop-up handling operations according to the preset pop-up processing strategy.

[0170] In one possible implementation, the apparatus further includes a prediction unit for:

[0171] The element waiting module predicts the loading time of the target element based on historical execution logs.

[0172] The timeout threshold for element lookup is dynamically adjusted based on loading time.

[0173] In one possible implementation, the window unit 602 is specifically used for:

[0174] The window handle locking module dynamically calls the local library to obtain the client's window handle and dynamically calls the local library to obtain the web page's window handle.

[0175] Window switching is performed based on the client-side window handle and the web-side window handle.

[0176] The automated testing device provided in this application can be used to execute the technical solutions of the automated testing methods in any of the above embodiments of this application. Its implementation principle and technical effect are similar, and will not be described again here.

[0177] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 7 As shown, the electronic device of this embodiment may include: at least one processor 701; and a memory 702 communicatively connected to the at least one processor; wherein the memory 702 stores instructions that can be executed by the at least one processor 701, and the instructions are executed by the at least one processor 701 to cause the electronic device to perform the method as described in any of the above embodiments.

[0178] Optionally, the memory 702 can be either standalone or integrated with the processor 701.

[0179] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0180] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method of any of the foregoing embodiments.

[0181] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the foregoing embodiments.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0183] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0184] It should be understood that the aforementioned processor can be a central processing unit (CPU) or other general-purpose processors. The processor can also be a digital signal processor (DSP) or an application-specific integrated circuit (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0185] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be various media that can store program code, such as USB flash drives, portable hard drives, read-only memory (ROM), disks or optical discs.

[0186] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof. Examples of storage media include Static Random-Access Memory (SRAM) or Electrically Erasable Programmable Read Only Memory (EEPROM).

[0187] Storage media can be, for example, erasable programmable read-only memory (EPROM) or programmable read-only memory (PROM). Storage media can also be read-only memory (ROM), magnetic storage, flash memory, magnetic disks, or optical disks. Storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0188] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components within an electronic device or host device.

[0189] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0190] The sequence numbers of the embodiments in this application are merely for description and do not represent the superiority or inferiority of the embodiments. Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0191] Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0192] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0193] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0194] It should be further noted that although the steps in the flowchart are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated in this document, there is no strict order requirement for the execution of these steps, and they can be executed in other orders.

[0195] Furthermore, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0196] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0197] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0198] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An automated testing method, characterized in that, The method includes: During automated testing, the client process is started or stopped through the process control module, which is a component used to control the local operating system processes; The window handle locking module obtains the window handles of the client and the web page, and switches between windows. The simulation operation module performs element operations on the client based on the window handle.

2. The method according to claim 1, characterized in that, After the simulation operation module performs client-side element operations based on the window handle, the method further includes: The pop-up detection module detects whether a pop-up exists in the client window. If a pop-up window is detected, the pop-up window handling module will perform pop-up window handling operations according to the preset pop-up window handling strategy.

3. The method according to claim 2, characterized in that, The step of detecting whether a pop-up exists in the client window via the pop-up detection module includes: Image features of the client window are extracted using an image recognition algorithm; The text content of the client window is extracted using an optical character recognition algorithm; The pop-up detection module performs joint matching of the image features and the text content to detect whether a pop-up exists in the client window.

4. The method according to claim 3, characterized in that, The method further includes: The semantic recognition results of the text content are obtained by performing semantic recognition on the text content using a natural language processing model. Semantic labels are generated based on the semantic recognition results, and these semantic labels are used for joint matching of pop-up detection.

5. The method according to claim 2, characterized in that, The step of executing pop-up handling operations through the pop-up handling module according to a preset pop-up handling strategy includes: The reinforcement learning model updates the pop-up handling strategies in the pop-up specification library based on the historical pop-up handling results. Based on the updated pop-up specification library, the pop-up processing module performs pop-up handling operations according to the preset pop-up processing strategy.

6. The method according to claim 2, characterized in that, After the pop-up handling module performs the pop-up handling operation according to the preset pop-up handling strategy, the method further includes: The element waiting module predicts the loading time of the target element based on historical execution logs. The timeout threshold for element lookup is dynamically adjusted based on the loading time.

7. The method according to any one of claims 1-6, characterized in that, The step of obtaining the window handles of the client and the web page through the window handle locking module and switching windows includes: The window handle locking module dynamically calls the local library to obtain the window handle of the client and dynamically calls the local library to obtain the window handle of the web page. The window switching is performed based on the window handle of the client and the window handle of the web page.

8. An automated testing device, characterized in that, The device includes: A process control unit is used to start or stop client processes during automated testing through a process control module, wherein the process control module is a component used to control local operating system processes; The window unit is used to obtain the window handles of the client and the web page through the window handle locking module, and to switch between windows. The simulation operation unit is used to perform client-side element operations based on the window handle through the simulation operation module.

9. An electronic device, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images