Website automation testing method, device and electronic equipment

Abstract

This invention provides a website automated testing method, apparatus, and electronic device, belonging to the field of automated testing technology. The method includes: acquiring test documents uploaded by users under a target project; parsing, segmenting, and vectorizing the test documents; storing the vectors in a vector database; and constructing a knowledge base associated with the target project; receiving test requirements input by the user; starting an AI orchestration engine; loading the large language model configuration, toolset, and knowledge base associated with the target project; and, based on the test requirements, having the AI ​​orchestration engine execute test tasks cyclically through an agent to obtain test execution results. This invention enables intelligent orchestration and execution of the entire website automated testing process, improving testing efficiency and robustness.

Description

Technical Field

[0001] This invention relates to the field of automated testing technology, specifically to a website automated testing method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of internet technology, the functional complexity and iteration frequency of website systems have increased significantly, making automated testing a key means of ensuring software quality. Traditional automated testing relies on testers manually writing test cases and scripts (such as those based on frameworks like Selenium and Playwright). Meanwhile, a few solutions have introduced artificial intelligence technology into the field of automated testing, utilizing the natural language understanding and generation capabilities of large language models to assist in tasks such as test case generation and script writing, aiming to reduce labor costs and improve the level of testing intelligence.

[0003] In the process of developing this invention, the inventors discovered that existing AI-based automated website testing methods have at least the following drawbacks: First, the test cases have a low degree of relevance to real-world business scenarios, and the knowledge cannot be reused. Existing methods typically only utilize the basic generation capabilities of large language models to directly convert user-inputted test requirements into test cases, resulting in incomplete coverage of generated test cases and difficulty in continuous optimization with project iterations.

[0004] Second, the intelligent orchestration capabilities of automated testing processes are insufficient, lacking state management and fault tolerance mechanisms. Existing intelligent testing tools mostly implement single functions (such as test case generation or script execution) and lack an end-to-end test process orchestration engine. When test tasks involve multiple rounds of interaction or abnormal scenarios, they struggle to handle complex situations such as model call failures, excessively long contexts, and high-risk operations requiring manual confirmation, resulting in poor traceability and robustness of the testing process.

[0005] Third, the integration methods of testing tools are rigid, limiting their scalability. Existing methods typically integrate specific automated testing frameworks using hard-coding (such as directly calling the Playwright API). When replacing or adding testing tools, the core code must be modified, making it difficult to flexibly connect to third-party services. At the same time, user-defined test scripts or dedicated tools cannot be easily integrated into automated processes, limiting the system's adaptability to the personalized testing needs of different projects.

[0006] Fourth, the testing process lacks project-level management and collaboration mechanisms. Existing tools lack isolation and access control for multiple projects and roles, making it difficult to achieve independent project-level management of test cases, knowledge bases, and tool resources; human-machine collaboration (such as approval before high-risk operations) is often ignored or requires additional development, making it impossible to seamlessly integrate into the automated testing process and failing to meet the collaborative testing needs of enterprise teams.

[0007] Therefore, how to fully utilize project-specific knowledge to achieve highly relevant intelligent generation of test cases without requiring extensive manual writing of test cases and scripts, while simultaneously building an end-to-end automated testing process with state persistence, exception tolerance, standardized tool extensions, and project-level collaboration capabilities to improve website testing efficiency and robustness, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] In view of this, it is necessary to provide a website automation testing method, device, and electronic device to solve the technical problems of low automation testing efficiency and poor system robustness caused by the low fit between test cases and business scenarios, lack of intelligent orchestration and status management of test processes, poor tool scalability, and insufficient project-level collaboration capabilities in existing methods.

[0009] To address the aforementioned technical problems, in a first aspect, the present invention provides a website automated testing method, comprising: The test documents uploaded by the user under the target project are obtained, the test documents are parsed, segmented and vectorized, and the processed vectors are stored in the vector database to build a knowledge base associated with the target project. The system receives test requirements input by the user and starts the AI ​​orchestration engine to load the large language model configuration, toolset, and knowledge base associated with the target project. The AI ​​orchestration engine integrates a middleware chain, which includes, in the order of loading, a model call retry middleware, a tool call retry middleware, a context summary middleware, a manual approval middleware, and a streaming output middleware. Based on the aforementioned testing requirements, the AI ​​orchestration engine executes the testing tasks through a proxy loop to obtain the test execution results.

[0010] In one possible implementation, the parsing, segmentation, and vectorization of the test document includes: Use a document parsing tool to extract the plain text content from the test document; The plain text content is segmented using a recursive character text segmenter to obtain multiple text fragments; The embedding model is invoked to convert each of the text fragments into a vector; The vectors are stored in the vector database, and the association between each text fragment and the target project and its source document is recorded.

[0011] In one possible implementation, the step of having the AI ​​orchestration engine execute test tasks through a proxy loop based on the test requirements to obtain test execution results includes: Based on the aforementioned testing requirements, the AI ​​orchestration engine retrieves relevant text fragments from the knowledge base as contextual information. The context information and the test requirements are input into the large language model to generate test cases corresponding to the test requirements; The AI ​​orchestration engine executes the test tasks corresponding to the test cases through a proxy loop to obtain the test execution results.

[0012] In one possible implementation, retrieving relevant text fragments from the knowledge base as context information using the AI ​​orchestration engine based on the test requirements includes: Using the test requirements as query conditions, perform vector retrieval and / or keyword retrieval in the knowledge base to obtain initial search results; The initial search results are reordered, and a preset number of text segments with the highest similarity are selected as the context information.

[0013] In one possible implementation, the step of having the AI ​​orchestration engine execute the test tasks corresponding to the test cases through a proxy loop to obtain the test execution results includes: Using the test case as the target of this test task, obtain the historical summary and task status of the current test session, wherein the historical summary records the compressed information of the historical loop steps, and the task status records the execution progress of the current test task; Based on the test cases, the historical summary, and the task status, a current context is constructed, and the current context is input into a large language model to obtain the decision result output by the large language model; Based on the decision result, the corresponding tools or skills in the toolset are invoked to perform the test operation; Record the proxy step information generated in this proxy loop and update the task status of the test task. The proxy step information includes the decision result in the current loop, the tools or skills called, and the corresponding execution result. Based on preset judgment conditions, determine whether the test task has been completed. If completed, use the execution result corresponding to the current test task as the test execution result; if not completed, return to the above steps of building the current context to update the task status.

[0014] In one possible implementation, the step of invoking the corresponding tools or skills in the toolset to perform the test operation based on the decision result includes: When the decision is to execute the test, the browser automation tool is invoked based on the test case through a standardized tool protocol, so that the browser automation tool generates and executes automated test scripts according to the test case.

[0015] In one possible implementation, the standardized tool protocol is a model context protocol; the step of invoking browser automation tools based on the test cases through the standardized tool protocol includes: An MCP session corresponding to the browser automation tool is created through the session manager. The MCP session is isolated based on the user identifier, project identifier, and session identifier. The automated test script is sent to the browser automation tool via the MCP session.

[0016] In one possible implementation, the website automated testing method further includes: Obtain user-uploaded custom skill packs, which include metadata description files and executable scripts; When the AI ​​orchestration engine is detected to be calling the custom skill, the skill package list is read to obtain the metadata description file corresponding to the custom skill in the skill package list; The executable script is invoked based on the metadata description file.

[0017] On the other hand, the present invention also provides a website automated testing device, comprising: The module is used to obtain test documents uploaded by users under the target project, parse, segment and vectorize the test documents, and store the processed vectors into a vector database to build a knowledge base associated with the target project. The receiving module is used to receive the test requirements input by the user and start the AI ​​orchestration engine to load the large language model configuration, toolset and knowledge base associated with the target project. The AI ​​orchestration engine integrates a middleware chain, which includes, in the order of loading, a model call retry middleware, a tool call retry middleware, a context summary middleware, a manual approval middleware, and a streaming output middleware. The testing module is used to execute test tasks through an agent loop by the AI ​​orchestration engine based on the test requirements, and obtain test execution results.

[0018] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the website automated testing method described in any of the above implementations.

[0019] The beneficial effects of this invention are as follows: The website automated testing method provided by this invention parses, segments, and quantifies user-uploaded test documents and stores them in a database, building a dedicated knowledge base for each project. Subsequent tests repeatedly utilize the business knowledge in this dedicated knowledge base, avoiding the need to re-organize documents each time and improving testing efficiency. When the AI ​​orchestration engine is started, this knowledge base, model configuration, and toolset are loaded, and middleware such as retry, context summarization, and manual approval are integrated into the engine. This enables automatic retry when model calls fail, automatic compression when dialogues are too long, and requests for manual confirmation when encountering high-risk operations, thereby improving the fault tolerance and controllability of the entire testing process. Based on the test requirements input by the user, the engine automatically executes test tasks through a proxy loop without manual intervention. This achieves end-to-end automation from test requirements to test case generation, script execution, and result feedback, improving the efficiency of website automated testing and the robustness of the system. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic flowchart of an embodiment of the website automated testing method provided by the present invention; Figure 2 For the present invention Figure 1 A schematic diagram of an embodiment of S101; Figure 3 For the present invention Figure 1 A schematic diagram of an embodiment of S103; Figure 4 For the present invention Figure 3 A schematic diagram of an embodiment of S301; Figure 5 For the present invention Figure 3 A schematic diagram of an embodiment of S303; Figure 6 A schematic flowchart of another embodiment of the website automated testing method provided by the present invention; Figure 7 This is a schematic flowchart of another embodiment of the website automated testing method provided by the present invention; Figure 8 This is a schematic diagram of an embodiment of the website automated testing device provided by the present invention; Figure 9 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0023] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0024] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0025] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0026] In some embodiments of the present invention, the website automated testing method is applied to a layered architecture system with a front-end and back-end separation. The system, from top to bottom, includes a front-end layer, an API gateway layer, a business logic layer, an AI orchestration engine layer, a data access layer, and an external service and container orchestration layer, with each layer communicating through standardized interfaces.

[0027] The front-end layer is developed based on Vue 3, TypeScript, Vite, and Arco Design, and provides a visual interactive interface for project management, requirement review, test case management, knowledge base operation, test execution, etc., and supports real-time test process feedback.

[0028] The API gateway layer is implemented based on the Django REST Framework and CORS, providing a unified HTTP / REST API and WebSocket interface to complete request routing, unified response format, access control, and JWT authentication functions.

[0029] The business logic layer comprises eight core modules: LangGraph API, Project Management API, Test Case API, Knowledge Base RAG API, MCP Tool API, Intelligent Orchestration API, Skills Management API, and Prompt Word Management API. It is responsible for the full lifecycle management of projects, test cases, knowledge bases, and tools.

[0030] The AI ​​orchestration engine layer uses LangGraph as its core workflow engine and integrates LLM configuration, Checkpointer state persistence, Tools layer, Middleware chain, and Agent Loop intelligent orchestration module.

[0031] The data access layer uses PostgreSQL or SQLite to store relational data (such as users, projects, and test cases) and Qdrant to store vector data (such as embedding vectors in knowledge base documents). The external service and container orchestration layer uses a Docker bridge network to implement containerized deployment of multiple services, including PostgreSQL, Redis, Qdrant, Xinference, Backend (Django service), MCP service, Playwright MCP browser automation service, and Draw.io chart editor.

[0032] Through this layered and decoupled architecture design, this embodiment enables the independent development, deployment, and expansion of each module, which helps to improve the maintainability and deployment flexibility of the system.

[0033] Figure 1 This is a flowchart illustrating an embodiment of the website automated testing method provided by the present invention. This website automated testing method is applied to the aforementioned layered architecture system with front-end and back-end separation, such as... Figure 1 As shown, the website's automated testing methods include: S101. Obtain the test documents uploaded by the user under the target project, parse, segment and vectorize the test documents, and store the processed vectors into a vector database to build a knowledge base associated with the target project.

[0034] Specifically, the system acquires various test documents uploaded by users under the target project (such as business requirement specifications, historical test reports, etc. in PDF, Word, Excel, PPT, or Markdown formats), parses, segments, and vectorizes these documents, and stores the resulting vectors in a vector database, thereby building a dedicated knowledge base for the project. This knowledge base achieves project-level resource isolation, ensuring that test documents from different projects do not interfere with each other.

[0035] S102. Receive the test requirements input by the user and start the AI ​​orchestration engine to load the large language model configuration, toolset and knowledge base associated with the target project. The AI ​​orchestration engine integrates a middleware chain, which includes, in the order of loading, a model call retry middleware, a tool call retry middleware, a context summary middleware, a manual approval middleware, and a streaming output middleware.

[0036] Specifically, users input website testing requirements (such as specifying functional modules, test scenarios, and test metrics) at the front-end layer and send the request to the LangGraph API workflow engine at the business logic layer through the API gateway layer.

[0037] Among these testing requirements is, for example, "testing the product ordering function of an e-commerce website, covering scenarios such as normal ordering, insufficient inventory, and payment failure."

[0038] The AI ​​orchestration engine is then launched, which automatically loads the configuration of the large language model associated with the target project (such as model name, context window size, whether streaming output is enabled, etc.), toolset (including knowledge base retrieval tools, browser automation tools, etc.), and the aforementioned knowledge base.

[0039] In a specific example, the LangGraph API loads the corresponding LLM configuration, MCP toolset, Skills, and knowledge base for the project, creates a LangGraph Agent executor, and configures the thread_id for Checkpointer state persistence. The Agent executor executes test tasks through the Agent Loop intelligent orchestration module. Combining the retrieval results from the RAG knowledge base, it generates test cases (including test points, test steps, and expected results) tailored to the business scenario using a large language model. The generated test cases are saved through the test case API and TestCase records are created in the data access layer.

[0040] Subsequently, based on the generated test cases, the AI ​​orchestration engine calls the Playwright MCP tool via the MCP tool protocol to automatically generate Playwright automated test scripts, or calls user-defined Skills to execute test operations. The Playwright MCP tool executes browser automation operations, and during the test process, it uses a middleware chain to implement retries for model or tool calls, context summaries, and manual approval for high-risk operations, and uses WebSocket to provide real-time feedback on the test process to the front-end layer.

[0041] It's worth noting that the AI ​​orchestration engine integrates a middleware chain, which, in the order of loading, includes model call retry middleware, tool call retry middleware, context summary middleware, manual approval middleware, and streaming output middleware. These middleware components are used to automatically retry when model or tool calls fail, intelligently compress dialogue contexts when they are too long, request manual confirmation before high-risk operations, and provide real-time feedback on the execution process to the front end, respectively.

[0042] The model call retry middleware is used to automatically initiate retries when large language model calls fail. As a preferred approach, this embodiment configures the number of retries to 3, employing an exponential backoff strategy, with the waiting time for each retry increasing sequentially to avoid call failures caused by instantaneous network fluctuations or service overload.

[0043] The tool call retry middleware is used to automatically retry when a tool or skill fails to perform a test operation. Similarly, this embodiment is configured to retries 3 times, also employing an exponential backoff strategy to ensure that calls that fail due to the temporary unavailability of external tool services can be effectively recovered.

[0044] The context summary middleware is used to address the issue of excessively long dialogue contexts during test task execution. When the token usage of the current session reaches a preset threshold (e.g., 75% of the context window limit), the middleware automatically calls the Large Language Model to compress the historical context into a summary and retains the most recent preset number of original dialogue records (e.g., retaining the 4 most recent core records), thereby avoiding exceeding the LLM's context window limit and ensuring the stable execution of long-running test tasks.

[0045] The manual approval middleware (i.e., the HITL middleware) is used for manual intervention and control of high-risk tool operations. Before the AI ​​orchestration engine calls the preset high-risk tools (such as database modification, payment simulation, etc.), this middleware triggers a manual approval process, waiting for the user's response via WebSocket or polling, and deciding whether to continue or terminate the call based on the user's choice, thereby improving the security and controllability of the testing process.

[0046] The streaming output middleware is used to provide real-time feedback on the execution process of test tasks to the front end. This middleware uses the Server Send Events (SSE) protocol to push information such as model output text, tool call logs, and execution status to the front end in a streaming manner, allowing users to understand the test progress in real time and improving the interactive experience.

[0047] These middleware components work together to address issues such as fault tolerance when models or tools fail to be invoked, performance issues when dialogue contexts are too long, security control of high-risk operations, and real-time feedback during execution, thereby significantly improving the robustness, controllability, and user experience of the testing process.

[0048] In one specific implementation of this embodiment, the aforementioned AI orchestration engine is implemented using an improved LangGraph AI orchestration engine module. This improved LangGraph AI orchestration engine module includes an LLM configuration management module, a Checkpointer state persistence module, a Tools layer, a Middleware chain, and an Agent Loop intelligent orchestration module. The specific implementation of each component is described below.

[0049] LLM configuration management supports adaptation of large language models from multiple vendors such as OpenAI, Azure, and Ollam. Configuration items include model name, whether vision is supported, context window size, whether to enable summarization, whether to enable manual approval, and whether to enable streaming output, which can be flexibly adjusted according to different testing scenarios.

[0050] Checkpointer state persistence uses "thread_id = username_id_project_id_session_id" as a unique identifier. SQLite is used in the development environment and PostgreSQL is used in the production environment to persistently store the execution state of test tasks, thereby supporting task breakpoint resumption and full process traceability.

[0051] The Tools layer integrates three major categories of tools: MCP Tools (remote tools), Knowledge Tools (knowledge base retrieval tools), and Skills Tools (user-defined skill tools), providing underlying support for AI orchestration.

[0052] The middleware chain is integrated in the following order: ModelRetryMiddleware (model call retry, using a 3-exponential backoff strategy), ToolRetryMiddleware (tool call retry, also using a 3-exponential backoff strategy), SummarizationMiddleware (context digest, triggered when token usage reaches 75% of the context window, retaining the 4 most recent core records), HumanInTheLoopMiddleware (triggers manual approval for high-risk tool operations), and StreamingMiddleware (streams output via the SSE protocol).

[0053] The agent executor creation process includes loading the specified LLM model, toolset, middleware chain, and system prompt words, and configuring recursion limits (supporting approximately 500 tool calls). During dialogue execution, the `ainvoke` method is called asynchronously, passing in a message list and a configuration item containing `thread_id`, and finally returning the execution status of the test task.

[0054] Through the above mechanism, this embodiment effectively solves problems such as model or tool call failure, context bloat, high-risk operation control, and real-time result feedback, and significantly improves the robustness and controllability of the testing process.

[0055] S103. Based on testing requirements, the AI ​​orchestration engine executes test tasks in a loop through agents to obtain test execution results.

[0056] Furthermore, after the test is completed, the system generates a test result report (including test case execution status, pass rate, and issue logs), which is fed back to the front-end layer for users to view. At the same time, the test results are associated with the test cases and stored to achieve full lifecycle traceability of the test process.

[0057] Additionally, it should be noted that users can also edit and optimize test cases at the front-end layer based on the test results, or supplement knowledge base documents. The system will then deposit the optimized information into the corresponding module to improve the quality of subsequent test case generation.

[0058] Through the complete process described above, this embodiment achieves closed-loop automation from project establishment, knowledge accumulation, test case generation to test execution and feedback optimization, significantly improving the efficiency and intelligence level of website automated testing.

[0059] In some embodiments of the present invention, such as Figure 2 As shown, step S101 involves parsing, segmenting, and vectorizing the test document, including: S201. Use a document parsing tool to extract the plain text content from the test document; S202. Use a recursive character text segmenter to segment the plain text content to obtain multiple text fragments; S203. Call the embedding model to convert each text fragment into a vector; S204. Store the vectors in the vector database and record the association between each text fragment and the target project and its source document.

[0060] Specifically, when extracting text content, the corresponding document parsing tool is automatically matched according to the text format. For example, the pypdf library is used for PDF files, the python-docx library is used for Word documents, and the openpyxl library is used for Excel files.

[0061] As a preferred method, when performing text segmentation, the segmentation block size is set to 512 characters and the overlap is set to 64 characters to ensure semantic coherence between adjacent segments. Alternatively, it can be configured to segment according to the heading level, depending on the actual application needs.

[0062] As a preferred approach, this embodiment employs the embedding service provided by the Xinference framework.

[0063] Finally, the vectors obtained from the text fragments are stored in the Qdrant vector database, and the association between each text fragment and the target project and its source document (such as project ID, document ID, fragment number, etc.) is recorded in the relational database.

[0064] This embodiment provides a structured knowledge base for subsequent retrieval enhancement generation, which helps to improve the fit between test cases and business scenarios.

[0065] In some embodiments of the present invention, such as Figure 3 As shown, step S103, based on testing requirements, involves the AI ​​orchestration engine executing test tasks through a loop via an agent to obtain test execution results, including: S301. Based on testing requirements, relevant text fragments are retrieved from the knowledge base as contextual information using an AI orchestration engine; S302. Input the context information and test requirements into the large language model to generate test cases corresponding to the test requirements. S303: The AI ​​orchestration engine executes the test tasks corresponding to the test cases through a proxy loop to obtain the test execution results.

[0066] The generated test cases include, but are not limited to, information such as test points, test steps, expected results, and priorities.

[0067] For example, for the requirement of "testing the product ordering function", the large language model may generate test cases such as "Test point: normal ordering process; Test steps: 1. Open the product details page, 2. Click to add to cart, 3. Proceed to checkout, 4. Fill in the shipping address, 5. Submit the order; Expected result: Order generated successfully, redirected to the payment page".

[0068] It should be noted that in this embodiment, when retrieving from the knowledge base using the AI ​​orchestration engine, a combination of RAG retrieval and large language model generation is used, which enables the generated test cases to make full use of project-specific knowledge and avoid omissions when manually sorting out test points.

[0069] In some embodiments of the present invention, such as Figure 4As shown, step S301, based on testing requirements, retrieves relevant text fragments from the knowledge base using an AI orchestration engine as contextual information, including: S401. Using the test requirements as query conditions, perform vector retrieval and / or keyword retrieval in the knowledge base to obtain initial search results; S402. Reorder the initial search results and select the text fragments with the highest similarity as context information.

[0070] Specifically, vector retrieval uses cosine similarity to calculate the distance between the query vector and vectors in the database, while keyword retrieval uses the BM25 algorithm for matching. These two retrieval methods can be used individually or in combination to obtain more comprehensive recall results.

[0071] After obtaining the initial search results, a re-ranking model (such as Xinference Reranker) is called to re-rank the initial results, and the text fragments with the highest similarity (such as Top 5) are selected as the final context information.

[0072] For example, in response to testing requirements for the "order placement function on e-commerce websites", the system may retrieve business document fragments such as "shopping cart checkout process", "inventory deduction rules", and "payment callback processing".

[0073] Through the above-described hybrid retrieval and reordering mechanism, this embodiment effectively improves the relevance and accuracy of contextual information, thereby enhancing the quality of subsequent test case generation.

[0074] In a preferred embodiment, the aforementioned retrieval and reordering functions are uniformly implemented by the RAG knowledge base retrieval enhancement module. The specific query and retrieval process includes: After a user initiates a test requirement query, the system performs vector retrieval in the knowledge base (filtered by TopK and similarity threshold), and can choose to combine it with BM25 keyword retrieval for mixed retrieval to obtain initial results; Then, Xinference Reranker is called to reorder the initial results and select the text fragments with the highest similarity (e.g., the top 5) as context information. The context, along with the test requirements, is input into the large language model to generate test cases or test suggestions; at the same time, a QueryLog is generated to record retrieval information for subsequent analysis.

[0075] Through the above-mentioned hybrid retrieval and re-ranking mechanism, this embodiment significantly improves the relevance and accuracy of retrieval results, thereby enhancing the fit between test cases and business scenarios.

[0076] In some embodiments of the present invention, such as Figure 5As shown, in step S303, the AI ​​orchestration engine executes the test tasks corresponding to the test cases through a proxy loop to obtain the test execution results, including: S501. Taking the test case as the target of this test task, obtain the historical summary and task status of the current test session. The historical summary records the compressed information of the historical loop steps, and the task status records the execution progress of the current test task. S502. Based on test cases, historical summaries, and task status, construct the current context and input the current context into the large language model to obtain the decision results output by the large language model; S503. Based on the decision result, invoke the corresponding tools or skills in the toolset to perform the test operation; S504. Record the proxy step information generated in this proxy loop and update the task status of the test task. The proxy step information includes the decision result in the current loop, the tools or skills called, and the corresponding execution result. S505. Based on the preset judgment conditions, determine whether the test task has been completed. If completed, use the execution result corresponding to the current test task as the test execution result; if not completed, return to the above steps of building the current context to update the task status.

[0077] The history summary records key information that was compressed in previous loop steps, such as completed operations and current progress; the task status records the current execution progress of the test task (such as the number of steps executed, operations to be executed, etc.).

[0078] The decision outcome could be something like "calling browser automation tools to execute the test" or "supplementing the knowledge base search".

[0079] After execution, the proxy step information generated in this proxy loop is recorded. This information includes the decision result of the current loop, the tools or skills invoked, and the corresponding execution results (such as success or failure of the operation, returned data, etc.). Simultaneously, the task status of the test task is updated. Then, based on preset judgment conditions, it is determined whether the test task has been completed. If completed, the execution result corresponding to the current test task is output as the test execution result; if not completed, the process returns to the steps of building the current context and updating the task status, repeating the loop until the task ends.

[0080] The preset judgment conditions can be that all test steps have been completed, or the maximum number of loops has been reached, etc. The specific conditions can be set according to the needs of the actual application scenario, and there are no restrictions here.

[0081] It should be noted that the proxy loop has a built-in context compression mechanism: when the token usage of the current session reaches a preset threshold (e.g., 90% of the context window limit), the large language model is automatically invoked to compress the historical context into a summary of 2-3 sentences, while retaining the most recent original dialogue record, thereby avoiding performance issues caused by excessively long contexts.

[0082] Through this cyclical execution and state management mechanism, this embodiment realizes the automated advancement and breakpoint resume of complex test tasks, which helps to improve the reliability and traceability of the test process.

[0083] In the preferred embodiment of this example, the agent loop described above is executed by the Agent Loop intelligent orchestration module. The internal execution details of the Agent Loop intelligent orchestration module are as follows: First, create AgentTask and AgentBlackboard. The blackboard is used to store various status information during task execution. Then it enters a loop for execution, with a configurable maximum number of steps (e.g., up to 500 steps). In each loop step, the system constructs the current context (including historical summaries and the current state), inputs the context into the large language model for consideration, and obtains the decision results output by the model. Use the appropriate tools or skills based on the decision-making results; Record the information for this agent step (i.e., AgentStep, including decision content, tools invoked, and execution results); update the task status in AgentBlackboard.

[0084] During this process, the token usage of the current session is continuously checked. When it reaches 90% of the context window limit, context compression is automatically triggered: the large language model is called to summarize the entire historical context into 2-3 sentences (covering completed operations, current progress, and next steps). The compression strategy is "all history → 1 summary + retain the most recent original record".

[0085] At the same time, it determines whether a human-machine loop (HITL) manual approval needs to be triggered, for example when the decision involves a high-risk operation.

[0086] Once the task is completed, a final response is generated, and an AutomationScript record is automatically created and saved.

[0087] Through this intelligent orchestration and context compression mechanism, this embodiment effectively solves the problem of context inflation in long dialogues and achieves stable progress of complex testing tasks.

[0088] In some embodiments of the present invention, based on the decision result, the corresponding tools or skills in the toolset are invoked to perform the test operation, including: When the decision is to execute the test, the browser automation tool is invoked based on the test cases through a standardized tool protocol, so that the browser automation tool can generate and execute automated test scripts according to the test cases.

[0089] For example, when a test case requires "clicking the Add to Cart button", browser automation tools (such as Playwright) will automatically generate and execute the corresponding click operation script.

[0090] As a preferred approach, the standardized tool protocol adopts the Model Context Protocol (MCP), which defines a unified set of tool invocation interfaces, enabling the system to flexibly connect to different browsers for automation implementation. Through this standardized invocation method, this embodiment avoids the scalability limitations of traditional hard-coded integration, facilitating subsequent replacement or addition of testing tools.

[0091] In some embodiments of the present invention, the standardized tool protocol is a model context protocol; such as Figure 6 As shown, browser automation tools are invoked based on test cases through a standardized tool protocol, including: S601. Create an MCP session corresponding to the browser automation tool through the session manager. The MCP session is isolated based on the user identifier, project identifier, and session identifier. S602. Send the automated test script to the browser automation tool via the MCP session.

[0092] In some embodiments of the present invention, the standardized tool protocol specifically adopts the Model Context Protocol (MCP). The process of calling browser automation tools based on test cases via MCP is as follows: First, an MCP session corresponding to the browser automation tool is created through a global session manager. This session manager uses a singleton pattern to uniformly manage all MCP sessions. To ensure data isolation between different users, projects, and test sessions, MCP sessions are isolated at three levels based on user ID, project ID, and session ID. For example, test sessions of the same user under different projects will be assigned different session IDs, ensuring they do not interfere with each other.

[0093] The automated test script is then sent through this MCP session to a browser automation tool (such as the PlaywrightMCP service), which executes the script in a separate browser instance.

[0094] Furthermore, the aforementioned MCP session management is uniformly handled by the MCP tool protocol call module.

[0095] Preferably, the MCP tool protocol call module adopts a four-layer architecture of "Django Backend – GlobalMCPSessionManager – MultiServerMCPClient – ​​MCP Server".

[0096] The GlobalMCPSessionManager, using a singleton pattern, is responsible for the unified management of the creation, retrieval, and destruction of all MCP sessions. Tool registration utilizes the `@mcp.tool()` decorator to define tool functions and configure tool descriptions, achieving standardized tool registration.

[0097] For persistent session management, a three-level isolation system is implemented based on the (user_id, project_id, session_id) triple, supporting state persistence across conversation rounds. In addition, an idle timeout cleanup mechanism is set up: when an MCP session is idle for more than 15 consecutive minutes, the session is automatically closed and related resources are released to avoid long-term resource occupation.

[0098] This standardized protocol call architecture replaces the traditional hard-coded tool integration method, enabling the system to flexibly interface with any third-party testing tool that supports the MCP protocol, thus enhancing the system's scalability. Simultaneously, the aforementioned session management method achieves secure, isolated, and efficient automated browser calls.

[0099] In some embodiments of the present invention, such as Figure 7 As shown, the website's automated testing methods also include: S701. Obtain the user-uploaded custom skill pack, which contains metadata description files and executable scripts; S702. When the AI ​​orchestration engine is detected to call a custom skill, the skill pack list is read to obtain the metadata description file corresponding to the custom skill in the skill pack list. S703, Build command call executable script based on metadata description file.

[0100] In some embodiments of the present invention, the above-described website automation testing method further includes a custom skills extension step.

[0101] Specifically, the system pre-acquires user-uploaded custom skill packages, which are typically ZIP archives containing a SKILL.md file (containing YAML-formatted pre-configuration and documentation), an executable script (such as script.py or script.js), and dependent resource files (in the assets directory).

[0102] Regarding skill pack loading, this embodiment employs a progressive loading mechanism: when the AI ​​orchestration engine starts, it first obtains a list of all installed skill packs (containing only skill names and brief descriptions). When it detects that the AI ​​orchestration engine needs to call a custom skill, it then reads the skill's metadata description file (SKILL.md detailed document) as needed, constructs the execution command according to the command format and parameter requirements defined in the metadata description file, and subsequently calls the execution function and passes in the session identifier to achieve state maintenance across calls.

[0103] In terms of execution and session management, it communicates with an independent Node.js process via the JSON-RPC protocol, creates an independent browser instance for each session, supports up to 20 concurrent sessions, and sets an automatic cleanup mechanism for 15 minutes of idle time.

[0104] For example, a user can upload a skill pack for "database status check". During the test, the AI ​​orchestration engine will call the skill based on the decision results, execute the script to query the order status in the database and return the results.

[0105] By using this plug-in-based skill extension approach, this embodiment meets the personalized testing needs of different projects, avoids the tedious process of modifying the core code for each new tool, and ensures the convenience and efficiency of personalized testing.

[0106] In some embodiments of the present invention, in order to ensure the safety and controllability of the testing process, a triple security mechanism is adopted, consisting of authentication and authorization, HITL human-in-the-Loop approval, and security protection.

[0107] In terms of authentication and authorization, JWT authentication (access token is valid for 12 hours and refresh token is valid for 7 days) and API Key authentication are used as fallback mechanisms. Combined with the Django permission system, role-based access control is implemented, which restricts the operation permissions of resources by the roles of project members (owner, administrator, member).

[0108] Regarding HITL manual approval, the require_hitl configuration item and the user's approval preferences are checked before the tool is called. For high-risk tool operations (such as database modification or high-risk browser operations), a manual approval process is triggered. The process waits for the user's response via WebSocket or polling, and then executes the continue or abort operation based on the user's choice.

[0109] In terms of security protection, it adopts Zip Slip protection (secure decompression skill package to prevent path traversal attacks), command injection protection, three-level session isolation, 30-minute task timeout protection, and full-process input verification.

[0110] Through these security mechanisms, this embodiment effectively avoids malicious attacks and resource abuse on the system, ensuring the safe and reliable operation of the automated testing process.

[0111] To better implement the website automated testing method in the embodiments of the present invention, based on the website automated testing method, correspondingly, as follows: Figure 8 As shown, this embodiment of the invention also provides a website automated testing device, the website automated testing device 800 including: Module 801 is used to obtain test documents uploaded by users under the target project, parse, segment and vectorize the test documents, and store the processed vectors into a vector database to build a knowledge base associated with the target project. The receiving module 802 is used to receive the test requirements input by the user and start the AI ​​orchestration engine to load the large language model configuration, toolset and knowledge base associated with the target project. The AI ​​orchestration engine integrates a middleware chain, which includes, in the order of loading, a model call retry middleware, a tool call retry middleware, a context summary middleware, a manual approval middleware, and a streaming output middleware. Test module 803 is used to execute test tasks in a loop through an agent based on test requirements, and obtain test execution results.

[0112] The website automated testing device 800 provided in the above embodiments can implement the technical solutions described in the above website automated testing method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above website automated testing method embodiments, and will not be repeated here.

[0113] like Figure 9 As shown, the present invention also provides an electronic device 900. The electronic device 900 includes a processor 901, a memory 902, and a display 903. Figure 9 Only some components of the electronic device 900 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0114] In some embodiments, processor 901 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 902 or process data, such as the website automated testing method of the present invention.

[0115] In some embodiments, processor 901 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 901 may be local or remote. In some embodiments, processor 901 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.

[0116] In some embodiments, memory 902 may be an internal storage unit of electronic device 900, such as a hard disk or memory of electronic device 900. In other embodiments, memory 902 may also be an external storage device of electronic device 900, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 900.

[0117] Furthermore, the memory 902 may include both internal storage units of the electronic device 900 and external storage devices. The memory 902 is used to store application software and various types of data installed on the electronic device 900.

[0118] In some embodiments, display 903 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 903 is used to display information from electronic device 900 and to display a visual user interface. Components 901-903 of electronic device 900 communicate with each other via a system bus.

[0119] In one embodiment, when processor 901 executes the website automation testing program in memory 902, the following steps can be performed: The system retrieves test documents uploaded by users under the target project, parses, segments, and vectorizes the test documents, and stores the resulting vectors into a vector database to build a knowledge base associated with the target project. Receive the test requirements input by the user and start the AI ​​orchestration engine. The AI ​​orchestration engine loads the large language model configuration, toolset and knowledge base associated with the target project. Based on testing requirements, the AI ​​orchestration engine executes test tasks in a loop through agents to obtain test execution results.

[0120] It should be understood that when the processor 901 executes the website automation test program in the memory 902, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0121] Furthermore, the embodiments of the present invention do not specifically limit the type of the electronic device 900 mentioned. The electronic device 900 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 900 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0122] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the website automated testing methods provided in the above-described method embodiments.

[0123] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0124] The website automated testing method, apparatus, and electronic equipment provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A website automated testing method, characterized in that, include: The test documents uploaded by the user under the target project are obtained, the test documents are parsed, segmented and vectorized, and the processed vectors are stored in the vector database to build a knowledge base associated with the target project. The system receives test requirements input by the user and starts the AI ​​orchestration engine to load the large language model configuration, toolset, and knowledge base associated with the target project. The AI ​​orchestration engine integrates a middleware chain, which includes, in the order of loading, a model call retry middleware, a tool call retry middleware, a context summary middleware, a manual approval middleware, and a streaming output middleware. Based on the aforementioned testing requirements, the AI ​​orchestration engine executes the testing tasks through a proxy loop to obtain the test execution results.

2. The website automated testing method according to claim 1, characterized in that, The parsing, segmentation, and vectorization of the test document includes: Use a document parsing tool to extract the plain text content from the test document; The plain text content is segmented using a recursive character text segmenter to obtain multiple text fragments; The embedding model is invoked to convert each of the text fragments into a vector; The vectors are stored in the vector database, and the association between each text fragment and the target project and its source document is recorded.

3. The website automated testing method according to claim 1, characterized in that, Based on the test requirements, the AI ​​orchestration engine executes test tasks through a proxy loop to obtain test execution results, including: Based on the aforementioned testing requirements, the AI ​​orchestration engine retrieves relevant text fragments from the knowledge base as contextual information. The context information and the test requirements are input into the large language model to generate test cases corresponding to the test requirements; The AI ​​orchestration engine executes the test tasks corresponding to the test cases through a proxy loop to obtain the test execution results.

4. The website automated testing method according to claim 3, characterized in that, The step of retrieving relevant text fragments from the knowledge base as context information using the AI ​​orchestration engine based on the test requirements includes: Using the test requirements as query conditions, perform vector retrieval and / or keyword retrieval in the knowledge base to obtain initial search results; The initial search results are reordered, and a preset number of text segments with the highest similarity are selected as the context information.

5. The website automated testing method according to claim 3, characterized in that, The process of the AI ​​orchestration engine executing the test tasks corresponding to the test cases through a proxy loop to obtain test execution results includes: Using the test case as the target of this test task, obtain the historical summary and task status of the current test session, wherein the historical summary records the compressed information of the historical loop steps, and the task status records the execution progress of the current test task; Based on the test cases, the historical summary, and the task status, a current context is constructed, and the current context is input into a large language model to obtain the decision result output by the large language model; Based on the decision result, the corresponding tools or skills in the toolset are invoked to perform the test operation; Record the proxy step information generated in this proxy loop and update the task status of the test task. The proxy step information includes the decision result in the current loop, the tools or skills called, and the corresponding execution result. Based on preset judgment conditions, determine whether the test task has been completed. If completed, use the execution result corresponding to the current test task as the test execution result; if not completed, return to the above steps of building the current context to update the task status.

6. The website automated testing method according to claim 5, characterized in that, The step of invoking the corresponding tools or skills in the toolset to perform test operations based on the decision result includes: When the decision is to execute the test, the browser automation tool is invoked based on the test case through a standardized tool protocol, so that the browser automation tool generates and executes automated test scripts according to the test case.

7. The website automated testing method according to claim 6, characterized in that, The standardized tool protocol is a model context protocol; the step of calling browser automation tools based on the test cases through the standardized tool protocol includes: An MCP session corresponding to the browser automation tool is created through the session manager. The MCP session is isolated based on the user identifier, project identifier, and session identifier. The automated test script is sent to the browser automation tool via the MCP session.

8. The website automated testing method according to any one of claims 1 to 7, characterized in that, The website automated testing method also includes: Obtain user-uploaded custom skill packs, which include metadata description files and executable scripts; When the AI ​​orchestration engine is detected to be calling the custom skill, the skill package list is read to obtain the metadata description file corresponding to the custom skill in the skill package list; The executable script is invoked based on the metadata description file.

9. A website automated testing device, characterized in that, include: The module is used to obtain test documents uploaded by users under the target project, parse, segment and vectorize the test documents, and store the processed vectors into a vector database to build a knowledge base associated with the target project. The receiving module is used to receive the test requirements input by the user and start the AI ​​orchestration engine to load the large language model configuration, toolset and knowledge base associated with the target project. The AI ​​orchestration engine integrates a middleware chain, which includes, in the order of loading, a model call retry middleware, a tool call retry middleware, a context summary middleware, a manual approval middleware, and a streaming output middleware. The testing module is used to execute test tasks through an agent loop by the AI ​​orchestration engine based on the test requirements, and obtain test execution results.

10. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the website automated testing method according to any one of claims 1 to 8.

Images