Agent-based object function inspection method, device and equipment and storage medium

Abstract

The application relates to an agent-based object function inspection method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: in response to a function inspection instruction for a detection object, a target inspection task matched with the detection object is determined from a plurality of candidate inspection tasks; according to each task action contained in the target inspection task, an agent is called to determine a detection tool matched with each task action from each candidate tool; according to the execution sequence of each task action in the target inspection task, each detection tool is called in sequence to execute the corresponding task action, and function inspection information is obtained; and a function inspection result of the detection object is generated based on the function inspection information. The method can automatically, efficiently and accurately complete function inspection.

Description

Technical Field

[0001] This application relates to the field of automated testing technology, and in particular to an object function inspection method, apparatus, computer equipment, storage medium and computer program product based on intelligent agents. Background Technology

[0002] As the informatization level of power systems and various enterprise-level web applications continues to deepen, power system-related business platforms typically include multiple web systems or modules such as equipment management, work order management, monitoring and alarm, report analysis, and access control. The functional modules of the system are becoming increasingly complex, and periodic inspections of key functions are required during daily operation to ensure system availability and business continuity.

[0003] Currently, the functional inspection of such systems mainly relies on manual operation, where maintenance personnel manually log into each system, perform operations, and record the results according to a predetermined process. However, manual inspection has two main drawbacks. First, it is inefficient. Faced with the need for inspection of multiple systems, multiple pages, and multiple time periods, manual operation is time-consuming and labor-intensive, making it difficult to achieve high-frequency, full coverage. Second, the quality of inspection is greatly affected by factors such as personnel experience and work status, which can easily lead to missed inspections, misjudgments, or inconsistent operations. Summary of the Invention

[0004] Therefore, it is necessary to provide an agent-based object function inspection method, device, computer equipment, computer-readable storage medium, and computer program product that can automatically, efficiently, and accurately complete function inspections, addressing the aforementioned technical problems.

[0005] Firstly, this application provides an object function inspection method based on intelligent agents. The method includes:

[0006] In response to a functional inspection command for a test object, a target inspection task matching the test object is determined from multiple candidate inspection tasks.

[0007] Based on the task actions included in the target inspection task, the intelligent agent is invoked to determine the detection tool that matches each of the task actions from each candidate tool.

[0008] According to the execution order of each task action in the target inspection task, each detection tool is called in sequence to execute the corresponding task action to obtain functional inspection information;

[0009] The functional inspection results of the detected object are generated based on the functional inspection information.

[0010] Secondly, this application also provides an object function inspection device based on an intelligent agent. The device includes:

[0011] The instruction response module is used to respond to the functional inspection instruction for the inspection object and determine the target inspection task that matches the inspection object from multiple candidate inspection tasks.

[0012] The detection tool determination module is used to call an intelligent agent to determine the detection tool that matches each of the task actions included in the target inspection task.

[0013] The functional inspection module is used to sequentially call each of the detection tools to execute the corresponding task actions according to the execution order of each task action in the target inspection task, so as to obtain functional inspection information.

[0014] The result generation module is used to generate the functional inspection results of the detected object based on the functional inspection information.

[0015] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described method.

[0016] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0017] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0018] The aforementioned agent-based object function inspection method, device, computer equipment, storage medium, and computer program product pre-set multiple candidate inspection tasks. When a function inspection of a specific object is required, a target inspection task matching the object can be determined from the multiple candidate tasks, ensuring a high degree of correlation between the target inspection task and the object, thus improving inspection accuracy and efficiency. Subsequently, based on the tasks and actions included in the target inspection task, the agent is invoked to determine the corresponding detection tool from the candidate tools. Following the execution order of the tasks and actions in the target inspection task, each detection tool is sequentially invoked to execute its corresponding task action, obtaining function inspection information. Based on this information, the function inspection result of the object is generated. This not only decouples and reuses detection tools and tasks, enhancing adaptability and scalability to different inspection scenarios, but also improves the standardization and repeatability of the inspection process, reducing inspection errors caused by manual operation or process errors. Therefore, it comprehensively improves the accuracy, stability, and flexibility of the system's automated function inspection, while reducing manual inspection costs and misjudgment analysis. Attached Figure Description

[0019] Figure 1 This is an application environment diagram of an agent-based object function inspection method in one embodiment.

[0020] Figure 2 This is a flowchart illustrating an agent-based object function inspection method in one embodiment.

[0021] Figure 3 This is a flowchart illustrating how, in one embodiment, each detection tool is sequentially invoked to execute its corresponding task action according to the execution order of each task action in the target inspection task, thereby obtaining functional inspection information.

[0022] Figure 4 This is a flowchart illustrating an agent-based object function inspection method in another embodiment.

[0023] Figure 5 This is a schematic diagram of the process of calling the login information acquisition tool to obtain login information from the non-login detection page in one embodiment.

[0024] Figure 6 This is a flowchart illustrating how an information input tool is invoked in one embodiment to simulate a delayed input action, inputting login information into the information input position of the non-login detection page, and generating a login status detection page.

[0025] Figure 7 This is a flowchart illustrating an agent-based object function inspection method in another embodiment.

[0026] Figure 8 This is a flowchart illustrating an agent-based object function inspection method in another embodiment.

[0027] Figure 9 This is a flowchart illustrating an agent-based object function inspection method in another embodiment.

[0028] Figure 10 This is a structural block diagram of an automated inspection intelligent agent system in one embodiment;

[0029] Figure 11 This is a schematic diagram of the overall processing flow of an automated inspection intelligent agent system in one embodiment;

[0030] Figure 12 This is a flowchart illustrating an agent-based object function inspection method in another embodiment.

[0031] Figure 13 This is a structural block diagram of an agent-based object function inspection device in one embodiment.

[0032] Figure 14This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] The object function inspection method based on intelligent agents provided in this application can be applied to, for example... Figure 1 In the application environment shown, the automated inspection intelligent agent system 102 communicates with the user terminal 104 via a network. A data storage system can store the data that the automated inspection intelligent agent system 102 needs to process. The data storage system can be integrated into the automated inspection intelligent agent system 102 or placed in the cloud or on other network servers. Responding to the functional inspection command for the inspection object from the user terminal 104, the automated inspection intelligent agent system 102 determines the target inspection task matching the inspection object from multiple candidate inspection tasks. Based on the task actions included in the target inspection task, it determines the inspection tool matching each task action from each candidate tool. Following the execution order of the task actions in the target inspection task, it sequentially calls each inspection tool to execute the corresponding task actions, obtaining functional inspection information. Based on the functional inspection information, it generates the functional inspection result of the inspection object, and finally displays the functional inspection result on the user terminal 104.

[0035] Among them, the automated inspection intelligent agent system 102 is an inspection system used for automated functional inspection of various inspection objects. It can be understood as an intelligent agent system that makes decisions based on a large language model and calls tools to execute based on the response results. It introduces a large language model as a decision engine, enabling the system to intelligently analyze and judge information collected during the inspection process, thereby dynamically adjusting inspection strategies and execution procedures. This intelligent agent system based on a large language model can better adapt to complex and changing inspection environments, improving inspection efficiency and accuracy.

[0036] In one embodiment, the automated inspection intelligent agent system 102 can be built based on the CrewAI intelligent agent framework, which encapsulates the inspection task into an intelligent agent task. The intelligent agent interacts with the underlying browser by automatically calling predefined tools. The underlying layer can use asynchronous frameworks such as Playback to control the headless / headed browser and use large language models, such as DashScope Qwen-VL-OCR and other large visual models, to realize the generation of inspection results.

[0037] The intelligent agent, based on a large language model decision engine, autonomously analyzes and judges the execution status of inspection tasks, and dynamically adjusts inspection strategies and execution processes based on the results, achieving intelligent and automated inspection. After the inspection task is completed, the intelligent agent can input all execution information, such as the tools used, tool parameters, and tool execution results, into the large language model. The model then analyzes and judges this information, outputting conclusions such as whether the inspection task was completed and whether any anomalies exist. In this way, a closed-loop intelligent agent system is formed, with task and tool inputs and large language model decision outputs, realizing the intelligent execution and dynamic adjustment of automated inspection tasks.

[0038] The user terminal 104 is the terminal device used by the object management user. It provides an interaction window between the object management user and the automated inspection intelligent agent system 102. The object management user can trigger functional inspection commands for each detected object to the automated inspection intelligent agent system 102 through the user terminal 104, and can also receive functional inspection results of the detected objects through the user terminal 104. It is understood that the user terminal 104 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc.

[0039] In one embodiment, the automated inspection intelligent agent system 102 can be mounted on a terminal, such as on a user terminal 104.

[0040] In one embodiment, the automated inspection intelligent agent system 102 can be implemented on a server, which can be a standalone server or a server cluster consisting of multiple servers.

[0041] In one embodiment, such as Figure 2 As shown, an object function inspection method based on an intelligent agent is provided, which is then applied to... Figure 1 Taking the automated inspection intelligent agent system 102 as an example, the following steps are included:

[0042] S202, in response to a functional inspection command for the object being inspected, determines a target inspection task that matches the object being inspected from multiple candidate inspection tasks.

[0043] The testing objects refer to software systems, application modules, or business components that require functional inspection, such as web systems or modules included in power system-related business platforms, including equipment management, work order management, monitoring and alarm, report analysis, and access control.

[0044] Functional inspection commands are instruction signals used to instruct the automated inspection intelligent agent system to perform functional inspections on the objects being inspected. They can be generated by maintenance personnel based on user terminals for specific objects being inspected, or they can be generated by maintenance personnel through user terminals by setting up timed tasks in the automated inspection intelligent agent system, such as automatically triggering functional inspection commands for the power report analysis webpage system at 2:00 AM every day.

[0045] Among them, candidate inspection tasks are inspection task instances that are pre-built and configured for different inspection objects. They can include a complete task description of a set of ordered task actions and their associated parameters that are pre-set for a specific inspection object, which is used to guide the automated inspection process.

[0046] For example, an automated inspection intelligent agent system can respond to functional inspection instructions for an object to be inspected, and search for a target inspection task that matches the object to be inspected from multiple candidate inspection tasks based on the object identifier of the object to be inspected.

[0047] S204, based on the various task actions included in the target inspection task, invoke the agent to determine the detection tool that matches each task action from the candidate tools.

[0048] In this context, a task action is a single executable operation unit that constitutes an inspection task. It can be understood as the smallest operational step in the inspection task. Each action corresponds to an operation type, such as click, input, navigation, wait, etc., and can be accompanied by parameters, such as element locator, input value, timeout, etc. For example, in the scenario of inspecting the functionality of a web system, clicking the login button is a task action, and its type is click.

[0049] Candidate tools are pre-defined functional modules designed to perform corresponding tasks during functional inspection. Each candidate tool encapsulates specific underlying operational capabilities and provides a unified calling interface. These underlying operational capabilities may include, but are not limited to, clicking elements, inputting text, and retrieving page content. Detection tools, on the other hand, are tools matched to the corresponding task actions.

[0050] In one embodiment, the candidate tool can be obtained by encapsulating the asynchronous browser automation capabilities of an automated browser testing tool. For example, automated browser testing tools may include, but are not limited to, Playwright, Puppeteer, and Selenium.

[0051] For example, after determining the target inspection task that matches the object to be inspected, the automated inspection agent system can, based on the task actions included in the target inspection task, call the agent to determine the inspection tool that matches each task action from each candidate tool.

[0052] In one embodiment, the target inspection task includes not only each task action, but also the tool identifier of the detection tool corresponding to each task action. The automated inspection intelligent agent system can determine the tool identifier corresponding to each task action based on the task information of the target inspection task, and for each task action, determine the detection tool that matches the corresponding task action from each candidate tool based on the tool identifier.

[0053] S206: According to the execution order of each task action in the target inspection task, each detection tool is called in sequence to execute the corresponding task action to obtain functional inspection information.

[0054] The execution order of each task action is the sequential arrangement of the tasks in the target inspection task, which can reflect the normal flow of the business logic of the inspection object. For example, the task action of entering a username must be executed before the task action of clicking login, which is determined by the execution order.

[0055] Among them, functional inspection information is intermediate information collected during the execution of each task action, such as intermediate results, logs, screenshots, execution results, etc. during the execution of each task action. In essence, it is the raw data and status record generated during the task execution process, which can be used to generate subsequent functional inspection results.

[0056] For example, the automated inspection intelligent agent system can sequentially call each detection tool to execute the corresponding task actions according to the execution order of each task action in the target inspection task, and obtain functional inspection information.

[0057] S208, Generate functional inspection results of the test object based on functional inspection information.

[0058] The functional inspection results are comprehensive conclusions formed by analyzing, summarizing, and judging the functional inspection information obtained during the execution of the target inspection task. They are usually presented in the form of structured data or reports and may include, but are not limited to, whether the target inspection task was successfully executed, the reasons for failure, a list of evidence, and statistical information.

[0059] For example, the automated inspection intelligent agent system can analyze the functional inspection results of the inspection object based on the functional inspection information and generate the functional inspection results of the inspection object.

[0060] In one embodiment, the functional inspection results may include structured results, JSON reports, HTML reports, etc. The structured results may include, but are not limited to, task execution status, action execution summary, final Uniform Resource Locator (URI), and screenshot path list, facilitating subsequent statistics or reuse. JSON reports can be generated by summarizing multi-task inspection results and may include, but are not limited to, timestamps, success / failure statistics, and JSON files containing output details for each task. HTML reports can be generated visual reports or single-file reports created by parsing screenshot paths recorded in the functional inspection information, reading the image files, and embedding them into HTML using a preset encoding mechanism, such as Base64 encoding. HTML reports can be opened and viewed directly without external image dependencies, facilitating offline archiving and distribution.

[0061] In the aforementioned agent-based object function inspection method, multiple candidate inspection tasks are pre-set. When a function inspection of a certain object is required, a target inspection task matching the object can be determined from the multiple candidate tasks, ensuring a high degree of correlation between the target inspection task and the object, thus improving inspection accuracy and efficiency. Subsequently, based on the task actions included in the target inspection task, the agent is invoked to determine the detection tool matching each task action from the candidate tools. Following the execution order of the task actions in the target inspection task, each detection tool is sequentially invoked to execute the corresponding task actions, obtaining function inspection information. Based on this information, the function inspection result of the object is generated. This method not only decouples and reuses detection tools and tasks, enhancing adaptability and scalability to different inspection scenarios, but also improves the standardization and repeatability of the inspection process, reducing inspection errors caused by manual operation or process errors. Therefore, it comprehensively improves the accuracy, stability, and flexibility of the system's automated function inspection, while reducing manual inspection costs and misjudgment analysis.

[0062] In one embodiment, after the intelligent agent in the automated inspection intelligent agent system completes the task execution, it can send back the tools called during the task execution process, the tool call parameters, and the tool execution results to the large language model, so that the large language model can continue to make decisions in subsequent rounds based on the returned information, until it is determined that all inspection tasks for the current inspection object have been completed or there is an anomaly.

[0063] In one embodiment, the target inspection task includes a detection environment creation task and a functional detection task. The detection environment creation task includes actions such as browser instance creation, session context creation, and detection page creation. Figure 3 As shown in S206, following the execution order of each task action in the target inspection task, each detection tool is sequentially invoked to execute the corresponding task action, thereby obtaining functional inspection information, including:

[0064] S302 invokes the browser instance creation tool to create a browser instance and configure startup parameters for the browser instance based on the detection environment information of the detection object.

[0065] The browser instance creation tool is specifically designed to generate browser instances. It encapsulates the browser startup functionality, can receive environment configuration parameters, actually start the browser process through a low-level automation interface, and return a handle to the browser instance. For example, it can use PlayWright to actually start the browser process.

[0066] Among them, the detection environment information is a set of data used to describe the environment configuration required for detection execution. It is used to define how the browser starts and runs. For example, the detection environment information may include, but is not limited to, browser type, startup mode, viewport size, proxy settings, and whether to ignore certificate errors.

[0067] The browser instance is the browser process object in the execution of the object detection task. It is an abstract identifier of the browser process actually launched by the underlying driver. All subsequent page operations will be based on the browser instance, such as creating a new page, navigation, and clicking.

[0068] Startup parameters are specific configuration items that control browser behavior. They are specific parameters passed to the underlying driver when a browser instance is created, used to customize how the browser runs, such as headless mode, headed mode, specifying a user agent, loading extensions, etc.

[0069] For example, the automated inspection intelligent agent system can call the browser instance creation tool to determine the creation parameters and startup parameters of the browser instance based on the detection environment information of the object being inspected, create the browser instance based on the creation parameters, and configure the startup parameters for the browser instance.

[0070] S304 If a browser state file corresponding to the detected object exists in local storage, and the login state represented by the browser state file is valid, call the context creation tool to load the browser state file and create a reused session context on the browser instance.

[0071] Local storage refers to the file system or persistent storage medium of the device on which the automated inspection intelligent agent system is running, used to save relevant state data of the browser session.

[0072] A browser state file is a file that represents the browser's login status. It stores browser-specific context information and can be loaded in subsequent sessions to restore the previous browsing state. The browser's login status represents the user's authentication status within the corresponding browser, indicating whether the user has successfully logged in. Login statuses can be understood as valid or invalid. If the browser state file represents a valid login status, it means the login information in the browser state file is still valid, has not been logged out by the server, or timed out, and can be directly used for subsequent checks. If the browser state file represents an invalid login status, it means the login information in the browser state file has expired and cannot be used directly.

[0073] Among them, the context creation tool is a tool used to generate browser contexts. It is a callable module that encapsulates the function of creating browser contexts. It can load contexts based on existing state files or create a completely new Gree environment.

[0074] Reusing a session context refers to a new browser context that loads historical state. The reused session context inherits from the previous browser state file, thus enabling session reuse.

[0075] For example, the automated inspection intelligent agent system can search in local storage for a browser state file corresponding to the detected object based on the object identifier of the detected object. If a browser state file corresponding to the detected object exists, it determines whether the login state represented by the browser state file is a valid state based on the information recorded in the file. If it is a valid state, it calls the context creation tool to create an initial session context, loads the browser state file, injects its login state into the initial context session, and obtains a reused session context.

[0076] In one embodiment, if the session cookie in the browser state file is still valid and has not been marked as invalid by the server, then the login status represented by the browser state file can be determined to be valid.

[0077] S306, The page creation tool is invoked to create a login state detection page in the reused session context based on the object address of the detection object.

[0078] Among them, the page creation tool is a tool specifically designed to generate new pages in a specific browser context. It encapsulates the function of creating a new page, can receive the parameters required for page creation, create a page instance through the underlying interface and return it.

[0079] The object address of the detected object refers to the Uniform Resource Locator (URL) of the detected object, which is the URL used by the browser to access the detected object and is used to identify the network location of the detected object.

[0080] The login state detection page is a page instance that has been created and is in a valid login state. That is, it is a new page created by the page creation tool in the reuse session context. The object address of the detection object has been loaded, and all subsequent functional detection actions are performed within this page.

[0081] For example, the automated inspection intelligent agent system can call a page creation tool to create a login-state detection page in a reused session context based on the object address of the detected object.

[0082] S308: According to the execution order of each function test action included in the function test task, the corresponding function test tools are called in sequence, and the corresponding function test actions are executed on the login state test page to obtain the function inspection information corresponding to each function test action.

[0083] Among them, the functional testing task refers to the specific functional testing task to be executed, which includes a complete definition of a series of ordered functional verification steps, and each functional verification step corresponds to a kinetic energy testing action.

[0084] Functional testing actions are the atomic operation units that constitute a functional testing task; that is, the smallest executable step in a functional testing task. Each action corresponds to an operation type and includes the parameters required for execution. The execution order of actions refers to the sequential arrangement of the functional testing actions within the functional testing task, reflecting the normal flow of business logic. A functional testing tool is a callable module selected to execute a specific functional testing action. It encapsulates low-level operational capabilities, can receive action parameters, and actually execute them.

[0085] For example, after creating the login state detection page, the automated inspection intelligent agent system can sequentially call the corresponding functional detection tools according to the execution order of each functional detection action included in the functional detection task, execute the corresponding functional detection action on the login state detection page, and obtain the functional inspection information corresponding to each functional detection action.

[0086] In one embodiment, it is understood that the functional inspection information may also include relevant information recorded during browser instance creation, session context creation, and detection page creation.

[0087] In one embodiment, after the target inspection task is successfully executed, the automated inspection agent system can automatically save the current browser context state back to the corresponding local storage area to ensure that the next inspection task can inherit the latest session credentials.

[0088] In the above embodiments, by checking the locally stored browser state file before each task starts, login state reuse can be quickly achieved based on the stored browser state file if a valid login state exists. This improves the persistence of browser session state and the success rate of cross-task reuse, thereby effectively improving the inspection efficiency of automated function inspection.

[0089] In one embodiment, such as Figure 4 As shown, the agent-based object function inspection method may also include:

[0090] S402, If a browser state file corresponding to the detected object does not exist in local storage, or if the login state represented by the browser state file is invalid, the context creation tool is invoked to create an initial session context on the browser instance.

[0091] The login status is invalid, which means that the login information in the browser's state file is no longer valid. This may be due to reasons such as server session timeout, user logout, or credential change, and cannot be used directly for subsequent operations.

[0092] The initial session context is a browser-isolated environment that has not loaded any historical state and does not contain any pre-saved cookies or stored data.

[0093] For example, if the automated inspection intelligent agent system determines that there is no browser state file corresponding to the detected object in the local storage, or that the login state represented by the browser state file is invalid, it can call the context creation tool to create an initial session context on the browser instance.

[0094] S404, The page creation tool is invoked to create an unlogged-in detection page in the initial session context based on the object address of the detected object.

[0095] Among them, the unauthenticated detection page is a page instance that is in an unauthenticated state. This page usually displays the login interface or the unauthenticated state and is used to perform login operations.

[0096] For example, the automated inspection intelligent agent system can call the page creation tool to create an unlogged-in inspection page in the initial session context based on the object address of the detected object.

[0097] S406, call the login information retrieval tool to obtain login information.

[0098] The login information retrieval tool is specifically designed to obtain the information required for login, encapsulating the function of obtaining login credentials. Login information refers to credential data used for identity authentication, such as username, password, and verification code.

[0099] For example, the automated inspection intelligent agent system can call a login information acquisition tool to obtain login information.

[0100] In one embodiment, the login information acquisition tool can obtain the login information from configuration, login failure detection page, password management service, or external sources.

[0101] S408, invoke the information input tool, simulate delayed input action to input login information into the information input position of the not logged in detection page, and generate the login status detection page.

[0102] Among them, the information input tool is a tool specifically designed for inputting information into page elements. It encapsulates the function of simulating user input, can simulate real keyboard or mouse input behavior, and is configured with input delay parameters.

[0103] Simulated delayed input refers to a gradual input behavior with time intervals. This means that when an information input tool performs input, it doesn't input instantly, like pasting or filling in all at once. Instead, it simulates the speed of human input by adding brief, random delays to each input step to reduce the risk of being detected as automated scripts by websites. For example, when inputting "4312", you can first input "4", wait a few milliseconds, and then input "3", and so on.

[0104] For example, the automated inspection intelligent agent system can call the information input tool to simulate a delayed input action to input login information into the information input position of the unlogged-in detection page, thereby generating a logged-in detection page.

[0105] In the above embodiments, when the saved session expires or does not exist, the automated inspection intelligent agent system can log in autonomously without human intervention, effectively ensuring the automatic execution of inspection tasks. Furthermore, the introduction of random delays in information input operations simulates delayed input actions, which can effectively reduce the risk of being identified as an automated script by the website and improve the inspection compatibility with systems that have anti-automation mechanisms.

[0106] In one embodiment, login information includes login verification information. For example... Figure 5 As shown in S406, the login information acquisition tool is invoked to obtain login information from the never-login detection page, including:

[0107] S502, if the login verification method on the non-login detection page is graphical verification, call the login information acquisition tool to locate the graphical verification element on the non-login detection page.

[0108] Image verification, in particular, is a verification mechanism based on graphic content. It requires users to identify and input characters, numbers, or text displayed in an image to prove that the user is human. For example, a login page might display an image containing distorted letters, requiring users to input content from the image for login verification.

[0109] Among them, the image verification element is the HTML element on the page that carries the image verification code, which is usually a... Label.

[0110] For example, when the automated inspection intelligent agent system determines that the login verification method of the unlogged detection page is graphical verification, it can call the login information acquisition tool to locate the graphical verification element on the unlogged detection page.

[0111] S504: Obtain the image source data of the graphic verification element based on the element attribute information of the graphic verification element.

[0112] The element attribute information consists of the various attribute values ​​of the image verification element, which can be used to locate and extract the image source. For example, the element attribute information can include the src attribute of the image verification element, which can be obtained directly via Base64 or downloaded via URL.

[0113] For example, after locating the graphic verification element, the automated inspection intelligent agent system can obtain the image source data of the graphic verification element based on the element attribute information of the graphic verification element.

[0114] S506 constructs a multimodal request message based on image source data and multiple preset message construction modalities.

[0115] The message construction modality is pre-defined and used to construct various modal types of multimodal request messages, such as text modality, image modality, etc.

[0116] Multimodal request messages are requests built to invoke visual recognition models. They contain request structures with multiple data modalities, enabling visual recognition models to understand recognition tasks and output results.

[0117] For example, the automated inspection intelligent agent system can construct multimodal request messages based on image metadata and multiple preset message building heads.

[0118] In one embodiment, the automated inspection intelligent agent system can construct modal information for each message construction modality based on image source data, and then construct a multimodal request message based on the modal information.

[0119] In one embodiment, the automated inspection intelligent agent system can first preprocess the image source data, and then construct multimodal request messages based on the preprocessed image source data and multiple preset message construction heads. For example, the automated inspection intelligent agent system can decode the image source data and convert it into RGB format, and then perform noise reduction and enhancement through median filtering and other methods to obtain preprocessed image source data, thereby improving the robustness of subsequent recognition.

[0120] S508 calls a pre-set visual recognition model to identify verification information in the multimodal request message and obtain login verification information.

[0121] Among them, the visual recognition model is a model tool capable of processing images and outputting login verification information. It can be trained based on deep learning models or visual language models. Leveraging its stronger image understanding capabilities, the visual recognition model can effectively identify distorted, overlapping, or distracting characters in verification images.

[0122] For example, the automated inspection intelligent agent system can call a pre-set visual recognition model to identify verification information in multimodal request messages and obtain login verification information.

[0123] In one embodiment, the multimodal request message may include an image and a text prompt, wherein the text prompt may request the visual recognition model to return in a structured format for easy parsing by the program, such as returning a structure containing a "number" field.

[0124] In the above embodiments, by integrating a visual recognition model into the automated inspection intelligent agent system, various types of image verification codes can be accurately identified, effectively improving the login success rate and reducing the risk of process interruption due to verification code recognition failure.

[0125] In one embodiment, such as Figure 6 As shown, in step S408, the information input tool is invoked to simulate a delayed input action, inputting login information into the information input position of the non-login detection page, generating a login status detection page, including:

[0126] S602, invoke the information input tool, simulate delayed input action to input login information into the information input position of the not logged in detection page, and trigger the information submission operation.

[0127] The information submission operation is the action performed after entering login information, used to send credentials to the server for verification. For example, clicking the login button.

[0128] For example, the automated inspection intelligent agent system can call the information input tool to simulate a delayed input action, inputting login information into the information input position of the unlogged-in inspection page, and triggering the information submission operation.

[0129] S604: If it is determined that a page has responded, obtain the page content of the responding page.

[0130] A page response refers to the action of the server updating the page and returning the corresponding response page after processing the request. A page response can refer to page redirection, partial refresh, or display of error messages.

[0131] For example, the automated inspection intelligent agent system can monitor unlogged-out detection pages in real time, and when a response is detected, obtain the page content of the response page. Page content refers to the information extracted from the response page used to determine the login result, such as text, elements, and other analyzable information.

[0132] S606: If the page content matches the preset verification error text and the number of verification retries has not reached the preset threshold, an update operation is triggered for the login verification information on the unlogged-in detection page.

[0133] The preset verification error text is a pre-configured error message template that may appear on the page when login fails. It is used to compare with the actual page content to determine whether login has failed. For example, the preset verification error text may include, but is not limited to, username and password mismatch, incorrect verification code, incorrect verification code, CAPTCHA error, etc.

[0134] The verification retry count is the number of times login information has been attempted to be identified and submitted in the current login process, used to control the retry limit. The preset threshold is the maximum number of retries allowed for the login task, used to prevent endless retries.

[0135] The login verification information update operation refers to the action performed to refresh the login verification information on the not-logged-in detection page.

[0136] For example, the automated inspection intelligent agent system can compare the page content with the preset verification error text. If the page content matches the preset verification error text, it can obtain the number of verification retries recorded in the current login process. If the number of verification retries does not reach the preset threshold, it means that the login action can be triggered again. The automated inspection intelligent agent system can trigger an update operation for the login verification information of the unlogged-in detection page.

[0137] S608, return to the step of calling the login information acquisition tool to locate the graphic verification element on the never-login detection page, until the page content does not match the preset error text, or the number of verification retries reaches the preset threshold.

[0138] For example, after triggering an update operation, the automated inspection intelligent agent system can return to the step of calling the login information acquisition tool to locate the graphic verification element on the unlogged detection page, until the page content does not match the preset error text, or the number of verification retries reaches the preset threshold.

[0139] In the above embodiments, by setting a preset verification error text, login failure caused by verification code recognition error can be accurately identified. At the same time, by setting a retry mechanism, login can be automatically retried if the retry threshold is not reached. Compared with the traditional one-time recognition scheme, it has a stronger self-recovery capability and effectively reduces the risk of overall inspection failure caused by a single recognition error.

[0140] In one embodiment, such as Figure 7 As shown, the agent-based object function inspection method may further include the following steps:

[0141] S702, for each function detection action, determine the detection waiting mechanism that matches the function detection action from among the candidate waiting mechanisms.

[0142] Among them, the candidate waiting mechanism is an event-driven waiting method predefined for specific action scenarios. It determines whether the page has reached a stable state after the action is executed by listening to specific page events. For example, specific page events may include the appearance of specific elements, URL changes, network idle time, etc.

[0143] For example, for each functional detection action, the automated inspection intelligent agent system can determine the detection waiting mechanism that matches the functional detection action from among the candidate waiting mechanisms.

[0144] S704 monitors the page response of the login state detection page after the function detection action is completed.

[0145] Page responsiveness refers to the dynamic changes that occur on the page after the functional testing actions are completed, including loading, rendering, and network requests. Examples include the appearance of new elements, URL changes, and the cessation of network activity.

[0146] For example, after the functional detection action is completed, the automated inspection intelligent agent system can continuously monitor the page response of the login state detection page.

[0147] S706: When the page response meets the triggering conditions of the detection waiting mechanism, determine the function inspection information corresponding to the function detection action based on the page content of the login state detection page.

[0148] The trigger condition for the detection waiting mechanism is the specific event condition that the detection waiting mechanism listens for, which is used to determine whether the page response has reached the expected state. When the trigger condition is met, it can be considered that the waiting has ended and the system can proceed with subsequent processing.

[0149] For example, when the automated inspection intelligent agent system detects a response on a page, it can compare the page response with the event triggering conditions corresponding to the detection waiting mechanism. If the comparison result indicates that the triggering conditions are met, it can determine that the page response meets the triggering conditions of the detection waiting mechanism. Based on the page content of the login-state detection page, it can determine the functional inspection information corresponding to the functional detection action.

[0150] For example, if the page displays the text "Welcome, admin" after the login action is completed, the page response can be considered to have met the triggering conditions of the detection waiting mechanism.

[0151] In one embodiment, the automated inspection intelligent agent system monitors the page response status of the login state detection page with a preset monitoring duration threshold. During the monitoring process, the automated inspection intelligent agent system will record the monitoring duration and compare it with the preset monitoring duration threshold. If the monitoring duration reaches the preset monitoring duration threshold, it will forcibly determine that the page response status does not meet the triggering conditions of the detection waiting mechanism, thereby preventing the occurrence of indefinite waiting.

[0152] In the above embodiments, by configuring a time-driven waiting mechanism for each function detection action, compared with the traditional fixed-time waiting mechanism, the system can accurately respond to changes in the actual state of the page. While reducing misjudgments caused by network fluctuations or page rendering delays, it achieves an adaptive waiting effect of "less waiting when fast and more waiting when slow".

[0153] In one embodiment, such as Figure 8 As shown, the agent-based object function inspection method may further include the following steps:

[0154] S802, in response to the task construction instruction for the newly added inspection object, obtains the inspection requirement information of the newly added inspection object.

[0155] Among them, newly added detection objects refer to detection entities that have not yet been configured with inspection tasks in the automated inspection intelligent agent system, such as software systems, application modules or business components that have not yet established corresponding inspection tasks.

[0156] Task creation instructions are signals used to instruct the creation of corresponding detection tasks for newly added detection objects. These instructions can be triggered and generated by operations and maintenance personnel through user terminals within the automated inspection intelligent agent system. For example, they can be generated directly by operations and maintenance personnel clicking on the interface, or indirectly by setting up scheduled tasks.

[0157] The inspection requirement information is provided by the operations and maintenance personnel and is used to define the descriptive information of the functional inspection requirements of newly added inspection objects. It may include the functional points to be verified, operation process, expected results, target URL, etc.

[0158] For example, the automated inspection intelligent agent system can respond to task construction instructions for newly added inspection objects and obtain inspection requirement information for the newly added inspection objects.

[0159] S804 extracts field information from the inspection requirement information according to the preset task configuration fields to obtain the inspection task configuration file.

[0160] The preset task configuration field is a predefined task configuration template structure, similar to a preset prompt word template. It is used to convert inspection requirement information into structured configuration, which may include task name, target URL, operation step list (each step includes action type, locator, input value, etc.), assertion conditions, timeout settings, etc.

[0161] The inspection task configuration file refers to a structured task definition file, such as a JSON file that is extracted from field information and conforms to the preset configuration field format, which fully describes the inspection task for the newly added inspection object.

[0162] For example, the automated inspection intelligent agent system can extract field information from the inspection requirement information according to the preset task configuration fields to obtain the inspection task configuration file.

[0163] S806, based on the inspection task configuration file, performs natural language conversion on each operation step included in the inspection task configuration, generating each task action and its corresponding task description.

[0164] Natural language processing (NLP) refers to the process of converting structured data into readable text. The automated inspection system can automatically translate the structured parameters of each operation step in the inspection task configuration into a natural language description based on a preset template, making the task intent clearer. The task description is the natural language explanation of the corresponding task action, that is, the intention or content of the action described in human language for each task action.

[0165] For example, the automated inspection intelligent agent system can perform natural language conversion on each operation step included in the inspection task configuration file to generate each task action and its corresponding task description.

[0166] In one embodiment, the automated inspection agent system can convert the various operational steps included in the inspection task configuration into natural language descriptions and structured parameters, dynamically generating a task description that can be understood by the agent development framework. The agent development framework may include, but is not limited to, CrewAI, langChain, langGraph, Vertex AI, etc.

[0167] In one embodiment, the process of the automated inspection intelligent agent system performing natural language conversion can be understood as constructing an inspection decision context for a large language model, and calling the large language model to perform reasoning and decision-making on the inspection decision context to obtain the inspection decision result for the current round.

[0168] S808 encapsulates each task action with its corresponding task description to obtain an inspection task that matches the newly added detection object.

[0169] For example, an automated inspection intelligent agent system can encapsulate each task action with its corresponding task description to obtain an inspection task that matches the newly added inspection object.

[0170] In one embodiment, the inspection task may include at least one of the following: a tool identifier to be executed, tool call parameters, and task completion judgment conditions.

[0171] In the above embodiments, by converting the inspection requirements information of newly added detection objects into a clear task description, the automated inspection intelligent agent system can no longer execute tasks based on rigid code, but can intelligently execute inspection tasks from the perspective of understanding the task objectives.

[0172] In one embodiment, such as Figure 9 As shown, the agent-based object function inspection method may further include the following steps:

[0173] S902: For any task action, if the execution result of the corresponding action is an execution error, obtain the error log of the task action.

[0174] Among them, the action execution result is the status returned by the automated inspection intelligent agent system after calling the detection tool to complete the corresponding task action, which can usually include execution success, execution error, etc.

[0175] The error log for a task action is text data that records detailed information about execution errors, such as context information captured when a task action fails. This may include, but is not limited to, error type, error message, stack trace, current page URL, screenshot, etc.

[0176] For example, for any task action, the automated inspection intelligent agent system can obtain the error log of the task action if the execution result of the corresponding action is an execution error.

[0177] S904, based on the task description and error log corresponding to the task action, performs error analysis to determine the cause of the error in the task action.

[0178] Error analysis refers to the analytical process of deducing the possible reasons for the failure of a task action.

[0179] For example, the automated inspection intelligent agent system can perform error analysis based on the task description and error log corresponding to the task action to determine the cause of the error. For example, the cause of the error may include element not being rendered, network timeout, locator failure, page jump abnormality, etc.

[0180] In one embodiment, the automated inspection intelligent agent system is pre-configured with an error analysis model. The task description and error log corresponding to the task action can be input into the error analysis model to determine the cause of the error. The error analysis model can be a pre-trained learning model or a large language model.

[0181] S906, adjust the task action according to the cause of the error to obtain the adjusted task action.

[0182] Adjusting task actions refers to the process of dynamically modifying task action parameters or execution strategies based on the cause of the error. This may include, for example, changing the locator, adding waiting conditions, retrying, skipping, or performing remedial operations.

[0183] For example, the automated inspection intelligent agent system can adjust the task actions according to the cause of the error to obtain the adjusted task actions.

[0184] For example, if the error is due to a selector failure, since the automated inspection agent system executes tasks based on the task description rather than fixed steps, it can automatically try to replace other available selectors to achieve the expected goals described in the task description, instead of simply judging the inspection as a failure because fixed steps cannot be executed.

[0185] S908 performs functional inspections on the detected object based on the adjusted task actions, and obtains functional inspection information.

[0186] For example, an automated inspection intelligent agent system can perform functional inspections on the objects to be inspected based on adjusted task actions, and obtain functional inspection information.

[0187] In the above embodiments, by introducing an error analysis mechanism and a dynamic action adjustment mechanism, the adaptive fault tolerance capability of the automated inspection intelligent agent system is realized. The system can automatically perform error analysis and make adjustment decisions based on feedback, effectively improving the robustness and success rate of inspection tasks.

[0188] In one embodiment, an agent-based object function inspection method is provided, which can be applied to, for example... Figure 10 The following explanation uses an automated inspection intelligent agent system as an example. Figure 10 As shown, the automated inspection intelligent agent system may include an automated inspection intelligent agent, a large language service (LLM service), a visual language model (VL model), a tool layer, a report generation module, and a browser automation engine.

[0189] The automated inspection intelligent agent system adopts an overall technical approach of "task configuration—agent orchestration—browser action execution—CAPTCHA recognition closed loop—session persistence—evidence chain reporting." The system is built on the CrewAI intelligent agent framework, encapsulating inspection tasks as agent tasks. The agent interacts with the underlying browser by autonomously invoking predefined tools. The underlying layer uses the Playwright asynchronous framework to control headless / headed browsers, utilizing large-scale visual models such as DashScope Qwen-VL-OCR to achieve CAPTCHA recognition, ultimately generating a visually rich inspection report.

[0190] In one embodiment, such as Figure 11 As shown, during the inspection process, the automated inspection agent system can first design prompts based on the inspection tasks set in the configuration file and the provided Playwright toolchain. It then calls a large language model to make decisions, obtains the necessary tools and parameters, executes the toolchain, and returns the execution results to the large language model. The model then decides whether the task is complete and whether there are any abnormalities in the inspected system's functionality. In this way, a closed-loop agent system is formed with task and tool inputs and a large language model decision output, achieving intelligent execution and dynamic adjustment of automated inspection tasks.

[0191] Overall, such as Figure 12 As shown, the agent-based object function inspection method may include the following steps:

[0192] S1201, Configuration-driven inspection task modeling and distribution.

[0193] For example, the automated inspection intelligent agent system defines the metadata of inspection tasks through a JSON configuration file, models the inspection requirements in a structured configuration manner, and each inspection task includes at least:

[0194] Task identification information may include task name, task description, and target URL.

[0195] An action sequence consists of several action objects, each of which contains at least an action type field `action` and carries parameter fields according to the action type.

[0196] The expected results are described, along with the semantic constraints used to generate the inspection conclusions (such as "the Dashboard title should appear after successful login").

[0197] The action object may include, but is not limited to:

[0198] navigate: The parameter includes the URL;

[0199] fill: Parameters include selector and value;

[0200] click: Parameters include selector;

[0201] wait: Parameters include selector and timeout;

[0202] wait_url: Parameters include url, timeout, and wait_until;

[0203] check_text: Parameters include selector and expected;

[0204] solve_captcha: Parameters include image_selector and input_selector;

[0205] submit_with_captcha_retry: parameters include image_selector, input_selector, submit_selector, error_text, max_retries;

[0206] screenshot: can be without parameters;

[0207] sleep: The parameter includes sleep (seconds).

[0208] This configuration-driven approach decouples the inspection process from business changes, enabling rapid iteration and expansion of inspection strategies.

[0209] S1202, Intelligent Agent Orchestration and Tool Calling Based on CrewAI.

[0210] For example, the automated inspection agent system can transform the operation steps in the JSON configuration into natural language descriptions and structured parameters, dynamically generating CrewAI's task description. This enables the agent to not only execute rigid code, but also understand the task objectives and make decisions based on feedback (such as analyzing error logs through LLM services and adjusting subsequent actions).

[0211] The inspection agent can perform the following processes:

[0212] First, read the environment variables and configurations (including LLM connection parameters, result output directory, task configuration file path, etc.).

[0213] Second, verify the validity of critical configurations.

[0214] Third, during the initialization of the intelligent agent's runtime, a "task-tool" mapping relationship is established, enabling the intelligent agent to select and call the browser control tool to execute inspection steps based on the task description.

[0215] Fourth, record key logs during the execution process for use in subsequent report generation and problem localization.

[0216] S1203, based on Playwright's browser context management and action scheduling execution.

[0217] For example, the automated inspection intelligent agent system can encapsulate the automation capabilities of the Playwright asynchronous browser into a CrewAI tool, forming a unified action scheduler whose execution process includes:

[0218] First, browser startup: Create a browser instance and configure startup parameters;

[0219] Second, context creation: Create BrowserContext and enable the ability to ignore HTTPS certificate errors in applicable scenarios to adapt to the self-signed certificate environment of the power intranet;

[0220] Third, page creation and access: Create a new page and access the target URL;

[0221] Fourth, action dispatch and execution: Traverse the action sequence and execute the corresponding process for each action according to its `action` type:

[0222] A human-like input strategy of "clearing → delayed key input" is implemented for the fill action;

[0223] The wait action is configured to time out before the element appears;

[0224] The wait_url action waits for URL matching as configured and supports the `wait_until` condition;

[0225] Retrieve the element text from the check_text action and perform an expected value inclusion check;

[0226] The screenshot action is timestamped to generate filenames and written to the results directory to form trace evidence;

[0227] Wait after performing the action on the sleep action to handle loading delays or asynchronous rendering.

[0228] Fifth, results summary: During the execution of the action, accumulate a list of action results and a list of screenshot paths, and record the final URL and execution status;

[0229] Sixth, exception handling: capture execution exceptions and return structured error messages to avoid an uncontrollable state of "no response / no conclusion".

[0230] S1204, browser session state persistence and cross-task reuse.

[0231] For example, to solve the problem of repeated logins, an automated browser state management (AuthState Management) mechanism can be designed, which may include:

[0232] First, session context loading: Before each task starts, the system checks the locally stored browser state file (such as auth_state.json) and attempts to load the state to create a browser context, enabling login state reuse.

[0233] Second, intelligent context injection: If a valid state file exists, inject the state when initializing BrowserContext, so that the agent can directly access pages that require login to view, skipping the login step.

[0234] Third, state update and saving: After the task is successfully executed, the system will automatically save the current browser context state back to the local file to ensure that the next inspection task can inherit the latest session credentials.

[0235] S1205, a CAPTCHA recognition and closed-loop retry mechanism based on a multimodal visual model.

[0236] For example, for the most difficult graphical CAPTCHA in web login scenarios, the automated inspection intelligent agent system can adopt a technical solution of "image acquisition - image enhancement - multimodal recognition - filling and submission - error judgment - closed-loop retry", specifically including:

[0237] First, image acquisition: Use Playwright's "image_selector" tool to locate the CAPTCHA image element, read its src attribute (supports direct acquisition via Base64 or download via URL), and obtain the image source data.

[0238] Second, image preprocessing: decode the image source data and convert it to RGB uniformly, and then use methods such as median filtering to remove noise and enhance the recognition robustness.

[0239] Third, multimodal recognition invocation: The preprocessed CAPTCHA image is used to construct a multimodal request message (image + text prompt), which is then used to invoke the visual recognition model service for recognition. The text prompt can request the model to return in a structured format (e.g., a structure containing a number field) for easier program parsing. Unlike traditional OCR, the large visual model has stronger image understanding capabilities and can recognize distorted, overlapping, or distracting characters.

[0240] Fourth, result parsing and filling: Parse the content returned by the model, extract the verification code text, clear the verification code input box and fill it in in a human-like delayed input manner.

[0241] Fifth, submission and error judgment: After the submit button is clicked, wait for the page to respond, obtain the page content and determine whether the preset error text error_text (such as "verification code error") appears.

[0242] Sixth, closed-loop retry: If the determination fails, it will automatically enter the next round of recognition and submission; the number of retries is limited by max_retries; if no error text appears on the page, it will be determined as successful and the retry will end.

[0243] Through the aforementioned closed-loop mechanism, automated login is equipped with a self-recovery capability of "detecting failure - automatically correcting errors - trying again".

[0244] S1206, based on Playwright's asynchronous high-concurrency execution.

[0245] For example, an automated inspection intelligent agent system can leverage Python's asynchronous programming features, combined with Playwright's asynchronous API, to achieve multi-task concurrent inspection capabilities and non-blocking execution of browser actions. Specifically, this includes:

[0246] First, a precise waiting mechanism: abandoning the traditional fixed-number sleep, it adopts event-driven waiting mechanisms such as wait_for_selector and wait_for_url to ensure that the operation is executed only after the page element has actually been loaded, which greatly improves the efficiency and stability of inspection.

[0247] Second, motion simulation: introduce random delays in the fill (input) operation to simulate human keyboard input behavior and reduce the risk of being identified as an automated script by WAF (Web Application Firewall).

[0248] Third, adaptive environment: It supports switching between headless and headed browser modes to adapt to different operating environment requirements (such as headless mode in a server environment without a GUI), and automatically configures to ignore HTTPS certificate errors, adapting to the power intranet environment.

[0249] S1207, Structured output of inspection results and generation of self-contained chain of evidence reports.

[0250] For example, after the inspection is completed, the result is not a simple log text, but a structured report generated in multiple formats, specifically including:

[0251] First, structured output results: The output should include at least fields such as execution status, action execution summary, final URL, and screenshot path list, to facilitate subsequent statistics and secondary consumption.

[0252] Second, JSON report: Summarize the results of multi-task inspections and generate a JSON file containing timestamps, success / failure statistics, and output details for each task.

[0253] Third, HTML report: Generates a visual report, parses the screenshot paths recorded in the inspection output, and embeds the image files into HTML in Base64 format to form a single-file report; this report can be opened and viewed directly without external image dependencies, which is convenient for offline archiving and distribution.

[0254] The aforementioned agent-based object function inspection method achieves several key improvements. First, through a combined mechanism of "image preprocessing + multimodal recognition + error text judgment + closed-loop retry," the inspection system can continuously run in login processes involving CAPTCHAs, reducing overall failures caused by a single recognition error and significantly improving the stability and availability of CAPTCHA scenarios. Second, inspection actions are described using a unified action protocol, allowing for rapid adaptation to business changes by adjusting action sequences (such as selectors, waiting conditions, and verification text), without frequent modifications to core code; transforming the inspection system from a "script engineering" to a "configuration engineering" approach, reducing operational costs. Third, through session state persistence and loading mechanisms, the system avoids re-interacting with logins and CAPTCHAs for each inspection, improving inspection efficiency and reducing the probability of triggering system security risk control or rate limiting policies, while also reducing additional pressure on business systems. Fourth, by combining action latency and environmental context simulation, the inspection behavior more closely resembles that of real users, reducing interference with the target system and the risk of being blocked. Fifth, to address common intranet issues such as self-signed certificates and network fluctuations, a browser context-level fault tolerance strategy is adopted (e.g., ignoring certificate errors, timeout waiting, exception handling, and structured error returns), making the inspection more continuous and robust in complex network environments. Sixth, screenshots are automatically taken and paths are recorded during the inspection process. When the report is generated, the screenshots are embedded in HTML as Base64 to form a self-contained report file. Combined with structured JSON reports, this can simultaneously meet the needs of both "manual review and viewing" and "system integration statistical analysis," making the inspection results traceable, auditable, and archiveable.

[0255] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0256] Based on the same inventive concept, this application also provides an agent-based object function inspection device for implementing the agent-based object function inspection method described above. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more agent-based object function inspection device embodiments provided below can be found in the limitations of the agent-based object function inspection method described above, and will not be repeated here.

[0257] In one embodiment, such as Figure 13 As shown, an object function inspection device 1300 based on an intelligent agent is provided, including: an instruction response module 1301, a detection tool determination module 1302, a function inspection module 1303, and a result generation module 1304, wherein:

[0258] The instruction response module 1301 is used to respond to the functional inspection instruction for the inspection object and determine the target inspection task that matches the inspection object from multiple candidate inspection tasks.

[0259] The detection tool determination module 1302 is used to call the intelligent agent to determine the detection tool that matches each task action according to each task action included in the target inspection task.

[0260] The functional inspection module 1303 is used to sequentially call each detection tool to execute the corresponding task actions according to the execution order of each task action in the target inspection task, so as to obtain functional inspection information.

[0261] The result generation module 1304 is used to generate functional inspection results of the inspection object based on the functional inspection information.

[0262] In one embodiment, the target inspection task includes a detection environment creation task and a function inspection task. The detection environment creation task includes tasks such as browser instance creation, session context creation, and detection page creation. The function inspection module 1303 is used to: call the browser instance creation tool to create a browser instance and configure startup parameters for the browser instance based on the detection environment information of the detection object; if a browser state file corresponding to the detection object exists in local storage and the login state represented by the browser state file is valid, call the context creation tool to load the browser state file and create a reused session context on the browser instance; call the page creation tool to create a login state detection page in the reused session context based on the object address of the detection object; and sequentially call the corresponding function inspection tools according to the execution order of the function inspection actions included in the function inspection task, execute the corresponding function inspection actions on the login state detection page, and obtain the function inspection information corresponding to each function inspection action.

[0263] In one embodiment, the functional inspection module 1303 is configured to: if there is no browser state file corresponding to the detected object in the local storage, or if the login state represented by the browser state file is an invalid state, call the context creation tool to create an initial session context on the browser instance; call the page creation tool to create an unlogged-in detection page in the initial session context based on the object address of the detected object; call the login information acquisition tool to acquire login information; and call the information input tool to simulate a delayed input action to input the login information into the information input position of the unlogged-in detection page, thereby generating a logged-in state detection page.

[0264] In one embodiment, login information includes login verification information. The functional inspection module 1303 is used to: when the login verification method on the unlogged-in detection page is graphical verification, invoke a login information acquisition tool to locate the graphical verification element on the unlogged-in detection page; obtain the image source data of the graphical verification element based on its element attribute information; construct a multimodal request message based on the image source data and multiple preset message construction modalities; and invoke a pre-set visual recognition model to identify the verification information in the multimodal request message to obtain the login verification information.

[0265] In one embodiment, the functional inspection module 1303 is used to: invoke an information input tool to simulate a delayed input action to input login information into the information input position of the login-not-logged-in detection page, triggering an information submission operation; if a response is determined to occur on the page, obtain the page content of the response page; if the page content matches the preset verification error text and the number of verification retries has not reached a preset threshold, trigger an update operation for the login verification information on the login-not-logged-in detection page; return to the step of invoking the login information acquisition tool to locate the graphic verification element on the login-not-logged-in detection page, until the page content does not match the preset error text, or the number of verification retries reaches a preset threshold.

[0266] In one embodiment, the agent-based object function inspection device 1300 further includes:

[0267] The waiting mechanism determination module is used to determine the detection waiting mechanism that matches the function detection action from among the candidate waiting mechanisms for each function detection action; the candidate waiting mechanisms are event-driven waiting mechanisms.

[0268] The response monitoring module is used to monitor the page response of the login status detection page after the function detection action is completed.

[0269] The condition triggering module is used to determine the function inspection information corresponding to the function detection action based on the page content of the login state detection page when the page response meets the triggering conditions of the detection waiting mechanism.

[0270] In one embodiment, the agent-based object function inspection device 1300 further includes:

[0271] The information acquisition module is used to obtain the inspection requirement information of the newly added inspection objects in response to the task construction instructions for the newly added inspection objects.

[0272] The information extraction module is used to extract field information of inspection requirements according to the preset task configuration fields, and obtain the inspection task configuration file.

[0273] The language conversion module is used to convert the operation steps contained in the inspection task configuration into natural language based on the inspection task configuration file, and generate each task action and its corresponding task description.

[0274] The encapsulation module is used to encapsulate each task action with its corresponding task description to obtain an inspection task that matches the newly added detection object.

[0275] In one embodiment, the agent-based object function inspection device 1300 further includes:

[0276] The error log acquisition module is used to acquire the error log of any task action when the execution result of the corresponding action is an execution error.

[0277] The error analysis module is used to analyze errors based on the task description and error logs corresponding to the task actions to determine the cause of errors in the task actions.

[0278] The action adjustment module is used to adjust the task actions according to the cause of the error, and obtain the adjusted task actions.

[0279] The functional inspection module is also used to perform functional inspections on the detection object based on the adjusted task actions, and obtain functional inspection information.

[0280] Each module in the aforementioned agent-based object function inspection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0281] In one embodiment, a computer device is provided, which may be a server integrated with an automated inspection intelligent agent system, and its internal structure diagram may be as follows: Figure 14As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores relevant data for an agent-based object function inspection method. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an agent-based object function inspection method.

[0282] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0283] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the specific implementation steps of the above-described method.

[0284] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the specific implementation steps of the above method.

[0285] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the specific implementation steps of the above-described method.

[0286] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the acquisition, storage, processing, and transmission of the data all comply with relevant laws and regulations.

[0287] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0288] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0289] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for inspecting object functions based on intelligent agents, characterized in that, The method includes: In response to a functional inspection command for a test object, a target inspection task matching the test object is determined from multiple candidate inspection tasks. Based on the task actions included in the target inspection task, the intelligent agent is invoked to determine the detection tool that matches each of the task actions from each candidate tool. According to the execution order of each task action in the target inspection task, each detection tool is called in sequence to execute the corresponding task action to obtain functional inspection information; The functional inspection results of the detected object are generated based on the functional inspection information.

2. The method according to claim 1, characterized in that, The target inspection task includes a detection environment creation task and a function detection task; the detection environment creation task includes tasks such as browser instance creation, session context creation, and detection page creation. The process involves sequentially calling each detection tool to execute its corresponding task action according to the execution order of the target inspection task, thereby obtaining functional inspection information, including: The browser instance creation tool is invoked to create a browser instance and configure startup parameters for the browser instance based on the detection environment information of the detection object. If a browser state file corresponding to the detected object exists in local storage, and the login state represented by the browser state file is valid, the context creation tool is invoked to load the browser state file and a reused session context is created on the browser instance. The page creation tool is invoked to create a login-state detection page in the reused session context based on the object address of the detected object; According to the execution order of each function detection action included in the function detection task, the corresponding function detection tools are called in sequence, and the corresponding function detection actions are executed on the login state detection page to obtain the function inspection information corresponding to each function detection action.

3. The method according to claim 2, characterized in that, The method further includes: If a browser state file corresponding to the detected object does not exist in the local storage, or if the login state represented by the browser state file is invalid, the context creation tool is invoked to create an initial session context on the browser instance. The page creation tool is invoked to create an unlogged-in detection page based on the object address of the detected object in the initial session context; Use the login information retrieval tool to obtain login information; The information input tool is invoked to simulate a delayed input action, inputting the login information into the information input position of the not-logged-in detection page, thereby generating a login status detection page.

4. The method according to claim 3, characterized in that, The login information includes login verification information; The step of calling the login information acquisition tool to obtain login information from the not-login detection page includes: If the login verification method on the not-login detection page is graphical verification, the login information acquisition tool is invoked to locate the graphical verification element from the not-login detection page; Based on the element attribute information of the graphic verification element, obtain the image source data of the graphic verification element; Based on the image source data and multiple preset message construction modalities, a multimodal request message is constructed; A pre-set visual recognition model is invoked to identify the verification information in the multimodal request message, thereby obtaining login verification information.

5. The method according to claim 3, characterized in that, The information input tool is invoked to simulate a delayed input action, inputting the login information into the information input position of the not-logged-in detection page, generating a login-state detection page, including: The information input tool is invoked to simulate a delayed input action, inputting the login information into the information input position of the not-logged-in detection page, thus triggering the information submission operation. Once it is determined that the page has responded, obtain the content of the responding page; If the page content matches the preset verification error text and the number of verification retries has not reached the preset threshold, an update operation is triggered for the login verification information of the not logged-in detection page. Return to the step of calling the login information acquisition tool to locate the graphic verification element from the not logged in detection page, until the page content does not match the preset error text, or the number of verification retries reaches the preset threshold.

6. The method according to any one of claims 2 to 5, characterized in that, The method further includes: For each function detection action, a detection waiting mechanism that matches the function detection action is determined from each candidate waiting mechanism; the candidate waiting mechanism is an event-driven waiting mechanism. After the function detection action is completed, monitor the page response of the login state detection page; If the page response meets the triggering conditions of the detection waiting mechanism, the function inspection information corresponding to the function detection action is determined based on the page content of the login state detection page.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: In response to a task construction instruction for a newly added detection object, obtain the inspection requirement information of the newly added detection object; According to the preset task configuration fields, the field information of the inspection requirement information is extracted to obtain the inspection task configuration file; Based on the inspection task configuration file, natural language conversion is performed on each operation step included in the inspection task configuration to generate each task action and the corresponding task description for each task action. Each task action is encapsulated with its corresponding task description to obtain an inspection task that matches the newly added detection object.

8. The method according to claim 7, characterized in that, The method further includes: For any task action, if the execution result of the corresponding task action is an execution error, obtain the error log of the task action; Based on the task description corresponding to the task action and the error log, perform error analysis to determine the cause of the error in the task action; Based on the cause of the error, the task action is adjusted to obtain the adjusted task action; Based on the adjusted task actions, a functional inspection is performed on the detection object to obtain functional inspection information.

9. An object function inspection device based on intelligent agents, characterized in that, The device includes: The instruction response module is used to respond to the functional inspection instruction for the inspection object and determine the target inspection task that matches the inspection object from multiple candidate inspection tasks. The detection tool determination module is used to call an intelligent agent to determine the detection tool that matches each of the task actions included in the target inspection task. The functional inspection module is used to sequentially call each of the detection tools to execute the corresponding task actions according to the execution order of each task action in the target inspection task, so as to obtain functional inspection information. The result generation module is used to generate the functional inspection results of the detected object based on the functional inspection information.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images