Webui adaptive testing system and method based on multi-agent cooperation and multi-modal perception

Abstract

The application relates to the technical field of adaptive testing, and particularly provides a WebUI adaptive testing system and method based on multi-agent cooperation and multi-modal perception, which comprises a multi-modal perception subsystem, an agent cooperative testing subsystem and an assertion and verification subsystem. The method comprises the following steps: adjusting element positioning strategies through a continuous interaction feedback mechanism for a multi-dimensional interface feature set, and outputting an element descriptor set; deploying a multi-agent cooperative decision unit comprising an interface understanding agent, a test logic agent and a data agent; outputting a semantic test script formed through a cooperative fusion mechanism; performing pixel-level difference calculation, identifying visual inconsistency areas through multi-scale image comparison; performing business-level logic difference, analyzing the consistency of interface element states and business rules, and outputting a hierarchical verification report. The application can maintain high coverage, low maintenance cost and reliable defect positioning capability of the test in an environment where pages frequently evolve.

Description

Technical Field

[0001] This invention relates to the field of adaptive testing technology, and in particular to a WebUI adaptive testing system and method based on multi-agent collaboration and multimodal perception. Background Technology

[0002] With the rapid iteration of web front-end technologies, such as single-page applications, micro-frontends, and dynamic rendering, traditional script-based UI automation testing can no longer meet the following needs: Diverse interfaces and frequently changing page structures lead to locator failure; cross-modal interaction, as modern web applications integrate text, images, and video, making single-text parsing insufficient to fully perceive the UI state; testing efficiency, as manually writing test cases is costly and regression testing cycles are long; and adaptive requirements, as different browsers, screen resolutions, and network conditions necessitate dynamic adjustments to testing strategies. To address these pain points, the industry has begun exploring the combination of multi-agent collaboration and multimodal perception to achieve more intelligent and adaptive Web UI testing systems. However, most solutions still suffer from key shortcomings such as single-agent perception, rigid collaboration strategies, insufficient cross-platform adaptability, and a lack of online learning.

[0003] Existing technology 1, Chinese patent application number 202510982642.4, discloses a web automation testing method and system, including automated script recording and dynamic adaptation, layered framework collaborative testing and data verification, and continuous integration testing and test report generation. The automated script recording and dynamic adaptation involves recording page operation scripts using Playwright and applying them to the UI layer. When the page UI changes, the script is re-recorded and replaces the original script, eliminating the need for element-by-element location and code modification. In the layered framework collaborative testing and data verification, the data access layer inputs test data, the UI layer executes page operations and retrieves display data, and the business logic layer calls API interfaces to obtain real data and compares it with the display data for verification. Although this method leverages Playwright's advantages to build a web automation testing framework, and the structured engineering project is more suitable for project operation and maintenance, and utilizes Jenkins to build a continuous integration testing environment, automating test input, execution, and output, the reliance on re-recording scripts after page UI changes still relies on specific DOM paths or coordinates for location, lacking a unified identifier for element semantics, leading to frequent script maintenance.

[0004] Prior art two, Chinese patent application number 202510855744.X, discloses a Python-based automated testing method for web systems, including the following steps: interface test data capture, page element acquisition, test framework construction, and creation of a visual testing platform combining UI and interface examples. Specifically, the interface test data capture uses the Charls tool for packet capture, sets up a proxy, installs an SSL certificate, and performs YAML file conversion after packet capture. Page element acquisition uses methods provided by Selenium to obtain relevant information about elements, analyzes and processes the page, and encapsulates it. While this method can improve the efficiency of web automated testing and avoids the problem of excessively high usage barriers, reducing project risks and labor costs, it only processes text / DOM or single visual information, failing to simultaneously utilize the visual, textual, and structural features of the page, resulting in incomplete perception of complex interactions.

[0005] Existing technology three, Chinese patent application number 202410063891.9, discloses an automated UI testing technology for web page Canvas elements based on deep learning, aiming to solve the limitations of traditional web testing methods when handling Canvas elements. It proposes a scriptless automated testing method by combining YOLO and OCR technologies. The core steps include: first, using YOLO technology to achieve object detection, and combining it with OCR technology to obtain component ID information; second, constructing a component structure tree based on the object detection results; then, generating random test cases through the component structure tree, and expanding the test cases through metamorphic relationships. Next, using Selenium to execute page operations, locating components based on the ID information in the script file. Finally, saving test cases with operational errors, as well as test results information for violating metamorphic relationships. Although it has higher efficiency and accuracy compared to traditional testing methods, and can effectively cope with the complexity and dynamism of Canvas elements; however, the object detection model needs to be retrained or its parameters tuned to adapt to new pages, limiting its adaptability.

[0006] Current technologies 1, 2, and 3 suffer from drawbacks such as unstable element localization semantics, single-modal awareness, insufficient business abstraction, weak cross-version adaptability, and a single verification level. Therefore, this invention provides a WebUI adaptive testing system and method based on multi-agent collaboration and multimodal awareness. Summary of the Invention

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In one aspect, the present invention provides a WebUI adaptive testing system based on multi-agent collaboration and multimodal perception, comprising:

[0009] The intelligent agent collaborative testing subsystem is used to deploy a multi-agent collaborative decision-making unit, including an interface understanding agent, a test logic agent, and a data agent; the output is a semantic test script formed through a collaborative fusion mechanism; the semantic test script describes the test scenario and operation sequence with high-level semantics;

[0010] The assertion and verification subsystem takes the actual interface state generated by the execution of the semantic test script as input and compares it with the expected state description embedded in the semantic test script. This includes: performing pixel-level difference calculations to identify visually inconsistent regions through multi-scale image comparison; performing business-level logic difference calculations to parse the consistency between the interface element state and business rules, and outputting a layered verification report; and integrating pixel-level and business-level difference details in the layered verification report to complete the test closed-loop verification.

[0011] In one optional implementation, the agent cooperative testing subsystem includes:

[0012] The semantic relation network construction component is used by the interface understanding agent to receive a set of element descriptors from the perception subsystem, and through its internal semantic relation network construction algorithm, to parse the hierarchy, interaction and logical relationship between elements and generate an interface structure graph rich in semantic relations.

[0013] The test logic skeleton weaving component is used by the test logic agent to obtain the interface structure graph. Through its internal dynamic logic weaving engine, it transforms the nodes and relationships in the graph into a series of ordered test action nodes, injects business logic constraints defined by the business rule library into this skeleton, and outputs a test process skeleton that carries the business rules.

[0014] The test script fusion generation component is used by the data intelligence agent to receive the test process skeleton. Through the internal contextual data binding mechanism, it associates and instantiates abstract data variables with specific test action nodes in the skeleton. The instantiation results and the test process skeleton are input into the collaborative fusion center of the data intelligence agent. After semantic alignment and logical verification, a semantic test script that directly drives the execution engine is generated.

[0015] In one optional implementation, the assertion and verification subsystem includes:

[0016] The multi-scale visual difference detection component is used to process the actual interface state generated by the execution of the semantic test script with the expected state description embedded in the script through a multi-resolution analysis program, extract visual features at different scales and perform alignment and comparison to generate a visual inconsistency region map; the visual inconsistency region map quantifies and records the spatial distribution and intensity of pixel-level differences.

[0017] The business logic state verification component is used to parse the actual interface element state and perform consistency checks with the business rules in the semantic test script, generating a business logic violation record; the business logic violation record describes in detail the deviation between the element state and the business rule.

[0018] The hierarchical verification report generation component is used to input business logic violation records and visual inconsistency area mappings into the report synthesizer for processing. It performs hierarchical classification and aggregation based on the difference type and severity, and outputs a hierarchical verification report that integrates pixel-level and business-level difference details. The hierarchical verification report completes the test closed-loop verification.

[0019] In one optional implementation, the multi-scale visual difference detection component includes:

[0020] The visual feature hierarchical decomposition sub-component is used to process the actual interface state and the expected state description through a frequency domain filter group, decomposing the input image into components of different frequency bands; the low-frequency components retain macroscopic layout features, the mid-frequency components capture the main body shape features, and the high-frequency components retain microscopic detail features, generating a structured multi-frequency band visual feature set.

[0021] The feature space adaptive alignment sub-component is used to establish and process multi-band visual feature sets through feature point correspondence, extract stable feature points in different frequency band layers and construct matching relationships; use the matched feature points to calculate geometric transformation parameters, perform coordinate system unification processing on each frequency band layer, and generate a multi-band feature set with spatially aligned position.

[0022] The difference region synthesis sub-component is used to process a set of spatially aligned multi-band features through weighted difference processing, assigning weight coefficients according to the importance of different bands in visual perception; the weighted difference data is then processed by connected component analysis, merging adjacent difference pixels into complete anomalous regions, and finally generating a visual inconsistency region map describing pixel-level differences.

[0023] In one optional implementation, the difference region synthesis sub-component includes:

[0024] The difference significance focusing module is used to process the weighted difference data through a sliding window scan, calculate the distribution statistics of pixel difference intensity in each local neighborhood, and generate potential extreme point location information. The potential extreme point location information is processed through neighborhood contrast analysis to calculate the degree of deviation of the difference intensity between each potential extreme point and its surrounding area, generating an extreme point set with significance scores. The extreme point set with significance scores is processed through spatiotemporal consistency checks to exclude transient extreme points caused by noise and generate a stable candidate seed point distribution map.

[0025] The spatial proximity growth processing module is used to process the candidate seed point distribution map through neighborhood similarity propagation, and to conditionally diffuse from the seed point to the surrounding pixels, including adjacent pixels that meet the similarity conditions into the growth range, and generating an initial set of abnormal regions.

[0026] The region boundary optimization processing module is used to process the initial set of abnormal regions through boundary continuity analysis, detect the fracture gaps at the edge of the region and perform bridging repair, and merge spatially overlapping adjacent regions to form a set of abnormal regions with complete outlines.

[0027] In one optional implementation, the spatial proximity growth processing module includes:

[0028] The initial neighborhood labeling processing submodule is used to process the candidate seed point distribution map through adjacent pixel scanning, establish an initial growth queue for each seed point, and mark its directly adjacent pixels as the state to be detected, thereby generating the initial growth structure.

[0029] The similarity neighborhood expansion processing submodule is used to process the initial growth structure through multiple similarity conditions. It sequentially detects the consistency of difference intensity, texture pattern continuity and spatial distance relationship between the pixel to be detected and the current growth region. Pixels that meet the comprehensive similarity conditions are incorporated into the growth region to generate the set of growth regions in the expansion process.

[0030] The region integrity determination submodule is used to process the set of growing regions in expansion through boundary activity detection, identify active growing boundaries that still have qualified neighboring pixels, use this boundary information to update the growth queue and trigger a new round of expansion until all region boundaries no longer have expandable pixels, and generate an initial set of abnormal regions.

[0031] In one optional implementation, the similarity neighborhood expansion processing submodule includes:

[0032] The difference intensity consistency measurement unit is used to process the initial growth structure through local intensity consistency measurement to obtain the statistical distribution matching degree of each pixel to be detected and the current growth region in terms of difference intensity. The initial growth structure is processed through sliding neighborhood statistics to calculate the distribution features of the difference intensity values ​​around each pixel to be detected, generating a statistical feature set containing the local intensity mean and variance. The statistical feature set is processed through relative entropy calculation to measure the similarity between the local intensity distribution of each pixel to be detected and the overall intensity distribution of the current growth region, generating a pixel-level distribution matching metric result. The pixel-level distribution matching metric result is processed through adaptive threshold segmentation to retain pixels with a matching degree higher than the dynamically calculated threshold, generating a candidate pixel set that satisfies intensity consistency.

[0033] The texture pattern continuity analysis unit processes the candidate pixel set that satisfies intensity consistency through a Gabor filter bank to extract texture response features at different directions and scales, generating pixel-level multi-scale texture feature vectors. The pixel-level multi-scale texture feature vectors are then processed by directional gradient consistency analysis to obtain the cosine similarity between each candidate pixel and the boundary pixels of the growing region in the texture feature space, generating a texture coherence score set. The texture coherence score set is then processed by region growing constraints to retain pixels with scores higher than the average level of the neighborhood, generating a candidate pixel set that satisfies texture continuity.

[0034] The spatial distance relationship comprehensive determination unit is used to process the candidate pixel set that satisfies texture continuity through proximity weight adjustment, and to perform final screening by combining the minimum spatial distance between the pixel and the growth region boundary and the density distribution relationship, so as to generate a pixel set that meets the similarity conditions; the pixel set that meets the similarity conditions is output as the pixel that can be incorporated into the growth region.

[0035] In one optional implementation, the spatial distance relationship comprehensive determination unit includes:

[0036] The geometric spatial distance mapping sub-unit is used to measure the spatial straight-line distance between the candidate pixel set that satisfies texture continuity and the pixel coordinate set of the growth region boundary, and to generate a spatial proximity dataset containing the minimum distance value.

[0037] The annular region density matching subunit is used to statistically analyze the pixel distribution density of the annular region guided by the spatial proximity dataset. It measures the similarity between the annular region density centered on the candidate pixel and the corresponding annular region density inside the growth region, forming a density matching degree dataset.

[0038] The spatial relationship comprehensive filtering subunit is used to scale and integrate the matching degree values ​​in the density matching degree dataset and the distance values ​​in the spatial proximity dataset using preset weight coefficients to generate an initial spatial relationship score for each candidate pixel. After normalization adjustment, the initial scores are aggregated to form a spatial relationship score set with unified dimensions. The spatial relationship score set is filtered through a preset threshold to output the final pixel set that meets the spatial relationship conditions.

[0039] In one optional implementation, a multimodal perception subsystem is further included, which is used to input a dynamic web interface, process the input through multimodal perception stream, integrate visual feature extraction, text semantic parsing and interface structure analysis to generate a multi-dimensional interface feature set; adjust the element positioning strategy of the multi-dimensional interface feature set through a continuous interactive feedback mechanism, and output a set of element descriptors with semantic consistency; the set of element descriptors contains stable semantic identifiers of dynamic interface elements.

[0040] In another aspect, the present invention provides a WebUI adaptive testing method based on multi-agent cooperation and multimodal awareness, for implementing the aforementioned WebUI adaptive testing system based on multi-agent cooperation and multimodal awareness. The WebUI adaptive testing method based on multi-agent cooperation and multimodal awareness includes the following steps:

[0041] The dynamic web interface is input and processed through multimodal perceptual stream processing, which integrates visual feature extraction, text semantic parsing, and interface structure analysis to generate a multi-dimensional interface feature set. The element localization strategy is adjusted through a continuous interactive feedback mechanism on the multi-dimensional interface feature set, and a set of element descriptors with semantic consistency is output. The set of element descriptors contains stable semantic identifiers of dynamic interface elements.

[0042] The deployment includes a multi-agent collaborative decision-making unit comprising an interface understanding agent, a test logic agent, and a data agent. The interface understanding agent parses the semantic relationships in the set of element descriptors to generate an interface structure graph. The test logic agent derives the test process skeleton based on the interface structure graph and injects business logic constraints. The data agent introduces test data variables in conjunction with the test process skeleton. The output is a semantic test script formed through a collaborative fusion mechanism. The semantic test script describes the test scenario and operation sequence with high-level semantics.

[0043] The actual interface state generated by the execution of the semantic test script is used as input and compared with the expected state description embedded in the semantic test script. This includes: performing pixel-level difference calculation and identifying visually inconsistent areas through multi-scale image comparison; performing business-level logic difference, analyzing the consistency between the interface element state and business rules, and outputting a layered verification report; the layered verification report integrates pixel-level and business-level difference details to complete the test closed-loop verification.

[0044] This invention utilizes a multimodal perception subsystem to perform unified abstraction of the page's visual, textual, and structural aspects. This enables the maintenance of stable semantic identifiers for key elements even when the page layout or content changes, achieving adaptive and robust element positioning. The intelligent agent collaborative testing subsystem, after parsing these semantic identifiers, constructs a complete interface structure graph. It generates a reusable test process skeleton using business constraints and combines it with variable test data to form a highly abstract and transferable semantic test script, allowing the test script to automatically adapt to different business scenarios and interface versions. After executing the script, the assertion and verification subsystem performs multi-level comparisons between the actual interface state and the expected description. Pixel-level comparisons capture subtle visual deviations, and multi-scale analysis ensures visual consistency across different resolutions and rendering environments. Business-level comparisons verify whether the interface state meets business rules, generating a unified layered verification report. Attached Figure Description

[0045] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0046] Figure 1 This is a block diagram of the WebUI adaptive testing system based on multi-agent collaboration and multimodal perception provided in Embodiment 1 of the present invention;

[0047] Figure 2 This is a schematic diagram of the WebUI adaptive testing system based on multi-agent collaboration and multimodal perception provided in Embodiment 1 of the present invention.

[0048] Figure 3 This is a block diagram of the multimodal sensing subsystem provided in Embodiment 2 of the present invention;

[0049] Figure 4 This is a block diagram of the intelligent agent collaborative testing subsystem provided in Embodiment 4 of the present invention;

[0050] Figure 5 This is a block diagram of the assertion and verification subsystem provided in Embodiment 7 of the present invention;

[0051] Figure 6 This is a flowchart of the WebUI adaptive testing method based on multi-agent cooperation and multimodal perception provided in Embodiment 13 of the present invention;

[0052] Figure 7 This is a schematic diagram of the WebUI adaptive testing method based on multi-agent cooperation and multimodal perception provided in Embodiment 13 of the present invention.

[0053] Figure 8 A block diagram of the electronic device provided by the present invention;

[0054] Figure 9 A block diagram of a computer-readable storage medium provided for this invention. Detailed Implementation

[0055] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0056] Hereinafter, the terms "first," "second," etc., are used for descriptive convenience only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0057] In this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed mechanical connection, a detachable mechanical connection, or an integral part; or, "connection" can be a direct connection or an indirect connection through an intermediate medium. Furthermore, unless otherwise explicitly specified and limited, the term "coupling" should be interpreted broadly. For example, "coupling" can be a direct electrical connection, such as physical contact and electrical conduction between two components; it can also be understood as an electrical connection between different components in a circuit structure through physical lines capable of transmitting electrical signals, such as copper foil or wires on a printed circuit board (PCB), to transmit electrical signals; or, "coupling" can be an indirect electrical connection between two components through an intermediate medium; or, "coupling" can be an electrical connection between two components in a non-contact manner, such as an electrical connection between two components using capacitive coupling to transmit electrical signals.

[0058] In this embodiment of the invention, directional terms such as "up," "down," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and can change accordingly depending on the orientation of the components in the accompanying drawings.

[0059] Example 1: As Figure 1 As shown, this embodiment of the invention provides a WebUI adaptive testing system based on multi-agent collaboration and multimodal perception, comprising:

[0060] The multimodal perception subsystem is used to take dynamic web interface input, process it through multimodal perception stream, integrate visual feature extraction, text semantic parsing and interface structure analysis to generate a multi-dimensional interface feature set; adjust the element localization strategy of the multi-dimensional interface feature set through a continuous interactive feedback mechanism, and output a set of element descriptors with semantic consistency; the set of element descriptors contains stable semantic identifiers of dynamic interface elements.

[0061] The interactive feedback mechanism refers to the closed-loop adjustment process in which the system continuously senses the difference between the actual interface state and the expected result during the test execution process, and dynamically transmits the difference information back to the multimodal perception subsystem. Specifically, when the test is executed, the system captures the visual changes, text content and structural responses of the interface in real time, and compares them with the expected state in the preset semantic test script. If element positioning deviation or semantic inconsistency is found, the feature extraction strategy or element descriptor generation logic in the multimodal perception flow is immediately adjusted. For example, the weight of visual features is recalibrated, the contextual association of text parsing is optimized, or the granularity of interface structure analysis is corrected. This enables the system to adapt to the dynamic changes of the Web interface, such as asynchronous loading and style switching, and ensures the stability of element positioning and semantic consistency. In this way, the accuracy and robustness of the test are maintained in the changing environment, and the adaptive loop of perception-execution-feedback is realized.

[0062] The intelligent agent collaborative testing subsystem is used to deploy a multi-agent collaborative decision-making unit, including an interface understanding agent, a test logic agent, and a data agent. The interface understanding agent parses the semantic relationships in the set of element descriptors and generates an interface structure graph. The test logic agent derives the test process skeleton based on the interface structure graph and injects business logic constraints. The data agent introduces test data variables in conjunction with the test process skeleton. The output is a semantic test script formed through a collaborative fusion mechanism. The semantic test script describes the test scenario and operation sequence with high-level semantics.

[0063] The assertion and verification subsystem takes the actual interface state generated by the execution of the semantic test script as input and compares it with the expected state description embedded in the semantic test script. This includes: performing pixel-level difference calculations to identify visually inconsistent regions through multi-scale image comparison; performing business-level logic difference calculations to parse the consistency between the interface element state and business rules, and outputting a layered verification report; and integrating pixel-level and business-level difference details in the layered verification report to complete the test closed-loop verification.

[0064] For details on the principles described in the above embodiments, please refer to the appendix. Figure 2This embodiment uses a multimodal perception subsystem to perform unified abstraction of the page's visual, textual, and structural aspects. This enables the maintenance of stable semantic identifiers for key elements even when the page layout or content changes, achieving adaptive and robust element positioning. The intelligent agent collaborative testing subsystem, after parsing these semantic identifiers, constructs a complete interface structure graph. It generates a reusable test process skeleton using business constraints and combines it with variable test data to form a highly abstract and transferable semantic test script, allowing the test script to automatically adapt to different business scenarios and interface versions. After executing the script, the assertion and verification subsystem performs multi-level comparisons between the actual interface state and the expected description. Pixel-level comparisons capture subtle visual deviations, and multi-scale analysis ensures visual consistency across different resolutions and rendering environments. Business-level comparisons verify whether the interface state meets business rules, generating a unified layered verification report.

[0065] In summary, this embodiment achieves a fully adaptive closed loop from page awareness and test generation to result verification, which can maintain high test coverage, low maintenance costs, and reliable defect localization capabilities in an environment where pages evolve frequently.

[0066] Example 2: As Figure 3 As shown, based on Embodiment 1, the multimodal sensing subsystem provided in this embodiment of the invention includes:

[0067] The multimodal feature semantic encoding component is used to process multi-dimensional interface feature sets through a deep semantic encoding network, mapping visual features, text semantics, and interface structural features to a unified semantic space to generate a set of primary semantic feature vectors; the set of primary semantic feature vectors contains preliminary semantic representations of interface elements;

[0068] The interactive feedback semantic optimization component is used to process the initial semantic feature vector set through a cross-modal attention mechanism, calculate the consistency score between visual, text, and structural features, and adjust the confidence of the semantic feature vectors based on the score; the adjusted semantic feature vectors are iteratively optimized through a reinforcement learning policy network to output an optimized semantic feature vector set.

[0069] The element descriptor generation component processes the optimized set of semantic feature vectors through a graph neural network to encode the semantic relationships and structural dependencies between interface elements, generating a set of element descriptors with semantic consistency; the set of element descriptors is output as a stable semantic identifier.

[0070] In the above embodiments, this embodiment uses deep semantic encoding to uniformly map visual, textual, and structural features, forming a preliminary semantic representation of elements. Cross-modal attention and reinforcement learning iteratively improve feature consistency and confidence, enabling semantic vectors to adapt to page changes and maintain reliability. Subsequently, graph neural networks are used to capture semantic relationships and structural dependencies between elements, generating stable and reusable element descriptors across versions. This embodiment achieves a closed loop of multimodal perception, semantic self-correction, and structured labeling for dynamic web interfaces, ensuring the robustness and semantic consistency of element localization and providing a reliable input foundation for testing agents.

[0071] Example 3: Based on Example 1, the interactive feedback semantic optimization component provided in this embodiment of the invention includes:

[0072] The intermodal attention weight generation subcomponent processes the primary semantic feature vector set through a cross-modal attention layer, calculates the interaction strength between visual, textual, and structural feature vectors, and generates an intermodal attention weight matrix; the intermodal attention weight matrix represents the dependencies between multimodal features;

[0073] The consistency score calculation sub-component processes the inter-modal attention weight matrix through a multimodal fusion function, aggregating the association strengths of visual, textual, and structural dimensions to generate a multimodal consistency score vector. The multimodal consistency score vector quantifies the degree of cross-modal consistency of each semantic feature vector.

[0074] The confidence adaptive adjustment sub-component processes the multimodal consistency score vector through a non-linear mapping mechanism, converting the consistency score into a confidence adjustment coefficient, which is then applied to the confidence value of the primary semantic feature vector set, outputting an adjusted semantic feature vector set. The adjusted semantic feature vector set serves as the optimization input for the reinforcement learning policy network.

[0075] In the above embodiments, this embodiment calculates the correlation strength between features through cross-modal attention to form a weight matrix; it uses a fusion function to aggregate the weights into a consistency score, quantifying the cross-modal matching degree of each feature; and then maps the score to a confidence coefficient to dynamically adjust the reliability of the feature vector. This embodiment achieves adaptive calibration of multimodal features, ensuring that semantic representations maintain high consistency among visual, textual, and structural information and possess adjustable confidence, providing reliable input for reinforcement learning optimization.

[0076] Example 4: Figure 4 As shown, based on Embodiment 1, the intelligent agent collaborative testing subsystem provided in this embodiment of the invention includes:

[0077] The semantic relation network construction component is used by the interface understanding agent to receive a set of element descriptors from the perception subsystem. Through its internal semantic relation network construction algorithm, it parses the hierarchy, interaction and logical relationship between elements to generate an interface structure graph rich in semantic relations. The interface structure graph describes the component composition and interaction context of the interface in a machine-readable form.

[0078] The test logic skeleton weaving component is used by the test logic agent to obtain the interface structure graph. Through its internal dynamic logic weaving engine, it transforms the nodes and relationships in the graph into a series of ordered test action nodes, injects business logic constraints defined by the business rule library into this skeleton, and outputs a test process skeleton that carries the business rules.

[0079] The test script fusion generation component is used by the data intelligence agent to receive the test process skeleton. Through the internal context data binding mechanism, it associates and instantiates abstract data variables with specific test action nodes in the skeleton. The instantiation result and the test process skeleton are input into the collaborative fusion center of the data intelligence agent. After semantic alignment and logical verification, a semantic test script that can directly drive the execution engine is finally generated.

[0080] Semantic alignment is the process by which the Collaborative Fusion Center addresses the inconsistency in the languages ​​of different intelligent agents when integrating information. The task is to establish a unified understanding framework that ensures that elements in the interface graph, the action intent in the test skeleton, and the specific values ​​in the data instances refer to the same thing and that the context matches. Specifically, this involves term mapping, such as associating a submit button in the graph with a click-to-submit operation in the skeleton; checking the compatibility of data variables with the types of the bound nodes, for example, the data bound to a username input box must be in text format; and verifying that the test logic intent matches the interactivity of the interface elements, for example, the node to be entered must correspond to an input text box in the graph. This ensures that all components collaborate under the same semantic consensus, avoiding action mismatches due to misunderstandings.

[0081] Logical verification, based on semantic consistency, involves conducting feasibility reviews and conflict resolution on the assembled test process. It primarily performs three checks: first, process integrity verification, ensuring the test action sequence is coherent and uninterrupted from start to finish, with all necessary steps in place; second, business rule conflict detection, analyzing whether multiple business constraints injected into the skeleton contradict each other—for example, one rule requiring a user to log in first, while another rule attempts to access a personal page without logging in; and third, precondition and postcondition verification, confirming that the execution prerequisites for each test action, such as specific page states and data readiness, can be correctly triggered and satisfied by their upstream actions. Only by passing all these verifications is the generated test script logically sound, reliable, and correctly executable.

[0082] In the above embodiments, this embodiment organizes the perceived element descriptors into a machine-readable structural graph through a semantic relationship network, capturing hierarchy, interaction, and logical associations. Based on this graph, the test logic skeleton transforms nodes and relationships into ordered test actions, embedding business rules into the action sequence to achieve automated expression of business constraints. Subsequently, the test script fusion component binds business data variables to action nodes, completes instantiation, and performs semantic alignment and logical verification in the collaborative fusion center, generating a semantic test script that can directly drive the execution engine. This embodiment achieves end-to-end automation from multimodal perception to structured semantic graph, business constraint test skeleton, and data instantiation, ensuring that the test script maintains semantic consistency, business accuracy, and direct executability when the page changes.

[0083] Example 5: Based on Example 4, the test script fusion generation component provided in this embodiment of the invention includes:

[0084] The data requirement mapping sub-component is used to test the process skeleton by extracting abstract requirements for data type, format and value range from the business logic constraints of the test action nodes through the data intent parsing program, and generating a data requirement profile.

[0085] The data context matching subcomponent is used to retrieve and match data requirement profiles through the context data space, and to filter out concrete data values ​​that match the current test context from the candidate data pool based on dynamic context features, generating a set of data context instances.

[0086] The data node binding sub-component is used to process the data context instance collection through the binding relationship generator, and to map and attach it one by one to the corresponding test action nodes in the test process skeleton, generating an instantiated test action sequence that carries specific data values.

[0087] In the above embodiments, this embodiment generates a data requirement profile by extracting the business constraints of test actions; retrieves and matches specific data values ​​that meet the requirements in the context data space; and then binds these specific data with the corresponding test actions one by one to form a complete instantiated test sequence with actual data, thereby realizing the automated data filling and executability of the test script.

[0088] Example 6: Based on Example 5, the data context matching sub-component provided in this embodiment of the invention includes:

[0089] The demand feature vectorization module is used to process the data demand profile through the demand vectorization model, transforming the abstract requirements regarding data type, format, and value range into a set of high-dimensional numerical features, generating a set of demand feature vectors.

[0090] The context space similarity retrieval module is used to process the set of demand feature vectors through an approximate nearest neighbor search algorithm in the context data space, and quickly locate potentially matching data entries from the candidate data pool based on vector similarity measurement to generate a preliminary candidate data set.

[0091] The dynamic context matching and filtering module is used to process the initial candidate data set through a context consistency filter, and to perform precise matching and verification by combining the dynamic context features acquired in real time, thereby filtering out concrete data values ​​that fully conform to the current test scenario and finally generating a set of data scenario instances.

[0092] In the above embodiments, this embodiment transforms abstract data requirements into high-dimensional feature vectors to achieve fast approximate proximity retrieval in the contextual data space; then, it uses real-time contextual features for precise filtering, eliminating mismatches and retaining only specific data values ​​that fully conform to the current test context; thus forming a closed loop from requirement abstraction to efficient similarity retrieval and then to precise contextual matching, so that the data context matching process has both the speed of large-scale retrieval and ensures the contextual consistency and accuracy of the results.

[0093] Example 7: Figure 5 As shown, based on Embodiment 1, the assertion and verification subsystem provided in this embodiment of the invention includes:

[0094] The multi-scale visual difference detection component is used to process the actual interface state generated by the execution of the semantic test script with the expected state description embedded in the script through a multi-resolution analysis program, extract visual features at different scales and perform alignment and comparison to generate a visual inconsistency region map; the visual inconsistency region map quantifies and records the spatial distribution and intensity of pixel-level differences.

[0095] The business logic state verification component is used to parse the actual interface element state and perform consistency checks with the business rules in the semantic test script, generating a business logic violation record; the business logic violation record describes in detail the deviation between the element state and the business rule.

[0096] The hierarchical verification report generation component is used to input business logic violation records and visual inconsistency area mappings into the report synthesizer for processing. It performs hierarchical classification and aggregation based on the difference type and severity, and outputs a hierarchical verification report that integrates pixel-level and business-level difference details. The hierarchical verification report completes the test closed-loop verification.

[0097] In the above embodiments, the multi-scale visual difference detection of this embodiment achieves pixel-level comparison across resolutions, accurately locates and quantifies visually inconsistent areas; the business logic state verification performs rule matching on the actual element state and records deviations at the business level; the hierarchical verification report categorizes and aggregates the two types of differences according to type and severity, forming a unified pixel and business level difference report, achieving complete closed-loop verification.

[0098] Example 8: Based on Example 7, the multi-scale visual difference detection component provided in this embodiment of the invention includes:

[0099] The visual feature hierarchical decomposition sub-component is used to process the actual interface state and the expected state description through a frequency domain filter group, decomposing the input image into components of different frequency bands; the low-frequency components retain macroscopic layout features, the mid-frequency components capture the main body shape features, and the high-frequency components retain microscopic detail features, generating a structured multi-frequency band visual feature set.

[0100] The feature space adaptive alignment sub-component is used to establish and process multi-band visual feature sets through feature point correspondence, extract stable feature points in different frequency band layers and construct matching relationships; use the matched feature points to calculate geometric transformation parameters, perform coordinate system unification processing on each frequency band layer, and generate a multi-band feature set with spatially aligned position.

[0101] The difference region synthesis sub-component is used to process a set of spatially aligned multi-band features through weighted difference processing, assigning weight coefficients according to the importance of different bands in visual perception; the weighted difference data is then processed by connected component analysis, merging adjacent difference pixels into complete anomalous regions, and finally generating a visual inconsistency region map describing pixel-level differences.

[0102] In the above embodiments, this embodiment obtains multi-band visual features of macroscopic layout, mesoscopic shape and microscopic details through frequency domain decomposition; stable feature points are extracted and matched in each frequency band to achieve cross-band geometric alignment and eliminate differences in viewpoint and scale; then, the differences are weighted according to the importance of the frequency band and connected regions are merged to obtain a structured pixel-level abnormal region mapping, so as to achieve accurate positioning of differences between the actual and expected interfaces at different visual levels.

[0103] Example 9: Based on Example 8, the differential region synthesis sub-component provided in this embodiment of the invention includes:

[0104] The difference significance focusing module is used to process the weighted difference data through a sliding window scan, calculate the distribution statistics of pixel difference intensity in each local neighborhood, and generate potential extreme point location information. The potential extreme point location information is processed through neighborhood contrast analysis to calculate the degree of deviation of the difference intensity between each potential extreme point and its surrounding area, generating an extreme point set with significance scores. The extreme point set with significance scores is processed through spatiotemporal consistency checks to exclude transient extreme points caused by noise and generate a stable candidate seed point distribution map.

[0105] The spatial proximity growth processing module is used to process the candidate seed point distribution map through neighborhood similarity propagation, and to conditionally diffuse from the seed point to the surrounding pixels, including adjacent pixels that meet the similarity conditions into the growth range, and generating an initial set of abnormal regions.

[0106] The similarity condition refers to the dual judgment criteria that a neighboring pixel to be examined must meet to be included in the current abnormal region. Specifically, it includes two core and quantified technical conditions: First, the difference features are similar, that is, the difference between the pixel's difference intensity value, such as color, brightness, and other quantitative indicators, and the average value of the current growth region or the value of the seed point is less than a preset dynamic threshold; second, the spatial location is adjacent, that is, the pixel must be directly adjacent to the current growth region in physical space, usually defined using 4-connected or 8-connected neighborhoods. Only when a pixel simultaneously meets both conditions of sufficiently similar feature values ​​and direct physical proximity will the system determine that it belongs to the same abnormality and include it in the region for growth and diffusion, ensuring that the features within the finally merged abnormal region are consistent and spatially continuous.

[0107] The region boundary optimization processing module is used to process the initial set of abnormal regions through boundary continuity analysis, detect the fracture gaps at the region edges and perform bridging repair, and merge spatially overlapping adjacent regions to finally form a set of abnormal regions with complete outlines.

[0108] In the above embodiments, this embodiment uses saliency focusing to locate reliable difference seed points and eliminate noise interference; similarity diffusion is performed starting from the seed points to quickly generate preliminary abnormal regions; then, continuity detection and bridging repair are performed on the region boundaries, and overlapping regions are merged to obtain a set of abnormal regions with complete contours and suppressed noise, thereby achieving accurate and robust extraction of visual differences.

[0109] Example 10: Based on Example 9, the spatial proximity growth processing module provided in this embodiment of the invention includes:

[0110] The initial neighborhood labeling processing submodule is used to process the candidate seed point distribution map through adjacent pixel scanning, establish an initial growth queue for each seed point, and mark its directly adjacent pixels as the state to be detected, thereby generating the initial growth structure.

[0111] The similarity neighborhood expansion processing submodule is used to process the initial growth structure through multiple similarity conditions. It sequentially detects the consistency of difference intensity, texture pattern continuity and spatial distance relationship between the pixel to be detected and the current growth region. Pixels that meet the comprehensive similarity conditions are incorporated into the growth region to generate the set of growth regions in the expansion process.

[0112] The region integrity determination submodule is used to process the set of growing regions in expansion through boundary activity detection, identify active growing boundaries that still have qualified neighboring pixels, use this boundary information to update the growth queue and trigger a new round of expansion until all region boundaries no longer have expandable pixels, and finally generate a complete set of initial abnormal regions.

[0113] In the above embodiments, this embodiment establishes an initial growth queue for candidate seed points and marks adjacent pixels to be detected to form a growth structure; then, under the constraints of multiple conditions, such as difference intensity, texture continuity, and spatial distance, pixels that meet the similarity are gradually incorporated to expand the growth region; finally, active boundaries are detected and updated, and the expansion is cyclically continued until there are no expandable pixels, resulting in a complete and coherent set of abnormal regions, thus achieving accurate and robust growth of the difference region.

[0114] Example 11: Based on Example 10, the similarity neighborhood expansion processing submodule provided in this embodiment of the invention includes:

[0115] The difference intensity consistency measurement unit is used to process the initial growth structure through local intensity consistency measurement to obtain the statistical distribution matching degree of each pixel to be detected and the current growth region in terms of difference intensity. The initial growth structure is processed through sliding neighborhood statistics to calculate the distribution features of the difference intensity values ​​around each pixel to be detected, generating a statistical feature set containing the local intensity mean and variance. The statistical feature set is processed through relative entropy calculation to measure the similarity between the local intensity distribution of each pixel to be detected and the overall intensity distribution of the current growth region, generating a pixel-level distribution matching metric result. The pixel-level distribution matching metric result is processed through adaptive threshold segmentation to retain pixels with a matching degree higher than the dynamically calculated threshold, generating a candidate pixel set that satisfies intensity consistency. The candidate pixel set that satisfies intensity consistency filters out pixels with large deviations in difference intensity.

[0116] Local intensity consistency measures whether a pixel belongs to the same visual feature region by measuring the statistical matching degree between the pixel to be detected and its growing region in terms of brightness or color intensity distribution. First, the sliding neighborhood statistical method is used to analyze the intensity values ​​around each pixel to be detected, generating a set of statistical features containing local mean and variance. Then, the local distribution is compared with the overall intensity distribution of the growing region using measures such as relative entropy, quantifying the degree of fit between the pixel and the region in terms of intensity. The aim is to select pixels that have coherent brightness changes, laying the foundation for texture and spatial analysis, and ensuring that the region growing result is both consistent with local uniformity and can adapt to natural transitions in the image.

[0117] The texture pattern continuity analysis unit processes the candidate pixel set that satisfies intensity consistency through a Gabor filter bank to extract texture response features at different directions and scales, generating pixel-level multi-scale texture feature vectors. The pixel-level multi-scale texture feature vectors are then processed by directional gradient consistency analysis to obtain the cosine similarity between each candidate pixel and the boundary pixels of the growing region in the texture feature space, generating a texture coherence score set. The texture coherence score set is then processed by region growing constraints to retain pixels with scores higher than the average level of the neighborhood, generating a candidate pixel set that satisfies texture continuity.

[0118] The spatial distance relationship comprehensive determination unit is used to process the candidate pixel set that satisfies texture continuity through proximity weight adjustment, and to perform final screening by combining the minimum spatial distance between the pixel and the growth region boundary and the density distribution relationship, so as to generate a pixel set that meets the similarity conditions; the pixel set that meets the similarity conditions is output as the pixel that can be incorporated into the growth region.

[0119] In the above embodiments, this embodiment achieves accurate screening of the consistency of intensity differences by using local intensity distribution statistics and relative entropy measurement; then, it uses multi-scale Gabor filtering to extract texture features and evaluates texture continuity by using directional gradient cosine similarity to retain pixels with coherent textures; finally, it combines the minimum distance from the pixel to the boundary of the growth region with density weights for comprehensive judgment to obtain a set of pixels that simultaneously meet the constraints of intensity, texture and spatial distance, thereby achieving high-precision and robust expansion of abnormal regions.

[0120] Example 12: Based on Example 11, the spatial distance relationship comprehensive determination unit provided in this embodiment of the invention includes:

[0121] The geometric spatial distance mapping sub-unit is used to measure the spatial straight-line distance between the candidate pixel set that satisfies texture continuity and the pixel coordinate set of the growth region boundary, and to generate a spatial proximity dataset containing the minimum distance value.

[0122] The annular region density matching subunit is used to statistically analyze the pixel distribution density of the annular region guided by the spatial proximity dataset. It measures the similarity between the annular region density centered on the candidate pixel and the corresponding annular region density inside the growth region, forming a density matching degree dataset.

[0123] The spatial relationship comprehensive filtering subunit is used to scale and integrate the matching degree values ​​in the density matching degree dataset and the distance values ​​in the spatial proximity dataset using preset weight coefficients to generate an initial spatial relationship score for each candidate pixel. After normalization adjustment, the initial scores are aggregated to form a spatial relationship score set with unified dimensions. The spatial relationship score set is filtered through a preset threshold to output the final pixel set that meets the spatial relationship conditions.

[0124] Among them, the minimum spatial distance refers to the shortest geometric straight-line distance between the candidate pixel and the boundary pixel of the growth region. It intuitively reflects the proximity of the pixels in spatial position. The smaller the distance, the more likely the pixel is to belong to the natural extension of the current growth region.

[0125] Meeting the spatial relationship condition involves a comprehensive judgment based on the minimum distance and the density matching degree of the annular region. Specifically, the similarity of pixel distribution density between the annular region centered on the candidate pixel and the corresponding annular region inside the growth region is obtained. Then, the distance value and density matching degree are integrated into a unified spatial relationship score through preset weights. Only pixels with scores exceeding a set threshold are judged to meet the spatial relationship condition. This pixel is not only close to the growth region in location, but its surrounding structural density is also consistent with the interior of the region, ensuring that the growth of the region is both natural and stable in spatial distribution.

[0126] In the above embodiments, this embodiment obtains the closest distance from the candidate pixel to the growth boundary by measuring the straight-line distance; the density of the ring-shaped pixels centered on this distance is statistically analyzed and compared with the corresponding density of the growth region to obtain the density matching degree; the distance and density matching degree are merged and normalized according to weight to generate a unified spatial relationship score; after threshold screening, pixels that meet the spatial relationship constraints are retained to achieve dual constraints on the geometric position and density consistency of the abnormal region expansion.

[0127] Example 13: As Figure 6 As shown, based on Examples 1-12, the WebUI adaptive testing method based on multi-agent cooperation and multimodal perception provided in this embodiment of the invention includes the following steps:

[0128] Step S100: Input the dynamic Web interface, and through multimodal perceptual stream processing, integrate visual feature extraction, text semantic parsing and interface structure analysis to generate a multi-dimensional interface feature set; adjust the element positioning strategy of the multi-dimensional interface feature set through a continuous interactive feedback mechanism, and output a set of element descriptors with semantic consistency; the set of element descriptors contains stable semantic identifiers of dynamic interface elements.

[0129] Step S200: Deploy a multi-agent collaborative decision-making unit including an interface understanding agent, a test logic agent, and a data agent; the interface understanding agent parses the semantic relationships in the element descriptor set and generates an interface structure graph; the test logic agent derives the test process skeleton based on the interface structure graph and injects business logic constraints; the data agent introduces test data variables in conjunction with the test process skeleton; the output is a semantic test script formed through a collaborative fusion mechanism; the semantic test script describes the test scenario and operation sequence with high-level semantics.

[0130] Step S300: Take the actual interface state generated by the execution of the semantic test script as input and compare it with the expected state description embedded in the semantic test script; including: performing pixel-level difference calculation, identifying visual inconsistency areas through multi-scale image comparison; performing business-level logic difference, parsing the consistency between the interface element state and business rules, and outputting a layered verification report; the layered verification report integrates pixel-level and business-level difference details to complete the test closed-loop verification.

[0131] For details on the principles described in the above embodiments, please refer to the appendix. Figure 7 This embodiment achieves unified abstraction of visual, textual, and structural information of a page through multimodal perception, generating stable semantic element identifiers. Multiple agents collaboratively parse these identifiers, construct a structured graph, and automatically generate semantic test scripts for variable data under business constraints. After executing the scripts, pixel-level and business-level multi-scale differential comparisons are performed to generate a unified hierarchical verification report. This embodiment forms a full-link adaptive testing process from perception and script generation to closed-loop verification, ensuring that testing maintains high coverage, low maintenance costs, and accurate defect localization capabilities even when the page changes.

[0132] Figure 8 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present invention is shown.

[0133] Electronic devices may include a central processing unit / microprocessor / main control chip; and a storage medium coupled to the central processing unit / microprocessor / main control chip, wherein computer-executable instructions are stored for performing the steps of various methods of embodiments of the present invention when executed by a processor.

[0134] The central processing unit / microprocessor / main control chip may include, but is not limited to, one or more processors or microprocessors.

[0135] Storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, and computer storage media (such as hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).

[0136] In addition, the electronic device may include (but is not limited to) a data bus, an input / output bus / external bus / device bus, a display, and input / output devices (e.g., keyboard, mouse, speaker, etc.).

[0137] The central processing unit / microprocessor / main control chip can communicate with external devices via wired or wireless networks (not shown) through input / output buses / external buses / device buses.

[0138] The storage medium may also store at least one computer-executable instruction for performing the steps of various functions and / or methods in the embodiments described herein when the central processing unit / microprocessor / main control chip is running.

[0139] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0140] Figure 9 A schematic diagram of a computer-readable storage medium according to an embodiment of the present invention is shown.

[0141] like Figure 9 As shown, instructions, such as computer-readable instructions, are stored on a non-transitory computer-readable storage medium. When the computer-readable instructions are executed by a processor, the various methods described above can be performed. The non-transitory computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-transitory non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, the non-transitory computer-readable storage medium can be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the non-transitory computer-readable storage medium, the various methods described above can be performed.

[0142] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0145] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods of the various embodiments of this invention through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0146] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A WebUI self-adaptive testing system based on multi-agent cooperation and multi-modal perception, characterized in that, Comprise: An agent collaborative testing subsystem for deploying a multi-agent collaborative decision unit comprising an interface understanding agent, a test logic agent, and a data agent; and outputting a semanticized test script formed through a collaborative fusion mechanism; The semanticized test script describes a test scene and an operation sequence in a high-level semantic; An assertion and verification subsystem for comparing an actual interface state generated by execution of the semanticized test script with an expected state description embedded in the semanticized test script; comprising: performing pixel-level difference calculation to identify visual inconsistency regions through multi-scale image comparison; performing business-level logic difference to analyze the consistency of interface element states and business rules, and outputting a hierarchical verification report; and integrating pixel-level and business-level difference details in the hierarchical verification report to complete test closed-loop verification; The assertion and verification subsystem comprises: A multi-scale visual difference detection component for processing the actual interface state generated by execution of the semanticized test script and the expected state description embedded in the script through a multi-resolution analysis procedure, extracting visual features at different scales and performing alignment comparison, and generating a visual inconsistency region map; the visual inconsistency region map quantitatively records the spatial distribution and intensity of pixel-level differences; A business logic state verification component for analyzing actual interface element states and detecting consistency with business rules in the semanticized test script, and generating a business logic violation record; the business logic violation record describes the deviation between element states and business rules in detail; A hierarchical verification report generation component for inputting the business logic violation record and the visual inconsistency region map into a report synthesizer for processing, classifying and aggregating according to difference types and severity, and outputting a hierarchical verification report integrating pixel-level and business-level difference details; the hierarchical verification report completes test closed-loop verification; The multi-scale visual difference detection component comprises: A visual feature hierarchical decomposition subcomponent for processing the actual interface state and the expected state description through a frequency domain filter group to decompose the input image into components at different frequency bands; low-frequency components retain macro layout features, mid-frequency components capture subject shape features, and high-frequency components retain microscopic detail features, generating a structured multi-band visual feature set; A feature space adaptive alignment subcomponent for processing the multi-band visual feature set through feature point correspondence relationship establishment to extract stable feature points in different frequency band layers and construct matching relationships; using matching feature points to calculate geometric transformation parameters and perform coordinate system unification processing on each frequency band layer to generate a spatial position-aligned multi-band feature set; A difference region synthesis subcomponent for processing the spatial position-aligned multi-band feature set through weighted difference processing to assign weight coefficients according to the importance of different frequency bands in visual perception; the weighted difference data is processed through connected region analysis to merge adjacent difference pixels into complete abnormal regions, and finally generate a visual inconsistency region map describing pixel-level differences.

2. The multi-agent collaboration and multi-modal perception based WebUI adaptive testing system of claim 1, wherein, The agent collaborative testing subsystem comprises: A semantic relationship network construction component is configured to receive a set of element descriptors from the perception subsystem by the interface understanding agent, analyze the hierarchy, interaction and logical association among the elements by a semantic relationship network construction algorithm inside the component, and generate an interface structure graph with rich semantic relationships; A test logic skeleton weaving component is configured to obtain the interface structure graph by the test logic agent, convert the nodes and relationships in the graph into a series of ordered test action nodes by a dynamic logic weaving engine inside the component, inject business logic constraints defined by the business rule library into the skeleton, and output a test flow skeleton carrying the business rules; A test script fusion generation component is configured to receive the test flow skeleton by the data agent, associate and instantiate abstract data variables with specific test action nodes in the skeleton by an internal context data binding mechanism, and input the instantiation result and the test flow skeleton into a collaborative fusion center of the data agent to generate a semantic test script directly driving the execution engine after semantic alignment and logical verification.

3. The multi-agent collaboration and multi-modal perception based WebUI adaptive testing system of claim 1, wherein, The difference region synthesis sub-component includes: A difference saliency focusing processing module is configured to calculate distribution statistics of pixel difference intensity in each local neighborhood by scanning the weighted difference data with a sliding window, generate potential extreme point position information, calculate the difference intensity deviation degree of each potential extreme point and its surrounding area by neighborhood contrast analysis processing, generate a set of extreme points with saliency scores, and exclude transient extreme points caused by noise by spatiotemporal consistency check processing to generate a stable candidate seed point distribution map; A spatial proximity growth processing module is configured to perform neighborhood similarity propagation processing on the candidate seed point distribution map, conditionally diffuse the seed points to the surrounding pixels, include adjacent pixels meeting the similarity condition into the growth range, and generate an initial abnormal region set; A region boundary optimization processing module is configured to perform boundary continuity analysis processing on the initial abnormal region set, detect and bridge the gaps of the region edges, and merge the adjacent regions with spatial overlap to form an abnormal region set with complete contours.

4. The multi-agent collaboration and multi-modal perception based WebUI adaptive testing system of claim 3, wherein the space The proximity growth processing module includes: An initial neighborhood labeling processing submodule is configured to perform neighborhood pixel scanning processing on the candidate seed point distribution map, establish an initial growth queue for each seed point, label the directly adjacent pixels as a to-be-detected state, and generate an initial growth structure; A similarity neighborhood expansion processing submodule is configured to perform a plurality of similarity condition judgment processes on the initial growth structure, sequentially detect the difference intensity consistency, texture pattern continuity and spatial distance relationship between the to-be-detected pixels and the current growth region, include the pixels meeting the comprehensive similarity conditions into the growth region, and generate an extended growth region set. A region integrity judging sub-module is configured to perform a boundary activity detection process on the growing region set in the expansion, identify active growing boundaries still having qualified neighboring pixels, update the growing queue and trigger a new round of expansion by using the boundary information, and generate an initial abnormal region set until all region boundaries no longer have expandable pixels.

5. The multi-agent collaboration and multi-modal perception based WebUI adaptive testing system of claim 4, wherein, The similarity neighborhood expansion processing sub-module comprises: A difference intensity consistency measurement unit is configured to perform a local intensity consistency measurement process on the initial growing structure, obtain a statistical distribution matching degree of each to-be-detected pixel and the current growing region in difference intensity, and perform a sliding neighborhood statistical process on the initial growing structure to calculate distribution features of the difference intensity values around each to-be-detected pixel, thereby generating a statistical feature set comprising a local intensity mean value and a variance; The statistical feature set is subjected to a relative entropy calculation process to perform a similarity measurement on the local intensity distribution of each to-be-detected pixel and the overall intensity distribution of the current growing region, thereby generating a pixel-level distribution matching quantification result; the pixel-level distribution matching quantification result is subjected to an adaptive threshold segmentation process to retain pixels having a matching degree higher than a dynamically calculated threshold, thereby generating a candidate pixel set satisfying intensity consistency; A texture pattern continuity analysis unit is configured to perform a Gabor filter set process on the candidate pixel set satisfying intensity consistency to extract texture response features in different directions and scales, thereby generating a pixel-level multi-scale texture feature vector; the pixel-level multi-scale texture feature vector is subjected to a direction gradient consistency analysis process to obtain a cosine similarity of each candidate pixel and a growing region boundary pixel in a texture feature space, thereby generating a texture continuity score set; the texture continuity score set is subjected to a region growing constraint process to retain pixels having a score higher than a neighborhood average level, thereby generating a candidate pixel set satisfying texture continuity; A spatial distance relationship comprehensive judgment unit is configured to perform a proximity weight adjustment process on the candidate pixel set satisfying texture continuity, combine a minimum spatial distance and a density distribution relationship of the pixel and the growing region boundary, and perform a final screening, thereby generating a pixel set meeting a similarity condition; the pixel set meeting the similarity condition is output as a pixel that can be incorporated into the growing region.

6. The multi-agent collaboration and multi-modal perception based WebUI adaptive testing system of claim 5, wherein, The spatial distance relationship comprehensive judgment unit comprises: A geometric spatial distance mapping sub-unit is configured to perform a spatial straight-line distance measurement on the candidate pixel set satisfying texture continuity and a growing region boundary pixel coordinate set, thereby generating a spatial proximity data set comprising a minimum distance value; A ring region density matching sub-unit is configured to perform a statistical process on a ring region pixel distribution density guided by the spatial proximity data set, perform a similarity measurement on a ring region density centered on the candidate pixel and a corresponding ring region density inside the growing region, and form a density matching degree data set; A spatial relationship comprehensive screening sub-unit is configured to scale and integrate a matching degree value in the density matching degree data set and a distance value in the spatial proximity data set by using a preset weight coefficient, thereby generating an initial spatial relationship score of each candidate pixel; The initial score is subjected to a normalization adjustment and aggregation, thereby forming a spatial relationship score set with a unified dimension; The spatial relationship score set is filtered by a preset threshold, and a final pixel set meeting the spatial relationship condition is output.

7. The multi-agent collaboration and multi-modal perception based WebUI adaptive testing system of claim 1, wherein, The multi-modal perception subsystem is further included, which is configured to input a dynamic Web interface, process the input through a multi-modal perception flow, fuse visual feature extraction, text semantic analysis and interface structure analysis, generate a multi-dimensional interface feature set, adjust element positioning strategies through a continuous interaction feedback mechanism for the multi-dimensional interface feature set, and output an element descriptor set with semantic consistency, wherein the element descriptor set includes stable semantic identifiers of dynamic interface elements.

8. A multi-agent collaboration and multi-modal perception based WebUI adaptive testing method, used for implementing the multi-agent collaboration and multi-modal perception based WebUI adaptive testing system according to any one of claims 1 to 7, characterized in that, The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-modal perception subsystem is further included, which is configured to input a dynamic Web interface, process the input through a multi-modal perception flow, fuse visual feature extraction, text semantic analysis and interface structure analysis, generate a multi-dimensional interface feature set, adjust element positioning strategies through a continuous interaction feedback mechanism for the multi-dimensional interface feature set, and output an element descriptor set with semantic consistency, wherein the element descriptor set includes stable semantic identifiers of dynamic interface elements. The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises the following steps: The multi-agent cooperation and multi-modal perception-based WebUI adaptive testing method comprises

Images