Multimodal based scenario driven automated testing method, apparatus and related device

Abstract

This invention relates to the field of artificial intelligence and discloses a multimodal scenario-driven automated testing method, apparatus, and related equipment. The method includes: receiving a test task objective and generating a corresponding set of test steps; acquiring multimodal data of the current interface of the application under test; generating and executing operation instructions for the next test step based on a scenario-driven strategy, according to the multimodal data of the current interface, the test task objective, and historical operation sequences; collecting multimodal data of the new interface after execution; confirming whether the test task objective has been completed based on the multimodal data of the interface before and after the execution of the operation instructions; if not completed, continuing to generate and execute operation instructions for the next test step based on the scenario-driven strategy until the task is completed. This invention can be applied to business scenarios such as fintech and healthcare. Through scenario-driven strategies and multimodal result verification, it effectively improves the perception capability of automated testing in complex business scenarios and the accuracy and robustness of test results.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology and can be applied to business areas such as fintech and healthcare. In particular, it relates to a multimodal scenario-driven automated testing method, apparatus, and related equipment. Background Technology

[0002] With the deepening of digital transformation in industries such as finance, insurance, and healthcare, mobile applications and web applications have become key carriers for core businesses such as transaction claims, credit services, online diagnosis and treatment, health management, and electronic medical record management. As a result, the requirements for software quality and stability in various industries are increasing. Therefore, software automated testing, as a core link to ensure application quality and avoid business risks, faces severe challenges in terms of efficiency and reliability.

[0003] Currently, existing automated testing tools mainly rely on interface structure information for script recording and playback. When facing critical node testing of complex business processes (such as insurance claims and loan applications in the fintech field, or the hierarchical diagnosis and treatment appointment process in the healthcare field), they have shortcomings such as high interface fragility, lack of understanding of business scenarios, and insensitivity to non-textual visual information. They are unable to ensure the accuracy and reliability of testing in complex business scenarios and cannot meet the needs of high-quality development in various industries. Summary of the Invention

[0004] The main objective of this invention is to provide a multimodal scenario-driven automated testing method, apparatus, and related equipment, aiming to solve the technical problems of insufficient accuracy and reliability of automated testing in complex business scenarios in the prior art, so as to improve the level of software automated testing and its adaptability to industry needs, and meet the high reliability testing needs of industries such as finance, insurance, and healthcare.

[0005] The technical solution of the present invention is as follows: The first aspect of this invention provides a multimodal, scenario-driven automated testing method, comprising: Receive test task objectives described in natural language, and generate a corresponding set of test steps based on the test task objectives; Obtain multimodal data of the current interface of the application to be tested, wherein the multimodal data includes at least screenshots of the interface and interface structure data; Based on the scenario-driven strategy, the operation instructions for the next test step are generated according to the multimodal data of the current interface, the test task objective, and the historical operation sequence. The operation command is executed on the application under test, and multimodal data of the new interface after execution is collected; Based on the comparison of multimodal data of the interface before and after the execution of the operation command, the test task objective is confirmed to be completed according to the comparison results. If the test task objective is not completed, a new interface will be used as the current interface, and the operation instructions for generating the next test step based on the scenario-driven strategy will be executed until all test steps for the test task objective are completed.

[0006] A second aspect of the present invention provides a multimodal-based scene-driven automated testing device, comprising: The receiving module is used to receive the test task objectives described in natural language and generate a corresponding set of test steps based on the test task objectives. A multimodal perception module is used to acquire multimodal data of the current interface of the application under test. The multimodal data includes at least an interface screenshot and interface structure data. The scenario-driven decision module is used to generate operation instructions for the next test step based on the scenario-driven strategy, the multimodal data of the current interface, the test task objective, and the historical operation sequence. The execution acquisition module is used to execute the operation instructions on the application under test and acquire multimodal data of the new interface after execution. The multimodal verification module is used to compare the multimodal data of the interface before and after the execution of the operation command, and to confirm whether the test task objective has been completed and whether an exception has been triggered based on the comparison results. The iterative testing module is used to, if the test task objective is not completed, use a new interface as the current interface, return to continue generating operation instructions for the next test step based on the scenario-driven strategy and execute them, until all test steps for the test task objective are completed.

[0007] A third aspect of the present invention provides a computer device including at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the above-described multimodal scenario-driven automated testing method.

[0008] A fourth aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described multimodal-based scenario-driven automated testing method.

[0009] Beneficial Effects: This invention discloses a multimodal scene-driven automated testing method, apparatus, and related equipment. Compared with existing technologies, this invention receives a test task objective described in natural language and generates a corresponding set of test steps based on the test task objective; acquires multimodal data of the current interface of the application under test, the multimodal data including at least screenshots and interface structure data; based on a scene-driven strategy, generates operation instructions for the next test step according to the multimodal data of the current interface, the test task objective, and historical operation sequences; executes the operation instructions on the application under test and collects multimodal data of the new interface after execution; compares the multimodal data of the interface before and after the execution of the operation instructions, and confirms whether the test task objective has been completed based on the comparison result; if the test task objective has not been completed, the new interface is used as the current interface, and the process returns to continue generating and executing operation instructions for the next test step based on the scene-driven strategy until all test steps of the test task objective are completed. This invention can be applied to business scenarios such as fintech and healthcare. By combining task intent and historical operation sequences with a scenario-driven strategy, it generates operation instructions for each test step and performs multimodal task completion verification before and after the operation, effectively improving the ability to perceive complex business scenarios and the accuracy and robustness of test result judgment. Attached Figure Description

[0010] To more clearly illustrate the solutions in this invention, the accompanying drawings used in the description of the embodiments of this invention will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0011] Figure 1 A schematic diagram of an application environment for the multimodal-based scenario-driven automated testing method provided in an embodiment of the present invention; Figure 2 A flowchart of a scene-driven automated testing method based on multimodality provided in an embodiment of the present invention; Figure 3 This is a system architecture diagram of an application embodiment of the multimodal-based scenario-driven automated testing method provided by the present invention; Figure 4 A flowchart illustrating multimodal verification comparison in one application embodiment of the present invention; Figure 5 A schematic diagram of the functional modules of the scene-driven automated testing device based on multimodality provided in an embodiment of the present invention; Figure 6 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments of the invention are described below in conjunction with the accompanying drawings.

[0013] The multimodal-based scene-driven automated testing method provided in this invention can be applied to, for example... Figure 1 In the application environment, it includes a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0014] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as financial clients, healthcare clients, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software, etc. (for example only).

[0015] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0016] Server 105 can be a server providing various services, such as a backend server supporting the content browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices. Server 105 can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the shortcomings of traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"), such as high management difficulty and weak business scalability. Server 105 can also be a server for a distributed system or a server combined with blockchain.

[0017] It should be noted that the multimodal scene-driven automated testing method provided in this application embodiment can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the multimodal scene-driven automated testing device provided in this embodiment can also be located in the first terminal device 101, the second terminal device 102, or the third terminal device 103. Alternatively, the multimodal scene-driven automated testing method provided in this embodiment can generally be executed by the server 105. Correspondingly, the multimodal scene-driven automated testing device provided in this embodiment can generally be located in the server 105.

[0018] It should be understood that the number of terminal devices, networks, and servers listed above is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be used.

[0019] like Figure 2 As shown, the scenario-driven automated testing method based on multimodality provided in this embodiment of the invention specifically includes the following steps: S201. Receive the test task objective described in natural language, and generate a corresponding set of test steps based on the test task objective.

[0020] In this embodiment, the system receives test task objectives input by the user in natural language. These objectives can directly align with actual business scenario requirements, such as "completing online car insurance claims by uploading photos and submitting cases" in the financial field, or "completing remote consultation by filling in patient information and applying for appointments" in the healthcare field. Natural language processing (NLP) technology is used to analyze the intent, break down the scenario, and extract the steps from the test task objectives. This transforms the abstract natural language objectives into a set of executable test steps ordered by business logic. Specifically, a pre-trained Named Entity Recognition (NER) model can be used to identify the core business entities, action verbs, and constraints in the task, thereby breaking it down into a set of test steps that can complete the test task objectives.

[0021] For example, in the financial sector's auto insurance claims scenario, after intent parsing the test task objective "completing online auto insurance claim photo upload and case submission," a set of test steps can be generated, including "entering the claims portal → selecting auto insurance claims → uploading photos of the damaged vehicle → filling in claims information → submitting the claims case." Similarly, in the healthcare sector's remote consultation scenario, based on the test task objective "completing remote consultation patient information filling and registration application," a set of test steps can be automatically generated, including "entering the consultation page → selecting the department and doctor → filling in basic patient information → filling in symptom description → submitting the registration application," thus achieving the conversion of natural language into standardized test steps.

[0022] This embodiment lowers the barrier to entry for automated testing by receiving test task objectives described in natural language. Testers do not need professional coding skills; business personnel can directly issue test tasks via natural language. Furthermore, by parsing the intent and breaking down the natural language task objectives into steps, a corresponding set of test steps is generated, providing a clear execution framework for subsequent iterative testing. This clarifies the overall test path and the objectives at each stage, making subsequent operation instructions more targeted.

[0023] S202. Obtain multimodal data of the current interface of the application to be tested. The multimodal data includes at least screenshots of the interface and interface structure data.

[0024] In this embodiment, when iteratively executing all test steps in the test step set, multimodal data is first collected from the current operation interface of the application to be tested. This multimodal data includes at least interface screenshots and interface structure data. The interface screenshots are the visual image data of the current interface, which can be obtained through the test framework interface, system screenshot API, etc. The interface structure data are the structured data of the current interface, such as the document object model (DOM) and XML view hierarchy tree, which include information such as the identifier, coordinates, attributes, and hierarchical relationship of the interface controls.

[0025] For example, in the financial sector, when collecting data for auto insurance claims, if the current interface is a photo upload page for auto insurance claims, then a visual screenshot of the page and DOM structure data containing controls such as "upload button," "photo preview box," and "next button" are collected simultaneously. In the healthcare sector, when collecting data for remote consultations, if the current interface is a remote consultation information entry page, then a screenshot of the page and structure data containing controls such as "name input box," "symptom input box," and "submit button" are collected simultaneously, providing a complete multimodal information foundation for subsequent feature extraction and decision generation.

[0026] This embodiment acquires visual and structural bimodal information through multimodal perception, breaking through the limitations of traditional automated testing tools that rely solely on interface structure data (DOM / XPath). It provides a complete information foundation for subsequent scenario-driven decisions, enabling the accurate capture of non-textual visual information such as icons, colors, animations, and the visual state of controls. This solves the problem of traditional tools being insensitive to visual feedback and is more suitable for scenarios such as finance and healthcare where business status is reflected by visual elements.

[0027] S203. Based on the scenario-driven strategy, generate the operation instructions for the next test step according to the multimodal data of the current interface, the test task objective, and the historical operation sequence.

[0028] In this embodiment, the scenario-driven strategy is a pre-built decision generation strategy based on scenario multimodal features and a test experience knowledge base. Specifically, the scenario-driven strategy integrates industry business rules, the test experience knowledge base, and multimodal feature reasoning logic. It fuses and analyzes the multimodal data of the current interface, the overall test task objective, and historical operation sequences to generate specific operation instructions for the next test step. These instructions include target control positioning information and action types, including but not limited to click, input, swipe, long press, return, and submit. Generating executable operation instructions based on the scenario-driven strategy ensures that the generated instructions align with industry business rules and historical test experience, avoiding meaningless test operations and improving the intelligence and rationality of test decisions.

[0029] For example, in the car insurance claims scenario in the financial field, if the historical operation sequence is "enter the claims portal → select car insurance claims", the current interface is the car insurance claims photo upload page, and the multimodal data shows that there is an interactive "upload button", then the operation instruction "click the upload button with coordinates (X,Y)" will be automatically generated after multimodal fusion and inference based on the scenario-driven strategy.

[0030] For example, in the remote consultation scenario in the medical and health field, if the historical operation sequence is "enter the consultation page → select the department and doctor", the current interface is the patient information filling page, and the multimodal data shows that there is a "name input box", then the operation instruction "enter the test name 'Zhang San' in the name input box" will be automatically generated to realize intelligent decision-making based on the scenario context.

[0031] This embodiment makes decisions by combining three-dimensional information: multimodal data, test task objectives, and historical operation sequences. This enables the generation of operation instructions to have context-aware capabilities, solving the problem that traditional testing tools lack the ability to understand business scenarios. It can adapt to complex multi-step business processes such as insurance claims and remote consultations.

[0032] S204. Execute the operation command on the application to be tested, and collect multimodal data of the new interface after execution.

[0033] In this embodiment, the generated operation instructions are executed on the current interface of the application under test by calling the execution interface of the automated testing framework, while recording the timestamp, execution status, and other information of the operation. After the operation instructions are executed, multimodal data is collected from the new interface. The collection method is consistent with the collection method of the current interface, that is, the visual screenshots and structural data of the new interface are acquired synchronously to ensure that the collected multimodal data can fully reflect the interface state after the operation is executed.

[0034] For example, in the car insurance claims scenario in the financial field, after the "click the upload button" operation is performed, a screenshot of the new interface after the photo upload pop-up window appears and the corresponding DOM structure data are collected; as another example, in the remote consultation scenario in the medical and health field, after the "enter name" operation is performed, a screenshot of the page after the name is entered is collected and the structure data are collected to provide comparative data for subsequent result verification.

[0035] This embodiment automates test actions by invoking an automated testing framework to execute operation commands, thereby improving test execution efficiency. Furthermore, it synchronously collects multimodal data of the new interface after the operation, fully capturing the interface state changes brought about by the operation and providing realistic and effective comparative data for subsequent task completion assessment.

[0036] Preferably, after executing the operation command on the application under test and before collecting multimodal data of the new interface after execution, the waiting time can be dynamically adjusted through an intelligent waiting strategy. Specifically, a response time prediction model is pre-built in an experience knowledge base. This model takes "operation type" (such as click, submit, upload), "business scenario" (such as large-amount transfer, image upload), and "current network status" as input features, and uses "time taken from operation execution to interface stabilization" recorded in historical tests as labels. It is trained using linear regression or random forest algorithms. For example, the model will learn that "the average response time for the 'submit claim' operation in the financial field is 2.3 seconds with a standard deviation of 0.5 seconds under a 4G network," allowing the trained response time prediction model to dynamically predict the response time based on the actual execution situation. Therefore, after executing the current operation command, the current operation type, the business scenario, and the real-time monitored network latency are input into the response time prediction model to obtain the predicted response time of the operation. After the predicted response time is reached, the subsequent multimodal data collection steps are entered. By training and predicting the operation response time based on historical data, compared with the traditional fixed waiting time, it can effectively avoid the problems of low test efficiency due to excessively long waiting time and test failure due to insufficient interface loading due to excessively short waiting time, thereby helping to improve the overall test execution efficiency.

[0037] S205. Compare the multimodal data of the interface before and after the execution of the operation instruction, and confirm whether the test task objective has been completed based on the comparison results.

[0038] In this embodiment, a comprehensive comparative analysis is performed based on the multimodal data of the interface collected before and after the execution of the operation command. Specifically, it extracts the change features at both the visual and structural levels, and combines this with the completion criteria of the test task to determine whether the current operation step is completed and whether the overall test task objective is achieved. Through the verification and comparison of multimodal data, the comparative analysis not only focuses on changes in text information but also covers changes in non-textual visual elements (icons, colors, animations, control states) and structural nodes (control additions / removals, attribute changes, hierarchy adjustments). This achieves comprehensive verification of the completion status of business steps. Furthermore, it uses actual interface state changes as the basis for judging task completion, rather than relying on preset script logic, making the judgment results more consistent with the actual business completion standards and improving the reliability of the test results.

[0039] For example, in the financial sector, the completion criteria for the photo upload step in car insurance claims are "the appearance of a photo thumbnail + a green checkmark icon + the upload control status changing to 'completed'". By comparing multimodal data before and after the operation, it is verified whether all of the above features have appeared to confirm whether the step has been completed. In the healthcare sector, the completion criteria for the patient information entry step are "all required input fields display entered content + the 'Next' button becomes clickable". By comparing multimodal data, it is verified whether this criterion is met, thereby judging the progress of the overall task objective.

[0040] This embodiment achieves accurate judgment of changes in business status by directly comparing multimodal data before and after the operation. It can not only identify changes in text information, but also accurately capture changes in visual elements and structural nodes. The multi-dimensional comparative analysis effectively avoids the problem of inaccurate judgment due to insensitivity to visual information, thereby achieving comprehensive verification of the completion status of business steps and avoiding misjudgments caused by the lack of information in a single dimension, thus improving the accuracy and reliability of test results.

[0041] S206. If the test task objective is not completed, the new interface is used as the current interface, and the operation instructions for generating the next test step based on the scenario-driven strategy are returned and executed until all test steps of the test task objective are completed.

[0042] In this embodiment, if multimodal data comparison confirms that the current test task objective has not been completed (i.e., there are still unexecuted test steps or the current step has not met the completion standard), the new interface after executing the operation instruction will be used as the new current interface. The process will return to the step of generating the operation instruction for the next test step based on the scenario-driven strategy, and the iterative process of multimodal data acquisition, operation instruction generation, execution, and result verification will continue. If it is confirmed that the test task objective has been completed (i.e., all test steps have been executed according to business logic and each step has met the completion standard), the test process will terminate.

[0043] For example, in the financial sector, if only the "upload photos of the damaged vehicle" step is completed, but the "fill in the claim information" and "submit the claim case" steps are not completed, the subsequent operation instructions will continue to be generated iteratively; in the healthcare sector, if all information has been filled in and the registration application has been submitted, and the interface displays "registration successful", the test will be terminated.

[0044] This embodiment employs an iterative testing process, adaptable to complex business scenarios involving multiple pages and steps. It can comprehensively cover the entire process of testing, such as insurance claims, loan applications, and remote registration. Furthermore, the iterative approach, using a new interface as the current screen, allows the testing process to progress in real-time following interface state changes, enhancing robustness to interface modifications. The process only terminates after all test steps are completed, ensuring the completeness of the testing task and comprehensively covering all aspects of the test objective, thus improving test coverage.

[0045] Preferably, after confirming that all test task objectives have been completed, the method further includes script generation and experience accumulation steps. This involves automatically retrieving the full execution log of the current test, filtering out invalid operations such as abnormal retries and rollbacks, and extracting the successful operation sequence from task start to successful completion. Based on a preset script template, the successful operation sequence is converted into a directly executable and reusable automated test script for subsequent direct invocation, thereby improving the reusability of test resources.

[0046] In addition, the entire process of this test can be reviewed. By comparing the successful and failed operation trajectories, rules containing "condition-action-effect" can be generated. For example, the review found that "when the interface displays a 'network timeout' prompt, executing the 'click retry' action can restore data loading." These rules are extracted and stored in the experience knowledge base to provide reliable rule guidance for subsequent automated testing, thereby continuously improving the scenario adaptability and decision accuracy of automated testing.

[0047] In the above embodiments, the present invention discloses a multimodal scenario-driven automated testing method. This method receives a test task objective described in natural language and generates a corresponding set of test steps based on the test task objective. It acquires multimodal data of the current interface of the application under test, including at least screenshots and interface structure data. Based on a scenario-driven strategy, it generates an operation instruction for the next test step according to the multimodal data of the current interface, the test task objective, and historical operation sequences. It executes the operation instruction on the application under test and collects multimodal data of the new interface after execution. It compares the multimodal data of the interface before and after the execution of the operation instruction and confirms whether the test task objective has been completed based on the comparison result. If the test task objective has not been completed, the new interface is used as the current interface, and the method returns to continue generating and executing the operation instruction for the next test step based on the scenario-driven strategy until all test steps of the test task objective are completed. This invention can be applied to business scenarios such as fintech and healthcare. By combining task intent and historical operation sequences with a scenario-driven strategy, it generates operation instructions for each test step and performs multimodal task completion verification before and after the operation, effectively improving the ability to perceive complex business scenarios and the accuracy and robustness of test result judgment.

[0048] In one embodiment, step S203 includes: S231. Visually encode the screenshot of the current interface and extract the visual feature vector. S232. Perform structural encoding on the interface structure data of the current interface and extract the structural feature vector; S233. The visual feature vector and the structural feature vector are fused through a cross-attention mechanism to obtain the multimodal feature representation of the current interface; S234. Based on the scenario-driven strategy, generate the operation instructions for the next test step according to the multimodal feature representation, test task objective and historical operation sequence.

[0049] In this embodiment, the multimodal data of the current interface is first visually encoded and structurally encoded. Specifically, the Visual Transformer (ViT) model is used to visually encode the screenshot of the current interface, dividing the screenshot into multiple image blocks. Through the model's multi-layer attention mechanism and feedforward neural network, local and global features of the image blocks are extracted and fused to generate a fixed-dimensional visual feature vector. This visual feature vector contains all key visual information in the screenshot, such as icons, colors, visual shapes of controls, animation states, and element layouts. For example, it can accurately represent visual features such as "whether the green checkmark icon appears," "whether the button is clickable and highlighted," and "whether the loading animation exists."

[0050] Furthermore, a Graph Neural Network (GNN) is used to encode the interface structure data of the current interface, transforming it into a graph structure with controls as nodes and hierarchical / associative relationships as edges. A graph convolutional layer is then used to aggregate and learn the features of each node and its neighboring nodes, extracting a structural feature vector. This structural feature vector contains all key information at the structural level, including the identifier, coordinates, interactivity, text value, hierarchical position, and node additions / removals of interface controls. For example, it can accurately represent structural features such as "the upload button has the ID XXX, coordinates (X,Y), and the attribute is interactive" and "a photo preview node has been added to the page."

[0051] Subsequently, a cross-attention fusion layer based on the Transformer architecture performs cross-attention calculations on visual and structural feature vectors as queries, keys, and values, respectively. This allows visual and structural features to perceive and fuse with each other, overcoming the limitations of single-modal information and ultimately generating a unified multimodal feature representation containing all key visual and structural information. This multimodal feature representation preserves both the visual details of the interface and its structured logic, achieving deep fusion of multimodal information. Based on a pre-built scenario-driven strategy, and combined with the fused multimodal feature representation, the overall test task objective, and historical operation sequences, decision generation is performed. This allows for the confirmation of the current interface's scenario state and the action requirements for the next test step, incorporating industry business rules and an experience knowledge base. Finally, it generates the next test step operation instruction, which includes the target control's location information and the specific action type.

[0052] This embodiment accurately encodes both visual and structural data, extracting feature vectors that fully represent the multimodal key information of the interface. A cross-attention mechanism achieves deep fusion of visual and structural features, generating a multimodal feature representation that avoids the limitations of single-modal information, providing a more comprehensive and accurate information foundation for generating operation instructions. Based on the fused multimodal feature representation, operation instructions are generated by combining task objectives with historical operation sequences, significantly improving the accuracy and rationality of instruction generation. This makes the generated instructions more aligned with the logical requirements of actual business scenarios, effectively solving the decision-making errors caused by limited information in traditional testing methods.

[0053] Preferably, this embodiment also provides a caching mechanism to reduce redundant calculations by caching the feature vectors of frequently used interface elements. Specifically, the cached objects are the interface scene feature vectors and the positioning information of frequently used controls. A unique scene hash value is generated in advance for each interface and cached. This scene hash value is calculated by combining the application package name, the current page route, and the core control ID. For example, the "login page" of mobile banking will generate a fixed hash value as the cache key.

[0054] After acquiring the multimodal data of the current interface of the application under test, its scene hash value is first calculated and queried in the local cache (such as Redis or memory cache). If the cache is hit, the location information and encoded feature vectors of commonly used controls in that scene (such as "login button" and "input box") are directly read from the cache, thus skipping the repeated ViT visual encoding and GNN structure encoding process and directly entering the feature fusion step, improving testing efficiency. If the cache is not hit, the complete encoding and location calculation process is executed. After obtaining the results, the hash value, feature vector, and location information of that scene are written to the cache, and an appropriate expiration time is set, such as 24 hours for high-frequency pages and 1 hour for low-frequency pages.

[0055] Using an LRU (Least Recently Used) eviction policy, the least recently used cache item is automatically removed when the cache capacity reaches a threshold. Simultaneously, when a version update of the application under test is detected (such as an app upgrade), all related caches are proactively cleared, and features are forced to be recalculated to ensure the validity of the cached data.

[0056] This embodiment effectively avoids a large number of repetitive coding operations by caching the feature vectors and positioning information of high-frequency interfaces, thereby reducing the amount of data processing and memory usage, and improving the real-time performance of test decisions. Furthermore, the eviction policy and the proactive clearing policy for application version updates also ensure the timeliness of cached data, achieving test performance optimization without sacrificing accuracy.

[0057] In one embodiment, step S233 includes: S2331. Confirm the scene state of the current interface based on the multimodal feature representation; S2332. Combine the test task objective with the historical operation sequence to perform scene action matching on the scene state and confirm the standard test action of the scene state. S2333. Based on the execution information of the standard test action and the positioning information of the current interface, generate the operation instructions for the next test step.

[0058] In this embodiment, when generating the operation instructions for the next test step, the current scene state of the interface is first confirmed based on the multimodal feature representation. Specifically, the business scene nodes of the current interface are identified, that is, feature similarity matching is performed between the multimodal feature representation and the features of each business scene node to clarify the specific position of the current position in the overall business process. For example, in the financial auto insurance claims scenario, there are "initial photo upload node", "claims information filling node", and "case submission confirmation node", and in the medical remote consultation scenario, there are "department selection node", "patient information filling node", and "registration submission node", etc. Furthermore, it is confirmed whether the current scene meets the preconditions for executing the target operation. For example, the "photo upload node" requires confirmation that "the upload button is interactive" and "there is no loading animation obscuring it", and the "information filling node" requires confirmation that "the required input box is displayed" and "the page is loaded". Only when all preconditions are met will the subsequent action matching stage be entered. If they are not met, the preconditions need to be completed.

[0059] After confirming that the current scenario node and operational conditions are met, the overall test task objective is used as a guide. The execution progress of the current test is determined by combining historical operation sequences. Simultaneously, a standard action library bound to the current scenario node is retrieved from the experience knowledge base. This standard action library contains pre-set test actions for each scenario node that conform to business logic and is continuously optimized and updated based on historical testing experience. By matching the scenario node, test task objective, and historical operation sequences, standard test actions suitable for the current test progress are selected from the standard action library. These actions are abstract action types that fit the business scenario, such as "clicking the photo upload button" or "submitting a claim" in a financial scenario, or "selecting the endocrinology department" or "submitting a registration application" in a medical scenario. This ensures that the matched standard actions are consistent with the overall test task objective and linked to historical operation sequences.

[0060] Next, the confirmed standard test actions are visualized. First, based on the execution information of the standard test actions, the specific action type (click, input, swipe, etc.) is determined. Then, the precise positioning information of the target control is extracted from the multimodal feature representation of the current interface, including the control's coordinates, label, and relative hierarchical position, prioritizing robust positioning methods (content-desc / aria-label / visible text + relative hierarchical positioning). Finally, the action type and the target control positioning information are fused to generate the next test step operation instruction that can be directly executed by the automated testing framework. The instruction format meets the execution requirements of the automated testing framework.

[0061] This embodiment confirms standard test actions by combining task objectives and historical operation sequences under scenario-driven conditions, ensuring consistency between test actions and business logic. It also automatically transforms abstract standard test actions into specific operation instructions containing location information and action types, achieving seamless connection from scenario decision-making to execution instructions and improving the standardization and executability of operation instruction generation.

[0062] In one embodiment, step S204 includes: S241. Execute the operation instruction on the application to be tested, and confirm the operation area of ​​the operation instruction; S242. After the operation instruction is executed, collect the multimodal data of the operation area after execution; S243. Overwrite the multimodal data of the interface before execution with the multimodal data of the operation area after execution to obtain the multimodal data of the new interface after execution.

[0063] In this embodiment, when executing an operation command and collecting multimodal data of the new interface after execution, the execution interface of the automated testing framework is first called to execute the generated operation command on the current interface of the application under test, ensuring that the operation action is applied to the target control. Simultaneously, based on the positioning information of the target control in the operation command, the operation area for this operation is determined. The operation area is the local interface area where the target control is located. For example, when clicking the "upload button," the operation area is the upload button and its surrounding local area; when entering content in an input box, the operation area is the local area where the input box is located. The range of the operation area can be adaptively adjusted according to the size of the control and the operation type.

[0064] After the operation command is executed, instead of collecting full multimodal data for the entire new interface, only local multimodal data is collected for the confirmed operation area. Visual screenshots and structural data of the operation area are acquired simultaneously. The collected local data fully reflects the changes to the interface caused by the operation. For example, after clicking the upload button, only a screenshot of the area where the upload button is located and the corresponding local DOM structure data are collected. The collected multimodal data of the operation area is then used to update the corresponding area data in the original interface multimodal data before the operation command was executed. Data in other areas where no operation occurred remains unchanged. Finally, the complete multimodal data of the new interface after execution is obtained, achieving incremental collection and updating of multimodal data.

[0065] This embodiment achieves incremental acquisition of multimodal data by identifying the operation area and collecting multimodal data only from that area. This significantly reduces the amount of data collected and the computational load of subsequent data processing. While ensuring data integrity, it further reduces system resource consumption and improves the efficiency of data acquisition and processing. Simultaneously, the incremental acquisition method can more accurately focus on interface changes caused by operations, facilitating rapid extraction of change features and improving the efficiency and accuracy of multimodal data comparison.

[0066] In one embodiment, step S205 includes: S251. Visually compare the screenshots of the current interface and the new interface to obtain the corresponding visual change features. S252. Compare the node differences of the interface structure data of the current interface and the new interface to obtain the corresponding structural change characteristics. S253. Based on the test task objective, perform a completion analysis on the visual change features and structural change features to obtain the visual completion and structural completion, respectively. S254. The visual completion and structural completion are weighted and fused to obtain the task completion confidence. S255. If the confidence level of the task completion is greater than or equal to the preset completion threshold, then it is confirmed that the test task objective has been completed.

[0067] In this embodiment, during multimodal result verification, visual comparison and structural comparison are performed separately. Specifically, multi-dimensional visual comparison is conducted on the interface screenshots before and after the operation. This can be achieved by using the SIFT / SURF algorithm to extract key point features and calculate the matching degree, using color histogram comparison to detect color changes, using the YOLO object detection algorithm to identify the appearance / disappearance of icons and controls, and using optical flow to detect animation state changes. By comprehensively analyzing the multi-dimensional visual analysis results, the visual change features after the operation are extracted. These visual change features include the addition / disappearance of visual elements, color changes, animation state changes, and changes in the visual shape of controls, such as "the appearance of a green checkmark icon," "the disappearance of the loading animation," and "the button changing from gray to a highlight color."

[0068] The interface structure data before and after the operation is compared by node differences. Specifically, the tree edit distance algorithm can be used to calculate the structural similarity of the DOM / XML tree, compare the changes in the attributes (visibility, interactivity, text value) of key control nodes, detect the addition / deletion of nodes and changes in hierarchical position, and comprehensively extract structural change features. That is, structural change features include the addition or removal of control nodes, attribute adjustments, hierarchical changes, page jumps, etc., such as "adding a photo preview node", "the upload button attribute changing from interactive to completed", "the next button attribute changing from non-clickable to clickable", etc.

[0069] Based on the overall test task objective, a completion analysis of visual and structural change features is performed. Specifically, we can first obtain the predefined visual and structural completion indicators for each test step in the test step set of the test task objective. For example, if the overall task objective is "to complete the entire online car insurance claim process", the test step set can be broken down as "enter the claim portal → select car insurance claim → upload photos of the damaged vehicle → fill in the claim information → submit the claim case". The visual indicator for the "upload photos" step is "the photo thumbnail appears", and the structural indicator is "the submit button becomes clickable". The same applies to other test steps.

[0070] The visual and structural change features extracted after the operation are executed are matched with the visual and structural completion criteria for the current test step to confirm the single-step visual and structural completion rates. For example, if all key visual elements (such as icons and status prompts) appear after matching, the single-step visual completion rate is 1; if only some are met, the single-step visual completion rate between 0 and 1 is calculated based on the matching ratio. Simultaneously, progress weights are pre-assigned to each step based on its importance and the number of remaining steps in the overall task. For instance, if the overall task has four steps and the current step is step 1, the progress weight is 0.25; if the current step is the last step, the progress weight is 1.

[0071] Based on the obtained single-step visual completion and single-step structural completion, the progress of the current test step in all steps is weighted and integrated to obtain visual completion and structural completion that reflect the overall task completion progress. Specifically: Visual completion = Single-step visual completion × Current step progress weight + Sum of visual progress of historically completed steps; Structural completion = Single-step structural completion × Current step progress weight + Sum of structural progress of historically completed steps. For example, in the "car insurance claim" task, if only step 1 "upload photos" is completed (both single-step visual completion and single-step structural completion are 1), since there are a total of 5 steps, the final visual completion and global structural completion are calculated as 0.2, thus accurately reflecting the actual state that the current step is completed, but the overall task is not yet finished.

[0072] Next, based on the industry attributes and business scenarios of the application under test, confidence weights are pre-configured for visual completion and structural completion. For example, in fields with high requirements for visual feedback, such as finance and insurance, the visual completion weight Wv can be set to 0.6 by default, and the structural completion weight Ws can be set to 0.4 by default. The specific weights can be dynamically adjusted according to the actual application scenario. The task completion confidence is calculated using the formula C=Wv×Sv+Ws×Ss, where C is the task completion confidence, Sv is the visual completion, and Ss is the structural completion. The calculated task completion confidence is compared with a preset completion threshold (such as 0.85, 0.9, etc.). If the task completion confidence is greater than or equal to the preset completion threshold, the current test task objective is confirmed to be completed; otherwise, it is confirmed to be incomplete.

[0073] This embodiment achieves accurate extraction of multimodal data change features before and after operation through multi-dimensional visual and structural comparison, covering key dimensions of visual and structural changes, avoiding omissions of change features, and thus ensuring the reliability of the test. Simultaneously, it combines visual and structural completion analysis with weighted fusion based on the test task objectives, making the calculation of task completion confidence more aligned with business needs, and achieving quantitative judgment of test task completion, thereby improving the accuracy, objectivity, and reliability of task completion determination.

[0074] In one embodiment, after step S205, the method further includes: S207. Extract negative features of the abnormal class from the multimodal data of the new interface to obtain visual abnormal features and structural abnormal features. S208. The negative features of visual anomalies and negative features of structural anomalies are weighted and fused to obtain the anomaly confidence level. S209. If the anomaly confidence level is greater than or equal to the preset anomaly threshold, then an anomaly is confirmed to be triggered, and the corresponding anomaly handling operation is performed according to the preset anomaly handling mechanism.

[0075] In this embodiment, after the operation command is executed, not only is the completion of the task objective determined, but also anomaly identification processing is performed based on the multimodal data of the new interface. Specifically, an anomaly negative feature library can be pre-constructed in an experience knowledge base, which stores the visual and structural features of various anomalies occurring in the test scenario. The visual features of anomalies include "error message icon appearing," "distorted control display," "blank page / screen tearing," etc., while the structural features include "target control node disappearing," "control attributes becoming non-interactive without a reasonable reason," "disordered page structure nodes," etc. Feature extraction is performed on the multimodal data of the new interface after the operation is executed, and the extracted features are matched with the anomaly negative feature library. Successfully matched anomaly features are selected, thereby obtaining visual anomaly features and structural anomaly features respectively.

[0076] Next, a visual anomaly score between 0 and 1 is calculated based on the number and severity of visual anomalies, and a structural anomaly score between 0 and 1 is calculated based on the number and severity of structural anomalies. Then, an anomaly confidence score is calculated using the formula Ca = Wv × Va + Ws × Sa (where Ca is the anomaly confidence score, Va is the visual anomaly score, and Sa is the structural anomaly score). Specifically, the same weighting method as the task completion confidence score can be used to assign confidence weights to visual and structural anomalies (for example, the visual weight Wv is 0.6 by default, and the structural weight Ws is 0.4 by default). The higher the anomaly confidence score, the greater the probability that the interface is abnormal.

[0077] The calculated anomaly confidence score is compared with a preset anomaly threshold (such as 0.25, 0.3, etc.). If the anomaly confidence score is greater than or equal to the preset anomaly threshold, it is confirmed that an anomaly has been triggered during the current test. At this time, the pre-built preset anomaly handling mechanism is invoked, and the corresponding anomaly handling operation is performed according to factors such as the type and severity of the anomaly. The anomaly handling operation includes retrying the current operation, returning to the previous screen, adjusting the operation command and re-executing, suspending the test and recording the anomaly information, etc.

[0078] This embodiment extracts abnormal features from the multimodal data of the new interface after the operation command is executed, and obtains the anomaly confidence score by quantifying and weighting the visual anomaly degree and structural anomaly degree. This achieves quantitative anomaly judgment and improves the objectivity and accuracy of anomaly judgment. At the same time, by triggering the corresponding anomaly handling mechanism through preset anomaly thresholds, anomalies can be automatically identified and handled during the testing process, which effectively improves the robustness and fault tolerance of automated testing, reduces test interruptions caused by anomalies, and improves the continuity and efficiency of testing.

[0079] In one embodiment, step S209 includes: S291. Analyze the multimodal data of the interface before and after the operation instruction is executed to determine the type of the currently triggered exception. S292. Match the exception type with the preset exception handling mechanism, confirm and execute the corresponding exception handling operation.

[0080] In this embodiment, after an exception is triggered, an exception type analysis is performed based on the multimodal data of the interface before and after the execution of the operation instruction. For example, exceptions occurring during the test are pre-classified into corresponding types, namely: interface exceptions (elements are invisible, non-interactive, page is unresponsive, control display is distorted, etc.), network exceptions (request timeout, connection failure, data loading failure, etc.), business exceptions (form validation failure, process interruption, business rules not met, etc.), and system exceptions (application crash, system error, app crash, etc.). Combining the multimodal data change characteristics before and after the execution of the operation instruction, the operation execution log, and the running log of the application under test, the current exception is matched with the exception type classification system to confirm the specific exception type triggered, and key information such as the time, location, and operation sequence of the exception are recorded.

[0081] The preset exception handling mechanism configures targeted handling strategies for different types and severity of exceptions. For example, for UI exceptions, the strategy is to "retry the current operation (maximum 3 times), and if it still fails, roll back to the previous step"; for network exceptions, the strategy is to "wait 5 seconds and retry (maximum 3 times), and if it still fails, record the exception and stop the test"; for business exceptions, the strategy is to "adjust the input parameters according to the error message and retry, and if it cannot be adjusted automatically, record the exception and stop the test"; and for system exceptions, the strategy is to "immediately record the exception and stop the test, and generate a crash report," etc. The confirmed current exception type is precisely matched with the preset exception handling mechanism, the corresponding handling strategy is extracted and the corresponding exception handling operation is automatically executed, and the exception information and handling process are recorded in the test log.

[0082] This embodiment classifies various anomalies during the testing process, clarifies the causes and manifestations of the anomalies, provides a foundation for subsequent targeted handling, and achieves targeted handling of various anomalies through a preset anomaly handling mechanism, avoiding a one-size-fits-all approach to anomaly handling, improving the effectiveness of anomaly handling, reducing test failures and interruptions caused by anomalies, and improving the success rate and efficiency of automated testing.

[0083] It should be noted that there is no necessary order between the above steps. Those skilled in the art will understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0084] The following combination Figure 3 and Figure 4 This paper introduces the system architecture and multimodal result verification process of the scenario-driven automated testing method based on multimodality in an application example: like Figure 3As shown, the system in this embodiment includes a task receiving module, a multimodal perception module, a multi-agent collaboration core, an experience knowledge base module, and a script generation module. The multimodal perception module obtains interface screenshots and structural data (such as a view hierarchy tree (DOM) / document object model (XML)) through the system API or test framework interface, and performs text optical character recognition (OCR), control attribute parsing, and visual feature extraction (such as icon template features, color histograms, key points, or depth feature vectors).

[0085] The system comprises the following modules: a task receiving module for receiving test task objectives described in natural language; a multimodal perception module for acquiring and processing interface screenshots and interface structure data; and a multi-agent collaboration core comprising: an observer module for outputting state vectors containing text elements, control sets (identifiers, coordinates, availability), page hierarchy, and visual feature sets; a planner module for generating operation instructions based on task objectives, historical operation sequences, and the current state vector, including target element location and action type; an executor module for calling an automation framework to execute instructions and providing feedback on execution results; a verifier module for comparing state vectors before and after execution, determining visual and structural differences, calculating task completion confidence, and outputting the determination; a reflector module for analyzing process logs, forming rule entries (such as preconditions and retry strategies), and persisting them to an experience knowledge base; a script generation module for generating automated test scripts from operation sequences; and an experience knowledge base module for storing test strategies and rules and providing reference for the multi-agent collaboration core.

[0086] The core of multi-agent collaboration adopts a message-passing-based collaboration mechanism, specifically including: The observer module publishes the extracted state vectors to the shared message bus; the planner module subscribes to the state vectors, combines the task objective with historical operation sequences to generate operation instructions, and publishes the instructions to the message bus; the executor module subscribes to the operation instructions, executes them, and then feeds back the execution results to the message bus; the verifier module subscribes to the state vectors before and after execution, performs comparative analysis, and publishes the verification results; the reflector module subscribes to the complete process log, analyzes it, and updates the experience knowledge base; each agent resolves decision conflicts through a priority mechanism, with the priority order being Verifier > Planner > Executor > Observer > Reflector.

[0087] like Figure 4As shown, in the "Upload Damaged Photos" step of the online auto insurance claims scenario, the screenshot before execution has a loading animation, and the screenshot after execution shows a photo thumbnail and a green checkmark icon; the status of the upload control in the structured data changes from interactive to completed. The verifier module combines visual evidence (icons, thumbnails, colors) and structural evidence (node ​​attributes and availability changes) to determine the completion of this step. When the final page displays the "Acceptance Number" text and the page layout matches the successful status template, the system determines that the overall task is completed and generates a script. The reflector module records the rule: "The upload step must be completed before submission; if the upload fails, retry once and return to the retry entry after timeout."

[0088] Further reference Figure 5 As a response to the above Figure 2 The present invention provides an embodiment of a multimodal, scene-driven automated testing device, which, in accordance with the implementation of the method shown, provides an embodiment of such a device. Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0089] like Figure 5 As shown, the multimodal-based scene-driven automated testing device 50 described in this embodiment includes: The receiving module 501 is used to receive the test task objective described in natural language and generate a corresponding set of test steps based on the test task objective; The multimodal perception module 502 is used to acquire multimodal data of the current interface of the application under test, wherein the multimodal data includes at least an interface screenshot and interface structure data. The scenario-driven decision module 503 is used to generate operation instructions for the next test step based on the scenario-driven strategy, the multimodal data of the current interface, the test task objective, and the historical operation sequence. The execution acquisition module 504 is used to execute the operation command on the application under test and acquire multimodal data of the new interface after execution; The multimodal verification module 505 is used to compare the multimodal data of the interface before and after the operation instruction is executed, and to confirm whether the test task objective is completed and whether an exception is triggered based on the comparison result. The iterative testing module 506 is used to, if the test task objective is not completed, use the new interface as the current interface, return to continue generating operation instructions for the next test step based on the scenario-driven strategy and execute them, until all test steps of the test task objective are completed.

[0090] The module referred to in this invention is a series of computer program instruction segments that can perform specific functions. It is more suitable than a program for describing the execution process of multimodal scenario-driven automated testing. For specific implementation methods of each module, please refer to the corresponding method embodiments above, which will not be repeated here.

[0091] In one embodiment, the scene-driven decision module 503 includes: A visual encoding unit is used to perform visual encoding on the screenshot of the current interface and extract visual feature vectors. The structure encoding unit is used to perform structure encoding on the interface structure data of the current interface and extract the structure feature vector. The feature fusion unit is used to fuse the visual feature vector and the structural feature vector through a cross-attention mechanism to obtain the multimodal feature representation of the current interface; The scenario-driven decision unit is used to generate operation instructions for the next test step based on the scenario-driven strategy, the multimodal feature representation, the test task objective, and the historical operation sequence.

[0092] In one embodiment, the scenario-driven decision unit includes: A scene confirmation unit is used to confirm the scene state of the current interface based on the multimodal feature representation. An action matching unit is used to perform scene action matching on the scene state by combining the test task objective with the historical operation sequence, and to confirm the standard test action of the scene state. The instruction generation unit is used to generate operation instructions for the next test step based on the execution information of the standard test action and the positioning information of the current interface.

[0093] In one embodiment, the execution acquisition module 504 includes: An execution confirmation unit is used to execute the operation instruction on the application under test and confirm the operation area of ​​the operation instruction; The data acquisition unit is used to acquire multimodal data of the operation area after the operation instruction is executed; The overlay update unit is used to overlay the multimodal data of the interface before execution with the multimodal data of the operation area after execution, so as to obtain the multimodal data of the new interface after execution.

[0094] In one embodiment, the multimodal verification module 505 includes: The visual comparison unit is used to visually compare the screenshots of the current interface and the new interface to obtain the corresponding visual change features. The structure comparison unit is used to compare the node differences of the interface structure data of the current interface and the new interface to obtain the corresponding structural change features. The analysis unit is used to perform completion analysis on the visual change features and structural change features based on the test task objectives, and obtain the visual completion and structural completion respectively. The weighted fusion unit is used to weightedly fuse the visual completion score and the structural completion score to obtain the task completion confidence score; The completion confirmation unit is used to confirm that the test task objective has been completed if the confidence level of the task completion is greater than or equal to a preset completion threshold.

[0095] In one embodiment, the device 50 further includes: An anomaly feature extraction module is used to extract anomaly-type negative features from the multimodal data of the new interface to obtain visual anomaly features and structural anomaly features. An anomaly feature fusion module is used to weightedly fuse the negative features of visual anomalies and the negative features of structural anomalies to obtain an anomaly confidence level. An exception handling module is used to confirm the triggering of an exception if the exception confidence level is greater than or equal to a preset exception threshold, and to perform corresponding exception handling operations according to a preset exception handling mechanism.

[0096] In one embodiment, the exception handling module includes: An anomaly type confirmation unit is used to perform anomaly type analysis based on the multimodal data of the interface before and after the operation instruction is executed, and to confirm the type of anomaly currently triggered. The mechanism matching and processing unit is used to match the exception type with a preset exception handling mechanism, confirm and execute the corresponding exception handling operation.

[0097] In the above embodiments, the present invention discloses a multimodal scenario-driven automated testing device, which generates operation instructions for each test step by combining task intent and historical operation sequence through scenario-driven strategy, and performs multimodal task completion verification based on the operation before and after, effectively improving the perception ability of complex business scenarios and the accuracy and robustness of test result judgment.

[0098] Specific limitations regarding the multimodal scene-driven automated testing device can be found in the limitations of the multimodal scene-driven automated testing method described above, and will not be repeated here. Each module in the aforementioned multimodal scene-driven automated testing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device, or stored in the memory of the computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0099] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0100] Another embodiment of the present invention provides a computer device, such as... Figure 6 As shown, computer device 60 includes: One or more processors 601 and memory 602, Figure 6 The following section uses a processor 601 as an example. The processor 601 and the memory 602 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0101] The processor 601 is used to perform various control logics of the computer device 60. It can be any conventional processor, microprocessor, state machine, general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), microcontroller, ARM (AcornRISC Cachine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0102] The memory 602, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the multimodal scene-driven automated testing method in this embodiment of the invention. The processor 601 executes various functional applications and data processing of the computer device 60 by running the non-volatile software programs, instructions, and units stored in the memory 602, thereby implementing the multimodal scene-driven automated testing method in the above-described method embodiment.

[0103] Another embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by one or more processors, perform the steps of the multimodal-based scene-driven automated testing method in any of the above method embodiments.

[0104] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0105] Based on the above description of the embodiments, those skilled in the art will understand that the methods described in the embodiments can be implemented using software plus necessary general-purpose hardware platforms. Of course, they can also be implemented using hardware, but in many cases, the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0106] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The computer program can be stored in a non-volatile, computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The storage medium can be a memory, magnetic disk, floppy disk, flash memory, optical storage, etc.

[0107] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0108] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multi-modal based scenario driven automated testing method, characterized in that, include: Receive test task objectives described in natural language, and generate a corresponding set of test steps based on the test task objectives; Obtain multimodal data of the current interface of the application to be tested, wherein the multimodal data includes at least screenshots of the interface and interface structure data; Based on the scenario-driven strategy, the operation instructions for the next test step are generated according to the multimodal data of the current interface, the test task objective, and the historical operation sequence. The operation command is executed on the application under test, and multimodal data of the new interface after execution is collected; Based on the comparison of multimodal data of the interface before and after the execution of the operation command, the test task objective is confirmed to be completed according to the comparison results. If the test task objective is not completed, a new interface will be used as the current interface, and the operation instructions for generating the next test step based on the scenario-driven strategy will be executed until all test steps for the test task objective are completed.

2. The scene-driven automated testing method based on multimodality according to claim 1, characterized in that, The scenario-driven strategy generates operation instructions for the next test step based on the multimodal data of the current interface, the test task objective, and the historical operation sequence, including: Visual encoding is performed on the screenshot of the current interface to extract visual feature vectors; The interface structure data of the current interface is structurally encoded, and the structural feature vector is extracted. The visual feature vector and the structural feature vector are fused by a cross-attention mechanism to obtain a multimodal feature representation of the current interface; Based on the scenario-driven strategy, the operation instructions for the next test step are generated according to the multimodal feature representation, the test task objective, and the historical operation sequence.

3. The scene-driven automated testing method based on multimodality according to claim 2, characterized in that, The scenario-driven strategy generates operation instructions for the next test step based on the multimodal feature representation, test task objective, and historical operation sequence, including: The scene state of the current interface is determined based on the multimodal feature representation; By combining the test task objective with the historical operation sequence, the scene state is matched with scene actions to confirm the standard test actions for the scene state. Based on the execution information of the standard test action and the positioning information of the current interface, the operation instructions for the next test step are generated.

4. The scene-driven automated testing method based on multimodality according to claim 1, characterized in that, The step of executing the operation command on the application under test and collecting multimodal data of the new interface after execution includes: Execute the operation command on the application under test and confirm the operation area of ​​the operation command; After the operation command is executed, multimodal data of the operation area after execution is collected; The multimodal data of the operation area after execution is used to overwrite the multimodal data of the interface before execution, so as to obtain the multimodal data of the new interface after execution.

5. The scene-driven automated testing method based on multimodality according to claim 1, characterized in that, The comparison of multimodal data from the interface before and after the execution of the operation command, and the confirmation of whether the test task objective has been achieved based on the comparison results, includes: Visual comparison is performed on the screenshots of the current interface and the new interface to obtain the corresponding visual change features; The node differences between the interface structure data of the current interface and the new interface are compared to obtain the corresponding structural change characteristics. Based on the test task objectives, a completion analysis is performed on the visual change features and structural change features to obtain the visual completion rate and structural completion rate, respectively. The visual completion score and structural completion score are weighted and fused to obtain the task completion confidence score. If the confidence level of task completion is greater than or equal to the preset completion threshold, then the test task objective is confirmed to be completed.

6. The scene-driven automated testing method based on multimodality according to claim 1, characterized in that, After comparing the multimodal data of the interface before and after the execution of the operation instruction, and confirming whether the test task objective has been completed based on the comparison results, the method further includes: Anomaly-type negative features are extracted from the multimodal data of the new interface to obtain visual anomaly features and structural anomaly features; The negative features of visual anomalies and negative features of structural anomalies are weighted and fused to obtain the anomaly confidence level. If the anomaly confidence level is greater than or equal to the preset anomaly threshold, then an anomaly is confirmed to be triggered, and the corresponding anomaly handling operation is performed according to the preset anomaly handling mechanism.

7. The scene-driven automated testing method based on multimodality according to claim 6, characterized in that, The confirmation of an exception triggering the exception involves performing corresponding exception handling operations according to a preset exception handling mechanism, including: Anomaly type analysis is performed on the multimodal data of the interface before and after the operation instruction is executed to confirm the type of anomaly currently triggered. The exception type is matched with the preset exception handling mechanism, and the corresponding exception handling operation is confirmed and executed.

8. A scene-driven automated testing device based on multimodal conditions, characterized in that, include: The receiving module is used to receive the test task objectives described in natural language and generate a corresponding set of test steps based on the test task objectives. A multimodal perception module is used to acquire multimodal data of the current interface of the application under test, wherein the multimodal data includes at least an interface screenshot and interface structure data. The scenario-driven decision module is used to generate operation instructions for the next test step based on the scenario-driven strategy, the multimodal data of the current interface, the test task objective, and the historical operation sequence. The execution acquisition module is used to execute the operation instructions on the application under test and acquire multimodal data of the new interface after execution. The multimodal verification module is used to compare the multimodal data of the interface before and after the execution of the operation command, and to confirm whether the test task objective has been completed and whether an exception has been triggered based on the comparison results. The iterative testing module is used to, if the test task objective is not completed, use a new interface as the current interface, return to continue generating operation instructions for the next test step based on the scenario-driven strategy and execute them, until all test steps for the test task objective are completed.

9. A computer device, characterized in that, Includes at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the multimodal-based scene-driven automated testing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the multimodal-based scene-driven automated testing method according to any one of claims 1-7.

Images