App automatic test method and agent system based on multi-modal perception

Abstract

This invention discloses an automated testing method and agent system for apps based on multimodal perception, relating to the field of image recognition or understanding. The method includes the following steps: a real-time perception layer takes screenshots of the app's screen interface in real time; when the interface changes, the screenshots are processed in three ways: detecting control bounding boxes; performing visual semantic encoding on interface elements; and simultaneously extracting text content to generate function tags. The three outputs are then fused to construct a UI tree. When a task instruction is input, a dynamic inference layer generates an action sequence based on the task instruction and the UI tree through a neural-symbolic collaborative mechanism, updates the dynamic document, and saves it to a real-time knowledge base. An execution optimization layer converts the actions of the action sequence into platform-specific operation code and executes the operation code, while automatically marking abnormal paths and updating avoidance strategies to the real-time knowledge base. This invention achieves a real-time perception-dynamic inference-execution closed loop through a three-level architecture, significantly improving generalization ability, exception coverage, and execution efficiency.

Description

Technical Field

[0001] This invention relates to the field of image or video recognition or understanding, specifically to an automated testing method and agent system for apps based on multimodal perception. Background Technology

[0002] With the increasing complexity of the mobile application ecosystem and the accelerating trend of multi-platform integration, automated testing technology faces increasingly severe challenges. Current mainstream automated testing frameworks (such as Appium and UI Automator) primarily operate based on predefined control coordinates or hierarchical structure attributes. Their technical limitations are becoming increasingly apparent in cross-device, cross-version, and dynamic scenarios, specifically manifesting in the following problems:

[0003] Insufficient dynamic interface adaptation capability: Traditional tools rely on static coordinates or control properties to locate elements, which cannot effectively cope with scenarios such as dynamic interface rendering, resolution adaptive adjustment or screen rotation, causing test cases to frequently fail when the device is migrated or the layout is updated.

[0004] Visual semantic understanding gap: Existing solutions lack the ability to model the association between the visual features of interface elements (such as icon style and color contrast) and functional semantics, making it difficult to identify the intent in complex interaction scenarios (such as distinguishing the functional differences between the "shopping cart" icon and the "favorites" icon), which limits the level of intelligence of the testing logic.

[0005] High version iteration and maintenance costs: Static scripts and fixed test knowledge bases cannot adapt to changes in interface logic during rapid APP iterations (such as control hierarchy reconstruction and business process jump rule adjustments), requiring manual relocation of elements and correction of test logic, which significantly increases maintenance overhead;

[0006] Lack of cross-platform generalization capability: The differences in UI description system and interaction protocol between heterogeneous agents such as Android and iOS force developers to design test scripts independently for different platforms, resulting in low test logic reuse rate and difficulty in verifying consistency across multiple platforms.

[0007] While existing research has attempted to improve the robustness of control localization through OCR text matching or image template recognition, it has not yet overcome the limitations of single-modal information processing and lacks the ability to dynamically reason about the semantics of the interface context. Furthermore, existing knowledge-driven testing methods mostly rely on predefined rule bases, making it difficult to achieve real-time environmental awareness and adaptive decision-making, thus hindering the long-term evolution of automated testing agent systems in complex scenarios. Summary of the Invention

[0008] The purpose of this invention is to provide an automated testing method and agent system for apps based on multimodal perception, which aims to overcome the aforementioned problems existing in the prior art.

[0009] To achieve the objective, the present invention provides the following technical solution:

[0010] An automated testing method for apps based on multimodal perception includes the following steps:

[0011] Step S1: Start and connect the app under test, while the causal chain learner completes initialization based on the dynamic knowledge base.

[0012] Step S2: The real-time perception layer takes a screenshot of the APP's screen interface in real time.

[0013] Step S3: Use screenshots to perform dynamic response detection.

[0014] Step S4: When a change in the interface is detected through dynamic response detection, the screenshot is processed in three ways: control bounding box detection; visual semantic encoding of interface elements; and simultaneous extraction of text content to generate function labels.

[0015] Step S5: Merge the three output results, construct a UI tree containing hierarchical topology, and pass the UI tree to the dynamic inference layer.

[0016] Step S6: When the dynamic inference layer receives a task instruction, it parses the task instruction through a neural-symbolic collaborative mechanism, combines it with the UI tree to generate an action sequence, and then performs symbol system verification and permission status checks on the action sequence.

[0017] Step S7: Update the action sequence to the dynamic document and save it to the real-time knowledge base, while passing the action sequence to the execution optimization layer.

[0018] Step S8: The optimization layer converts the action sequence into platform-specific operation code; executes the operation code, and stores the operation path and operation result in the multimodal memory pool.

[0019] Step S10: Repeat steps S3-S8 until the current task instruction is completed.

[0020] Furthermore, in step S4 above, the YOLOv5 model is used to detect the bounding boxes of controls; the CLIP model is used to perform visual semantic encoding of interface elements; and the OCR module is used to extract text content and generate function labels.

[0021] Furthermore, in step S5 above, the UI tree is in XML or JSON format.

[0022] Furthermore, in step S6 above, the dynamic inference layer parses task instructions through a neural-symbolic collaborative mechanism, combines it with the UI tree to generate an action sequence, and then performs symbolic system verification and permission status checks on the action sequence. Specifically:

[0023] The system uses a large language model to semantically parse the user's task commands to obtain the command intent; a causal chain learner generates an action sequence based on the UI tree and the command intent; and a symbolic system performs symbolic system verification and permission status checks on the action sequence to correct any errors and ensure that the user has the authority to execute the action.

[0024] Furthermore, in step S6 above, the generation of action sequences is further optimized by combining the temporal memory stored in the multimodal memory pool.

[0025] Furthermore, in step S6 above, after receiving a task instruction, the task instruction entry point is closed;

[0026] Between steps S5 and S6 above, the following steps are also included: the dynamic inference layer detects whether the current task instruction has been completed; if completed, the detection result is saved to the real-time knowledge base, the task instruction entry is opened, and the next task instruction is received; if not completed, the current task instruction is processed, and the task instruction entry remains closed.

[0027] Furthermore, step S8 also includes automatically marking abnormal paths, triggering path rollback, feeding back to the causal chain learner, and updating the avoidance strategy to the real-time knowledge base when symbol system verification fails or operation results are incorrect.

[0028] An APP automated testing agent system (i.e., intelligent agent) based on multimodal perception is used to execute any of the APP automated testing methods mentioned above. The agent system is based on a large language model and includes a real-time perception layer, a dynamic reasoning layer, an execution optimization layer, a real-time knowledge base, and a multimodal memory pool.

[0029] The aforementioned real-time perception layer is used to convert real-time screenshots of the APP's screen interface into a UI tree containing hierarchical topology relationships.

[0030] The aforementioned dynamic reasoning layer is used to parse task instructions through a neural-symbolic collaborative mechanism, combine them with the UI tree to generate action sequences, and then perform symbolic system verification and permission status checks on the action sequences.

[0031] The aforementioned execution optimization layer is used to convert action sequences into platform-specific operation code and execute the operation code;

[0032] The aforementioned real-time knowledge base is used to store updated dynamic documents, avoidance strategies, and save test results;

[0033] The aforementioned multimodal memory pool is used to store the operation path and operation result of the executed operation code.

[0034] Furthermore, the aforementioned real-time perception layer includes a dynamic response detection module, a YOLOv5 model, a CLIP model, an OCR module, and a UI structure generator. The dynamic response detection module is used to perform differential analysis on interface changes using the SSIM algorithm. When interface changes are detected, the YOLOv5 model is used to detect control bounding boxes, the CLIP model is used to perform visual semantic encoding of interface elements, and the OCR module is used to simultaneously extract text content and generate function labels. The UI structure generator is used to fuse the output results of the three models to construct a UI tree containing hierarchical topological relationships.

[0035] The aforementioned dynamic inference layer includes a multifunctional module, a causal chain learner, a symbol system, and a conflict resolution engine. The multifunctional module is used for LLM task parsing and task state detection; the causal chain learner generates action sequences based on the UI tree and task instructions; the symbol system validates the action sequences; and the conflict resolution engine automatically marks abnormal paths, triggers path rollback, and feeds back to the causal chain learner when symbol system validation fails or operation results are incorrect. Furthermore, the causal chain learner also triggers path rollback and generates a rollback action sequence when it detects an abnormal interface.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] Compared to existing technologies, this invention achieves a real-time perception-dynamic reasoning-execution closed loop through a three-tier architecture, resulting in significant improvements in generalization ability, maintenance costs, anomaly coverage, and execution efficiency. This invention can dynamically update the knowledge base according to task needs, ensuring rapid adaptation to different application scenarios, and automatically optimizes its decision-making process by continuously updating historical information and operation results. Attached Figure Description

[0038] Figure 1 This is a flowchart (i.e., an interaction diagram of the Agent system) of the APP automated testing method in this invention.

[0039] Figure 2 This is a schematic diagram illustrating the principle of visual-semantic alignment achieved through the CLIP model in this invention.

[0040] Figure 3 This is a flowchart illustrating the hierarchical verification acceleration process for the memory component in this invention. Detailed Implementation

[0041] Specific embodiments of the present invention will now be described with reference to the accompanying drawings. Many details are described below to provide a comprehensive understanding of the invention; however, those skilled in the art will be able to implement the invention without these details.

[0042] like Figures 1-3 As shown, an automated testing method for apps based on multimodal perception includes the following steps:

[0043] Step S1: Start and connect the app under test, while the causal chain learner completes initialization based on the dynamic knowledge base.

[0044] Specifically, this app automation testing method begins with the app automation testing agent system establishing a connection with the app. This connection can be either wired or wireless.

[0045] Step S2: The real-time perception layer takes a screenshot of the APP's screen interface in real time.

[0046] Specifically, after the connection is established, the real-time perception layer will continuously capture images of the APP screen interface at a frequency of 15 frames per second (i.e., 15fps).

[0047] Step S3: Use screenshots to perform dynamic response detection.

[0048] Specifically, the real-time perception layer includes a dynamic response detection module, which uses the SSIM algorithm to perform difference analysis on captured screenshots and detect the dynamic response of the screen interface when the user operates or the system changes. This allows for timely detection of interface changes, abnormal interfaces, or unresponsiveness.

[0049] Step S4: When a change in the interface is detected through dynamic response detection, the screenshot is processed in three ways: control bounding box detection; visual semantic encoding of interface elements; and simultaneous extraction of text content to generate function labels.

[0050] like Figure 1 and Figure 2 As shown, specifically, the real-time perception layer includes a YOLOv5 model, a CLIP model, and an OCR module. When a change in the interface is detected, screenshots are processed in three ways: the YOLOv5 model detects the bounding boxes of controls; Figure 2 The diagram illustrates the principle of visual-semantic alignment for the CLIP model. The CLIP model performs visual and semantic encoding on interface elements (e.g., mapping a red circular button to the "close" function); the OCR module simultaneously extracts the text content and generates function labels.

[0051] Step S5: Merge the three output results, construct a UI tree containing hierarchical topology, and pass the UI tree to the dynamic inference layer.

[0052] Specifically, the real-time perception layer includes a UI structure generator. This generator integrates the outputs of the YOLOv5 model, the CLIP model, and the OCR module to construct a UI tree containing hierarchical topological relationships, thereby converting screenshots of the screen interface into structured data. The UI tree can be in XML or JSON format, or other structured languages.

[0053] Step S6: The dynamic inference layer checks whether the current task instruction has been completed; if the task instruction has been completed, the detection result is saved to the real-time knowledge base, the task instruction entry is opened, and the next task instruction in the task instruction sequence is received; if it has not been completed, the current task instruction is processed.

[0054] The aforementioned task instructions include preset test instructions and user-issued operation instructions. In actual production, a large number of preset test instructions are first used as task instructions to cover and train the APP's functions. When the test accuracy reaches a high level, such as 95%, it indicates that the Agent system is ready for release or launch. After the Agent system goes live, unknown APPs can be selected as test subjects for testing. Users construct the test instruction set for the APP under test and test the APP.

[0055] Step S7: When the dynamic inference layer receives a task instruction, it parses the task instruction through a neural-symbolic coordination mechanism and generates an action sequence by combining it with the UI tree.

[0056] Specifically, the signal entry point of the dynamic inference layer has a multi-functional module with two states: waiting and running, corresponding to the two functions of LLM task parsing and task state detection, respectively.

[0057] When no task command is input, the multi-functional module is in a waiting state, and the LLM task parsing function is enabled. This is used to parse the task command using the LLM method. After the LLM task parsing is complete, the multi-functional module automatically switches from the waiting state to the running state, and the task status detection function is enabled. This is used to detect whether the current task command has been completed, i.e., to execute step S6. After the current task command is completed, the multi-functional module automatically switches from the running state to the waiting state, waiting for the input of the next task command, and this process repeats.

[0058] Of course, the aforementioned multifunctional module can also be replaced by a task status detection module and an LLM task parsing module.

[0059] Specifically, step S7 includes the following sub-steps:

[0060] Step S701: After receiving a task instruction, the dynamic inference layer closes the task instruction entry point. Simultaneously, the multi-functional module uses a large language model (LLM) to perform semantic parsing of the task instruction to obtain the instruction intent.

[0061] Step S702: The causal chain learner generates a series of actions that conform to the causal relationship of the context (such as "modify avatar → click album → select photo", or "click search bar → enter keywords → filter results") based on the UI tree and instruction intent, i.e., action sequence.

[0062] The causal chain is whereby the Agent system executes a predefined action (such as clicking or swiping), records the interface changes after the operation, and establishes a causal chain of "action → element → result" (e.g., clicking the "search bar" → popping up the keyboard → entering text → displaying the results list).

[0063] Step S703: The causal chain learner combines the temporal memory stored in the multimodal memory pool to further optimize the action sequence.

[0064] Step S704: The symbol system performs symbol system verification on the action sequence, and if the symbol system verification fails, it triggers the conflict resolution engine to fix any possible errors, and then performs a permission status check to ensure that there is permission to execute the action.

[0065] Symbol system validation checks whether actions conform to UI rules, such as "when entering image information in a chat window, UI rules require a '+' button to the right of the input box." The conflict resolution engine refers to contextual rule updates, i.e., the functional changes of the same element in different scenarios (e.g., a "share button" is a share link on a product page but saves content on an article page), updating the document through historical interaction data and annotating conditional trigger logic.

[0066] Step S8: Update the action sequence to the dynamic document and save it to the real-time knowledge base, while passing the action sequence to the execution optimization layer.

[0067] Step S9: The optimization layer converts the action sequence into platform-specific operation code; executes the operation code, and stores the operation path and operation result in the multimodal memory pool.

[0068] Specifically, the execution optimization layer includes a cross-device abstraction module, which converts the actions of the action sequence into platform-specific operation code.

[0069] Preferably, step S9 further includes: automatically marking abnormal paths and updating avoidance strategies to the real-time knowledge base when symbol system verification fails or operation results are incorrect.

[0070] Specifically, when symbol system verification fails or the operation result is incorrect, the conflict resolution engine will trigger path rollback, feeding back to the causal chain learner so that the learner can perform reinforcement learning. Simultaneously, it will automatically label abnormal paths and update the avoidance strategy to the real-time knowledge base. Furthermore, when the causal chain learner detects an abnormal interface (such as a crash pop-up or an advertising page), it will also trigger path rollback, generating a rollback action sequence.

[0071] Step S10: Repeat steps S3-S9 until the current task instruction is completed.

[0072] Specifically, after the current task instruction is completed, the causal chain learner initializes according to the dynamic knowledge base, and the multi-functional module starts the LLM task parsing function to allow the reception of the next task instruction, so as to complete all task instructions in the task instruction sequence one by one according to steps S1-S10.

[0073] Simple example:

[0074] Example 1: E-commerce App Product Search Test (Training Phase)

[0075] 1. Initialization: Connect to the electronic device with the e-commerce app installed. If the electronic device is an iOS device, use the Xcode official toolchain and the libimobiledevice open-source library; if the electronic device is an Android device, use Android DebugBridge (ADB) to establish a connection with the electronic device. Load the e-commerce app's industry semantic rule set (e.g., ecommerce_rules_v2). For brevity, the initialization step is omitted in the description of other embodiments.

[0076] 2. Perception stage:

[0077] The Python code for parsing e-commerce product search pages based on YOLOv5 and Tesseract (OCR model) is as follows (including three elements: dynamic perception, element association, and rule validation).

[0078] # E-commerce App Product Search Page Parsing System (YOLOv5 + Tesseract Solution)

[0079] import cv2

[0080] import pytesseract

[0081] import torch

[0082] import numpy as np

[0083] from yolov5.models.experimental import attempt_load

[0084] from yolov5.utils.general import non_max_suppression

[0085] class SearchPageAnalyzer:

[0086] def __init__(self):

[0087] # Initialize the multimodal parsing module

[0088] self.device = 'cuda' if torch.cuda.is_available() else 'cpu'

[0089] self.yolo = self._init_yolo_model('yolov5s_ecommerce.pt')

[0090] self.ocr_config = r'--oem 3 --psm 6 -l chi_sim+eng'

[0091] # Dynamically Sensing Parameters

[0092] self.min_ocr_confidence = 0.7

[0093] self.element_classes = ['search_box', 'filter_btn', 'product_card', 'price_tag']

[0094] # Rule Validation Configuration

[0095] self.required_elements = {

[0096] 'search_page': ['search_box', 'search_btn'],

[0097] 'product_card': ['product_image', 'price_tag']

[0098] }

[0099] def _init_yolo_model(self, weights_path):

[0100] Loading a customized YOLOv5 model.

[0101] model = attempt_load(weights_path).to(self.device)

[0102] model.eval()

[0103] return model

[0104] def _dynamic_preprocess(self, img):

[0105] "Dynamic Sensing Preprocessing"

[0106] # CLAHE Contrast Enhancement

[0107] lab = cv2.cvtColor(img, cv2.COLOR_BGR2LAB)

[0108] l, a, b = cv2.split(lab)

[0109] clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))

[0110] cl = clahe.apply(l)

[0111] enhanced = cv2.merge((cl,a,b))

[0112] # Edge-preserving filtering

[0113] return cv2.bilateralFilter(enhanced, 9, 75, 75)

[0114] def _element_relationship_analysis(self, elements):

[0115] Element Association Analysis (UI Structure Parsing)

[0116] # Create hierarchical levels by sorting by Y coordinate

[0117] sorted_elements = sorted(elements, key=lambda x: x['position'][1])

[0118] hierarchy = {}

[0119] for idx, elem in enumerate(sorted_elements):

[0120] x1, y1, x2, y2 = elem['position']

[0121] center = ((x1+x2) / 2, (y1+y2) / 2)

[0122] # Find the parent containerfor ancestor in reversed(sorted_elements[:idx]):

[0123] a_x1, a_y1, a_x2, a_y2 = ancestor['position']

[0124] if a_x1<center[0]<a_x2 and a_y1<center[1]<a_y2:

[0125] hierarchy[elem['id']] = ancestor['id']

[0126] break

[0127] return hierarchy

[0128] def _rule_validation(self, elements, page_type):

[0129] """Symbolic rule validation"""

[0130] missing = []

[0131] for req in self.required_elements.get(page_type, []):

[0132] if not any(e['type'] == req for e in elements):

[0133] missing.append(req)

[0134] if missing:

[0135] raise ValueError(f"Violation of {page_type} rule: missing element {missing}")

[0136] def parse_search_page(self, screenshot):

[0137] "Complete Analysis Process"

[0138] # Dynamic Perception Stage

[0139] processed_img = self._dynamic_preprocess(screenshot)

[0140] img_tensor = torch.from_numpy(processed_img).permute(2,0,1).float().div(255).unsqueeze(0)

[0141] # YOLOv5 Element Detection

[0142] with torch.no_grad():

[0143] pred = self.yolo(img_tensor.to(self.device))[0]

[0144] detections = non_max_suppression(pred, 0.5, 0.5)[0]

[0145] elements = []

[0146] for det in detections.cpu().numpy():

[0147] x1, y1, x2, y2, conf, cls = det

[0148] elem_type = self.element_classes[int(cls)]

[0149] # Element ROI Extraction

[0150] roi = screenshot[int(y1):int(y2), int(x1):int(x2)]

[0151] # Tesseract OCR recognition

[0152] gray = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)

[0153] text = pytesseract.image_to_string(gray, config=self.ocr_config).strip()

[0154] # Building a dynamic element document

[0155] elements.append({

[0156] "id": f"elem_{len(elements)}",

[0157] "type": elem_type,

[0158] "position": (x1, y1, x2, y2),

[0159] "text": text,

[0160] "confidence": float(conf)

[0161] })

[0162] # Element Association Analysis

[0163] hierarchy = self._element_relationship_analysis(elements)

[0164] # Rule validation (complies with symbolic constraints)

[0165] self._rule_validation(elements, 'search_page')

[0166] return {

[0167] "version": "ecommerce_1.0",

[0168] "page_type": "search_page",

[0169] "elements": elements,

[0170] "hierarchy": hierarchy,

[0171] "validation": {

[0172] "status": "passed",

[0173] "timestamp": "2024-03-20T14:30:00Z"

[0174] }

[0175] }

[0176] 3. Reasoning and Decision-Making Stage:

[0177] Generate a sequence of actions: Click the search bar → Enter "smartphone" → Select price sorting → Verify results;

[0178] Symbol system verification: Confirm that the search button exists and is not grayed out.

[0179] 4. Execution Phase:

[0180] The command is converted to an ADB command: adb shell input tap 520 1800; the test result list is updated, and if the price sorting label does not appear, the abnormal recovery strategy is triggered.

[0181] Example 2: Payment Page Anomaly Handling (Training Phase)

[0182] Anomaly detection: An unknown pop-up window was detected using the SSIM algorithm (similarity to a known pop-up window < 0.7).

[0183] Determining payment failure: The causal chain learner matches the "payment failure pop-up" scenario in the history and detects unexpected interfaces (i.e. abnormal interfaces).

[0184] Automatic recovery (i.e., path rollback): Click the retry button → check network status → return to the preset path on the product page.

[0185] Example 3: Agent system online testing

[0186] The Agent system achieves intelligent test coverage across the entire process from requirement to deployment by maintaining test knowledge through dynamic documentation, optimizing operation paths through causal chain learning, and ensuring operational reliability through symbolic verification. When the Agent system's test accuracy reaches a high level, such as 95%, it indicates that the Agent system is ready for release or deployment. After the Agent system is deployed and launched, unknown apps can be selected as test subjects. Users can build test instruction sets for the app under test and perform tests on it.

[0187] E-commerce APP testing process code:

[0188] def test_checkout_flow():

[0189] # Initialize the multimodal awareness agent

[0190] Agent = TestAgent("amazon.apk")

[0191] Agent.start_recording()

[0192] # Execute test

[0193] try:

[0194] Agent.execute("Search for iPhone 15")

[0195] Agent.execute("Add to cart")

[0196] if Agent.detect_element("Insufficient inventory warning"):

[0197] raise TestError("Inventory error")

[0198] Agent.execute("Entering the checkout page")

[0199] # Symbol verification payment conditions

[0200] SymbolValidator.check(

[0201] "total_amount>0",

[0202] "shipping_address.exists()" )

[0204] finally:

[0205] report = Agent.generate_report()

[0206] upload_to_jira(report).

[0207] This invention also discloses an APP automated testing agent system based on multimodal perception, used to execute the above-mentioned APP automated testing method.

[0208] The agent system operates in two phases: training and deployment. During training, the system first targets specific apps, executing pre-defined test instruction sets. Based on multimodal perception, it learns and recognizes the visual and semantic perception of page elements, generating a UI tree. This UI tree is stored in a multimodal memory pool and a real-time knowledge base. Combining the current task requirements and UI tree information, it generates action sequences, converts these sequences into operation code, and executes the code. In the deployment phase, the agent system selects an unknown app as the test subject. Users construct a test instruction set for the app and test it. The agent system continuously optimizes its decision-making process by updating historical information and operation results.

[0209] The agent system is based on a large language model (LLM) and includes a real-time perception layer, a dynamic reasoning layer, an execution optimization layer, a real-time knowledge base, a dynamic knowledge base, and a multimodal memory pool.

[0210] The real-time perception layer is used to convert real-time screenshots of the APP's screen interface into a UI tree containing hierarchical topology relationships.

[0211] Specifically, the real-time perception layer includes a dynamic response detection module, a YOLOv5 model, a CLIP model, an OCR module, and a UI structure generator. The dynamic response detection module is used to perform difference analysis on interface changes using the SSIM algorithm. When an interface change is detected, the YOLOv5 model is used to detect control bounding boxes, the CLIP model is used to perform visual semantic encoding on interface elements, and the OCR module is used to extract text content and generate function labels simultaneously. The UI structure generator is used to fuse the output results of the three models to construct a UI tree containing hierarchical topological relationships.

[0212] The dynamic reasoning layer is used to parse task instructions through a neural-symbolic collaborative mechanism, combine them with the UI tree to generate action sequences, and then perform symbol system verification and permission status checks on the action sequences.

[0213] Specifically, the dynamic reasoning layer includes a multifunctional module, a causal chain learner, a symbol system, and a conflict resolution engine. The multifunctional module is used for LLM task parsing and task state detection. The causal chain learner is used to generate action sequences based on the UI tree and task instructions. The symbol system is used to verify the action sequences. The conflict resolution engine is used to automatically mark abnormal paths, trigger path rollback, and feed back to the causal chain learner when it detects that the symbol system verification has failed or the operation result is incorrect.

[0214] The execution optimization layer is used to convert action sequences into platform-specific operation code and execute the operation code.

[0215] As can be seen, this Agent system achieves a real-time perception-dynamic reasoning-execution closed loop through a three-level architecture, and has significantly improved in indicators such as: 1) generalization ability (reusability of test scripts across apps); 2) maintenance cost (adaptation time of test cases after interface changes); 3) exception coverage (automatic identification and handling of 23 common exception scenarios (such as permission pop-ups, network timeouts); 4) execution efficiency.

[0216] Real-time knowledge bases are used to store action sequences and existing knowledge, and belong to short-term memory. They maintain dynamic documents for each app, recording the functional descriptions of the current page elements (such as "back button → return to the previous page") and related operations (such as "a form must be filled out before submission" or "the payment process includes a verification code pop-up").

[0217] The dynamic knowledge base is a long-term, general-purpose memory. It is based on test task sets (instructions, processes, results) and causal chain learning to accumulate general rules (such as "all settings entries include a gear icon"). After the task instructions are executed, the real-time knowledge base is merged into the dynamic knowledge base.

[0218] Multimodal memory pools are used to store the operation paths and results of executed operation codes, and belong to short-term hybrid memory. They integrate visual perception, operation history, semantic parsing records, and physical interaction features into temporal memory, establish cross-dimensional feature indexes, and assist agent systems in predicting the next action (such as associating button colors with historical click success rates, or in e-commerce apps, completing the coherent operation of searching → browsing products → adding to cart → checkout).

[0219] As can be seen, the real-time knowledge base, dynamic knowledge base, and multimodal memory pool serve as memory components. Through the long-term pattern accumulation of the dynamic knowledge base, the context awareness of the real-time knowledge base, and the cross-dimensional association of the multimodal memory pool, this agent system can achieve the following characteristics: 1) Pre-generation of operation paths, reducing inference requests from large language models (LLMs). 2) Rapid anomaly localization, shortening the diagnosis time through a memory backtracking mechanism. 3) Cross-scenario knowledge reuse, allowing existing knowledge to be reused when adapting to new devices. This memory component effectively solves the problems of "repeated exploration" and "contextual breakage" existing in traditional AI agents, enabling this agent system to maintain efficient decision-making capabilities during continuous learning.

[0220] Figure 3 To accelerate the layered verification process of the memory component, its layered design is the core path to reduce the inference process. Key steps are represented in code, as shown in the following example:

[0221] Pattern pre-matching (Python code)

[0222] class MemoryRetriever:

[0223] def match_operation_pattern(self, current_state):

[0224] # Multimodal memory pool priority matching

[0225] visual_hash = self._generate_visual_hash(current_state.screenshot)

[0226] matched = self.memory_pool.search(visual_hash, threshold=0.9)

[0227] if matched:

[0228] # Directly call historical successful paths

[0229] return matched['action_chain']

[0230] else:

[0231] # Fuzzy matching in dynamic knowledge base

[0232] return self.dynamic_kb.fuzzy_match(current_state.elements).

[0233] More specifically, taking the handling of app pop-ups as an example, the real-time data structure of each memory component is as follows:

[0234] 1. When a "Insufficient storage space" pop-up is detected during file upload, the historical operation records of the multimodal memory pool are retrieved:

[0235] {

[0236] "visual_hash": "a83d2e",

[0237] "action_chain": [

[0238] {"action": "click", "target": "Clean Now"},

[0239] {"expect": "progress_bar", "timeout": 30}

[0240] ],

[0241] "success_rate": 92.7%

[0242] }

[0243] 2. Matching pre-stored rules for dynamic knowledge base:

[0244] rule = {

[0245] "condition": "dialog.text contains 'storage space'",

[0246] "recommend_action": "click('Clean Now')",

[0247] "exception_cases": ["Backup mode", "Agent updating"]

[0248] }

[0249] 3. The real-time knowledge base records the current element and confirms the action that can be performed:

[0250] {

[0251] "element_type": "dialog",

[0252] "buttons": ["Clean Now", "Do Not Process Now"],

[0253] "context": "file_upload_page

[0254] }

[0255] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. An automated testing method for apps based on multimodal perception, characterized in that: Includes the following steps: Step S1: Start and connect the app under test, while the causal chain learner completes initialization based on the dynamic knowledge base; Step S2: The real-time perception layer takes a screenshot of the APP's screen interface in real time; Step S3: Perform dynamic response detection using screenshots; Step S4: When a change in the interface is detected through dynamic response detection, the screenshot is processed in three ways: the YOLOv5 model is used to detect the bounding box of the control; the CLIP model is used to perform visual semantic encoding of the interface elements; and the OCR module is used to extract the text content and generate function labels simultaneously. Step S5: Merge the three outputs, construct a UI tree containing hierarchical topology, and pass the UI tree to the dynamic inference layer; the UI tree is in XML or JSON format. Step S6: When the dynamic inference layer receives a task instruction, it parses the task instruction through a neural-symbolic collaborative mechanism, combines it with the UI tree to generate an action sequence, and then performs symbol system verification and permission status check on the action sequence. Step S7: Update the action sequence to the dynamic document and save it to the real-time knowledge base, while passing the action sequence to the execution optimization layer; Step S8: The optimization layer converts the action sequence into platform-specific operation code; executes the operation code and stores the operation path and operation result in the multimodal memory pool; when the symbol system verification fails or the operation result is incorrect, the abnormal path is automatically marked, the path rollback is triggered, and feedback is sent to the causal chain learner, while the avoidance strategy is updated to the real-time knowledge base. Step S9: Repeat steps S3-S8 until the current task instruction is completed.

2. The APP automated testing method based on multimodal perception according to claim 1, characterized in that: In step S6, the dynamic inference layer parses task instructions through a neural-symbolic collaborative mechanism, combines it with the UI tree to generate an action sequence, and then performs symbolic system verification and permission status checks on the action sequence. Specifically: The semantics of user-issued task commands are semantically parsed using a large language model to obtain the command intent. The causal chain learner generates an action sequence based on the UI tree and instruction intent; the symbol system performs symbol system verification and permission status checks on the action sequence, corrects existing errors, and ensures that there is permission to execute the action.

3. The APP automated testing method based on multimodal perception according to claim 1 or 2, characterized in that: In step S6, the generation of action sequences is further optimized by combining the temporal memories stored in the multimodal memory pool.

4. The APP automated testing method based on multimodal perception according to claim 1, characterized in that: In step S6, after receiving a task instruction, the task instruction entry is closed. Between steps S5 and S6, the following is also included: the dynamic reasoning layer detects whether the current task instruction has been completed; if completed, the detection result is saved to the real-time knowledge base, the task instruction entry is opened, and the next task instruction is received. If not completed, continue processing the current task instruction.

5. An automated testing agent system for apps based on multimodal perception, characterized in that: This agent system is used to execute the APP automated testing method as described in any one of claims 1-4. The agent system is based on a large language model and includes a real-time perception layer, a dynamic reasoning layer, an execution optimization layer, a real-time knowledge base, and a multimodal memory pool. The real-time perception layer is used to convert real-time screenshots of the APP's screen interface into a UI tree containing hierarchical topology. The dynamic reasoning layer is used to parse task instructions through a neural-symbolic collaborative mechanism, combine them with the UI tree to generate action sequences, and then perform symbol system verification and permission status checks on the action sequences. The execution optimization layer is used to convert action sequences into platform-specific operation code and execute the operation code; The real-time knowledge base is used to store updated dynamic documents, avoidance strategies, and save test results; The multimodal memory pool is used to store the operation path and operation result of the executed operation code.

6. The APP automated testing agent system based on multimodal perception according to claim 5, characterized in that: The real-time perception layer includes a dynamic response detection module, a YOLOv5 model, a CLIP model, an OCR module, and a UI structure generator. The dynamic response detection module is used to perform difference analysis on interface changes using the SSIM algorithm. When an interface change is detected, the YOLOv5 model is used to detect control bounding boxes, the CLIP model is used to perform visual semantic encoding of interface elements, and the OCR module is used to simultaneously extract text content and generate function labels. The UI structure generator is used to fuse the output results of the three models to construct a UI tree containing hierarchical topological relationships. The dynamic reasoning layer includes a multifunctional module, a causal chain learner, a symbol system, and a conflict resolution engine. The multifunctional module is used for LLM task parsing and task status detection. The causal chain learner is used to generate action sequences based on the UI tree and task instructions. The symbol system is used to verify the action sequences. The conflict resolution engine is used to automatically mark abnormal paths, trigger path rollback, and feed back to the causal chain learner when it detects symbol system verification failure or operation result error.

Images