Large model driven cross-application function test method and system based on memory implantation

Abstract

The invention belongs to the field of cross-application function automatic testing in mobile applications, and relates to a large model driven cross-application function testing method and system based on memory implantation. The method comprises the following steps: constructing a cross-application function knowledge base for storing and organizing complete information of historical cross-App tasks; searching a historical path most similar to the current test task from the cross-application function knowledge base, reconstructing the historical path into a dialogue memory format conforming to the natural interaction habit of the large language model, and implanting the large language model; and dynamically generating a test path matched with task semantics by utilizing a large language model according to an implanted dialogue memory format, and realizing test state monitoring and feedback correction in the whole process in combination with a local state sensing mechanism. According to the method, a cross-application function knowledge base is constructed, and a global + local memory implantation mechanism is designed, so that the model has a long-term memory ability and a real-time state sensing ability, and therefore, the model can stably run in a complex scene, and a test task can be efficiently completed.

Description

Technical Field

[0001] This invention belongs to the field of cross-application functional automated testing in mobile applications, specifically involving a large model-driven cross-application functional testing method and system based on memory implantation. Background Technology

[0002] This invention focuses on the automated testing of cross-application functions in mobile applications. In the context of an increasingly open and complex software ecosystem, more and more mobile applications are no longer isolated systems, but rather co-operate with other applications to complete user tasks as part of an overall ecosystem. For example, from sharing photos from the gallery to social media platforms, completing shopping processes through third-party payment, and granting permissions by navigating to settings pages, all involve collaborative operations between multiple apps. These cross-application functions are already widespread in real-world applications; statistics show that over 87% of apps contain at least one cross-application interaction, making them an indispensable part of the mobile ecosystem.

[0003] While cross-application functionality is crucial in practical use, current automated GUI testing methods still face numerous challenges in handling this type of functionality. Traditional testing methods typically rely on predefined operational models or path generation mechanisms based on components within a single app. These methods assume the test process runs within a closed, controlled app environment, making it difficult to cover complex scenarios such as app-to-app transitions and contextual changes. Especially in cross-app functionality, the interfaces, interaction logic, and state changes of different apps are highly uncertain, making existing models prone to semantic ambiguity or misoperation during transitions, thus leading to test failures.

[0004] In recent years, Large Language Models (LLMs) have been increasingly applied to GUI testing tasks due to their powerful reasoning and language understanding capabilities. They can overcome the limitations of traditional methods to some extent by understanding interface semantics and generating operation sequences. However, existing LLM testing methods are mostly based on real-time image perception and general reasoning capabilities, lacking the ability to model cross-app semantic relationships and historical context. This makes it difficult to accurately determine the correct operation path when facing complex scenarios such as interface transitions and function shifts. Furthermore, many operations across app functions are irreversible; once a misoperation occurs, the testing process cannot be recovered, further increasing the risk of test failure.

[0005] Therefore, there is an urgent need for an automated testing method with memory and contextual reasoning capabilities, capable of simulating the thought processes of real users completing tasks across multiple apps, and effectively addressing the uncertainties and high risks in cross-app functional testing. The method and system proposed in this invention are based on this premise, innovatively introducing a "memory implantation" mechanism. This mechanism injects historical testing paths into a large language model in the form of human-like dialogue, thereby enhancing its perception, reasoning, and decision-making capabilities in cross-app scenarios, providing a novel paradigm for automated GUI testing.

[0006] The shortcomings of existing technologies are as follows:

[0007] 1) Lack of cross-App semantic modeling capabilities leads to low path reasoning accuracy: Existing automated GUI testing methods based on large language models mostly rely on current page screenshots and general natural language reasoning capabilities for decision-making, without considering the task context relationships between multiple applications. In cross-App scenarios, due to the lack of direct semantic connections between different Apps, traditional models struggle to accurately determine the component corresponding to the target operation. This is especially true when the semantics of components in intermediate App interfaces are ambiguous, making it prone to misoperations, leading to test path deviations or even failures.

[0008] 2) Weak perception of state changes and poor fault tolerance: Cross-application operations often have the characteristic of being irreversible once executed, such as payments, permission authorization, and account switching. If a step in the testing process is judged incorrectly, the entire path will often enter an unrecoverable error state. However, current testing methods lack mechanisms for tracking and remembering the operation state, making it impossible to detect in real time whether the test path deviates from expectations, and it is also difficult to adjust strategies in a timely manner. This results in insufficient fault tolerance, seriously affecting test stability and coverage completeness.

[0009] 3) Lack of test context and inability to reuse existing experience: Existing automated GUI testing methods are usually based on single-inference, lacking a structured representation and memory storage mechanism for historical testing experience, making it impossible to reuse existing successful paths in similar scenarios. This means that even when facing similar cross-app functions (such as sharing, login, payment, etc.), the testing system still needs to perform reasoning "from scratch," wasting a lot of computing resources and easily repeating the same false positives, reducing overall testing efficiency.

[0010] 4) Lack of dynamic monitoring mechanisms makes it difficult to ensure the completion of test tasks: Traditional testing methods often rely on fixed paths or task templates, making it difficult to cope with dynamic changes that occur during execution, such as interface loading failures, authorization pop-up interference, and incomplete redirects. The lack of real-time status monitoring and feedback capabilities leads to uncontrollable task completion status during testing, posing a potential risk of "incomplete testing or misjudged success." Summary of the Invention

[0011] This invention aims to address key challenges in current automated GUI testing across application functional scenarios, namely, improving the accuracy of path reasoning, enhancing the stability of the testing process, and achieving test universality across multiple scenarios. To this end, this invention proposes a memory-implanted, large language model-driven testing method and system. It utilizes structured historical cross-App functional testing experience to construct a "memorable and reasonable" test context, guiding the model to make accurate decisions in multi-App collaborative scenarios and generate test paths that conform to business logic.

[0012] The technical solution adopted in this invention is as follows:

[0013] A large-scale model-driven cross-application functional testing method based on memory implantation includes the following steps:

[0014] Build a cross-application functional knowledge base to store and organize complete information on historical cross-app tasks;

[0015] Retrieve the historical path most similar to the current test task from the cross-application functional knowledge base, reconstruct it into a dialogue memory format that conforms to the natural interaction habits of the large language model, and then embed it into the large language model;

[0016] Based on the implanted dialogue memory format, a test path that matches the task semantics is dynamically generated using a large language model, and a local state awareness mechanism is combined to achieve full-process test state monitoring and feedback correction.

[0017] Furthermore, the construction of the cross-application functional knowledge base includes:

[0018] Extract complete cross-App interaction information from historical test data or open-source mobile applications, including information from multiple dimensions such as static structure, dynamic behavior, and visual semantics;

[0019] The extracted information is stored in a unified structure and standardized and vectorized through multimodal coding technology to form a cross-application functional knowledge base.

[0020] Furthermore, the extraction of complete cross-App interaction information from historical test data or open-source mobile applications includes: extraction of basic App information, acquisition of interface screenshots and component hierarchy information, extraction of operation paths and function descriptions, annotation of multi-App jump relationships, and annotation of function categories.

[0021] Furthermore, the extraction of information is stored in a unified structure and standardized and indexed using multimodal coding techniques, including:

[0022] Input GUI screenshots of the start page, key navigation pages, and target pages in cross-App tasks into the image encoder to capture page structure and visual style features;

[0023] Input the functional description, component attributes, and interaction sequence into the language model encoder to extract semantic-level functional intent information;

[0024] By using a projection layer to map visual and linguistic features to a shared space, a unified cross-app functional embedding representation is output.

[0025] Furthermore, the step of retrieving the historical path most similar to the current test task from the cross-application functional knowledge base, reconstructing it into a dialogue memory format that conforms to the natural interaction habits of the large language model, and embedding it into the large language model includes:

[0026] The current App's GUI page screenshot is input into the image encoder to extract its layout structure, visual hierarchy, and spatial arrangement information; the current App's component attributes are input into the language model encoder to extract the semantic layer's component functional features; the visual features and language features are projected onto a unified embedding space through the fusion layer to generate the current GUI page's multimodal representation vector.

[0027] Using multimodal representation vectors as query keys for functional retrieval, a two-stage similarity retrieval strategy is employed to search for historical paths with similar functional semantics and structure in a cross-application knowledge base. The two-stage similarity retrieval strategy includes: in the first stage, based on the cosine similarity of embedded vectors, rapid matching of the starting page vectors of all paths in the cross-application knowledge base is performed to obtain the Top-K candidate functional paths as preliminary retrieval results; in the second stage, a fine-grained semantic re-ranking mechanism based on a cross-encoder is used to concatenate each candidate functional path with the component information and screenshot information of the current page into an input pair, which is then input into a pre-trained visual-language contrastive model to deeply analyze the contextual consistency and semantic coherence between the input pairs; finally, the Top-K paths are re-ranked based on the semantic matching score, and the most relevant historical paths are selected for subsequent memory construction.

[0028] The retrieved historical function paths are transformed into conversational memory inputs that are "understandable and reusable" for the model. This includes two memory forms: global memory and local memory. Global memory is used to simulate the model's past experience with similar tasks that it has "already completed." The entire function path is reconstructed into a conversational format and input into the model context, enabling the model to "recall" previously completed similar tasks and thus understand the current goal and path. Local memory is used to record real-time state information during the test and is dynamically updated as the test is executed. This helps the model maintain its understanding of task continuity and avoids repeated clicks, accidental touches of controls, or path interruptions.

[0029] Furthermore, the step of dynamically generating test paths that match the task semantics using a large language model based on the implanted dialogue memory format, and combining this with a local state awareness mechanism to achieve full-process test state monitoring and feedback correction, includes:

[0030] The test page is automatically annotated, and all interactive controls are automatically identified. Based on the physical location, component type, and hierarchical structure information of the control on the page, a unique number is generated for each operable component, and the corresponding GUI screenshot is annotated to form an annotated screenshot. The annotated screenshot, component attribute information, current page description, local memory state, and global memory content are organized together into a multimodal input, which is then passed into the multimodal large language model as context information.

[0031] After acquiring complete contextual information, the multimodal big oracle model generates the test action to be executed based on the current page state and historical task memory, and converts it into specific touch operations. These operations are then sent to the tested app for execution via automated instructions. After the operation is completed, a new page screenshot and control structure are captured, and it is determined whether app jumps, permission pop-ups, or state switching behaviors have occurred. If the new page deviates significantly from the path recorded in memory or fails to achieve the expected goal, the local memory is updated and the input is reorganized. This prompts the multimodal big oracle model to regenerate the operation strategy and automatically correct the path.

[0032] Set up a test status monitor to track the current test progress throughout the entire process and perform semantic-level completion checks.

[0033] Furthermore, the test status monitor is automatically triggered at two key nodes: first, after an app-to-app transition occurs; and second, when the model determines that the task is complete and attempts to return to the original app. At this time, a multimodal comparison is performed between the current local path and the remembered "standard path," including: visual similarity between the current page screenshot and the remembered page screenshot, operational consistency between control operation sequences, and semantic alignment between the functional goal description and the current state. If the monitor determines that the current path has successfully completed the functional goal, it marks the current task as "completed." If it determines that the current state has not yet covered all necessary steps, it generates a semantic feedback prompt and re-guides the model to perform supplementary operations until the completion judgment criteria are met. The test status monitor is also responsible for identifying typical abnormal situations and triggering a correction mechanism.

[0034] A large-scale model-driven cross-application functional testing system based on memory implantation includes:

[0035] The cross-application functional knowledge base building module is used to build a cross-application functional knowledge base for storing and organizing complete information on historical cross-App tasks;

[0036] The similar function retrieval and memory implantation module is used to retrieve the most similar historical path to the current test task from the cross-application function knowledge base, reconstruct it into a dialogue memory format that conforms to the natural interaction habits of the large language model, and implant it into the large language model.

[0037] The test path generation and status monitoring module is used to dynamically generate test paths that match the task semantics based on the embedded dialogue memory format using a large language model, and to achieve full-process test status monitoring and feedback correction by combining a local state awareness mechanism.

[0038] The beneficial effects of this invention are as follows:

[0039] This invention constructs a multimodal, cross-app functional knowledge base, integrating information such as interface structure, text descriptions, and component operation sequences. It also designs a novel "global + local" memory implantation mechanism, enabling the model to possess long-term memory capabilities and real-time state awareness. This allows for robust operation even in complex scenarios such as ambiguous semantics of operational components, irreversible paths, or abnormal states. Furthermore, the system design supports automatic startup without task descriptions and features a highly versatile interface coding and similar function retrieval mechanism. It is applicable to various types of mobile apps and multiple cross-application interaction tasks, efficiently completing testing tasks without the need for additional manual annotation.

[0040] Compared to existing automated GUI testing technologies, this invention offers significant advantages in test path reasoning capabilities, execution robustness, functional coverage breadth, and cross-task versatility. It effectively addresses the inefficiencies, low accuracy, and susceptibility to interruptions faced by traditional testing methods in complex cross-App scenarios. Through these innovative technologies, this invention effectively improves the coverage and accuracy of cross-App automated testing, enhances the model's decision robustness and context adaptability, and provides new technical support for achieving highly reliable, low-cost, and low-intervention mobile software testing. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method of the present invention.

[0042] Figure 2 This is a Prompt example diagram of the method of the present invention. Detailed Implementation

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0044] The main contents and innovations of this invention are as follows:

[0045] 1) Memory-Impregnated Testing Enhancement Mechanism. This invention proposes a novel memory enhancement technology for large language models, breaking through the traditional single-round perceptual testing method. By constructing a "global memory" and "local memory" structure, historical cross-App functional paths are implanted into the LLM in a dialogue-like manner, giving it a human-like ability to recall task experience. The global memory simulates the process of the model having completed similar functional tests, providing prior guidance for the current task; the local memory dynamically records the state changes at each step of the test, achieving continuous awareness of the context. This innovation significantly improves the model's continuous reasoning ability and fault tolerance in cross-App functional testing, effectively avoiding operational ambiguity and path deviation.

[0046] 2) Multimodal Functional Knowledge Base Construction and Semantic Retrieval Mechanism. Addressing the semantic ambiguity and component heterogeneity issues across application functions, this invention designs a knowledge base construction method that integrates visual and structural information. By extracting multimodal content such as GUI screenshots, component hierarchy information, and operation steps, a unified encoding is generated to produce an embedded representation, which is then stored in a cross-App functional knowledge base. During testing, embedding similarity is used to quickly retrieve historical test cases with similar structures and functions. This is then combined with a semantic reordering model (such as a cross-encoder) for refined matching, achieving accurate alignment of the current test task and providing strong semantic support for subsequent path generation.

[0047] 3) Dynamic Path Generation Mechanism Driven by Test Monitoring and Feedback. To ensure the continuity and integrity of test execution, this invention proposes a dynamic path monitoring mechanism based on semantic reasoning. During model execution testing, the system continuously compares the text and image information of the current page state with that of the historical path to determine whether the test has deviated or terminated prematurely. Once a deviation is detected, the system will immediately generate feedback and adjust the model strategy to restore the correct path, avoiding test interruptions or incorrect result reports due to interface jump failures, component anomalies, etc. This mechanism significantly improves the stability and reliability of cross-App automated testing.

[0048] This invention proposes a method and system for cross-application functional testing based on memory implantation using a large language model. The method mainly consists of three steps: a cross-application functional knowledge base construction step, a similar function retrieval and memory implantation step, and a test path generation and status monitoring step. Correspondingly, the system mainly consists of three modules: a cross-application functional knowledge base construction module, a similar function retrieval and memory implantation module, and a test path generation and status monitoring module. The overall structure diagram is shown below. Figure 1 As shown, the prompt example used for each step / module is as follows: Figure 2 As shown below, the structure and workflow of each step will be explained in detail.

[0049] Step 1: Building a Cross-Application Functional Knowledge Base

[0050] To enable the large language model to acquire rich historical testing experience and use it to guide current tasks, this invention first constructs a structured and searchable cross-application functional knowledge base. This knowledge base stores and organizes complete information on historical cross-app tasks, including the interaction flow between the starting app and the target app, GUI screenshots, component hierarchy, interactive controls, operation logs, etc., and generates a unified representation through multimodal encoding for subsequent retrieval of similar tasks and invocation by the large language model.

[0051] 1) Process 1: Information Extraction

[0052] The first step in building a cross-app functional knowledge base is to extract complete cross-app interaction information from historical test data or open-source mobile applications. This information includes multiple dimensions such as static structure, dynamic behavior, and visual semantics, and is an important foundation for subsequent memory structure building and path transfer reasoning.

[0053] App basic information extraction: Extract the app package name, version number, declared Activities, and the jump relationships between Activities triggered by intents from the application's AndroidManifest.xml. Using static analysis tools, parse intent-filter tags, call graphs, and implicit / explicit intents to construct the initial Activity Transition Graph (ATG) within the app to represent the app's internal UI structure.

[0054] Interface screenshots and component hierarchy information acquisition: GUI screenshots are captured at each step of the historical test path, and the corresponding view hierarchy files are exported using tools such as UIAutomator. This file provides attribute information for each component (including ID, type, hierarchy, interactivity, text content, and position coordinates) to reconstruct the complete visual interaction state. All interactive controls are automatically identified and labeled so that subsequent models can accurately identify the objects to be manipulated.

[0055] Operation path and function description extraction: Operation logs from historical test tasks are extracted and organized into reproducible interaction sequences, such as "Click button A → Select image → Redirect to Gmail → Confirm send," etc. Simultaneously, through a combination of rules and manual verification, natural language descriptions for cross-app functions are extracted and standardized, such as "Share image from gallery to Gmail." This facilitates the subsequent construction of a searchable functional semantic index.

[0056] Multi-App Navigation Relationship Labeling: For test processes involving navigation between multiple apps, explicitly label each step that crosses application boundaries, including information such as "App status before navigation," "intermediate App name," and "initial page after navigation." Also, record the first operable page after the navigation (such as permission authorization, account selection, etc.) as a key anchor point in the functional path.

[0057] Functional Classification and Labeling: Combining manual labeling and rule templates, different cross-App functions are divided into several categories, such as "content sharing", "third-party login / authorization", "system permission jump", "payment function", "data synchronization", etc., to facilitate similarity modeling and generalization transfer for subsequent tasks.

[0058] 2) Process 2: Knowledge Base Construction

[0059] The cross-App functional knowledge base stores the extracted information in a unified structure and uses multimodal coding technology for standardized representation and vectorized indexing, making it suitable for efficient retrieval enhancement testing tasks. A graph-text fusion coding model (such as GME) is used to represent each cross-App test path as a unified multimodal vector. Specifically, this includes:

[0060] Visual encoding: Input GUI screenshots of the start page, key navigation pages, and target pages in cross-App tasks into the image encoder to capture page structure and visual style features.

[0061] Text encoding: Input the function description, component attributes, and interaction sequence into the language model encoder to extract semantic-level functional intent information.

[0062] Structural Fusion: Visual and linguistic features are mapped to a shared space through a projection layer, outputting a unified cross-App function embedding representation for similar path retrieval. Each function path is ultimately stored as a structure, including: function name, function category, function starting page ID, function embedding vector, screenshot path, component tree, interaction log, jump chain, target App name, and other fields.

[0063] Step Two: Similar Function Retrieval and Memory Implantation

[0064] To enable large language models to possess real-world experience memory and context awareness when performing cross-App functional tests, this invention designs an enhancement mechanism of "similar function retrieval + structured memory implantation." This mechanism retrieves the most similar historical paths to the current test task from the cross-App functional knowledge base and reconstructs them into a dialogue memory format that conforms to the natural interaction habits of the large language model. This allows the model to perceive that it has "performed similar functions before," thereby effectively improving the accuracy and stability of cross-App path reasoning.

[0065] 1) Process 1: Multimodal encoding of the current page

[0066] At the start of the test, the system first extracts visual and structural information from the current app's GUI page. Specifically, the system obtains a screenshot of the page and its corresponding view hierarchy file. Using a graph encoding model (such as GME), the page screenshot is input into an image encoder to extract information such as its layout structure, visual hierarchy, and spatial arrangement. Simultaneously, component attributes extracted from the view hierarchy (such as control type, text, and hierarchical relationship) are input into a language model encoder to extract semantic component functional features. Subsequently, a fusion layer projects the visual and linguistic features onto a unified embedding space to generate a multimodal representation vector of the current GUI page. This vector will serve as the query key for function retrieval, used to find historical paths with similar functional semantics and structure in the knowledge base.

[0067] 2) Process 2: Two-stage retrieval of similar functional paths

[0068] To ensure that the retrieved functional paths possess both structural similarity and semantic relevance, this invention designs a two-stage similarity retrieval strategy. In the first stage, the system quickly matches the starting page vectors of all paths in the knowledge base based on the cosine similarity of the embedded vectors, obtaining the Top-K candidate functional paths as preliminary retrieval results. In the second stage, to further improve accuracy, the system introduces a refined semantic re-ranking mechanism based on a cross-encoder. This mechanism concatenates each candidate functional path with the component information and screenshot information of the current page as input pairs, inputting them into a pre-trained visual-language contrastive model to deeply analyze their contextual consistency and semantic coherence. Finally, the Top-K paths are re-ranked based on this semantic matching score, selecting the most relevant historical paths for subsequent memory construction.

[0069] 3) Process 3: Generation and Implantation of Structured Memories

[0070] To enable large language models to sustainably understand test tasks, this invention proposes a structured memory implantation mechanism that transforms retrieved historical functional paths into "understandable and reusable" conversational memory inputs for the model. This mechanism comprises two memory forms: global memory and local memory. Global memory simulates the model's past experiences with similar tasks, reconstructing the entire functional path into a dialogue-like format, including user task intent (i.e., function description), key page screenshots (labeled with component numbers), and each step of the operation and its interface state changes. This content is organized into an interactive dialogue similar to a "user-assistant," input into the model's context, enabling the model to "recall" previously completed similar tasks, thereby understanding the current goal and path. Local memory records real-time state information during testing, including visited pages, clicked components, whether a jump occurred, the target app, and the operation history in the current state. This information is dynamically updated as the test executes and embedded into the model as input, helping the model maintain its understanding of task continuity and avoiding repeated clicks, accidental touches of controls, or path interruptions.

[0071] Step 3: Test Path Generation and Status Monitoring

[0072] After acquiring the relevant memories, the system enters the automatic path generation stage. Guided by the Multimodal Large Language Model (MLLM), it dynamically generates test paths that match the semantics of the task. Combined with the local state awareness mechanism, it realizes full-process test state monitoring and feedback correction to ensure the correctness, integrity and robustness of the test paths.

[0073] 1) Process 1: Automatic annotation and input preprocessing of the test page

[0074] To enable MLLM to accurately identify interactive controls on the current page and make appropriate operation choices, the system first automatically annotates the test page. Specifically, the system uses tools such as UIAutomator to extract the view hierarchy structure of the current app page and automatically identifies all interactive controls (including buttons, text input boxes, image pickers, permission confirmation boxes, etc.). Based on the physical location, component type, and hierarchy of the control on the page, the system generates a unique number for each operable component and annotates it on the corresponding GUI screenshot, forming an "annotated screenshot." Subsequently, the system organizes this annotated screenshot, component attribute information, current page description, local memory state, and global memory content into a multimodal input, which is passed as a context prefix to MLLM to guide it in generating the specific operation to be performed in the current step.

[0075] 2) Process 2: Path Decision Generation and Dynamic Adjustment

[0076] After acquiring complete context information, MLLM generates the test action to be executed based on the current page state and historical task memory. The generated result is usually in the form of "click control number 3" or "swipe to the bottom and click continue". The system converts this result into a specific touch operation and sends it to the tested app for execution via Android automation instructions. After the operation is executed, the system immediately captures a new page screenshot and control structure, and determines whether there have been any app jumps, permission pop-ups, state transitions, or other behaviors. If the new page deviates significantly from the path recorded in memory or fails to achieve the expected goal, the system will update the local memory and reorganize the input, prompting MLLM to regenerate the operation strategy and automatically correct the path. For example, when jumping to Gmail and not seeing the email form, the system will backtrack to determine if there are any additional authorization operations that have not been completed, and guide the model back to the previous step to complete the operation. This mechanism gives the test path good adaptability and can flexibly cope with heterogeneous interaction logic and interruption factors between apps.

[0077] 3) Process 3: Status monitoring and task completion assessment

[0078] To ensure the achievement of cross-App functional testing objectives, this invention introduces a dedicated "test status monitor" to track the current test progress throughout the entire process and perform semantic-level completion judgments. The monitor automatically triggers at two key nodes: first, after an inter-App jump occurs; and second, when the model determines the task is complete and attempts to return to the original App. At this time, the system performs a multimodal comparison between the current local path and the remembered "standard path," including: visual similarity between the current page screenshot and the remembered page screenshot, operational consistency between control operation sequences, and semantic alignment between the functional goal description and the current state. If the monitor determines that the current path has successfully completed the functional goal (e.g., content has been successfully shared to the target App and returned to the original App), the current task is marked as "completed." If the monitor determines that the current state has not covered all necessary steps (e.g., not yet submitted, still stuck in an intermediate App), semantic feedback is generated to guide the model to perform supplementary operations until the completion judgment criteria are met.

[0079] In addition, the monitor is also responsible for identifying typical anomalies, such as getting stuck on the authorization page, accidentally jumping back to the starting page, or entering the wrong third-party app. These can trigger a correction mechanism to prevent path interruption or false success. This "semantic-driven + context-monitoring" mechanism greatly improves the robustness and task closure capability of cross-app testing.

[0080] The present invention proposes a cross-application functional automated testing method and system based on a large language model driven by memory implantation, called InterDroid. Compared with existing automated GUI testing technologies, it has significant advantages in terms of test path reasoning ability, execution robustness, functional coverage, and cross-task migration versatility. It can effectively solve the problems of low efficiency, low accuracy, and easy interruption faced by traditional testing methods in complex cross-App scenarios.

[0081] 1) Improve test path reasoning ability, significantly improve accuracy and coverage.

[0082] This invention constructs a structured cross-application functional knowledge base and designs a dual global and local memory mechanism, endowing the large language model with "experience-driven" reasoning capabilities. This allows it to deduce complete paths consistent with business logic even when facing scenarios involving transitions between different apps, ambiguous component semantics, or complex state changes. According to experimental data, InterDroid achieved an average page coverage of 0.84 and an operation coverage of 0.85 in 100 real-world cross-app functional path tests, both significantly exceeding the current best baseline method (VisionDroid) at 0.36 and 0.38, respectively. It also improved Exact Match accuracy by 268%, demonstrating superior semantic understanding and path reasoning capabilities.

[0083] 2) Enhance the stability and fault tolerance of the testing process, and support interruption recovery and path correction.

[0084] Since cross-app testing processes typically involve a series of irreversible operations, such as permission redirection, account authorization, and payment processes, misoperation can lead to test failure and task interruption. To address this, this invention designs a dynamic local memory tracking mechanism and a test state monitor. During path execution, the system continuously records the context state and performs semantic comparison and deviation detection on the results of each step. When path deviation, execution interruption, or incompleteness is detected, the system promptly adjusts the testing strategy and supplements the missing steps, significantly improving test stability and fault tolerance. In experiments, InterDroid's average test time is only 7.1 minutes, significantly better than similar LLM methods (such as GPTDroid's average of 17.3 minutes), demonstrating superior test loop efficiency.

[0085] 3) It achieves broad compatibility across functions and apps, and has good versatility.

[0086] The functional knowledge base constructed in this invention adopts a standardized multimodal coding method that integrates text and graphics, enabling historical paths to be migrated and reused across different apps. Whether it's social sharing, third-party login, system settings navigation, or data synchronization, as long as the task logic is similar, they can be migrated and adapted through a similar function retrieval and memory implantation mechanism, eliminating the need for repetitive training or manual script design, fundamentally breaking through the strong dependence of traditional methods on specific apps. Even in new app scenarios not seen in the knowledge base, InterDroid maintains stable inference performance, demonstrating high task generalization ability and platform adaptability.

[0087] 4) Possesses practical defect detection capabilities and is suitable for industrial-grade testing tasks.

[0088] This invention not only performed exceptionally well in performance metrics tests but also demonstrated outstanding defect detection capabilities in real-world apps. In testing 180 real-world apps available on Google Play, InterDroid discovered 43 previously undetected crash-related defects. Of these, 31 have been fixed by developers, and 12 were officially confirmed as valid issues; none were disproven. This result demonstrates that this invention possesses powerful practical problem detection capabilities and is a cutting-edge intelligent technology truly applicable to real-world industrial testing processes.

[0089] In summary, this invention comprehensively improves the performance and applicability of cross-App automated testing from multiple dimensions, including path semantic reasoning, test process robustness, cross-application adaptability, and intelligent testing efficiency. It can provide efficient, intelligent, and low-cost automated testing solutions for mobile software development, and has significant industrial application value and promising prospects for technology promotion.

[0090] Another embodiment of the present invention provides a large-scale model-driven cross-application functional testing system based on memory implantation, comprising:

[0091] The cross-application functional knowledge base building module is used to build a cross-application functional knowledge base for storing and organizing complete information on historical cross-App tasks;

[0092] The similar function retrieval and memory implantation module is used to retrieve the most similar historical path to the current test task from the cross-application function knowledge base, reconstruct it into a dialogue memory format that conforms to the natural interaction habits of the large language model, and implant it into the large language model.

[0093] The test path generation and status monitoring module is used to dynamically generate test paths that match the task semantics based on the embedded dialogue memory format using a large language model, and to achieve full-process test status monitoring and feedback correction by combining a local state awareness mechanism.

[0094] The above division of modules is merely illustrative. In practical applications, the functions described above can be assigned to different functional modules as needed to complete all or part of the functions described in the aforementioned method. The specific working process of each module can be found in the corresponding process in the aforementioned method embodiments, and will not be repeated here.

[0095] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing steps of the method of the present invention.

[0096] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) that stores a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0097] Another embodiment of the present invention provides a computer program product, the computer program product including a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0098] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A method for cross-application functional testing based on a large model driven by memory implantation, characterized in that, Includes the following steps: Build a cross-application functional knowledge base to store and organize complete information on historical cross-app tasks; Retrieve the historical path most similar to the current test task from the cross-application functional knowledge base, reconstruct it into a dialogue memory format that conforms to the natural interaction habits of the large language model, and then embed it into the large language model; Based on the implanted dialogue memory format, a test path that matches the task semantics is dynamically generated using a large language model, and a local state awareness mechanism is combined to achieve full-process test state monitoring and feedback correction.

2. The method according to claim 1, characterized in that, The construction of the cross-application functional knowledge base includes: Extract complete cross-App interaction information from historical test data or open-source mobile applications, including information from multiple dimensions such as static structure, dynamic behavior, and visual semantics; The extracted information is stored in a unified structure and standardized and vectorized through multimodal coding technology to form a cross-application functional knowledge base.

3. The method according to claim 2, characterized in that, The process of extracting complete cross-App interaction information from historical test data or open-source mobile applications includes: extraction of basic App information, acquisition of interface screenshots and component hierarchy information, extraction of operation paths and function descriptions, annotation of multi-App jump relationships, and annotation of function categories.

4. The method according to claim 2, characterized in that, The process of storing extracted information in a unified structure and performing standardized representation and vectorized indexing using multimodal coding technology includes: Input GUI screenshots of the start page, key navigation pages, and target pages in cross-App tasks into the image encoder to capture page structure and visual style features; Input the functional description, component attributes, and interaction sequence into the language model encoder to extract semantic-level functional intent information; By using a projection layer to map visual and linguistic features to a shared space, a unified cross-app functional embedding representation is output.

5. The method according to claim 1, characterized in that, The process of retrieving the historical path most similar to the current test task from the cross-application functional knowledge base, reconstructing it into a dialogue memory format that conforms to the natural interaction habits of the large language model, and embedding it into the large language model includes: The current App's GUI page screenshot is input into the image encoder to extract its layout structure, visual hierarchy, and spatial arrangement information; the current App's component attributes are input into the language model encoder to extract the semantic layer's component functional features; the visual features and language features are projected onto a unified embedding space through the fusion layer to generate the current GUI page's multimodal representation vector. Using multimodal representation vectors as query keys for functional retrieval, a two-stage similarity retrieval strategy is employed to search for historical paths with similar functional semantics and structure in a cross-application knowledge base. The two-stage similarity retrieval strategy includes: in the first stage, based on the cosine similarity of embedded vectors, rapid matching of the starting page vectors of all paths in the cross-application knowledge base is performed to obtain the Top-K candidate functional paths as preliminary retrieval results; in the second stage, a fine-grained semantic re-ranking mechanism based on a cross-encoder is used to concatenate each candidate functional path with the component information and screenshot information of the current page into an input pair, which is then input into a pre-trained visual-language contrastive model to deeply analyze the contextual consistency and semantic coherence between the input pairs; finally, the Top-K paths are re-ranked based on the semantic matching score, and the most relevant historical paths are selected for subsequent memory construction. The retrieved historical function paths are transformed into conversational memory inputs that are "understandable and reusable" for the model. This includes two memory forms: global memory and local memory. Global memory is used to simulate the model's past experience with similar tasks that it has "already completed." The entire function path is reconstructed into a conversational format and input into the model context, enabling the model to "recall" previously completed similar tasks and thus understand the current goal and path. Local memory is used to record real-time state information during the test and is dynamically updated as the test is executed. This helps the model maintain its understanding of task continuity and avoids repeated clicks, accidental touches of controls, or path interruptions.

6. The method according to claim 1, characterized in that, The process involves dynamically generating test paths that match the task's semantics using a large language model based on the implanted dialogue memory format, and combining this with a local state awareness mechanism to monitor and correct the test state throughout the entire process. This includes: The test page is automatically annotated, and all interactive controls are automatically identified. Based on the physical location, component type, and hierarchical structure information of the control on the page, a unique number is generated for each operable component, and the corresponding GUI screenshot is annotated to form an annotated screenshot. The annotated screenshot, component attribute information, current page description, local memory state, and global memory content are organized together into a multimodal input, which is then passed into the multimodal large language model as context information. After acquiring complete contextual information, the multimodal big oracle model generates the test action to be executed based on the current page state and historical task memory, and converts it into specific touch operations. These operations are then sent to the tested app for execution via automated instructions. After the operation is completed, a new page screenshot and control structure are captured, and it is determined whether app jumps, permission pop-ups, or state switching behaviors have occurred. If the new page deviates significantly from the path recorded in memory or fails to achieve the expected goal, the local memory is updated and the input is reorganized. This prompts the multimodal big oracle model to regenerate the operation strategy and automatically correct the path. Set up a test status monitor to track the current test progress throughout the entire process and perform semantic-level completion checks.

7. The method according to claim 1, characterized in that, The test status monitor is automatically triggered at two key nodes: first, after an app-to-app transition occurs; and second, when the model determines that the task is complete and attempts to return to the original app. At this time, a multimodal comparison is performed between the current local path and the remembered "standard path," including: visual similarity between the current page screenshot and the remembered page screenshot, operational consistency between control operation sequences, and semantic alignment between the functional goal description and the current state. If the monitor determines that the current path has successfully completed the functional goal, it marks the current task as "completed." If it determines that the current state has not yet covered all necessary steps, it generates a semantic feedback prompt and re-guides the model to perform supplementary operations until the completion judgment criteria are met. The test status monitor is also responsible for identifying typical abnormal situations and triggering correction mechanisms.

8. A large-scale model-driven cross-application functional testing system based on memory implantation, characterized in that, include: The cross-application functional knowledge base building module is used to build a cross-application functional knowledge base for storing and organizing complete information on historical cross-App tasks; The similar function retrieval and memory implantation module is used to retrieve the most similar historical path to the current test task from the cross-application function knowledge base, reconstruct it into a dialogue memory format that conforms to the natural interaction habits of the large language model, and implant it into the large language model. The test path generation and status monitoring module is used to dynamically generate test paths that match the task semantics based on the embedded dialogue memory format using a large language model, and to achieve full-process test status monitoring and feedback correction by combining a local state awareness mechanism.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.

Images