Graphical user interface forensic analysis method based on multi-modal large language model

Abstract

The invention provides a graphical user interface forensic analysis method based on a multi-mode large language model, and belongs to the technical field of man-machine interaction, artificial intelligence safety and digital forensic. The method comprises the steps of evidence capture and distillation, semantic analysis and record generation, storage and indexing, and retrieval and inquiry. Through evidence capture and distillation, on the premise that evidence integrity is guaranteed, the visual data volume needing to be analyzed is greatly reduced, and long-term and efficient evidence obtaining analysis on a mobile terminal becomes possible; by introducing the analysis capability of a multi-modal large language model, original and structurality-free visual evidence is converted into a structuralization intelligent evidence which can be understood and deeply excavated; a designed two-stage natural language evidence obtaining query engine supports a complete analysis process from macroscopic semantic retrieval to microscopic detail inquiry; the problems that a traditional evidence obtaining method depends on manual troubleshooting and is low in efficiency are solved, and unprecedented high efficiency and convenience are provided for backtracking and auditing of mass interaction records.

Description

TECHNICAL FIELD

[0001] The present application provides a method for forensics analysis of graphical user interface (GUI) interaction on intelligent terminals, specifically relates to a technology for converting visual event streams distilled by intelligent distillation into structured semantic provenance records using multi-modal large language models, and belongs to the fields of human-computer interaction, artificial intelligence security and digital forensics. BACKGROUND

[0002] In today's intelligent terminal environment, in order to conduct operation audit or user behavior analysis, recording and forensics of graphical user interface (GUI) interaction process has become an important technical requirement. Such technology aims to obtain a series of operations of the user on the device screen and provide a basis for subsequent analysis.

[0003] Currently, the mainstream technical path to achieve this goal includes relying on application built-in log recording and using continuous screen recording. Although the built-in log of the application (such as browsing history, payment record) has low overhead, its main disadvantage is the fragmentation of data and the lack of context. Forensic personnel must manually integrate scattered records from different applications with different formats, which is tedious and unreliable. More importantly, these text logs cannot provide rich visual context when the operation occurs, making it difficult to accurately restore the user's true intention, and these records are extremely easy to tamper with or delete, seriously affecting the integrity of the evidence. Another method, continuous screen recording, can capture all visual information completely, but its fundamental defect is the high resource consumption. Long-term, high-frame-rate screen recording on mobile terminals will generate a huge amount of data storage and significantly increase the power consumption and heat of the device, which is not sustainable for a forensics system that needs to run continuously. Even the most advanced desktop screen recording system has very high requirements for hardware and cannot be directly applied to mainstream mobile devices. In addition, regardless of which recording method, the existing technology lacks deep semantic understanding of the interaction sequence. They either only provide isolated log entries or only provide raw video streams, and cannot automatically convert a series of user operations (such as a series of clicks and swipes) into an event summary with clear logic and intention that is understandable by humans.

[0004] Therefore, there is an urgent need for a new technical solution that can capture rich GUI interaction evidence on resource-constrained mobile devices in a low-overhead manner and automatically perform deep semantic analysis on these evidence to generate structured forensics records that can be efficiently queried. SUMMARY

[0005] The purpose of the present application is to provide a method for efficient forensic analysis of graphical user interface (GUI) interaction on a smart terminal, aiming to solve the problem of forensic difficulty caused by data fragmentation, context loss and resource limitation of mobile terminals in the prior art. By introducing an innovative evidence distillation combined with multi-modal semantic analysis technical solution, the present application can create interactive traceability records rich in semantics and efficiently queryable under the premise of protecting user privacy and low resource consumption.

[0006] The technical solution of the present application is as follows:

[0007] A graphical user interface forensic analysis method based on a multi-modal large language model, for use on a smart terminal, the smart terminal integrating a lightweight visual model and a multi-modal large language model (MLLM), and a database for storage and query, the method comprising the following steps:

[0008] Evidence capture and distillation phase:

[0009] A. Adaptive capture of interaction events: In the evidence capture phase, first monitor user interaction in a low-power mode, and only collect data at key moments in high density to obtain a set of original candidate evidence;

[0010] B. Optimization of visual evidence distillation: Through multi-level redundant data elimination, the highest information density core visual evidence sequence is extracted;

[0011] Semantic analysis and record generation phase:

[0012] C. Multi-modal semantic analysis of visual evidence sequence: The condensed visual evidence sequence is converted into structured, human-readable traceability records using a multi-modal large language model MLLM;

[0013] Storage and indexing phase:

[0014] D. Storage and indexing of traceability records: Based on the generated structured traceability record data, a hybrid storage mechanism is introduced to uniformly manage multi-modal evidence and store it in the database of the smart terminal, and a text embedding model and database vector index are constructed;

[0015] Retrieval and interrogation phase:

[0016] E. Real-time forensic query and deep interrogation: To meet different levels of forensic needs, a two-stage natural language forensic query engine is designed to support a complete analysis process from macroscopic retrieval to microscopic interrogation, realizing graphical user interface forensic analysis.

[0017] Further, the A adaptive capture of interaction events of the evidence capture and distillation phase specifically comprises:

[0018] A1. Key event trigger: the terminal device is in a standby state with almost zero power consumption by default, and only when the user's key GUI operation is monitored, the event capture process is triggered;

[0019] A2. Time window sampling: once the user's key GUI operation is monitored, the fragments are captured within the preset time window after the key GUI operation at the preset sampling frequency to form the original candidate evidence set containing context information.

[0020] Further, the key GUI operation in A1 includes screen clicking, sliding, and inputting.

[0021] Further, the evidence capture and distillation phase B optimizes the distillation of visual evidence, specifically including:

[0022] B1. Inter-frame redundancy elimination: a fast redundancy elimination technique based on pixel similarity is applied to calculate the pixel difference between adjacent frames in the original candidate evidence set, and according to the set pixel difference threshold, static or repeated frames with no significant changes in content are discarded;

[0023] B2. Intra-frame semantic information redundancy compression: after redundancy elimination, each frame of the video frame is divided into multiple patches according to spatial segmentation, and a lightweight visual model is used to extract the features of each patch. By comparing the feature similarity of the patches at the same spatial position in consecutive frames, only those patches with significant semantic changes are retained according to the set semantic similarity threshold. Through this step, a core visual evidence sequence composed of a series of key patches is finally generated.

[0024] Further, the comparison of the feature similarity of the patches at the same spatial position in consecutive frames in B2 is cosine similarity.

[0025] Further, the semantic analysis and record generation phase C is specifically:

[0026] The distilled core visual evidence sequence is input into a multi-modal large language model (MLLM), and the MLLM is guided to perform chain reasoning analysis on the interactive behavior through structured prompt engineering, generating a structured data object containing a natural language summary, event type classification, and event classification reasons as a traceability record.

[0027] Further, the storage and indexing phase specifically includes:

[0028] D1. Data persistence, unified data storage: the timestamp of the key event capture, the structured data object generated by C, and the core visual evidence sequence generated by B are stored as a whole record in a relational database with vector extension on the intelligent terminal;

[0029] D2. Vector index construction: using the text embedding model, convert the text content in the record generated by C into a high-dimensional vector embedding, and use the vector support extension of the relational database to build a vector index, providing support for subsequent natural language semantic queries.

[0030] Further, the two-stage natural language forensic query engine of the retrieval and challenge stage specifically comprises:

[0031] E1. Semantic macro event retrieval: the user makes a fuzzy semantic query through natural language; the query engine first converts the question into a query vector using the same text embedding model as in D, and then uses the vector index constructed in D for fast semantic matching to recall the top-K most relevant interactive event records from the massive records;

[0032] E2. Visual micro detail challenge: when the user locks on a specific event from the retrieval results and asks specific follow-up questions about the visual details of the event, the query engine will take out the core visual evidence corresponding to the event from the event records of E1 in the database, and call the MLLM together with the question to conduct visual question answering (VQA), locate and extract the answer to the question from the visual evidence, and then return it to the user in text form.

[0033] Further, the evidence capture and distillation stage is performed online in real time; the semantic analysis and record generation stage is processed online in real time or offline in batches; the storage and index stage is processed online in real time or offline in batches.

[0034] The technical effects of the present application are:

[0035] (1) In view of the defect of high resource consumption in the prior art, the present application can greatly reduce the amount of visual data that needs to be analyzed under the premise of ensuring the integrity of the evidence through evidence capture and distillation. This completely solves the problem of high power consumption and storage overhead of the continuous screen recording scheme, making it possible to perform long-term and efficient forensic analysis on mobile devices.

[0036] (2) In view of the defect of data fragmentation and context loss in the prior art, the present application converts the original, unstructured visual evidence into understandable, structured intelligent evidence through the introduction of the analysis capability of the multi-modal large language model.

[0037] (3) In addition, the two-stage natural language forensic query engine designed by the present application supports the complete analysis process from macro semantic retrieval to micro detail challenge. This solves the problem of low efficiency of traditional forensic methods relying on manual investigation, and provides unprecedented efficiency and convenience for the backtracking and auditing of massive interactive records. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the overall flowchart of the inventive method.

[0039] Figure 2 is the flowchart of the evidence capture and distillation stage.

[0040] Figure 3 is the flowchart of the semantic parsing and record generation stage.

[0041] Figure 4 is the flowchart of the storage and indexing stage.

[0042] Figure 5 is the flowchart of the retrieval and challenge stage. DETAILED DESCRIPTION

[0043] The present application will be further clarified by the following embodiments, with reference to the accompanying drawings.

[0044] The present application is a graphical user interface forensics analysis method based on a multi-modal large language model, which can be used on a smart terminal.

[0045] The present embodiment is to realize the method of efficiently and deeply analyzing the graphical user interface (GUI) interaction on a mainstream smart phone terminal.

[0046] The configuration of the smart phone terminal is as follows:

[0047] Hardware platform: the smart phone terminal of the present embodiment is a consumer-level smart phone equipped with a standard ARM architecture CPU and NPU (neural processing unit), such as an Android device with Qualcomm Snapdragon series chips. This platform can complete medium and light visual model and large language model inference tasks locally.

[0048] Model preparation: in the present embodiment, the smart phone terminal integrates two core AI models. One is a lightweight visual model / encoder, such as ViT-Small, used for feature extraction in the evidence distillation stage. The other is a multi-modal large language model (MLLM), such as a 3 billion parameter (3B) model (e.g. Qwen2.5-VL-3B), which is specifically used to convert visual sequences into structured text.

[0049] The present application is a graphical user interface forensics analysis method based on a multi-modal large language model, which is used on a smart phone terminal and includes the following steps:

[0050] The evidence capture and distillation stage (online real-time) is shown in the flowchart of this stage. Figure 2

[0051] ​A. Adaptive capture of interaction events: In the evidence capture phase, the user interaction is first monitored in a low-power way, and only when the key event moment is triggered, the high-density data collection is performed to balance the evidence integrity and resource consumption.

[0052] A1. Key event trigger: The terminal device is in a standby state with almost zero power consumption by default, and through the auxiliary function interface of the intelligent terminal operating system, only when the key GUI operation of the user is monitored (such as the user's click and sliding event), the event capture process is triggered;

[0053] A2. Time window sampling: Once the key GUI operation of the user is monitored, the screen video stream in the preset time window after the key GUI operation is captured at a preset sampling frequency, forming a raw candidate evidence set containing rich context information.

[0054] In this embodiment, once the user's click and sliding event occurs, the screen video stream in the 2-second window after the event is captured immediately, the sampling frequency is set to high frame rate 2fps, and a raw candidate evidence set containing 4 frames of images with rich context information is generated.

[0055] B. Optimization distillation of visual evidence: In order to greatly reduce the data amount sent to the subsequent large model analysis, the present application introduces a set of visual evidence optimization distillation strategy, through multi-level redundant data elimination, the core visual evidence sequence with the highest information density is extracted, which specifically includes:

[0056] B1. Inter-frame redundancy removal: In order to eliminate the visual redundancy, the fast deduplication technology based on pixel similarity is applied, the L1 distance of the pixels between the adjacent frames (such as the 1st frame and the 2nd frame, the 2nd frame and the 3rd frame) in the raw candidate evidence set is calculated. In this embodiment, the pixel difference threshold is set to 30. If the L1 distance between the two frames is less than 30, it is considered that the latter frame is a redundant frame and is discarded, which effectively deals with the scenes such as UI loading delay or user rapid repeated click.

[0057] B2. Intra-frame semantic information redundancy compression: Apply intra-frame compression technology based on semantic features to segment the video frames after redundancy deduplication into multiple patches, and use the lightweight visual model ViT-Small model to extract features at the patch level for each remaining frame. Then, calculate the cosine similarity of the feature vectors of the patches at the same spatial position between adjacent key frames. According to the set semantic similarity threshold, only the patches with significant semantic changes in content are retained, thereby eliminating a large amount of redundant information such as background and static toolbars. The semantic similarity threshold is set to 0.5 in this embodiment. Only when the similarity is less than 0.5, the patch of the latter frame is considered to have a significant semantic change and is retained, and the remaining patches are discarded. Through this step, a highly condensed core visual evidence sequence is finally generated.

[0058] The semantic analysis and record generation stage (which can be processed online in real time or offline in batches) is shown in the flowchart of this stage as Figure 3

[0059] C. Multi-modal semantic analysis of visual evidence sequence: After the optimization of visual evidence distillation, a multi-modal semantic analysis module is introduced to convert the condensed visual evidence sequence into structured, human-readable traceability records using a multi-modal large language model, which specifically includes:

[0060] The core visual evidence sequence (i.e., a series of key patches) generated in step B2 is input into the locally deployed 3B-scale MLLM (Qwen2.5-VL-3B). By designing a pre-set structured prompt engineering prompt containing role-playing, strategy definition, and output format requirements, the MLLM is guided to perform Chain-of-Thought reasoning analysis on the interactive behavior. The model finally outputs a JSON object containing abstractive_summary (natural language summary), event_type (event type such as "payment operation"), event_description (event classification basis / reason), etc. as traceability records. If the event_type is marked as meaningless daily operation, the record can be discarded, and only records with analytical value are retained.

[0061] The storage and indexing stage (which can be processed online in real time or offline in batches) is shown in the flowchart of this stage as Figure 4

[0062] D. Storage and indexing of traceability records: Based on the generated structured data, a hybrid storage mechanism is introduced to uniformly and efficiently manage multi-modal evidence to facilitate subsequent quick retrieval and deep analysis, which specifically includes: ​​

[0063] D1. Data persistence, unified data storage: The timestamps of key events capture, the structured data objects generated by C, along with the core visual evidence sequence generated by B2 for scenario reproduction, are recorded as a whole in the PostgreSQL database local to the device.

[0064] D2. Vector index construction: Using the bge-base-en-v1.5 text embedding model, the text summary content in each record (such as the natural language summary abstractive_summary and event classification basis / reason event_description field content) is calculated to generate a 768-dimensional vector embedding. Using the pgvector extension of the PostgreSQL database, efficient IVFFlat or HNSW vector indexes are constructed for subsequent natural language semantic high-speed queries.

[0065] Retrieval and interrogation phase, as shown in the flowchart of this phase. Figure 5

[0066] E. Real-time forensic query and deep interrogation: To meet different levels of forensic needs, a two-stage natural language forensic query engine is introduced to support the complete analysis process from macro retrieval to micro interrogation, which includes two stages:

[0067] E1. Semantic macro event retrieval: The user inputs a natural language question in the query interface. The query engine first converts the question into a query vector using the text embedding model in D2, and then uses the vector index constructed in step D2 to perform efficient Cosine similarity retrieval (ANN) in the vector database, recalling the Top-K most relevant interaction event records from the massive records.

[0068] E2. Visual micro detail interrogation: When the user selects an event from the retrieval results and asks a specific question (for example, "What is the transfer amount?"), the query engine will take out the core visual evidence stored in the event record from the E1 event record in the database, and submit it together with the question to the MLLM again. At this time, the MLLM performs a visual question and answer (VQA) task to locate and extract the answer to the question from the visual evidence, and then returns it to the user in text form.

[0069] ​Through the above-described embodiments, the application not only realizes low-cost and long-period GUI interaction forensics on ordinary smart phones, but also endows the smart terminal with unprecedented deep semantic understanding and intelligent human-computer interaction query capability by combining a multi-modal large language model, converts original and unstructured visual evidence into understandable and deep-digging structured intelligent evidence, and significantly improves the feasibility and efficiency of GUI interaction forensics analysis on mobile terminals. This method is particularly suitable for various scenarios that require fine and intelligent auditing, analysis and backtracking of terminal device operations, and is especially suitable for application environments with high requirements for privacy protection, resource consumption and analysis depth, such as enterprise data leakage prevention, fine parental control and high-quality operation data collection.

[0070] Finally, it should be noted that the purpose of the disclosed embodiments is to help further understand the application, but those skilled in the art can understand that various replacements and modifications are possible without departing from the spirit and scope of the application and the appended claims. Therefore, the application should not be limited to the disclosed embodiments, and the scope of the application claimed is defined by the scope of the claims.

Claims

1. A graphical user interface forensic analysis method based on a multimodal large language model, characterized in that, For use on smart terminals, the smart terminal integrates a lightweight visual model and a multimodal large language model (MLLM), as well as a database for storage and querying. The method includes the following steps: Evidence capture and distillation phase: A. Adaptive capture of interaction events: In the evidence capture phase, user interactions are first monitored in a low-power manner, and high-density data collection is only performed at critical event moments to obtain the original candidate evidence set; B. Optimized distillation of visual evidence: Extracting the core visual evidence sequence with the highest information density through multi-level redundant data removal; Semantic parsing and record generation stage: C. Multimodal semantic parsing of visual evidence sequences: Using the multimodal large language model MLLM, the condensed visual evidence sequences are transformed into structured, human-readable source traceability records; Storage and indexing phase: D. Storage and indexing of source traceability records: Based on the generated structured source traceability record data, a hybrid storage mechanism is introduced to uniformly manage multimodal evidence, store it in the database of the smart terminal, and build a vector index using a text embedding model and the database; Search and inquiry phase: E. Real-time Evidence Collection and In-depth Interrogation: To meet the evidence collection needs at different levels, a two-stage natural language evidence collection query engine is designed, supporting a complete analysis process from macro-level retrieval to micro-level interrogation, and realizing evidence collection analysis through a graphical user interface.

2. The method as described in claim 1, characterized in that, The adaptive capture of the evidence capture and distillation phase A interaction event specifically includes: A1. Critical event triggering: The terminal device is in a near-zero power standby state by default. The event capture process is only triggered when a critical GUI operation by the user is detected. A2. Time-series window sampling: Once a key GUI operation by the user is detected, a segment is captured at a preset sampling frequency within a preset time window following the key GUI operation to form an original candidate evidence set containing contextual information.

3. The method as described in claim 2, characterized in that, Key GUI operations in A1 include screen taps, swipes, and input.

4. The method as described in claim 1, characterized in that, The optimized distillation of B-visual evidence in the evidence capture and distillation stage specifically includes: B1. Inter-frame redundancy deduplication: Applying a fast deduplication technique based on pixel similarity, calculate the pixel differences between adjacent frames in the original candidate evidence set, and discard static or duplicate frames whose content has not changed significantly according to the set pixel difference threshold. B2. Intra-frame semantic information redundancy compression: Applying intra-frame compression technology based on semantic features, each frame of the redundant deduplicated video frame is spatially divided into multiple patch tiles. The features of each patch are extracted using a lightweight visual model. Then, by comparing the feature similarity of patches at the same spatial position in consecutive frames, based on the set semantic similarity threshold, only those patches whose content has undergone significant semantic changes are retained. Through this step, a core visual evidence sequence composed of a series of key patches is finally generated.

5. The method as described in claim 4, characterized in that, In B2, the feature similarity of corresponding blocks in the same spatial position in consecutive frames is cosine similarity.

6. The method as described in claim 1, characterized in that, The semantic parsing and record generation stage C specifically includes: The distilled core visual evidence sequence is input into the multimodal large language model MLLM, and the MLLM is guided by structured prompting engineering to perform thought chain reasoning analysis on the interaction behavior, generating a structured data object containing natural language summary, event type classification and event classification reasons, as a source tracing record.

7. The method as described in claim 1, characterized in that, The storage and indexing stage specifically includes: D1. Data persistence and unified data storage: The timestamps of key events, the structured data objects generated by C, and the core visual evidence sequences generated by B are recorded as a whole and stored in a relational database with vector extension on the smart terminal. D2. Vector Index Construction: Using a text embedding model, the text content in the records generated by C is converted into high-dimensional vector embeddings, and the vector index is constructed by leveraging the vector support of relational databases to support subsequent natural language semantic queries.

8. The method as described in claim 1, characterized in that, The two-stage natural language forensics query engine for the retrieval and interrogation phases specifically includes: E1. Semantic Macro Event Retrieval: Users perform fuzzy semantic queries using natural language; the query engine first uses the same text embedding model as in D to transform the question into a query vector, and then uses the vector index built in D to perform fast semantic matching, retrieving the Top-K most relevant interaction event records from massive records; E2. Visual Micro-details Inquiry: When a user identifies a specific event from the search results and asks specific follow-up questions about the visual details of that event, the query engine will retrieve the core visual evidence corresponding to the event from the event records in E1 of the database, and call MLLM to perform visual question answering (VQA) along with the questions. The engine will locate and extract the answers to the questions from the visual evidence and then return them to the user in text form.

9. The method as described in claim 1, characterized in that, The evidence capture and distillation stages are performed online in real time; the semantic parsing and record generation stages are performed online in real time or offline in batch processing; and the storage and indexing stages are performed online in real time or offline in batch processing.

Images