End-side multimodal memory construction method and system for time-series screen data

Abstract

The application provides a terminal-side multi-modal memory construction method and system for timing screen data, comprising: collecting continuous screen frames and window meta information of a terminal-side device; performing GUI structural analysis on the reserved screen frames to form a structured text index; writing the structured meta data, the structured text index and the multi-modal embedding into a relational index track and a vector index track respectively, and establishing a correspondence relationship of the double-track index based on frame identification; performing streaming activity semantic labeling on the reserved screen frames, reusing existing activity labels, aggregating unmatched screen frames into a to-be-labeled activity cluster, and calling a visual language model when creating a representative sample of a new to-be-labeled activity cluster; aggregating screen frames into activity segments with a time range; and in response to a retrieval request, filtering candidate frames using the activity segments, and extracting traceable visual evidence. The application can reduce the high-cost model calling frequency and screen stream storage overhead in long-term operation.

Description

Technical Field

[0001] This invention belongs to the fields of edge-side artificial intelligence agents, multimodal retrieval, screen content understanding, and agent memory technology. Specifically, it relates to a method and system for constructing edge-side multimodal memory for time-series screen data. More specifically, it is a method and system for constructing long-term, continuous, and high-frequency edge-side screen streams into searchable, traceable, and consumer-friendly multimodal memory for agents. "High-frequency" refers to a screen acquisition time interval of less than or equal to 3 seconds, i.e., an acquisition frequency greater than or equal to 1 / 3 Hz; "long-term" refers to the screen time span of continuous or intermittent acquisition and cumulative storage by the edge device reaching several weeks or even months. Background Technology

[0002] With the development of large language models and multimodal models, intelligent agents are evolving from one-off question-answering tools into systems capable of long-term environmental awareness, understanding user context, and performing cross-application tasks.

[0003] In edge-side intelligent agents, the screen is the most direct entry point for digital behavior. Most user actions in tasks such as document editing, web browsing, code development, meeting communication, and data analysis are reflected in the screen display, window switching, control layout, and text content. Therefore, converting continuous screen streams into long-term memory is a crucial foundation for edge-side intelligent agents to obtain user context.

[0004] In existing technologies, various GUI agents or computer-controlled agents typically input the current screen screenshot into a multimodal model for understanding and generate the next action based on the current interface state. This approach is suitable for real-time operations, but it usually treats the screen as a temporary observation object in a single task, rather than a long-term, searchable, and traceable memory object. If this approach is directly extended to a continuous screen stream, multimodal understanding of a large number of screen frames is required, resulting in high multimodal lexical costs and inference latency.

[0005] Existing long-term screen recording or personal screen memory systems, such as Microsoft Recall, Rewind.ai, Screenpipe, and Paperboy, typically record the user's screen continuously and provide timeline playback, keyword search, or OCR text search capabilities. While these solutions can cover a relatively long period of screen history, they are primarily geared towards human review and lack visual semantics. They have not yet fully organized screen recordings into multimodal memories that can be directly consumed by intelligent agents.

[0006] Existing screen memory solutions typically suffer from the following drawbacks.

[0007] First, the cost of single-screenshot understanding or frame-by-frame multimodal understanding is too high. GUI agents typically input the entire current screen screenshot into a visual language model for understanding. This approach is suitable for real-time operations, but continuous screen streams have high frequency, long duration, and high resolution characteristics. Performing complete multimodal inference for each frame would result in high computational, latency, and model input costs, making it difficult to run on the device for extended periods.

[0008] Second, screen recording systems relying on OCR lack visual semantics. Existing long-term screen recording systems mostly employ continuous screenshots, OCR extraction, and keyword retrieval. While OCR can retain some textual information, it loses visual semantics such as window hierarchy, control positions, table relationships, chart structures, page layout, and application context. For queries like "Which page was I debugging code on?", "Which chart did I look at this morning?", or "What's written near this button?", retaining only flat text is insufficient to reliably locate evidence.

[0009] Third, simple video compression is insufficient to directly serve intelligent agents. Recording the screen as video can reduce the storage overhead of the original image, but the video itself is only pixel evidence and does not contain structured text, semantic embeddings, activity tags, or retrieval relationships. Without frame-by-frame addressing, index mapping, and semantic organization, higher-level intelligent agents still struggle to locate usable evidence from long-form video.

[0010] Fourth, calling the visual language model frame-by-frame for activity annotation would lead to uncontrollable costs. Screen activity is continuous and highly repetitive; for example, users may spend a considerable amount of time on the same document, webpage, or IDE interface. If the visual language model is called to generate activity tags for each frame, a large number of semantically identical images would be processed repeatedly. Conversely, if sampling is only based on fixed time intervals, key behavioral boundaries such as window switching, pop-up appearances, and changes in reading position may be missed.

[0011] Therefore, the edge screen memory system needs to solve three technical problems simultaneously: how to reduce storage and processing overhead while retaining traceable visual evidence; how to transform the GUI structure in the screen into searchable visual semantics; and how to abstract the continuous screen stream into stable activity-level semantic objects without calling the visual language model frame by frame.

[0012] Patent document CN120832362A discloses a method and system for multimodal understanding and question answering of long videos based on large models and retrieval enhancement, which achieves efficient understanding and accurate question answering of long videos. However, it does not solve the defects that are usually present in the existing screen memory solutions.

[0013] This problem urgently needs to be solved. Summary of the Invention

[0014] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for constructing edge-side multimodal memory for time-series screen data.

[0015] A method for constructing edge-side multimodal memory for time-series screen data according to the present invention includes: Step S1: Collect continuous screen frames and their window metadata from the end-side device, filter out redundant frames based on the inter-frame differences between the current screen frame and the previous retained screen frame, perform video block compression on the retained screen frames, and record the addressing information of the retained screen frames in the video block for subsequent extraction. Step S2: Perform GUI structured parsing on the retained screen frame to generate interface region candidates associated with the window metadata. Extract the corresponding region from the retained screen frame according to the coordinates of each region candidate and perform OCR to form a structured text index. The structured text index includes: the region OCR result and region coordinates, the application to which it belongs, the window title, and the source frame identifier.

[0016] Step S3: Generate a multimodal embedding based on the retained screen frame and store it in the vector database. Write the structured metadata, the structured text index and the multimodal embedding into the relational index track and the vector index track respectively, and establish the correspondence between the two-track index based on the frame identifier. Step S4: Based on the multimodal embedding similarity of continuous screen content, perform streaming activity semantic annotation on the retained screen frames, reuse existing activity tags, aggregate retained screen frames that do not match existing activity clusters into activity clusters to be annotated, and call the visual language model to generate activity tags when creating representative samples of new activity clusters to be annotated. Step S5: Based on the activity tag, time continuity, and application context, aggregate screen frames with the same activity tag, consistent application context, and time continuity into activity segments with a time range; Step S6: When responding to a retrieval request, candidate activity segments are selected from the activity segments according to the time range, application context, or activity tag in the retrieval request. Within the frame range corresponding to the candidate activity segments, relevant frames are recalled by combining the structured text index and the vector index, and traceable visual evidence is extracted according to the video block addressing information.

[0017] Preferably, in step S1, the inter-frame difference is calculated using the normalized root mean square error (RMSE). It is determined whether the RMSE is less than the difference threshold. If the result is yes, the current screen frame is determined to be a redundant frame; if the result is no, the current screen frame is used as a reserved screen frame. The video block compression includes: writing multiple retained screen frames into the same video block according to a preset time length or refresh condition, compressing the video block using video encoding, and recording the video block identifier and intra-block frame sequence number for each retained screen frame, so that any historical screen frame can be decoded on demand using a composite key of "video block identifier + intra-block frame sequence number".

[0018] Preferably, in step S2, the GUI structured parsing includes: The screen frame is converted into a grayscale image and the gradient magnitude is calculated. Adaptive thresholding is performed on the gradient magnitude to obtain the interface boundary mask. Dilation, closing operation and connected component analysis are performed on the interface boundary mask to obtain region candidates. The overlapping region candidates are merged according to the intersection-union ratio, inclusion degree or text density. According to the coordinates of each candidate region, the corresponding region image is cropped from the screen frame, and OCR is performed on each region image to obtain the region OCR result including the recognized text, local coordinates and recognition confidence. The local coordinates of the region OCR result are converted into absolute screen coordinates, and the region OCR result is bound to the region candidate, application, window, and source frame identifier, and written into the structured text index.

[0019] Preferably, the dual-track index includes: a relational index track and a vector index track; The relational index track stores frame identifiers, timestamps, video blocks to which they belong, frame numbers within the blocks, application names, window titles, region coordinates, OCR text, activity tags, and full-text search indexes. The vector index track stores the multimodal embedding, timestamp, application name, activity tag, and corresponding frame identifier of the screen frame; The structured conditions are filtered through the relational index track, the semantic similarity conditions are recalled through the vector index track, and the recall results are merged into traceable evidence by the same frame identifier.

[0020] Preferably, the streaming activity semantic annotation in step S4 includes a four-layer progressive process, including: L1 layer: The current frame is embedded and matched with the centroid of the mature activity cluster under the corresponding application. If the similarity reaches the threshold of the mature activity cluster, the activity label of the mature activity cluster is reused; if the similarity does not reach the threshold of the mature activity cluster, the process proceeds to L2 layer. In L2 layer, if L1 layer does not match, the current frame is embedded and matched with the candidate activity clusters for similarity. If a candidate activity cluster is matched, the support of the candidate activity cluster is increased, and when the support reaches the promotion threshold, the candidate activity cluster is promoted to a mature activity cluster. If no candidate activity cluster is matched, the process proceeds to L3 layer. L3 layer: If neither L1 nor L2 layer is hit, the current frame is used as the representative sample of the active cluster to be labeled, or it is added to the active cluster to be labeled as a subordinate sample of the existing representative sample. L4 layer: When a representative sample of a new activity cluster to be labeled is generated, the visual evidence, structured OCR text, application context and existing activity labels corresponding to the representative sample are input into the visual language model to obtain the activity label, and the activity label is backfilled into all samples of the activity cluster to be labeled. The labeled activity cluster is then converted into a candidate activity cluster or merged into an existing mature activity cluster.

[0021] Preferably, the centroids of mature active clusters in the L1 layer are updated using an exponential moving average method after reusing the active labels, as expressed by:

[0022] in, Indicates the first The centroid of an active cluster; symbol " " indicates that the value of the variable on the left is updated to the result of the expression on the right; Indicates the current frame embedding; Indicates the update rate; When the support of the mature active cluster exceeds the stability threshold, the centroid update rate is stopped or reduced to suppress abnormal sample interference during long-term operation. The visual language model calls in the L4 layer are executed asynchronously. When the number of consecutive failures of the visual language model calls exceeds the preset failure threshold, it enters a circuit breaker state and suspends new calls during the cooling-off period. When the system restarts or the user triggers a recalculation, the historical samples that are still in the unlabeled state are reorganized into unlabeled activity clusters and labeled.

[0023] Preferably, the aggregation of the activity fragments includes: If adjacent frames belong to the same activity cluster, have the same application context, and the time interval does not exceed the session break threshold, then the adjacent frames will be merged into the same activity segment. Different activity segments are generated if adjacent frames have different activity clusters, inconsistent application contexts, or time intervals exceeding the session break threshold. For frames in the unlabeled state, a temporary active fragment is first attached, and the active tag and time range of the active fragment are updated after the active tag is backfilled.

[0024] According to the present invention, an edge-side multimodal memory construction system for time-series screen data is provided, employing an edge-side multimodal memory construction method for time-series screen data, including: Screen capture module: captures continuous screen frames, focused application, window title, and monitor information; Change-driven storage module: Based on inter-frame differences, redundant frames are filtered out, the remaining screen frames are compressed into video blocks, and video block addressing information is maintained; GUI Structured Parsing Module: Generates interface structure units, binds OCR text with application, window, area coordinates and layout relationships to form a structured text index; Dual-track indexing module: Writes structured metadata and full-text search data into the relational index track, writes multimodal embeddings into the vector index track, and maintains the frame identifier mapping between the two; Activity semantic annotation module: performs mature cluster reuse, candidate cluster promotion, aggregation of activity clusters to be annotated, and visual language model annotation, and generates activity fragments based on activity cluster labels, application context, and temporal continuity; The retrieval and evidence return module retrieves relevant screen evidence based on activity segments, structured text indexes, and vector indexes, and returns corresponding visual and text evidence based on video block addressing information.

[0025] According to the present invention, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the steps of an edge-side multimodal memory construction method for time-series screen data are implemented. An electronic device according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of an edge-side multimodal memory construction method for time-series screen data.

[0026] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention is used to construct a continuous screen stream into a multimodal memory that can be invoked by an intelligent agent, preserving visual semantics and reducing the cost of multimodal lexical units and the storage and processing overhead of the continuous screen stream, thereby improving the retrieval accuracy and positioning efficiency of long-term screen data; furthermore, it aggregates continuous screen frames into activity segments with behavioral semantics and time range, providing upper-layer intelligent agents with more stable memory objects than single frames, single OCR text, or single video segments, facilitating question answering, backtracking, daily report generation, and task context recovery.

[0027] 2. This invention processes continuous screen streams as temporal data, filters out redundant frames based on inter-frame changes, and performs video block compression on the retained frames. This reduces the storage and processing burden of long-term screen streams while preserving traceable pixel evidence. Furthermore, this invention utilizes the characteristics of screen GUIs, such as windows, controls, coordinates, text, and layout relationships, to bind OCR text to the interface structure, preventing screen memory from degenerating into flat text. This allows for the preservation of some visual semantics and improves evidence localization capabilities.

[0028] 3. This invention combines structured text indexing and multimodal vector indexing into a dual-track index, enabling precise time, application, window, and keyword filtering to work in conjunction with fuzzy visual semantic recall, making it suitable for handling complex historical context queries in intelligent agents. Furthermore, this invention employs a four-layer progressive annotation process—mature clusters, candidate clusters, unannotated activity clusters, and visual language models—to ensure that visual language model invocation is primarily triggered by activity changes rather than by the number of screen frames, thereby reducing the frequency of high-cost model invocations during long-term operation and reducing multimodal lexical consumption.

[0029] 4. The processor of the end-side device provided by this invention first filters redundant screen frames by normalizing inter-frame differences, so that screen frames that do not reach the difference threshold do not enter the OCR, multimodal embedding and active semantic annotation processing links, thereby reducing the amount of data that the processor needs to process and the number of read and write operations to the memory; the video encoder organizes the retained screen frames into video blocks and performs H.265 / HEVC inter-frame compression to utilize the temporal redundancy between consecutive screen frames and reduce the amount of data written to the storage medium; the processor performs stepwise similarity matching on screen frames through mature active clusters, candidate active clusters and unannotated active clusters, and only calls the visual language model when creating representative samples of new unannotated active clusters, thereby reducing the hardware computing resources occupied for visual language model inference and the model call waiting time; In a specific test, screen trajectories were captured for 30 minutes on the same end device at a 1-second acquisition interval. Under the conditions of setting the difference threshold to 0.006, forming a video block every 60 frames, and using H.265 / HEVC encoding, the original frame-by-frame PNG file occupied 3533.27 MB, while after inter-frame difference filtering and video block compression, it occupied 270.15 MB, reducing storage usage by approximately 92.35%. In 893 active frame processing events, only 20 events entered the visual language model annotation layer, reducing the number of visual language model calls by approximately 97.76% compared to the frame-by-frame calling method. Attached Figure Description

[0030] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A schematic diagram of the edge-side multimodal memory construction system architecture for time-series screen data provided by the present invention; Figure 2 This is a schematic diagram of the overall process of the end-side multimodal memory construction method provided by the present invention; Figure 3 A schematic diagram of the change-driven storage and video block addressing process provided by the present invention; Figure 4 This is a schematic diagram of the GUI structured text index and dual-track index construction process provided by the present invention; Figure 5 This is a schematic diagram of the four-layer progressive streaming activity semantic annotation process provided by the present invention. Detailed Implementation

[0031] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0032] To address the shortcomings of existing technologies, this invention provides a method and system for constructing edge-side multimodal memory for time-series screen data. Through change-driven storage, GUI structured parsing, dual-track indexing, and streaming activity semantic annotation, it constructs a continuous screen stream into a multimodal memory that can be consumed by intelligent agents. Specifically, it filters out redundant frames based on inter-frame differences and performs video block compression on the retained frames; it utilizes windows, controls, coordinates, and layout relationships to carry screen visual semantics, binding OCR text with the interface structure to form a structured text index; and it generates multimodal embeddings of screen frames, which, together with the structured text index, constitute... Dual-track indexing; based on embedded similarity reuse of activity tags, within the same application context, when the highest cosine similarity (hereinafter referred to as similarity) between the current frame and the centroids of each mature activity cluster, each candidate activity cluster centroid, and each representative sample of an existing unlabeled activity cluster does not reach the corresponding threshold, the current frame is determined as a screen frame that does not match an existing activity cluster, hereinafter referred to as "unmatched activity frame". The current frame is used as a representative sample to create a new unlabeled activity cluster, and a visual language model is called to generate activity tags. Screen frames with similar semantics and continuous time are aggregated into activity fragments to locate relevant evidence.

[0033] In one specific embodiment, the similarity threshold for mature activity clusters, candidate activity clusters, and activity clusters to be labeled is 0.82.

[0034] Specifically, according to the present invention, a method for constructing edge-side multimodal memory for time-series screen data includes: Step S1: Collect continuous screen frames and window metadata from the end-side device, filter out redundant frames based on inter-frame differences, perform video block compression on the retained screen frames, and record the addressing information of each retained screen frame in the video block.

[0035] Step S2: Perform GUI structured parsing on the retained screen frames to generate interface structure units corresponding to windows, regions, controls, or text regions. Bind the OCR text to the coordinates, application, window title, and layout relationship of the interface structure units to form a structured text index.

[0036] Step S3: Generate the multimodal embedding of the retained screen frame, and write the structured text index and the multimodal embedding into the relational index track and the vector index track respectively to form a dual-track index with the frame identifier as the common primary key.

[0037] Step S4: Based on the multimodal embedding similarity of continuous screen content, perform streaming activity semantic annotation on the retained screens. For matched activity frames, reuse existing activity labels; for unmatched activity frames, first aggregate them into activity clusters to be labeled, and only call the visual language model to generate labels when the representative frame of the activity cluster to be labeled appears.

[0038] Step S5: Aggregate screen frames with the same or similar activity semantics, continuous time and consistent application context into activity fragments, so that candidate screen frames can be filtered according to time range, application and behavioral semantics in the future.

[0039] Step S6: In response to retrieval requests from users or agents, the candidate range is narrowed down using activity segments, and relevant evidence is recalled by combining structured text indexes and vector indexes. The corresponding visual evidence is decoded or returned based on video block addressing information.

[0040] A multimodal memory construction system for time-series screen data provided by the present invention includes: The screen capture module is used to capture continuous screen frames, monitor information, focused application, window title, and window area information. The change-driven storage module is used to perform inter-frame difference determination, redundant frame filtering, low-information frame suppression, video block compression, and frame-by-frame addressing. The GUI structured parsing module is used to generate interface region candidates, allocate regions to OCR text, and transform flat OCR text into structured text objects with coordinates, windows, applications, and layout relationships. The dual-track index module is used to maintain relational index tracks and vector index tracks. The relational index track carries metadata such as time, application, window, OCR text and video addressing, while the vector index track carries multimodal embedding and semantic nearest neighbor retrieval fields. The activity semantic annotation module is used to perform mature activity cluster matching, candidate activity cluster promotion, unannotated activity cluster aggregation, asynchronous annotation by visual language model, label backfilling, and activity fragment generation. The retrieval and evidence return module is used to perform mixed retrieval based on time, application, keywords, activity tags and multimodal semantic conditions, and return traceable visual evidence and structured text evidence.

[0041] like Figure 1As shown, the edge-side multimodal memory construction system for time-series screen data provided by this invention can be deployed on laptops, desktops, or other edge devices with screen capture capabilities. The system takes a continuous screen stream as input and includes a screen capture module, a memory construction module, an activity semantic annotation module, an edge-side multimodal memory bank, a retrieval and evidence interface, and an upper-layer intelligent agent.

[0042] Figure 1 The functions of each module are described in detail below: Firstly, the screen capture module is used to continuously acquire consecutive screen frames from the terminal device and simultaneously acquire window metadata such as the focused application, window title, window position, monitor number, and capture timestamp.

[0043] The output of the screen capture module goes into the memory construction module to form searchable and traceable underlying memory evidence; on the other hand, it provides temporal continuity and application context for the activity semantic annotation module.

[0044] Secondly, the memory building module is used to perform change filtering on screen frames, video block compression, GUI structured parsing, OCR text binding, multimodal embedding generation, and dual-track index writing.

[0045] Third, the activity semantic annotation module is used to read the multimodal embeddings, structured text and application context generated by the memory building module, perform activity tag reuse, new activity annotation and activity fragment generation, and write the activity tags and activity fragments into the edge multimodal memory library.

[0046] Fourth, the edge-side multimodal memory is used to store video blocks, structured text indexes, vector indexes, frame-level metadata, and active segments.

[0047] Fifth, the retrieval and evidence interface is used to locate relevant activity fragments, structured text, and visual evidence from the edge multimodal memory based on user or agent queries, and return the evidence to the upper-layer agent.

[0048] In one specific embodiment, the basic data objects in the system include: Firstly, screen frames: full-screen screenshots or application screenshots obtained by the screen capture module, containing metadata such as frame identifier, capture timestamp, monitor number, focused application, and focused window title, which is the execution granularity of embedding, retrieval, and semantic annotation; Secondly, video blocks: video files formed by compressing multiple screen frames over a period of time, used for long-term preservation of pixel evidence; Third, structured text objects: text evidence units consisting of OCR text, region coordinates, application name, window title, confidence level, and source region; Fourth, multimodal embedding: the vector representation of screen frames generated by the multimodal embedding model is used to express the visual and textual semantics of screen content; Fifth, activity clusters: clustered objects composed of semantically similar screen frames, including cluster identifiers, centroid vectors, support, and activity labels; Sixth, Activity Segment: A time segment object formed by merging frames that are continuous in time and have consistent activity clusters, including start and end times, application, window, activity label, and representative evidence frame.

[0049] The system is logically divided into a data plane and a control plane. The data plane is responsible for acquisition, deduplication, OCR, embedding, video compression, and index building, ensuring the stable and continuous operation of the processing chain. The control plane is responsible for system-level scheduling and policy management, including acquisition frequency control, video block refresh policy, batch write scheduling, active semantic annotation task scheduling, visual language model call control, anomaly recovery, and privacy policy enforcement. The data plane and control plane are interconnected, enabling high-cost semantic processing, model calls, and batch tasks to be executed asynchronously, ensuring the system's efficient operation.

[0050] A method for constructing edge-side multimodal memory for time-series screen data according to the present invention includes: Step S1: Collect continuous screen frames and their window metadata from the end-side device, filter out redundant frames based on the inter-frame differences between the current screen frame and the previous retained screen frame, perform video block compression on the retained screen frames, and record the addressing information of the retained screen frames in the video block for subsequent extraction. Step S2: Perform GUI structured parsing on the retained screen frame obtained in step S1 to generate interface region candidates associated with window metadata. Extract the corresponding region from the retained screen frame according to the coordinates of each region candidate and perform OCR. Bind the region OCR result with the region coordinates, the application to which it belongs, the window title and the source frame identifier to form a structured text index. Step S3: Generate multimodal embeddings based on the screen frames retained in S1 and store them in the vector database. Write the structured metadata, structured text index and multimodal embeddings into the relational index track and the vector index track respectively, and establish the correspondence between the two-track indexes based on the frame identifier. Step S4: Based on the multimodal embedding similarity of continuous screen content, perform streaming activity semantic annotation on the retained screen frames, reuse existing activity tags, aggregate unmatched activity frames into activity clusters to be annotated, and call the visual language model to generate activity tags when creating a representative sample of a new activity cluster to be annotated. The activity cluster to be annotated includes a representative sample and zero or more subordinate samples similar to the representative sample. Step S5: Based on activity tags, temporal continuity, and application context, aggregate screen frames with the same or similar activity tags, consistent application context, and temporal continuity into activity segments with a time range. Step S6: When responding to a retrieval request, candidate activity segments are selected from the activity segments based on the time range, application context, or activity tag in the retrieval request. Relevant frames are recalled by combining structured text index and vector index, and traceable visual evidence is extracted based on the video block addressing information.

[0051] Specifically, in step S1, the inter-frame difference is calculated using the normalized root mean square error:

[0052] in, This represents the grayscale image of the current screen frame. This represents the grayscale image of the previous retained screen frame. and These represent the image height and width, respectively. When M t When the difference is less than the threshold, the current screen frame is determined to be a redundant frame and removed from the main processing chain of OCR, embedding, and semantic annotation; when M t If the difference threshold is greater than or equal to the current screen frame, the current screen frame will be retained for subsequent processing.

[0053] Specifically, the video block compression in step S1 includes: Write multiple retained screen frames into the same video block according to a preset time length or refresh condition; The video block is compressed using H.265 / HEVC video encoding to generate an MP4 video block; Record the video block identifier and intra-block frame number for each retained screen frame, so that any historical screen frame can be decoded on demand using a composite key of "video block identifier + intra-block frame number".

[0054] Specifically, in step S2, the GUI structured parsing includes: converting the screen frame into a grayscale image and calculating the gradient magnitude; Adaptive thresholding is performed on the gradient magnitude to obtain the interface boundary mask; Perform dilation, closing, and connected component analysis on the interface boundary mask to obtain region candidates; Candidate overlapping regions are merged based on intersection-union ratio, inclusion degree, or text density; Based on the coordinates of each candidate region, the corresponding region image is extracted from the screen frame; OCR is performed on each region image to obtain the region OCR result, which includes the recognized text, local coordinates, and recognition confidence. The local coordinates of the region OCR results are converted to absolute screen coordinates, and the region OCR results are bound to region candidates, application, window, and source frame identifiers, and written into a structured text index.

[0055] Specifically, the dual-track index includes: Relational index tracks are used to store frame identifiers, timestamps, video blocks to which they belong, frame numbers within blocks, application names, window titles, region coordinates, OCR text, activity tags, and full-text search indexes; The vector index track is used to store the multimodal embedding, timestamp, application name, activity tag, and corresponding frame identifier of the screen frame; Among them, structured conditions are filtered through relational index tracks, semantic similarity conditions are recalled through vector index tracks, and the recall results are merged into traceable evidence by the same frame identifier.

[0056] Specifically, the streaming activity semantic annotation in step S4 includes a four-layer progressive process: In the L1 layer, the current frame is embedded and matched with the centroid of the mature activity cluster under the corresponding application. If the similarity reaches the threshold of the mature cluster, the activity label of the mature activity cluster is reused. In the L2 layer, if the L1 layer does not match, the current frame is embedded and matched with the candidate activity cluster for similarity. If the candidate activity cluster is matched, the support of the candidate activity cluster is increased, and when the support reaches the promotion threshold, the candidate activity cluster is promoted to a mature activity cluster. If L1 and L2 are both not hit, the current frame is used as the representative sample of the active cluster to be labeled, or as a subordinate sample similar to the existing representative sample and added to the active cluster to be labeled. In layer L4, when a representative sample of a new activity cluster to be labeled is generated, the visual evidence, structured OCR text, application context and existing activity labels corresponding to the representative sample are input into the visual language model to obtain the activity label. The activity label is then backfilled into all samples in the activity cluster to be labeled, and the labeled activity cluster is converted into a candidate activity cluster or merged into an existing mature activity cluster.

[0057] Specifically, the centroids of mature activity clusters in the L1 layer are updated using an exponential moving average method after reusing activity tags. This is used to absorb changes in interface theme, layout fine-tuning, or window size changes under the same activity, and to improve the stability of reusing activity tags in subsequent similar frames.

[0058] Where, μ j This represents the centroid vector of the j-th mature activity cluster; the symbol "←" indicates that the value of the variable on the left is updated to the result of the expression on the right. Indicates the current frame embedding. Indicates the update rate; When the support of mature active clusters exceeds the stability threshold, the centroid update rate is stopped or reduced to suppress interference from anomalous samples during long-term operation.

[0059] Specifically, the visual language model calls in the L4 layer are executed asynchronously and include: Set a timeout window for the active clusters to be labeled; if no response is received after the timeout, retain the labeling status. When the number of consecutive failed calls to the visual language model exceeds the failure threshold, it enters a circuit breaker state, suspending new calls during the cooldown period; Once the visual language model returns the active labels, all samples within the active cluster to be labeled are backfilled in a batch transaction manner. When the system restarts, the model is switched, or a user triggers a recalculation, historical samples that are still in the unlabeled state or marked as needing recalculation are reorganized into unlabeled activity clusters and labeled.

[0060] Specifically, the aggregation of the activity fragments includes: If adjacent frames belong to the same activity cluster, have the same application context, and the time interval does not exceed the session break threshold, then the adjacent frames will be merged into the same activity segment. If adjacent frames have different activity clusters, inconsistent application contexts, or a time interval exceeding the session break threshold, different activity segments are generated. For frames in the unlabeled state, a temporary active fragment is first attached, and the active label and time range of the active fragment are updated after the active label is backfilled.

[0061] A multimodal memory construction system for time-series screen data provided by the present invention includes: The screen capture module is used to capture continuous screen frames, focused applications, window titles, and monitor information. The change-driven storage module is used to filter out redundant frames based on inter-frame differences, perform video block compression on the retained screen frames, and maintain video block addressing information. The GUI structured parsing module is used to generate interface structure units, bind OCR text with application, window, area coordinates and layout relationships to form a structured text index; The dual-track indexing module is used to write structured metadata and full-text search data into the relational index track, write multimodal embeddings into the vector index track, and maintain the frame identifier mapping between the two. The activity semantic annotation module is used to perform mature cluster reuse, candidate cluster promotion, aggregation of activity clusters to be annotated, and visual language model annotation, and to generate activity fragments based on activity cluster labels, application context, and temporal continuity. The retrieval and evidence return module is used to retrieve relevant screen evidence based on activity fragments, structured text indexes, and vector indexes, and return corresponding visual and text evidence based on video block addressing information.

[0062] Specifically, the system further includes an interface module for intelligent agents, the interface module comprising: Atomic interface, the finest-grained query, used to read frame, OCR region, and video block evidence; The aggregation interface integrates underlying atomic evidence for querying, and is used to return activity fragments and mixed search results by time, application, window, or activity tag. The intelligent interface directly encapsulates common tasks of intelligent agents, and is used to generate question-and-answer results, activity explanations, or time period summaries based on retrieved evidence.

[0063] like Figure 2 As shown, the main flow of the method of the present invention includes steps 101 to 107.

[0064] In step 101, the system collects continuous screen frames and window metadata, including the focused application, window title, window position, monitor number, and collection timestamp. For multi-monitor devices, the system supports collecting screen frames from each monitor separately; for platforms that support window enumeration, the system can also collect sub-screenshots of the first few active windows. The window metadata is then processed along with the screen frames in subsequent steps.

[0065] In step 102, the system filters screen frames that will enter subsequent processing based on the degree of change in content between adjacent screens. This step determines which screen frames from the acquisition results enter the memory building chain, and its output serves as the common processing object for steps 103 to 106. Specific inter-frame difference calculations, threshold judgments, and branch processing will be handled later. Figure 3 Further explanation is provided in steps 201 to 207.

[0066] In step 103, the system further compresses and organizes the screen frames into underlying visual evidence that can be referenced by subsequent indexing. This step, by compressing video blocks and recording frame addressing information, provides the corresponding original screen content for the text objects, vector records, and activity segments generated in steps 104 to 106.

[0067] In step 104, the system divides the screen frame into GUI regions and, combined with the corresponding application, window coordinate information, and OCR results, forms a structured text object with spatial attribution and interface context. The processes of GUI region candidate generation, OCR text allocation, and relational index writing will be performed within the specified timeframes. Figure 4 Further explanation is provided in steps 301 to 306.

[0068] In step 105, the system generates a multimodal embedding for the screen frame and associates the multimodal embedding with the structured text object and window metadata obtained in step 104 through frame identifiers. This step forms a relational index track that can perform precise conditional filtering and a vector index track that can perform semantic nearest neighbor retrieval, enabling keywords, time, and application conditions to participate in retrieval together with visual semantics.

[0069] In step 106, the system utilizes multimodal embedding of screen frames, timestamps, and application context to identify user activities, organizing semantically similar and temporally continuous frames into activity segments with activity tags and start and end times. This step elevates scattered frame-level records into time-segment objects that can be retrieved by behavioral semantics, and reduces the overhead of calling the visual language model by reusing existing tags and labeling based on new activity triggers; the specific activity tag generation process will be discussed later. Figure 5 Further explanation is provided in steps 401 to 415.

[0070] In step 107, based on the design of the memory module, the system provides a query interface designed for the upper-layer agent: first, it parses and uses the time and application information in the query prompts to perform relational data filtering; then, it uses the activity tag information generated in step 106 to filter out candidate activity fragments to further narrow down the scope; next, it combines the structured text index and vector index generated in step 105 to recall relevant frames; finally, it obtains the corresponding visual content based on the video block and frame sequence number within the block generated in step 103; and obtains text and pixel evidence and returns it to the agent to provide context for answering questions. In this process, activity fragments are used to quickly locate the retrieval scope from the long-term screen stream, the structured text index is used for precise text location, the vector index is used to supplement fuzzy semantic recall, and the video block addressing information is used to trace back the original pixel evidence.

[0071] like Figure 3 As shown, steps 102 and 103 can be further refined into steps 201 to 209.

[0072] In steps 201 and 202, the system inputs the current screen frame and obtains the previous retained frame. In step 203, the system calculates the normalized inter-frame difference between the current screen frame and the previous retained frame, and determines whether the normalized inter-frame difference is lower than the difference threshold.

[0073] Because continuous screen streams contain a significant amount of spatiotemporal redundancy—for example, when a user is lingering on a document page, reading a webpage, or waiting for a program to run—adjacent frames may only differ in cursor blinking or mouse movement. Failure to filter out these frames would incur unnecessary processing costs. Therefore, the system uses a frame-level deduplication tool to determine changes in the current screen frame.

[0074] In one specific embodiment, the system converts the current screen frame and the previous retained screen frame into grayscale representations and calculates the normalized root mean square error:

[0075] like <R diff In step 204, the current screen frame is identified as a redundant frame; wherein, the R diff R represents the normalized inter-frame difference threshold used to distinguish redundant frames from reserved screen frames. diff The value of R ranges from 0 to 1; in one specific embodiment, R diff Take 0.006.

[0076] In step 205, the system does not send the redundant frame into the subsequent memory construction link, but instead updates the time anchor point corresponding to the previously retained frame, so that the timeline can record the screen state until the current acquisition time, and then proceeds to step 207 to continue acquiring the next screen frame. ≥R diff If so, proceed to step 206, where the current screen frame is identified as a reserved frame and written into the buffer to be compressed.

[0077] As a supplement to the redundancy deduplication logic: In order to avoid high-resolution screen noise and reduce processing costs, the system can perform fixed-ratio cropping and downsampling of screen frames before calculating differences; In order to avoid low-information frames such as lock screens occupying processing resources, low-information frames are eliminated in advance by calculating the grayscale standard deviation.

[0078] In step 206, the retained frames are not directly written to disk, but are first accumulated in the compression buffer by the video block writer. After writing, the system determines whether the video block refresh conditions have been met. These refresh conditions include reaching a preset duration, a frame rate threshold, or an externally triggered interrupt. If the video block refresh conditions have not been met, the system proceeds to step 207 to continue acquisition, and subsequent retained screen frames are written to the buffer.

[0079] If the refresh condition is met, proceed to step 208, where H.265 / HEVC video encoding is used to encode the reserved screen frames in the buffer to be compressed into MP4 video blocks, thereby reducing the storage overhead of long-term pixel evidence through inter-frame compression.

[0080] Finally, in step 209, the system records the corresponding video block metadata, the intra-block frame sequence number of each screen frame within the video block, and ultimately writes the video block to storage. The intra-block frame sequence number is used to support subsequent evidence retrieval, allowing the agent to decode the screen frame at a specified time based on the addressing information. Steps 208 and 209 are performed asynchronously and will not interfere with the foreground screen capture and preprocessing process.

[0081] like Figure 4As shown, the screen is not an ordinary natural image, but a structured scene with windows, controls, text, coordinates, and layout. This invention utilizes this characteristic to transform some visual semantics into a structured text index.

[0082] In step 301, the system inputs the retained screen frame identified and written to the compression buffer in step 206, and reads the window metadata corresponding to the retained screen frame. The window metadata includes the application name, window title, window position, and focus state. Subsequently, the retained screen frame enters the structured text index construction branch and the vector index construction branch, respectively.

[0083] In step 302, the system generates GUI region candidates on the reserved screen frame. In one specific embodiment, the system adopts a lightweight region candidate generation process inspired by the UIED method, converts the RGB screen image to a grayscale image, uses the Sobel operator to calculate the gradient magnitude and performs adaptive thresholding to obtain edge masks corresponding to controls, text blocks, separator lines and window boundaries; then, it performs dilation and closing operations on the edge masks to connect discrete edges belonging to the same GUI region into connected regions; then, it performs connected component analysis to obtain multiple region candidate boxes, and merges overlapping region candidates according to the intersection-union ratio, inclusion degree and text density to obtain a more stable set of regions.

[0084] The above method does not require precise identification of every control category, but rather divides the GUI area with low overhead to accommodate sufficient retrieval semantics. For example, for a document editor, the system can divide the area into a body text area, a sidebar area, and a toolbar area; for a browser, the system can divide the area into a webpage body, a tab bar, an address bar, and a side navigation area; for an IDE, the system can divide the area into a code editor, a file tree, a terminal output, and a debugging panel area.

[0085] In step 303, the system binds the coordinates of each region candidate to window metadata, so that the region candidate has the application to which it belongs, the window to which it belongs, the region coordinates, and the source frame identifier.

[0086] In step 304, the system extracts the corresponding region images from the retained screen frames according to the coordinates of each region candidate, and uses OCR to extract the text information in each region image.

[0087] Let R be the candidate region of the i-th region. i R i =(b i , a i , w i ), where b i Let a be the region coordinates. i For the application, w iThis is the window to which the image belongs; after performing OCR on the image of this region, the region OCR result is obtained. i O i =(b ij ,t ij , c ij ), where b ij Let t be the local coordinates of the j-th text within the region image. ij To recognize text, c ij To determine the confidence level, the system can obtain the absolute coordinates of the text within the original screen frame based on the local coordinates.

[0088] In step 305, the system combines the OCR results of each region with the region identifier, region coordinates and window metadata to generate a structured text object.

[0089] In step 306, the system writes the structured text objects into a relational database, forming a relational index track. The relational index includes a region detail index and a full-text search index: the region detail index is used to store the coordinates, text, and window affiliation of each region to support spatial positioning and fine-grained evidence display; the full-text search index is used to summarize the text of multiple regions within the same screen frame according to the frame identifier to support keyword and phrase retrieval.

[0090] Corresponding to the structured text indexing construction branch, in step 307, the system generates a multimodal embedding based on the retained screen frame input in step 301. The multimodal embedding is generated by the edge-side lightweight multimodal embedding model based on the visual content of the screen image and is used to represent semantic features.

[0091] In step 308, the system writes the multimodal embedding into the vector database to form a vector index track. By preserving visual information and supporting fuzzy matching, relevant frames can be recalled through visual semantics even if there are no completely matching keywords in the structured text index.

[0092] In step 309, the system establishes a consistent reference relationship between the relational index record formed in step 306 and the vector index record formed in step 308 using frame identifiers. The relational index track is used to store filterable and interpretable data tables; the vector index track is used to store multimodal embeddings and their filtering fields.

[0093] In step 310, the system forms a dual-track index consisting of a relational index track and a vector index track to enhance the agent's retrieval capabilities. A typical retrieval process can first filter candidates by time range and application name in the relational index track, then perform semantic nearest neighbor retrieval in the vector index track, and finally link the results back to the relational index track to obtain OCR regions, video block addresses, and activity tags. Simultaneously, frames obtained from keyword retrieval can also undergo semantic rearrangement through the vector index track.

[0094] like Figure 5 As shown, this invention designs the activity semantic annotation of a continuous screen stream as a four-layer progressive process: L1 mature activity clusters, L2 candidate activity clusters, L3 activity clusters to be annotated, and L4 visual language model annotation. The first three layers sequentially use low-cost vector similarity matching and state update to process the current frame, and only representative samples of newly created activity clusters to be annotated trigger the visual language model task.

[0095] In step 401, the system inputs the multimodal embedding of the current frame. And obtain the application context a corresponding to the frame. t and timestamp r t The system maintains mature activity clusters, candidate activity clusters, and activity clusters to be labeled separately to avoid erroneous merging of frames with similar visual layouts but different behavioral semantics in different applications.

[0096] In step 402, the system compares the cosine similarity between the multimodal embedding of the current frame and the centroids of mature activity clusters under the same application, and determines whether a match is found. If the similarity of a mature activity cluster reaches the mature cluster threshold, then in step 403, the current frame directly reuses the activity label of that mature activity cluster, and updates the cluster centroids using an exponential moving average.

[0097] in, For the first The centroid of a mature active cluster is indicated by the symbol "←", which means updating the value of the variable on the left to the result of the calculation of the expression on the right. Indicates the current frame embedding. This indicates the update rate. This update method allows the system to adapt to long-term changes and avoids a few abnormal frames significantly altering the meaning of active clusters.

[0098] If a mature activity cluster is not found in step 402, then in step 404, the multimodal embedding of the current frame is compared with the centroid of candidate activity clusters under the same application to determine whether a match is found. Candidate activity clusters are used to carry activity modes that have obtained activity tags but whose support has not yet reached the stability requirement. If a candidate activity cluster is found, then in step 405, the activity tag of the candidate activity cluster is reused in the current frame, the system increases the support of the candidate activity cluster, and determines whether its support reaches the promotion threshold.

[0099] If the promotion threshold is reached, the candidate activity cluster is promoted to a mature activity cluster in step 406. If the promotion threshold is not reached, the activity cluster remains in the candidate state and waits for further improvement in cluster support. Thus, short-lived activity patterns with insufficient support do not immediately become mature activity clusters, while recurring stable activities can gradually enter the set of mature activity clusters.

[0100] If step 404 fails to find a candidate activity cluster, then in step 407, the multimodal embedding of the current frame is compared with the representative sample embeddings of each activity cluster to be labeled under the same application to determine whether a match has been found. Specifically, let the multimodal embedding of the current frame be... , No. The representative sample embeddings of each unlabeled activity cluster are: Calculate the cosine similarity using the following formula. :

[0101] Among them, the superscript " " indicates transpose, symbol " " represents the L2 norm.

[0102] Within the same application context, if the maximum value of the cosine similarity among the various activity clusters to be labeled is greater than or equal to the similarity threshold of the activity clusters to be labeled, then it is determined as a hit, and the current frame is added to the activity cluster to be labeled with the highest cosine similarity; if there is no activity cluster to be labeled that reaches the similarity threshold, then it is determined as a miss. In a specific embodiment, the similarity threshold of the activity clusters to be labeled is 0.82.

[0103] If a match is found, in step 408, the system adds the current frame as a subordinate sample to the corresponding active cluster to be labeled. The subordinate sample no longer triggers a new visual language model call, but waits for the visual language model labeling task triggered by the sample representing the active cluster to return an active label.

[0104] If no target activity cluster is found, in step 409, the system uses the current frame as a representative sample to create a new target activity cluster. In step 410, the system asynchronously invokes the visual language model for the representative sample. The input to the visual language model includes: representative sample pixel evidence obtained based on the video block identifier and frame sequence number within the block, the corresponding structured OCR text, application name, window title, nearby time context, and a list of existing activity tags. The asynchronous invocation allows the current frame to exit the real-time processing after completing the target state registration, without blocking the acquisition and indexing of subsequent screen frames.

[0105] In step 411, the visual language model generates activity labels based on representative sample evidence, or reuses existing activity labels if the generated result is semantically consistent with existing activity labels.

[0106] In step 412, after the visual language model in step 411 returns the annotation results, a callback event is triggered. In the callback event, the system backfills the activity labels in batches to the representative samples and all subordinate samples in the activity cluster to be annotated, and in step 413, the activity cluster to be annotated is promoted to a candidate activity cluster.

[0107] To ensure stability during long-term operation, visual language model calls are executed asynchronously by independent worker threads. If the visual language model service times out, it remains in the unlabeled state and retryes after a period of time; if the number of consecutive failures exceeds a threshold, it enters a circuit breaker state to suspend call attempts; when the service is restored and the system restarts, the system can scan historical unlabeled samples and reorganize model calls for supplementary labeling.

[0108] All the branches mentioned above eventually converge to step 414: Once the activity cluster affiliation and activity tag of the frame have been determined, the system generates an activity segment based on the determined activity cluster identifier, activity tag, application context, and temporal continuity. If adjacent frames belong to the same activity cluster, have the same application context, and the time interval does not exceed the session break threshold, they are merged into the same activity segment; otherwise, a new activity segment is created.

[0109] In step 415, the system writes information about the activity segment, including activity tag, start and end time, application name, window title, representative frame and internal frame number, into storage so that the upper-layer agent can filter candidate activity segments according to behavioral semantics and time range.

[0110] A method for constructing edge-side multimodal memory for time-series screen data according to the present invention includes: Step S1: Read the continuous screen frames and their window metadata from the display controller via the screen capture interface; the CPU calculates the inter-frame difference between the current screen frame and the previous retained screen frame, filters out redundant frames, and writes the retained screen frames into the memory buffer; the CPU calls the video encoder to perform video block compression on the retained screen frames in the memory buffer; the compressed video blocks are stored on the disk, and the addressing information of the retained screen frames in the video blocks is stored for subsequent extraction. Step S2: The CPU performs GUI structured parsing on the retained screen frame to generate interface region candidates associated with the window metadata; the CPU extracts the corresponding region according to the coordinates of each region candidate and performs OCR to form a structured text index and write it to the disk. Step S3: The GPU runs a multimodal embedding model to generate multimodal embeddings based on the retained screen frames; the CPU organizes the structured metadata and the structured text index into relational index records, organizes the multimodal embeddings into vector index records, and establishes a correspondence between the relational index records and the vector index records based on the frame identifier; the storage controller writes the relational index records and the vector index records to the disk; Step S4: The CPU reads the activity cluster information from the disk, calculates the cosine similarity between the multimodal embedding of the current frame and the corresponding embedding of each activity cluster, and reuses existing activity labels or aggregates unmatched activity frames into activity clusters to be labeled based on the calculation results; when creating representative samples for new activity clusters to be labeled, the CPU sends the visual evidence, structured OCR text and application context corresponding to the representative samples to the visual language model service running on the remote server through the network interface; the remote server generates activity labels and returns the activity labels to the end device through the network interface; the CPU backfills the activity labels into the representative samples and subordinate samples in the activity clusters to be labeled, and the storage controller writes the backfilling results to the disk; Step S5: The CPU reads the activity tag, timestamp, and application context corresponding to each screen frame from the disk. Based on the activity tag, time continuity, and application context, screen frames with the same activity tag, consistent application context, and continuous time are aggregated into an activity segment with a time range. The storage controller writes the activity tag, start and end time, application name, window title, and representative frame identifier of the activity segment to the disk. Step S6: The CPU queries the activity segments, structured text index, and vector index in the disk according to the retrieval request to recall relevant frames; the storage controller reads the corresponding video block from the disk and loads it into memory; the video decoder decodes the corresponding screen frame evidence according to the video block identifier and the frame sequence number within the block, and combines the visual evidence, structured OCR text, and activity tags into evidence results, which are then returned to the user or the upper-level intelligent agent.

[0111] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0112] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for constructing edge-side multimodal memory for time-series screen data, characterized in that, include: Step S1: Collect continuous screen frames and their window metadata from the end-side device, filter out redundant frames based on the inter-frame differences between the current screen frame and the previous retained screen frame, perform video block compression on the retained screen frames, and record the addressing information of the retained screen frames in the video block for subsequent extraction. Step S2: Perform GUI structured parsing on the retained screen frame to generate interface region candidates associated with the window metadata. Extract the corresponding region from the retained screen frame according to the coordinates of each region candidate and perform OCR to form a structured text index. Step S3: Generate a multimodal embedding based on the retained screen frame and store it in the vector database. Write the structured metadata, the structured text index and the multimodal embedding into the relational index track and the vector index track respectively, and establish the correspondence between the two-track indexes based on the frame identifier. Step S4: Based on the multimodal embedding similarity of continuous screen content, perform streaming activity semantic annotation on the retained screen frames, reuse existing activity tags, aggregate retained screen frames that do not match existing activity clusters into activity clusters to be annotated, and call the visual language model to generate activity tags when creating representative samples of new activity clusters to be annotated. Step S5: Based on the activity tag, time continuity, and application context, aggregate screen frames with the same activity tag, consistent application context, and time continuity into activity segments with a time range. Step S6: When responding to a retrieval request, candidate activity segments are selected from the activity segments based on the time range, application context, or activity tag in the retrieval request. Relevant frames are recalled by combining the structured text index and the vector index, and traceable visual evidence is extracted based on the video block addressing information.

2. The method for constructing edge-side multimodal memory for time-series screen data according to claim 1, characterized in that, In step S1, the inter-frame difference is calculated using the normalized root mean square error. It is determined whether the normalized root mean square error is less than the difference threshold. If the result is yes, the current screen frame is determined to be a redundant frame. If the result is negative, the current screen frame will be kept as the screen frame. The video block compression includes: writing multiple retained screen frames into the same video block according to a preset time length or refresh condition, compressing the video block using video encoding, and recording the video block identifier and the intra-block frame sequence number for each retained screen frame, so that any historical screen frame can be decoded on demand using a composite key of "video block identifier + intra-block frame sequence number".

3. The method for constructing edge-side multimodal memory for time-series screen data according to claim 1, characterized in that, In step S2, the GUI structured parsing includes: The screen frame is converted into a grayscale image and the gradient magnitude is calculated. Adaptive thresholding is performed on the gradient magnitude to obtain the interface boundary mask. Dilation, closing operation and connected component analysis are performed on the interface boundary mask to obtain region candidates. The overlapping region candidates are merged according to the intersection-union ratio, inclusion degree or text density. According to the coordinates of each candidate region, the corresponding region image is cropped from the screen frame, and OCR is performed on each region image to obtain the region OCR result including the recognized text, local coordinates and recognition confidence. The local coordinates of the region OCR result are converted into absolute screen coordinates, and the region OCR result is bound to the region candidate, application, window, and source frame identifier, and written into the structured text index.

4. The method for constructing edge-side multimodal memory for time-series screen data according to claim 1, characterized in that, The dual-track index includes: a relational index track and a vector index track; The relational index track stores frame identifiers, timestamps, video blocks to which they belong, frame numbers within the blocks, application names, window titles, region coordinates, OCR text, activity tags, and full-text search indexes. The vector index track stores the multimodal embedding, timestamp, application name, activity tag, and corresponding frame identifier of the screen frame; The structured conditions are filtered through the relational index track, the semantic similarity conditions are recalled through the vector index track, and the recall results are merged into traceable evidence by the same frame identifier.

5. The method for constructing edge-side multimodal memory for time-series screen data according to claim 1, characterized in that, The streaming activity semantic annotation in step S4 includes a four-layer progressive process, including: L1 layer: The current frame is embedded and matched with the centroid of the mature activity cluster under the corresponding application. If the similarity reaches the threshold of the mature activity cluster, the activity label of the mature activity cluster is reused; if the similarity does not reach the threshold of the mature activity cluster, the process proceeds to L2 layer. In L2 layer, if L1 layer does not match, the current frame is embedded and matched with the candidate activity clusters for similarity. If a candidate activity cluster is matched, the support of the candidate activity cluster is increased, and when the support reaches the promotion threshold, the candidate activity cluster is promoted to a mature activity cluster. If no candidate activity cluster is matched, the process proceeds to L3 layer. L3 layer: If neither L1 nor L2 layer is hit, the current frame is used as the representative sample of the active cluster to be labeled, or it is added to the active cluster to be labeled as a subordinate sample of the existing representative sample. L4 layer: When a representative sample of a new activity cluster to be labeled is generated, the visual evidence, structured OCR text, application context and existing activity labels corresponding to the representative sample are input into the visual language model to obtain the activity label, and the activity label is backfilled into all samples of the activity cluster to be labeled. The labeled activity cluster is then converted into a candidate activity cluster or merged into an existing mature activity cluster.

6. The method for constructing edge-side multimodal memory for time-series screen data according to claim 5, characterized in that, The centroids of mature activity clusters in the L1 layer are updated using an exponential moving average method after reusing activity labels, as expressed by: in, Indicates the first Centroid of an active cluster; symbol " " indicates that the value of the variable on the left is updated to the result of the expression on the right; Indicates the current frame embedding; Indicates the update rate; When the support of the mature active cluster exceeds the stability threshold, the centroid update rate is stopped or reduced to suppress abnormal sample interference during long-term operation. The visual language model calls in the L4 layer are executed asynchronously. When the number of consecutive failures of the visual language model calls exceeds the preset failure threshold, it enters a circuit breaker state and suspends new calls during the cooling-off period. When the system restarts or the user triggers a recalculation, the historical samples that are still in the unlabeled state are reorganized into unlabeled activity clusters and labeled.

7. The method for constructing edge-side multimodal memory for time-series screen data according to claim 1, characterized in that, The aggregation of the activity segments includes: If adjacent frames belong to the same activity cluster, have the same application context, and the time interval does not exceed the session break threshold, then the adjacent frames will be merged into the same activity segment. If adjacent frames have different activity clusters, inconsistent application contexts, or a time interval exceeding the session break threshold, different activity segments are generated. For frames in the unlabeled state, a temporary active fragment is first attached, and the active tag and time range of the active fragment are updated after the active tag is backfilled.

8. A multimodal memory construction system for time-series screen data, characterized in that, The edge-side multimodal memory construction method for time-series screen data according to any one of claims 1 to 7 includes: Screen capture module: captures continuous screen frames, focused application, window title, and monitor information; Change-driven storage module: Based on inter-frame differences, redundant frames are filtered out, the remaining screen frames are compressed into video blocks, and video block addressing information is maintained; GUI Structured Parsing Module: Generates interface structure units, binds OCR text with application, window, area coordinates and layout relationships to form a structured text index; Dual-track indexing module: Writes structured metadata and full-text search data into the relational index track, writes multimodal embeddings into the vector index track, and maintains the frame identifier mapping between the two; Activity semantic annotation module: performs mature cluster reuse, candidate cluster promotion, aggregation of activity clusters to be annotated, and visual language model annotation, and generates activity fragments based on activity cluster labels, application context, and temporal continuity; The retrieval and evidence return module retrieves relevant screen evidence based on activity segments, structured text indexes, and vector indexes, and returns corresponding visual and text evidence based on video block addressing information.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the end-side multimodal memory construction method for time-series screen data as described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the end-side multimodal memory construction method for time-series screen data as described in any one of claims 1 to 7.

Images