Stream video compression system based on agent long-term memory and optimized retrieval

Abstract

The application discloses a kind of based on agent long-term memory and optimization search's streaming video compression system, including: memory graph processing module, no edge minimum maximum sampling module and edge perception weighted pruning module and time decay memory search module, the structural characteristics of the present application based on memory graph, text node in long-term memory is distinguished according to whether it is connected with entity node, and differential compression strategy is used, effectively removes redundant memory node, while maintaining the integrity of key information is significantly reduced in storage, based on time decay memory search mechanism, different time segments of memory are given dynamic weight, pay more attention to recent memory more relevant to current query in search stage, so as to alleviate the performance loss caused by memory compression, while improve retrieval efficiency.Through the above technical scheme, the application realizes efficient compression and fast retrieval of agent long-term memory under the premise of guaranteeing streaming video understanding accuracy, significantly improves the real-time performance and scalability of the system.

Description

Technical Field

[0001] This invention relates to a technology in the field of image processing, specifically a streaming video compression system based on agent long-term memory and optimized retrieval. Background Technology

[0002] With the rapid development of applications such as streaming media, real-time monitoring, and human-computer interaction, streaming video understanding has gradually become a core research direction in the fields of computer vision and multimodal intelligent agents. Unlike offline video understanding, streaming video understanding requires models to store, retrieve, and utilize historical information for reasoning and decision-making in real time under continuous input and incomplete information environments. Most existing methods focus on lexical compression and key-value caching compression, but they still struggle to guarantee the consistency of long-term entities (such as human identities) when faced with a continuous stream of large amounts of visual information. To address this, methods based on agent long-term memory have emerged. These methods model video content as a memory graph, using face and voice nodes to store human entity information, while text nodes are used to store video content. Summary of the Invention

[0003] This invention addresses the shortcomings of existing streaming video agents in long-term memory modeling, including memory size expansion, low retrieval efficiency, and performance degradation after compression, as well as the inability to effectively and specifically compress the memory graph generated by the agent. It proposes a streaming video compression system based on agent long-term memory and optimized retrieval. Based on the structural characteristics of the memory graph, text nodes in long-term memory are distinguished according to whether they are connected to entity nodes, and a differentiated compression strategy is adopted to effectively remove redundant memory nodes. While significantly reducing storage while maintaining the integrity of key information, a time-decay memory retrieval mechanism dynamically assigns weights to memories of different time segments. During the retrieval phase, recent memories more relevant to the current query are prioritized, thereby mitigating the performance loss caused by memory compression and improving retrieval efficiency. Through the above technical solution, this invention achieves efficient compression and fast retrieval of agent long-term memory while ensuring the accuracy of streaming video understanding, significantly improving the system's real-time performance and scalability.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a streaming video compression system based on agent-based long-term memory and optimized retrieval, comprising: a memory graph processing module, a borderless min-max sampling module, an edge-aware weighted pruning module, and a time-decay memory retrieval module. The memory graph processing module processes visual information from the outside world into a memory graph using an agent, and then divides the nodes into isolated text nodes and connected text nodes. The borderless min-max sampling module clusters isolated text nodes into several clusters and performs min-max sampling within each cluster. Simultaneously, the edge-aware weighted pruning module calculates the edge weight matrix and similarity matrix of connected text nodes in the memory graph, and then filters connected nodes based on the edge weight score and similarity score to achieve memory graph compression. The time-decay memory retrieval module calculates the total similarity of all text nodes within each time segment, and then uses a non-linear time decay function to decay the memory content within each time segment range to optimize memory retrieval performance.

[0006] The intelligent agent mentioned above is, but is not limited to, the open-source M3-agent (github.com / ByteDance-Seed / m3-agent).

[0007] Technical effect

[0008] This invention employs structured compression of the long-term memory graph of streaming video agents and combines it with a time-decay memory retrieval mechanism for efficient retrieval. Compared to existing technologies, this invention effectively maintains or even improves the performance of streaming video understanding and reasoning while significantly reducing memory size and retrieval overhead. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the structure of the present invention;

[0010] Figure 2 This is a schematic diagram of the intelligent agent in an embodiment;

[0011] In the figure: (a) is the agent's memory map; (b) is the agent's memory storage and retrieval.

[0012] Figure 3 This is a schematic diagram illustrating the effect of an example. Detailed Implementation

[0013] This embodiment relates to a streaming video compression system based on agent long-term memory and optimized retrieval, including: a memory graph processing module, a borderless minimum and maximum sampling module, an edge-aware weighted pruning module, and a time decay memory retrieval module.

[0014] The borderless min-max sampling module includes a spherical K-means clustering unit and a min-max sampling unit, wherein: spherical K-means clustering divides isolated text nodes into a series of clusters; and min-max sampling selects representative memory nodes from the series of clusters after division.

[0015] The edge-aware weighted pruning module includes an edge weight matrix unit, a similarity matrix unit, and a fusion unit. The edge weight matrix unit calculates the importance of each connected text node relative to the entity set. The similarity matrix unit calculates the similarity between connected text nodes. The fusion unit performs fusion filtering based on importance and similarity to obtain the most important node.

[0016] The time decay memory retrieval module includes: a sum similarity calculation unit within each segment and an exponential decay unit, wherein: the sum similarity calculation unit calculates the sum similarity between all memories within each time segment and the user query, and then calculates the number of memories to be retrieved within each segment; the exponential decay unit calculates the decayed sum similarity based on the sum similarity and determines the final retrieved memory.

[0017] like Figure 1 As shown, this embodiment illustrates a streaming video compression method based on the aforementioned system, comprising:

[0018] Step 1: Based on the connectivity of the memory graph, the memories in the memory graph are divided into two categories: isolated text nodes and connected text nodes. Specifically, this includes:

[0019] 1.1 Unbounded min-max sampling, specifically including:

[0020] i) Cluster all isolated text nodes using the spherical K-means clustering algorithm: The embedding vector of the text node. The clustering ratio is used to control the retention of isolated text nodes. For the first One cluster.

[0021] ii) Calculate the minimum and maximum distances between text nodes within each cluster, and retain the most representative text nodes, specifically: , ,in: For candidate nodes, Selected node set and The corresponding embedding vector.

[0022] iii) Repeat steps i) and ii) until the preset number of selected nodes is reached, ensuring that the furthest node is selected from each cluster, until the required number of isolated text nodes to be retained is met. So far, we have obtained the most representative isolated text nodes, where: | For the first Number of nodes within a cluster. Clustering ratio. Control the retention of isolated text nodes.

[0023] 1.2 Edge-aware weighted pruning, specifically including:

[0024] i) After constructing the edge weight matrix based on the connected text nodes obtained in step 1.1, sum the matrix row by row to obtain the edge weight score of each connected text node relative to the set of entity nodes, specifically: The number of face and voice nodes, The number of connected text nodes, weight matrix elements For the first The text node and the first Edge weight scores between individual face or voice nodes.

[0025] ii) After constructing a similarity matrix based on the connected text nodes, sum the rows of the matrix and subtract the summed matrix from the vector of all ones to obtain the similarity score of each connected text node relative to the entire set of connected text nodes, specifically: , where: similarity matrix elements For the first The text node and the first The similarity between text nodes.

[0026] iii) Combining the edge weight score and similarity score of each connected text node, calculate the final fusion score for each text node, and select the one with the highest score. One set of text nodes is retained, the remaining text nodes are pruned, and the final set of retained connected text nodes is obtained, specifically: ,in: For the first Edge weight scores for each connected text node For the first Similarity score of each text node This is the weighting balancing factor.

[0027] Step 2: Employing time-decayed memory retrieval, after calculating the total similarity of text nodes within each time segment of the memory graph, the memory content of each time segment is decayed using a non-linear time decay function to optimize memory retrieval performance. Specifically, this includes:

[0028] 2.1 Apply a nonlinear time decay function to the memory content of each time segment, specifically including:

[0029] i) Calculate the sum of similarities between the retrieval content of the M3 agent and the memories obtained in each time segment in step 1. ,in: For the first The total number of memories within a time segment, This represents the similarity between the retrieved content and the embedding vectors of each memory content.

[0030] ii) Apply time decay to the summation similarity within each time segment: ,in: / The average similarity of each time segment. The time decay coefficient, The time when the user's problem was encountered.

[0031] 2.2 Based on the average similarity of each time segment after attenuation, the number of memories to be returned is assigned to each time segment. Then, the corresponding number of memories retrieved from each time segment are concatenated and returned. Specifically: in: This represents the total number of memory entries that need to be retrieved. This mechanism ensures that the most recent memory content is retrieved, thereby guaranteeing a more accurate search.

[0032] Through practical application scenario experiments, the experimental data obtained by running the streaming video compression system based on intelligent agent long-term memory and optimized retrieval of this invention in an environment with two A100 graphics cards is shown in Table 1.

[0033] Table 1

[0034] As shown in Table 1, with 70% memory compression, this invention improved accuracy by 1% compared to the original M3 agent on three highly challenging benchmark datasets. For example... Figure 3 As shown, compared to the original M3 agent, the present invention also achieves an average speedup of 1.87 times in memory retrieval time.

[0035] Compared with existing technologies, this invention decouples the original memory graph compression and retrieval processes, efficiently compresses the memory graph based on its structure, and applies a time-decay memory retrieval mechanism to optimize the memory retrieval process. Experimental results show that the accuracy of this method on multiple benchmark datasets is significantly higher than other compression methods, and it also has a faster memory retrieval speed than the original M3 agent. On several challenging open-source streaming and long video understanding benchmark datasets, under conditions of high-proportion memory graph compression, the method of this invention can still maintain an accuracy comparable to or even better than the uncompressed system, and significantly shorten the memory retrieval time, thereby improving the real-time performance, stability, and scalability of the system, verifying the effectiveness and practical value of the technical solution of this invention.

[0036] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A streaming video compression system based on agent-based long-term memory and optimized retrieval, characterized in that, include: The system comprises a memory graph processing module, a borderless min-max sampling module, an edge-aware weighted pruning module, and a time-decay memory retrieval module. Specifically: the memory graph processing module processes visual information from the outside world into a memory graph through an intelligent agent, then divides the nodes into isolated text nodes and connected text nodes; the borderless min-max sampling module clusters isolated text nodes into several clusters and performs min-max sampling within each cluster; simultaneously, the edge-aware weighted pruning module calculates the edge weight matrix and similarity matrix of connected text nodes in the memory graph, and then filters connected nodes based on the edge weight score and similarity score to achieve memory graph compression; the time-decay memory retrieval module calculates the total similarity of all text nodes within each time segment, and then uses a non-linear time decay function to decay the memory content within each time segment to optimize memory retrieval performance.

2. The streaming video compression system based on agent long-term memory and optimized retrieval according to claim 1, characterized in that, The borderless min-max sampling module includes a spherical K-means clustering unit and a min-max sampling unit, wherein: spherical K-means clustering divides isolated text nodes into a series of clusters; and min-max sampling selects representative memory nodes from the series of clusters after division.

3. The streaming video compression system based on agent long-term memory and optimized retrieval according to claim 1, characterized in that, The edge-aware weighted pruning module includes an edge weight matrix unit, a similarity matrix unit, and a fusion unit. The edge weight matrix unit calculates the importance of each connected text node relative to the entity set. The similarity matrix unit calculates the similarity between connected text nodes. The fusion unit performs fusion filtering based on importance and similarity to obtain the most important node.

4. The streaming video compression system based on agent long-term memory and optimized retrieval according to claim 1, characterized in that, The time decay memory retrieval module includes: a sum similarity calculation unit within each segment and an exponential decay unit, wherein: the sum similarity calculation unit calculates the sum similarity between all memories within each time segment and the user query, and then calculates the number of memories to be retrieved within each segment; the exponential decay unit calculates the decayed sum similarity based on the sum similarity and determines the final retrieved memory.

5. A streaming video compression method based on agent-based long-term memory and optimized retrieval, using the system described in any one of claims 1-4, characterized in that... include: Step 1: Based on the connectivity of the memory graph, the memories in the memory graph are divided into two categories: isolated text nodes and connected text nodes. Specifically, this includes: 1.1 Unbounded min-max sampling; 1.2 Edge-aware weighted pruning; Step 2: Employing time-decayed memory retrieval, after calculating the total similarity of text nodes within each time segment of the memory graph, the memory content of each time segment is decayed using a non-linear time decay function to optimize memory retrieval performance. Specifically, this includes: 2.1 Apply a nonlinear time decay function to the memory content of each time segment; 2.2 Based on the average similarity of each time segment after attenuation, the number of memories to be returned is assigned to each time segment. Then, the corresponding number of memories retrieved from each time segment are concatenated and returned. Specifically: in: This mechanism ensures that the most recent memory content is retrieved, thus guaranteeing a more accurate search, based on the total number of memory content entries that need to be searched.

6. The streaming video compression method based on agent long-term memory and optimized retrieval according to claim 5, characterized in that, The aforementioned unbounded minimum-maximum sampling specifically includes: i) Cluster all isolated text nodes using the spherical K-means clustering algorithm: The embedding vector of the text node. The clustering ratio is used to control the retention of isolated text nodes. For the first One cluster; ii) Calculate the minimum and maximum distances between text nodes within each cluster, and retain the most representative text nodes, specifically: , ,in: For candidate nodes, Selected node set and The corresponding embedding vector; iii) Repeat steps i) and ii) until the preset number of selected nodes is reached, ensuring that the furthest node is selected from each cluster, until the required number of isolated text nodes to be retained is met. So far, we have obtained the most representative isolated text nodes, where: | For the first Number of nodes within a cluster, clustering ratio Control the retention of isolated text nodes.

7. The streaming video compression method based on agent long-term memory and optimized retrieval according to claim 5, characterized in that, The edge-aware weighted pruning specifically includes: i) After constructing the edge weight matrix based on the connected text nodes obtained in step 1.1, sum the matrix row by row to obtain the edge weight score of each connected text node relative to the set of entity nodes, specifically: The number of face and voice nodes, The number of connected text nodes, weight matrix elements For the first The text node and the first Edge weight scores between individual face or voice nodes; ii) After constructing a similarity matrix based on the connected text nodes, sum the rows of the matrix and subtract the summed matrix from the vector of all ones to obtain the similarity score of each connected text node relative to the entire set of connected text nodes, specifically: , where: similarity matrix elements For the first The text node and the first Similarity between text nodes; iii) Combining the edge weight score and similarity score of each connected text node, calculate the final fusion score for each text node, and select the one with the highest score. One set of text nodes is retained, the remaining text nodes are pruned, and the final set of retained connected text nodes is obtained, specifically: ,in: For the first Edge weight scores for each connected text node For the first Similarity score of each text node This is the weighting balancing factor.

8. The streaming video compression method based on agent long-term memory and optimized retrieval according to claim 5, characterized in that, Step 2.1 specifically includes: i) Calculate the sum of similarities between the agent's retrieved content and the memories obtained in each time segment in step 1. ,in: For the first The total number of memories within a time segment, The similarity between the content retrieved by the model and the embedding vectors of each memory content; ii) Apply time decay to the summation similarity within each time segment: ,in: / The average similarity of each time segment. The time decay coefficient, The time when the user's problem was encountered.

Images