Emergency-oriented graph stream data storage optimization method, device and equipment

Abstract

The invention relates to an emergency-oriented graph stream data storage optimization method, device and equipment, and relates to the technical field of graph stream data storage. The method comprises the following steps: constructing an elastic graph flow abstract structure comprising a hash address and storage block distributor, an abstract storage structure, a storage block manager, a kick-out playback strategy and a graph query interface; according to the Hash address and a linear congruence algorithm and a jump consistency Hash algorithm which are sequentially adopted by the storage block distributor, performing insertion, deletion and query operation on each edge; when the edges are inserted, the edges are redistributed according to a kick-out replay strategy, and the number of the storage blocks is adaptively expanded according to a burst coefficient of the storage block manager based on the edges; during deletion, carrying out migration and storage block deletion according to the storage block manager; and during edge query, performing multi-type query according to the graph query interface. The method is suitable for efficient and accurate storage management of the graph stream data under the emergency situation.

Description

Technical Field

[0001] This application relates to the field of image stream data storage technology, and in particular to an optimized method, apparatus and equipment for image stream data storage in the event of an emergency. Background Technology

[0002] Graph flows, as an evolutionary graph represented by an infinite sequence of edges, are becoming a popular data format in big data applications such as smart IoT devices, cybersecurity systems, and social networks. However, due to their massive scale and rapid update frequency, storing all continuously generated edges is impractical. Therefore, graph flow summarization structures that support approximate storage and management of graph flows have attracted considerable attention in recent years.

[0003] However, existing dynamic graph flow summarization structures still have significant limitations, including a lack of dynamic scalability, lack of support for deletion operations, and performance bottlenecks during structural adjustments. In particular, when faced with bursts of graph flow data, existing structures cannot adaptively adjust storage capacity, easily triggering frequent edge remapping and increasing the computational and time overhead of data storage. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, and equipment for optimizing image and stream data storage in response to the above-mentioned technical problems.

[0005] A method for optimizing graph flow data storage in response to emergencies, the method comprising: Construct a resilient graph flow digest structure, including hash addresses and storage block allocators, digest storage structure, storage block manager, kick-out replay policy, and graph query interface; wherein, the digest storage structure consists of multiple homogeneous sequentially arranged storage blocks, each storage block is a fixed-size compressed matrix composed of buckets with an equal number of rows and columns; For each edge in the input graph flow data, the candidate bucket hash address index and storage block index of the edge are generated iteratively based on the hash address and the linear congruential algorithm and the jump consistent hash algorithm used by the storage block allocator in turn, and the insertion, deletion and query operations are performed on each edge. During edge insertion, the kick-out and replay strategy is invoked to redistribute edges based on the occupancy status of buckets in the storage block, and the storage block manager is invoked to adaptively expand the number of storage blocks by calculating the burst coefficient of the edges; during edge deletion, the storage block manager is invoked to perform edge migration and storage block deletion based on the load status of the summary storage structure; and during edge query, the graph query interface is invoked to perform edge query, 1-hop successor query, 1-hop predecessor query, and path reachability query.

[0006] A graph stream data storage optimization device for handling emergencies, the device comprising: The graph stream summary building module is used to build a flexible graph stream summary structure, including hash addresses and storage block allocators, summary storage structure, storage block manager, kick-out replay strategy and graph query interface; wherein, the summary storage structure consists of multiple homogeneous sequentially arranged storage blocks, each storage block is a fixed-size compressed matrix, which is composed of buckets with an equal number of rows and columns; The storage allocation module is used to iteratively generate the candidate bucket hash address index and storage block index of each edge in the input graph flow data, based on the hash address and the linear congruential algorithm and the jump consistent hash algorithm used by the storage block allocator in turn, and to perform insertion, deletion and query operations on each edge. The edge operation module is used to redistribute edges during edge insertion by invoking the kick-out and replay strategy based on the occupancy status of buckets in the storage block, and to adaptively expand the number of storage blocks by invoking the storage block manager by calculating the burst coefficient of the edge; during edge deletion, it calls the storage block manager to perform edge migration and storage block deletion based on the load status of the summary storage structure; and during edge query, it calls the graph query interface to perform edge query, 1-hop successor query, 1-hop predecessor query, and path reachability query.

[0007] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Construct a resilient graph flow digest structure, including hash addresses and storage block allocators, digest storage structure, storage block manager, kick-out replay policy, and graph query interface; wherein, the digest storage structure consists of multiple homogeneous sequentially arranged storage blocks, each storage block is a fixed-size compressed matrix composed of buckets with an equal number of rows and columns; For each edge in the input graph flow data, the candidate bucket hash address index and storage block index of the edge are generated iteratively based on the hash address and the linear congruential algorithm and the jump consistent hash algorithm used by the storage block allocator in turn, and the insertion, deletion and query operations are performed on each edge. During edge insertion, the kick-out and replay strategy is invoked to redistribute edges based on the occupancy status of buckets in the storage block, and the storage block manager is invoked to adaptively expand the number of storage blocks by calculating the burst coefficient of the edges; during edge deletion, the storage block manager is invoked to perform edge migration and storage block deletion based on the load status of the summary storage structure; and during edge query, the graph query interface is invoked to perform edge query, 1-hop successor query, 1-hop predecessor query, and path reachability query.

[0008] The aforementioned graph flow data storage optimization method, apparatus, and device for sudden events construct a flexible graph flow summary structure including a hash address and storage block allocator, a summary storage structure, a storage block manager, a kick-out replay strategy, and a graph query interface. Based on the linear congruential algorithm and jump-consistent hash algorithm used in the hash address and storage block allocator, it can accurately locate the storage position of each edge in the graph flow data and create a uniform edge-to-storage block mapping, achieving linear space overhead adjustment proportional to the graph flow cardinality to ensure scalability and low-latency operation. Based on the summary storage structure, storage block manager, and kick-out replay strategy, it can adaptively and dynamically adjust the storage block allocation based on the graph flow scale. When the graph flow is in a burst, it can adaptively expand the number of storage blocks by calculating the burst coefficient, seamlessly expanding the storage capacity. When the graph flow scale decreases, it can delete edges and automatically reclaim underutilized storage blocks, timely reclaiming storage resources, ensuring efficient scaling and resource utilization of storage space in dynamic scenarios, and significantly improving the dynamic scalability and storage performance of graph flow data storage. Furthermore, based on the graph query interface, it can realize highly accurate graph flow data query operations, making this application applicable to efficient and accurate storage management of graph flow data in sudden events. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating a graph flow data storage optimization method for contingency situations in one embodiment. Figure 2 This is a schematic diagram of the elastic graph flow summary structure in one embodiment; Figure 3 This is a schematic diagram of the edge insertion operation in one embodiment; Figure 4 This is a schematic diagram illustrating the expansion of a storage block in one embodiment; Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0011] In one embodiment, such as Figure 1 As shown, a method for optimizing graph flow data storage in response to emergencies is provided, including the following steps: Step 1: Construct a resilient graph flow digest structure, including hash addresses and storage block allocators, digest storage structure, storage block manager, kick-out replay policy, and graph query interface; wherein, the digest storage structure consists of multiple homogeneous sequentially arranged storage blocks, each storage block is a fixed-size compressed matrix composed of buckets with an equal number of rows and columns.

[0012] Among them, the Elastic Graph Flow Summary Structure (EGSS) Opt )like Figure 2 As shown.

[0013] Step 2: For each edge in the input graph flow data, based on the hash address and the linear congruential algorithm and the jump consistent hash algorithm used by the storage block allocator in sequence, the candidate bucket hash address index and storage block index of the edge are generated iteratively, and insertion, deletion and query operations are performed on each edge.

[0014] For example, in network traffic analysis, the edges of graph flow data represent packet flows, such as packets from source IP address 110.71.87.27 to destination IP address 203.78.135.92, generated at timestamp 1641013200.090725.

[0015] It should be understood that the skip-consistent hashing algorithm is improved through the function ch(x,n), where x is the label of the data element, n is the cell count, and K is the total number of elements, aiming to uniformly map K / n elements to each cell. As the number of cells increases from n to n+1, approximately K / (n+1) elements are remapped to the new cell, using pseudo-random numbers seeded with the element ID, checking if the nth random number is less than 1 / (n+1) to determine the migration. Initially requiring n-1 comparisons, it is optimized to O(ln(n)) by skipping checks when elements might remain in their original positions. The skip-consistent hashing algorithm achieves minimal remapping through pseudo-random number generation, ensuring consistency and uniformity without additional space overhead, making it highly suitable for dynamic hash representations.

[0016] Step 3: During edge insertion, the kick-out and replay strategy is invoked to redistribute edges based on the occupancy status of buckets in the storage block, and the storage block manager is invoked to adaptively expand the number of storage blocks by calculating the burst coefficient of the edges; During edge deletion, the storage block manager is invoked to migrate edges and delete storage blocks based on the load status of the summary storage structure; And during edge query, the graph query interface is invoked to perform edge query, 1-hop successor query, 1-hop predecessor query, and path reachability query.

[0017] The aforementioned graph flow data storage optimization method for sudden events constructs a flexible graph flow summary structure, including a hash address and storage block allocator, a summary storage structure, a storage block manager, a kick-out replay strategy, and a graph query interface. Based on the linear congruential algorithm and the jump-consistent hash algorithm used in the hash address and storage block allocator, it can accurately locate the storage position of each edge in the graph flow data and create a uniform edge-to-storage block mapping, achieving linear space overhead adjustment proportional to the graph flow cardinality to ensure scalability and low-latency operation. Based on the summary storage structure, storage block manager, and kick-out replay strategy, it can adaptively and dynamically adjust the storage block allocation based on the graph flow size. When the graph flow is in a burst, it can adaptively expand the number of storage blocks by calculating the burst coefficient, seamlessly expanding the storage capacity. When the graph flow size decreases, it can delete edges and automatically reclaim underutilized storage blocks, timely reclaiming storage resources, ensuring efficient scaling and resource utilization of storage space in dynamic scenarios, and significantly improving the dynamic scalability and storage performance of graph flow data storage. Furthermore, based on the graph query interface, it can realize highly accurate graph flow data query operations, making this application applicable to efficient and accurate storage management of graph flow data under sudden events.

[0018] In one embodiment, the candidate bucket hash address index of the edge is iteratively generated and the fingerprint pairs of the nodes at both ends of the edge are calculated according to the linear congruent algorithm used by the hash address allocator; then, the storage block index of the edge is calculated according to the source node fingerprint of the edge using the jump consistent hash algorithm used by the storage block allocator, and the edge insertion, deletion and query operations are performed based on the candidate bucket hash address index and the storage block index.

[0019] Specifically, for each node First, the hash address allocator uses a linear congruential algorithm to generate nodes via a recursive formula. The seed sequence is represented as: ; in, Indicates a multiplier. Represents small prime numbers. Represents the modulus. It is a node fingerprints, It is the initial seed, { | } is the seed set. This represents the total number of iterations.

[0020] based on The nodes were calculated. The hash address is: ; in, Represents a node hash value, Indicates fingerprint length. The number of rows and columns of the storage block, and the size of each storage block is [value missing]. ; This is the initial address. Based on this formula, any address can be understood. and its iteration rounds i The initial address can be reconstructed. It can also enable the kick-out replay strategy to reassign edges to other candidate buckets.

[0021] For edges in graph flow data The hash address allocator allocates addresses to the source node. and target node implement The iteration produced a total of Individual address and Each column address generates Each candidate position is represented as a candidate bucket hash address index. Each candidate location corresponds to a candidate bucket in the storage block. Candidate bucket Store the inserted edges f-bit fingerprint pairs of the source node fingerprint and the target node fingerprint Iteration rounds and the edge weight ;in, and These represent the row address and column address, respectively, used to locate candidate buckets within a storage block; The timestamp generated for the edge.

[0022] Secondly, the skip-consistent hashing algorithm used by the storage block allocator is based on the edge... Source node fingerprint Calculate the edges Storage block index , is represented as: ; in, This represents the current number of storage blocks. Then, edge insertion, deletion, and query operations can be performed based on the candidate bucket hash address index and the storage block index.

[0023] In one embodiment, during edge insertion, the edge is reallocated according to the bucket occupancy status in the storage block using a kick-out and replay strategy. Furthermore, by calculating the edge burst coefficient, the storage block manager is invoked to adaptively expand the number of storage blocks, including: Edge insertion: For each newly arriving edge in the graph flow data, it is processed through the hash address and the storage block allocator. After each iteration, based on the generated candidate bucket hash address index and storage block index, an attempt is made to insert the edge fingerprint pair, iteration round, and edge weight into the corresponding storage block. There are 10 candidate buckets; edges of shared nodes are mapped to storage blocks in the same row or column using a row-first layout. Check if a candidate bucket is empty. If empty, insert the edge directly into storage. If not empty, further check the fingerprint pairs and iteration rounds in the candidate bucket. If the fingerprint pair and iteration round of the edge to be inserted are consistent with the fingerprint pair and iteration round of the edge stored in the candidate bucket, it indicates that the edge to be inserted is the same edge as the edge stored in the candidate bucket. In this case, the weight of the edge to be inserted is added to the weight already stored in the candidate bucket. Otherwise, it indicates that the candidate bucket is already occupied, and a kick-out and replay strategy is adopted. The edge with the smallest iteration count is selected from the candidate buckets and kicked out. The kicked-out edge is then reassigned to the spare bucket to allow for the insertion of a new edge. If the spare bucket storing the kicked-out edge is full, the kick-out replay strategy is repeated to reassign the kicked-out edge to a new spare bucket. The strategy of excluding the last inserted edge is also adopted to prevent the formation of a cycle.

[0024] For example, Figure 3 Displaying the inserted edge Example. Calculate fingerprint pairs ( a , b And assign edges to storage blocks. .by It generates four candidate positions. Since the first three buckets are occupied and do not match 'e', ​​EGSS Opt In the bucket of storage block B2 Store edge information ( a , b ,2,2,1).

[0025] Storage block expansion: When the number of kickouts reaches a preset maximum value δ, the burst coefficient of the edge is further calculated. According to the suddenness coefficient The number of new storage blocks added by the adaptive expansion of the storage block manager is calculated. And the kicked-out edges will be reallocated to new storage blocks; among them, For the time period The number of edges arriving in the internal graph stream data, and the burst time window. Indicates the typical duration of an emergency. This represents the number of rows and columns in each storage block. It should be understood that adaptively expanding the number of storage blocks based on the burst coefficient allows for determining the number of additional storage blocks and achieving adaptive scaling when graph flow data bursts. This strategy minimizes unnecessary edge remapping operations and reduces the computational and time overhead of graph flow data storage.

[0026] Specifically, when the number of new storage blocks added in the expansion is 1, the kicked-out edges will be reallocated to the new storage blocks, including: By the storage block manager, in the current summary storage structure Storage blocks B0 to B n-1 Then, add a new homogeneous storage block to the expansion. Traverse storage blocks B0 to B n-1 In the empty bucket of the Central African Republic, for any edge, if Remap this edge to In the middle, and try to move this edge to If the same position in the candidate bucket as the original storage block, If the corresponding position is empty, the migration is successful; if it is not empty, try inserting this edge. Other candidate positions in the list, until If all positions in the set are not empty, the migration fails. This edge is added to the set of affected edges, and the storage block manager then... Then add a new homogeneous storage block Remap the affected edges until all edges in the graph flow data are successfully inserted; where, The fingerprint of the source node of the edge. This uses a jump-consistent hashing algorithm. It's worth noting that the edge migration is efficient; it only requires recalculating the target block without changing the positions of buckets within the block, preserving spatial locality and improving storage efficiency.

[0027] For example, Figure 4 This demonstrates the process of expanding storage blocks. (Edge) All four candidate buckets are full, and the number of kicks exceeds the threshold δ. EGSS Opt Add new storage block B n And traverse block B0 to B n-1 The edges in the middle. For the edges If jumpHash( a , n +1)= n It is remapped to B n Then, the side that was kicked out Remapped to block jumpHash( d ,n +1).

[0028] In one embodiment, during edge deletion, the storage block manager is invoked to perform edge migration and storage block deletion based on the load status of the summary storage structure, including: Edge deletion: For each edge to be deleted in the graph flow data, the hash address and storage block allocator are used to perform the deletion. After the first iteration, based on the generated candidate bucket hash address index and storage block index, an attempt is made to remove the fingerprint pair, iteration round, and edge weight of the edge to be deleted from the corresponding storage block. Delete from candidate buckets; check the fingerprint pairs and iteration rounds in the candidate buckets. If the fingerprint pair and iteration round of the edge to be deleted are consistent with the fingerprint pair and iteration round of the edge stored in the candidate buckets, it indicates that the candidate buckets match. Clear the edge information stored in the candidate buckets and update the load factor of the calculated information after deletion of the summary storage structure. ;in, This represents the number of edges currently stored in all storage blocks of the summary storage structure. This represents the number of rows and columns in each storage block. This represents the current number of storage blocks.

[0029] Storage block deletion: If the load factor is lower than the preset threshold Traverse the last storage block B n-1 For any edge in the empty bucket, calculate its target storage block index as follows: Then, combining the candidate bucket hash address index of this edge, the edge is re-inserted into the corresponding candidate bucket of the target storage block, until B. n-1 All edges in non-empty buckets were successfully migrated to B0 to B1. n-2 After the target storage block is reached, the storage block manager deletes B. n-1 ; where, if B n-1 If any edge in B fails to be re-inserted into the target storage block, the failed edge is added to the set of affected edges constructed by the kick-out replay strategy, and the edges successfully inserted into the target storage block are restored to B. n-1 The error message indicates that the storage block deletion operation failed.

[0030] In one embodiment, during the edge query process, the graph query interface is called to perform edge query, one-hop successor query, one-hop predecessor query, and path reachability query, including: Edge queries include: for a given edge, the hash address and the storage block allocator. After each iteration, the fingerprint pair, candidate bucket hash address index, and storage block index of the given edge are calculated. Based on the candidate bucket hash address index and storage block index, the fingerprint pair and iteration round stored in all candidate buckets of the corresponding storage block are queried to see if they are consistent with the fingerprint pair and iteration round of the given edge. If they are consistent, it means that the candidate bucket is matched. By feeding back and accumulating the weights stored in the matching candidate buckets, the aggregate weight of the given edge is obtained.

[0031] A 1-hop successor lookup or a 1-hop predecessor lookup includes: for a given node, traversing the hash address and the storage block allocator. After each iteration, the fingerprint of the given node is calculated, the storage block index of the given node is identified, and the row address index is calculated. Based on the storage block index and the row address index, it is queried whether the fingerprint and iteration round stored in all candidate buckets in the corresponding storage block are consistent with the fingerprint and iteration round of the given node. If they are consistent, it means that the candidate buckets are matched. By feeding back and accumulating the outgoing edge weights or incoming edge weights stored in the matching candidate buckets, the total outgoing edge weights or the total incoming edge weights of the given node are obtained.

[0032] The path reachability query includes the following steps: For a given source node and target node, a direct connectivity check is first performed to confirm whether the source node is directly connected to the target node. If a direct connection is found, the path reachability query ends. Otherwise, a breadth-first search algorithm is used to add the source node and its unvisited neighbor nodes to a first-in-first-out (FIFO) queue. Visited nodes are tracked, and a node is removed from the head of the queue and its fingerprint is calculated. For each dequeued fingerprint, a hash address allocator is used to calculate the row address index, and a storage block allocator is used to identify the target storage block index. Based on the storage block index and row address index, the buckets in the corresponding row address of the target storage block are searched. The fingerprint and iteration count of the current node are checked against the fingerprint and iteration count stored in the bucket. If they match, a reachable path exists between the source node and the current node. The search continues until the target node is found or the queue is empty, at which point the path reachability query result from the source node to the target node is returned.

[0033] Furthermore, the EGSS constructed in this application can be verified through the following analysis. Opt Storage performance, including: 1. Memory Cost Analysis: For those with n EGSS of isomorphic compression matrices Opt The total memory cost is: ; in m This refers to the width / depth of each compression matrix, also known as the number of rows and columns. b Indicates the size of the bucket. This indicates the number of storage blocks. To ensure EGSS... OptAble to store all edges in the flow graph, assuming the storage block has an average load factor α, we obtain Here, |E| is the number of edges in the flow graph. Generally, the number of distinct edges is less than |E|. Based on this, the space complexity can be obtained as O(|E|). As a dynamic structure, EGSS Opt Adjust by adding or removing storage in response to changes in |E|. n This maintains a space complexity of O(|E|).

[0034] 2. Time Cost Analysis: Insertion, Deletion, and Kick-out Replay. For insertion or deletion of edge e... i EGSS Opt Calculate the block number block(e) in O(1) time. i Then, in block(e) i In this process, up to r×r buckets are checked, resulting in O(1+r) time complexity. 2 The time complexity is approximately O(1) because r is a small constant. Therefore, insertion and deletion are constant-time operations. If the kick-out replay operation has l kicks, then the time complexity is O(l), which is also constant since l is bounded.

[0035] EdgeQuery: For edge query EdgeQuery(e=(s,d)), EGSS Opt Computing block(e) and checking at most r×r buckets takes O(1) time, thus requiring only O(1) time complexity. This demonstrates that using efficient hash-based edge queries achieves high accuracy while maintaining constant query time complexity.

[0036] Node query: For nodes v 1-hop successor query, EGSS Opt The target storage block is located using the jump-consistent river hash algorithm, and r×m buckets are checked within that block. Since r m, the time complexity is O(m), which is usually considered O(1) for a fixed-size matrix. For a 1-hop predecessor query, EGSS Opt Traversing all n storage blocks, each block checks at most r×m buckets. For a constant m, the time complexity is O(n×m) ≈ O(n). In practice, for a bounded matrix, the time complexity of two queries is approximately O(1).

[0037] Path reachability query: The path reachability query employs a Breadth-First Search (BFS)-based approach, essentially performing multiple one-hop successor queries during graph traversal. In the best case, when a direct edge exists between the source and target nodes, the query is completed in O(1) time complexity via an initial connectivity check. However, in the worst case, a large-scale BFS exploration of the graph may be required. Since each one-hop successor query has a time complexity of O(m), and BFS may visit up to |V| nodes, the worst-case time complexity is O(m×|V|). Actual performance depends on the graph structure and path length; shorter paths and sparse graphs require fewer operations, thus the path reachability query in this application can still maintain good efficiency in most real-world scenarios.

[0038] 3. Conflict Rate Analysis: EGSS Opt Only one-sided errors occur, meaning the returned result is not less than the exact value. That is, EGSS... Opt Only false positive errors occur. If edge e i and e j A conflict occurs if and only if the other edge e j =(v,u)(s≠v or d≠u) satisfies: (1) the same fingerprint pair (v,u)(s≠v or d≠u) s = v , d = u The probability of this happening is (1 / 2). f ) 2 (2) The probability of this happening in the same candidate bucket is r. 2 / m 2 ; and (3) the same storage block, the probability of this happening is 1 / n. Note that due to the excellent uniformity of the JumpConsistentHash algorithm, each edge maps to different storage blocks with the same probability, and the above condition (3) holds. Since the JumpConsistentHash algorithm uses the source node fingerprint as a seed, condition (3) is implicit in condition (1). Therefore, edge e i The probability of an edge collision is: ; Where |E| represents EGSS Opt The number of edges in EGSS. This can be calculated. Opt The probability of a conflict between the two sides is very small.

[0039] 4. False Positive Analysis: Let ζ represent the false positive rate of edge queries. A false positive occurs when a query returns a non-existent edge e. i When =(s,d), EGSSOpt The existence of the hash collision was incorrectly reported. This situation arises from the sequential occurrence of two independent collision events. First, the stored edge e must exist. j =(v,u), whose fingerprint pairs < v , u >Fingerprint pair with the query edge< s , d >Conflict occurs because the fingerprint is an f-bit value, therefore the probability of it occurring is 1 / (2^f). f )×1 / (2 f )=1 / 2 2f Secondly, the aforementioned fingerprint collision has occurred, only when the stored edge e j and query edge e i Report false positives when at least one candidate bucket is shared. Due to edge e i It has r determined by its allocated address 2 There are candidate buckets, and edge e j It also has r 2 There are 10 candidate buckets, considering that each bucket has approximately r... 2 / m 2 The probability of being selected by two edges, therefore the probability of these two candidate bucket sets overlapping is approximately r. 2 / m 2 Due to EGSS Opt All storage blocks can store a maximum of n×m 2 There are 10 edges, and each stored edge may cause a false positive with a probability of analysis. Therefore, the upper bound of the false positive rate is: ; The false positive rate is affected by three parameters, which have different effects: fingerprint length f with respect to 2 -2f The exponential form reduces false positives, while the address allocation parameter r increases with the number of candidate positions. 2 The quadratic form of the fingerprint increases false positives, and the number of storage blocks *n* increases linearly with the number of stored edges. The interaction between *r* and *f* is particularly important because a larger *r* provides better load balancing and kick-out replay performance, but also increases the probability of collisions. Therefore, an appropriate fingerprint length *f* needs to be chosen to compensate for the increased collision risk caused by a larger *r* value. This analysis shows that EGSS... Opt The false positive rate essentially depends on the hash collision probability, not just the number of storage edges. Therefore, the fingerprint length needs to be set according to the desired false positive rate and the selected r value, with the relationship f≥log2(r)+(1 / 2)log2(n)+k / 2, where k is the target false positive rate exponent.

[0040] 5. Load Factor Analysis: For each inserted edge, EGSS... Opt Assign r candidate hash addresses to each node, thereby generating r² candidate buckets in the corresponding storage block. Analyze the successful insertion probability of the i-th edge by analyzing the occupancy status of the first (i-1) edges. Let p i This represents the probability that the first i edges can all be successfully inserted into the storage block without triggering an eviction. This probability follows a recursive relationship: ; in, Let represent the probability that all r² candidate buckets of the first i edges are occupied, assuming that the first (i-1) edges are uniformly distributed among the m² buckets. Therefore, The probability of having at least one available candidate bucket is given.

[0041] The load factor is defined as the ratio of the total number of successfully inserted edges to the total capacity m². When exactly the first (i-1) edges are successfully inserted but the ith edge fails, the load factor is: The probability of this event is... , representing the difference between the successful insertion of the first (i-1) edges and the successful insertion of the first i edges. Therefore, the expected load factor is calculated as follows:

[0042] The above equations reveal that the block size m² and the number of candidate buckets r² jointly influence the achievable load factor. Larger block sizes increase memory overhead but provide more buckets for storing edges, while increasing the number of candidate buckets per edge increases computational overhead during insertion but improves the probability of finding an available location. The trade-offs between these parameters must be carefully balanced: increasing r improves load factor performance and reduces eviction frequency, but at the cost of higher insertion time complexity and a higher false positive rate. Therefore, the optimal choice of m and r should be based on the target load factor requirement, available memory resources, and acceptable query performance trade-offs.

[0043] Furthermore, a 300-second subset of the network traffic information dataset WIDE was selected as input graph flow data, containing 2.72 million network traffic entries (source and destination IP addresses) and 4.73 million network traffic edges (timestamped communication flows), and this data was input into the EGSS constructed in this application. OptStorage performance was compared with existing graph flow summarization structures, and all experiments were conducted on a single server. Existing graph flow summarization structures include: First, GSS: a chained structure that achieves elastic capacity proportional to the graph flow size through a compressed matrix linked list. Second, Auxo: a dynamic graph flow summarization structure based on a binary tree structure. Both were extended to support edge deletion and storage block reduction for fair comparison. In the experiments, the compressed matrix size was set to the same, the Blizzard hash algorithm was used for position calculation, and the MurmurHash3 algorithm was used for fingerprint generation. A node's fingerprint is generated using the formula... Calculated, default Then, the number of hash addresses of a node is determined using a formula derived from load factor analysis.

[0044] The WIDE dataset was used to comprehensively evaluate edge insertion, query, and deletion operations to assess overall performance under real-world workload conditions. Building upon storage block reduction experiments, query operations were introduced. A query dataset was constructed by randomly selecting edges, and query operations were performed on the graph flow summary structure in chronological order to simulate query tasks encountered in real-world applications. The overall time cost results in Table 1 demonstrate that EGSS... Opt It offers superior performance compared to existing graph flow summarization structures. EGSS Opt The performance improvement achieved is directly related to its adaptive memory block adjustment mechanism, which dynamically responds to graph flow burst patterns, thereby reducing unnecessary edge remapping operations. The degree of this improvement is strongly positively correlated with the burst coefficient of the input data stream: a higher burst coefficient makes EGSS... Opt Significant performance improvements can be achieved by avoiding frequent small-scale expansions, while a lower burst factor results in more limited improvements. Comparative analysis shows that EGSS... Opt EGSS offers significant advantages over existing methods across a wide range of workload scenarios. Specifically, when processing wide datasets, EGSS... Opt Compared to GSS and Auxo, EGSS achieved time cost reductions of 15.59% and 17.80%, respectively. Its consistent performance advantage across insert, query, and delete operations demonstrates its superior performance. Opt Its architecture successfully addresses the fundamental challenges of dynamic graph flow processing, providing excellent scalability and efficiency for practical applications requiring high-throughput graph operations.

[0045] Table 1 Total Time Cost (seconds)

[0046] In one embodiment, a graph stream data storage optimization device for contingency situations is provided, comprising: The graph stream summary building module is used to build a flexible graph stream summary structure, including hash addresses and storage block allocators, summary storage structure, storage block manager, kick-out replay strategy and graph query interface; wherein, the summary storage structure consists of multiple homogeneous sequentially arranged storage blocks, each storage block is a fixed-size compressed matrix, which is composed of buckets with an equal number of rows and columns; The storage allocation module is used to iteratively generate the candidate bucket hash address index and storage block index of each edge in the input graph flow data, based on the hash address and the linear congruential algorithm and the jump consistent hash algorithm used by the storage block allocator in turn, and to perform insertion, deletion and query operations on each edge. The edge operation module is used to redistribute edges during edge insertion by invoking the kick-out and replay strategy based on the occupancy status of buckets in the storage block, and to adaptively expand the number of storage blocks by invoking the storage block manager by calculating the burst coefficient of the edge; during edge deletion, it calls the storage block manager to perform edge migration and storage block deletion based on the load status of the summary storage structure; and during edge query, it calls the graph query interface to perform edge query, 1-hop successor query, 1-hop predecessor query, and path reachability query.

[0047] Specific limitations regarding the image stream data storage optimization device for contingency situations can be found in the limitations of the image stream data storage optimization method for contingency situations described above, and will not be repeated here. Each module in the aforementioned image stream data storage optimization device for contingency situations can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0048] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for optimizing graph stream data storage in response to contingencies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0049] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0050] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps: Construct a resilient graph flow digest structure, including hash addresses and storage block allocators, digest storage structure, storage block manager, kick-out replay policy, and graph query interface; wherein, the digest storage structure consists of multiple homogeneous sequentially arranged storage blocks, each storage block is a fixed-size compressed matrix composed of buckets with an equal number of rows and columns; For each edge in the input graph flow data, the candidate bucket hash address index and storage block index of the edge are generated iteratively based on the hash address and the linear congruential algorithm and the jump consistent hash algorithm used by the storage block allocator in turn, and the insertion, deletion and query operations are performed on each edge. During edge insertion, the kick-out and replay strategy is invoked to redistribute edges based on the occupancy status of buckets in the storage block, and the storage block manager is invoked to adaptively expand the number of storage blocks by calculating the burst coefficient of the edges; during edge deletion, the storage block manager is invoked to perform edge migration and storage block deletion based on the load status of the summary storage structure; and during edge query, the graph query interface is invoked to perform edge query, 1-hop successor query, 1-hop predecessor query, and path reachability query.

[0051] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0052] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A burst-oriented graph stream data storage optimization method, characterized in that, The method comprises: constructing a resilient graph stream summary structure comprising a hash address and storage block allocator, a summary storage structure, a storage block manager, a kick-out replay strategy and a graph query interface; wherein the summary storage structure is composed of a plurality of homogenous storage blocks arranged in sequence, each storage block being a fixed-size compressed matrix composed of a plurality of buckets with equal row and column numbers; for each edge in the input graph stream data, iteratively generating a candidate bucket hash address index and a storage block index of the edge according to the linear congruential algorithm and the jump consistent hashing algorithm successively adopted by the hash address and storage block allocator, and performing insertion, deletion and query operations on each edge; in the insertion process of the edge, the kick-out replay strategy is called for re-distribution of the edge according to the occupation of the buckets in the storage block, and the storage block manager is called for adaptive expansion of the number of storage blocks by calculating the burst coefficient of the edge; in the deletion process of the edge, the storage block manager is called for edge migration and storage block deletion according to the load of the summary storage structure; and in the query process of the edge, the graph query interface is called for edge query, 1-hop successor query, 1-hop predecessor query and path reachability query.

2. The burst-oriented graph stream data storage optimization method of claim 1, wherein, iteratively generating a candidate bucket hash address index and a storage block index of the edge according to the linear congruential algorithm and the jump consistent hashing algorithm successively adopted by the hash address and storage block allocator, and performing insertion, deletion and query operations on each edge, comprising: iteratively generating a candidate bucket hash address index of the edge according to the linear congruential algorithm adopted by the hash address allocator and calculating a fingerprint pair of the nodes at both ends of the edge; then, according to the source node fingerprint of the edge, calculating the storage block index of the edge by the jump consistent hashing algorithm adopted by the storage block allocator, and performing insertion, deletion and query operations on the edge based on the candidate bucket hash address index and the storage block index.

3. The burst-oriented graph stream data storage optimization method of claim 2, wherein, iteratively generating a candidate bucket hash address index of the edge according to the linear congruential algorithm adopted by the hash address allocator and calculating a fingerprint pair of the nodes at both ends of the edge, comprising: For each node , the linear congruential algorithm employed by the hash address allocator generates a seed sequence for the node , expressed by the recursive formula: ； wherein, represents a multiplier, represents a small prime number, represents a modulus, is a fingerprint of a node is an initial seed, is a set of seeds, | is a set of seeds, is the total number of iterations; Based on , the hash address of the node is calculated as: ； wherein, represents a hash value of a node , represents a fingerprint length, is the number of rows and columns of a storage block, and the size of each storage block is ; is an initial address; For edges in graph flow data The hash address allocator allocates addresses to the source node. and target node implement The iteration produced a total of Individual address and Each column address generates Each candidate position is represented as a candidate bucket hash address index. Each candidate location corresponds to a candidate bucket in the storage block. Candidate bucket Store the inserted edges fingerprint pairs of source node fingerprints and target node fingerprints Iteration rounds and the edge weight ;in, and These represent the row address and column address, respectively, used to locate candidate buckets within a storage block; The timestamp generated for the edge.

4. The burst-oriented graph stream data storage optimization method of claim 2, wherein, calculating the storage block index of the edge according to the source node fingerprint of the edge by the jump consistent hashing algorithm adopted by the storage block allocator, comprising: The skip-consistent hashing algorithm used by the storage block allocator is based on the edge. Source node fingerprint Calculate the edges Storage block index , represented as: ； wherein, is the current number of storage blocks.

5. The burst-oriented graph stream data storage optimization method of claim 1, wherein, in the insertion process of the edge, the kick-out replay strategy is called for re-distribution of the edge according to the occupation of the buckets in the storage block, and the storage block manager is called for adaptive expansion of the number of storage blocks by calculating the burst coefficient of the edge, comprising: For each newly arrived edge of the graph stream data, the edge is hashed to obtain a hash address and a storage block allocator is used to allocate a storage block After the sub-iteration, according to the generated candidate bucket hash address index and storage block index, the fingerprint pair of the edge, the iteration round, and the weight of the edge are attempted to be inserted into the storage block corresponding to the candidate bucket; wherein the edges of the shared nodes are mapped into the storage blocks in the same row or the same column in a row-major layout. checking whether the candidate bucket is empty, if empty, directly inserting storage; if not empty, further checking the fingerprint pair and iteration round in the candidate bucket, if the fingerprint pair and iteration round of the current edge to be inserted are consistent with the fingerprint pair and iteration round of the edge stored in the candidate bucket, it indicates that the current edge to be inserted and the edge stored in the candidate bucket are the same edge, at this time, the weight of the current edge to be inserted is accumulated to the stored weight in the candidate bucket; otherwise, it indicates that the candidate bucket has been occupied, adopting the kick-out replay strategy, selecting the iteration round and the smallest edge from the candidate bucket to kick out, and re-distributing the kicked-out edge to the standby bucket to realize the insertion of the new edge; wherein, if the standby bucket storing the kicked-out edge is full, repeatedly adopting the kick-out replay strategy, re-distributing the kicked-out edge to a new standby bucket, and adopting the strategy of excluding the last inserted edge to prevent forming a cycle. checking whether the candidate bucket is empty, if empty, directly inserting storage; if not empty, further checking the fingerprint pair and iteration round in the candidate bucket, if the fingerprint pair and iteration round of the current edge to be inserted are consistent with the fingerprint pair and iteration round of the edge stored in the candidate bucket, it indicates that the current edge to be inserted and the edge stored in the candidate bucket are the same edge, at this time, the weight of the current edge to be inserted is accumulated to the stored weight in the candidate bucket; otherwise, it indicates that the candidate bucket has been occupied, adopting the kick-out replay strategy, selecting the iteration round and the smallest edge from the candidate bucket to kick out, and re-distributing the kicked-out edge to the standby bucket to realize the insertion of the new edge; wherein, if the standby bucket storing the kicked-out edge is full, repeatedly adopting the kick-out replay strategy, re-distributing the kicked-out edge to a new standby bucket, and adopting the strategy of excluding the last inserted edge to prevent forming a cycle. When the number of kicks reaches a preset maximum value δ, the burst coefficient of the edge is further calculated as According to the burst coefficient The number of new storage blocks added by the storage block manager for adaptive expansion is calculated as And the kicked edge is reassigned to the new storage block; wherein, is the number of edges arriving in the graph stream data within the time period The burst time window represents the typical burst duration, is the number of rows and columns of each storage block.

6. The burst-oriented graph stream data storage optimization method of claim 5, wherein, when the number of new storage blocks added by expansion is 1, the kick-out edge is re-distributed to the new storage block, comprising: by the storage block manager, at a current state of the summary storage structure storage blocks B0 to B n-1 After that, a new homogeneous storage block ; traverse the storage blocks B0 to B n-1 the edges in the non-empty buckets, for any edge, if , remap the edge to , and try to migrate the edge to the same position as the original storage block in , if , the same position is empty, the migration is successful; if not, try to insert the edge into other candidate positions in , until all positions in are not empty, the migration fails, the edge is added to the affected edge set, and a new isomorphic storage block is added by the storage block manager after , remapping the affected edges until all edges in the graph stream data are inserted successfully; wherein is the fingerprint of the source node of the edge, is the consistent hashing algorithm.

7. The burst-oriented graph stream data storage optimization method of claim 1, wherein, in the deletion process of the edge, the storage block manager is called for edge migration and storage block deletion according to the load of the summary storage structure, comprising: For each edge to be deleted in the graph stream data, the fingerprint pair, the iteration round and the weight of the edge are deleted from the corresponding storage block in the After the sub-iteration, according to the generated candidate bucket hash address index and storage block index, the fingerprint pair, the iteration round and the weight of the edge to be deleted are attempted to be deleted from the candidate bucket corresponding to the storage block. checking the fingerprint pair and iteration round of the current edge to be deleted and the fingerprint pair and iteration round of the edge stored in the candidate bucket, if the fingerprint pair and iteration round of the current edge to be deleted and the fingerprint pair and iteration round of the edge stored in the candidate bucket are consistent, indicating that the candidate bucket matches, clearing the information of the edge stored in the candidate bucket, and updating the load factor of the summary storage structure after the calculation information is deleted as ; wherein, represents the number of edges currently stored in all storage blocks in the summary storage structure, represents the number of rows and columns of each storage block, represents the current number of storage blocks; if the load factor is lower than a preset threshold , traverses edges in non-empty buckets in the last storage block B n-1 , for any edge, calculates its target storage block index as , then, combined with the candidate bucket hash address index of the edge, re-inserts the edge into the corresponding candidate bucket of the target storage block, until edges in all non-empty buckets in B n-1 are successfully migrated to target storage blocks between B0 and B n-2 , the storage block manager deletes B n-1 ; wherein, if any edge in B n-1 fails to be re-inserted into a target storage block, adds the edge failing to be inserted into the affected edge set constructed by the kick-out replay strategy, and restores edges successfully inserted into the target storage block to B n-1 , indicating that the storage block deletion operation fails.

8. The burst-oriented graph stream data storage optimization method of claim 1, wherein, in the query process of the edge, the graph query interface is called for edge query, 1-hop successor query, 1-hop predecessor query and path reachability query, comprising: The edge query includes: for a given edge, passing through a hash address and a storage block allocator After the sub-iteration, the fingerprint pair of the given edge, the candidate bucket hash address index and the storage block index are calculated; according to the candidate bucket hash address index and the storage block index, it is queried whether the fingerprint pairs stored in all candidate buckets of the corresponding storage block are consistent with the fingerprint pair of the given edge and the iteration round, and whether the iteration round is consistent with the iteration round, if both are consistent, it indicates that the candidate bucket matches, the weight stored in the matched candidate bucket is fed back and accumulated, and the aggregation weight of the given edge is obtained through the query. The 1-hop successor query or 1-hop predecessor query includes: for a given node, passing through a hash address and a storage block allocator After the sub-iteration, the fingerprint of the given node is calculated, the storage block index of the given node is identified and the row address index is calculated; whether the fingerprints stored in all candidate buckets in the corresponding storage block and the iteration round are consistent with the fingerprint and iteration round of the given node is queried according to the storage block index and the row address index, if they are all consistent, it indicates that the candidate buckets match, the out-edge weight or in-edge weight stored in the matched candidate buckets is fed back and accumulated, and the out-edge weight sum or in-edge weight sum of the given node is obtained through the query. The path reachability query comprises: for a given source node and a target node, first performing a direct connectivity check to determine whether the source node is directly connected to the target node, and if so, ending the path reachability query; otherwise, using a breadth-first search algorithm to add the source node and unvisited neighbor nodes of the source node to a first-in-first-out queue, and tracking visited nodes, taking a node from the head of the queue and calculating a fingerprint, for each dequeued fingerprint, using a hash address allocator to calculate a row address index, and using a storage block allocator to identify a target storage block index; searching a bucket in a corresponding row address of the target storage block according to the storage block index and the row address index, checking whether the fingerprint and iteration round of the current node match the fingerprint and iteration number stored in the bucket, and if so, indicating that there is a reachable path between the source node and the current node; traversing all nodes in the first-in-first-out queue until the target node is searched or the queue is empty, and returning a path reachability query result from the source node to the target node.

9. A burst-oriented graph stream data storage optimization apparatus, characterized by, The device comprises: a graph stream summary construction module configured to construct a resilient graph stream summary structure comprising a hash address and storage block allocator, a summary storage structure, a storage block manager, a kick-out replay strategy, and a graph query interface; wherein the summary storage structure is composed of a plurality of isomorphic storage blocks arranged in sequence, and each storage block is a fixed-size compressed matrix composed of a plurality of buckets with equal row and column numbers; a storage allocation module configured to, for each edge in input graph stream data, iteratively generate a candidate bucket hash address index and a storage block index of the edge according to a linear congruential algorithm and a jump consistent hashing algorithm adopted by the hash address and storage block allocator in sequence, and perform insertion, deletion, and query operations on each edge; an edge operation module configured to, in the insertion process of the edge, call the kick-out replay strategy to perform re-allocation of the edge according to the occupation of the buckets in the storage block, and call the storage block manager to perform adaptive expansion of the number of storage blocks by calculating the burst factor of the edge; in the deletion process of the edge, call the storage block manager to perform edge migration and storage block deletion according to the load of the summary storage structure; and in the query process of the edge, call the graph query interface to perform edge query, 1-hop successor query, 1-hop predecessor query, and path reachability query. 10.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-9. The processor, when executing the computer program, implements the steps of the method of any one of claims 1 to 8.

Images