Field granularity storage layout optimization method for JSON data incremental version management

Abstract

This invention provides a field-granular storage layout optimization method for incremental version management of JSON data, belonging to the field of data version management. The method includes: S1, constructing a storage graph model that supports field-granular materialization; S2, converting the version graph into a storage graph with minimized total storage overhead based on the storage graph model; S3, if the minimized total storage overhead is less than the storage space constraint, performing optimization operations on all versions at both the field and version granularity levels to generate a set of candidate change edges for optimizing the storage graph; S4, introducing user access preferences as a moderating factor to evaluate the value of all change edge sets; S5, adding the change edge sets sequentially to the storage graph in descending order of value without violating storage space constraints to update the storage graph. This invention can correctly adapt to user preferences for full version queries and historical queries for specific fields while ensuring that the total storage overhead is less than the storage space constraint.

Description

Technical Field

[0001] This invention relates to the field of data version management technology, and in particular to a field-granular storage layout optimization method for incremental version management of JSON data. Background Technology

[0002] In today's rapidly developing information technology landscape, data has become a crucial asset for all types of enterprises and organizations. JSON (JavaScript Object Notation), as a semi-structured data format, has been widely adopted in various fields such as machine learning, scientific computing, and engineering applications due to its high readability and flexible support for nested structures. Traditional version control systems (VCS) are primarily geared towards file version management and lack optimization for JSON data, making it difficult to support fine-grained querying of data and its historical versions. Document-oriented databases (DODB), while possessing excellent data query capabilities, do not support automatic version control, typically requiring the construction of a dedicated version structure during the data modeling phase and manual maintenance of the evolution process, which can easily lead to redundancy and errors. Temporal databases are suitable for linear time queries but struggle to adapt to the non-linear version management needs of JSON data, such as structural changes and branching evolution.

[0003] However, JSON data is often highly dynamic in large-scale and complex systems. Taking deep learning scenarios as an example, training data, model parameters, and interface inputs may frequently generate new versions during the preprocessing, model iteration, and inference deployment phases. In this context, incrementally storing multiple versions of JSON data is significant, as it saves storage overhead while preserving the data evolution process. However, relying solely on incremental storage can lead to historical queries requiring the backtracking of numerous changed data, impacting query performance.

[0004] One feasible approach is to optimize storage layout based on materialization. This involves creating snapshots of certain fields for some critical versions of incremental storage to reduce the computational burden of backtracking incremental data. A reasonable materialization strategy can strike a balance between storage and query efficiency, but choosing which versions to materialize is crucial. If too few materialize versions, queries will still require backtracking a large amount of incremental data, impacting query performance; if too many materialize versions, it will consume excessive storage resources, diminishing the advantages of incremental storage.

[0005] Materialized version storage strategies explore how to materialize certain versions—that is, to fully store partial versions—to construct a storage graph, thereby improving the performance of version control systems in terms of storage, version switching, version commits, and cross-version queries and operations. The goal of these methods is to build an optimized physical structure based on the logical relationships between versions. Version nodes without in-degree are fully stored versions, i.e., materialized nodes, meaning these versions are stored completely and can be directly accessed without depending on data from other versions. Nodes with in-degree are incrementally stored versions, indicating that these versions store data that differs from their predecessor, requiring the application of these differences to reconstruct the complete version data.

[0006] Most materialized version storage strategies focus on archiving and rebuilding linear version chains, making them ill-suited to the non-linear, multi-branch version graph structures prevalent in JSON data. For example, the semantic merging strategy proposed by Buneman et al. is suitable for XML archiving, and Seering et al. optimized version storage for array-type databases based on minimum spanning trees, but neither supports branch merging or field-level queries. The LMG algorithm proposed by Bhattacherjee et al. is built on a balanced framework of global minimum spanning trees and shortest path trees, which can balance storage and rebuilding overhead to some extent, but its optimization granularity remains at the document level and does not model field-level access preferences. Zhou et al. and Guo et al. further improved the insertion and evaluation strategies for rebuilding edges based on heuristic optimization methods, but still failed to effectively handle non-uniform query behavior at the field level.

[0007] Based on the above needs, choosing the appropriate materialized version is key to optimizing storage layout. The selection of the materialized version typically needs to consider the following three issues:

[0008] First, since storage space is usually constrained and snapshots cannot be created without limit, the storage layout must be optimized within a limited space.

[0009] Secondly, users often access data not through global queries, but through local field queries. If the storage layout is not optimized for field-level queries, it may require backtracking and rebuilding the entire JSON document during the query, resulting in unnecessary computational overhead.

[0010] Furthermore, user access preferences also play a crucial role in storage layout optimization. Some users focus on minimizing the overall cost of a complete query, aiming to minimize the overhead of querying any version of a data when storage resources are limited; while other users, whose data access is often unevenly distributed, are more concerned with the cost of queries that access data frequently.

[0011] Therefore, the selection of materialized versions should, under the premise of meeting storage space constraints, take into account user access preferences and achieve an optimal trade-off between the overhead of global full version queries and historical query overhead. To this end, the system needs to adopt a flexible storage layout optimization strategy to dynamically adjust the location and number of materialized versions to adapt to different query patterns and resource limitations, thereby improving system performance. Summary of the Invention

[0012] To address the technical challenge of achieving an optimal trade-off between the overhead of globally complete version queries and the overhead of querying specific fields' historical data while meeting storage space constraints and considering user access preferences, this invention provides a field-granular storage layout optimization method for incremental version management of JSON data. The technical solution is as follows:

[0013] On the one hand, a field-granular storage layout optimization method for incremental version management of JSON data is provided, including:

[0014] S1. Construct a storage graph model that supports field-level materialization;

[0015] S2. Based on the constructed storage graph model, the version graph is converted into a storage graph with minimized total storage overhead using the MCA algorithm.

[0016] S3. If the total storage overhead is less than the storage space constraint, then perform optimization operations on all versions at both the field granularity and version granularity levels to generate a set of candidate change edge sets for optimizing the storage graph. The generated set of candidate change edge sets includes: fully materialized change edge sets, field-granular materialized change edge sets built based on query load, direct incremental change edge sets, and change edge sets removed by the MCA algorithm.

[0017] S4. Introduce user access preferences as a moderating factor to evaluate the value of all variable edge sets;

[0018] S5. In descending order of value, without violating storage space constraints, add the changing edge set to the storage graph that minimizes the total storage overhead to update the storage graph.

[0019] Furthermore, the storage graph model adopts a storage graph representation based on edge attribute graphs, which supports incremental representation at the field level; among which, it adopts... This indicates a storage graph that supports field-level materialization. These represent the set of nodes in the stored graph, the set of edges in the stored graph, and the set of incremental JSON data for each version, respectively.

[0020] Furthermore, the storage graph representation based on edge attribute graphs includes:

[0021] for ,definition ;in, For storage graph A version node in a document represents a version. It can be reconstructed using different preceding versions and different incremental JSON data. Representing the storage graph The set of nodes; For version A set of incremental JSON data, representing a set of data that can be used to rebuild versions. Incremental JSON data; Indicates from version To version Incremental JSON data;

[0022] Benchmark for JSON data increment This is used to indicate the incremental JSON data it depends on, rather than directly relating the node connections in the storage graph; where, for indicating version JSON data increment ,if If it is unmaterialized, then the version The version node corresponding to the incremental JSON data pointed to by the preceding node in the storage graph; if If it is fully materialized or its base is empty, then the version The preceding node in the storage graph is ;if If it is materialized at the field granularity, then the version The preceding node in the storage graph also contains and The corresponding version nodes, the former used to indicate the source of its materialized portion, and the latter used to provide the version context for the remaining non-materialized portions; among them, empty nodes... It represents an empty document and is the preceding node for all materialized versions;

[0023] An attributed edge representation is used to characterize the structural representation of incremental JSON data in the storage graph; where, for an edge in the storage graph... , Representing the storage graph The set of edges is represented using a quadruple. , Indicates the starting point of the edge, i.e., the previous version; This indicates the endpoint of the edge, i.e., the target version; This represents the increment of the JSON data corresponding to the edge; This represents the version change operation contained in the edge; each edge This indicates that in the previous version Based on this, apply incremental JSON data. Version change operation , obtain the target version Data status;

[0024] Define the set of variable edges , Indicates all increments of JSON data The set of associated edges, express of Attributes; during layout optimization, the set of edges is changed. As an operational atomic unit, it must be processed as a whole when added or deleted; that is, it must either be retained or... All edges in the graph, or delete all edges; for a stored graph that has already been optimized. Each version node can be represented by at most one JSON data increment, i.e. .

[0025] Furthermore, the MCA algorithm is the Zhu-Liu / Edmonds algorithm;

[0026] The process of converting the version graph into a storage graph with minimized total storage overhead based on the constructed storage graph model and the MCA algorithm includes:

[0027] Based on the constructed storage graph model and version graph, construct a dictionary. , Save all The set of changing edges The mapping of its storage overhead; where, Indicates all increments of JSON data The set of associated edges, Indicates incremental JSON data Storage overhead, For version An incremental collection of JSON data;

[0028] Based on dictionary The version graph is converted into a weighted directed acyclic graph (DAG), and the minimum-cost tree graph is obtained through the Zhu-Liu / Edmonds algorithm, which is the storage graph that minimizes the total storage overhead. In DAG, Seen as a direction The edge weight is .

[0029] Furthermore, the construction of the field-granular materialized change edge set based on query load includes:

[0030] Based on query load, the incremental JSON data corresponding to the query version is materialized at the field granularity with the query path as the target; the query load includes: query version, query path, and query frequency;

[0031] Specifically, a prefix check strategy is used to construct a materialized change edge set at the field granularity. If the query path is already covered by the current processing path or is a prefix of an already materialized path, the generation of the change edge set for that query path is skipped.

[0032] Furthermore, generating the direct incremental change edge set includes:

[0033] The current version and its previous materialized ancestor node The version sequence between them is compressed to form a new JSON data increment, which is then... As a baseline; for incremental JSON data Let the pointer iterative access Preorder node Is the preceding version fully materialized? If so, return [value]. arrive All JSON data incremental sequences Otherwise Set it as its predecessor version and repeat the above iterative operation; merge the final JSON data increment sequence. Used to represent Incremental JSON data directly reconstructed.

[0034] Furthermore, the evaluation of the value of all changed edge sets includes:

[0035] Determine the evaluation function for the variable edge set. Using the evaluation function Calculate the value of adding each modified edge set from the candidate modified edge set to a storage graph that minimizes the total storage overhead; where the evaluation function is... Represented as:

[0036]

[0037] in, Indicates value; This indicates the increase in storage overhead after adding the corresponding modified edge set; This represents the reduction in the cost of complete reconstruction after adding the corresponding modified edge set; This indicates the reduction in historical query overhead for a specific field after adding the corresponding modified edge set; Represents the normalization function; Representing the storage graph as This represents the number of child nodes of the root node, used to reflect the impact of modifying the current version on all versions contained in the search path; and These are the full reconstruction cost weight and the historical query cost weight for specific fields, which are used to control the different contributions of different costs to the value of the variable edge set under different user access preferences. , , Representing versions respectively Storage overhead, version Full reconstruction cost, version The overhead of historical queries for specific fields.

[0038] Furthermore, version Storage overhead Represented as:

[0039]

[0040] in, Indicates incremental JSON data Storage overhead, For version An incremental collection of JSON data;

[0041] Version Full reconstruction cost Represented as:

[0042]

[0043] in, Minimize the function; Indicates incremental JSON data The total reconstruction cost;

[0044] Version query overhead Represented as:

[0045]

[0046] in, Indicates incremental JSON data Query overhead; This indicates the query load.

[0047] Furthermore, the method also includes:

[0048] Get its current version, and use it as the predecessor version and branch to which the new version belongs;

[0049] Registered users submit incremental JSON data for a new version, update the current version information to the newly submitted version, and update the version graph to indicate the logical relationship between the new version and existing versions;

[0050] After the version layout optimization time is reached, the operation steps S1-S5 are called to generate a new storage graph for the version graph;

[0051] Based on the results of the generated new storage graph, the corresponding JSON data increments are materialized / dematerialized and saved to JVD, where JVD represents a multi-granularity version control system for JSON data.

[0052] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for optimizing the field-granular storage layout for incremental version management of JSON data.

[0053] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0054] This invention provides a field-granularity storage layout optimization method for incremental version management of JSON data. It supports query loads based on version graphs and version history as input, fully leveraging the ability of incremental JSON data to support both full materialization and field-granularity materialization. While meeting storage space constraints, it improves access performance at both the field and version granularity levels at the cost of storage overhead. Furthermore, it introduces user access preferences as a moderating factor, achieving a balance between the overhead of full version queries (also known as full reconstruction overhead) and the overhead of specific field history queries. This method expands the applicability of existing materialization optimization techniques in semi-structured, multi-version scenarios, improving system response performance and resource utilization efficiency in real-world query environments. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart of a field-granular storage layout optimization method for incremental version management of JSON data provided in an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of a multi-granularity version management system structure for JSON data provided in an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of a version submission process provided by an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of a historical query process provided by an embodiment of the present invention. Detailed Implementation

[0060] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0061] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0062] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0063] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0064] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0065] This invention provides a method for optimizing the field-granular storage layout for incremental version management of JSON data, such as... Figure 1 The flowchart shown is for a field-granular storage layout optimization method for incremental version management of JSON data. The processing flow of this method may include the following steps:

[0066] S1. Construct a storage graph model that supports field-level materialization;

[0067] In practical applications, storage space is often limited, necessitating a balance between storage overhead and system performance. However, existing research often focuses on single metrics and fails to adequately consider user access patterns. In reality, user queries tend to exhibit biases, such as frequent access to specific versions or optimization needs for specific query loads. Therefore, introducing materialization-based storage layout optimization is crucial.

[0068] Define version diagram It is a directed acyclic graph (DAG) where an edge represents a derivation relationship between two versions. (Storage graph) It's also a directed graph used to represent the storage layout for incremental data across multiple versions. (Similar to a version graph) The difference lies in the version diagram. It describes the fixed logical relationships between versions, while the storage graph... It reflects the current storage status and may change as storage policies are adjusted.

[0069] Version materialization refers to selectively storing incrementally reconstructed data snapshots to accelerate version history queries. Leveraging the field-level materialization mechanism of JSON data, materialization can cover the entire document or only specific fields. Due to differences in materialization locations, the same version graph can correspond to multiple storage graphs. By materializing only a portion of the nodes of the complete document, the storage overhead of materializing infrequently accessed parts is reduced, providing a foundation for optimizing storage graphs for field-level query loads. In the case of field-granular materialization, a node in the storage graph may simultaneously contain both incremental and materialized representations. Therefore, the first step is to design a storage graph representation method that supports field-granular materialization to flexibly adapt to field-level query requirements.

[0070] For a given version of the storage graph The following storage options are available:

[0071] 1) Incremental storage: No optimization is performed.

[0072] 2) Fully materialized storage: The rebuilt document will be stored completely.

[0073] 3) Field-level materialized storage: Only the target path is materialized, while the rest is still stored in incremental form.

[0074] In different application scenarios, user query preferences significantly influence the choice of optimization methods and locations: In data tracing scenarios, users focus more on queries of specific fields to accurately track key data; this query pattern is typically reflected in historical query load. In data recovery or random query scenarios, users prefer to perform a complete reconstruction of random versions to ensure the system can access data from any historical point in time. These two scenarios reflect users' preferences for local optimization of historical access and balanced optimization of version queries, respectively. In practical applications, user access behavior often exhibits mixed characteristics, meaning there is both frequent access to a few fields and extensive access to multiple versions. Therefore, how to flexibly balance different preferences under limited resources and construct a multi-version layout with both query access efficiencies becomes a key challenge in version storage optimization.

[0075] Based on the above problems, the objective of this embodiment is to optimize the storage graph to correctly adapt to user preferences for full version queries and historical queries for specific fields, while ensuring that the total storage overhead is less than the storage space constraint. Regarding versions on the storage graph... This embodiment evaluates the overhead from the following three dimensions to determine which of the above storage methods should be selected. :

[0076] 1. Storage overhead The storage space required for the version on the hard drive.

[0077] 2. Cost of querying the full version The time cost of a complete version reconstruction; the cost of querying the complete version is also called the cost of a complete reconstruction.

[0078] 3. Cost of historical queries for specific fields Given a set of query loads {<query version, query path, query frequency>, ...}, performs query load on the version. The total time cost required; among which, the historical query cost for a specific field is referred to as the historical query cost.

[0079] In this embodiment, the query path is used to represent a specific field.

[0080] To better represent the dispersion of queries, the overhead here is the overhead required when the result data is not cached.

[0081] Formalistically, the research problem in this embodiment is as shown in Formula 1:

[0082]

[0083] in, It is a storage diagram. Represents the set of all feasible storage graph layout schemes; This is the final storage graph obtained after optimization; and These are the full reconstruction cost weight and the historical query cost weight factors, used to reflect user access preferences; This indicates the system's storage space constraints; Representing the storage graph The set of nodes; A version node in the storage graph represents a version; , , These represent storage overhead, full version query overhead, and query overhead, respectively.

[0084] This problem is an MSR (Min-Sum Retrieval) problem. Existing research has sufficiently demonstrated that MSR problems are NP-hard; therefore, a heuristic method is needed to generate the memory graph.

[0085] Traditional storage graphs typically use complete documents as the version granularity and represent incremental relationships through edges. For example, in a storage graph, a node with in-degree represents incremental storage, while a node without in-degree represents materialized storage. In this structure, querying a version usually means accessing the complete data. However, querying JSON data involves not only accessing the complete document but also accessing specific fields. Existing storage graph structures cannot effectively support the latter because their incremental or materialization strategies are based on the document level and lack optimization for field-level access.

[0086] Therefore, in this embodiment, a storage graph model supporting field-level materialization will be provided based on the field-level materialization technology available in JSON data increments. .

[0087] Table 1 Key Parameter Description

[0088]

[0089] To address the aforementioned issues, this embodiment proposes a storage graph model based on edge-attribute graphs. This representation supports incremental representation at the field level, overcoming the limitations of traditional methods that only support document-level incremental or materialized representations. Consider the storage graph. The key parameters are defined as shown in Table 1. To ensure the connectivity of the storage graph and simplify the optimization process, [further details are needed]. Introduce an empty node , indicates an empty document.

[0090] for This can be reconstructed using different preceding versions and different incremental JSON data. Therefore, the following definition is provided. Indicates a set of versions that can be rebuilt. The incremental JSON data, where each This is a possible JSON data increment, representing a change from version [version number missing]. To version The incremental JSON data.

[0091] Benchmark for JSON data increment This is used to indicate the incremental JSON data it depends on, rather than directly relating the node connections in the storage graph. Specifically, for representing versions... JSON data increment ,if If it is unmaterialized, then the version The version node corresponding to the incremental JSON data pointed to by the preceding node in the storage graph; if If it is fully materialized or its base is empty, then the version The preceding node in the storage graph is ;if If it is materialized at the field granularity, then the version The preceding node in the storage graph also contains and The corresponding version nodes are used to indicate the source of their materialized parts and to provide the version context for the remaining non-materialized parts.

[0092] Therefore, a JSON data increment often corresponds to multiple edges in the storage graph, rather than a single edge structure. To accurately depict the structural representation of a JSON data increment in the storage graph, this embodiment uses an edge representation with attributes. Use a quadruple to represent .in, Indicates the starting point of the edge. Indicates the endpoint of the edge. This indicates the increment of the JSON data corresponding to that edge. This indicates the version change operation contained in the edge. Each edge represents the version change operation in the previous version. Based on this, apply incremental JSON data. Version change operation The target version can be obtained. The data status.

[0093] To uniformly manage edges related to incremental JSON data, a set of variable edges is defined. This indicates all increments related to JSON data. The set of associated edges, express of Attributes. During layout optimization, the set of edges is changed. As an operational atomic unit, it must be processed as a whole when added or deleted; that is, it must either be retained or... All edges in the graph, or delete all edges. For a stored graph that has already been optimized. Each version node can be represented by at most one JSON data increment, i.e. .

[0094] S2. Based on the constructed storage graph model, the version graph is transformed into a storage graph with minimized total storage overhead using the MCA algorithm. In this embodiment, an incremental cost model for JSON data is first given, and based on this, a cost model for a single version on the storage graph is given. A cost model is a mathematical model used to quantify system overhead, which helps the system evaluate the performance and economy of different solutions during the design, optimization, and decision-making process.

[0095] In this embodiment, since each set of changing edges directly corresponds to the JSON data increment, before discussing the calculation method of each version's various costs, it is necessary to first give the cost model of the JSON data increment.

[0096] 1) Storage overhead Thanks to the incremental representation method, the storage overhead of the JSON data increment D is equivalent to the number of bytes of the result of D serialization.

[0097] 2) Total reconstruction costs The reconstruction cost is affected by multiple factors such as I / O and communication, making it difficult to model directly. The complete reconstruction of incremental JSON data is related to the total number of path nodes traversed during the collection and application of version change operations; therefore, this can be used as a metric.

[0098] 3) Historical query overhead : The total query cost is related to the query load. When calculating the total query cost, it is necessary to first calculate the query cost of each query path. Indicates in version Query target path The query cost. Given a set of query loads, the total query cost is shown in Formula 2:

[0099]

[0100] in, Represents a set of query loads, where To query the version, indicating the number. The target version number corresponding to each query; Indicates the first The query path for each query; Query frequency, representing The Middle The frequency of each query access. The higher the access frequency, the higher the contribution to the total query overhead of the query load.

[0101] Cost model based on incremental JSON data, for version and its potential JSON data increment set (Abbreviation: Variation Set), which can give the following version of the cost model:

[0102] 1) Storage overhead :Version The storage overhead is equivalent to the total storage overhead of the incremental JSON data in its potential change set. The more incremental JSON data is stored, the more data is saved for representation. The more incremental JSON data there is, the more storage space it will occupy. The calculation is shown in Formula 3:

[0103]

[0104] 2) Total reconstruction costs :Version The full reconstruction cost is equivalent to the minimum reconstruction cost of the incremental JSON data in its potential set of changes. The calculation is shown in Formula 4:

[0105]

[0106] 3) Historical query overhead Similar to the overhead of a full rebuild, version The query overhead is equivalent to the total query overhead that minimizes the query load on that version by the incremental JSON data in its potential change set. The calculation is shown in Formula 5:

[0107]

[0108] Optimize storage graph There are two main challenges. First, existing research has shown that this is an NP-hard problem with a solution space that exhibits a super-exponential relationship with the number of versions, making it impossible to search for the optimal storage layout within a reasonable timeframe. Second, there is no clear correlation between the optimization objective and the constraints, making it difficult to reduce the problem's complexity.

[0109] This embodiment achieves the optimization task through steps S2-S5. First, a storage graph with the minimum total storage overhead is constructed. If the total storage overhead is still less than the storage constraint, specific optimization operations are performed on certain versions at both the field granularity and version granularity levels to generate candidate change edge sets for optimizing the storage graph. Finally, user access preferences are introduced as a regulating factor to evaluate the value of all change edge sets, and the storage graph is updated without violating storage space constraints, thereby achieving the optimization task.

[0110] In this embodiment, the MCA algorithm is the Zhu-Liu / Edmonds algorithm.

[0111] In this embodiment, in order to obtain a storage graph that minimizes total storage overhead, a dictionary is first constructed based on the constructed storage graph model and version graph. , Save all The set of changing edges And its mapping to storage overhead. Even The starting points of the edges in the equation may be different, but their ending points are the same, i.e. The minimum cost tree graph (MCA) focuses on the endpoints of edges rather than their origins, ensuring that every node is reachable. Therefore, it can be based on a dictionary. The version graph is transformed into a regular weighted directed acyclic graph (DAG), whose minimum-cost tree graph can be obtained using the Chu-Liu / Edmonds algorithm. Since the version graph is a DAG, its minimum-cost tree graph cannot be a cyclic graph. In the actual algorithm execution, the cycle detection steps were removed. The Chu-Liu / Edmonds algorithm requires a starting point to be specified during execution; the previous storage graph model introduced nodes... This node is reachable from any other node in the graph, therefore... Starting with storage overhead as the weight, a storage graph that minimizes total storage overhead can be obtained with only one execution. .

[0112] S3. If the total storage overhead is less than the storage space constraint, then perform optimization operations on all versions at both the field granularity and version granularity levels to generate a set of candidate change edge sets for optimizing the storage graph. The generated set of candidate change edge sets includes: fully materialized change edge sets, field-granular materialized change edge sets built based on query load, direct incremental change edge sets, and change edge sets removed by the MCA algorithm.

[0113] In this embodiment, the generated set of candidate modified edges for optimizing the storage graph will be used to accelerate version queries and version reconstruction. In fact, any two versions... , There is one between them Version arrive The changes are complex. Exploring all potential variable edge sets is computationally too complex, and many variable edge sets do not significantly improve performance. Therefore, this study adopts a greedy strategy, focusing only on the following four methods for generating variable edge sets:

[0114] 1) Fully materialize the change edge set: The JSON data increment corresponding to the version will be materialized with the document root node as the target path.

[0115] 2) Constructing a field-granular materialized change edge set based on query load: Based on query load, the incremental JSON data corresponding to the query version is materialized at the field granularity with the query path as the target; the query load includes: query version, query path and query frequency.

[0116] Consider a scenario where the query path has already been covered or materialized in the current JSON data increment. Continuing to materialize it would lead to redundancy, thus requiring conflict detection. This embodiment employs a prefix checking strategy: if the path has been covered by the currently processed path or is a prefix of an already materialized path, the generation of the modified edge set for that path is skipped.

[0117] 3) Direct incremental change edge set: the current version and its previous materialized ancestor node. The version sequence between them is compressed to form a new JSON data increment, which is then... Based on this, the target version for reconstruction was reduced. The number of previous versions that need to be accessed at any time is reduced, improving reconstruction speed while still storing data in incremental form. To reduce storage overhead. For incremental JSON data. Let the pointer iterative access Preorder node Is it fully materialized? If so, return... arrive All JSON data incremental sequences Otherwise Set it to its predecessor version and repeat the above steps. Merge the final incremental sequence of JSON data. Used to represent Incremental JSON data directly reconstructed.

[0118] 4) The set of modified edges removed by the MCA algorithm: This is used in constructing a storage graph that minimizes the total storage overhead. During this process, some edges are removed because they incur higher storage costs compared to others. However, this does not mean they fail to provide improvements in storage graph reconstruction and query performance. Therefore, all edges removed by the MCA algorithm are retained for use in subsequent steps, despite the potential additional storage overhead.

[0119] The JSON data increments obtained by the four methods described above will ultimately be stored in the candidate change edge set. In the next step, it is used for evaluation and added to the storage graph.

[0120] In this embodiment, the field granularity corresponds to the materialized change edge set constructed based on the query load, and the version granularity corresponds to the fully materialized change edge set, the directly incremental change edge set, and the change edge set removed by the MCA algorithm.

[0121] S4. Introduce user access preferences as a moderating factor to evaluate the value of all variable edge sets;

[0122] In this embodiment, local modifications to a particular version of the modified edge set can impact the overall overhead of the storage graph. This impact is not always a case of increased storage overhead or decreased overhead for complete reconstruction / historical query. This embodiment first discusses the potential impact on storage graph performance in detail, considering the incremental JSON data and the aforementioned candidate modified edge sets.

[0123] , Adding this modified edge set increases storage overhead without improving rebuilding or query costs. Such edge sets offer no benefit and should be removed.

[0124] , Adding it increases storage overhead, but can significantly reduce rebuild or query costs. It is commonly seen in cases of full materialization, field-level materialization, or direct incremental conversion.

[0125] , This simultaneously reduces storage, rebuild, and query overhead, a "free lunch" scenario. For example, when a version contains a large number of delete operations, materializing that version can actually reduce overall storage and rebuild costs. As you can see, while total storage overhead decreases significantly, the number of nodes required to rebuild the document also becomes smaller.

[0126] Based on this, and in conjunction with the objectives of this embodiment, an evaluation function for the variable edge set is given. :

[0127]

[0128] in, Indicates value, This indicates the increase in storage overhead after adding the corresponding modified edge set. This represents the reduction in the cost of complete reconstruction after adding the corresponding modified edge set. This indicates the reduction in historical query overhead for a specific field after adding the corresponding modified edge set. This represents the normalization function, where all increases and decreases are normalized. It is normalized by Min-Max. Representing the storage graph as This represents the number of child nodes of the root node, used to reflect the impact of modifying the current version on all versions contained in the search path. and These are the full reconstruction cost weight and the historical query cost weight for specific fields, used to control the different contributions of different costs to the value of the edge set under different user access preferences.

[0129] S5. In descending order of value, without violating storage space constraints, add the changing edge set to the storage graph that minimizes the total storage overhead to update the storage graph.

[0130] In this embodiment, after completing the... After calculating the value of all variable edge sets, they are sorted in descending order of value. If two variable edge sets have the same value, the one with lower storage overhead is prioritized. Based on the sorting results, the variable edge sets are added to the storage graph sequentially without violating storage constraints. superior.

[0131] In this embodiment, in a JSON-based version control system (JVD) that supports multiple granularities, the commit operation is the core mechanism driving version changes of data objects. For example... Figure 2 As shown, firstly, the system retrieves the current version based on the record identifier, which serves as the predecessor version and the branch to which the new version belongs. Then, the registered user submits a new version JSON data increment, updating the current version information to the newly submitted version and updating the version graph to indicate the logical relationship between the new version and existing versions. After the version layout optimization time is reached, the layout optimization method (i.e., steps S1-S5) is called on the version graph to generate a new storage graph. Based on the results in the generated new storage graph, the corresponding JSON data increments are materialized / dematerialized, and these JSON data increments are saved to the JVD. Dematerialization is the reverse process of materialization.

[0132] In this embodiment, as Figure 3 As shown, a JSON-based version control system (JVD) that supports multiple granularities of JSON data consists of six core layers, enabling efficient JSON version management:

[0133] 1) User Interface Layer: This layer provides the user interface, including version management and data access interfaces, enabling end users or applications to easily perform version-related operations. This embodiment implements a RESTful API based on Flask to enable interaction with users and further calls functions in other layers to meet users' version management needs.

[0134] 2) Data Management Layer: This layer is responsible for the state management and branch management of data, including: Data Manager and Branch Manager.

[0135] 3) Incremental Management Layer: This layer is responsible for managing incremental JSON data, including the Incremental Manager.

[0136] 4) Layout Optimization Layer: This layer implements the proposed layout optimization method, including a layout optimizer. Specifically, the system uses depth-first search (DFS) to construct the version evolution path and sorts and filters materialized nodes based on access frequency and reconstruction cost, thereby dynamically adjusting the physical layout of incremental JSON data to improve query efficiency and storage utilization.

[0137] 5) Storage Communication Layer: This layer encapsulates the access logic of the underlying storage, including communicators and storage connectors. It abstracts the differences in the underlying storage systems through an abstract interface, enabling Java Virtual Machines (JVDs) to adapt to different storage solutions. Storage connectors are components that encapsulate the details of interaction with specific storage systems, providing a unified access interface. Each storage connector inherits from the storage system through a different storage connector implementation.

[0138] 6) Underlying storage layer: This layer is the specific implementation of the storage communication layer. It provides adaptation interfaces for different storage systems to ensure that JVD can seamlessly integrate with various storage backends, including dynamic container databases and document-oriented databases.

[0139] In this embodiment, in order to support efficient access to history and version control operations, the system provides a unified version reconstruction mechanism based on incremental JSON data, and implements a version history query function on this basis.

[0140] like Figure 4As shown, JVD first initializes version pointers based on the query type: single-point queries point to the target version, and range queries point to the endpoint version of the query range. It then traverses the version pointers forward along the storage graph. If a version pointer already exists in the memory cache (Buffered), the corresponding JSON data increment is retrieved directly from the cache; otherwise, it is loaded from the database and written to the Buffered cache. The version pointer is then updated to the previous version, and this loading process is repeated until any termination condition is met (the version pointer materializes the target field, the modification path set contains only the JSON root node, or the version pointer is the starting version of a range query). Once all required JSON data increments are loaded into memory, a reconstruction operation is applied to the JSON data increment of the user's query target.

[0141] This invention provides a field-granularity storage layout optimization method for incremental version management of JSON data. It supports query loads based on version graphs and version history as input, fully leveraging the ability of incremental JSON data to support both full materialization and field-granularity materialization. While meeting storage space constraints, it improves access performance at both the field and version granularity levels at the cost of storage overhead. Furthermore, it introduces user access preferences as a moderating factor, achieving a balance between the overhead of full version queries (also known as full reconstruction overhead) and the overhead of specific field history queries. This method expands the applicability of existing materialization optimization techniques in semi-structured, multi-version scenarios, improving system response performance and resource utilization efficiency in real-world query environments.

[0142] In summary, the field-granularity storage layout optimization method for incremental version management of JSON data provided in this embodiment of the invention has at least the following beneficial effects:

[0143] 1) High space utilization: Under different storage constraints set to the range of 125%–250% of the MCA optimization results, the average storage overhead of the method of this invention is only slightly higher than that of LMG-ALL (+0.85%) and DyVer (+0.41%), indicating that this solution can accurately allocate resources close to the storage limit and achieve more effective version materialization and structure reuse.

[0144] 2) Significant query performance advantages: When storage constraints are small (≤175%), the average full version query time of the method of this invention is reduced by 16.29% compared with LMG-ALL and by 6.24% compared with DyVer, which fully demonstrates its stronger ability to optimize query paths in resource-constrained environments.

[0145] 3) Significantly superior historical access efficiency: Under the constraint of 150%–175%, the total time spent on historical queries is reduced by up to 85.14% compared to DyVer and by up to 86.53% compared to LMG-ALL, significantly reducing the overall system response latency, and is especially suitable for high-concurrency multi-version access scenarios.

[0146] 4) High sensitivity to user preferences: Under different weight settings, the method of this invention has a good response capability to changes in query preferences: when the user preference weight α decreases from 1.0 to 0.4, the complete reconstruction time increases from 0.70ms to 3.64ms, but the total historical access time decreases by about 11%; indicating that the method can flexibly adjust the storage-performance trade-off strategy according to user goals and adapt to personalized optimization needs.

[0147] 5) Superior combined strategy performance: In the ablation experiment, the full optimization strategy, under the condition of meeting the 150% storage constraint, only increased the storage overhead by 22%, which reduced the full version query time by 53.21% and the total historical access time by 86.17%, which is significantly better than using the minimum storage (MSS) or query optimization (CAN) strategies alone.

[0148] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0149] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0150] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0151] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0152] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0153] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0154] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0157] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for optimizing field-granular storage layout for incremental version management of JSON data, characterized in that, The method includes: S1. Construct a storage graph model that supports field-level materialization; S2. Based on the constructed storage graph model, the version graph is converted into a storage graph with minimized total storage overhead using the MCA algorithm. S3. If the total storage overhead is less than the storage space constraint, then perform optimization operations on all versions at both the field granularity and version granularity levels to generate a set of candidate change edge sets for optimizing the storage graph. The generated set of candidate change edge sets includes: fully materialized change edge sets, field-granular materialized change edge sets built based on query load, direct incremental change edge sets, and change edge sets removed by the MCA algorithm. S4. Introduce user access preferences as a moderating factor to evaluate the value of all variable edge sets; S5. In descending order of value, without violating storage space constraints, add the changing edge set to the storage graph that minimizes the total storage overhead to update the storage graph.

2. The field-granularity storage layout optimization method for incremental version management of JSON data according to claim 1, characterized in that, The storage graph model adopts an edge-attribute graph-based storage graph representation, which supports incremental representation at the field level; among which, it uses... This indicates a storage graph that supports field-level materialization. These represent the set of nodes in the stored graph, the set of edges in the stored graph, and the set of incremental JSON data for each version, respectively.

3. The field-granularity storage layout optimization method for incremental version management of JSON data according to claim 2, characterized in that, The storage graph representation method based on edge attribute graphs includes: for ,definition ;in, For storage graph A version node in a document represents a version. It can be reconstructed using different preceding versions and different incremental JSON data. Representing the storage graph The set of nodes; For version A set of incremental JSON data, representing a set of data that can be used to rebuild versions. Incremental JSON data; Indicates from version To version Incremental JSON data; Benchmark for JSON data increment This is used to indicate the incremental JSON data it depends on, rather than directly relating the node connections in the storage graph; where, for indicating version JSON data increment ,if If it is unmaterialized, then the version The version node corresponding to the incremental JSON data pointed to by the preceding node in the storage graph; if If it is fully materialized or its base is empty, then the version The preceding node in the storage graph is ;if If it is materialized at the field granularity, then the version The preceding node in the storage graph also contains and The corresponding version nodes, the former used to indicate the source of its materialized portion, and the latter used to provide the version context for the remaining non-materialized portions; among them, empty nodes... It represents an empty document and is the preceding node for all materialized versions; An attributed edge representation is used to characterize the structural representation of incremental JSON data in the storage graph; where, for an edge in the storage graph... , Representing the storage graph The set of edges is represented using a quadruple. , Indicates the starting point of the edge, i.e., the previous version; This indicates the endpoint of the edge, i.e., the target version; This represents the increment of the JSON data corresponding to the edge; This represents the version change operation contained in the edge; each edge This indicates that in the previous version Based on this, apply incremental JSON data. Version change operation , obtain the target version Data status; Define the set of variable edges , Indicates all increments of JSON data The set of associated edges, express of Attributes; during layout optimization, the set of edges is changed. As an operational atomic unit, it must be processed as a whole when added or deleted; that is, it must either be retained or... All edges in the graph, or delete all edges; for a stored graph that has already been optimized. Each version node can be represented by at most one JSON data increment, i.e. .

4. The field-granularity storage layout optimization method for incremental version management of JSON data according to claim 1, characterized in that, The MCA algorithm is the Zhu-Liu / Edmonds algorithm; The process of converting the version graph into a storage graph with minimized total storage overhead based on the constructed storage graph model and the MCA algorithm includes: Based on the constructed storage graph model and version graph, construct a dictionary. , Save all The set of changing edges The mapping of its storage overhead; where, Indicates all increments of JSON data The set of associated edges, Indicates incremental JSON data Storage overhead, For version An incremental collection of JSON data; Based on dictionary The version graph is converted into a weighted directed acyclic graph (DAG), and the minimum-cost tree graph is obtained through the Zhu-Liu / Edmonds algorithm, which is the storage graph that minimizes the total storage overhead. In DAG, Seen as a direction The edge weight is .

5. The field-granularity storage layout optimization method for incremental version management of JSON data according to claim 1, characterized in that, The construction of field-granular materialized change edge sets based on query load includes: Based on query load, the incremental JSON data corresponding to the query version is materialized at the field granularity with the query path as the target; the query load includes: query version, query path, and query frequency; Specifically, a prefix check strategy is used to construct a materialized change edge set at the field granularity. If the query path is already covered by the current processing path or is a prefix of an already materialized path, the generation of the change edge set for that query path is skipped.

6. The field-granularity storage layout optimization method for incremental version management of JSON data according to claim 1, characterized in that, Generating a direct incremental change edge set includes: The current version and its previous materialized ancestor node The version sequence between them is compressed to form a new JSON data increment, which is then... As a baseline; for incremental JSON data Let the pointer iterative access Preorder node Is the preceding version fully materialized? If so, return [value]. arrive All JSON data incremental sequences Otherwise Set it as its predecessor version and repeat the above iterative operation; merge the final JSON data increment sequence. Used to represent Incremental JSON data directly reconstructed.

7. The field-granularity storage layout optimization method for incremental version management of JSON data according to claim 1, characterized in that, The evaluation of the value of all modified edge sets includes: Determine the evaluation function for the variable edge set. Using the evaluation function Calculate the value of adding each modified edge set from the candidate modified edge set to a storage graph that minimizes the total storage overhead; where the evaluation function is... Represented as: in, Indicates value; This indicates the increase in storage overhead after adding the corresponding modified edge set; This represents the reduction in the cost of complete reconstruction after adding the corresponding modified edge set; This indicates the reduction in historical query overhead for a specific field after adding the corresponding modified edge set; Represents the normalization function; Representing the storage graph as This represents the number of child nodes of the root node, used to reflect the impact of modifying the current version on all versions contained in the search path; and These are the full reconstruction cost weight and the historical query cost weight for specific fields, which are used to control the different contributions of different costs to the value of the variable edge set under different user access preferences. , , Representing versions respectively Storage overhead, version Full reconstruction cost, version The overhead of historical queries for specific fields.

8. The field-granularity storage layout optimization method for incremental version management of JSON data according to claim 7, characterized in that, Version Storage overhead Represented as: in, Indicates incremental JSON data Storage overhead, For version An incremental collection of JSON data; Version Full reconstruction cost Represented as: in, Minimize the function; Indicates incremental JSON data The total reconstruction cost; Version query overhead Represented as: in, Indicates incremental JSON data Query overhead; This indicates the query load.

9. The field-granularity storage layout optimization method for incremental version management of JSON data according to claim 4, characterized in that, The method further includes: Get its current version, and use it as the predecessor version and branch to which the new version belongs; Registered users submit incremental JSON data for a new version, update the current version information to the newly submitted version, and update the version graph to indicate the logical relationship between the new version and existing versions; After the version layout optimization time is reached, the operation steps S1-S5 are called to generate a new storage graph for the version graph; Based on the results of the generated new storage graph, the corresponding JSON data increments are materialized / dematerialized and saved to JVD, where JVD represents a multi-granularity version control system for JSON data.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 9.

Images