Hybrid row-column storage method for graph database edge attribute sparsity

Abstract

This invention discloses a hybrid row-column storage method for graph database edge attribute sparsity, comprising: collecting edge attribute features; calculating edge attribute sparsity scores and access popularity scores; determining edge attribute storage categories based on the above scores; constructing an in-row storage structure for the edge body and an out-of-column sparse attribute storage structure; establishing an edge attribute row-column mapping index; writing each edge attribute into the corresponding storage structure according to the storage category and synchronously updating the index; performing staged attribute reading during query execution, where the query executor first accesses the index to determine the attribute storage location and migration status, directly reading in-row attributes, delaying the reading of out-of-column attributes, and selecting the reading location for attributes in migration according to stages; dynamically executing attribute migration based on sparsity and popularity changes; and maintaining consistency during the migration process through status management and version verification. This invention solves the problems of wasted sparse attribute space and high reading costs for frequently used attributes.

Description

Technical Field

[0001] This invention belongs to the field of graph database technology, and specifically relates to a hybrid row-column storage method for graph database edge attribute sparsity. Background Technology

[0002] Graph databases typically use nodes, edges, and attributes to describe complex, interconnected data. Edges not only represent relationships between two nodes but can also carry attributes such as time, weight, amount, status, source, confidence level, business type, and version number. Publicly available information generally describes graph databases as a data model composed of nodes, edges, and attributes; edges can contain attributes that describe the context of the relationships.

[0003] Existing edge attribute storage methods in graph databases mainly include: row-based edge record storage, which stores the edge identifier, start point, end point, relation type, and all attributes or attribute sets in the same edge record; key-value pair attribute storage, which appends edge attributes to edge records or independent attribute tables in the form of key-value pairs and associates them through edge identifiers; unified columnar storage, which splits different attribute fields into columns to facilitate scanning a specific attribute or performing filtering calculations; serialized attribute block storage, which packages edge attributes into objects or extended attribute blocks and stores them together with the edge body; and table or column family creation based on relation type, which creates different table structures or column families for different relation types to ensure a relatively consistent attribute layout for edges of the same type.

[0004] However, the above-mentioned existing technical solutions have the following drawbacks: First, storing all edges in a row results in sparse attributes occupying a large amount of invalid space. If all possible attribute fields are placed in the edge record, many edge pairs will have empty or non-existent low-frequency attributes, which can easily lead to problems such as null values ​​occupying space, field metadata bloat, and increased serialization overhead. For business graphs with hundreds of extended attributes, most edges may only contain a few core attributes, and uniform row-based storage will result in excessively wide edge records.

[0005] Second, full columnar storage increases the cost of reading frequently used attributes. If all edge attributes are split into off-column storage, querying frequently used attributes requires accessing multiple column files, column families, or attribute tables. For example, path traversal often requires reading attributes such as time, weight, and status. If these attributes are all stored off-column, each edge expansion requires additional column retrieval, increasing the cost of random reads, column concatenation, and table lookups.

[0006] Third, existing solutions lack fine-grained layering to address the sparsity of edge attributes. Ordinary row storage or ordinary column storage typically determines the storage method statically at the table level, relation type level, or attribute field level, lacking a comprehensive judgment mechanism for edge attributes in graph databases. For example, factors such as attribute occurrence rate, access popularity, filter hit rate, query stage, update frequency, and compression benefits are not uniformly incorporated into the decision-making process.

[0007] Fourth, the integration of attribute access with the graph traversal process is not tight enough. Graph queries typically begin with adjacency expansion along edges, followed by filtering, sorting, or path scoring based on edge attributes. Existing common attribute storage methods often fail to distinguish between attributes necessary for the traversal phase and expanded attributes needed only for the result return phase, resulting in the reading of a large number of temporarily unnecessary sparse attributes during query execution.

[0008] Fifth, the attribute sparsity cannot be adaptively adjusted after changes. The usage frequency of business fields changes over time. Some attributes that were originally infrequent may become high-frequency filtering conditions due to the launch of new businesses; some attributes that were originally high-frequency may gradually become less popular. Static storage layout cannot adapt to changes in attribute popularity in a timely manner, leading to storage expansion or decreased query performance.

[0009] In summary, existing edge attribute storage technologies for graph databases have shortcomings in terms of space efficiency, query performance, and dynamic adaptation. There is an urgent need for a hybrid row-column storage method that can comprehensively evaluate edge attribute sparsity and access popularity, and dynamically adjust the storage layout. Summary of the Invention

[0010] This invention provides a hybrid row-column storage method for graph databases with sparse edge attributes to solve the aforementioned technical problems. Specifically, the technical solution is as follows: A hybrid row-column storage method for graph databases with sparsity of edge attributes includes the following steps: Collect edge attribute features; Calculate edge attribute sparsity scores to assess the degree of attribute sparsity; Calculate the edge attribute access popularity score to assess the frequency of attribute access; By combining the sparsity score and the access popularity score, the storage category of the edge attribute is determined. The storage category includes in-row high-frequency attributes, out-of-column sparse attributes, candidate out-of-column attributes, candidate back-of-column attributes, and bi-state observation attributes. Construct an edge body inline storage structure. The edge body record contains a fixed edge field and an inline high-frequency attribute area. The inline high-frequency attribute area is used to store edge attributes that are determined to be inline high-frequency attributes. Construct an off-column sparse attribute storage structure. The off-column sparse attribute area is organized according to relation type and attribute field. It is used to store edge attributes that are determined to be off-column sparse attributes, and only records the edge identifier and attribute value that actually exist. Establish an edge attribute row-column mapping index to record the current storage location, storage mode, and migration status of each edge attribute. According to the storage category, write each edge attribute into the corresponding storage structure and update the edge attribute row-column mapping index synchronously. The query executor first accesses the edge attribute row and column mapping index to determine the storage location and migration status of each attribute. For in-row attributes in a stable storage state, they are read directly along with the edge main record. For out-of-column attributes in a stable storage state, they are read only when required by the filtering conditions or the returned field. For attributes in a migration state, they are read from the source location or the double-write location according to the migration stage. Based on changes in edge attribute sparsity and access frequency, the system dynamically performs migration operations, moving in-row attributes to out-of-column or out-of-column attributes back to in-row. Consistency is maintained during attribute migration, and query continuity and write consistency are ensured through migration status management and version verification.

[0011] Furthermore, the calculation of edge attribute sparsity scoring includes: For attribute field p under relation type r, calculate the attribute occurrence rate: , Among them, E r E represents the total number of edges under relation type r. r,p This represents the number of edges with attribute p under relation type r; Attribute sparsity is calculated using the following formula: Sparse(r,p) = 1 - AppearRate(r,p) In this case, the closer Sparse(r,p) is to 1, the sparser the attribute p is; the closer it is to 0, the more common the attribute p is.

[0012] Furthermore, the calculation of edge attribute sparsity score further includes calculating a comprehensive sparsity score:

[0013] Where SPS(r,p) represents the comprehensive sparsity score of attribute p under relation type r, NullRate(r,p) represents the proportion of null values ​​of the attribute, AvgLen(r,p) represents the average length of the attribute, CompressGain(r,p) represents the columnar compression gain, norm represents the normalization function, and α1 to α4 are weight parameters.

[0014] Furthermore, the calculation of edge attribute access popularity score includes: For attribute field p under relation type r, calculate the access popularity score:

[0015] Where AHS(r,p) represents the attribute access popularity score, ReadFreq(r,p) represents the attribute reading frequency, FilterFreq(r,p) represents the frequency of the attribute being used as a filter condition, PathHit(r,p) represents the frequency of the attribute being hit in path queries, EarlyUse(r,p) represents whether the attribute is used in the early stage of adjacency expansion, UpdateFreq(r,p) represents the attribute update frequency, norm represents the normalization function, and β1 to β5 are weight parameters.

[0016] Furthermore, determining the storage category of the edge attributes includes calculating a mixed row-column storage decision value:

[0017] Where HCS(r,p) represents the hybrid row and column storage decision value, ColumnReadCost(r,p) represents the off-column read cost, RowBenefit(r,p) represents the traversal benefit brought by in-row storage, MigrationCost(r,p) represents the attribute migration cost, norm represents the normalization function, and γ1 to γ5 are weight parameters. Based on HCS(r,p) and SPS(r,p), the attributes are divided into in-row high-frequency attributes, out-of-column sparse attributes, candidate out-of-column migration attributes, candidate back-of-column migration attributes, and bimorphic observation attributes.

[0018] Furthermore, the constructed edge body in-row storage structure includes: Set a fixed edge field in the edge body record. The fixed edge field includes edge identifier, start point identifier, end point identifier, relationship type, direction marker, version and timestamp. An inline attribute bitmap is set in the edge body record, and the inline attribute bitmap is used to mark the existence state of each attribute in the inline high-frequency attribute area; In the edge body record, a high-frequency attribute area is set up, and attribute fields that are determined to be high-frequency attributes are written into the high-frequency attribute area in a preset order. In the edge body record, off-column sparse attribute positioning information is set, which is used to point to the corresponding record of the edge in the off-column sparse attribute area.

[0019] Furthermore, the construction of the out-of-column sparse attribute storage structure includes: The storage area is divided according to the relation type and attribute field, and an independent off-column sparse attribute area is created for each attribute field under each relation type; Write the relation type and attribute field identifier in the sparse attribute area outside the column; Scan the edge records with this attribute, write the set of edge identifiers with this attribute into the edge identifier list, write the corresponding attribute values ​​into the attribute value column in order, and write the version information into the version column. Set a null bitmap for the attribute value column to mark the default state, set an offset index for the variable-length attribute, and configure compressed metadata to record the compression algorithm and encoding method; For edges that do not have this attribute, do not write an empty placeholder value.

[0020] Furthermore, establishing the edge attribute row-column mapping index includes: Create an attribute layout directory, which uses the relation type and attribute field identifier as index keys; Write the storage mode of each attribute in the attribute layout directory. The storage mode includes in-row stable storage, out-of-column stable storage, in-row to out-of-column migration, and out-of-column to in-row migration. Write the sparse score, access popularity score, and decision score for each attribute in the attribute layout directory; The offset position of inline attributes and the column file or storage block location of off-column attributes are recorded in the attribute layout directory; Set a migration status field in the attribute layout directory to record whether the attribute is currently in the migration process and record the update time of the layout information; When accessing edge attributes, the query executor first queries the attribute layout directory and decides whether to read directly from the edge body record or from the off-column sparse attribute area based on the storage mode.

[0021] Furthermore, the phased attribute reading during query execution includes: Parse the query statement and extract the relation type, edge attribute filtering conditions, path pattern, return fields, and sorting fields; The query executor accesses the attribute layout directory to determine the storage mode and migration status of the attributes involved in the query; For attributes that are in a stable storage state within the row, they are read directly along with the main record. For attributes that are in stable storage outside the column, read them only when needed by filter conditions, return fields, or subsequent operators; For attributes in the migration state, read from the source storage location, the target storage location, or the double-write location, depending on the current migration stage. For queries with large result sets, pre-filtering is performed using inline attributes first, and then out-of-column attributes are read in batches according to the candidate edge identifier; The results of batch reading of external attributes are concatenated to form the final edge attribute result.

[0022] Furthermore, dynamically performing attribute migrations includes: The triggering conditions for migrating inline attributes to column-side attributes include: attribute sparsity consistently exceeding the threshold, attribute access frequency consistently falling below the threshold, attribute average length increasing leading to excessively wide side records, attribute columnar compression yielding significantly higher benefits than inline storage, attributes no longer participating in path filtering or early traversal, and inline attribute region expansion leading to decreased adjacency scan performance. Triggering conditions for migrating out-of-column attributes back to in-row include: continuously increasing attribute access popularity, attributes becoming high-frequency filtering conditions, attributes being frequently used in the early stages of path queries, out-of-column read costs exceeding in-row storage benefits, increased attribute occurrence rates leading to decreased sparsity, and frequent out-of-column table lookups during query execution. During the migration process, the following steps are executed sequentially: marking migration status, scanning and copying attribute values, dual-write synchronization, version verification, route switching, and cleaning up source location data, to ensure data consistency during the migration.

[0023] The advantage of this invention lies in the hybrid row-column storage method for graph database edge attribute sparsity provided. By collecting edge attribute structural features and access features, sparsity scores and access popularity scores are calculated to determine the storage category. Based on this, an in-row storage structure, an out-of-column storage structure, and a row-column mapping index are constructed to realize phased query reading and dynamic attribute migration. At the same time, it solves the problems of sparse attribute space waste, high cost of reading frequently used attributes, static layout failure, and lack of migration consistency, so that graph databases have both traversal performance and space efficiency.

[0024] The advantages of this invention also lie in the hybrid row-column storage method for edge attribute sparsity in graph databases. By defining an attribute occurrence rate formula, a comprehensive sparsity scoring formula, and an access frequency scoring formula, it incorporates multiple factors such as attribute occurrence rate, null value ratio, average length, compression benefit, read frequency, filter frequency, path hit frequency, and early usage frequency into a unified quantitative evaluation. This makes the determination of attribute sparsity and access frequency more objective and accurate, avoids storage misjudgments caused by a single indicator, and improves the reliability of row-column storage decisions.

[0025] The advantages of this invention also lie in the hybrid row-column storage method for edge attribute sparsity in graph databases. By first accessing the attribute layout directory during query execution to determine the storage mode and migration status, direct reading, delayed reading, and stage-adaptive reading strategies are adopted for the stable state of in-row, stable state of out-of-column, and state in migration, respectively. It also supports batch reading of out-of-column attributes after pre-filtering of in-row attributes, so that only necessary core attributes are loaded during the traversal stage, avoiding invalid reading of irrelevant sparse attributes, and significantly reducing the attribute reading overhead and the number of out-of-column table lookups in multi-hop queries. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the hybrid row-column storage method for graph database edge attribute sparsity proposed in this application. Detailed Implementation

[0028] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0029] In the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installation", "connection" and "linkage" should be interpreted broadly, and can refer to mechanical or electrical connections, or internal connections between two components, or direct connections. "Up", "down", "left", "right", etc., are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may change.

[0030] like Figure 1The diagram illustrates a hybrid row-column storage method for edge attribute sparsity in graph databases, comprising the following steps: S1: Collect edge attribute features. S2: Calculate edge attribute sparsity scores to assess the degree of attribute sparsity. S3: Calculate edge attribute access popularity scores to assess the frequency of attribute access. S4: Combine sparsity scores and access popularity scores to determine the storage category of edge attributes. The storage categories include intra-row high-frequency attributes, out-of-column sparse attributes, candidate outgoing attributes, candidate returning attributes, and bi-state observation attributes. S5: Construct an intra-row storage structure for the edge body. The edge body record contains a fixed edge field and an intra-row high-frequency attribute area. The intra-row high-frequency attribute area is used to store edge attributes determined to be intra-row high-frequency attributes. S6: Construct an out-of-column sparse attribute storage structure. The out-of-column sparse attribute area is organized according to relation type and attribute field, used to store edge attributes determined to be out-of-column sparse attributes, and only records the edge identifier and attribute value that actually exist for that attribute. S7: Establish an edge attribute row-column mapping index to record the current storage location, storage mode, and migration status of each edge attribute. Based on the storage category, write each edge attribute to the corresponding storage structure and synchronously update the edge attribute row-column mapping index. S8: Perform phased attribute reading during query execution. The query executor first accesses the edge attribute row-column mapping index to determine the storage location and migration status of each attribute. For in-row attributes in a stable storage state, read directly along with the edge body record. For out-of-column attributes in a stable storage state, read delayed only when required by filtering conditions or return fields. For attributes in a migration state, read from the source location or dual-write location depending on the migration stage. S9: Dynamically execute migration operations based on changes in edge attribute sparsity and access frequency, moving in-row attributes out to out-of-column or out-of-column attributes back to in-row. Maintain consistency during attribute migration, ensuring query continuity and write consistency during migration through migration status management and version verification. This invention collects edge attribute features, calculates sparsity scores and access popularity scores respectively, comprehensively determines the attribute storage category, and constructs in-row storage structures, out-of-column storage structures, and row-column mapping indexes accordingly to achieve attribute-level hybrid row-column storage initialization deployment. During queries, it reduces invalid attribute loading by reading attributes in stages and adapts to attribute hot and cold changes through dynamic migration and consistency maintenance mechanisms. This solution simultaneously solves technical problems such as storage space waste caused by edge attribute sparsity, high cost of reading frequently used attributes, inability of static storage layout to balance traversal performance and space efficiency, indiscriminate attribute reading during queries, and read / write consistency during migration. It enables graph databases to achieve both traversal performance and space efficiency in edge attribute storage, and has adaptive adjustment capabilities. The following details the above steps.

[0031] For step S1: Collect edge attribute features.

[0032] During the operation of the graph database, the system collects edge attribute statistics based on relation type, edge attribute field, and time window. For attribute field p under relation type r, the following information is collected: The attribute exhibits characteristics. It contains the total number of attributes E under relation type r. r The number of edges E with attribute p r，p The proportion of non-empty values ​​of attribute p, the proportion of empty values ​​of attribute p, the length of the average value of attribute p, and the cardinality of the value range of attribute p.

[0033] Attribute access characteristics include the number of times attribute p is read, the number of times attribute p is filtered, the number of times attribute p is used in sorting, aggregation, and path scoring, the number of times attribute p is hit in one-hop, two-hop, and multi-hop queries, and the access stage of attribute p in the query execution plan.

[0034] The attribute update characteristics include the number of times attribute p is added, the number of times attribute p is modified, the number of times attribute p is deleted, the update frequency of attribute p per unit time, and the number of version conflicts of attribute p.

[0035] Storage and compression features include the space usage of attribute p when stored in a row, the space usage of attribute p when stored out of a column, the columnar compression ratio of attribute p, the in-row read cost of attribute p, the out-of-column read cost of attribute p, and the table lookup or concatenation cost of attribute p.

[0036] The query phase features include whether it is a required attribute for the adjacency traversal phase, whether it is a path filtering attribute, whether it is a result return attribute, whether it is a low-frequency audit attribute, and whether it is an offline analysis attribute.

[0037] The edge attributes collected above provide a data foundation for subsequent score calculations.

[0038] For step S2: Calculate the edge attribute sparsity score to assess the degree of attribute sparsity.

[0039] In the embodiments of this application, calculating the edge attribute sparsity score includes: For attribute field p under relation type r, calculate the attribute occurrence rate: , Among them, E r E represents the total number of edges under relation type r. r,p This represents the number of edges with attribute p under relation type r. max(E) r , 1) Used to avoid a denominator of 0.

[0040] Attribute sparsity is calculated using the following formula: Sparse(r,p) = 1 - AppearRate(r,p) In this case, the closer Sparse(r,p) is to 1, the sparser the attribute p is; the closer it is to 0, the more common the attribute p is.

[0041] By defining an attribute occurrence rate formula, the actual proportion of an attribute appearing in edge records is calculated using the total number of edges under a relation type as the denominator and the number of edges with that attribute as the numerator. Subtracting the occurrence rate from 1 yields the attribute sparsity, thus quantifying the sparseness of the attribute's distribution in the edge set. This step provides a foundational indicator for subsequent comprehensive sparsity scoring. By clarifying the calculation formulas for attribute occurrence rate and sparsity, the assessment of attribute sparsity has a quantifiable mathematical basis, avoiding subjective judgment and providing an objective basis for subsequent storage category classification.

[0042] In embodiments of this application, calculating the edge attribute sparsity score further includes calculating the comprehensive sparsity score:

[0043] Where SPS(r,p) represents the comprehensive sparsity score of attribute p under relation type r, NullRate(r,p) represents the proportion of null values ​​in the attribute, AvgLen(r,p) represents the average length of the attribute, CompressGain(r,p) represents the columnar compression gain, norm represents the normalization function, and α1 to α4 are weight parameters. When SPS(r,p) is higher than the preset threshold S... th When this is the case, it indicates that the attribute is suitable for off-column sparse storage.

[0044] Building upon attribute sparsity, this paper introduces three dimensions: null value ratio, average attribute length, and columnar compression benefit. Through normalization and weighted summation, a comprehensive sparsity scoring model is constructed. When the score exceeds a preset threshold, the attribute is deemed suitable for off-column sparse storage, thus incorporating multi-dimensional structural features into storage decisions. By introducing multiple factors such as null value ratio, average length, and compression benefit, sparsity assessment becomes more comprehensive and accurate, avoiding misjudgments caused by relying solely on occurrence rate as a single indicator, and improving the reliability of off-column storage decisions.

[0045] For step S3: Calculate the edge attribute access popularity score to assess the frequency of attribute access.

[0046] In the embodiments of this application, calculating the edge attribute access popularity score includes: For attribute field p under relation type r, calculate the access popularity score:

[0047] Where AHS(r,p) represents the attribute access popularity score, ReadFreq(r,p) represents the attribute read frequency, FilterFreq(r,p) represents the frequency of the attribute being used as a filter condition, PathHit(r,p) represents the frequency of the attribute being hit in path queries, EarlyUse(r,p) indicates whether the attribute is used in the early stage of adjacency expansion, UpdateFreq(r,p) represents the attribute update frequency, norm represents the normalization function, and β1 to β5 are weight parameters. AHS(r,p) is higher than the access popularity threshold A. th When this is the case, it indicates that the attribute is suitable for inline storage to reduce the additional read cost during queries.

[0048] By defining an access popularity scoring formula, read frequency, filter frequency, path hit frequency, and early use frequency are used as positive factors, while update frequency is used as a negative factor. After normalization and weighting, a comprehensive access popularity score is obtained. When the score exceeds a threshold, the attribute is deemed suitable for in-row storage, thus quantifying the frequency of attribute usage in query scenarios. By incorporating multi-dimensional access features such as reads, filters, path hits, and early use into the popularity evaluation, and distinguishing between positive and negative factors, the evaluation of attribute access frequency better reflects the actual query scenarios of graph databases, improving the accuracy of in-row storage decisions.

[0049] For step S4: Combine the sparsity score and access popularity score to determine the storage category of the edge attribute. The storage category includes high-frequency attributes within rows, sparse attributes outside columns, candidate outgoing attributes, candidate returning attributes, and bi-state observation attributes.

[0050] In the embodiments of this application, determining the storage category of the edge attribute includes calculating a mixed row-column storage decision value:

[0051] Where HCS(r,p) represents the hybrid row and column storage decision value, ColumnReadCost(r,p) represents the off-column read cost, RowBenefit(r,p) represents the traversal benefit brought by in-row storage, MigrationCost(r,p) represents the attribute migration cost, norm represents the normalization function, and γ1 to γ5 are weight parameters.

[0052] Based on sparsity scoring and access popularity scoring, three decision factors are introduced: off-column read cost, in-row storage benefit, and migration cost. A hybrid row-column storage decision value is calculated, and the attribute is classified into five storage categories by combining the decision value with the sparsity score. This achieves refined, multi-factor comprehensive decision-making for attribute storage location. By introducing engineering constraints such as read cost, storage benefit, and migration cost, the storage category classification considers not only the characteristics of the attribute itself but also system operating costs and migration overhead, improving the practicality and operability of storage layout decisions.

[0053] Based on HCS(r,p) and SPS(r,p), attributes are categorized into in-row high-frequency attributes, out-of-column sparse attributes, candidate outbound attributes, candidate backbound attributes, and bi-state observation attributes. The specific categorization logic includes setting an in-row decision threshold Hrow, an out-of-column decision threshold Hcol, a high sparsity threshold Shigh, a low sparsity threshold Slow, and a continuous window number M, where Hrow is greater than Hcol and Shigh is greater than Slow. When HCS(r,p) ≥ Hrow and SPS(r,p) ≤ Slow, it indicates that the attribute has high access benefit and low sparsity, and the attribute is determined as an in-row high-frequency attribute. When HCS(r,p) ≤ Hcol and SPS(r,p) ≥ Shigh, it indicates that the attribute has low in-row benefit and high sparsity, and the attribute is determined as an out-of-column sparse attribute. When an attribute is currently in stable inline storage (ROW_INLINE), and within M consecutive statistical windows, HCS(r,p) ≤ Hcol or SPS(r,p) ≥ Shigh, it indicates that the original inline attribute has become cold or sparse, and the attribute is identified as a candidate for outbound migration. When an attribute is currently in stable outbound storage, and within M consecutive statistical windows, HCS(r,p) ≥ Hrow and SPS(r,p) ≤ Slow, it indicates that the original outbound attribute has become hot and its sparsity has decreased, and the attribute is identified as a candidate for return migration. If the above stable partitioning conditions are not met, if HCS(r,p) is between Hcol and Hrow and SPS(r,p) is between Slow and Shigh, or if within the most recent M consecutive statistical windows, the initial storage determination result of the attribute changes back and forth between inline and outbound tendencies, then the attribute is identified as a bimorphic observation attribute, and migration is not triggered temporarily. If HCS(r,p) is between Hcol and Hrow and SPS(r,p) is between Slow and Shigh, it indicates that the attribute is currently statically undefined. If the initial storage determination result of the attribute fluctuates between in-row bias and out-of-column bias within the most recent M consecutive statistical windows, it indicates that the attribute is not statically undefined, but rather dynamically unstable. In-row high-frequency attributes represent attributes with high occurrence rate, high access frequency, high filtering frequency, and requiring fast reading during adjacency traversal. Out-of-column sparse attributes are suitable for attributes with low occurrence rate, low access frequency, long attribute values, high compression benefits, and only need to be read in result display or auditing scenarios. Candidate out-of-column attributes are suitable for attributes currently stored in-row but with increased sparsity, decreased access popularity, or significant row width expansion. Candidate back-of-column attributes are suitable for attributes currently stored out-of-column but with increased access popularity, increased filtering frequency, or frequent path query hits. Bimorphic observation attributes are suitable for attributes whose scores fluctuate around a threshold; they are not migrated temporarily and are placed in the observation window.

[0054] For step S5: Construct the edge body inline storage structure. The edge body record contains a fixed edge field and an inline high-frequency attribute area. The inline high-frequency attribute area is used to store the edge attributes that are determined to be inline high-frequency attributes.

[0055] An in-row storage structure for the edge body is constructed. This structure includes a fixed edge field, an in-row attribute bitmap, an in-row high-frequency attribute area, and out-of-column sparse attribute location information within the edge body record. This provides a physical storage medium for high-frequency attributes that are read along with the edge body. In the embodiments of this application, constructing the in-row storage structure for the edge body includes: In the edge body record, a fixed edge field is set, which includes edge identifier, start identifier, end identifier, relation type, direction marker, version, and timestamp. An in-row attribute bitmap is set in the edge body record to mark the existence status of each attribute in the in-row high-frequency attribute area. An in-row high-frequency attribute area is set in the edge body record, and attribute fields determined to be in-row high-frequency attributes are written into the in-row high-frequency attribute area in a preset order. Out-of-column sparse attribute positioning information is set in the edge body record to point to the corresponding record of the edge in the out-of-column sparse attribute area.

[0056] By sequentially setting a fixed edge field, an in-row attribute bitmap, an in-row high-frequency attribute area, and out-of-column sparse attribute location information in the edge body record, a hierarchical in-row storage structure is formed. The in-row attribute bitmap is used to dynamically mark the existence status of each attribute, and the out-of-column location information points to the corresponding record of the edge in the out-of-column area, achieving both compactness of in-row storage and its correlation with out-of-column storage. Dynamically marking the existence status of attributes through the in-row attribute bitmap avoids the waste of space due to null values ​​caused by fixed field widths; establishing the correlation between in-row and out-of-column storage through out-of-column location information allows a single edge record to quickly access in-row high-frequency attributes and locate out-of-column sparse attributes as needed, balancing access efficiency and storage flexibility.

[0057] For step S6: Construct an off-column sparse attribute storage structure. The off-column sparse attribute area is organized according to relation type and attribute field. It is used to store edge attributes that are determined to be off-column sparse attributes, and only records the edge identifier and attribute value that actually exist.

[0058] An off-column sparse attribute storage structure is constructed, creating independent off-column storage areas according to relation type and attribute fields. Only edge identifiers and attribute values ​​that actually exist for the attribute are recorded. Null value bitmaps, offset indexes, and compressed metadata are configured to provide a space-saving physical storage medium for sparse attributes. In the embodiments of this application, constructing the off-column sparse attribute storage structure includes: Storage areas are divided according to relation type and attribute fields. A separate off-column sparse attribute area is created for each attribute field under each relation type. Relation type and attribute field identifiers are written into the off-column sparse attribute area. Edge records with the specified attribute are scanned, and the set of edge identifiers with the specified attribute is written to the edge identifier list. The corresponding attribute values ​​are written sequentially to the attribute value column, and version information is written to the version column. A null bitmap is set for the attribute value column to mark the default state, an offset index is set for variable-length attributes, and compressed metadata is configured to record the compression algorithm and encoding method. For edges where the attribute does not exist, no null placeholder is written.

[0059] By creating independent storage areas based on relation type and attribute fields, the edge identifier set, attribute value column, and version column are organized separately, and a null value bitmap, offset index, and compressed metadata are configured to form a columnar storage structure for sparse attributes. No data is written to edges where the attribute does not exist, completely eliminating null value placeholders. By storing only the edge identifiers and attribute values ​​that actually exist, the waste of null value placeholders in row-based storage for sparse attributes is completely eliminated. Through columnar organization and compressed metadata configuration, sparse attributes possess high compression ratios and batch read advantages, significantly reducing the storage space occupied by sparse attributes.

[0060] For step S7: Establish an edge attribute row-column mapping index to record the current storage location, storage mode and migration status of each edge attribute. According to the storage category, write each edge attribute into the corresponding storage structure and update the edge attribute row-column mapping index synchronously.

[0061] An edge attribute row-column mapping index is established, and an attribute layout directory is created with relation type and attribute field identifier as keys. This directory records the storage mode, scoring results, physical location, and migration status of each attribute, providing the query executor and migration executor with a basis for attribute location query and routing decisions. In the embodiments of this application, establishing the edge attribute row-column mapping index includes: Create an attribute layout catalog, which uses the relation type and attribute field identifier as index keys.

[0062] Write the storage mode for each attribute in the attribute layout directory. The storage modes include in-row stable storage, out-of-column stable storage, in-row to out-of-column migration, and out-of-column to in-row migration.

[0063] Write the sparse score, access popularity score, and decision score for each attribute in the attribute layout directory.

[0064] Record the offset position of inline attributes and the column file or storage block location of off-column attributes in the attribute layout directory.

[0065] Set a migration status field in the attribute layout directory to record whether the attribute is currently in the migration process and record the update time of the layout information.

[0066] When accessing edge attributes, the query executor first queries the attribute layout directory and decides whether to read directly from the edge body record or from the off-column sparse attribute area based on the storage mode.

[0067] This application creates an attribute layout directory with relation type and attribute field identifiers as keys to record key information such as storage mode, scoring results, physical location, and migration status. It also explicitly allows the query executor to determine the attribute read path by querying this directory, achieving centralized management of attribute storage locations and unified scheduling of query routes. Furthermore, by centrally managing the storage location and migration status of each attribute through the attribute layout directory, the query executor and migration executor can make routing decisions based on a unified index, avoiding read path confusion caused by scattered attribute location management. Simultaneously, the migration status field supports correct route selection during migration, providing a foundational index-level support for dynamic migration.

[0068] For step S8: phased attribute reading during query execution, the query executor first accesses the edge attribute row and column mapping index to determine the storage location and migration status of each attribute. For in-row attributes in a stable storage state, they are read directly along with the edge main record. For out-of-column attributes in a stable storage state, they are read only when required by the filtering conditions or returned fields. For attributes in a migration state, they are read from the source location or the double-write location according to the migration stage.

[0069] The query executor performs phased attribute reading during query execution. First, it accesses the attribute layout directory to determine attribute position and migration status. Inline attributes are read directly along with the main body, while out-of-column attributes are read on demand with delayed reading. For attributes in migration, the reading position is selected based on the stage. For large-scale queries, inline attributes are pre-filtered before out-of-column attributes are read in batches, achieving fine-grained control over attribute reading during query execution. In the embodiments of this application, phased attribute reading during query execution includes: Parse the query statement to extract the relationship type, edge attribute filtering conditions, path pattern, return fields, and sorting fields.

[0070] The query executor accesses the attribute layout directory to determine the storage mode and migration status of the attributes involved in the query.

[0071] For attributes that are in a stable storage state within the row, they are read directly from the subject record along the edge.

[0072] For attributes that are in a stable, off-column storage state, read them only when required by filter conditions, return fields, or subsequent operators.

[0073] For attributes in the migration state, read from the source storage location, the target storage location, or the double-write location, depending on the current migration stage.

[0074] For queries with large result sets, pre-filtering is performed using inline attributes first, and then out-of-column attributes are read in batches according to the candidate edge identifier.

[0075] The results of batch reading of external attributes are concatenated to form the final edge attribute result.

[0076] Specifically, by parsing the query statement to extract each element, the query executor first accesses the attribute layout directory to determine the attribute position and migration status. Then, it adopts direct reading, delayed reading, and stage-adaptive reading strategies for stable in-row, stable out-of-column, and migrating states, respectively. It also supports batch reading of out-of-column attributes after pre-filtering in-row attributes, realizing fine-grained stage control of attribute reading during query execution. By differentiating attribute storage status and migration status and adopting differentiated reading strategies, the query executor only reads the necessary core in-row attributes during the traversal phase, avoiding loading a large number of irrelevant sparse attributes and significantly reducing the attribute reading overhead in multi-hop queries. At the same time, by pre-filtering and batch reading of out-of-column attributes, the number of out-of-column table lookups is reduced, improving query execution efficiency.

[0077] For step S9: Based on the changes in edge attribute sparsity and access frequency, dynamically execute migration operations to move in-row attributes out of column or move out-of-column attributes back to in-row. Maintain consistency during attribute migration, and ensure query continuity and write consistency during migration through migration status management and version verification.

[0078] The system periodically or through event-triggered recalculations of sparsity and access frequency scores for each attribute. When an attribute meets the migration or return trigger conditions, a complete migration process is executed, including marking the migration status, copying attribute values, dual-write synchronization, version verification, route switching, and cleaning up source data. This ensures that the attribute storage location can adapt to changes in business query patterns. Consistency is maintained during the migration process. By defining multi-level migration states such as normal, copying, dual-write, and switching, coupled with version verification and exception rollback mechanisms, query continuity, write consistency, and exception recovery are guaranteed during dynamic migration of attributes.

[0079] In the embodiments of this application, dynamically performing attribute migration includes: The triggering conditions for migrating inline attributes to column-side attributes include: attribute sparsity consistently exceeding a threshold; attribute access frequency consistently falling below a threshold; attribute average length increasing, leading to excessively wide edge records; attribute columnar compression yielding significantly higher benefits than inline storage; attributes no longer participating in path filtering or early traversal; and inline attribute region expansion causing a decrease in adjacency scan performance.

[0080] When one or more of the above conditions are met, the system will migrate the attribute from in-row to out-of-column. The migration process includes: Mark the attribute as ROW_TO_COLUMN_MIGRATING; Scan the edge records that have this attribute; Write the attribute values ​​to the sparse attribute area outside the column; Record attribute version and edge ID; Perform double write on attributes added or updated during migration; Verify that the values ​​in the row and outside the column are consistent; Switch the layout directory to COLUMN_EXTERNAL; Clean up inline attribute values ​​or mark them as placeholders for migration.

[0081] The triggering conditions for migrating out-of-column attributes back to in-row include: continuously increasing attribute access frequency; the attribute becoming a high-frequency filtering condition; the attribute being frequently used in the early stages of path queries; the cost of out-of-column reads exceeding the storage benefits of in-rows; increased attribute occurrence rate leading to decreased sparsity; and frequent out-of-column table lookups during query execution.

[0082] When one or more of the above conditions are met, the system will move the attribute back from outside the column to inside the row. The migration process includes: Mark the attribute as COLUMN_TO_ROW_MIGRATING; Batch read out-of-column attribute values ​​by edge ID; Write the attributes into the attribute area within the edge body row; Update the inline attribute bitmap and offset; Perform double-write or incremental log synchronization on updates during migration; Complete consistency verification; Switch the layout directory to ROW_INLINE; Depending on the circumstances, retain the extra-listed copies for a period of observation or clear the extra-listed copies.

[0083] To ensure the accuracy of query results during dynamic attribute migration, the system implements attribute migration status and version verification mechanisms. During the migration process, the following steps are executed sequentially: marking migration status, scanning and copying attribute values, dual-write synchronization, version verification, route switching, and cleaning up data in the source location, to guarantee data consistency during the migration.

[0084] By clearly defining the triggering conditions for in-row attribute migration and out-of-column attribute migration, the system covers various business scenarios such as changes in sparsity, popularity, length, compression benefits, query patterns, and performance bottlenecks. It also defines a complete migration process including status marking, data replication, dual-write synchronization, version verification, route switching, and data cleanup, achieving automated and secure adjustment of attribute storage locations. Through clear triggering conditions and a standardized migration process, the system can automatically detect changes in business query patterns and adaptively adjust attribute storage locations, avoiding the long-term invalidation of static storage layouts. Simultaneously, dual-write synchronization and version verification mechanisms ensure data integrity and consistency during migration, achieving a balance between dynamic optimization of storage layout and data security.

[0085] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A hybrid row-column storage method for graph databases with sparse edge attributes, characterized in that, Includes the following steps: Collect edge attribute features; Calculate edge attribute sparsity scores to assess the degree of attribute sparsity; Calculate the edge attribute access popularity score to assess the frequency of attribute access; By combining the sparsity score and the access popularity score, the storage category of the edge attribute is determined. The storage category includes in-row high-frequency attributes, out-of-column sparse attributes, candidate out-of-column attributes, candidate back-of-column attributes, and bi-state observation attributes. Construct an edge body inline storage structure. The edge body record contains a fixed edge field and an inline high-frequency attribute area. The inline high-frequency attribute area is used to store edge attributes that are determined to be inline high-frequency attributes. Construct an off-column sparse attribute storage structure. The off-column sparse attribute area is organized according to relation type and attribute field. It is used to store edge attributes that are determined to be off-column sparse attributes, and only records the edge identifier and attribute value that actually exist. Establish an edge attribute row-column mapping index to record the current storage location, storage mode, and migration status of each edge attribute. According to the storage category, write each edge attribute into the corresponding storage structure and update the edge attribute row-column mapping index synchronously. The query executor first accesses the edge attribute row and column mapping index to determine the storage location and migration status of each attribute. For in-row attributes in a stable storage state, they are read directly along with the edge main record. For out-of-column attributes in a stable storage state, they are read only when required by the filtering conditions or the returned field. For attributes in a migration state, they are read from the source location or the double-write location according to the migration stage. Based on changes in edge attribute sparsity and access frequency, the system dynamically performs migration operations, moving in-row attributes to out-of-column or out-of-column attributes back to in-row. Consistency is maintained during attribute migration, and query continuity and write consistency are ensured through migration status management and version verification.

2. The hybrid row-column storage method for graph database edge attribute sparsity according to claim 1, characterized in that, The calculation of edge attribute sparsity score includes: For attribute field p under relation type r, calculate the attribute occurrence rate: ， Among them, E r E represents the total number of edges under relation type r. r,p This represents the number of edges with attribute p under relation type r; Attribute sparsity is calculated using the following formula: Sparse(r,p) = 1 - AppearRate(r,p) In this case, the closer Sparse(r,p) is to 1, the sparser the attribute p is; the closer it is to 0, the more common the attribute p is.

3. The hybrid row-column storage method for graph database edge attribute sparsity according to claim 2, characterized in that, The calculation of edge attribute sparsity score further includes calculating a comprehensive sparsity score: SPS(r,p)=α1 Sparse(r,p)+α2 norm(NullRate(r,p))+α3 norm(AvgLen(r,p))-α4 norm(CompressGain(r,p)) Where SPS(r,p) represents the comprehensive sparsity score of attribute p under relation type r, NullRate(r,p) represents the proportion of null values ​​of the attribute, AvgLen(r,p) represents the average length of the attribute, CompressGain(r,p) represents the columnar compression gain, norm represents the normalization function, and α1 to α4 are weight parameters.

4. The hybrid row-column storage method for graph database edge attribute sparsity according to claim 3, characterized in that, The calculation of edge attribute access popularity score includes: For attribute field p under relation type r, calculate the access popularity score: Where AHS(r,p) represents the attribute access popularity score, ReadFreq(r,p) represents the attribute reading frequency, FilterFreq(r,p) represents the frequency of the attribute being used as a filter condition, PathHit(r,p) represents the frequency of the attribute being hit in path queries, EarlyUse(r,p) represents whether the attribute is used in the early stage of adjacency expansion, UpdateFreq(r,p) represents the attribute update frequency, norm represents the normalization function, and β1 to β5 are weight parameters.

5. The hybrid row-column storage method for graph database edge attribute sparsity according to claim 4, characterized in that, The determination of the storage category for edge attributes includes calculating a mixed row-column storage decision value: Where HCS(r,p) represents the hybrid row and column storage decision value, ColumnReadCost(r,p) represents the off-column read cost, RowBenefit(r,p) represents the traversal benefit brought by in-row storage, MigrationCost(r,p) represents the attribute migration cost, norm represents the normalization function, and γ1 to γ5 are weight parameters. Based on HCS(r,p) and SPS(r,p), the attributes are divided into in-row high-frequency attributes, out-of-column sparse attributes, candidate out-of-column migration attributes, candidate back-of-column migration attributes, and bimorphic observation attributes.

6. The hybrid row-column storage method for graph database edge attribute sparsity according to claim 1, characterized in that, The constructed edge body inline storage structure includes: Set a fixed edge field in the edge body record. The fixed edge field includes edge identifier, start point identifier, end point identifier, relationship type, direction marker, version and timestamp. An inline attribute bitmap is set in the edge body record, and the inline attribute bitmap is used to mark the existence state of each attribute in the inline high-frequency attribute area; In the edge body record, a high-frequency attribute area is set up, and attribute fields that are determined to be high-frequency attributes are written into the high-frequency attribute area in a preset order. In the edge body record, off-column sparse attribute positioning information is set, which is used to point to the corresponding record of the edge in the off-column sparse attribute area.

7. The hybrid row-column storage method for graph database edge attribute sparsity according to claim 1, characterized in that, The construction of the out-of-column sparse attribute storage structure includes: The storage area is divided according to the relation type and attribute field, and an independent off-column sparse attribute area is created for each attribute field under each relation type; Write the relation type and attribute field identifier in the sparse attribute area outside the column; Scan the edge records with this attribute, write the set of edge identifiers with this attribute into the edge identifier list, write the corresponding attribute values ​​into the attribute value column in order, and write the version information into the version column. Set a null bitmap for the attribute value column to mark the default state, set an offset index for the variable-length attribute, and configure compressed metadata to record the compression algorithm and encoding method; For edges that do not have this attribute, do not write an empty placeholder value.

8. The hybrid row-column storage method for graph database edge attribute sparsity according to claim 1, characterized in that, The establishment of the edge attribute row-column mapping index includes: Create an attribute layout directory, which uses the relation type and attribute field identifier as index keys; Write the storage mode of each attribute in the attribute layout directory. The storage mode includes in-row stable storage, out-of-column stable storage, in-row to out-of-column migration, and out-of-column to in-row migration. Write the sparse score, access popularity score, and decision score for each attribute in the attribute layout directory; The offset position of inline attributes and the column file or storage block location of off-column attributes are recorded in the attribute layout directory; Set a migration status field in the attribute layout directory to record whether the attribute is currently in the migration process and record the update time of the layout information; When accessing edge attributes, the query executor first queries the attribute layout directory and decides whether to read directly from the edge body record or from the off-column sparse attribute area based on the storage mode.

9. The hybrid row-column storage method for graph database edge attribute sparsity according to claim 8, characterized in that, The phased attribute reading during query execution includes: Parse the query statement and extract the relation type, edge attribute filtering conditions, path pattern, return fields, and sorting fields; The query executor accesses the attribute layout directory to determine the storage mode and migration status of the attributes involved in the query; For attributes that are in a stable storage state within the row, they are read directly along with the main record. For attributes that are in stable storage outside the column, read them only when needed by filter conditions, return fields, or subsequent operators; For attributes in the migration state, read from the source storage location, the target storage location, or the double-write location, depending on the current migration stage. For queries with large result sets, pre-filtering is performed using inline attributes first, and then out-of-column attributes are read in batches according to the candidate edge identifier; The results of batch reading of external attributes are concatenated to form the final edge attribute result.

10. The hybrid row-column storage method for graph database edge attribute sparsity according to claim 1, characterized in that, Dynamically performing attribute migrations includes: The triggering conditions for migrating inline attributes to column-side attributes include: attribute sparsity consistently exceeding the threshold, attribute access frequency consistently falling below the threshold, attribute average length increasing leading to excessively wide side records, attribute columnar compression yielding significantly higher benefits than inline storage, attributes no longer participating in path filtering or early traversal, and inline attribute region expansion leading to decreased adjacency scan performance. Triggering conditions for migrating out-of-column attributes back to in-row include: continuously increasing attribute access popularity, attributes becoming high-frequency filtering conditions, attributes being frequently used in the early stages of path queries, out-of-column read costs exceeding in-row storage benefits, increased attribute occurrence rates leading to decreased sparsity, and frequent out-of-column table lookups during query execution. During the migration process, the following steps are executed sequentially: marking migration status, scanning and copying attribute values, dual-write synchronization, version verification, route switching, and cleaning up source location data, to ensure data consistency during the migration.

Images