Cloud native resource description framework data storage system, method and computer device

Abstract

The application relates to a cloud-native resource description framework data storage system, method and computer device. The system comprises a storage layer for storing triples sharing the same attribute in resource description framework graph data as a subgraph and storing the subgraph in cloud object storage, and constructing a range-aware offset index for each subgraph; a computing layer for receiving a resource description framework query, decomposing the resource description framework query into multiple subqueries according to attributes, querying the range-aware offset index, obtaining byte offsets of target data values in each subquery in the corresponding subgraph, determining a target access method for each subquery from multiple preset data access methods based on a query processing efficiency model according to the byte offsets, executing the target access method to output a subquery result, and connecting the results of the subqueries to generate a query answer. The system can greatly reduce the operation cost of storage and query while maintaining competitive query performance.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a cloud-native resource description framework data storage system, method and computer device. Background Technology

[0002] With the development of cloud computing and web data management technologies, Resource Description Framework (RDF), as the standard model for data exchange on the web, has experienced explosive growth in data volume, and managing RDF data in the cloud has become a mainstream solution. The basic unit of RDF is the triple, and an RDF dataset can be abstracted as a graph. SPARQL is the standard query language for querying and manipulating RDF graphs. Its query essence is to find a subgraph in the RDF graph that is homomorphic to the query graph.

[0003] Currently, most traditional RDF storage systems are built on a compute-deposit integrated architecture, deploying data storage and query computation on the same node, and completing SPARQL query processing through fixed resource allocation. Meanwhile, while modern cloud-native relational databases adopt a compute-deposit separation architecture, they are not optimized for the RDF data model or SPARQL query semantics; and some big data RDF engines that borrow vertical partitioning strategies are only built on HDFS or Spark, failing to fully utilize the characteristics of cloud object storage. However, all of the above approaches have significant shortcomings in a cloud environment: compute-deposit integrated architectures struggle to dynamically scale compute and storage resources according to workload fluctuations when deployed directly, often resulting in low resource utilization and unnecessary operational expenses; databases and engines under compute-deposit separation architectures either lack compatibility with RDF and SPARQL or suffer from high usage costs due to a failure to optimize for cloud storage I / O patterns and cost structures. Summary of the Invention

[0004] Therefore, it is necessary to provide a cloud-native resource description framework data storage system, method, and computer device to address the aforementioned technical issues.

[0005] A cloud-native resource description framework data storage system, the system comprising: The storage layer is used to store triples that share the same attribute in the resource description framework graph data as a subgraph and store them in cloud object storage. A range-aware offset index is constructed for each subgraph. The range-aware offset index records the byte offset range of the data value in the corresponding subgraph. The data value includes the value of the subject or object of the triple. The computation layer receives resource description framework queries, decomposes the resource description framework queries into multiple sub-queries according to attributes, queries the range-aware offset index to obtain the byte offset of the target data value in each sub-query in the corresponding subgraph, and, based on the query processing efficiency model, determines the target access method for each sub-query from multiple preset data access methods according to the byte offset. It executes the target access method by calling the cloud object storage access interface and outputs the sub-query results. The results of each sub-query are connected to generate the query answer. The query processing efficiency model is constructed with cloud storage access cost and execution time as parameters, with the goal of minimizing the total cost while meeting preset time constraints.

[0006] A cloud-native resource description framework data storage method, the method comprising: Triples sharing the same attribute in the resource description framework graph data are stored as a subgraph and stored in cloud object storage. A range-aware offset index is constructed for each subgraph. The range-aware offset index records the byte offset range of the data value in the corresponding subgraph. The data value includes the value of the subject or object of the triple. The system receives a resource description framework query, decomposes the query into multiple sub-queries based on attributes, queries the range-aware offset index to obtain the byte offset of the target data value in each sub-query within the corresponding subgraph, and, based on a query processing efficiency model, determines the target access method for each sub-query from multiple preset data access methods according to the byte offset. The system then executes the target access method by calling the cloud object storage access interface, outputting the sub-query results, and concatenates the results of each sub-query to generate the query answer. The query processing efficiency model is constructed with cloud storage access cost and execution time as parameters, aiming to minimize the total cost while satisfying preset time constraints.

[0007] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Triples sharing the same attribute in the resource description framework graph data are stored as a subgraph and stored in cloud object storage. A range-aware offset index is constructed for each subgraph. The range-aware offset index records the byte offset range of the data value in the corresponding subgraph. The data value includes the value of the subject or object of the triple. The system receives a resource description framework query, decomposes the query into multiple sub-queries based on attributes, queries the range-aware offset index to obtain the byte offset of the target data value in each sub-query within the corresponding subgraph, and, based on a query processing efficiency model, determines the target access method for each sub-query from multiple preset data access methods according to the byte offset. The system then executes the target access method by calling the cloud object storage access interface, outputting the sub-query results, and concatenates the results of each sub-query to generate the query answer. The query processing efficiency model is constructed with cloud storage access cost and execution time as parameters, aiming to minimize the total cost while satisfying preset time constraints.

[0008] The aforementioned cloud-native resource description framework data storage system, method, and computer device, by vertically dividing the resource description framework graph data into subgraphs according to attributes and storing them in cloud object storage, can shift storage costs from expensive block storage to low-cost object storage. A range-aware offset index is built for each subgraph, recording the byte offset range of the subject or object value within the subgraph. This enables precise location of the required data's physical location during data access, avoiding unnecessary data transfer overhead from full file scanning. Queries are decomposed into multiple subqueries according to attributes, and the byte offset of the target data value is obtained by querying the index, transforming global queries into targeted access to specific subgraphs, reducing the amount of data involved in a single query. By constructing and solving a query processing efficiency model to determine the target access method, the query execution engine can automatically select the lowest-cost data access path while ensuring response time meets user requirements, achieving a dynamic balance between query performance and operational costs. This invention significantly reduces storage and query operational costs while maintaining competitive query performance. Attached Figure Description

[0009] Figure 1 This is an application scenario diagram of a cloud-native resource description framework data storage method in one embodiment. Figure 2 This is a schematic diagram illustrating an example of the influencedBy attribute data file and ROI index in one embodiment. Figure 3 This is a performance comparison chart of access methods under different attribute file sizes in one embodiment, where, Figure 3 (a) is a performance comparison chart for a 1M property file. Figure 3 (b) is a performance comparison chart for a 10M property file. Figure 3 (c) is a performance comparison chart for a 100M property file. Figure 3 (d) is a performance comparison chart for a 1000M property file; Figure 4 This is a diagram illustrating the cost comparison of access methods under different attribute file sizes in one embodiment. Figure 4 (a) is a cost comparison chart under a 1M attribute file. Figure 4 (b) is a cost comparison chart under the 10M attribute file. Figure 4 (c) is a cost comparison chart for a 100M attribute file. Figure 4 (d) is a cost comparison chart for a 1000M attribute file; Figure 5 This is a performance and cost diagram illustrating how the access method varies with the size of the attribute file in one embodiment; wherein, Figure 5 (a) is a performance diagram of access methods under different attribute file sizes. Figure 5 (b) A schematic diagram illustrating the cost of access methods for different attribute file sizes; Figure 6 This is a comparison chart of index hit rates under different α values ​​in one embodiment; Figure 7 This is a schematic diagram of query decomposition in one embodiment; Figure 8 This is a pseudocode diagram of the branch and bound algorithm in one embodiment; Figure 9 This is a schematic diagram illustrating the performance comparison results in one embodiment, wherein, Figure 9 (a) is a comparison chart of query response times on the WatDiv 100M dataset. Figure 9 (b) is a comparison chart of query response times on the WatDiv 1B dataset. Figure 9 (c) is a comparison chart of query response times on the DBpedia dataset. Figure 9 (d) is a comparison chart of query response times on the Wikidata dataset; Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0011] In one embodiment, such as Figure 1 As shown, a cloud-native resource description framework data storage system is provided, including: The storage layer is used to store triples that share the same attribute in the resource description framework graph data as a subgraph and store them in cloud object storage. A range-aware offset index is built for each subgraph. The range-aware offset index records the byte offset range of the data value in the corresponding subgraph. The data value includes the value of the subject or object of the triple. The computation layer receives resource description framework queries, decomposes them into multiple subqueries based on attributes, uses a query range-aware offset index to obtain the byte offset of the target data value in each subquery within the corresponding subgraph, and, based on a query processing efficiency model, determines the target access method for each subquery from multiple preset data access methods according to the byte offset. It then executes the target access method by calling the cloud object storage access interface, outputs the subquery results, and concatenates the results of each subquery to generate the query answer. The query processing efficiency model is constructed with cloud storage access cost and execution time as parameters, aiming to minimize the total cost while satisfying preset time constraints.

[0012] Specifically, such as Figure 1 As shown, HuGraph follows a storage-compute separation architecture, consisting of a storage layer and a compute layer. The storage layer is responsible for storing the RDF graph in the cloud, leveraging low-cost cloud object storage services (such as OSS) to reduce storage costs. The compute layer handles user-submitted SPARQL queries, built on elastic cloud computing services (such as EC2). HuGraph is cloud-independent and can be easily deployed on mainstream cloud platforms such as AWS and Alibaba Cloud. In the storage layer, the core design grouped the RDF graph into subgraphs by attributes, storing each subgraph as an OSS object (with the attribute name as the object key), supplemented by range-aware offset indexes to optimize access performance. In the compute layer, the input SPARQL query is first decomposed into multiple subqueries (each subquery corresponding to an attribute subgraph); then, the optimal access method is selected for each subquery (balancing cost and performance); finally, a complete query execution plan is generated and executed, and the results of all subqueries are concatenated to obtain the final answer. The storage layer and compute layer interact with each other through standard network protocols and access interfaces provided by cloud storage services. After determining the target access method, the computing layer initiates a data request to the storage layer by calling the cloud object storage's retrieval operation interface or selection operation interface; the storage layer returns the corresponding subgraph data or data within a specified range based on the request type and parameters.

[0013] Figure 1The example uses GET and S3 Select to illustrate the connection between the compute and storage layers. However, it's important to note that these operation names are specific to AWS S3 services, and the actual connection methods used in the system are not limited to these. On different cloud platforms, the corresponding functions may be provided with different names; for example, in Alibaba Cloud OSS, they are called GetObject and SelectObject operations. Furthermore, any cloud storage access interface that supports retrieving the entire object on request or a portion of an object by range, and that supports data filtering queries on the storage side, can serve as a connection between the two layers. The system's query processing efficiency model abstracts the common characteristics of these operations in terms of billing rules and transmission characteristics—i.e., whether billing is based on the number of requests or the amount of data, and whether filtering pushdown is supported—rather than relying on the specific implementation of a particular cloud platform. Therefore, the connection method between the two layers can be flexibly replaced with other cloud storage access interfaces with the same technical characteristics depending on the actual cloud environment deployed, while the overall system architecture and optimization strategy remain unchanged.

[0014] In the aforementioned cloud-native resource description framework (RMR) data storage system, by vertically dividing the RMR graph data into subgraphs according to attributes and storing them in cloud object storage, storage costs can be shifted from expensive block storage to low-cost object storage. A range-aware offset index is built for each subgraph, recording the byte offset range of the subject or object value within the subgraph. This allows for precise location of the required data's physical location during data access, avoiding unnecessary data transfer overhead from full file scans. Queries are decomposed into multiple subqueries based on attributes, and the byte offset of the target data value is retrieved from the index, transforming global queries into targeted access to specific subgraphs. This reduces the amount of data involved in a single query. By constructing and solving a query processing efficiency model to determine the target access method, the query execution engine can automatically select the lowest-cost data access path while ensuring response time meets user requirements, achieving a dynamic balance between query performance and operational costs. This embodiment of the invention can significantly reduce storage and query operational costs while maintaining competitive query performance.

[0015] In one embodiment, the range-aware offset index includes a file header and an index table; the file header is used to record the total number of bytes of the corresponding subgraph; each record in the index table includes a data value, a position identifier, a start offset, and an end offset, and the position identifier is used to indicate whether the data value is the value of the subject or object of the triple.

[0016] Specifically, storing the entire RDF graph as a single file object results in scanning the entire file for every query, leading to extremely low efficiency. Therefore, this invention employs a vertical partitioning strategy: dividing the RDF graph into subgraphs based on attributes, with each subgraph containing all triples sharing the same attribute, stored as an independent file object. The rationale for this design lies in the fact that the number of attributes in RDF data is typically limited, and SPARQL query variables rarely appear at attribute positions. To further improve retrieval locality, triples in each subgraph are sorted by their object values—this leverages the observation that object positions are often constant in actual SPARQL queries. To improve performance, a mapping dictionary replaces all IRIs and literals with unique integer IDs, compressing triple storage and simplifying query processing. The mapping dictionary is also stored as a file object, loaded into memory during EC2 instance initialization, and uses LevelDB for fast ID lookup.

[0017] There are two basic ways to access OSS or S3 data: GET: Low-cost way to retrieve the entire file, but requires filtering on the computing side and takes a long time when processing large amounts of data.

[0018] SELECT: Pushing the filter down to the storage layer results in better performance, but costs are higher as it requires payment based on the amount of data scanned and returned.

[0019] To combine the advantages of both, this invention designs a range-aware offset index (ROI) that enables efficient data retrieval based on the Range GET and Range SELECT functions of OSS or S3.

[0020] After partitioning triples sharing the same attribute into a subgraph, each subgraph corresponds to an independent object in cloud object storage. For each such object, a range-aware offset index file is constructed and associated with it. The index file is logically divided into two parts: a file header and an index table. The file header records the total number of bytes of the subgraph object, which is used as a baseline for the total data volume when calculating access costs and time estimations. Each record in the index table corresponds to a data value with index value (i.e., the value of a subject or the value of an object), records the start and end byte offsets of the value at all occurrences of the value in the subgraph object, and distinguishes whether the value appears as a subject or an object through position identifiers.

[0021] For the file object corresponding to attribute p, its ROI index contains an index table and a file header used to record the total number of bytes of the corresponding file object. Each record in the index table is (Value, Position, StartOffset, EndOffset), where Value is the value of the subject or object in the triple, Position indicates whether the value is the subject or object, and StartOffset / EndOffset: the byte range of all triples containing this value in the file object.

[0022] Accessing data using this index involves two stages: First, a GET or SELECT operation is used to query the index and obtain the byte offset corresponding to the target value. Second, a Range GET or Range SELECT operation is used to directly access the specified byte range of the subgraph object, retrieving only the required data. This approach significantly reduces data transfer and scanning volume, balancing performance and cost.

[0023] Furthermore, larger indexes are not always better: excessively large indexes increase storage costs and index scan time, while excessively small indexes have low hit rates. Therefore, this invention adds index entries incrementally from high to low selectivity, and defines index usefulness conditions to ensure index effectiveness: ; in, For attributes The corresponding range-aware offset index, Function for calculating byte size. For each index entry, 0 < α ≤ 1 is a threshold parameter that controls the strictness of the index size. Only indexes that meet this condition will be retained. For the data selectivity of attribute P. For example... Figure 2 The image shows an example of the data file and ROI index for the attribute influenceBy.

[0024] In addition to attribute-based partitioning, auxiliary data file objects (SPO or OSP layouts) organized by subject or object can be created to accelerate triple pattern queries where constants appear in the subject or object positions. However, this method introduces data copying, increasing storage costs and maintenance complexity; therefore, such auxiliary files are only generated for low-selectivity data.

[0025] For data maintenance, HuGraph supports dynamic insertion and deletion of triples. Deletion involves: downloading the file object corresponding to the attribute, marking the triple to be deleted as an invalid ID (placeholder), and re-uploading the file. The offsets in the index do not need to be modified. Insertion involves: downloading the file object corresponding to the attribute, adding the new triple and reordering it, uploading the new file; and simultaneously updating the offset records for the corresponding subject and object in the ROI index.

[0026] In this embodiment, by constructing a range-aware offset index of the above structure for each subgraph, a fast mapping from data values ​​to physical storage locations can be established with a small index storage overhead. This allows for accurate location of the target data's byte range without traversing the entire subgraph file during query processing, reducing unnecessary data transmission.

[0027] In one embodiment, a variety of preset data access methods include a method to directly obtain the entire object, a method to directly perform a selection query on the object, a method to obtain the index first and then obtain the data by range, a method to obtain the index first and then select the data by range, a method to select the index first and then obtain the data by range, and a method to select the index first and then select the data by range.

[0028] Specifically, combining the basic operations of object storage services and range-aware offset indexing, a total of six data access methods were designed: The direct GET method (DG) retrieves the entire subgraph object without using an index, using the standard GET operation. The cost is only charged based on the number of requests, regardless of the amount of data.

[0029] The Direct Selection method (DS, Direct SELECT) does not use indexes; it directly scans and filters subgraph objects on the storage side using SELECT operations, and is billed based on the amount of data scanned and returned.

[0030] The index range retrieval method (GG, index range GET) is executed in two steps: First, a GET operation is used to query the range-aware offset index to obtain the byte offset corresponding to the target data value; then, a Range GET operation is used to retrieve data from the subgraph object according to that offset range. Both steps are billed based on the number of requests and are suitable for scenarios that reduce data transfer volume by narrowing the data reading range through indexing.

[0031] The index range selection method (GS, index range SELECT) first uses a GET operation to query the index to obtain the offset, and then uses a Range SELECT operation to select data on the storage side according to the offset range. Index retrieval costs are low, while data filtering is performed on the storage side.

[0032] Choosing the index range retrieval method (SG, SELECT index range GET) involves first using a SELECT operation to query the index and obtain the offset, then using a Range GET operation to retrieve the data within that offset range. Index queries can be pushed down for filtering, resulting in low data retrieval costs.

[0033] The Select Index Range (SS, SELECT Index Range SELECT) method uses SELECT operations in both steps, allowing for pushdown filtering in both index lookup and data retrieval, making it suitable for scenarios with high performance requirements.

[0034] like Figure 3 As shown, the performance gap between GET and SELECT operations gradually narrows as the proportion of returned data increases. This indicates that for low-selectivity triple patterns (i.e., when retrieving a large proportion of data from a subgraph object), the GET-based access method can achieve comparable performance to the SELECT-based access method, while incurring significantly lower monetary costs. Figure 5 As shown in (a), the performance and cost of the GET operation are linearly positively correlated with the size of the subgraph file and are basically insensitive to the size of the result data; the performance and cost of the SELECT operation are positively correlated with both the size of the subgraph file and the size of the result data.

[0035] like Figure 4 As shown, when querying large-scale subgraph files, although the SELECT operation has a performance advantage when the result set is small, the significant scan overhead caused by scanning the entire file dominates the overall query cost. For example... Figure 5 As shown in (b), as the size of the subgraph file decreases, the proportion of computational cost in the total query cost increases. At this point, a trade-off exists between the performance advantage of the SELECT operation and the cost advantage of the GET operation. Therefore, for large subgraph files, GET methods are less expensive; for small subgraph files, a choice must be made considering both performance and cost.

[0036] Experimental results show that for any attribute, the performance and cost of data retrieval are closely related to the size ratio of the subgraph object and its corresponding index file. As the threshold parameter α (used to control the strictness of index entries) increases, the performance of the access method using range-aware offset indexing first improves and then decreases, while the cost increases monotonically. Simultaneously, the index hit rate increases monotonically with increasing α, such as... Figure 6 As shown. Therefore, for each attribute, there exists an optimal threshold parameter α that determines the appropriate index entry size, thus affecting the choice of the optimal access method. Since each subquery can only use one data access method at a time, based on the above experimental conclusions, the choice of the optimal access method depends on a comprehensive trade-off between the subgraph file size, the amount of data returned, and the index size.

[0037] In summary, the experimental results show that the performance and cost of GET methods are linearly positively correlated with the size of the subgraph file and largely independent of the amount of data returned. For SELECT methods, both the performance and cost are positively correlated with the size of the subgraph file and the amount of data returned. For large subgraph files, GET methods are less expensive; for small subgraph files, a trade-off exists between the performance advantage of SELECT methods and the cost advantage of GET methods. As the proportion of returned data increases, the performance gap between GET and SELECT methods gradually narrows. Therefore, the optimal access method depends on the subgraph file size, the amount of data returned, and the index size. Based on these conclusions, the query processing efficiency model calculates the cloud storage access cost and execution time for each data access method by differentiating the calculations based on the per-request billing characteristic of GET methods (independent of data volume) and the per-scan and per-return billing characteristic of SELECT methods (positively correlated with data volume). This allows for the determination of the target access method for each subquery, achieving a refined trade-off between cost and performance.

[0038] In one embodiment, querying the range-aware offset index to obtain the byte offset of the target data value in each subquery in the corresponding subgraph includes: extracting the target data value as a constant from the triple pattern of the subquery; querying the range-aware offset index to obtain the start offset and end offset corresponding to the target data value, and using the start offset and end offset as the byte offset of the target data value in the corresponding subgraph.

[0039] In this embodiment, after the query is decomposed into multiple subqueries by attribute, each subquery corresponds to a triplet pattern. Since the object position in the actual query is often a constant, this constant value is extracted as the target data value. The target data value is matched with the range-aware offset index of the corresponding attribute: if a record of the target data value exists in the index, its start and end offsets are directly read; if a record of the target data value does not exist in the index, it means that the constant value has no matching data in the subgraph, the subquery result is empty, and no further actual data access is required. By extracting constants from subqueries and querying the index to obtain byte offsets, the query conditions can be transformed into precise data reading ranges. On the one hand, this reduces the operation that originally required scanning the entire subgraph file to reading only the relevant byte range, significantly reducing the amount of data read and network transmission overhead; on the other hand, it allows determining whether the query is likely to return results during the index query stage, terminating the process in advance for cases without a match, and avoiding invalid calls to the cloud storage service.

[0040] Specifically, such as Figure 7The query decomposition diagram shown illustrates that, because the RDF graph is stored as independent file objects by attribute, the SPARQL query is first decomposed into multiple subqueries, each containing all triple schemas that share the same attribute. This way, each subquery only needs to access one file object to retrieve all relevant data. Figure 7 The query contains 5 triple patterns involving 4 different attributes, and therefore is broken down into 4 subqueries. Two of the triple patterns share the attribute `influencedBy`, and are merged into one subquery.

[0041] In one embodiment, determining the target access method for each subquery from multiple preset data access methods based on the query processing efficiency model and according to the byte offset includes: calculating the cloud storage access cost and execution time of each data access method when obtaining the corresponding subquery result based on the byte offset, the billing rules and transmission characteristics of each preset data access method; searching for access method allocation schemes for each subquery with the goal of minimizing the total cost and the constraint of satisfying preset time constraints; calculating the upper and lower bounds of the cost for each partial allocation scheme in the search process, and pruning the partial allocation schemes whose lower bound exceeds the current global optimal upper bound of cost, until the target access method for each subquery is determined; the upper bound of cost is the total cost obtained by using the preset high-cost method for all unallocated subqueries, and the lower bound of cost is the total cost obtained by using the preset low-cost method for all unallocated subqueries.

[0042] In this embodiment, the acquisition operation interface is charged based on the number of requests, which is low-cost but requires the transmission of the entire subgraph data, making it suitable for subgraphs with small data volumes. The selection operation interface pushes filtering down to the storage side for execution, offering better performance but is charged based on the amount of data scanned and returned, resulting in higher costs. For a given subquery, the byte offset determines the amount of data to be acquired. Combining the characteristics of each access method, such as whether an index is used and whether filtering is pushed down, the estimated cost and time for each method can be calculated. When searching for the optimal allocation scheme, the entire process is constructed as a search tree, with each node corresponding to a partial allocation scheme (i.e., the access method for some subqueries has been determined, while the access methods for the remaining subqueries have not yet been determined). For each partial allocation scheme, its upper and lower cost bounds are calculated. During the search process, if a complete allocation scheme that satisfies the time constraint and has a lower total cost is found, the global optimal cost upper bound is updated. If the cost lower bound of a certain partial allocation scheme exceeds the current global optimal cost upper bound, then no matter how the scheme is subsequently allocated, its final cost cannot be better than the currently found optimal solution, so the branch is pruned and the search is stopped. Through the above search and pruning process, the optimal access method for each subquery can be determined from an exponential number of candidate solutions within a limited time. Under the premise of ensuring that the query response time meets the user requirements, the combination solution with the lowest total cost is automatically selected, so as to achieve the optimal trade-off between cost and performance between query processing on the computing side and data access on the storage side.

[0043] Specifically, each subquery can choose one of six access methods, with the goal of minimizing the total query cost while satisfying the user-specified time constraint. Since actual SPARQL queries typically contain no more than 10 edges, this invention employs a branch and bound algorithm to find the globally optimal solution. First, a search tree is constructed, with each node representing a partial allocation of access methods. For each partial solution, upper and lower bounds of the cost are calculated: upper bound: assuming all unallocated subqueries use the highest-cost SELECT method; lower bound: assuming all unallocated subqueries use the lowest-cost GET method. A depth-first strategy is used to explore the search tree, prioritizing the deepest unpruned node with the smallest upper bound for branching. If the lower bound of a subproblem exceeds the current globally optimal upper bound, the branch is pruned.

[0044] The worst-case time complexity of the algorithm is However, in practical applications, branch and bound pruning is highly effective and can quickly find the optimal solution. The algorithm pseudocode is as follows: Figure 8 As shown. After determining the access method for each subquery, HuGraph calls the corresponding cloud API to obtain the subquery results, and then concatenates all the results to obtain the final answer.

[0045] In one embodiment, executing a target access method by calling the cloud object storage access interface and outputting a subquery result includes: according to the target access method, calling the cloud object storage's get operation interface or select operation interface to obtain all data of the corresponding subgraph or to obtain data of a specified byte range in the corresponding subgraph based on the byte offset, and outputting the obtained data as the subquery result; wherein, the get operation interface is used to obtain subgraph data and is billed based on the number of requests, and the select operation interface is used to execute a select query on the subgraph and is billed based on the amount of data scanned and the amount of data returned.

[0046] Specifically, the cloud object storage service provides two basic data access interfaces: the retrieval operation interface returns the entire object or a specified byte range of data to the computing end, with billing calculated based on the number of requests, independent of data volume, resulting in extremely low cost; the selection operation interface scans and filters data on the storage end, returning only data that meets the criteria to the computing end, with billing calculated separately for the scanned and returned data volumes, offering better performance but at a higher cost. Furthermore, both interfaces support range read functionality, allowing the retrieval of only a portion of the object's data by specifying a byte offset. When the target access method is an index-based range read type, the start and end offsets obtained by querying the index are passed to the access interface, retrieving only the data segment corresponding to that offset range in the subgraph, rather than the entire subgraph file. By selecting the appropriate access interface and parameters based on the target access method, the actual cost of each cloud storage access can be precisely matched to the query requirements: for scenarios requiring the retrieval of a small amount of data, range reads transmit only the necessary bytes; for scenarios requiring the full retrieval, the low request cost advantage of the retrieval operation interface is utilized—achieving an optimal balance between cost and performance in different scenarios.

[0047] In one embodiment, generating a query answer by connecting the results of each subquery includes: using a dynamic programming algorithm to determine the connection order of the results of each subquery, estimating the cloud storage access cost of each candidate connection plan based on the cardinality of intermediate results, selecting the connection plan with the lowest cloud storage access cost as the target connection order, and connecting the results of each subquery according to the target connection order to generate a query answer; wherein, when determining the connection order, the most selective triplet pattern is not processed first.

[0048] Specifically, to determine the optimal join order, execution plans are iteratively constructed for subsets of subqueries: starting with a single subquery, more subqueries are gradually merged, and the size of intermediate results generated by different join methods is evaluated for each subset. Based on the cardinality of the intermediate results, the cloud storage access cost of each candidate plan is estimated. Unlike traditional databases that prioritize processing the most selective subqueries, this system does not prioritize processing the most selective triplet patterns when determining the join order. This is because, in a compute-storage separation architecture, while processing highly selective subqueries first can reduce the size of intermediate results, it leads to multiple scattered requests to the cloud storage service with small data volumes, thus increasing the number of requests, data transmission rounds, and overall latency. By determining the join order based on cloud storage access cost rather than simply selectivity, the number of requests to the cloud storage service and the total amount of data transmitted can be reduced, achieving a better balance between overall query cost and execution efficiency.

[0049] In one embodiment, the computation layer also supports performing at least one of the following operations on the query results: union operation, left outer join operation, and predicate filtering operation.

[0050] Specifically, to support the complex query features of SPARQL 1.0 and some SPARQL 1.1 versions, the system performs these operations by post-processing the generated basic result sets after completing the basic graph pattern query. For the UNION operation, the system merges the result sets of multiple basic graph pattern queries; for the OPTIONAL operation, the system performs a left outer join on the relevant result sets; for the FILTER operation, the system applies a preset predicate to filter each record in the result set. These complex operators do not affect the core processes of query decomposition, access method determination, and join order optimization; they only perform secondary processing on the result sets before the final output. Through this design that decouples basic graph pattern queries from complex operators, the system can support rich query semantics while maintaining a simple and efficient core query optimization process, meeting diverse query needs in practical applications.

[0051] This invention proposes HuGraph, a cloud-native resource description framework storage system designed specifically for cloud environments. It employs a storage-compute separation architecture, with the core objective of minimizing operational costs while meeting user query performance requirements. At the storage layer, HuGraph organizes the resource description framework graph by grouping all triples sharing the same attribute into dedicated subgraphs. Each subgraph is stored as an independent object in a remote shared object storage, with costs an order of magnitude lower than remote block storage. A range-aware offset index is also designed to record the byte offset range of data values ​​within the subgraph object, balancing data access performance and overhead to achieve efficient range-based data retrieval. At the compute layer, HuGraph uniformly models the cost and execution time of various cloud storage operations, transforming query execution into a time-constrained cost minimization optimization problem, proving it to be NP-hard, and proposing an efficient branch-and-bound algorithm to find the optimal execution strategy. Comprehensive experimental results on multiple cloud platforms demonstrate that HuGraph, compared to traditional resource description framework storage systems, significantly reduces operational costs while maintaining competitive query performance in real-world workloads, and exhibits good scalability and cloud platform independence.

[0052] The effectiveness of this invention was verified through experiments. A comprehensive experiment was conducted in an Amazon cloud environment using three synthetic datasets and two real datasets. The comparison systems included six traditional resource description framework storage systems and one cloud-hosted graph database. Performance results are as follows: Figure 9 As shown, Virtuoso, gStorePL, and HuGraph generally outperform other systems. On the WatDiv dataset, gStorePL achieved the highest performance, followed closely by HuGraph in second place. On the DBpedia dataset, HuGraph, using a lightweight elastic compute configuration with 2 cores and 8GB of memory, completed all queries in 709 seconds, close to gStore's 443-second execution time after preloading. However, gStore requires a configuration with 64 cores and 256GB of memory to load the index into memory; Virtuoso takes 1100 seconds to complete the same workload, and despite its significantly higher resource consumption, its execution time is still longer. On the Wikidata dataset, HuGraph, using the same lightweight configuration, completed all queries in approximately 1500 seconds, faster than Virtuoso's 1713 seconds with a 4-core, 16GB memory configuration; gStorePL takes 2884 seconds to complete the same workload, slower and more expensive than HuGraph. Figure 9(b) It can be seen that HuGraph's query cost is significantly lower than other systems on all datasets. On the WatDiv 1B, DBpedia, and Wikidata datasets, HuGraph's cost is only 1 / 3.5, 1 / 14.1, and 1 / 2.29 of Virtuoso's, respectively. By comparing the results on the WatDiv 100M and WatDiv 1B datasets, HuGraph's query time and cost increase linearly with the amount of data and at a slow rate, indicating that it has good scalability.

[0053] The experimental results above demonstrate that the present invention significantly reduces the operating costs of storage and querying while maintaining competitive query performance. Moreover, the performance and cost advantages become more and more obvious as the data scale grows, effectively verifying the technical effectiveness of the cloud-native resource description framework data storage system based on range-aware offset indexing and query processing efficiency model under the storage-compute separation architecture.

[0054] In one embodiment, a cloud-native resource description framework data storage method is provided, including: Triples sharing the same attribute in the resource description framework graph data are stored as a subgraph and stored in cloud object storage. A range-aware offset index is built for each subgraph. The range-aware offset index records the byte offset range of the data value in the corresponding subgraph. The data value includes the value of the subject or object of the triple. The system receives resource description framework queries, decomposes them into multiple subqueries based on attributes, uses a query range-aware offset index to obtain the byte offset of the target data value in each subquery within the corresponding subgraph, and, based on a query processing efficiency model, determines the target access method for each subquery from multiple preset data access methods according to the byte offset. It then executes the target access method by calling the cloud object storage access interface and outputs the subquery results. Finally, it connects the results of each subquery to generate the query answer. The query processing efficiency model is constructed with cloud storage access cost and execution time as parameters, aiming to minimize the total cost while satisfying preset time constraints.

[0055] For specific limitations regarding the data storage method of the cloud-native resource description framework, please refer to the limitations of the cloud-native resource description framework data storage system mentioned above, which will not be repeated here. Each module in the aforementioned cloud-native resource description framework data storage system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0056] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a cloud-native resource description framework data storage method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0057] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0058] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0059] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0060] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A cloud-native resource description framework data storage system, characterized in that, The system includes: The storage layer is used to store triples that share the same attribute in the resource description framework graph data as a subgraph and store them in cloud object storage. A range-aware offset index is constructed for each subgraph. The range-aware offset index records the byte offset range of the data value in the corresponding subgraph. The data value includes the value of the subject or object of the triple. The computation layer receives resource description framework queries, decomposes these queries into multiple sub-queries based on attributes, queries the range-aware offset index to obtain the byte offset of the target data value in each sub-query within the corresponding subgraph, and, based on a query processing efficiency model, determines the target access method for each sub-query from multiple preset data access methods according to the byte offset. It then executes the target access method by calling the cloud object storage access interface, outputs the sub-query results, and connects the results of each sub-query to generate the query answer. The query processing efficiency model is constructed with cloud storage access cost and execution time as parameters, aiming to minimize the total cost while satisfying preset time constraints. The range-aware offset index includes a file header and an index table; the file header is used to record the total number of bytes of the corresponding subgraph; each record in the index table includes the data value, a position identifier, a start offset, and an end offset, and the position identifier is used to indicate whether the data value is the value of the subject or the object of the triple; The query processing efficiency model, which determines the target access method for each subquery from multiple preset data access methods based on the byte offset, includes: Based on the byte offset, the billing rules and transmission characteristics of the various preset data access methods, calculate the cloud storage access cost and execution time of each data access method when obtaining the corresponding subquery result; With the goal of minimizing the total cost and the constraint of satisfying a preset time limit, a search is performed on the access method allocation scheme for each subquery. For each partial allocation scheme in the search process, an upper and lower cost bound are calculated, and allocation schemes whose lower cost bound exceeds the current global optimal upper cost bound are removed until the target access method for each subquery is determined. The upper cost bound is the total cost obtained by using the preset high-cost method for all unallocated subqueries, and the lower cost bound is the total cost obtained by using the preset low-cost method for all unallocated subqueries. The various preset data access methods include methods for directly obtaining the entire object, methods for directly performing selection queries on the object, methods for obtaining the index first and then obtaining data by range, methods for obtaining the index first and then selecting data by range, methods for selecting the index first and then obtaining data by range, and methods for selecting the index first and then selecting data by range.

2. The system according to claim 1, characterized in that, The query of the range-aware offset index yields the byte offset of the target data value in each subquery within the corresponding subgraph, including: Extract the target data value as a constant from the triple pattern of the subquery; Query the range-aware offset index to obtain the start offset and end offset corresponding to the target data value, and use the start offset and end offset as the byte offset of the target data value in the corresponding subgraph.

3. The system according to claim 1, characterized in that, The step of executing the target access method by calling the cloud object storage access interface and outputting the subquery result includes: According to the target access method, by calling the cloud object storage's get operation interface or select operation interface, all data of the corresponding subgraph is obtained, or data of a specified byte range in the corresponding subgraph is obtained based on the byte offset, and the obtained data is output as the subquery result; wherein, the get operation interface is used to obtain subgraph data and is charged according to the number of requests, and the select operation interface is used to execute a select query on the subgraph and is charged according to the amount of scanned data and the amount of returned data.

4. The system according to claim 1, characterized in that, The results of connecting the subqueries to generate the query answer include: A dynamic programming algorithm is used to determine the connection order of the results of each subquery. The cardinality of the intermediate results is used to estimate the cloud storage access cost of each candidate connection plan. The connection plan with the lowest cloud storage access cost is selected as the target connection order. The results of each subquery are connected according to the target connection order to generate the query answer. In determining the connection order, the most selective triplet pattern is not processed first.

5. The system according to claim 1, characterized in that, The computation layer is also used to perform at least one of the following operations on the query answer: union operation, left outer join operation, and predicate filtering operation.

6. A cloud-native resource description framework data storage method implemented in the system described in any one of claims 1-5, characterized in that, The method includes: Triples sharing the same attribute in the resource description framework graph data are stored as a subgraph and stored in cloud object storage. A range-aware offset index is constructed for each subgraph. The range-aware offset index records the byte offset range of the data value in the corresponding subgraph. The data value includes the value of the subject or object of the triple. The system receives a resource description framework query, decomposes the query into multiple sub-queries based on attributes, queries the range-aware offset index to obtain the byte offset of the target data value in each sub-query within the corresponding subgraph, and, based on a query processing efficiency model, determines the target access method for each sub-query from multiple preset data access methods according to the byte offset. The system then executes the target access method by calling the cloud object storage access interface, outputting the sub-query results, and concatenates the results of each sub-query to generate the query answer. The query processing efficiency model is constructed with cloud storage access cost and execution time as parameters, aiming to minimize the total cost while satisfying preset time constraints.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 6.

Images