Parallel Processing Method and Device for SPARQL Queries Based on Granularity Blocks

Abstract

The present invention provides a parallel processing method and apparatus for SPARQL queries based on granularity blocks, including: obtaining and parsing a query request for querying RDF data, and generating a query graph based on the query request; in response to the size of the intermediate result buffer being greater than a preset threshold when executing a query plan based on the query graph, dividing the intermediate result buffer into multiple blocks of a preset size, where the intermediate result buffer is a buffer storing intermediate results; when the task queue of any thread is empty, calling a task scheduler to schedule tasks to be executed to the thread for execution until the task queues of all threads are empty; wherein the tasks to be executed are scheduled by the task scheduler from the task queues of other threads, and the task queues of other threads are not empty, and the tasks to be executed include scanning and joining the blocks. By parallelly processing multiple data blocks with this method and achieving load balancing through a global task scheduling strategy, the query efficiency can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular, to a parallel processing method and device for SPARQL queries based on granular blocks. Background Art

[0002] The Resource Description Framework (RDF) data model was proposed to construct a network object model in the semantic web. It has been widely used in multiple fields such as the Web, government, and biology. For example, in computational biology, RDF datasets such as UniProt RDF are constructed to record experimental data. Currently, the scale of RDF datasets is growing significantly, which brings a major challenge of how to efficiently process SPARQL queries on large-scale RDF datasets.

[0003] Common RDF storage systems, such as BRAHMS, Yars2, Hexastore, SW-Store, SpiderStore, BitMat, RDF-3X, Hadoop-RDF-3X, gStore, etc., the structures of these storage systems can be roughly classified into the following types: triple tables, clustered property tables, vertically partitioned tables, and graph-based storage methods. The above systems aim to achieve scalable processing of RDF data, and they accelerate SPARQL query processing through compact storage structures, multiple indexes, and advanced query plan generation methods. For example, RDF-3X constructs six B+ tree indexes for all permutations of S, O, and P in six independent indexes to quickly locate triples that match any triple pattern. Although these index technologies can improve the join speed by orders of magnitude, the storage space overhead of the indexes limits the scalability of the query processor because the cost of storing and accessing these indexes is high. When the pattern selectivity in the query is low, even if the final result is small, these systems will scan a large amount of data, so the query efficiency is low. Summary of the Invention

[0004] The present invention provides a parallel processing method and device for SPARQL queries based on granular blocks, which is used to solve the defect of low query efficiency when performing SPARQL queries on RDF data in the prior art, and realizes parallel processing of multiple data blocks through this method, and realizes load balancing through a global task scheduling strategy, effectively improving the efficiency of SPARQL queries.

[0005] The present invention provides a parallel processing method for SPARQL queries based on granular blocks, including the following steps.

[0006] Obtain and parse a query request for querying RDF data, and generate a query graph based on the query request;

[0007] When, in response to executing a query plan based on the query graph, the size of the intermediate result buffer is greater than a preset threshold, the intermediate result buffer is divided into multiple blocks of a preset size, where the intermediate result buffer is a buffer storing intermediate results;

[0008] When the task queue of any one thread is empty, the task scheduler is called to schedule the tasks to be executed to the thread for execution until the task queues of all threads are empty;

[0009] Among them, the tasks to be executed are scheduled by the task scheduler from the task queues of other threads, and the task queues of the other threads are not empty. The tasks to be executed include scanning and joining the blocks.

[0010] According to a parallel processing method based on granularity blocks for SPARQL queries provided by the present invention, the task queue includes a starting endpoint and a preset number of ending endpoints. The thread starts executing the tasks to be executed from the starting endpoint of the task queue. The preset number is the difference between the total number of threads and 1, and the ending endpoints correspond to all other threads one by one.

[0011] According to a parallel processing method based on granularity blocks for SPARQL queries provided by the present invention, the step of, when the task queue of any one thread is empty, calling the task scheduler to schedule the tasks to be executed to the thread for execution until the task queues of all threads are empty, includes:

[0012] When the task queue of any one thread is empty, the task scheduler is called to schedule the tasks to be executed at the specified ending endpoint of the task queue of other threads to the starting endpoint of the thread, so that the thread executes the tasks to be executed until the task queues of all threads are empty;

[0013] Among them, the specified ending endpoint is the ending endpoint corresponding to the thread in the task queue of the other thread.

[0014] According to a parallel processing method based on granularity blocks for SPARQL queries provided by the present invention, the joining process includes:

[0015] Based on the quicksort algorithm, the multiple blocks divided from the intermediate result buffer are sorted to obtain sorted blocks; the merge join algorithm is iteratively used to merge adjacent sorted blocks into an ordered buffer until all the multiple blocks divided from the intermediate result buffer have been merged.

[0016] According to a parallel processing method based on granularity blocks for SPARQL queries provided by the present invention, the joining process includes:

[0017] For any two of the intermediate result buffers, divide the intermediate result buffers into multiple buckets with the same number based on a hash function;

[0018] Iteratively use the merge join algorithm to sort and join the buckets in one of the intermediate result buffers with the corresponding buckets in the other intermediate result buffer until all the multiple buckets divided from the intermediate result buffers have been merged to obtain an ordered buffer.

[0019] According to a parallel processing method based on granularity blocks for SPARQL queries provided by the present invention, the scanning process includes scanning through known predicate scanning patterns and unknown predicate scanning patterns, and the unknown predicate scanning patterns include a single unknown variable scanning pattern and a double unknown variable scanning pattern.

[0020] The present invention also provides a parallel processing device based on granularity blocks for SPARQL queries, including the following modules: a parsing module, configured to obtain and parse a query request for querying RDF data, and generate a query graph based on the query request; a partitioning module, configured to, when the size of the intermediate result buffer is greater than a preset threshold when executing a query plan based on the query graph, partition the intermediate result buffer into multiple blocks of a preset size, where the intermediate result buffer is a buffer storing intermediate results;

[0021] A calling module, configured to, when the task queue of any thread is empty, call a task scheduler to schedule a to-be-executed task to the thread for execution until the task queues of all threads are empty;

[0022] Wherein, the to-be-executed task is scheduled by the task scheduler from the task queue of another thread, and the task queue of the other thread is not empty, and the to-be-executed task includes performing a scanning process and a joining process on the block.

[0023] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, it implements the parallel processing method based on granularity blocks for SPARQL queries as described in any one of the above.

[0024] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the parallel processing method based on granularity blocks for SPARQL queries as described in any one of the above.

[0025] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the parallel processing method based on granularity blocks for SPARQL queries as described in any one of the above.

[0026] The parallel processing method based on granularity blocks for SPARQL queries provided by the present invention obtains and parses a query request for querying RDF data, generates a query graph based on the query request. When executing a query plan based on the query graph, if the size of the intermediate result buffer is greater than a preset threshold, the intermediate result buffer is divided into multiple blocks of a preset size. When the task queue of any thread is empty, the task scheduler is called to schedule the tasks to be executed to the thread for execution until the task queues of all threads are empty. By this method, the intermediate result buffer is divided into multiple blocks for parallel processing, and load balancing is achieved through a global task scheduling strategy, which can effectively improve the efficiency of SPARQL queries on RDF data. Brief Description of the Drawings

[0027] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0028] Figure 1 It is a flowchart of the parallel processing method based on granularity blocks for SPARQL queries provided by the present invention.

[0029] Figure 2 It is a schematic diagram of the system architecture of the parallel processing method based on granularity blocks for SPARQL queries provided by the present invention.

[0030] Figure 3 It is a schematic diagram of task scheduling for the task queue provided by the present invention.

[0031] Figure 4 It is a schematic diagram of merge-join processing for the intermediate result buffer provided by the present invention.

[0032] Figure 5 It is a schematic diagram of the structure of the parallel processing device based on granularity blocks for SPARQL queries provided by the present invention.

[0033] Figure 6 It is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Embodiments

[0034] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0035] Figure 1 FIG. 4 is a schematic flowchart of a parallel processing method based on granularity blocks for SPARQL queries provided by the present invention. As Figure 1 shown, the method includes the following steps 101 to 103.

[0036] Step 101: Obtain and parse a query request for querying RDF data, and generate a query graph based on the query request.

[0037] In some embodiments, the large-scale RDF data system used in the present invention is TriParaPipe, which can perform parallel processing of SPARQL queries based on granularity blocks for RDF data sets containing billions of triples.

[0038] Refer to Figure 2 , Figure 2 FIG. 5 is a schematic system architecture diagram of a parallel processing method based on granularity blocks for SPARQL queries provided by the present invention. As Figure 2 shown, the system includes a storage layer, a parallel processing layer, and a query analysis layer. Among them, the storage layer saves relevant information after the RDF data is imported into the system, including a bit matrix, a statistical index, a dictionary, and an entity predicate index. The dictionary is used to maintain the mapping structure between URIs and element IDs; the bit matrix is a data structure for storing RDF data after triples are imported; the statistical index is used to calculate the selectivity of triple patterns and join variables. The parallel processing layer is used to accelerate the query speed through coarse-grained and fine-grained processing methods, including a parallel processing module based on granularity blocks and an operation engine. The parallel processing module is responsible for accessing and processing data so that after the intermediate results are divided into numerous blocks, a higher degree of parallelism can be obtained during the query execution process; the dynamic load balancing module is used to perform load balancing among the task queues of each thread. The query analysis layer is responsible for parsing the query request, constructing the query graph, and generating a query plan. The query graph maintains the relationship between variables and triple patterns after parsing the query request, and then the plan generator generates a query graph for the query request and generates a query plan according to the query graph, while the operation engine executes the query task according to the query plan.

[0039] Step 102: When the size of the intermediate result buffer is greater than a preset threshold in response to executing the query plan based on the query graph, divide the intermediate result buffer into multiple blocks of a preset size.

[0040] In some embodiments, the intermediate result buffer is a buffer storing intermediate results.

[0041] In some embodiments, when TriParaPipe executes a query plan based on a query graph, the intermediate results generated are all stored in a buffer as the input for the next operation. The buffer storing the intermediate results is the intermediate result buffer. When querying RDF data, intermediate results refer to the data temporarily generated or partially processed during the query execution process. These intermediate results play an important role in tracking the query execution process, optimizing query performance, and understanding the final result. For example, when executing a complex SPARQL query involving multiple JOIN operations, each JOIN step may generate intermediate results, which are then used in the next JOIN step. The database management system may select different physical execution plans to optimize these JOIN operations, so as to generate intermediate results and final results as efficiently as possible.

[0042] In some embodiments, for the purpose of achieving parallel processing, the intermediate result buffer storing the intermediate results is divided into many smaller data blocks. First, an initial block size needs to be set. In the operating system, by default, the page size is 4KB, and in the system used in the present invention, each ID occupies 4 bytes. Therefore, a block is set to a size containing 1024 IDs. Next, the preset threshold for buffer block division needs to be determined. Generally, if the size of the intermediate result buffer is less than the buffer size itself, then the buffer obviously does not need to be divided. Through multiple experiments in the present invention, it is verified that when the size of an intermediate result buffer exceeds 4 times the block size, for the purpose of performing parallel processing, the intermediate result buffer needs to be divided into smaller blocks, for example, divided into 4 smaller blocks, and then the query processing can be carried out separately and independently on the pairs of these blocks, so as to achieve parallel processing and improve processing efficiency.

[0043] Step 103: When the task queue of any thread is empty, call the task scheduler to schedule the to-be-executed tasks to the thread for execution until the task queues of all threads are empty.

[0044] Wherein, the to-be-executed tasks are scheduled by the task scheduler from the task queues of other threads, and the task queues of the other threads are not empty. The to-be-executed tasks include scanning and joining processes on the blocks.

[0045] In some embodiments, the task queue includes a starting endpoint and a preset number of ending endpoints. The thread starts executing the task to be executed from the starting endpoint of the task queue. The preset number is the difference between the total number of threads and 1, and each ending endpoint corresponds to one of all the other threads.

[0046] In some embodiments, step 103 can be implemented in the following manner: when the task queue of any one thread is empty, the task scheduler is called to schedule the task to be executed at the specified ending endpoint of the task queues of other threads to the starting endpoint of the thread, so that the thread executes the task to be executed until the task queues of all threads are empty; wherein, the specified ending endpoint is the ending endpoint corresponding to the thread in the task queues of the other threads.

[0047] Specifically, since there are a large number of triples matching the triple pattern, the intermediate query results of RDF data may be very large, resulting in a very large number of blocks. It becomes particularly important to ensure that each thread processes an equal amount of work, so that the time required for each thread to process the blocks allocated to it is almost the same. Load balancing can be described by multiple task queues. A task to be executed is a pairing of two blocks, and each task to be executed is executed in parallel computing. The ratio of the task queue to the thread can be adjusted. Each thread maintains its own task queue, obtains tasks from the task queue, and processes these tasks. The task execution model determines the order in which tasks are pushed into or popped from the task queue. The task queue provides a convenient abstraction to express parallel computing as a set of task sets.

[0048] The present invention introduces a multi-ended queue (meque) to replace the traditional double-ended queue (deque) to execute tasks. Each thread has a multi-ended queue (i.e., task queue) of which it is the owner. Each task queue includes a starting endpoint and a preset number of ending endpoints. The tasks to be executed in the task queue can be assigned to other threads for processing by a work stealing scheduler (i.e., task scheduler). Each thread executes the tasks to be executed by taking them out one by one from the front end (i.e., starting endpoint) of the task queue corresponding to the thread. However, different from the double-ended queue that only allows work stealing from the back end, the multi-ended queue used in the present invention allows the "thief" (i.e., the thread with an empty task queue) to perform work stealing at multiple ending endpoints of the sequence container in addition to the front end. It should be further noted that all the stealing operations in the present invention are realized by the task scheduler for task scheduling.

[0049] Specifically, referring to Figure 3 , Figure 3 is a schematic diagram of task scheduling for the task queue provided by the present invention. In Figure 3In it, a stealing endpoint startpos is assigned to the task queue of each thread. Suppose there are n threads: T 1 ,... T n . Each meque has a starting position for work stealing, denoted as startpos (i.e., the above-mentioned endpoint, and the number of endpoints is the difference between the total number of threads and 1, and each endpoint corresponds one-to-one with each other thread). The thief can steal tasks from the endpoints after startpos to avoid conflicts with the owner thread of the task queue (the owner thread of the task queue starts executing the tasks to be executed from the starting endpoint front). Starting from startpos, each other thread will be assigned a stealing endpoint for future stealing of the task queue. For example, for the task queue in Figure 3 , the stealing endpoint assigned to thread 1 is T′ 1 end, the stealing endpoint assigned to thread j is T′ j end, the stealing endpoint assigned to thread n is T′ n end, and the stealing endpoint assigned to thread m is T′ m end. The endpoints assigned to consecutive threads are adjacent. For example, the endpoints assigned to thread j and thread j + 1 are adjacent.

[0050] In some cases, if a thread needs to steal tasks from the same meque again, a new endpoint is re-assigned to the thread. Suppose when thread j steals from thread i for the first time, the endpoint of the task queue corresponding to thread i is pre_end. Then when thread j selects thread i as the stealing target again and there are still tasks to be stolen in the meque of thread i, a new endpoint will be assigned to thread j in the meque of thread i: (pre_end + n - 1) % mq:length, where % is the modulo operator and mq:length is the queue length of the meque. The above task stealing process will continue throughout the task execution until all the tasks to be executed in the meque of each thread are checked and executed. When a thread finds that its meque is empty, it will globally steal tasks from the specified endpoints corresponding to this thread in the meques of other threads.

[0051] The multi - ended queue used in the present invention assigns a non - decreasing end point to each thread, and each thief obtains the stolen tasks from the end point of the meque assigned to it through a work - stealing scheduler. By using a multi - ended queue instead of a double - ended queue, conflicts among thieves are eliminated when there are sufficient tasks. When there are insufficient tasks in the meque, thieves are allowed to steal the same tasks. In this case, this method can still avoid some conflicts because not all threads choose the same end point. Therefore, through the task - stealing method based on a multi - ended queue provided by this application, it can be ensured that each thread is executing tasks in parallel, thus effectively improving the task - processing efficiency.

[0052] In some embodiments, the tasks to be executed include scanning and joining the block, where the joining process includes: based on the quick - sort algorithm, sorting the multiple blocks divided from the intermediate result buffer to obtain sorted blocks; iteratively using the merge - join algorithm to merge adjacent sorted blocks into an ordered buffer until all the multiple blocks divided from the intermediate result buffer have been merged.

[0053] In some embodiments, the scanning process includes scanning through a known - predicate scanning pattern and an unknown - predicate scanning pattern, and the unknown - predicate scanning pattern includes a single - unknown - variable scanning pattern and a double - unknown - variable scanning pattern.

[0054] Specifically, in the system provided by the present invention, the query plan is executed by the query execution engine according to the query graph. Although the query graph may be very complex, the basic query operators are scan and join. The scan is used to directly obtain data from the data storage layer, while the join is used to obtain records from two table buffers with the same key. According to whether the records are sorted by the key, the join can be divided into a merge join and a hash join.

[0055] In the system provided by the present invention, the data is partitioned according to the predicate, so the query patterns can be divided into two types: the predicate - known pattern and the predicate - unknown pattern. The scan operation scans data from the triple matrix according to the above two patterns. Therefore, the scan can be divided into predicate - known scanning (PKS) and predicate - unknown scanning (PUS). All scan operations are performed by using a lower - bound - upper - bound pair in specific components (S, P, O) of the pattern.

[0056] Known predicate scanning is a very common operation in SPARQL queries. First, a hash function is used to locate the predicate buffer by the ID of the predicate. Second, the ID-Chunks index is used to locate the chunks where the ID might be. With the ID-Chunks index, the upper and lower bounds of the chunk number can be obtained in O(1) time complexity by hashing. Then, since the IDs within a chunk are sorted by entity ID, binary search can be used to locate the ID in the specified chunk. In addition, the triple matrix adopts a lightweight compression scheme, so individual triples can be decompressed independently. Finally, these chunks are scanned and the data in the buffer is decompressed until the ID is greater than the upper bound or the end of the predicate buffer is reached, and the ID is placed in the result buffer. Since the compression scheme used in the triple matrix is lightweight, the decompression speed is very fast.

[0057] Performing unknown predicate scanning must know the ID of the subject or object in the pattern. To accelerate the scanning, an ID-predicate index is used, which is stored in bitmap format and the storage space is reduced by an adaptive bit-based compression algorithm. By quickly decompressing the bit matrix structure of the ID-predicate index, all scanning operations can be completed in almost linear time complexity. If there is another unknown component, such as the subject or object, the PKS program is used to find the answer. PUS can be further divided into two types: single variable-unknown scanning (SVUS, Single Variable-Unknown Scanning) and double variable-unknown scanning (DVUS, Double Variable-Unknown Scanning).

[0058] In some embodiments, the system provided by the present invention can also perform a reduction operation, that is, use a semi-join to filter out the mismatched results. Obviously, the semi-join reducer should only be adopted when the semi-join is inexpensive and can significantly reduce the size of the intermediate query results. The semi-join can be applied at all levels of the query plan, so the semi-join technology is adopted in the system.

[0059] Specifically, assume the triple pattern is P i and P j , and the predicate common to the two patterns is denoted as p. The semi-join between P i and P j can be expressed as or For , after performing the semi-join, only the intermediate results of P j that match P i will be retained. Similarly, will eliminate P i that do not match P jIntermediate results. The reduction operation can not only trim the results, reduce the communication cost and memory consumption during query evaluation, but also remove useless patterns in subsequent join operations.

[0060] In some embodiments, the tasks to be executed include scanning and joining the blocks. The joining process includes: sorting the multiple blocks divided from the intermediate result buffer based on the quicksort algorithm to obtain sorted blocks; iteratively using the merge join algorithm to merge adjacent sorted blocks into an ordered buffer until all the multiple blocks divided from the intermediate result buffer have been merged.

[0061] Specifically, the merge join is used to obtain results from the buffers sorted by the join key. Its advantage is low time complexity. If the records in the buffer are r m and r n , then the time complexity is O(r m +r n ). Although the merge join is very effective, if one of the join buffers is not sorted (if neither is sorted, the hash join will be used), the unsorted buffer needs to be sorted before using the merge join algorithm.

[0062] In this system, the buffer to be sorted is first divided into multiple smaller blocks, and the size of each block is equal to the size of the memory page, that is, 4KB. This is done to reduce the TLB (Translation Lookaside Buffer) miss rate. Secondly, the quicksort algorithm is used to sort these blocks, and the worker threads are used to sort each block separately. Then, by iteratively using the merge join method, the adjacent sorted blocks are merged into a larger ordered buffer until the entire buffer is sorted.

[0063] In the system provided by the present invention, at the beginning of query execution, since the system stores data in an ordered manner, the buffer of the query pattern has been sorted. Therefore, the merge join can be directly used without pre-sorting the data. However, after the join operation, the data is no longer in an ordered state, and at this time, the buffer must be sorted again. Thus, it is necessary to execute the block sorting stage and the block merging stage in parallel. When the engine scans the data, logically, the small buffer is sliced into multiple blocks, and the boundary IDs of each block are collected. Then, these boundary IDs are used to filter the second buffer in parallel, so that the engine can first remove the unmatched triples.

[0064] In some embodiments, the connection processing includes: for any two of the intermediate result buffers, dividing the intermediate result buffers into a plurality of buckets with the same number based on a hash function; iteratively using a merge join algorithm to sort and join the buckets in one of the intermediate result buffers with the corresponding buckets in the other intermediate result buffer until all the buckets divided from the intermediate result buffers have been merged to obtain an ordered buffer.

[0065] Specifically, in general, the execution of the hash join algorithm is divided into two phases: a build phase and a probe phase. In the build phase, the smaller of the two input buffers (denoted as R) is scanned to fill a hash table. Then, in the probe phase, the second input buffer (denoted as S) is scanned, and for each set of data therein, a probe is performed in the hash table to find the tuples that match those in R.

[0066] In the embodiments provided by the present invention, referring to Figure 4 , Figure 4 is a schematic diagram of the merge join processing for the intermediate result buffer provided by the present invention. As shown in Figure 4 , this solution includes a build phase and a sort-merge join phase, rather than the traditional probe phase. Figure 4 In 1 , ① and ③ represent the build phase, and ② represents the sort-merge join phase. First, in the build phase, the buffer R and the buffer S are divided into several buckets b k ,....b 11 using the hash function h, so that the records with the same join key in the two buffers will surely be hashed into the same corresponding bucket. In specific implementation, first calculate the size of each bucket, and then allocate memory according to this bucket size to reduce memory consumption. If the bucket is still too large (for example, setting 100 bytes as a threshold size), it will be further divided into smaller buckets b 1j ,...b ; then in the merge-sort join phase, since the records with the same join key have been divided into the same bucket, the sort-merge join can be performed respectively between the corresponding buckets. Here, worker threads can be used to execute the join operation between each pair of buckets. After the task is submitted to the task queue, the thread fetches the task for execution. Inside the thread, first sort the records in the bucket, and then use the merge join to generate a merge result to obtain an ordered buffer.

[0067] The parallel processing method based on granularity blocks for SPARQL queries provided by the present invention obtains and parses a query request for querying RDF data, generates a query graph based on the query request. When executing a query plan based on the query graph, if the size of the intermediate result buffer is greater than a preset threshold, the intermediate result buffer is divided into multiple blocks of a preset size. When the task queue of any one thread is empty, the task scheduler is called to schedule the tasks to be executed to the thread for execution until the task queues of all threads are empty. By this method, the intermediate result buffer is divided into multiple blocks for parallel processing, and load balancing is achieved through a global task scheduling strategy, which can effectively improve the efficiency of SPARQL queries on RDF data.

[0068] The parallel processing device based on granularity blocks for SPARQL queries provided by the present invention will be described below. The parallel processing device based on granularity blocks for SPARQL queries described below can be correspondingly referred to the parallel processing method based on granularity blocks for SPARQL queries described above.

[0069] As Figure 5 shown, Figure 5 is a schematic structural diagram of the parallel processing device based on granularity blocks for SPARQL queries provided by the present invention. Figure 5 The parallel processing device 500 based on granularity blocks for SPARQL queries shown in it includes a parsing module 501, a dividing module 502, and a calling module 503. Among them, the parsing module 501 is used to obtain and parse a query request for querying RDF data, and generate a query graph based on the query request; the dividing module 502 is used to divide the intermediate result buffer into multiple blocks of a preset size in response to that when executing a query plan based on the query graph, the size of the intermediate result buffer is greater than a preset threshold, where the intermediate result buffer is a buffer storing intermediate results; the calling module 503 is used to call the task scheduler to schedule the tasks to be executed to the thread for execution when the task queue of any one thread is empty until the task queues of all threads are empty; where the tasks to be executed are scheduled by the task scheduler from the task queues of other threads, and the task queues of the other threads are not empty, and the tasks to be executed include scanning and joining processing of the blocks.

[0070] Figure 6 Illustrates a schematic structural diagram of an electronic device entity, as Figure 6As shown in the figure, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640. Among them, the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logical instructions in the memory 630 to execute a parallel processing method based on granularity blocks for SPARQL queries. The method includes: obtaining and parsing a query request for querying RDF data, and generating a query graph based on the query request; when the size of the intermediate result buffer is greater than a preset threshold when executing a query plan based on the query graph, dividing the intermediate result buffer into multiple blocks of a preset size, where the intermediate result buffer is a buffer storing intermediate results; when the task queue of any thread is empty, calling a task scheduler to schedule a task to be executed to the thread for execution until the task queues of all threads are empty; where the task to be executed is scheduled by the task scheduler from the task queue of other threads, and the task queue of the other thread is not empty, and the task to be executed includes scanning and joining the blocks.

[0071] In addition, when the logical instructions in the above-mentioned memory 630 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0072] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the parallel processing method based on granularity blocks for SPARQL queries provided by the above-mentioned various methods. The method includes: obtaining and parsing a query request for querying RDF data, and generating a query graph based on the query request; when the size of the intermediate result buffer is greater than a preset threshold in response to executing a query plan based on the query graph, dividing the intermediate result buffer into multiple blocks of a preset size, where the intermediate result buffer is a buffer storing intermediate results; when the task queue of any one thread is empty, calling a task scheduler to schedule a task to be executed to the thread for execution until the task queues of all threads are empty; where the task to be executed is scheduled by the task scheduler from the task queue of other threads, and the task queue of the other threads is not empty, and the task to be executed includes scanning and joining the blocks.

[0073] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the parallel processing method based on granularity blocks for SPARQL queries provided by the above-mentioned various methods. The method includes: obtaining and parsing a query request for querying RDF data, and generating a query graph based on the query request; when the size of the intermediate result buffer is greater than a preset threshold in response to executing a query plan based on the query graph, dividing the intermediate result buffer into multiple blocks of a preset size, where the intermediate result buffer is a buffer storing intermediate results; when the task queue of any one thread is empty, calling a task scheduler to schedule a task to be executed to the thread for execution until the task queues of all threads are empty; where the task to be executed is scheduled by the task scheduler from the task queue of other threads, and the task queue of the other threads is not empty, and the task to be executed includes scanning and joining the blocks.

[0074] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0075] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A granular block-based parallel processing method for SPARQL queries, characterized in that: include: Obtaining and parsing a query request for querying RDF data, and generating a query graph based on the query request; In response to a size of an intermediate result buffer being greater than a preset threshold when executing a query plan based on the query graph, dividing the intermediate result buffer into a plurality of blocks of preset sizes, wherein the intermediate result buffer is a buffer storing intermediate results; When the task queue of any thread is empty, the task scheduler is called to schedule the task to be executed to the thread for execution until the task queues of all threads are empty; The to-be-executed task is scheduled by the task scheduler from the task queue of other threads, and the task queue of the other threads is not empty, and the to-be-executed task includes scanning and connecting the blocks; The task queue includes a starting endpoint and a preset number of end points, the thread starts to execute the task to be executed from the starting endpoint of the task queue, the preset number is the difference between the number of all threads and 1, and the end points correspond to all other threads one by one; When the task queue of any thread is empty, calling the task scheduler to schedule the to-be-executed task to the thread for execution until the task queues of all threads are empty, including: When the task queue of any thread is empty, the task scheduler is called to schedule the pending tasks at the designated end points of the task queues of other threads to the starting end point of the thread, so that the thread executes the pending tasks until the task queues of all threads are empty; Wherein, the designated end point is the end point corresponding to the thread in the task queue of the other thread; Wherein, the task queue is a multi-terminal queue; When thread j steals thread i for the first time, the end point of the multi-end queue corresponding to thread i is pre_end. When thread j selects thread i as the stealing target again and there are still stealable tasks in the multi-end queue of thread i, a new end point is allocated to thread j through the multi-end queue of thread i. The new end point is expressed as (pre_end+n-1)%mq:length, where pre_end represents the end point of the multi-end queue corresponding to thread i, n represents the total number of threads, % represents the modulo operator, and mq:length represents the queue length of the multi-end queue corresponding to thread i. Wherein, the stealing is realized by performing task scheduling through the task scheduler.

2. The granular block-based parallel processing method for SPARQL queries according to claim 1, characterized in that: The connection process includes: Based on a quick sorting algorithm, sorting is performed on the multiple blocks divided into the intermediate result buffer to obtain sorted blocks; The merge join algorithm is used iteratively to merge adjacent sorted blocks into an ordered buffer until the multiple blocks divided into the intermediate result buffer are merged.

3. The parallel processing method based on granular blocks for SPARQL queries according to claim 1, characterized in that: The connection process includes: For any two of the intermediate result buffers, dividing the intermediate result buffers into multiple buckets of the same number based on a hash function; Iteratively use a merge join algorithm to sort and connect the buckets in one of the intermediate result buffers with the corresponding buckets in another intermediate result buffer until the multiple buckets divided by the intermediate result buffer are merged to obtain an ordered buffer.

4. The granular block-based parallel processing method for SPARQL queries according to claim 1, characterized in that: The scanning process includes scanning through a known predicate scanning mode and an unknown predicate scanning mode, and the unknown predicate scanning mode includes a single unknown variable scanning mode and a double unknown variable scanning mode.

5. A parallel processing device based on granular blocks for SPARQL queries, characterized in that: include: A parsing module, used to obtain and parse a query request for querying RDF data, and generate a query graph based on the query request; a partitioning module, configured to, in response to a size of an intermediate result buffer being greater than a preset threshold when executing a query plan based on the query graph, partition the intermediate result buffer into a plurality of blocks of preset sizes, wherein the intermediate result buffer is a buffer storing intermediate results; A calling module, used for calling a task scheduler to schedule the to-be-executed task to the thread for execution when the task queue of any thread is empty, until the task queues of all threads are empty; The to-be-executed task is scheduled by the task scheduler from the task queue of other threads, and the task queue of the other threads is not empty, and the to-be-executed task includes scanning and connecting the blocks; The task queue includes a starting endpoint and a preset number of end points, the thread starts to execute the task to be executed from the starting endpoint of the task queue, the preset number is the difference between the number of all threads and 1, and the end points correspond to all other threads one by one; When the task queue of any thread is empty, calling the task scheduler to schedule the to-be-executed task to the thread for execution until the task queues of all threads are empty, including: When the task queue of any thread is empty, the task scheduler is called to schedule the pending tasks at the designated end points of the task queues of other threads to the starting end point of the thread, so that the thread executes the pending tasks until the task queues of all threads are empty; Wherein, the designated end point is the end point corresponding to the thread in the task queue of the other thread; Wherein, the task queue is a multi-terminal queue; When thread j steals thread i for the first time, the end point of the multi-end queue corresponding to thread i is pre_end. When thread j selects thread i as the stealing target again and there are still stealable tasks in the multi-end queue of thread i, a new end point is allocated to thread j through the multi-end queue of thread i. The new end point is expressed as (pre_end+n-1)%mq:length, where pre_end represents the end point of the multi-end queue corresponding to thread i, n represents the total number of threads, % represents the modulo operator, and mq:length represents the queue length of the multi-end queue corresponding to thread i. Wherein, the stealing is realized by performing task scheduling through the task scheduler.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the parallel processing method based on granular blocks for SPARQL queries according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the parallel processing method based on granular blocks for SPARQL queries according to any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the parallel processing method based on granular blocks for SPARQL queries according to any one of claims 1 to 4 is implemented.

Images