Parallel Reasoning Algorithm for Streaming RDF Data Based on Spark Streaming

Abstract

The invention relates to a parallel reasoning algorithm for streaming RDF data based on Spark Streaming. First, combine the OWL Horst inference rules to construct the corresponding rule connection variable relationship table; in the iterative parallel inference stage, the batch new data in the Streaming data stream and the data generated by the previous inference are regularly obtained as input data, and the input pattern data and instance data are processed. Carry out classification processing and store it in the corresponding Redis cluster; then, connect the variable relationship table according to the rules, judge the rules that can be activated in this reasoning, and combine the corresponding instance data to generate reasoning data; finally, delete the duplicate data generated in this reasoning and storage, and the iterative reasoning ends. The present invention reduces the number of tasks of MapReduce, combines Spark to perform iterative reasoning of flow data; design rules to connect variable relational tables to store data and new data generated during reasoning, ensuring the completeness of the algorithm; design example triples The storage solution, combined with the characteristics of Redis, trades space for time to achieve fast reading of instance data.

Description

technical field [0001] The invention belongs to the technical field of massive streaming RDF data reasoning, and in particular relates to a parallel reasoning algorithm for streaming RDF data based on Spark Streaming. Background technique [0002] Most of the existing reasoning methods based on OWL rules are centralized processing of fixed-size static data sets. Due to the limitation of centralized processing mechanism, the existing algorithms are inefficient when processing massive real-time data. In order to respond to this growing demand, many scholars have studied and proposed their own RDF stream reasoning architecture: Barbieri DF[1] et al. proposed an incremental reasoning algorithm based on stream and rich background knowledge. Add expiry time information to RDF triples, when new streaming data arrives, perform reasoning calculations on new data, and terminate explicit facts and delete invalid triples. The IDRM[2] algorithm can efficiently and scalablely perform RDF...

AI Technical Summary

Problems solved by technology

The IDRM algorithm is specially modeled for RDFS rules, so the efficiency of OWL Horst rule reasoning is not high

Slider is only designed for RDFS rules, so it is not suitable for complex OWL Horst rule reasoning

The PRAS algorithm performs reasoning on streaming data by designing a pseudo-bidirectional network, but due to the high consumption of pseudo-bidirectional network communication, the efficiency of processing a large amount of streaming data is not high

Images