Real-time knowledge graph construction system

Abstract

The invention discloses a real-time knowledge graph construction system, and relates to the technical field of graph analysis platforms. The real-time knowledge graph construction system comprises anapplication program part, a knowledge storage module, a knowledge consumption module, a knowledge bus, a data source bus and a data source module, wherein the specific work of each part is as follows:A1, a data source module is responsible for data acquisition and is accessed to a KAFKA message queue; C1, a source data bus is responsible for transmitting basic data needing entity / relationship extraction; C1, a knowledge extraction module is responsible for carrying out entity extraction and relationship extraction by using basic data in SparkStreaming consumption KAFKA. The application program part, the knowledge storage module, the data source bus, the data source module and other modules are used in cooperation. Entity extraction and relationship extraction are carried out by using basic data in Spark Streaming consumption KAFKA, and failure data is recorded in an external memory by adopting HBase so that problems of loss or repetition of extracted data and single data source accesscan be solved at the same time.

Description

technical field [0001] The invention relates to the technical field of graph analysis platforms, in particular to a real-time knowledge graph construction system. Background technique [0002] Kafka is an open source stream processing platform written in Scala and Java. It is a high-throughput distributed publish-subscribe message system that can handle all consumer action stream data in the website. This action (web browsing, searches and other user actions) is a key factor for many social functions on the modern web, these data are usually solved by processing logs and log aggregation due to throughput requirements, for log data and offline analysis like Hadoop system, but requires real-time processing limitations. This is a feasible solution. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time messages through clusters. [0003] In the existing technology, there are few applications ...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the deficiencies in the prior art, the present invention provides a real-time knowledge map construction system, which solves the problems in the field of actual real-time map construction in the prior art There are very few applications, basically T+1 (building a graph relationship once a day), so it is impossible to extract accurate knowledge extraction results at one time, and it is difficult to well adapt to businesses that require high data accuracy. At the same time, knowledge extraction exists High latency, prone to loss or duplication of knowledge extraction data, fewer data sources to support and consider, and low scalability of the solution

Images