Crawler-based knowledge graph processing method and device, computer equipment and storage medium

Abstract

The invention relates to the field of data acquisition. Specifically, a knowledge graph is constructed by using a data crawling mode. The invention further discloses a crawler-based knowledge graph processing method and device, computer equipment and a storage medium. The method comprises the following steps of crawling data information in webpage information by utilizing a crawler system; performing data cleaning on the crawled data information; performing information completion on the crawled data information subjected to data cleaning; target information, and performing feature extraction on the target information, constructing a relationship classification model according to the extracted feature information, obtaining a series of relationship entity pair data from the data informationby using the relationship classification model, and storing the relationship entity pair as a basic knowledge carrier of the knowledge graph to complete the construction of the processing knowledge graph. By means of the mode, the information can be rapidly obtained, the working efficiency can be greatly improved, and knowledge provided by the knowledge graph is beneficial for better understanding information content and carrying out information public opinion analysis.

Description

technical field [0001] The present application relates to the field of network information acquisition, and in particular to a crawler-based knowledge map processing method, device, computer equipment and storage medium. Background technique [0002] In the technical field of network information acquisition, a web crawler (also known as a web spider, a web robot, and more often referred to as a web chaser in the FOAF community) is a method that automatically crawls the World Wide Web according to certain rules. information programs or scripts. Other less commonly used names include ant, autoindex, emulator, or worm. [0003] With the rapid development of the network, the World Wide Web has become the carrier of a large amount of information, how to effectively extract and use this information has become a huge challenge. Search engines, such as the traditional general search engines Baidu and Google, as a tool to assist people in retrieving information, have become the ent...

AI Technical Summary

Problems solved by technology

[0006] (3) With the abundance of data forms on the World Wide Web and the continuous development of network technology, a large number of different data such as pictures, databases, audio, and video multimedia appear. General search engines are often powerless to these data with dense information content and a certain structure, and cannot perform well. discover and acquire

[0007] (4) Most general search engines provide keyword-based retrieval, and it is difficult to support queries based on semantic information

[0008] As we all know, in the field of financial technology, information, public opinion, etc. need to be understood in a timely manner, and at the same time, it needs to be quickly analyzed and processed by category. However, the existing network information acquisition technology has the above-mentioned defects, which makes it unable to meet the needs of users. Difficult to meet the needs of technological development

Images