Academic resource acquisition method based on LDA (latent Dirichlet allocation)

Abstract

The invention provides an academic resource acquisition method based on LDA (latent Dirichlet allocation). According to the academic resource acquisition method, a topical crawler is used; an LDA topical model is also used; firstly, a training corpus is provided for the LDA topical model to train to obtain a topical document; the topical crawler further comprises a topical determination module, a similarity calculation module and a URL (Uniform Resource Locator) priority ranking module on the basis of a general network crawler; in the crawling process of the topical crawler, the topical document is adopted to guide calculation of topical similarity, a URL of which the topical similarity is greater than a set threshold is selected, the topical crawler maintains an URL queue of webpages which are not accessed, and the topical crawler sequentially and continuously accesses a webpage of each URL according to a ranking sequence of the URL queue, crawls corresponding academic resources, and continuously stores the crawled academic resources into a database after carrying out classification labeling until the URL in the queue of the webpages which are not accessed is empty; an API (Application Program Interface) of which an academic resource database is open is provided for display and calling; machine learning is fused into the academic resource acquisition method, and academic resource acquisition quality and efficiency are improved.

Description

technical field [0001] The invention relates to machine learning, information retrieval and web page data mining, in particular to an LDA-based academic resource acquisition method. Background technique [0002] With the electronicization of academic resources, it has gradually become a research hotspot to discover and excavate academic resources in the field of interest of researchers from massive academic resources. In order to adapt to the characteristics of massive, multi-source and heterogeneous digital academic resources, some new methods and models based on machine learning and data mining are constantly being used to find topics different from traditional keyword frequency-based topic discovery methods such as co-word analysis and citation analysis. Applied to the field of academic resource classification, such as the latent Dirichlet allocation model (latent Dirichlet allocation, LDA), social network analysis (SNA), etc., practice has found that this method has achi...

AI Technical Summary

Problems solved by technology

With the rapid development of the Internet, the number of web pages has increased rapidly, but due to the limited computing resources, network tool resources and storage resources, traditional search technologies have been difficult to cover the different needs of mass users

Images