Capital market public opinion monitoring method based on distributed web crawler and NLP

Abstract

The invention discloses a capital market public opinion monitoring method based on a distributed web crawler and an NLP. The capital market public opinion monitoring method comprises a distributed crawler module based on a cloud server and a financial text NLP analysis system. Based on a distributed crawler of a cloud server, multi-process and periodically updated public information capture is realized, and the system scale can be quickly adjusted according to use requirements; according to the method, a financial segmented word bank and a positive and negative emotion word bank are constructed, the cost of manual annotation is reduced in the corpus construction process through mixed sample inspection, fuzzy clustering and other algorithms, the positive and negative emotion tendency probability of a text is calculated based on a supervised learning algorithm, emotion indexes are synthesized by adopting an auto-encoder algorithm, and the emotion judgment accuracy is improved; accordingto the distributed architecture, system crash caused by a single node fault is avoided, the validity of the emotion index is improved through the reconstructed financial emotion word bank and the textanalysis algorithm, the market emotion is dynamically reflected, and real-time capital market data is provided for a user.

Description

technical field [0001] The invention belongs to the technical field of the Internet, and in particular relates to a method for monitoring public opinion in a capital market based on distributed web crawlers and NLP. Background technique [0002] Public opinion analysis has the characteristics of a large amount of data and fast update speed. The real-time capture system faces difficulties such as high scalability requirements, high long-term operating costs, and high system stability requirements. In the existing technology, although the system architecture for stock market sentiment and stock public opinion analysis exists, the relevant architecture is based on a complex server cluster system, which has high operating costs and long deployment time, and does not have the basis for large-scale popularization. [0003] On the other hand, the actual performance of algorithms for public opinion text analysis has always been a problem. First of all, there is a lack of corpora fo...

AI Technical Summary

Problems solved by technology

[0004] The technical problem solved by the present invention: the information grabbing system of a single node faces the problem of poor system stability, while the traditional distributed architecture, such as Docker, has relatively complicated configuration and high cost; the existing lexicon is designed based on the scenes of daily life There is a relative lack of proprietary vocabulary and financial vocabulary, and the cost of manual labeling is high; traditional text sentiment analysis algorithms have the problem of judgment bias, and synthetic indicators need to be produced to improve the accuracy of sentiment judgments

Images