Public opinion monitoring method in capital market based on distributed web crawler and nlp

Abstract

The invention discloses a capital market public opinion monitoring method based on distributed network crawler and NLP, including a cloud server-based distributed crawler module and a financial text NLP analysis system; the cloud server-based distributed crawler realizes multi-process and periodic update Capture public information, and quickly adjust the scale of the system according to the needs of use; build a financial thesaurus, positive and negative emotional thesaurus, and reduce the cost of manual annotation in the process of corpus construction through mixed sample testing, fuzzy clustering and other algorithms, based on The supervised learning algorithm calculates the probability of positive and negative emotional tendencies of the text, and uses the self-encoder algorithm to synthesize the emotional index, which improves the accuracy of emotional judgment; the distributed architecture of the present invention avoids system collapse caused by a single node failure, and the reconstructed The financial sentiment lexicon and text analysis algorithm improve the validity of the sentiment index, dynamically reflect market sentiment, and provide users with real-time capital market data.

Description

technical field [0001] The invention belongs to the technical field of the Internet, and in particular relates to a method for monitoring public opinion in a capital market based on distributed web crawlers and NLP. Background technique [0002] Public opinion analysis has the characteristics of a large amount of data and fast update speed. The real-time capture system faces difficulties such as high scalability requirements, high long-term operating costs, and high system stability requirements. In the existing technology, although the system architecture for stock market sentiment and stock public opinion analysis exists, the relevant architecture is based on a complex server cluster system, which has high operating costs and long deployment time, and does not have the basis for large-scale popularization. [0003] On the other hand, the actual performance of algorithms for public opinion text analysis has always been a problem. First of all, there is a lack of corpora fo...

AI Technical Summary

Problems solved by technology

[0004] The technical problem solved by the present invention: the information grabbing system of a single node faces the problem of poor system stability, while the traditional distributed architecture, such as Docker, has relatively complicated configuration and high cost; the existing lexicon is designed based on the scenes of daily life There is a relative lack of proprietary vocabulary and financial vocabulary, and the cost of manual labeling is high; traditional text sentiment analysis algorithms have the problem of judgment bias, and synthetic indicators need to be produced to improve the accuracy of sentiment judgments

Images