Recent message sending priority processing method and system based on crawler texts

Abstract

The invention discloses a recent message sending priority processing method and system based on a crawler text. The recent message sending priority processing method comprises the steps: a producer process stores all crawled data files in time folders with the message sending time of the producer process as a file name, and generates a mark file correlated with the data files in each time folder;and the consumer process extracts a time folder with the latest current time, regularizes the corresponding data file according to the mark file under the time folder, and moves the regularized data file to a historical folder. The recent message sending priority processing adopts a double-file control method, so that a producer and a consumer can be prevented from operating one file at the same time without using a system lock, and the accuracy of data is ensured, and the logic complexity of the system is reduced, and the maintainability of the system is improved.

Description

technical field [0001] The invention relates to the technical field of text processing, in particular to a crawler text-based method and system for prioritizing processing of recently published text. Background technique [0002] Companies that display news streams often grab news data from the "World Wide Web" and send it locally. The captured articles cannot be directly pushed to production. It is necessary to remove external links from articles, download pictures in articles, and tag and classify articles. After normalization processing, articles after normalization processing can be published. Usually some articles are extremely time-sensitive, so they cannot be processed according to the order in which the articles are crawled, but the most recently published articles need to be processed first to ensure that time-sensitive articles are processed first. [0003] Due to the large data volume of individual articles, for example, an article containing multiple base64 pict...

AI Technical Summary

Problems solved by technology

However, Redis is an in-memory database after all. If consumers have problems such as image download exceptions and blockages, it is very easy to cause redis memory to be insufficient, resulting in queue crashes and affecting system stability. It is an extremely troublesome thing, and it is difficult to ensure complete recovery of data

[0005] Mysql database method: Another common method is to store the posting time of files in the mysql database, and filter out the article data of the latest posting time through the posting time. The disadvantage of this method is that it needs to use database transactions, and the system will appear very cumbersome. Moreover, there may be multiple processes operating the same record in the data table, triggering the mysq lock table problem, causing all consumers to be stuck

Images