Recent message sending priority processing method and system based on crawler texts

A technology of priority processing and text, applied in the field of text processing, can solve problems such as insufficient redis memory, difficulty in ensuring complete recovery of data, and affecting system stability, etc., to reduce the number of consumers, improve maintainability, and increase system operation and maintenance effect of difficulty

CN110727845APending Publication Date: 2020-01-24DATAGRAND TECH INC
3 Cites 1 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2020-01-24

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
Patent Text Reader

Abstract

The invention discloses a recent message sending priority processing method and system based on a crawler text. The recent message sending priority processing method comprises the steps: a producer process stores all crawled data files in time folders with the message sending time of the producer process as a file name, and generates a mark file correlated with the data files in each time folder;and the consumer process extracts a time folder with the latest current time, regularizes the corresponding data file according to the mark file under the time folder, and moves the regularized data file to a historical folder. The recent message sending priority processing adopts a double-file control method, so that a producer and a consumer can be prevented from operating one file at the same time without using a system lock, and the accuracy of data is ensured, and the logic complexity of the system is reduced, and the maintainability of the system is improved.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the technical field of text processing, in particular to a crawler text-based method and system for prioritizing processing of recently published text. Background technique

[0002] Companies that display news streams often grab news data from the "World Wide Web" and send it locally. The captured articles cannot be directly pushed to production. It is necessary to remove external links from articles, download pictures in articles, and tag and classify articles. After normalization processing, articles after normalization processing can be published. Usually some articles are extremely time-sensitive, so they cannot be processed according to the order in which the articles are crawled, but the most recently published articles need to be processed first to ensure that time-sensitive articles are processed first.

[0003] Due to the large data volume of individual articles, for example, an article containing multiple base64 pict...

Examples

Embodiment Construction

[0044] In order to better understand the technical solutions of the present invention, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0045] It should be clear that the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0046] The present application will be described in further detail below through specific embodiments and in conjunction with the accompanying drawings.

[0047] Embodiments of the present invention provide a method and system for prioritizing processing of recently published documents based on crawler texts.

[0048]The recent document priority processing system based on the crawler text of this application includes a web service int...