Data continuous grabbing method and system based on Actor model

Abstract

The invention provides a data continuous capture method and system based on an Actor model. The method comprises the steps: defining a persistent Actor singleton object, and transmitting a message fortriggering data capture to the persistent Actor singleton object after the persistent Actor singleton object is started; generating a data request message in combination with a message type for triggering data capture, and sending the data request message to itself; initiating an HTTP request to a data providing interface by using a play-ws library, and obtaining data in a specified time range inthe data request message; if the HTTP request is successful, analyzing the data in the response to obtain captured data all Data; judging whether the all Data is the latest data or not, if so, filtering out the new data filterd Data, and storing the filterd Data in a relational database in a primary key unique mode, otherwise, storing the filterd Data in the relational database in a secondary keyunique mode; if not, storing the all Data in a relational database in a primary key unique mode; and when a data capture failure message is received, generating a new data request message and tryingto capture the data again by using the timer. According to the method, high-concurrency capture of the data is realized based on the Actor model, and data loss and data repetition are avoided.

Description

technical field [0001] The invention relates to the field of data capture, in particular to a continuous data capture method and system based on an Actor model. Background technique [0002] For applications that use third-party data for data analysis, it is usually necessary to solve the problem of data capture. For example, you want to continuously capture the latest recruitment information from certain recruitment websites, or continuously obtain the latest log records from the log interface provided by a video-on-demand or live broadcast service provider. Data capture itself is not difficult. It is nothing more than using legal HTTP requests to continuously obtain data, and then parse and store it. The technical difficulties of data capture are mainly manifested in three aspects, one is how to increase the speed of data capture, the other is how to solve the data loss caused by request failure, and the third is how to avoid repeated storage of data. [0003] For the fi...

AI Technical Summary

Problems solved by technology

[0008] In view of this, the present invention proposes a method and system for continuous data capture based on the Actor model, which is used to solve the problems of low data capture application speed, data loss, and data duplication, and is suitable for continuous data capture in a production environment need

Images