A method and server for data parallel processing

Abstract

The invention discloses a data parallel processing method and server, wherein the method includes: loading first data from a database into the memory, the first data being the latest data corresponding to the current first moment; when detecting When data processing is blocked in the database, the second data is read from the memory, and the second data is old data corresponding to the previous second moment; billing processing is performed according to the second data, and the data to be submitted is obtained. For the data to be submitted in the database, put the data to be submitted in the buffer; perform asynchronous persistence processing of the database according to the data to be submitted obtained by continuously reading from the buffer.

Description

technical field [0001] The invention relates to data processing technology, in particular to a data parallel processing method and server. Background technique [0002] In data processing, the simplest processing logic is usually executed sequentially. For example, there are three steps a, b, and c. The processing logic is to execute step a first, then step b, and finally step c. However, the processing logic of sequential execution seems simple, but there are various problems in practical applications. For example, in the reading and writing scenario, writing and reading are processed serially in order, which is inefficient. As another example, taking the billing business processing operation as an example, each time a billing task is processed, 1) data loading is performed first, for example, the balance data of the information recommender (such as an advertiser) and information related to information recommendation will be loaded from the database. Information such as bu...

AI Technical Summary

Problems solved by technology

[0003] It can be seen that the above operations are performed serially. The previous step in 1)-3) has not been executed, and the latter step cannot be performed. With the further development of the business, more and more data needs to be processed, which will lead to Data loading and database persistence operations are taking longer and longer

In addition, some data that needs to be processed is shared by the foreground and the background. If the front-end database has a large number of concurrent accesses, the read and write operations of the database will be blocked, which will affect the normal execution of the entire billing business process. Not only the processing efficiency Low, but also affect the accuracy of billing processing

However, in related technologies, there is no effective solution to this problem

Images