A Batch Storage Method Based on Distributed Messages

Abstract

The invention discloses a batch storage method based on distributed messages, which belongs to the technical field of Internet finance and solves the I / O performance bottleneck problem of database reading and writing caused by simultaneous writing of a large amount of high-concurrency data within the same period of time. Including asynchronous message processing service and timing scheduling service; the asynchronous message processing service is responsible for performing anti-duplication processing before writing business data according to different business scenarios. After the processing is completed, it is grouped into distributed memory according to business scenarios In the cache database: the scheduled scheduling service is responsible for starting the scheduling task every few minutes, querying the business data registered in the distributed memory cache server, and if there is, the business data is read and then batch storage operation is performed. Its purpose is to greatly reduce database interaction, solve the problem that the CPU, memory, and disk IO of the database are continuously fully loaded, and ensure the stable operation of the distributed consumer credit core.

Description

technical field [0001] The invention belongs to the technical field of Internet finance, and in particular relates to a batch storage method based on distributed messages. Background technique [0002] In the consumer credit business, with the rapid development of the business and the evolution of the distributed and microservices of the software architecture, a large amount of business data has been brought. In high concurrency scenarios, the consumer side of the server will receive a large number of concurrent asynchronous business message data and perform database storage operations. With the rapid increase in business volume, the application architecture layer of the distributed consumer credit system can be solved by a microservice-based horizontal expansion solution, but the horizontal expansion of the application server also transfers the bottleneck problem of concurrent access to the database. . [0003] Distributed consumer credit carries out message reception and...

AI Technical Summary

Problems solved by technology

[0003] Distributed consumer credit carries out message reception and warehousing of the business system. This processing method strongly binds the processing of business processes with a single data warehousing process, and in the time period of evening interest accrual batches, each message request Both need to update and query operations with the database, and the coupling degree of business information and data processing information is high, which does not conform to the principle of microservice design

[0004] At the same time, the database will generate a large number of read and write requests in the same period of time, which will bring huge pressure to the database. Even under continuous high-concurrency reading, the database will respond slowly, the processing will not be timely, and the request response will decline. High concurrency The request is equivalent to a large-traffic attack on the intranet and this attack is aimed at the database, which will cause the database to respond slowly. When performing disk IO operations, the database process cannot respond in time, causing the risk of database downtime

Since batch processing must be completed within a specified time period, this processing method puts a lot of pressure on the database, and batch processing cannot be completed within the time window, which affects downstream big data extraction and post-loan business system business development

Images