A multi-condition pagination query method in a distributed system

Abstract

The invention discloses a multi-condition paging query method under a distributed system. By constructing data subsets and intersecting the obtained data subsets, ID list data is obtained, and the whole data is replaced by querying key ID tags, only for The ID list is used for paging instead of paging the entire data, which reduces the overall data loading time and memory usage when the query conditions are distributed in multiple data sources and the query result data volume is large.

Description

technical field [0001] The invention relates to the field of data processing, and is particularly suitable for paging query of multiple data sources in a distributed system. Background technique [0002] In recent years, with the evolution of single applications to distributed systems and the popularity of microservices, many software systems have been split into multiple business subsystems, and each business subsystem has its own independent database. For software products that provide enterprise services for the B-side, it is first necessary to manage the employee accounts of the enterprise. Usually, a set of background user management system is provided for enterprise customers, and the IT administrator of each enterprise can manage the employee accounts of their own enterprise. [0003] For the convenience of explanation, we will refer to the background user management system as the management system below, and an example will be given to explain the problem in detail. ...

AI Technical Summary

Problems solved by technology

[0006] The above methods, either through data synchronization, change multiple data sources into a single data source, and simplify multi-subsystem queries into database single-table queries, thereby avoiding the paging problem of multiple data source queries, but there are also many problems: data consistency It is difficult to synchronize multiple data sources to ensure that the data in the two subsystems is completely consistent. Downtime, network disconnection, and data processing failures within the system are unavoidable problems; the impact on the architecture is great, and the entire architecture is Need to make adjustments, to introduce mq, new subsystems and corresponding DB storage, etc., need to make relatively large adjustments to the architecture, which is not conducive to the stability of the system

[0007] At the same time, the scalability is poor. Once a new screening field needs to be added, such as a position, not only the data synchronization mechanism for the position field needs to be added, but also the existing position data needs to be washed to the subsystem. No matter the difficulty of operation or the risk Very large; Secondly, as long as the number of users is slightly large, the performance of multi-table joint query will be very poor. A complex joint table SQL may not be returned within tens of seconds, directly slowing down the entire DB, not only the performance of this complex query cannot be guaranteed, but even will affect the availability of the entire system

[0008] At the same time, in the current background user management system, advanced filtering functions are often provided for the convenience of retrieval. Advanced filtering will have multiple filtering conditions. The data sources corresponding to these filtering conditions may be distributed in multiple business subsystems. Traditional SQL-based Conditional query can only query one business subsystem at a time. It is impossible to get the screening results that meet all the conditions in a single business subsystem. The conditional query results of each business subsystem must be aggregated to merge the final qualified data set. Based on the conditions The data set can be displayed in pages for the terminal

But in the end, there may be tens of thousands of qualified data sets, each with more than a dozen fields, including large fields such as mailboxes and departments that occupy dozens of bytes, and one piece of data occupies at least 100B+, and 100,000 pieces of data The data is at least 10MB+. It may take several seconds or even tens of seconds to query and load tens of thousands of pieces of data from each subsystem into the memory. This response delay is unacceptable for terminal use

Images