Multi-data source aggregation query methods, devices, equipment, media and products
By introducing an aggregation query component at the application layer, encapsulating data source logic and sorting rules, the high transformation cost and dependency issues caused by the introduction of distributed middleware are resolved, achieving low-intrusion and highly flexible multi-data source aggregation queries.
Patent Information
- Application Number
- CN202610987396.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-03
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, the introduction of distributed middleware for multi-data source aggregation queries is highly invasive, resulting in high costs for modifying existing application systems and excessive reliance on middleware, which affects system flexibility.
An aggregation query component is introduced at the application layer, which encapsulates the data retrieval logic and data sorting rules of multiple data sources. It receives aggregation query requests, parses task parameters, calls the data retrieval logic to obtain sharded data, performs ordered aggregation, and returns the results. It supports caching and dynamic data source management, and monitors the response status in real time to isolate abnormal data sources.
It enables low-intrusion multi-data source aggregation queries, reduces the cost of transforming existing application systems, improves system flexibility and maintainability, and avoids dependence on middleware.
Smart Images

Figure CN123115208A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of big data and financial technology, and in particular to a method, apparatus, device, medium and product for multi-data source aggregation query. Background Technology
[0002] As financial institutions undergo architectural transformation of their application systems, some customer-facing applications experience high online transaction volumes that cannot be supported by a single underlying database architecture. Therefore, horizontal sharding of the database is necessary. However, in certain business scenarios, the sharded application systems require aggregated queries and summary displays of data from multiple shards. Therefore, a solution suitable for aggregated queries from multiple data sources and supporting sorting is required.
[0003] Current industry solutions for aggregating queries from multiple data sources require the development of multiple distributed middleware to provide comprehensive SQL parsing capabilities and support cross-shard JOIN, subqueries, and aggregate functions (SUM / AVG / COUNT). However, introducing distributed middleware is highly intrusive to existing application systems, incurs high engineering costs, and leads to excessive reliance on middleware by the application side, potentially causing bottlenecks in subsequent updates and upgrades and impacting the flexibility of the application system.
[0004] Therefore, existing solutions that introduce middleware are costly to implement and rely excessively on middleware, and there is an urgent need for a low-intrusion, highly flexible solution. Summary of the Invention
[0005] This application provides a method, apparatus, device, medium, and product for multi-data source aggregation query, which achieves the technical effect of multi-data source aggregation query while maintaining high system flexibility and with minimal intrusion into existing application systems.
[0006] Firstly, embodiments of this application provide a multi-data source aggregation query method, applied to an aggregation query component. The aggregation query component encapsulates: data retrieval logic for each of the multiple data sources, and data sorting rules among the multiple data sources. The data retrieval logic is configured by the caller. The aforementioned multi-data source aggregation query method includes:
[0007] Receive multi-data source aggregation query requests sent by the caller, and parse the aggregation query task parameters from the multi-data source aggregation query requests;
[0008] The data retrieval logic is invoked according to the parameters of the aggregate query task to obtain the sharded data returned by each data source;
[0009] The system invokes the data sorting rules to perform ordered aggregation on each data segment, obtains the aggregated query results, and returns the aggregated query results to the caller.
[0010] In one possible implementation, the data source that has returned sharded data is the processed data source, and the data source that has not returned sharded data is the data source to be processed; the data retrieval logic is invoked according to the aggregate query task parameters to obtain the sharded data returned by each data source, including:
[0011] Based on the parameters of the aggregate query task, multiple threads call at least a portion of the data retrieval logic from the data source to be processed.
[0012] After obtaining the returned fragmented data, the thread occupied by the processed data source is released, and the process of calling at least part of the data retrieval logic of the pending data source is returned to continue until there is no more pending data source.
[0013] In one possible implementation, returning the aggregated query results to the caller includes:
[0014] Extract the target data of the specified page from the aggregate query results based on the pagination parameters indicated by the aggregate query task parameters;
[0015] Return the target data to the caller.
[0016] In one possible implementation, after returning the aggregated query results to the caller, the process further includes:
[0017] Cache the aggregated query results to the preset cache space;
[0018] When a new multi-data source aggregation query request is received, it checks whether there is valid cached data in the preset cache space that matches the new multi-data source aggregation query request;
[0019] If valid cached data exists, the valid cached data will be returned to the caller that sent the new multi-data source aggregation query request.
[0020] In one possible implementation, after parsing the aggregation query task parameters, the method further includes:
[0021] If a data source update operation is detected, the data source pool is updated based on the data source update operation, and multiple updated data sources in the updated data source pool are determined.
[0022] The data retrieval logic is invoked according to the parameters of the aggregate query task to obtain the sharded data returned by each updated data source;
[0023] The system invokes data sorting rules to perform ordered aggregation of the sharded data returned by each updated data source, obtains the aggregated query results, and returns the aggregated query results to the caller.
[0024] In one possible implementation, the multi-data source aggregation query method further includes:
[0025] After calling the data retrieval logic according to the parameters of the aggregate query task, the response status of each data source is monitored in real time;
[0026] If an abnormal response status is detected, the data source in the abnormal response status is isolated, and the data sorting rules are invoked to perform ordered aggregation on the sharded data returned by the data source in the normal response status to obtain the aggregation query result. The missing sharded data information is marked in the aggregation query result.
[0027] Secondly, embodiments of this application provide a multi-data source aggregation query device applied to an aggregation query component. The aggregation query component encapsulates: data retrieval logic for each of the multiple data sources, and data sorting rules among the multiple data sources. The data retrieval logic is configured by the caller. The aforementioned multi-data source aggregation query device includes:
[0028] The task parameter parsing module is used to receive multi-data source aggregation query requests sent by the caller and parse the aggregation query task parameters from the multi-data source aggregation query requests;
[0029] The data retrieval logic call module is used to call the data retrieval logic according to the parameters of the aggregate query task to obtain the sharded data returned by each data source;
[0030] The sorting and aggregation module is used to invoke data sorting rules, perform ordered aggregation of data from each data segment, obtain aggregated query results, and return the aggregated query results to the caller.
[0031] In one possible implementation, the data source that has returned fragmented data is the processed data source, and the data source that has not returned fragmented data is the pending data source. The data retrieval logic call module includes:
[0032] The thread calling unit is used to call the data retrieval logic of at least a part of the data source to be processed in multiple threads according to the parameters of the aggregate query task.
[0033] The thread release unit is used to release the thread occupied by the processed data source after obtaining the returned fragmented data, and return the steps of multi-threaded calls to at least part of the data retrieval logic of the pending data source, until there is no pending data source.
[0034] In one possible implementation, the sorting aggregation module includes:
[0035] The local data extraction unit is used to extract target data from the aggregate query results for a specified page based on the pagination parameters indicated by the aggregate query task parameters.
[0036] The local data return unit is used to return the target data to the caller.
[0037] In one possible implementation, the above-mentioned multi-data source aggregation query device further includes a query result caching module, which includes:
[0038] The result caching unit is used to cache the results of aggregated queries to a preset cache space;
[0039] The cache matching unit is used to detect whether there is valid cached data in the preset cache space that matches the new multi-data source aggregation query request when a new multi-data source aggregation query request is received.
[0040] The cache return unit is used to return valid cached data to the caller that sent the new multi-data source aggregation query request if valid cached data exists.
[0041] In one possible implementation, the above-mentioned multi-data source aggregation query device further includes a data source update module, which includes:
[0042] The data source pool update unit is used to update the data source pool based on the data source update operation if a data source update operation is detected, and to determine multiple updated data sources in the updated data source pool.
[0043] The update shard data return unit is used to call the data retrieval logic according to the parameters of the aggregation query task to obtain the shard data returned by each update data source;
[0044] The update aggregation result return unit is used to invoke the data sorting rules, perform ordered aggregation on the sharded data returned by each updated data source, obtain the aggregation query result, and return the aggregation query result to the caller.
[0045] In one possible implementation, the multi-data source aggregation query device further includes a data source status monitoring module, which includes:
[0046] The response status monitoring unit is used to monitor the response status of each data source in real time after the data retrieval logic is called according to the parameters of the aggregate query task.
[0047] The missing data marking unit is used to isolate the data source in an abnormal response state if an abnormal response state is detected, and to call the data sorting rules to perform ordered aggregation on the fragmented data returned by the data source in a normal response state to obtain the aggregation query result, and to mark the missing fragmented data information in the aggregation query result.
[0048] Thirdly, embodiments of this application provide a computer device, including: a memory and a processor; the memory stores a computer program, and the processor executes the computer program to perform the first aspect and / or various possible implementations of the first aspect.
[0049] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0050] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0051] The multi-data source aggregation query method, apparatus, device, medium, and product provided in this application embodiment are applied to an aggregation query component, that is, an aggregation query component is introduced at the application layer. This aggregation query component encapsulates the data retrieval logic for each of the multiple data sources and the data sorting rules among the multiple data sources, and the data retrieval logic is configured by the caller. Based on this, by receiving a multi-data source aggregation query request sent by the caller, the aggregation query task parameters are parsed from the multi-data source aggregation query request. Then, the data retrieval logic is called according to the aggregation query task parameters to obtain the fragmented data returned by each data source, thereby calling the data sorting rules to perform ordered aggregation of each fragmented data to obtain the aggregation query result, and the aggregation query result is returned to the caller. The entire process introduces an aggregation query component at the application layer to realize multi-data source query, fragmented data result aggregation, and sorting. On the one hand, the caller only needs to integrate the aggregation query component at the application layer, thereby reducing the intrusion into the existing application system and reducing the transformation cost of the existing application system. On the other hand, it does not rely on middleware, which can avoid the limitations of middleware on the system and improve system flexibility. Therefore, multi-data source aggregation query can be realized with low intrusion into the existing application system and while maintaining high system flexibility. Attached Figure Description
[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0053] Figure 1 A schematic diagram illustrating a scenario for the multi-data source aggregation query method provided in this application;
[0054] Figure 2 A flowchart illustrating the multi-data source aggregation query method provided in this application;
[0055] Figure 3 A flowchart illustrating the process of cross-shard database aggregation query of resource borrowing progress provided in this application;
[0056] Figure 4 This is a schematic diagram of the structure of the multi-data source aggregation query device provided in this application;
[0057] Figure 5A schematic diagram of the structure of the computer device provided in this application.
[0058] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in any of the foregoing embodiments.
[0060] It should be noted that all information (including but not limited to device information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, and necessary confidentiality measures have been taken. These measures do not violate public order and good morals, and corresponding operation entry points are provided for users to choose to authorize or refuse. Moreover, the technical solution involved in this application, which conducts big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, and electronic terminal operation data) and uses artificial intelligence technology for automated decision-making, making decisions that have a significant impact on individual rights based on the automated decision results, provides users with corresponding operation entry points for users to choose to agree to or refuse the automated decision results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.
[0061] It should be noted that the multi-data source aggregation query method, apparatus, computer equipment, computer-readable storage medium and computer program product provided in this application can be used in the fields of big data and fintech, as well as in any other fields. This application does not limit the application fields of the multi-data source aggregation query method, apparatus, computer equipment, computer-readable storage medium and computer program product.
[0062] Figure 1 This is a schematic diagram illustrating a scenario for the multi-data source aggregation query method provided in this application, such as... Figure 1As shown, taking a resource borrowing progress aggregation query service provided in the fintech field as an example, after database sharding is implemented in the existing application systems of financial institutions, an aggregation query component 100 is provided for multiple data sources formed by multiple sharded databases to achieve aggregation queries across multiple sharded databases. This aggregation query component 100 encapsulates: the data retrieval logic for each of the multiple data sources 200, and the data sorting rules among the multiple data sources 200. The data retrieval logic is configured by the caller 300. Based on this, the aggregation query component 100 can receive multi-data source aggregation query requests sent by the caller 300, parse the aggregation query task parameters from the multi-data source aggregation query requests, and then call the data retrieval logic according to the aggregation query task parameters to obtain the sharded data returned by each data source 200. Furthermore, the aggregation query component 100 can call the data sorting rules to perform ordered aggregation of the sharded data, obtain the aggregation query result, and return the aggregation query result to the caller 300.
[0063] The multi-data source aggregation query method provided in this application can realize multi-data source query, fragmented data result aggregation, and sorting processing by introducing an aggregation query component 100 at the application layer. On the one hand, the caller 300 only needs to integrate the aggregation query component 100 at the application layer, reducing the intrusion and modification of existing application systems and lowering the modification costs of existing application systems. On the other hand, it does not rely on middleware, avoiding the limitations imposed by middleware on existing application systems and improving system flexibility. Therefore, multi-data source aggregation queries can be implemented with low intrusion into existing application systems and while maintaining high system flexibility. This solves the technical problems in existing technologies where the introduction of middleware leads to high intrusion into existing application systems, resulting in high implementation and modification costs for existing application systems, and excessive reliance on middleware, which affects system flexibility.
[0064] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0065] In one embodiment, a multi-data source aggregation query method is provided. This embodiment illustrates the application of the multi-data source aggregation query method to an aggregation query component. It is understood that the multi-data source aggregation query method can also be applied to a terminal or a server, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server.
[0066] Figure 2This is a flowchart illustrating a multi-data source aggregation query method provided in an embodiment of this application. The method is applied to an aggregation query component, which encapsulates the data retrieval logic for each of the multiple data sources, as well as data sorting rules among the multiple data sources. The data retrieval logic is configured by the caller. That is, a lightweight aggregation query component is integrated on the application side. Through the logic encapsulated within the aggregation query component, the application side provides a multi-data source data aggregation query service between the application layer and the database, achieving aggregation queries across multiple data sources, and the results support sorting. It should be noted that the aggregation query component does not rely on distributed middleware; instead, it can coordinate the data acquisition, aggregation, sorting, and ordered result return from multiple data sources based on the data retrieval logic and / or data sorting rules provided by the caller. This application significantly reduces the transformation cost of existing application systems, reduces dependence on middleware, and improves the flexibility and maintainability of system business scenarios by delegating the aggregation logic to the application layer. Based on this, as... Figure 2 As shown, this multi-data source aggregation query method includes:
[0067] S201. Receive the multi-data source aggregation query request sent by the caller, and parse the aggregation query task parameters from the multi-data source aggregation query request.
[0068] Specifically, multiple data sources can refer to the multiple sharded databases generated after database sharding is implemented in the existing application systems of financial institutions. Each sharded database can correspond to one data source. Each data source can be a database instance or interface service that supports SQL queries. SQL (Structured Query Language) is a domain-specific language used to manage relational database management systems. Its core function is to define, manipulate, query, and control the data stored in relational databases.
[0069] Optionally, the aggregation query component can receive multi-data source aggregation query requests sent by the caller and parse the aggregation query task parameters from the multi-data source aggregation query requests, such as parsing standardized query statements (SQL statements), multi-data source information (including a list of data sources containing multiple sharded databases), pagination parameters (including page numbers and the number of records per page), etc.
[0070] For example, if the caller sends a multi-data source aggregation query request that is "query content B as required by A", the aggregation query component can parse the multi-data source aggregation query request, locate the storage location of content B, thereby confirming the data source, and then calling the data retrieval logic corresponding to the two data sources.
[0071] S202. Call the data retrieval logic according to the parameters of the aggregation query task to obtain the sharded data returned by each data source.
[0072] The data retrieval logic can be configured and implemented by the caller. Specifically, the data retrieval logic refers to the SQL execution logic or interface call logic implemented by the caller for the corresponding data source type.
[0073] Optionally, the aggregation query component can call the data retrieval logic corresponding to each data source according to the aggregation query task parameters, obtain the sharded data returned by each data source, store the obtained sharded data in memory, and then perform ordered aggregation.
[0074] For example, the aggregation query component can call the data retrieval logic encapsulated by the caller (such as executing the SQL statement "SELECT * FROM account WHERE shard_id=1"), and through the SQL execution component deployed on the application side, execute the query statement (SQL statement) indicated by the called data retrieval logic to obtain the returned sharded data.
[0075] S203. Call the data sorting rules to perform ordered aggregation of each data segment, obtain the aggregated query results, and return the aggregated query results to the caller.
[0076] Specifically, ordered aggregation refers to merging fragmented data in memory according to data sorting rules to ensure that the order of the final result is consistent with the requirements of the sorting rules.
[0077] Optionally, the aggregation query component can invoke data sorting rules, perform ordered aggregation of each data segment according to the data sorting rules, obtain the aggregation query result, and return the aggregation query result to the caller. The caller can implement the aggregation query function without writing complex code.
[0078] For example, if the data sorting rule indicates that the data should be sorted in descending order by the "balance" field, the aggregation query component can sort the data of each shard in descending order by the "balance" field and then merge the results.
[0079] The aforementioned multi-data source aggregation query method is applied to an aggregation query component, that is, an aggregation query component is introduced at the application layer. This component encapsulates the data retrieval logic for each of the multiple data sources, as well as the data sorting rules among the multiple data sources, with the data retrieval logic configured by the caller. Based on this, by receiving multi-data source aggregation query requests sent by the caller, the aggregation query task parameters are parsed from the requests. Then, the data retrieval logic is invoked according to the aggregation query task parameters to obtain the sharded data returned by each data source. The data sorting rules are then invoked to perform ordered aggregation of the sharded data, obtaining the aggregation query result, which is then returned to the caller. The entire process introduces an aggregation query component at the application layer, realizing multi-data source querying, sharded data result aggregation, and sorting. On the one hand, the caller only needs to integrate this aggregation query component at the application layer, thereby reducing the intrusion into existing application systems and lowering the cost of modifying existing application systems.
[0080] On the other hand, not relying on middleware avoids the limitations imposed by middleware on the system and improves system flexibility. Therefore, it is possible to achieve aggregated queries from multiple data sources with minimal intrusion into existing application systems while maintaining high system flexibility.
[0081] In one possible implementation, the data source that has returned sharded data is the processed data source, and the data source that has not returned sharded data is the pending data source. Based on this, the data retrieval logic is invoked according to the aggregation query task parameters to obtain the sharded data returned by each data source, including: invoking the data retrieval logic of at least a portion of the pending data sources in multiple threads according to the aggregation query task parameters; after obtaining the returned sharded data, the threads occupied by the processed data sources are released, and the steps of invoking the data retrieval logic of at least a portion of the pending data sources in multiple threads are returned, until there are no more pending data sources.
[0082] Optionally, the aggregation query component can determine the data sources to be processed in this aggregation query based on the multi-data source information carried in the aggregation query task parameters. Then, based on numerous idle threads in the thread pool, multiple threads call the data retrieval logic corresponding to at least a portion of the data sources to be processed, thereby executing the query statements (SQL statements) indicated by the called data retrieval logic in parallel. This yields the fragmented data returned by each of the corresponding data sources to be processed, and marks the data sources that have returned fragmented data as processed data sources. Furthermore, after obtaining the returned fragmented data, the aggregation query component can control the release of the threads occupied by the processed data sources back to the thread pool, returning to the step of multiple threads calling the data retrieval logic of at least a portion of the data sources to be processed, so as to utilize idle threads to repeatedly execute the query task until there are no more data sources to be processed.
[0083] For example, the aggregation query component can invoke a thread pool deployed on the application side based on a "polling + dynamic replenishment" scheduling strategy. The number of threads in this thread pool can be dynamically set according to the number of CPU cores on the application side. Furthermore, during scheduling, the aggregation query component can set an upper and lower limit on the number of threads to be scheduled, ensuring that the application side has enough idle threads to handle other tasks, while also allowing the aggregation query component to schedule enough idle threads to execute query tasks. The aggregation query component can terminate polling when there are no more data sources to process. This multi-threaded scheduling strategy maximizes thread utilization, providing high throughput and low latency thread resource support for aggregation queries in large-scale sharded databases.
[0084] In this embodiment, a parallel data retrieval mechanism is introduced into the aggregation query component. By acquiring data from each shard in parallel through multiple threads, the total time spent acquiring shard data can be effectively reduced. This is especially suitable for scenarios with a large number of sharded databases and can reduce the overall query time.
[0085] In one example, returning the aggregate query result to the caller includes: extracting the target data of a specified page from the aggregate query result according to the pagination parameters indicated by the aggregate query task parameters; and returning the target data to the caller.
[0086] The pagination parameters may include at least: page number (including start and end page number), number of items per page (e.g., page A contains 10 data items), and may also include offset, which can be used to skip a portion of the data on a specified page (e.g., skip data items X to Y on page A).
[0087] Specifically, the aggregation query component can first perform compliance checks on the pagination parameters indicated by the aggregation query task parameters. For example, the page number in the pagination parameters must not be less than 1, nor exceed a preset upper limit (such as a maximum of 1000), to prevent memory overflow or network timeouts caused by returning too much data at once. Furthermore, after the pagination parameters pass the compliance check, the aggregation query component can extract the target data for a specified page from the aggregation query results according to the pagination parameters indicated by the aggregation query task parameters, and then return the target data to the caller. For example, extracting the target data from page a1 to page a2 from the aggregation query results and then returning the target data to the caller.
[0088] For example, during the truncation process, the aggregation query component can perform truncation in batches to avoid lag caused by truncation of too much data at once. The aggregation query component can truncate only a fixed number of consecutive pages of data each time, and then merge the data from multiple truncations in the order of truncation to form the final target data.
[0089] In this embodiment, the aggregated query results after aggregation and sorting can be further processed. For example, the sorted aggregated query results can be partially retrieved and returned, thereby optimizing query performance and providing more refined and diversified query services.
[0090] In one possible implementation, after returning the aggregated query result to the caller, the method further includes: caching the aggregated query result in a preset cache space; upon receiving a new multi-data source aggregated query request, detecting whether there is valid cached data in the preset cache space that matches the new multi-data source aggregated query request; and if there is valid cached data, returning the valid cached data to the caller that sent the new multi-data source aggregated query request.
[0091] Optionally, a result caching mechanism can be introduced into the aggregation query component to cache the aggregation query results in a preset cache space on the application side. Upon receiving a new multi-data source aggregation query request, the aggregation query component can check whether there is valid cached data in the preset cache space that matches the new multi-data source aggregation query request. If the new multi-data source aggregation query request indicates the extraction of a specified type of data from multiple specified data sources, and the preset cache space has cached this specified type of data within a preset time period (e.g., 12 hours, or other durations set according to the actual application scenario), then it can be determined that there is valid cached data in the preset cache space that matches the new multi-data source aggregation query request. Furthermore, if valid cached data exists, the aggregation query component can return the valid cached data to the caller that sent the new multi-data source aggregation query request, thus reusing the aggregation query results.
[0092] For example, for aggregated query results cached in the preset cache space, if the caching duration of a certain aggregated query result has reached the caching duration threshold (such as 24 hours, or other durations set according to the actual application scenario), the aggregated query component can delete the aggregated query result that has reached the caching duration threshold from the preset cache space to maintain the timeliness of the cached data in the preset cache space and reduce the redundancy of the cached data in the preset cache space. In addition, if a data invalidation instruction is received for the cached data in the preset cache space, such as "invalidate the aggregated query result cached at the specified time point XX year XX month XX day XX hour XX minute XX second", the aggregated query component can delete the aggregated query result cached at the specified time point from the preset cache space.
[0093] In this embodiment, the aggregated query results can be cached and reused in subsequent identical or similar queries, reducing the repeated retrieval of aggregated query results for high-frequency queries. The caching mechanism can alleviate the pressure on aggregated queries and improve query response speed and resource utilization.
[0094] In one example, after parsing the aggregation query task parameters, the process also includes: if a data source update operation is detected, updating the data source pool based on the data source update operation, determining multiple updated data sources in the updated data source pool; calling the data retrieval logic according to the aggregation query task parameters to obtain the sharded data returned by each updated data source; calling the data sorting rules to perform ordered aggregation on the sharded data returned by each updated data source to obtain the aggregation query result, and returning the aggregation query result to the caller.
[0095] Specifically, the aggregation query component can monitor for data source update operations based on data source change listeners deployed on the application side. If a data source update operation is detected, the aggregation query component can update the data source pool based on the data source update operation, which includes, but is not limited to, adding, deleting, or switching data sources (i.e., sharded databases), thereby identifying multiple updated data sources in the updated data source pool. Furthermore, the aggregation query component can invoke data retrieval logic according to the aggregation query task parameters to execute the query statement indicated by the invoked data retrieval logic, obtain the sharded data returned by each updated data source, and then invoke data sorting rules to perform ordered aggregation of the sharded data returned by each updated data source to obtain the aggregation query result, which is then returned to the caller.
[0096] For example, the data source change listener can flexibly combine the following monitoring methods according to the deployment environment on the application side: (1) Configuration center proactive push: Integrate mainstream configuration centers in the industry. When the data source connection information (such as connection address, account, password, interface or port name) or sharding routing rules change, the configuration center pushes the change event through long polling. After receiving the change event, the aggregation query component determines that a data source change operation has been detected. (2) Periodic polling comparison: The aggregation query component starts a lightweight background thread (the default interval is 20 seconds, or other intervals can be flexibly configured according to actual needs), periodically loads the latest data source configuration, and performs a deep comparison with the current data source configuration in memory. When a difference is found, it determines that a data source change operation has been detected.
[0097] Optionally, while invoking the data retrieval logic according to the parameters of the aggregation query task, the aggregation query component can obtain the version numbers of multiple data sources in the current data source pool and bind these version numbers to threads. When executing the query operation corresponding to the current multi-data source aggregation query request, it can always be executed based on multiple data sources under this version number. Even if the data source pool is updated again during this period, the query operation being executed will not be affected in any way, and the obtained connection will still come from the data source pool in the record, ensuring the consistency of the query process.
[0098] In this embodiment, a dynamic data source configuration mechanism is introduced into the aggregation query component, which allows the data source to be dynamically adjusted at runtime according to needs (such as adding, deleting or switching sharded databases) without restarting the service or modifying the code, significantly improving the adaptability of the existing application system to business needs.
[0099] In some optional embodiments, the multi-data source aggregation query method further includes: after calling the data retrieval logic according to the aggregation query task parameters, monitoring the response status of each data source in real time; if an abnormal response status is detected, isolating the data source in the abnormal response status, and calling the data sorting rules to perform ordered aggregation of the sharded data returned by the data source in the normal response status to obtain the aggregation query result, and marking the missing sharded data information in the aggregation query result.
[0100] The abnormal response status includes at least the following: (1) Connection timeout: If no response is received after the preset timeout threshold (e.g., 5 seconds) during the database connection establishment phase, the connection is considered abnormal. (2) Query timeout: If the specified query duration (e.g., 30 seconds) is exceeded during the query execution phase, the query is considered to have timed out. (3) Abnormal data format: The data structure of the returned result set does not match the expected data result (column name, data type), such as missing fields or type conversion failure. (4) Network unreachable: The communication connection is interrupted. (5) Empty result: The data source returns an empty result set, but according to the data source history or metadata statistics, the sharded database should have data within this time range.
[0101] Optionally, after invoking the data retrieval logic according to the aggregation query task parameters, the aggregation query component can monitor the response status of each data source in real time based on preset response status monitoring rules. If an abnormal response status is detected, the aggregation query component can isolate the data source in the abnormal response status. After receiving the sharded data returned by the data source in the normal response status, the aggregation query component can invoke data sorting rules to perform ordered aggregation of the sharded data returned by the data source in the normal response status, obtain the aggregation query result, and mark the missing sharded data information in the aggregation query result for manual traceability.
[0102] For example, the aggregate query component can directly skip data sources that are in an abnormal response state in this query, mark them, reclaim the thread allocated to the abnormal data source, and not retry the data source in subsequent parallel tasks of this query, thus avoiding unnecessary waiting and ensuring that the overall query time is not prolonged by a single point of failure.
[0103] In this embodiment, an anomaly sharding detection and fault tolerance mechanism is introduced into the aggregation query component. When a data source (sharded database) responds abnormally, that data source can be automatically skipped and recorded, while the result is generated using the sharded data returned by the remaining data sources. This fault tolerance mechanism improves the stability and availability of data aggregation queries and facilitates the tracking and handling of anomalies.
[0104] Based on the various embodiments of the above multi-data source aggregation query method, such as Figure 3 As shown, taking the resource borrowing progress aggregation query service provided by a financial institution as an example, a flowchart of cross-shard database aggregation query of resource borrowing progress is provided. Specifically, users of financial institutions can trigger multi-data source aggregation query requests on the application side. The aggregation query component can receive the multi-data source aggregation query request through the application connection pool, and parse out the multi-data source aggregation query request carried in the multi-data source aggregation query request through the SQL parsing component, SQL optimization component, and SQL routing component deployed on the application side. Further, the aggregation query component can call the data retrieval logic according to the aggregation query task parameters, and execute the query statement (SQL statement) corresponding to the data retrieval logic through the SQL execution component deployed on the application side to obtain the shard data returned by each data source in the database connection pool (which can include shard database 1 to shard database N). Further, the aggregation query component can call the data sorting rules to perform ordered aggregation of each shard data to obtain the aggregation query result, and return the aggregation query result to the caller.
[0105] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0106] Based on the same inventive concept, this application also provides a multi-data source aggregation query apparatus for implementing the multi-data source aggregation query method described above. The solution provided by this multi-data source aggregation query apparatus is similar to the implementation scheme described in the multi-data source aggregation query method above. Therefore, the specific limitations in one or more apparatus embodiments provided below can be found in the limitations of the multi-data source aggregation query method described above, and will not be repeated here.
[0107] Figure 4 This is a schematic diagram of a multi-data source aggregation query device provided in an embodiment of this application. The device is applied to an aggregation query component, which encapsulates: the data retrieval logic for each of the multiple data sources, and the data sorting rules among the multiple data sources. The data retrieval logic is configured by the caller. For example... Figure 4 As shown, the multi-data source aggregation query device 400 provided in this embodiment includes:
[0108] The task parameter parsing module 401 is used to receive multi-data source aggregation query requests sent by the caller and parse the aggregation query task parameters from the multi-data source aggregation query requests.
[0109] The data retrieval logic call module 402 is used to call the data retrieval logic according to the parameters of the aggregation query task to obtain the sharded data returned by each data source;
[0110] The sorting and aggregation module 403 is used to call the data sorting rules, perform ordered aggregation on each data segment, obtain the aggregation query results, and return the aggregation query results to the caller.
[0111] The multi-data source aggregation query device provided in this application embodiment is applied to an aggregation query component, that is, an aggregation query component is introduced into the application layer. This aggregation query component encapsulates the data retrieval logic for each of the multiple data sources, as well as the data sorting rules among the multiple data sources, and the data retrieval logic is configured by the caller. Based on this, by receiving a multi-data source aggregation query request sent by the caller, the aggregation query task parameters are parsed from the multi-data source aggregation query request. Then, the data retrieval logic is called according to the aggregation query task parameters to obtain the fragmented data returned by each data source, thereby calling the data sorting rules to perform ordered aggregation of each fragmented data, obtaining the aggregation query result, and returning the aggregation query result to the caller. The entire process introduces an aggregation query component into the application layer to realize multi-data source query, fragmented data result aggregation and sorting. On the one hand, the caller only needs to integrate the aggregation query component at the application layer, thereby reducing the intrusion into the existing application system and reducing the transformation cost of the existing application system. On the other hand, it does not rely on middleware, which can avoid the limitations of middleware on the system and improve system flexibility. Therefore, multi-data source aggregation query can be realized with low intrusion into the existing application system and while maintaining high system flexibility.
[0112] In one possible implementation, the data source that has returned fragmented data is the processed data source, and the data source that has not returned fragmented data is the pending data source. The data retrieval logic call module includes:
[0113] The thread calling unit is used to call the data retrieval logic of at least a part of the data source to be processed in multiple threads according to the parameters of the aggregate query task.
[0114] The thread release unit is used to release the thread occupied by the processed data source after obtaining the returned fragmented data, and return the steps of multi-threaded calls to at least part of the data retrieval logic of the pending data source, until there is no pending data source.
[0115] In one possible implementation, the sorting aggregation module includes:
[0116] The local data extraction unit is used to extract target data from the aggregate query results for a specified page based on the pagination parameters indicated by the aggregate query task parameters.
[0117] The local data return unit is used to return the target data to the caller.
[0118] In one possible implementation, the above-mentioned multi-data source aggregation query device further includes a query result caching module, which includes:
[0119] The result caching unit is used to cache the results of aggregated queries to a preset cache space;
[0120] The cache matching unit is used to detect whether there is valid cached data in the preset cache space that matches the new multi-data source aggregation query request when a new multi-data source aggregation query request is received.
[0121] The cache return unit is used to return valid cached data to the caller that sent the new multi-data source aggregation query request if valid cached data exists.
[0122] In one possible implementation, the above-mentioned multi-data source aggregation query device further includes a data source update module, which includes:
[0123] The data source pool update unit is used to update the data source pool based on the data source update operation if a data source update operation is detected, and to determine multiple updated data sources in the updated data source pool.
[0124] The update shard data return unit is used to call the data retrieval logic according to the parameters of the aggregation query task to obtain the shard data returned by each update data source;
[0125] The update aggregation result return unit is used to invoke the data sorting rules, perform ordered aggregation on the sharded data returned by each updated data source, obtain the aggregation query result, and return the aggregation query result to the caller.
[0126] In one possible implementation, the multi-data source aggregation query device further includes a data source status monitoring module, which includes:
[0127] The response status monitoring unit is used to monitor the response status of each data source in real time after the data retrieval logic is called according to the parameters of the aggregate query task.
[0128] The missing data marking unit is used to isolate the data source in an abnormal response state if an abnormal response state is detected, and to call the data sorting rules to perform ordered aggregation on the fragmented data returned by the data source in a normal response state to obtain the aggregation query result, and to mark the missing fragmented data information in the aggregation query result.
[0129] The multi-data source aggregation query device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0130] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 5 As shown, the computer device 500 may include a memory 501 and a processor 502. Optionally, the computer device may further include a communication component 503, wherein the memory 501 and the processor 502 communicate; for example, the memory 501, the processor 502 and the communication component 503 may communicate via a communication bus 504, the memory 501 is used to store a computer program, and the processor 502 executes the computer program to implement the method of the above embodiments.
[0131] Optionally, the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps in the method embodiments disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0132] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.
[0133] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.
[0134] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.
[0135] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0138] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of this application and its equivalents, this application also intends to include these modifications and variations.
[0139] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0140] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.
[0141] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0142] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.
[0143] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.
[0144] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.
[0145] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0146] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.
[0147] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.
[0148] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.
Claims
1. A method for aggregation querying multiple data sources, characterized in that, The method is applied to an aggregation query component, which encapsulates: data retrieval logic for each of multiple data sources, and data sorting rules among the multiple data sources, wherein the data retrieval logic is configured by the caller; the method includes: Receive a multi-data source aggregation query request sent by the caller, and parse the aggregation query task parameters from the multi-data source aggregation query request; The data retrieval logic is invoked according to the parameters of the aggregation query task to obtain the sharded data returned by each data source; The data sorting rules are invoked to perform ordered aggregation on each of the data segments, resulting in an aggregated query result, which is then returned to the caller.
2. The method according to claim 1, characterized in that, Data sources that have returned sharded data are considered processed data sources, while data sources that have not returned sharded data are considered pending data sources. The step of calling the data retrieval logic according to the aggregation query task parameters to obtain the sharded data returned by each data source includes: According to the aggregate query task parameters, multiple threads call the data retrieval logic of at least a portion of the data sources to be processed; After obtaining the returned fragmented data, the thread occupied by the processed data source is released, and the process of calling at least part of the data retrieval logic of the pending data source is returned to continue until there is no more pending data source.
3. The method according to claim 1, characterized in that, Returning the aggregated query result to the caller includes: Based on the pagination parameters indicated by the aggregation query task parameters, extract the target data of the specified page from the aggregation query results; The target data is returned to the caller.
4. The method according to claim 1, characterized in that, After returning the aggregated query result to the caller, the method further includes: The aggregated query results are cached in a preset cache space; When a new multi-data source aggregation query request is received, it is checked whether there is valid cached data in the preset cache space that matches the new multi-data source aggregation query request; If the valid cached data exists, the valid cached data is returned to the caller that sent the new multi-data source aggregation query request.
5. The method according to claim 1, characterized in that, After parsing the parameters of the aggregation query task, the method further includes: If a data source update operation is detected, the data source pool is updated based on the data source update operation, and multiple updated data sources in the updated data source pool are determined. The data retrieval logic is invoked according to the parameters of the aggregation query task to obtain the sharded data returned by each of the updated data sources; The data sorting rules are invoked to perform ordered aggregation of the sharded data returned by each of the updated data sources, resulting in an aggregated query result, which is then returned to the caller.
6. The method according to claim 1, characterized in that, The method further includes: After the data retrieval logic is invoked according to the parameters of the aggregate query task, the response status of each data source is monitored in real time. If an abnormal response state is detected, the data source in the abnormal response state is isolated, and the data sorting rules are invoked to perform ordered aggregation on the fragmented data returned by the data source in the normal response state to obtain the aggregated query result. The fragmented data missing information is marked in the aggregated query result.
7. A multi-data source aggregation query device, characterized in that, The device is applied to an aggregation query component, which encapsulates: data retrieval logic for each of multiple data sources, and data sorting rules among the multiple data sources, wherein the data retrieval logic is configured by the caller; the device includes: The task parameter parsing module is used to receive the multi-data source aggregation query request sent by the caller and parse the aggregation query task parameters from the multi-data source aggregation query request; The data retrieval logic calling module is used to call the data retrieval logic according to the parameters of the aggregation query task to obtain the sharded data returned by each data source; The sorting and aggregation module is used to call the data sorting rules, perform ordered aggregation on each of the data segments, obtain the aggregation query results, and return the aggregation query results to the caller.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
