Data query method and device and server

Abstract

The invention relates to the technical field of big data processing, in particular to a data query method and device and a server. The data query method is applied to the server and comprises the steps that a request message sent by a data consumption end is received, and the request message at least comprises a query time interval and a query condition; based on the relationship between the query time interval and the update date of the local data source, determining a target data source, the target data source being at least one of the business data source and the local data source; wherein the update date is the date of the business data backed up to the local data source for the last time; querying in the target data source based on the query condition to obtain a query result; the query result is preprocessed, a response message is generated according to the preprocessed query result, and the response message is fed back to the data consumption end. By adopting the method provided by the invention, the risk that the data source has a fault when facing a high-concurrency query service can be reduced.

Description

Technical Field

[0001] The present application relates to the technical field of big data processing, and in particular to a data query method, device and server. Background Art

[0002] In the informatization construction of banks, in order to ensure the performance of the business processing scenarios of the core business system and to maintain a fast response to query services, the data processing system is usually implemented by combining online libraries and hot standby libraries. That is, the online library is responsible for real-time storage of business data generated by users and business platforms, and the online library writes the data to the hot standby library in real time and synchronously, so that the data of the two are consistent; the hot standby library is responsible for responding to all query requests, that is, all query requests are to search for data in the hot standby library.

[0003] However, the hot standby database has to synchronously write data to the online database in real time on the one hand, and also has to handle data request processing for query services on the other hand. This leads to a greater risk of failure for the data processing system when facing high-concurrency query services.

[0004] Therefore, how to reduce the risk of failure of data processing systems when facing high-concurrency query services is an issue that needs to be addressed urgently. Summary of the invention

[0005] Based on this, it is necessary to provide a data query method, device and server that can reduce the risk of failure of the data processing system when facing high-concurrency query services in response to the above technical problems.

[0006] In a first aspect, the present application provides a data query method, applied to a server, the method comprising:

[0007] Receive a request message sent by a data consumer, wherein the request message includes at least a query time interval and a query condition;

[0008] Based on the relationship between the query time interval and the update date of the local data source, determine the target data source, the target data source is at least one of the business data source and the local data source; wherein, at the beginning of each synchronization cycle, the business data source backs up the business data of the previous synchronization cycle to the local data source, and the update date is the date of the last backup of the business data to the local data source;

[0009] Query the target data source based on the query condition to obtain a query result;

[0010] The query result is preprocessed, a response message is generated from the preprocessed query result, and the response message is fed back to the data consumption end.

[0011] In one embodiment, determining the target data source based on the relationship between the query time interval and the update date includes:

[0012] If it is determined that the right end point of the query time interval is before the update date, the local data source is determined as the target data source;

[0013] If it is determined that the left end point or the right end point of the query time interval is within the update date, the business data source and the local data source are determined as target data sources;

[0014] If it is determined that the left end point of the query time interval is after the update date, the business data source is determined as the target data source.

[0015] In one embodiment, the request message further includes a query ID, and the querying in the target data source based on the query condition to obtain the query result includes:

[0016] Determining a query type based on the query ID;

[0017] If the query type is the first query, the target data source is queried based on the query time interval and the query conditions to obtain the target quantity of target data corresponding to the query conditions, and the business data source and / or the local data source are queried based on the query conditions and the target quantity to obtain the query results; the target quantity includes the quantity of target data corresponding to each of the business data source and the local data source.

[0018] In one embodiment, the method further comprises:

[0019] If the query type is a page-turning query, the target quantity determined by the query ID during the first query is called;

[0020] The business data source and / or the local data source is queried based on the query condition and the target quantity to obtain a query result.

[0021] In one embodiment, the request message further includes a request time, and determining the query type based on the query ID includes:

[0022] When it is determined that the query ID does not exist in the cache library, determining the query type is a first query, and writing the request time, the target quantity and the query ID into the cache library;

[0023] When it is determined that the query ID exists in the cache library, the query type is determined to be a page-turning query.

[0024] In one embodiment, the query condition includes a query order, a query start row, and a query quantity, and querying the business data source and / or the local data source based on the query condition and the target quantity to obtain a query result includes:

[0025] Determine the query end row according to the number of target data corresponding to the initial data source, the query order, the query start row and the query quantity; the query end row and the query start row both represent the time sequence of the data in the data source; the initial data source is the data source corresponding to the query start row;

[0026] Based on the query start row and the query end row, the business data source and / or the local data source is queried to obtain a query result.

[0027] In one embodiment, the method further comprises:

[0028] If the query condition input by the user does not include a query start row and a query quantity, the reference start row is used as the query start row, and the reference quantity is used as the query quantity.

[0029] In a second aspect, the present application further provides a data query device, the device comprising a receiving module, a target data source determination module, a query module and a preprocessing module, wherein:

[0030] A receiving module, used to receive a request message sent by a data consumer, wherein the request message includes at least a query time interval and a query condition;

[0031] A target data source determination module is used to determine the target data source based on the relationship between the query time interval and the update date of the local data source, wherein the target data source is at least one of the business data source and the local data source; wherein, at the beginning of each synchronization cycle, the business data source backs up the business data of the previous synchronization cycle to the local data source, and the update date is the date of the last backup of the business data to the local data source;

[0032] A query module, used to query the target data source based on the query condition to obtain a query result;

[0033] The preprocessing module is used to preprocess the query result, generate a response message from the preprocessed query result, and feed back the response message to the data consumption end.

[0034] In a third aspect, the present application further provides a server, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Receive a request message sent by a data consumer, wherein the request message includes at least a query time interval and a query condition;

[0036] Based on the relationship between the query time interval and the update date of the local data source, determine the target data source, the target data source is at least one of the business data source and the local data source; wherein, at the beginning of each synchronization cycle, the business data source backs up the business data of the previous synchronization cycle to the local data source, and the update date is the date of the last backup of the business data to the local data source;

[0037] Query the target data source based on the query condition to obtain a query result;

[0038] The query result is preprocessed, a response message is generated from the preprocessed query result, and the response message is fed back to the data consumption end.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0040] Receive a request message sent by a data consumer, wherein the request message includes at least a query time interval and a query condition;

[0041] Based on the relationship between the query time interval and the update date of the local data source, determine the target data source, the target data source is at least one of the business data source and the local data source; wherein, at the beginning of each synchronization cycle, the business data source backs up the business data of the previous synchronization cycle to the local data source, and the update date is the date of the last backup of the business data to the local data source;

[0042] Query the target data source based on the query condition to obtain a query result;

[0043] The query result is preprocessed, a response message is generated from the preprocessed query result, and the response message is fed back to the data consumption end.

[0044] In the above-mentioned data query method, device and server, the data between the local data source and the business data source are not updated synchronously in real time. Instead, at the end of each synchronization cycle, the business data source migrates the stored business data to the local data source, thereby reducing the burden of the local data source writing data in real time, thereby reducing the risk of failure; the local data source stores the business data of the ended synchronization cycle, and the business data source stores the business data in the current synchronization cycle. Therefore, when facing a query request, the target data source is determined according to the query time interval and the order of the update date of the last backup business data of the local data source, and then the local data source and / or the business data source are queried in a targeted manner, instead of just one data source independently undertaking all query services; and when facing data before and after the update date, the local data source and the business data source are queried separately, thereby reducing the processing pressure of the local data processing system when facing high-concurrency tasks, thereby reducing the risk of failure of the local data system when facing high-concurrency query services. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a structural diagram of a data processing system in the related art;

[0047] Figure 2 An application environment diagram of a data query method in an embodiment;

[0048] Figure 3 A schematic diagram of a flow chart of a data query method in an embodiment;

[0049] Figure 4 A schematic diagram of several possible situations when determining a target data source in an embodiment;

[0050] Figure 5 is a structural block diagram of a data processing system configured in a server in one embodiment;

[0051] Figure 6 A schematic diagram of a query path when the target data source is a local data source in an embodiment;

[0052] Figure 7 A schematic diagram of a query path when the target data source is a business data source in an embodiment;

[0053] Figure 8 A schematic diagram of a process for determining a query result in one embodiment;

[0054] Fig. 9 A schematic diagram of a process for determining a query result in another embodiment;

[0055] Fig.10 A schematic diagram of a query path when the target data source is a business data source and a local data source in an embodiment;

[0056] Fig.11 is a structural block diagram of a data query device in an embodiment;

[0057] Fig.12 FIG. 4 is a diagram showing the internal structure of a server in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] In the information construction of banks, in order to ensure the performance of the core business system business processing scenarios and to maintain a fast response to query services, the data processing system is usually implemented by combining online databases and hot standby databases. Figure 1 In the related art, the online database is responsible for storing the business data generated by users and business platforms in real time, and the online database writes the data to the hot standby database in real time and synchronously, so that the data of the two are consistent; the hot standby database is responsible for responding to all query requests, that is, all query requests are to search for data in the hot standby database. However, on the one hand, the hot standby database has to synchronously write the data to the online database in real time, and on the other hand, it has to deal with the data request processing of the query business, which leads to a greater risk of failure of the data processing system when facing high-concurrency query business. In addition, in order to reduce the burden of high-concurrency query business on the hot standby database, the span of a single query cycle is limited in the related art to ensure the optimal and stable response performance of the hot standby database.

[0060] Therefore, in order to reduce the risk of failure of the data processing system when facing high-concurrency query services, and also to enable the data consumer to query data with a larger period span at a time, the present application provides a data query method.

[0061] The data query method provided in the embodiment of the present application can be applied to Figure 2In the application environment shown, the terminal (data consumption end) 102 communicates with the server (service end) 104 through the network. The local data source and the business data source can store the business data that the server 104 needs to process, and the local data source and the business data source respectively store the business data in different synchronization cycles; the local data source and the business data source can be integrated on the server 104, or placed on the cloud or other network servers.

[0062] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0063] In an exemplary embodiment, Figure 3 As shown, a data query method is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 01-04:

[0064] Step 01: Receive a request message sent by a data consumer, where the request message includes at least a query time interval and a query condition.

[0065] For the embodiments of the present application, the data consumption end is the terminal device, which can be a self-service query terminal, or an electronic device such as a mobile phone, PC, and Pad. The request message may include a variety of information. The request message includes at least a query time interval and a query condition. The request message also includes a query ID; the query ID usually refers to an identifier used to uniquely identify and track this query request. The query condition represents the user's query needs, and the query condition supports a combination of conditions in a variety of business scenarios, which is not specifically limited in the embodiments of the present application.

[0066] Furthermore, if the request message includes a request time, it will be recognized by the server, and then the initiation time of the final request message will be determined to be the request time; if the request message does not include a request time, the current time in milliseconds is defaulted as the initiation time of the request message.

[0067] Step 02: Based on the relationship between the query time interval and the update date of the local data source, determine the target data source, where the target data source is at least one of the business data source and the local data source; wherein, at the beginning of each synchronization cycle, the business data source backs up the business data of the previous synchronization cycle to the local data source, and the update date is the date of the last backup of the business data to the local data source.

[0068] For the embodiment of the present application, two data sources are configured, namely a local data source and a business data source; the local data source and the business data source are both pre-configured databases; wherein the local data source stores the business data of the ended synchronization cycle, and the business data source stores the business data in the current synchronization cycle. Furthermore, adjacent synchronization cycles are continuous, for example, the 0 o'clock of each day is used as the division node of adjacent synchronization cycles; however, the specific duration and division nodes of the synchronization cycle are not specifically limited in the embodiment of the present application.

[0069] Furthermore, the moment when a synchronization cycle ends is also the moment when the next synchronization cycle begins. At the beginning of each synchronization cycle, the business data source backs up the stored business data to the local data source. Take the duration of the synchronization cycle as a full natural day, and take the division node of the adjacent cycle as 0 o'clock as an example; set the current time to 14:00 on September 3, 2024, then the current synchronization cycle is 0 o'clock on September 3, 2024-0 o'clock on September 4, 2024. The business data generated in the current synchronization cycle is stored in the business data source, and the business data generated before 0 o'clock on September 3, 2024 is stored in the local data source; and, at 0 o'clock on September 4, 2024, the current synchronization cycle ends and enters the next synchronization cycle. At this time, the business data source will back up the business data between 0 o'clock on September 3, 2024 and 0 o'clock on September 4, 2024 to the local data source for storage.

[0070] However, in some special time periods, such as holidays or month-end / quarterly accounting cycles, the business data source has a large amount of data that needs to be backed up to the local data source. At this time, it often causes delayed data backup. That is, at the beginning of each synchronization cycle, the business data of the business data source often cannot be backed up to the local data source on time.

[0071] The query time interval is a time period for the business data requested by the data consumer. The query time interval can be within a synchronization cycle that has ended, or within the current synchronization cycle, or within both the synchronization cycle that has ended and the current synchronization cycle. The query time interval is specifically determined by the needs of the data consumer. In an embodiment of the present application, it is necessary to obtain the update date of the last backup of the business data of the local data source, and then determine whether there is data in the time interval corresponding to the query request in the local data source based on the update date; that is, if the query time interval includes the time period before the update date, the target data source includes the local data source, and if the query time interval does not include the time period before the update date, the target data source does not include the local data source. In other words, the target data source can be the local data source itself, or it can be the business data source itself, or it can be both the local data source and the business data source.

[0072] Step 03: Query the target data source based on the query conditions to obtain the query results;

[0073] Step 04: pre-process the query results, generate a response message from the pre-processed query results, and feed the response message back to the data consumer.

[0074] For the embodiment of the present application, when the data consumer sends a query message to the server, an encrypted communication link will be established between the server and the data consumer; after determining the target data source, the target data source is queried based on the query conditions to obtain the query results, and the query results will be fed back to the data consumer based on the communication link.

[0075] Furthermore, the server preprocesses the query results to obtain preprocessed query results. Among them, preprocessing includes one or more of data desensitization, data format conversion, data alignment, data deduplication, data clustering, and data calculation and data visualization processing; the specific processing method of preprocessing is not specifically limited in the embodiment of the present application. The implementation method of the query result preprocessing can be to configure the algorithm or model for implementation and processing in the server, or to configure each module that implements the preprocessing function into an API interface, and then the server calls the corresponding preprocessing function module through the API interface to implement the preprocessing of the query results. The implementation method of preprocessing in the embodiment of the present application is not specifically limited.

[0076] In the above data query method, the data between the local data source and the business data source are not updated synchronously in real time. Instead, at the end of each synchronization cycle, the business data source migrates the stored business data to the local data source, which reduces the burden of real-time data writing of the local data source, thereby reducing the risk of failure. The local data source stores the business data of the ended synchronization cycle, and the business data source stores the business data in the current synchronization cycle. Therefore, when facing a query request, the target data source is determined based on the query time interval and the order of the update date of the last backup of the business data by the local data source, and then the local data source and / or the business data source are queried in a targeted manner, instead of just one data source independently undertaking all query services. In addition, when facing data before and after the update date, the local data source and the business data source will be queried separately, thereby reducing the processing pressure of the local data processing system when facing high-concurrency tasks, thereby reducing the risk of failure of the local data system when facing high-concurrency query services.

[0077] In an exemplary embodiment, referring to Figure 4 , step 02 may specifically include steps 021 to 023, wherein:

[0078] Step 021: If it is determined that the right endpoint of the query time interval is before the update date, the local data source is determined as the target data source.

[0079] Specifically, if the right endpoint of the query time interval is before the update date, it means that the data being queried are all data before the update date, that is, the data to be queried are all stored in the local data source, so the target data source to be queried is the local data source.

[0080] Step 022: If it is determined that the left endpoint or the right endpoint of the query time interval is within the update date, the business data source and the local data source are determined as target data sources.

[0081] Specifically, if the left endpoint or the right endpoint of the query time interval is within the update date, it means that part of the queried data is stored in the local data source, and part of the data is stored in the business data source. That is to say, the target data sources that need to be queried at this time are the local data source and the business data source.

[0082] Step 023: If it is determined that the left endpoint of the query time interval is after the update date, the business data source is determined as the target data source.

[0083] Specifically, if the right endpoint of the query time interval is after the update date, it means that the data being queried is all data after the update date, that is, the data to be queried is stored in the business data source, so the target data source to be queried is the business data source.

[0084] By querying the relationship between the time interval and the date (update date) when the local data source last backed up business data, the target data source is determined, thereby clarifying the specific object (target data source) to be queried; and when the target data source only involves one of the data sources, the query on the target data source does not affect the other data source at all, that is, it does not increase the workload of the other data source; when the target data source includes two data sources at the same time, the task of a query request is performed by the two data sources respectively. The corresponding part of the query can also be reduced, thereby reducing the burden of each of the two data sources, thereby reducing the risk of failure of the local data source when facing high-concurrency query business.

[0085] Furthermore, the server is configured with three units, such as Figure 5As shown, they are respectively a data combination service unit, a local data source query service unit, and a business data source query service unit. The data combination service unit parses the received request message and organizes the query conditions, and determines the target data source to be queried based on step 02, and then calls the local data source query service unit and / or the business data source query service unit based on the query time interval and query conditions to query the target data source to obtain the returned query results; the data combination service unit is also used for cache management, data alignment, response message organization, and pre-processes the query results and feeds them back to the data consumer.

[0086] It should be made clear that when querying the local data source, unified SQL semantics and SQL optimization are constructed through SQL (Structured Query Language) semantic parsing and data source dialect conversion, and data query is implemented by reaching the local data source through connection pool management and JDBC access, and query results can be obtained directly. The business data source needs to be called through the corresponding API (Application Programming Interface). When querying the business data source, the business data source query service unit is called, and the mainstream API service protocol is supported through HTTP (Hyper Text Transfer Protocol) access and RPC (Remote Procedure Call) access functions. Through authentication management, authentication information transmission is realized during the API access process corresponding to the business data source. Through anti-corrosion isolation, environmental adaptation is realized, and the code intrusion of the service module during the adaptation process of different environmental requirements is avoided to the greatest extent; the query time interval and query conditions are constructed into query parameters, and the query parameters are input into the corresponding API interface for query, and the query results are fed back by the API interface.

[0087] like Figure 6 As shown, when the data combination service unit determines that the target data source only includes the local data source, based on the query time interval and query conditions, the local data source query service unit is called to query from the local data source, and the query results obtained are pre-processed and fed back to the data consumer.

[0088] like Figure 7 As shown, when the data combination service unit determines that the target data source only includes the business data source, based on the query time interval and query conditions, the business data source query service unit is called to query the business data source, and the query results obtained are pre-processed and fed back to the data consumer.

[0089] When the data combination service unit determines that the target data source includes a business data source and a local data source, based on the query time interval and query conditions, the business data source query service unit is called to query from the business data source, and the local data source query service unit is called to query from the local data source, and the query results are pre-processed and fed back to the data consumer.

[0090] Further, refer to Figure 8 In step 03, based on the query condition, the query is performed in the target data source to obtain the query result, which may specifically include steps 031 to 033, wherein:

[0091] Step 031: Determine the query type based on the query ID.

[0092] Step 032: If the query type is the first query, the target data source is queried based on the query time interval and the query conditions to obtain the target quantity of target data corresponding to the query conditions, and the business data source and / or local data source are queried based on the query conditions and the target quantity to obtain the query results; the target quantity includes the quantity of target data corresponding to each of the business data source and the local data source.

[0093] Specifically, the request message includes a query ID, and the data combination service unit queries the cache library to determine whether the query ID exists in the cache library. When it is determined that the query ID does not exist in the cache library, the query type is determined to be a first query, and when it is determined that the query ID exists in the cache library, the query type is determined to be a page query.

[0094] Furthermore, for the query request of the first query, the data volume (target quantity), request time and query ID of the target result set corresponding to the first query are written into the cache library; specifically, the query ID is used as the primary key, the data volume of the queried target result set is cached in the cache library of the platform, and a validity period is set for the request time and / or query ID. The validity period can be a set value or a value determined based on the target quantity, wherein the length of the validity period is positively correlated with the size of the target quantity. Among them, the cache library is a cache database built based on Redis (Remote Dictionary Server) or a cache database built based on server memory.

[0095] Step 033: If the query type is a page query, the target quantity determined by the query ID during the first query is called; based on the query conditions and the target quantity, the business data source and / or the local data source is queried to obtain the query result.

[0096] If there are many corresponding data items in the query result set, all target data will not be fed back to the data consumer at once. Instead, the page-turning query logic is adopted to paginate the corresponding target data in the query result set according to the preset paging rules or based on the query conditions of the data consumer, and then the target data corresponding to a certain page is fed back to the data consumer.

[0097] Since the bank's business system generates a lot of data, and the business data is generated in real time, that is, the data in the business data source will be updated in real time; therefore, if the data consumer sends a request message for a page-turning query, it is necessary to re-query the corresponding target data from the business data source or the local data source. In the scheme of the embodiment of the present application, by writing the corresponding target quantity in the first query into the cache queue, the target data for the first query can be locked; when the data consumer performs a page-turning query, the corresponding target data can be found from the target quantity corresponding to the first query, without having to re-traverse the data in the business data source and the local data source; therefore, the embodiment of the present application adopts a method of locking the target quantity cache for the first query, which reduces the amount of data that needs to be queried during page-turning queries, and reduces the query pressure on the business data source and the local data source, thereby supporting the data consumer to query business data with a longer query cycle span.

[0098] In one embodiment, the business data in the business data source occurs during the transaction. Therefore, the business data stored in the business data source and the data backed up to the local data source through the business data source are associated with a transaction time; and in the business data source and the local data source, the business data are sorted and stored in the order of the transaction time. When the query type is the first query, the data combination service unit queries the local data source and the business data source respectively based on the query time interval and the query condition, and then obtains the transaction time and number of data items of the target data corresponding to each of the business data source and the local data source, and then constructs the target data set corresponding to the first query. Then, further, the target data set, query ID and request time corresponding to the first query are written into the cache library. Specifically, the query ID is used as the primary key, the queried target data set is written into the cache library of the platform, and a validity period is set for the request time and / or query ID. The validity period can be a set value or a value determined based on the target quantity, wherein the length of the validity period is positively correlated with the size of the target quantity. Among them, the cache library is a cache database built based on Redis or a cache database built based on server memory.

[0099] Furthermore, if Fig. 9As shown, in step 032 and step 033, based on the query condition and the target quantity, the business data source and / or the local data source is queried to obtain the query result, which may specifically include step 03A1 and step 03A2, wherein:

[0100] Step 03A1. Determine the query end row according to the number of target data corresponding to the initial data source, the query order, the query start row and the query quantity; the query end row and the query start row both represent the time sequence of the data in the data source; the initial data source is the data source corresponding to the query start row;

[0101] Step 03A2: Based on the query start row and the query end row, query the business data source and / or the local data source to obtain the query result.

[0102] Specifically, the query conditions include the query order, the query start row, and the query quantity. Fig.10 As shown, for the query ID of the first query, the local data source is queried based on the query conditions, as shown in path A1; the business data source is queried based on the query conditions, as shown in path A2; and finally the target data set is obtained.

[0103] The sum of the number of target data items in the local data source and the business data source is the number of target data items in the target data set, that is, the target quantity. When the target quantity is greater than the number of items filled in a single page (the number of queries), the target data is fed back to the data consumer in the form of paging. Specifically, the query result fed back by the first query is the number of requests in the query conditions at the time of the first query. Specifically, if the query conditions entered by the user do not include the query start row and the query quantity, the base start row is used as the query start row, and the base quantity is used as the query quantity. For example, if the query conditions do not include the query start row and the query quantity, the default query start row is 1 and the query quantity is 10.

[0104] Specifically, the query order is positive or reverse order, where positive order means the order from earliest to latest transaction time; reverse order means the order from latest to earliest transaction time. The query start row represents the transaction time of the business data, that is, the order of its position in the target data set according to the query order. The data source where the query start row is located is the initial data source, which can be a local data source or a business data source. The query end row is further determined based on the query start row and the query quantity, where the query end row = query start row + query quantity - 1.

[0105] If the query order in the query condition is in reverse order, the query is performed in the order of business data source first and local data source second; if the query order in the query condition is in forward order, the query is performed in the order of local data source first and business data source second. The business data corresponding to the transaction time between the query start row and the query end row is the query result.

[0106] If the query start row and the query end row are located in different data sources, the target data of the first part is determined from the initial data source according to the query order based on the query start row and the query quantity; and the corresponding data is supplemented from the data source where the query end row is located according to the query order and the query end row, and the insufficient data from the data source where the query end row is located is the target data of the second part. The target data of the first part and the second part are the query results, and the number of the target data of the first part and the second part is equal to the query quantity.

[0107] In one of the embodiments, if the number of target data items corresponding to the business data source is greater than or equal to the query end row, the target data from the query start row to the query end row is queried in reverse order from the business data source as the query result; if the number of target data items corresponding to the business data source is less than the query start row, the target data from (query start row - number of target data items from the business data source) to (query end row - number of target data items from the business data source) is queried in reverse order from the local data source as the query result; if the number of target data items corresponding to the business data source is greater than or equal to the query start row but less than the query end row, all target data starting from the query start row is queried in reverse order from the business data source as the query result, and the target data from row 1 to (query end row - number of target data items from the business data source) is queried in reverse order from the local data source as a supplement to the query result.

[0108] Specifically, when searching by page, if there is no query order in the query condition, the query order of the first query is still followed. Fig.10 When both the query start row and the query end row are located in the business data source, path B1 is executed, and only the business data source is queried to obtain the target data as the query result.

[0109] When the query start row and the query end row are located in different data sources, execute path B2, query the business data source to obtain the first part of the target data, query the local data source to obtain the second part of the target data, and splice the first part of the target data and the second part of the target data into one page as the query result and feedback to the data consumer.

[0110] When both the query start row and the query end row are located in the local data source, path B3 is executed to query only the local data source to obtain the target data as the query result.

[0111] In one example; if the target number of data obtained by the first query is 24, the order of the storage address is 24, then the target data set includes 24 transaction times arranged in ascending or descending order, and each transaction time corresponds to a target data. Among them, the local data source includes 11 target data, which are sorted 1-11, and the local data source includes 13 items sorted 12-24. If the query start line is 1, the query quantity is 8. The initial data source is the local data source, and the query start line and the query end line are both located in the local data source, and the query end line is 8; then the query result corresponding to the first query is the target data corresponding to the serial numbers 1-8, that is, it is completely the target data in the business data, that is, the first query only needs to execute path B1, and query the target data corresponding to the serial numbers 1-8 in the business data source as the query result.

[0112] Furthermore, when turning pages for query, if the query start line is 6 and the query quantity is 8, that is, the initial data source is the local data source, and the query end line is 13, then the query end line is located in the business data source, then the query result of this time corresponds to the target data corresponding to sequence numbers 6-11 and sequence numbers 12-13; the query data corresponding to the query result of this time includes both the data in the business data source and the data in the local data source, that is, path B2 needs to be executed.

[0113] Furthermore, when turning pages for query, if the query start behavior is 14 and the query quantity is 6, the initial data source is the business data source, and the query end behavior is 19, and the query end row is also located in the business data source. At this time, the query result is the target data corresponding to the serial numbers 14-19, and the execution path B3.

[0114] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a data query device for implementing the data query method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data query device embodiments provided below can refer to the limitations on the data query method above, and will not be repeated here.

[0116] In an exemplary embodiment, Fig.11 As shown, a data query device is provided, including a receiving module 1101, a target data source determination module 1102, a query module 1103 and a preprocessing module 1104, wherein:

[0117] The receiving module 1101 is used to receive a request message sent by a data consumer, where the request message includes at least a query time interval and a query condition;

[0118] The target data source determination module 1102 is used to determine the target data source based on the relationship between the query time interval and the update date of the local data source, where the target data source is at least one of the business data source and the local data source; wherein at the beginning of each synchronization cycle, the business data source backs up the business data of the previous synchronization cycle to the local data source, and the update date is the date of the last backup of the business data to the local data source;

[0119] A query module 1103 is used to query the target data source based on the query condition to obtain the query result;

[0120] The preprocessing module 1104 is used to preprocess the query results, generate a response message from the preprocessed query results, and feed back the response message to the data consumer.

[0121] In the above-mentioned data query device, the data between the local data source and the business data source are not updated synchronously in real time. Instead, at the end of each synchronization cycle, the business data source migrates the stored business data to the local data source, thereby reducing the burden of the local data source to write data in real time, thereby reducing the risk of failure; the local data source stores the business data of the ended synchronization cycle, and the business data source stores the business data in the current synchronization cycle. Therefore, when facing a query request, the target data source is determined based on the query time interval and the order of the update date of the last backup of the business data by the local data source, and then the local data source and / or the business data source are queried in a targeted manner, instead of just one data source independently undertaking all query services; and, when facing data before and after the update date, the local data source and the business data source will be queried separately, thereby reducing the processing pressure of the local data processing system when facing high-concurrency tasks, thereby reducing the risk of failure of the local data system when facing high-concurrency query services.

[0122] In one embodiment, the target data source determination module 1102 is specifically configured to:

[0123] If it is determined that the right end point of the query time interval is before the update date, the local data source is determined as the target data source;

[0124] If it is determined that the left end point or the right end point of the query time interval is within the update date, the business data source and the local data source are determined as the target data source;

[0125] If it is determined that the left end point of the query time interval is after the update date, the business data source is determined as the target data source.

[0126] In one embodiment, the query module 1103 is specifically used to:

[0127] Determine the query type corresponding to the request message;

[0128] If the query type is the first query, the target data source is queried based on the query time interval and query conditions to obtain the target number of target data corresponding to the query conditions, and the business data source and / or local data source are queried based on the query conditions and the target number to obtain the query results; the target number includes the number of target data corresponding to each of the business data source and the local data source.

[0129] In one embodiment, the query module 1103 is specifically used to:

[0130] Determine the query type corresponding to the request message;

[0131] If the query type is a page query, the target quantity determined by the query ID during the first query is called;

[0132] Based on the query conditions and the target quantity, the business data source and / or the local data source are queried to obtain the query results.

[0133] In one embodiment, the query module 1103 is further configured to:

[0134] When it is determined that the query ID does not exist in the cache library, the query type is determined to be a first query, and the request time, target quantity and query ID are written into the cache library;

[0135] When it is determined that the query ID exists in the cache library, the query type is determined to be a page-turning query.

[0136] In one embodiment, the query conditions include a query order, a query start row, and a query quantity, and the query module 1103 is further used to:

[0137] Determine the query end row according to the number of target data corresponding to the initial data source, the query order, the query start row and the query quantity; the query end row and the query start row both represent the time sequence of the data in the data source; the initial data source is the data source corresponding to the query start row;

[0138] Based on the query start row and the query end row, the business data source and / or the local data source is queried to obtain the query result.

[0139] In one embodiment, the device further includes a benchmark determination module, specifically configured to:

[0140] If the query conditions entered by the user do not include the query start row and query quantity, the base start row is used as the query start row, and the base quantity is used as the query quantity.

[0141] Each module in the above data query device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a server in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0142] In an exemplary embodiment, a server is provided. The server may be a terminal, and its internal structure diagram may be as follows: Fig.12As shown. The server includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the server is used to provide computing and control capabilities. The memory of the server includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the server is used to exchange information between the processor and the external device. The communication interface of the server is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data query method is realized. The display unit of the server is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the server can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the server shell, or an external keyboard, touchpad or mouse.

[0143] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the server to which the solution of the present application is applied. The specific server may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0144] In an exemplary embodiment, a server is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0145] Receive a request message sent by a data consumer, where the request message includes at least a query time interval and a query condition;

[0146] Based on the relationship between the query time interval and the update date of the local data source, determine the target data source, the target data source is at least one of the business data source and the local data source; wherein, at the beginning of each synchronization cycle, the business data source backs up the business data of the previous synchronization cycle to the local data source, and the update date is the date of the last backup of the business data to the local data source;

[0147] Query the target data source based on the query conditions to obtain the query results;

[0148] The query results are preprocessed, a response message is generated from the preprocessed query results, and the response message is fed back to the data consumer.

[0149] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0150] Receive a request message sent by a data consumer, where the request message includes at least a query time interval and a query condition;

[0151] Based on the relationship between the query time interval and the update date of the local data source, determine the target data source, the target data source is at least one of the business data source and the local data source; wherein, at the beginning of each synchronization cycle, the business data source backs up the business data of the previous synchronization cycle to the local data source, and the update date is the date of the last backup of the business data to the local data source;

[0152] Query the target data source based on the query conditions to obtain the query results;

[0153] The query results are preprocessed, a response message is generated from the preprocessed query results, and the response message is fed back to the data consumer.

[0154] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0155] Receive a request message sent by a data consumer, where the request message includes at least a query time interval and a query condition;

[0156] Based on the relationship between the query time interval and the update date of the local data source, determine the target data source, the target data source is at least one of the business data source and the local data source; wherein, at the beginning of each synchronization cycle, the business data source backs up the business data of the previous synchronization cycle to the local data source, and the update date is the date of the last backup of the business data to the local data source;

[0157] Query the target data source based on the query conditions to obtain the query results;

[0158] The query results are preprocessed, a response message is generated from the preprocessed query results, and the response message is fed back to the data consumer.

[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, data source or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The data source involved in each embodiment provided in this application may include at least one of a relational data source and a non-relational data source. Non-relational data sources may include distributed data sources based on blockchain, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but are not limited to this.

[0161] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data query method, characterized in that: Applied to a server, the method comprises: Receive a request message sent by a data consumer, wherein the request message includes at least a query time interval and a query condition; Based on the relationship between the query time interval and the update date of the local data source, determine the target data source, the target data source is at least one of the business data source and the local data source; wherein, at the beginning of each synchronization cycle, the business data source backs up the business data of the previous synchronization cycle to the local data source, and the update date is the date of the business data last backed up to the local data source; Query the target data source based on the query condition to obtain a query result; The query result is preprocessed, a response message is generated from the preprocessed query result, and the response message is fed back to the data consumption end.

2. The method according to claim 1, characterized in that The determining of the target data source based on the relationship between the query time interval and the update date includes: If it is determined that the right end point of the query time interval is before the update date, the local data source is determined as the target data source; If it is determined that the left end point or the right end point of the query time interval is within the update date, the business data source and the local data source are determined as target data sources; If it is determined that the left end point of the query time interval is after the update date, the business data source is determined as the target data source.

3. The method according to claim 2, characterized in that The request message also includes a query ID, and the query is performed in the target data source based on the query condition to obtain a query result, including: Determining a query type based on the query ID; If the query type is the first query, the target data source is queried based on the query time interval and the query conditions to obtain the target quantity of target data corresponding to the query conditions, and the business data source and / or the local data source are queried based on the query conditions and the target quantity to obtain the query results; the target quantity includes the quantity of target data corresponding to each of the business data source and the local data source.

4. The method according to claim 3, characterized in that The method further comprises: If the query type is a page-turning query, the target quantity determined by the query ID during the first query is called; The business data source and / or the local data source is queried based on the query condition and the target quantity to obtain a query result.

5. The method according to any one of claims 3 or 4, characterized in that: The request message also includes a request time, and the determining of the query type based on the query ID includes: When it is determined that the query ID does not exist in the cache library, determining the query type is a first query, and writing the request time, the target quantity and the query ID into the cache library; When it is determined that the query ID exists in the cache library, the query type is determined to be a page-turning query.

6. The method according to claim 3 or 4, characterized in that: The query conditions include a query order, a query start row, and a query quantity, and the querying the business data source and / or the local data source based on the query conditions and the target quantity to obtain a query result includes: Determine the query end row according to the number of target data corresponding to the initial data source, the query order, the query start row and the query quantity; the query end row and the query start row both represent the time sequence of the data in the data source; the initial data source is the data source corresponding to the query start row; Based on the query start row and the query end row, the business data source and / or the local data source is queried to obtain a query result.

7. The method according to claim 6, characterized in that The method further comprises: If the query condition input by the user does not include a query start row and a query quantity, the reference start row is used as the query start row, and the reference quantity is used as the query quantity.

8. A data query device, characterized in that: The device comprises a receiving module, a target data source determination module, a query module and a preprocessing module, wherein: A receiving module, used to receive a request message sent by a data consumer, wherein the request message includes at least a query time interval and a query condition; A target data source determination module is used to determine the target data source based on the relationship between the query time interval and the update date of the local data source, wherein the target data source is at least one of the business data source and the local data source; wherein, at the beginning of each synchronization cycle, the business data source backs up the business data of the previous synchronization cycle to the local data source, and the update date is the date of the last backup of the business data to the local data source; A query module, used to query the target data source based on the query condition to obtain a query result; The preprocessing module is used to preprocess the query result, generate a response message from the preprocessed query result, and feed back the response message to the data consumption end.

9. A server, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images