A data query method and device, electronic equipment and computer readable medium

Abstract

This application discloses a data query method, apparatus, electronic device, and computer-readable medium, relating to the field of big data processing technology. The method includes: receiving a data query request and obtaining a corresponding time identifier and database identifier; determining the corresponding number of partitions based on the database identifier, and then performing modulo operation based on the time identifier and the number of partitions to obtain a query partition identifier; responding to a change in the time identifier, determining a first historical partition from the historical partitions corresponding to the number of partitions and determining a second historical partition corresponding to the query partition identifier, and copying statistical information corresponding to the first historical partition to the second historical partition; generating a query statement based on the query partition identifier, accessing the second historical partition after copying the statistical information, and then executing the query statement to return query results. This achieves optimal query efficiency while improving the accuracy of data queries.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a data query method, apparatus, electronic device, and computer-readable medium. Background Technology

[0002] As the number of users handling a particular service increases and the service is fully integrated with the credit reporting system, the related data grows exponentially, necessitating more refined database management. Due to the poor performance of online transactions for large-scale data queries, it's necessary to partition the database tables containing large amounts of data. Existing methods for creating new partitions simply add a new partition to the database table. However, since the data volume in this partition is still relatively small, if the database collects statistical information at this time, a situation can arise where the data volume surges dramatically over a period of time while the statistical data recorded is still small. This can lead to frequent changes in the execution plan of database query statements, resulting in low accuracy of data queries. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data query method, apparatus, electronic device, and computer-readable medium, which can solve the problem of low data query efficiency and low accuracy caused by frequent changes in the execution plan of existing database SQL statements.

[0004] To achieve the above objectives, according to one aspect of the embodiments of this application, a data query method is provided, comprising:

[0005] Receive data query requests and obtain the corresponding time identifier and database identifier;

[0006] Based on the database identifier, determine the corresponding number of partitions, and then perform modulo operation based on the time identifier and the number of partitions to obtain the query partition identifier;

[0007] In response to a change in the time stamp, the first historical partition is determined from the historical partitions corresponding to the number of partitions, and the second historical partition corresponding to the query partition identifier is determined. The statistical information corresponding to the first historical partition is copied to the second historical partition.

[0008] A query statement is generated based on the query partition identifier, the second historical partition after copying the statistics is accessed, the query statement is executed, and the query result information is returned.

[0009] Optionally, before copying the statistics corresponding to the first historical partition to the second historical partition, the method further includes:

[0010] Based on the time identifier and the number of partitions, determine the identifier of the partition to be cleaned;

[0011] Clear the data in the partition corresponding to the partition identifier to be cleaned in the history partition.

[0012] Optionally, based on the time identifier and the number of partitions, the identifier of the partition to be cleaned is determined, including:

[0013] Increment the time stamp by 1, and then perform a modulo operation with the number of partitions to obtain the identifier of the partition to be cleaned.

[0014] Optionally, the first historical partition is determined from the historical partitions corresponding to the number of partitions, including:

[0015] Based on the time identifier, obtain the statistical information of the corresponding historical partition;

[0016] Calculate the average value of the statistical information of the historical partitions, and then determine the first historical partition in the corresponding historical partitions based on the average value.

[0017] Optionally, based on the time identifier, obtain the statistical information of the corresponding historical partition, including:

[0018] The system updates the statistical information of historical partitions corresponding to the number of partitions in real time, and then obtains the statistical information of historical partitions at a preset time interval before the time marker to determine the statistical information of the historical partition corresponding to the time marker.

[0019] Optionally, the average of the statistics for historical partitions is calculated, including:

[0020] Determine the amount of statistical information for the historical partitions corresponding to the time stamps and the number of historical partitions corresponding to the time stamps;

[0021] The average value of the data volume is calculated based on the data quantity and number of data points.

[0022] Optionally, the first historical partition in the corresponding historical partition is determined based on the average value, including:

[0023] Based on the average value, determine the Mahalanobis distance of each data point in the statistics of the historical partition corresponding to the time stamp;

[0024] Based on Mahalanobis distance, the historical partition whose statistical information is closest to the average value corresponding to the time stamp is identified as the first historical partition.

[0025] In addition, this application also provides a data query device, including:

[0026] The receiving unit is configured to receive data query requests and obtain the corresponding time identifier and database identifier;

[0027] The modulo unit is configured to determine the corresponding number of partitions based on the database identifier, and then perform modulo processing based on the time identifier and the number of partitions to obtain the query partition identifier;

[0028] The information replication unit is configured to, in response to a change in the time stamp, determine the first historical partition from the historical partitions corresponding to the number of partitions and determine the second historical partition corresponding to the query partition identifier, and copy the statistical information corresponding to the first historical partition to the second historical partition.

[0029] The data query unit is configured to generate a query statement based on the query partition identifier, access the second historical partition after copying the statistical information, execute the query statement, and return the query result information.

[0030] Specifically, the data query device also includes a data cleaning unit, configured to:

[0031] Based on the time identifier and the number of partitions, determine the identifier of the partition to be cleaned;

[0032] Clear the data in the partition corresponding to the partition identifier to be cleaned in the history partition.

[0033] Specifically, the data cleaning unit is further configured as follows:

[0034] Increment the time stamp by 1, and then perform a modulo operation with the number of partitions to obtain the identifier of the partition to be cleaned.

[0035] Specifically, the information replication unit is further configured as follows:

[0036] Based on the time identifier, obtain the statistical information of the corresponding historical partition;

[0037] Calculate the average value of the statistical information of the historical partitions, and then determine the first historical partition in the corresponding historical partitions based on the average value.

[0038] Specifically, the information replication unit is further configured as follows:

[0039] The system updates the statistical information of historical partitions corresponding to the number of partitions in real time, and then obtains the statistical information of historical partitions at a preset time interval before the time marker to determine the statistical information of the historical partition corresponding to the time marker.

[0040] Specifically, the information replication unit is further configured as follows:

[0041] Determine the amount of statistical information for the historical partitions corresponding to the time stamps and the number of historical partitions corresponding to the time stamps;

[0042] The average value of the data volume is calculated based on the data quantity and number of data points.

[0043] Specifically, the information replication unit is further configured as follows:

[0044] Based on the average value, determine the Mahalanobis distance of each data point in the statistics of the historical partition corresponding to the time stamp;

[0045] Based on Mahalanobis distance, the historical partition whose statistical information is closest to the average value corresponding to the time stamp is identified as the first historical partition.

[0046] In addition, this application also provides a data query electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the data query method described above.

[0047] In addition, this application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data query method described above.

[0048] To achieve the above objectives, according to another aspect of the embodiments of this application, a computer program product is provided.

[0049] A computer program product according to an embodiment of this application includes a computer program that, when executed by a processor, implements the data query method provided in an embodiment of this application.

[0050] One embodiment of the above invention has the following advantages or beneficial effects: This application receives a data query request, obtains the corresponding time identifier and database identifier; determines the corresponding number of partitions based on the database identifier, and then performs modulo processing based on the time identifier and the number of partitions to obtain the query partition identifier; in response to a change in the time identifier, determines the first historical partition from the historical partitions corresponding to the number of partitions and determines the second historical partition corresponding to the query partition identifier, copies the statistical information corresponding to the first historical partition to the second historical partition; generates a query statement based on the query partition identifier, accesses the second historical partition after copying the statistical information, and then executes the query statement to return query result information. This ensures that the index does not become invalid when executing the query statement, and the execution plan of the query statement does not change frequently, thereby improving the accuracy of data query while achieving optimal query efficiency.

[0051] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0052] The accompanying drawings are provided to better understand this application and do not constitute an undue limitation thereof. Wherein:

[0053] Figure 1This is a schematic diagram illustrating the main flow of a data query method according to an embodiment of this application;

[0054] Figure 2 This is a schematic diagram illustrating the main flow of a data query method according to an embodiment of this application;

[0055] Figure 3 This is a schematic diagram of the main flow of a data query method according to an embodiment of this application;

[0056] Figure 4 This is a schematic diagram of the main flow of a data query method according to an embodiment of this application;

[0057] Figure 5 This is a schematic diagram of the process of cleaning up old data partitions according to an embodiment of the data query method of this application;

[0058] Figure 6 This is a schematic diagram of the statistical information copying process of a data query method according to an embodiment of this application;

[0059] Figure 7 This is a schematic diagram of the main units of a data query device according to an embodiment of this application;

[0060] Figure 8 This is an exemplary system architecture diagram to which embodiments of this application can be applied;

[0061] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers in the embodiments of this application. Detailed Implementation

[0062] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with relevant national laws and regulations.

[0063] Figure 1 This is a schematic diagram of the main flow of a data query method according to an embodiment of this application, as shown below. Figure 1 As shown, the data query methods include:

[0064] Step S101: Receive a data query request and obtain the corresponding time identifier and database identifier.

[0065] In this embodiment, the execution entity of the data query method (e.g., a server) can receive data query requests via wired or wireless connections. The execution entity can obtain the time identifier carried in the data query request. The time identifier can be, for example, the month initiating the data query request, such as June, July, or August. Alternatively, the time identifier can be a quarter or year; this embodiment does not limit the specific content of the time identifier.

[0066] The executing entity can also obtain the database identifier carried in the data query request. This database identifier can be, for example, a database table name, a database address, etc. This application embodiment does not specifically limit the database identifier. The database identifier can be used to determine the database currently needed, so that data queries can be performed based on the determined database.

[0067] Step S102: Determine the corresponding number of partitions based on the database identifier, and then perform modulo operation based on the time identifier and the number of partitions to obtain the query partition identifier.

[0068] The executing entity can locate the corresponding database based on the database identifier and obtain the number of partitions into which the tables in the located database are divided, such as... Figure 3 As shown, for example, there are n partitions. Then, the executing entity can perform a modulo operation based on the time identifier (e.g., the current month y) and the number of partitions (e.g., n) to obtain the modulo r (this modulo r is the partition identifier being queried, for example, the partition number corresponding to the partition being accessed for data query), for example, r = y % n. Wherein, the value range of the month y is: 0... <y<12。

[0069] In step S103, in response to the change of time stamp, the first historical partition is determined from the historical partitions corresponding to the number of partitions, and the second historical partition corresponding to the query partition stamp is determined. The statistical information corresponding to the first historical partition is copied to the second historical partition.

[0070] In this embodiment, a change in the time identifier refers to a change in the month corresponding to the current data query request compared to the month corresponding to the previous data query request. For example, if the current data query request corresponds to June, and the most recent previous data query request corresponds to May, then the executing entity can determine that the time identifier has changed. The statistical information copying process is only required when the time identifier changes. This statistical information copying process is an initialization operation when querying data in a different month, and it is not required for every data query.

[0071] The statistical information replication process includes: determining the first historical partition from the historical partitions corresponding to the number of partitions corresponding to the database identifier and determining the second historical partition corresponding to the query partition identifier, and replicating the statistical information corresponding to the first historical partition to the second historical partition.

[0072] For example, the execution entity first needs to calculate the average of the statistical information of all historical partitions over the past m months for the current month. The first historical partition refers to the historical partition corresponding to the statistical information whose data volume best matches the average value μ. "Best match" means that the data volume corresponding to the statistical information of the historical partition is the same as or close to the average value μ. "Closest" means that the difference between the data volume corresponding to the statistical information of the historical partition and the average value μ is the smallest. The second historical partition refers to the historical partition corresponding to the query partition identifier, that is, the historical partition currently needed among the various historical partitions corresponding to the database identifier.

[0073] Specifically, determining the first historical partition from the historical partitions corresponding to the number of partitions includes: obtaining the statistical information of the corresponding historical partitions based on the time identifier; calculating the average value of the statistical information of the historical partitions, and then determining the first historical partition from the corresponding historical partitions based on the average value.

[0074] Specifically, based on the time identifier, the statistical information of the corresponding historical partition is obtained, including: real-time updating the statistical information of the historical partition corresponding to the number of partitions, and then obtaining the statistical information of the historical partitions within a preset time interval (e.g., m months) before the time identifier, so as to determine the statistical information of the historical partition corresponding to the time identifier (e.g., the current month). This application embodiment does not specifically limit the value of m.

[0075] Specifically, calculating the average value of the statistical information for historical partitions includes: determining the data volume of the statistical information for the historical partitions corresponding to the time markers and the number of historical partitions corresponding to the time markers; and calculating the average value of the data volume based on the data volume and the number. Specifically, the data volumes of each historical partition can be summed, and the ratio of this sum to the number of historical partitions can be calculated to obtain the average value of the data volume.

[0076] For example, the average value μ refers to the sum of the data volume p corresponding to the statistical information of all historical partitions in the m months prior to the current month, expressed in units of rows, i.e., μ = p / s, where s is the number of all historical partitions corresponding to the database identifier of the data query request. The historical partition whose statistical data volume is closest to this average value μ among all historical partitions in the past m months of the current month will be identified and designated as the first historical partition. The statistical information includes the data volume and data distribution information of the corresponding partition.

[0077] Specifically, determining the first historical partition within the corresponding historical partition based on the average value includes: determining the Mahalanobis distance of each data point in the statistics of the historical partition corresponding to the time identifier based on the average value; for example, the Mahalanobis distance formula is shown below:

[0078]

[0079] Where Σ is the covariance matrix of the multidimensional random variables, and μ is the sample mean. If the covariance matrix is ​​a unit vector, that is, each dimension is independent and identically distributed, the Mahalanobis distance becomes the Euclidean distance.

[0080] Based on Mahalanobis distance, the historical partition whose statistical information is closest to the average value corresponding to the time stamp is identified and thus designated as the first historical partition. Specifically, as follows... Figure 6 As shown, firstly, the average value μ of the historical partition statistical information data volume is calculated. Then, the Mahalanobis distance of a single data point is obtained through an algorithm. Based on the obtained Mahalanobis distance of a single data point, the data volume of statistical information with large deviations is filtered out, and the partition number of the historical partition closest to the average value μ is obtained, which is the partition number of the first historical partition. The statistical information of the first historical partition corresponding to the partition number of the first historical partition is used as the benchmark statistical information J. Then, the benchmark statistical information J is copied to the currently used partition, that is, copied to the second historical partition, so as to achieve the purpose of stabilizing the execution plan corresponding to the query statement.

[0081] Step S104: Generate a query statement based on the query partition identifier, access the second historical partition after copying the statistics, execute the query statement, and return the query result information.

[0082] like Figure 3 As shown, after obtaining the query partition identifier (i.e., the partition number) in step S102, the execution entity can concatenate the query partition identifier into the SQL statement to generate a query statement. Then, the query statement is executed to query the database and obtain the query results. Next, the second historical partition, which replicates the statistical information from the first historical partition, is accessed to execute the query statement, obtain the corresponding query results, and output the query results.

[0083] This embodiment receives a data query request and obtains the corresponding timestamp and database identifier. Based on the database identifier, it determines the corresponding number of partitions, and then performs a modulo operation based on the timestamp and the number of partitions to obtain the query partition identifier. In response to a change in the timestamp, it determines the first historical partition from the historical partitions corresponding to the number of partitions and the second historical partition corresponding to the query partition identifier. It then copies the statistical information corresponding to the first historical partition to the second historical partition. Based on the query partition identifier, it generates a query statement, accesses the second historical partition after copying the statistical information, executes the query statement, and returns the query results. This ensures that the index does not become invalid when executing the query statement, and the execution plan of the query statement does not change frequently, thereby improving the accuracy of data queries while achieving optimal query efficiency.

[0084] Figure 2 This is a schematic diagram of the main flow of a data query method according to an embodiment of this application, as follows: Figure 2 As shown, the data query methods include:

[0085] Step S201: Receive a data query request and obtain the corresponding time identifier and database identifier.

[0086] Step S202: Determine the corresponding number of partitions based on the database identifier, and then perform modulo operation based on the time identifier and the number of partitions to obtain the query partition identifier.

[0087] Step S203: Determine the identifier of the partition to be cleaned based on the time identifier and the number of partitions.

[0088] like Figure 5 As shown, the executing entity can obtain the current time T, which is the current month T, and then periodically clean up the data in the partition corresponding to the modulo of T-2 months each month. This ensures that each partition retains only one month's data, and the entire database table retains only two months' data at any given time.

[0089] Specifically, based on the time stamp and the number of partitions, the identifier of the partition to be cleaned is determined, including: incrementing the time stamp by 1, and then performing a modulo operation with the number of partitions to obtain the identifier of the partition to be cleaned. For example... Figure 5 As shown, the time identifier, such as T, represents the current month, and n is the number of partitions. By taking the modulo between T+1 and n, the partition identifier to be cleaned is obtained, then the partition identifier to be cleaned is l = (T+1)%n. This allows us to clean the data in partition (T+1)%n, where the data from month T-2 is stored.

[0090] Step S204: Clear the data in the partition corresponding to the partition identifier to be cleaned in the historical partition.

[0091] Once the partition to be cleaned is identified, the executing entity can delete the data in the partition corresponding to that partition. This ensures that as much valid data as possible is retained in the partition, reducing data storage pressure, freeing up data storage space, and improving data query speed.

[0092] In step S205, in response to the change of time stamp, the first historical partition is determined from the historical partitions corresponding to the number of partitions, and the second historical partition corresponding to the query partition stamp is determined. The statistical information corresponding to the first historical partition is copied to the second historical partition.

[0093] The time identifier can be, for example, a month. For instance, a change in the time identifier could mean that the month corresponding to the current data query request is different from the month corresponding to the previous data query request; in other words, the month for the data query has changed.

[0094] This application embodiment saves the data volume and corresponding statistical information of each historical partition as learning data for a machine learning model. The program is implemented using the PyOD framework and utilizes the Mahalanobis distance algorithm to maintain and analyze the statistical information of each historical partition, removing outliers. By aggregation, relatively stable and average statistical information is found in the historical partitions. Statistical information with a data volume close to the average value μ (i.e., the average data volume, unit: records) is copied to the currently used historical partition. Based on the changing trend of the data volume of historical partitions, the statistical information of the historical partitions that best matches the average value μ is continuously selected using the moving average method. When switching partitions (i.e., when the time stamp changes, i.e., when the month of the data query changes), the statistical information of the historical partition that dynamically best matches the average value μ is automatically copied to the currently used historical partition (the currently used historical partition is obtained by modulo the time stamp and the number of partitions). This prevents the index from becoming invalid due to frequent changes in statistical information, achieving optimal query efficiency.

[0095] Step S206: Generate a query statement based on the query partition identifier, access the second historical partition after copying the statistics, execute the query statement, and return the query result information.

[0096] The principles of steps S205 and S206 are similar to those of steps S103 and S104, and will not be repeated here.

[0097] In this embodiment, the implementation program in the execution body takes the monthly modulo of the received data query request to return the query partition identifier of the partition where the data is located. It then automatically generates an SQL statement to find the required data in the partition corresponding to that query partition identifier, connects to the database, queries the data, retrieves the returned data, and terminates the program. After cleaning the partitions, this embodiment can copy the most suitable statistical information to the currently used partition, preventing the index from becoming invalid and thus achieving optimal query efficiency.

[0098] This application embodiment partitions the database tables according to conditions such as time, ensuring that the data accessed by the program is always within a single partition, reducing the frequency of cross-partition queries involving large amounts of data. According to another aspect of this application embodiment, machine learning methods are used to maintain and statistically analyze the partition information. Statistical information with a data volume close to the average value μ (i.e., the average data volume, unit: records) is copied to the currently used historical partition. Based on the changing trend of the data volume in historical partitions, and combined with the moving average method, the statistical information of the historical partition that best matches the average value μ is continuously selected. When switching partitions (i.e., when the time stamp changes, i.e., when the month of the data query changes), the statistical information of the historical partition that dynamically best matches the average value μ is automatically copied to the currently used historical partition (the currently used historical partition is obtained by modulo the time stamp and the number of partitions). This prevents the index from becoming invalid due to frequent changes in statistical information, achieving optimal query efficiency.

[0099] Figure 4 This is a schematic diagram of the main flow of a data query method according to an embodiment of this application. Figure 4 As shown, by partitioning the tables in the database to obtain partition 1, partition 2, partition 3, and partition 4, and storing data in the corresponding partitions by taking the month modulo, the data for the same month is always stored in the same partition (e.g., partition 1) during data write operations. Therefore, data queried by the same or multiple transactions, after being processed by the partition location program and concatenated into SQL query statements, can always find the required information in the same partition (e.g., partition 1). Here, "the same or multiple transactions" refers to one or more query transactions performed within the same month.

[0100] Figure 7 This is a schematic diagram of the main units of a data query device according to an embodiment of this application. For example... Figure 7 As shown, the data query device 700 includes a receiving unit 701, a model taking unit 702, an information copying unit 703, and a data query unit 704.

[0101] The receiving unit 701 is configured to receive data query requests and obtain the corresponding time identifier and database identifier.

[0102] Modulo unit 702 is configured to determine the corresponding number of partitions based on the database identifier, and then perform modulo processing based on the time identifier and the number of partitions to obtain the query partition identifier.

[0103] The information replication unit 703 is configured to, in response to a change in the time stamp, determine the first historical partition from the historical partitions corresponding to the number of partitions and determine the second historical partition corresponding to the query partition identifier, and copy the statistical information corresponding to the first historical partition to the second historical partition.

[0104] The data query unit 704 is configured to generate a query statement based on the query partition identifier, access the second historical partition after copying the statistical information, execute the query statement, and return the query result information.

[0105] In some embodiments, the apparatus further includes Figure 7 The data cleaning unit (not shown) is configured to: determine the partition identifier to be cleaned based on the time identifier and the number of partitions; and clear the data in the partition corresponding to the partition identifier to be cleaned in the historical partitions.

[0106] In some embodiments, the data cleaning unit is further configured to: increment the time identifier by 1, and then perform a modulo operation with the number of partitions to obtain the identifier of the partition to be cleaned.

[0107] In some embodiments, the information copying unit 703 is further configured to: obtain statistical information of the corresponding historical partition based on the time identifier; calculate the average value of the statistical information of the historical partition, and then determine the first historical partition in the corresponding historical partition based on the average value.

[0108] In some embodiments, the information copying unit 703 is further configured to: update the statistical information of the historical partitions corresponding to the number of partitions in real time, and then obtain the statistical information of the historical partitions at a preset time interval before the time stamp, so as to determine the statistical information of the historical partitions corresponding to the time stamp.

[0109] In some embodiments, the information copying unit 703 is further configured to: determine the amount of statistical information of the historical partition corresponding to the time stamp and the number of historical partitions corresponding to the time stamp; and calculate the average amount of data based on the amount of data and the number of historical partitions.

[0110] In some embodiments, the information copying unit 703 is further configured to: determine the Mahalanobis distance of each data point in the statistical information of the historical partition corresponding to the time stamp based on the average value; and determine the partition in the historical partition corresponding to the time stamp whose statistical information is closest to the average value based on the Mahalanobis distance, and identify it as the first historical partition.

[0111] It should be noted that the data query method and data query device in this application are related in terms of specific implementation content, so repeated content will not be described again.

[0112] Figure 8 An exemplary system architecture 800 is shown that can be applied to the data query method or data query apparatus of the embodiments of this application.

[0113] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, and 803, a network 804, and a server 805. Network 804 serves as the medium for providing communication links between terminal devices 801, 802, and 803 and server 805. Network 804 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0114] Users can use terminal devices 801, 802, and 803 to interact with server 805 via network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0115] Terminal devices 801, 802, and 803 can be various electronic devices with data query and processing screens and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0116] Server 805 can be a server providing various services, such as a backend management server supporting data query requests submitted by users using terminal devices 801, 802, and 803 (this is just an example). The backend management server can receive data query requests, obtain the corresponding timestamp and database identifier; based on the database identifier, determine the corresponding number of partitions, and then perform modulo operation based on the timestamp and the number of partitions to obtain the query partition identifier; in response to a change in the timestamp, determine the first historical partition from the historical partitions corresponding to the number of partitions and determine the second historical partition corresponding to the query partition identifier, copy the statistical information corresponding to the first historical partition to the second historical partition; generate a query statement based on the query partition identifier, access the second historical partition after copying the statistical information, and then execute the query statement, returning the query results. This ensures that the index does not become invalid when executing the query statement, and the execution plan of the query statement does not change frequently, thereby achieving optimal query efficiency while improving the accuracy of data queries.

[0117] It should be noted that the data query method provided in this application embodiment is generally executed by server 805, and correspondingly, the data query device is generally set in server 805.

[0118] It should be understood that Figure 8 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0119] The following is for reference. Figure 9 It shows a schematic diagram of the structure of a computer system 900 suitable for implementing a terminal device according to the embodiments of this application. Figure 9 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0120] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the computer system 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0121] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0122] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this application.

[0123] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a receiving unit, a modulus-taking unit, an information copying unit, and a data querying unit. The names of these units do not necessarily limit the specific unit itself.

[0126] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to receive a data query request, obtain the corresponding time identifier and database identifier; determine the corresponding number of partitions based on the database identifier, and then perform modulo processing based on the time identifier and the number of partitions to obtain a query partition identifier; in response to a change in the time identifier, determine a first historical partition from the historical partitions corresponding to the number of partitions and determine a second historical partition corresponding to the query partition identifier, copy the statistical information corresponding to the first historical partition to the second historical partition; generate a query statement based on the query partition identifier, access the second historical partition after copying the statistical information, and then execute the query statement to return query result information.

[0127] The computer program product of this application includes a computer program that, when executed by a processor, implements the data query method in the embodiments of this application.

[0128] According to the technical solution of the embodiments of this application, the index will not become invalid when executing the query statement, and the execution plan of the query statement will not change frequently, thereby improving the accuracy of data query while achieving the optimal query efficiency.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data query method, characterized by, include: Receive data query requests and obtain the corresponding time identifier and database identifier; Based on the database identifier, the corresponding number of partitions is determined, and then a modulo operation is performed based on the time identifier and the number of partitions to obtain the query partition identifier; In response to a change in the time identifier, determining a first historical partition from the historical partitions corresponding to the number of partitions includes: obtaining statistical information of the corresponding historical partition based on the time identifier; Calculate the average value of the statistical information of the historical partitions, and then determine the first historical partition in the corresponding historical partitions based on the average value; the first historical partition refers to the historical partition corresponding to the statistical information that best matches the average value among the data volume corresponding to the statistical information of the historical partition; and determine the second historical partition corresponding to the query partition identifier, and copy the statistical information corresponding to the first historical partition to the second historical partition; the second historical partition refers to the historical partition that is currently needed among the various historical partitions corresponding to the database identifier. A query statement is generated based on the query partition identifier, the second historical partition after copying the statistical information is accessed, the query statement is executed, and the query result information is returned.

2. The method of claim 1, wherein, Before copying the statistical information corresponding to the first historical partition to the second historical partition, the method further includes: Based on the time identifier and the number of partitions, determine the identifier of the partition to be cleaned; Clear the data in the partition corresponding to the partition identifier to be cleaned in the historical partition.

3. The method of claim 2, wherein, The step of determining the partition identifier to be cleaned based on the time identifier and the number of partitions includes: The time identifier is incremented by 1, and then moduloed with the number of partitions to obtain the identifier of the partition to be cleaned.

4. The method of claim 1, wherein, The step of obtaining the statistical information of the corresponding historical partition based on the time identifier includes: The statistical information of the historical partitions corresponding to the number of partitions is updated in real time, and then the statistical information of the historical partitions at a preset time interval before the time marker is obtained, so as to determine the statistical information of the historical partitions corresponding to the time marker.

5. The method of claim 1, wherein, The calculation of the average value of the statistical information of the historical partitions includes: Determine the amount of statistical information for the historical partition corresponding to the time identifier and the number of historical partitions corresponding to the time identifier; The average value of the data volume is calculated based on the data volume and the quantity.

6. The method of claim 1, wherein, Determining the first historical partition in the corresponding historical partition based on the average value includes: Based on the average value, determine the Mahalanobis distance of each data point in the statistical information of the historical partition corresponding to the time identifier; Based on the Mahalanobis distance, the historical partition whose statistical information is closest to the average value is determined and identified as the first historical partition.

7. A data query device, characterized in that, include: The receiving unit is configured to receive data query requests and obtain the corresponding time identifier and database identifier; The modulo unit is configured to determine the corresponding number of partitions based on the database identifier, and then perform modulo processing based on the time identifier and the number of partitions to obtain the query partition identifier; The information replication unit is configured to, in response to a change in the time identifier, determine a first historical partition from the historical partitions corresponding to the number of partitions and determine a second historical partition corresponding to the query partition identifier, and copy the statistical information corresponding to the first historical partition to the second historical partition. The second historical partition refers to the historical partition that is currently needed among the various historical partitions corresponding to the database identifier. The data query unit is configured to generate a query statement based on the query partition identifier, access the second historical partition after copying the statistical information, execute the query statement, and return query result information. The information copying unit is further configured to: obtain statistical information of the corresponding historical partition based on the time identifier; Calculate the average value of the statistical information of the historical partitions, and then determine the first historical partition in the corresponding historical partitions based on the average value; the first historical partition refers to the historical partition corresponding to the statistical information that best matches the average value among the data volume corresponding to the statistical information of the historical partition.

8. The apparatus according to claim 7, characterized in that, The device further includes a data cleaning unit, configured to: Based on the time identifier and the number of partitions, determine the identifier of the partition to be cleaned; Clear the data in the partition corresponding to the partition identifier to be cleaned in the historical partition.

9. The apparatus according to claim 8, characterized in that, The data cleaning unit is further configured to: The time identifier is incremented by 1, and then moduloed with the number of partitions to obtain the identifier of the partition to be cleaned.

10. The apparatus according to claim 7, characterized in that, The information copying unit is further configured to: The statistical information of the historical partitions corresponding to the number of partitions is updated in real time, and then the statistical information of the historical partitions at a preset time interval before the time marker is obtained, so as to determine the statistical information of the historical partitions corresponding to the time marker.

11. The apparatus according to claim 7, characterized in that, The information copying unit is further configured to: Determine the amount of statistical information for the historical partition corresponding to the time identifier and the number of historical partitions corresponding to the time identifier; The average value of the data volume is calculated based on the data volume and the quantity.

12. A data query electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

13. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images