A data processing method, electronic device and computer readable storage medium

Abstract

The present application discloses a data processing method, an electronic device, and a computer-readable storage medium, the method comprising: receiving a current query request to a database, wherein the database comprises multiple partitions, the current query request comprises at least one partition condition, and each partition condition is used to query data from a partition; for each partition condition, searching for cache partition data matching the partition condition from a target cache space as matching partition data corresponding to the partition condition, wherein each cache partition data is cached separately in the target cache space, and each cache partition data is data in a partition that matches a historical query request; and using the found matching partition data as the query result of the current query request. In the above manner, the present application reduces the query resource overhead and query time consumption, thereby improving the query performance of the data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of information technology, the demand for data processing and analysis is gradually increasing. When processing and analyzing data, it is often necessary to query the data. Existing data queries are all performed in the database that stores the original data, which requires a large amount of resource overhead and takes a long time, so the query performance is low. Therefore, there is an urgent need for a way to improve the data query performance. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a data processing method, an electronic device and a computer-readable storage medium, which can improve the query performance of data.

[0004] To solve the above technical problems, a technical solution adopted in the present application is: providing a data processing method, the method comprising: receiving a current query request for a database, wherein the database includes multiple partitions, the current query request includes at least one partition condition, and each partition condition is used to query data from a partition; for each partition condition, searching for cache partition data matching the partition condition from a target cache space as matching partition data corresponding to the partition condition, wherein each cache partition data is cached separately in the target cache space, and each cache partition data is data in a partition that matches the historical query request; and using the found matching partition data as the query result of the current query request.

[0005] Among them, before searching for cache partition data matching the partition condition from the target cache space for each partition condition as the matching partition data corresponding to the partition condition, it also includes: splitting the current query request into at least one sub-request, each sub-request containing a partition condition; for each partition condition, searching for cache partition data matching the partition condition from the target cache space as the matching partition data corresponding to the partition condition, including: for each sub-request, searching for cache partition data matching the sub-request from the target cache space as the matching partition data corresponding to the partition condition in the sub-request.

[0006] Among them, the target cache space caches historical type characterization values ​​of historical query requests corresponding to each of the cache partition data; the searching for cache partition data matching the sub-request from the target cache space as matching partition data corresponding to the partition condition in the sub-request includes: determining the current type characterization value of the sub-request; searching from each of the historical type characterization values ​​cached in the target cache space whether there is a matching historical type characterization value consistent with the current type characterization value; in response to the existence of a matching historical type characterization value, searching for cache partition data matching the partition condition corresponding to the sub-request from each of the cache partition data corresponding to the matching historical type characterization value as the matching partition data corresponding to the partition condition in the sub-request.

[0007] Among them, each partition condition is divided according to the query field of the current query request, and the query field at least includes the partition field, and the current type characterization value of the sub-request is determined, including: removing the partition field corresponding to the sub-request, and asymmetrically encrypting the sub-request after removal to obtain the encryption result of the sub-request as the current type characterization value; and / or, the cache partition data corresponding to each historical type characterization value is cached in a key-value pair data structure, wherein each historical type characterization value is a first key, and each cache partition data corresponding to the historical type characterization value is used as the first value corresponding to the first key.

[0008] Among them, each cache partition data in the first value is cached in a key-value pair data structure, wherein, in a first value, the partition information corresponding to each partition is the second key, the cache partition data corresponding to the partition is the second value corresponding to the second key, and the partition information includes at least one of the partition name, partition version number and partition version update time.

[0009] The method further includes: in response to a partition condition being an unmatched condition, respectively searching for original partition data that matches the unmatched condition from multiple partitions of the database as matching partition data corresponding to the unmatched condition, wherein the unmatched condition is a target partition condition in which no matching cache partition data exists in the target cache space; and caching the matching partition data corresponding to the unmatched condition as new cache partition data in the target cache space.

[0010] Among them, multiple partitions include at least one cold partition and at least one hot partition; each cache partition data of the target cache space includes cache partition data corresponding to the cold partition and cache partition data corresponding to the first hot partition; the matching partition data that do not match the matching conditions are respectively used as new cache partition data and cached in the target cache space, including: in response to the partition corresponding to the matching partition data being a cold partition or a first hot partition, the matching partition data are used as new cache partition data and cached in the target cache space, wherein the first hot partition is a hot partition that is not currently in a frequently updated state; in response to the partition condition being an unmatched condition, after matching partition data that do not match the matching conditions are respectively searched out from multiple partitions of the database, it also includes: in response to the partition corresponding to the matching partition data being a second hot partition, the matching partition data is not cached, and the second hot partition is a hot partition that is currently in a frequently updated state.

[0011] Among them, the multiple partitions include at least one cold partition; the method also includes: in response to a data update in a cold partition in the database, and the cold partition has corresponding cache partition data, updating the cache partition data of the cold partition where the data update occurs; searching for cache partition data matching the partition condition from the target cache space as the matching partition data corresponding to the partition condition, including: searching for cache partition data matching the partition condition from the target cache space; in response to the existence of cache partition data matching the partition condition in the target cache space, and the partition condition is the first partition condition corresponding to the cold partition, reading the cache partition data matching the first partition condition as the matching partition data corresponding to the first partition condition.

[0012] Among them, the multiple partitions include at least one hot partition; the method also includes: in response to a hot partition in the database having data updated, and the hot partition has corresponding cache partition data, the cache partition data of the hot partition having data updated is not updated; searching for cache partition data matching the partition condition from the target cache space as matching partition data corresponding to the partition condition, including: searching for cache partition data matching the partition condition from the target cache space; in response to the existence of cache partition data matching the partition condition in the target cache space, and the partition condition is the second partition condition corresponding to the first hot partition, updating the cache partition data matching the second partition condition in the target cache space, and using it as matching partition data corresponding to the second partition condition, wherein the first hot partition is a hot partition that is not currently in a frequently updated state; in response to the existence of cache partition data matching the partition condition in the target cache space, and the partition condition is the third partition condition corresponding to the second hot partition, searching for original partition data matching the third partition condition from the second hot partition of the database as matching partition data corresponding to the third partition condition, wherein the second hot partition is a hot partition that is currently in a frequently updated state.

[0013] Among them, the multiple partitions include at least one hot partition; the method also includes: in response to a data update in a first hot partition in the database, and the first hot partition has corresponding cache partition data, updating the cache partition data of the first hot partition where the data is updated, wherein the first hot partition is a hot partition that is not currently in a frequent update state; in response to a data update in a second hot partition in the database, and the second hot partition has corresponding cache partition data, deleting the cache partition data of the second hot partition where the data is updated, or setting the cache partition data of the second hot partition where the data is updated to an invalid state, wherein the cache partition data in the invalid state will no longer be read, and will be restored to a valid state when an update occurs next time, and the second hot partition is a hot partition that is currently in a frequent update state. state; searching for cache partition data matching the partition condition from the target cache space as matching partition data corresponding to the partition condition, including: searching for cache partition data matching the partition condition from the target cache space; in response to the existence of valid cache partition data matching the partition condition in the target cache space, and the partition condition is a fourth partition condition corresponding to the hot partition, reading cache partition data matching the fourth partition condition as matching partition data corresponding to the fourth partition condition; or, in response to the existence of invalid cache partition data matching the partition condition in the target cache space, and the partition condition is the fourth partition condition corresponding to the hot partition, searching for original partition data matching the fourth partition condition from the corresponding hot partition of the database as matching partition data corresponding to the fourth partition condition.

[0014] The hot partition in the frequently updated state is a hot partition whose update time interval is less than the first time period, or a hot partition that has been updated in the latest second time period.

[0015] The method further includes: in response to the target cache space satisfying the deletion condition, deleting at least one cache partition data in the target cache space. The deletion condition includes that the storage capacity of the target cache space reaches the target upper limit value; and / or, deleting at least one cache partition data in the target cache space, including: determining the deletion order of each cache partition data in the target cache space according to the weight of each cache partition data in the target cache space; deleting the cache partition data in the target cache space according to the deletion order until the number of cache partition data deleted reaches the target number or the target cache space no longer meets the deletion condition.

[0016] The weight of the cache partition data is determined by the number of occurrences of the historical query requests corresponding to the cache partition data within the third time period.

[0017] In which, the data of the database is stored in partitions of several hard disks, and query requests of the same type are assigned to the same hard disk for processing; and / or, the data of the database is stored in partitions of several hard disks, wherein the target cache space includes cache space in processes of several hard disks, wherein the hard disk to which the historical query request is assigned is the request assigned hard disk, and the cache partition data corresponding to the historical query request is cached in the cache space in the process of the request assigned hard disk; and / or, the routing forwarding of the current query request and the historical query request is implemented by consistent hashing; and / or, each partition condition is divided according to the query field of the current query request, and the query field at least includes the partition field.

[0018] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, the device includes a memory and a processor, the memory is coupled to the processor, and program instructions are stored in the memory to implement any of the data processing methods as described above.

[0019] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium on which a computing program is stored, and the program is processed and executed to implement any of the data processing methods as described above.

[0020] The beneficial effect of the present application is as follows: different from the prior art, the present application independently caches each partition data of the database historically queried in the target cache space as different cache partition data. When the current query request is received, the matching cache partition data is directly queried from the target cache space as the query result of the current query request, that is, for the partition data that has been historically queried, the corresponding cache partition data can be directly read from the cache space without re-querying from the database, thereby reducing the query resource overhead and query time, thereby improving the data query performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of an embodiment of the data processing method of the present application;

[0022] Figure 2 This is a flow chart of a data matching method according to an embodiment of a data processing method of the present application;

[0023] Figure 3 This is a schematic diagram of the key-value pair cache structure of an embodiment of the data processing method of the present application;

[0024] Figure 4 This is a routing forwarding schematic diagram of an embodiment of a data processing method of the present application;

[0025] Figure 5 This is a schematic diagram of cache partition data hits in an embodiment of the data processing method of the present application;

[0026] Figure 6 It is a schematic diagram of the framework of an embodiment of a data processing device of the present application;

[0027] Figure 7 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0028] Figure 8 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0030] It should be understood that the reference to "embodiment" in this application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0032] See also Figure 1 , Figure 1 This is a flowchart of an embodiment of the data processing method of the present application. Among them, data will be generated in many business scenarios. The data processed by the present application can be images, such as image data obtained by a camera or a camera, text data, and any other type of data. After the data is written into the database, it will be queried. The data query consumes a lot of computing resources and is time-consuming. The data processing method provided by the present application can improve the performance of data query. The method includes the following steps:

[0033] Step S11: receiving a current query request to a database, wherein the database includes multiple partitions, and the current query request includes at least one partition condition, and each partition condition is used to query data from a partition.

[0034] Among them, the database is used to store data, and the data stored in the database can be data generated in various business scenarios. In some application scenarios, the storage format of the data in the database may include structured data, for example, data organized in a table with clear rows and columns, each column represents a specific attribute or field of the data, and each row represents an instance or record of data that conforms to this data model, which may specifically include the following columns: component ID, manufacturer, specification, etc., each of which records the specific information of the component. In some application scenarios, the data in the database may also include unstructured data, that is, data without a predefined data model or format, such as text files, pictures, audio, video, email, documents, etc. In some application scenarios, the data in the database may also include unstructured data, for example, file data with tags or attributes, and the number and order of these tags or attributes may change in different data instances.

[0035] The data in the database is stored in partitions. For example, in an application scenario where the data is in a table in the database, the table in the database is a partitioned table, that is, the data in the table in the database is divided into multiple partitions. By storing the data in the database in partitioned tables, query requests for each partition only need to be performed on the relevant partitioned tables, reducing the scope of data retrieval and thus improving data query efficiency.

[0036] In some embodiments, the data in the database is stored in multiple hard disks. For example, each physical node sets up several hard disks according to the resources it has, or in combination with actual business needs. Each hard disk is an independent storage and computing unit, and stores the full amount of metadata information. The hard disk is divided into several partitions, and the data of the database is stored in each partition of the several hard disks. In some embodiments, the multiple partitions of the database are divided by partition fields, and the partition fields can be business attributes shared by all data in the database. In a specific embodiment, the data in the database can refer to the following Table 1: License Plate Data Table. The data in the database can include pictures of license plates taken. The information of each piece of data includes the license plate number in the picture, the shooting time of the picture, and the name of the shooting device. The partition field can be a field in the license plate, or the shooting time of the picture, or the name of the shooting device. For example, when the partition field is the shooting time (day) of the picture, Table 1 can be divided into Table 2: License Plate Data Table Partition 1, and Table 3: License Plate Data Table Partition 2.

[0037] Table 1: License plate data table

[0038]

[0039] Table 2: License plate data table partition 1

[0040]

[0041] Table 3: License plate data table partition 2

[0042]

[0043] From Table 1, Table 2 and Table 3 above, we can see that after the data in the data table is divided into several partitions according to the partition field, the amount of data on each sub-table is reduced. When querying the data in the data table, the query requests for different partitions will be distributed on different sub-tables, reducing the occurrence of query blocking and improving the concurrent processing capability of the database.

[0044] In some embodiments, the query request may include at least one partition condition, which is used to query data from a partition. The partition condition is divided by the query field of the current query request, and the query field at least includes the partition field. For example, when the partition field of the database is time (day), the query request may include at least one partition condition divided by time (day). Specifically, taking the data in Table 1 and Table 2 as an example, the query field of the query request includes "2024-05-31", then the partition condition of the query request is "2024-05-31", or the query field of the query request includes "from 2024-05-30 to 2024-06-01", then the partition conditions of the query request are "2024-05-30", "2024-05-31", and "2024-06-01". It can be understood that the query field of the query request includes "May 2024", that is, the query range is "May 1, 2024 to May 31, 2024". If the partition field is day (that is, the data of different days in the database are stored in different partitions), the partition condition of the query request is the days between May 1, 2024 and May 31, 2024. If the partition field is month (that is, the data of different months in the database are stored in different partitions), the partition condition of the query request is a partition condition for May 2024. Similarly, the partition condition in the query request in this embodiment can also be in years.

[0045] In some application scenarios, the query field in the query request may also include other fields besides the partition field, such as location, name, etc. Then the multiple partition conditions are divided by partition field + other fields. For example, the query field of the query request may include the partition field "2024-05-31", and other fields "Shanghai" and "Hangzhou", then the partition conditions of the query request include two, one of which is "2024-05-31" + "Shanghai" (i.e. query the data of "Shanghai" in the partition corresponding to "2024-05-31"), and the other is "2024-05-31" + "Hangzhou" (i.e. query the data of "Hangzhou" in the partition corresponding to "2024-05-31").

[0046] In some embodiments, the partition field is not necessarily time, but can also be location, and other fields can include time, weather, etc. For example, the database is partitioned and stored with data from different locations, and the query fields of the query request include "Shanghai + June" and "Hangzhou + May". Then the partition conditions of the query request include two, one of which is "Shanghai + June", corresponding to the query of the data belonging to "June" in the "Shanghai" partition, and the other is "Hangzhou + May", corresponding to the query of the data belonging to "May" in the "Hangzhou" subtable. It can be understood that in some application scenarios, multiple partition conditions can be queried for the same partition. For example, the query fields of the partition condition can include "Shanghai + May" and "Shanghai + July", then the query field "Shanghai + May" corresponds to the data belonging to May in the "Shanghai" partition, and the query field "Shanghai + July" corresponds to the data in July in the "Shanghai" partition.

[0047] It can be understood that for some query requests, the query field may not contain a partition field, for example, the partition field is day, and the query field is "Shanghai", that is, the query request does not contain a partition field for a certain partition, and the subsequent steps of this embodiment are for query requests containing partition fields (that is, the query field contains partition fields), so this step S11 may specifically include: receiving the current query request for the database, parsing the current query request to determine whether it contains at least one partition condition, for example, it can be determined whether it contains a partition field, and the query partition range is within a certain range, for example, greater than or equal to one partition (specifically, the partition field is day, and the partition range in the query request is one or more days, not half a day or a few hours), or greater than or equal to one partition and less than the upper limit of the range. In response to the current query request containing at least one partition condition, step S12 and subsequent steps are executed.

[0048] Step S12: For each partition condition, search the target cache space for cache partition data that matches the partition condition as matching partition data corresponding to the partition condition, wherein each cache partition data is cached separately in the target cache space, and each cache partition data is data in a partition that matches a historical query request.

[0049] In this embodiment, the historical query results are cached in the target cache space. When the same query request is encountered, the results can be directly obtained from the target cache space without executing the complete database query process. This can significantly reduce the query time and improve the response speed.

[0050] Specifically, the target cache space includes the cache space corresponding to the hard disk process that processes the corresponding query request. The data matching each partition condition in the historical query request is cached in the target cache space as cache partition data. When a subsequent query request is received, the corresponding cache partition data is matched in the target cache space using each partition condition of the subsequent query request, and the query data matching the partition condition is extracted as the matching partition data corresponding to the partition condition.

[0051] In some embodiments, before executing step S12, it also includes: splitting the current query request into at least one sub-request, each sub-request includes a partition condition, and specifically, the current query request can be split according to the partition field in the current database. In a specific embodiment, the current query request includes a query partition range, and the query partition range is split using the partition field of the database, and the current query request is split into sub-requests corresponding to the partition fields corresponding to the query partition range. For example: the current query request is: "Query data from May 31 to June 1", the partition field of the database is time (day), that is, the query partition range is from May 31 to June 1, which can be split into the partition fields "May 31" and "June 1", so after the first query request is split according to the partition field, the sub-requests obtained include at least: "Query data on May 31" The first sub-request and "Query data on June 1" are a second sub-request.

[0052] In some embodiments, before splitting the current query request into at least one sub-request, it also includes: rewriting the current query request according to the grammatical rules. Specifically, rewriting the current query request according to the grammatical rules includes: unifying the grammatical rules of the current query request and the historical query request, and unifying the grammatical rules of the partition fields and other fields and the query request. For example, the grammar for "greater than" is unified as "≥", the grammar for "less than" is unified as "≤", and the grammar for "between..." is unified as "≤...≥" to avoid processing differences due to different grammars, standardize the data operation process, reduce the need for manual intervention, and thus improve the efficiency of data processing.

[0053] In some embodiments, for each partition condition, cache partition data that matches the partition condition is searched from the target storage space as matching partition data corresponding to the partition condition, including: for each sub-request, cache partition data that matches the sub-request is searched from the target cache space as matching partition data corresponding to the partition condition in the sub-request.

[0054] In some specific embodiments, see Figure 2 , Figure 2This is a flow chart of a data matching method of an embodiment of the data processing method of the present application. Searching for cache partition data matching a sub-request from a target cache space as matching partition data corresponding to a partition condition in the sub-request includes the following steps:

[0055] Step S21: Determine the current type characterization value of the sub-request.

[0056] Specifically, the type characterization values ​​of different sub-requests of the same application are different, and the type characterization values ​​of sub-requests of different applications may be the same. In some embodiments, the partition field corresponding to the sub-request can be removed, and a hash value can be obtained by hashing the sub-request after removal, and the obtained hash value can be used as the type characterization value of the current sub-request. For example, the sub-request "Get all the data of the Hangzhou sub-table", where the partition field is Hangzhou, then the sub-request after removing the partition field is "Get the data of the partition field", and a hash value is obtained by hashing "Get the data of the partition field", and the hash value is used as the type characterization value of the sub-request. It can be understood that if there are several sub-requests with different partitioning conditions but the other fields except the partitioning conditions are the same or there are no other fields, then the request type characterization values ​​obtained after these sub-requests remove the partition field and are encrypted with MD5 are the same.

[0057] In some embodiments, the partition condition is divided based on the query field of the current query request, and the query field includes at least the partition field. For example, if the partition condition of a sub-request is "Hangzhou + July", where the partition field is Hangzhou, a hash operation is performed on the sub-request after removing the partition field "Hangzhou" to obtain a hash value, and the hash value is used as the type characterization value of the sub-request. Specifically, the hash operation can be an MD5 operation, where MD5 (Message-Digest Algorithm 5) is a widely used cryptographic hash function that can calculate information of any length (such as text, files, etc.) and output a 128-bit (16-byte) hash value, usually expressed as 32 hexadecimal numbers. This hash value is highly unique and irreversible, that is, it is almost impossible for different inputs to obtain the same output, and it is almost impossible to infer the input from the output, thereby improving the security of the request data. In a specific scenario, the current query request is a SQL (Structured Query Language). After removing the partition field, the SQL is as follows: sql>SELECT cap_time, count(record_id) FROM carTable GROUP BY cap_time. The result after MD5 encryption is: f2d6ec7c594ed5d8285a96e8fe8d9cfe.

[0058] In other embodiments, the partition field corresponding to the sub-request may be removed, and the removed sub-request may be asymmetrically encrypted to obtain the encryption result of the sub-request as the current type representation value. The asymmetrical encryption method may include an elliptic curve encryption algorithm, etc.

[0059] Step S22: searching among the historical type characterization values ​​cached in the target cache space whether there is a matching historical type characterization value that is consistent with the current type characterization value.

[0060] The historical type characterization value includes the type characterization value of the sub-request in the historical query request. In some embodiments, the cache partition data corresponding to each historical type characterization value is cached in a key-value pair data structure, wherein each historical type characterization value is a first key, and each cache partition data corresponding to the historical type characterization value is used as the first value corresponding to the first key. It can be understood that the first value may include one or more partition data.

[0061] In some embodiments, the partition information corresponding to each partition of a first value is the second key, the cache partition data corresponding to the partition is the second value corresponding to the second key, and the partition information includes at least one of the partition name, the partition version number, and the update time of the partition version. The cache partition data can be assembled into a hash map (hash map) for storage, and the data structure of the hash map can be represented as:

[0062] HashMap <String,ResultPartitionSet<HashMap<PartitionInfo,ResultPartition> >>.

[0063] Among them, in the above data structure: HashMap represents hash mapping, String represents the first key, ResultPartitionSet represents the first value corresponding to the first key, PartitionInfo represents the second key in the first value, and ResultPartition is the second value corresponding to the second key.

[0064] In a specific embodiment, if Figure 3 As shown, Figure 3This is a schematic diagram of the key-value cache structure of an embodiment of the data processing method of the present application. The first key (key) is the MD5 signature of the sub-request with the partition field removed, and the first value (value(ResultPartitionSet)) is the partition information corresponding to the first key. Among them, the first value is a set of data sets of set type, each element in the first value is composed of a map (Map), each element is a partition data set, the second key (key) of each partition data set is the partition information, and the second value (value) of each partition data set is the cache partition data corresponding to the partition (that is, the historical query data of the partition). For example, Figure 3 The cache data hash map shown in the first row in the figure has a first key of f2d6ec7c594ed5d8285a96e8fe8d9cfe, and the first value corresponding to the first key includes the third partition cache data set, the fourth partition cache data set, etc. Among them, the second key of the third partition cache data set includes the partition name third partition, and the partition version update time 20240601. The second value of the third partition cache data set is the historical query result of the query request type corresponding to the first key in the third partition of the database.

[0065] In some embodiments, the current type characterization value is the current MD5 signature of the current sub-request after the partition field is removed, and the historical type characterization value includes the historical MD5 signature of the sub-request in the historical query request after the partition field is removed. From each historical type characterization value cached in the target cache space, searching whether there is a matching historical type characterization value that is consistent with the current type characterization value includes: comparing the current MD5 signature corresponding to the current sub-request with the historical MD5 signatures corresponding to each historical query request cached in the target cache space, if there is a current MD5 signature that is the same as the historical MD5 signature, then it means that the historical type characterization value matches the current type characterization value, and the historical type characterization value is a matching type characterization value of the current type characterization value.

[0066] Step S23: In response to the existence of a matching history type characterization value, cache partition data matching the partition condition corresponding to the sub-request is searched from each cache partition data corresponding to the matching history type characterization value as the matching partition data corresponding to the partition condition in the sub-request.

[0067] The historical query request corresponding to the matching history type characterization value is the matching history query request. It can be understood that the matching history query request after removing the partition field is the same as the current query request after removing the partition field. There may be cache partition data matching the partition condition corresponding to the current sub-request in each cache partition data corresponding to the matching history type characterization value of the current type characterization value, or there may be no cache partition data matching the partition condition corresponding to the current sub-request. In some embodiments, the partition field is the location, where the location includes: Shanghai, Hangzhou, and Beijing; the other fields are time (month), and the time includes: June, July, and August. The other fields corresponding to the matching history query request after removing the partition field are "June", and the partitions corresponding to the matching partition data corresponding to the matching history query request include Shanghai, Hangzhou, and Beijing. The request field of the current query sub-request includes "June + Shanghai", then the Shanghai partition data in the matching partition data is the matching partition data corresponding to the current sub-request, and the Shanghai partition data in the matching partition data is extracted as the matching partition data corresponding to the current sub-request.

[0068] In some embodiments, if there is no historical type characterization value consistent with the current type characterization value of the current sub-request in the target cache space, or if there is a partition condition of the sub-request and there is no matching cache partition data in the target cache space, such partition conditions can be regarded as unmatched conditions, and partition data matching each unmatched condition can be searched from the database. Specifically, the following steps can be performed in the case of unmatched conditions:

[0069] Step S24: In response to the partition condition being an unmatched condition, original partition data matching the unmatched condition is searched out from multiple partitions of the database respectively as matching partition data corresponding to the unmatched condition, wherein the unmatched condition is a target partition condition in which there is no matching cache partition data in the target cache space.

[0070] In some embodiments, the current query request may be the first query request received in the hard disk process, wherein the first query request is the first query request received in the current hard disk. It is understandable that when the first query request is received, the target cache space of the current hard disk does not contain cache partition data, that is, there is no partition data matching the partition condition in the first query request in the target cache space. All partition conditions in the first query request are unmatched conditions, the first query request is split into several sub-requests, each sub-request contains a partition condition, for each partition condition, the corresponding matching partition data is searched from each corresponding partition of the database, and each matching partition data is used as a new cache partition data, and cached in the target cache space. Specifically, at least one current type characterization value of several sub-requests can be obtained, each current type characterization value is used as the first key of a cache partition data, and the matching partition data of at least one sub-request corresponding to each current type characterization value is used as the first value corresponding to the first key. In the first value, the partition information corresponding to each matching partition data is the second key, and the data set in the matching partition data is the second value corresponding to the second key, wherein the partition information may include the partition name, the partition version number, and the partition version update time.

[0071] In some embodiments, the current query request may be the second query request received in the hard disk process, wherein the second query request is a query request other than the first query request received in the current hard disk. The second query request is split into several sub-requests, each of which contains a partition condition, and the current type characterization value of the current sub-request is obtained. In each historical type characterization value in the target cache space, a historical type characterization value consistent with the current type characterization value is searched as a matching historical type characterization value, and in the cache partition data corresponding to the matching historical type characterization value, cache data matching the partition condition in the current sub-request is searched. If there is no cache data matching the partition condition in the current sub-request in the cache partition data corresponding to the matching historical type characterization value, the original partition data matching the partition condition in the current sub-request is searched from the partition of the database as the matching partition data corresponding to the current sub-request, and the partition data is stored in the cache partition data corresponding to the matching historical type characterization value in the target cache space. The query results are cached in units of partitions, and subsequent query requests with the same query type and the same partition condition as the historical query request can directly obtain data from the cache data, which can significantly improve the query efficiency of the database.

[0072] Step S13: taking the found matching partition data as the query result of the current query request.

[0073] The found matching partition data may include at least one of data obtained from the target cache partition and data obtained from the original partition of the database. Specifically, the current query request includes several query sub-requests, each of which corresponds to matching partition data, and the matching partition data of all sub-requests are used as the query result data of the current query request.

[0074] Through the above scheme, each partition data of the database historically queried is cached independently in the target cache space as different cache partition data. When the current query request is received, the matching cache partition data is directly queried from the target cache space as the query result of the current query request. That is, for the partition data that has been historically queried, the corresponding cache partition data can be directly read from the cache space without re-querying from the database, thereby reducing the query resource overhead and query time, thereby improving the data query performance.

[0075] In some application scenarios, the original partition data in the database will be written, updated, deleted, and other operations. In order to ensure that the data queried by the query request is the latest data, the data of the cache partition needs to be consistent with the data of the original table partition.

[0076] In some embodiments, each partition of the database is divided into a cold partition and a hot partition according to the update frequency of the original partition data in one cycle. Specifically, a preset update threshold within a certain preset cycle can be set, and a cold partition is a partition whose update amount within the preset update threshold within the preset cycle, and a hot partition is a partition whose update amount within the preset cycle is greater than the preset threshold. The update amount can be at least one of the update data amount or the update number. For example, a partition with a daily update data amount of less than 10,000 and a daily update number of less than 100 times is a cold partition, and a partition with a daily update data amount of more than 100 times or a daily update data amount of more than 10,000 times is a hot partition.

[0077] In some embodiments, the multiple partitions of the database include at least one cold partition. Once a data update occurs in a cold partition, the cache partition data corresponding to the partition is also updated to keep the cache data fresh. Specifically, in response to a data update in a cold partition in the database, and the corresponding cache partition data exists in the cold partition, the cache partition data of the cold partition where the data update occurs is updated. In a specific embodiment, there is a cold partition A in the database, and the cold partition A (a1, a2) has corresponding cache partition data a (a1, a2). At a certain moment, the data a1 in the cold partition is changed to a3, then a1 in the cache partition a is changed to a3 following the data modification in partition A.

[0078] In some embodiments, when multiple partitions of a database include at least one cold partition, searching for cache partition data matching the partition condition from the target cache space as matching partition data corresponding to the partition condition includes: searching for cache partition data matching the partition condition from the target cache space; in response to the presence of cache partition data matching the partition condition in the target cache space, and the partition condition is a first partition condition corresponding to the cold partition, reading cache partition data matching the first partition condition as matching partition data corresponding to the first partition condition. In a specific embodiment, the current sub-request includes the first partition condition corresponding to the cold partition, obtaining the request type characterization value of the current sub-request, and searching from each historical type characterization value cached in the target cache space whether there is a matching historical type characterization value that is consistent with the current type characterization value. In response to the presence of a matching historical type characterization value of the current type characterization value in the target cache space, obtaining the first partition field in the first partition condition of the sub-request, and obtaining cache partition data matching the first partition field from the cache partition data corresponding to the matching historical type characterization value as matching partition data corresponding to the first partition condition. In response to the partition condition being an unmatched condition and the partition condition being the first partition condition corresponding to the cold partition, original partition data matching the unmatched condition is searched from the partitions of the database as the matched partition data corresponding to the unmatched condition. The matched partition data is cached in the target cache space in the form of a key-value pair as cache partition data. In response to a data update occurring in the cold partition corresponding to the cache partition data in the database, the cache partition data is updated.

[0079] In some embodiments, the multiple partitions of the database include at least one hot partition. For the hot partition, the cache update strategy needs to be adjusted to avoid frequent cache updates that reduce query performance. Specifically, the hot partition can be divided into a first hot partition and a second hot partition according to whether the hot partition is currently in a frequent update state, wherein the first hot partition is a hot partition that is not currently in a frequent update state, and the second hot partition is a hot partition that is currently in a frequent update state.

[0080] In some specific embodiments, the hot partition in the frequently updated state is a hot partition whose update time interval is less than the first time period. For example, the first time period can be set to one hour, and the two most recent update times in a hot partition are t1 and t2 respectively. If the time interval between t1 and t2 is less than one hour, the hot partition is the second hot partition, and if the time interval between t1 and t2 is not less than one hour, the hot partition is the first hot partition.

[0081] In another specific embodiment, the hot partition in the frequently updated state is the hot partition that has been updated in the second time period recently. For example, the second time period can be set to one hour, and the most recent update time of a hot partition from the current time t0 is t1. If the time interval between t1 and t0 is less than one hour, the hot partition is the second hot partition, and if the time interval between t1 and t0 is not less than one hour, the hot partition is the first hot partition.

[0082] In some application scenarios, different cache strategies can be set for the first hot partition and the second hot partition, respectively. Specifically, the multiple partitions of the database include at least one cold partition and at least one hot partition, and the cache partition data of the target cache space includes the cache partition data corresponding to the cold partition and the cache partition data corresponding to the first hot partition. The cache data of the target cache space does not include the result data corresponding to the second hot partition. Since the second hot partition is a hot partition with a large amount of updated data and is currently in a frequently updated state, it takes more background processes to maintain the validity of the partition, so it is not cached.

[0083] In some embodiments, there is an unmatched condition in the query request, and matching partition data matching the unmatched condition is found from multiple partitions of the database respectively, and the matching partition data matching the unmatched condition is used as new cache partition data and cached in the target cache space. Specifically, it is determined that the partition of the matching partition data belongs to a cold partition, or a first hot partition, or a second hot partition, and in response to the partition corresponding to the matching partition data being a cold partition or a first hot partition, the matching partition data is used as new cache data and cached in the target cache space. In response to the partition corresponding to the matching partition data being the second hot partition, the matching partition data is not cached.

[0084] In some application scenarios, a hot partition in the database has data updated, and the hot partition has corresponding cache partition data, and the cache partition data of the hot partition where the data update occurs is not updated. In some embodiments, the partitions of the current database include a first hot partition, and cache partition data that matches the partition condition is searched from the target cache space as matching partition data corresponding to the partition condition, including: searching from the target cache space for cache partition data that matches the partition condition. In response to the existence of cache partition data that matches the partition condition in the target cache space, and the partition condition is the second partition condition corresponding to the first hot partition, the cache partition data that matches the second partition condition in the target cache space is updated as the matching partition data corresponding to the second partition condition. In some specific embodiments, the current sub-request includes a second partition condition. When querying the current sub-request, the request type characterization value of the current sub-request can be first obtained, and the historical type characterization values ​​in the target cache space are searched to see whether there is a matching historical type characterization value that is consistent with the current type characterization value. In response to the existence of a matching historical type characterization value of the current type characterization value in the target cache space, the second partition field in the second partition condition of the sub-request is obtained, and the cache partition data matching the second partition field is obtained from the cache partition data corresponding to the historical type characterization value, and the cache partition data is updated, and the updated cache partition data is used as the matching partition data corresponding to the second partition condition of the current sub-request. When the hot partition is not currently in a state of frequent updates, when a query for the cache corresponding to the hot partition occurs, the update of the cache partition data is triggered, and the cache is updated asynchronously through a background process, which reduces the immediate impact of the update operation on the performance, and at the same time ensures that the queried data is the most recent data.

[0085] In other embodiments, the partition of the current database includes a second hot partition. In response to the presence of cache partition data matching the partition condition in the target cache space, and the partition condition is the third partition condition corresponding to the second hot partition, the original partition data matching the third partition condition is searched out from the second hot partition of the database as the matching partition data corresponding to the third partition condition. In a specific embodiment, the current sub-request includes a third partition condition, the third partition field in the third partition condition is obtained, the second hot partition corresponding to the third partition field is searched out from the database, and the original partition data matching the third partition condition is obtained from the second hot partition according to the third partition condition of the current sub-request as the matching partition data of the third partition condition of the sub-request. When the hot partition is currently in a state of frequent data updates, when a query for the hot partition occurs, the query is directly made from the database without updating the corresponding cache, thereby avoiding frequent cache updates.

[0086] In other application scenarios, different cache partition data processing strategies are set for the first hot partition and the second hot partition, respectively. In some embodiments, the partitions of the current database include the first hot partition, and in response to a data update of a first hot partition in the database, and the first hot partition has corresponding cache partition data, the cache partition data of the first hot partition where the data is updated is updated.

[0087] In some embodiments, the current database includes a second hot partition, and the cache processing strategy for the second hot partition includes: in response to a data update of a second hot partition in the database, and the corresponding cache partition data exists in the second hot partition, deleting the cache partition data of the second hot partition where the data update occurs, or setting the cache partition data of the second hot partition where the data update occurs to an invalid state, wherein the cache partition data in the invalid state is no longer read, and is restored to a valid state when the next update occurs. It can be understood that in this embodiment, if the cache partition data is valid, then the cache partition data is the latest data. Searching for cache partition data matching the partition condition from the target cache space as the matching partition data corresponding to the partition condition. Specifically, it includes: searching for valid cache partition data matching the partition condition from the target cache space, in response to the existence of valid cache partition data matching the partition condition in the target cache space, and the partition condition is the fourth partition condition corresponding to the hot partition, reading the cache partition data matching the fourth partition condition as the matching partition data corresponding to the fourth partition condition. Wherein, the fourth partition condition may include the partition condition of the first hot partition, and may also include the partition condition of the second hot partition. In some embodiments, in response to the existence of invalid cache partition data matching the partition condition in the target cache space, and the partition condition is the fourth partition condition corresponding to the hot partition, original partition data matching the fourth partition condition is searched from the hot partition corresponding to the database as matching partition data corresponding to the fourth partition condition. The fourth partition condition may correspond to the second hot partition, or the fourth partition condition may correspond to the first hot partition.

[0088] In other application scenarios, the cache partition data of the second hot partition where data update occurs can be set with an invalid mark. In subsequent queries, the cache partition data with the invalid mark does not participate in the matching of the partition condition. Specifically, when querying, whether the cache partition data is valid is directly determined by whether the cache partition data has an invalid mark. If the cache partition data has an invalid mark, the cache partition data is not read, and the corresponding original partition data is directly matched from the database.

[0089] In other application scenarios, the cache partition data corresponding to the second hot partition where data is updated may not be updated directly. In subsequent queries, the partition version number or partition version update time in the cache partition data is compared with the original partition data in the database to see if they are the same. If they are the same, the cache partition data is valid and can be read as the matching partition data for the query request; if they are different, the cache partition data is invalid and is not read, and the corresponding original partition data is directly matched from the database.

[0090] In a specific usage scenario, the database includes partition A and partition B, where both partition A and partition B are hot partitions, and the target cache space contains cache partition data a corresponding to partition A. Take partition A as an example for updating cache data: if data is updated in partition A at time t1, obtain the most recent data update time t0 before time t1 in partition A, and determine whether the time interval between t0 and t1 is less than one hour. If it is not less than one hour, the data corresponding to the updated data of partition A in cache partition data a is updated following the update of partition A; if it is less than one hour, the cache partition data a is marked as invalid, and the update of partition A does not update the data of cache partition a. At time t2, data is updated in partition A, obtain the most recent data update time t1 before time t2 in partition A, and determine whether the time interval between t2 and t1 is less than one hour. If it is not less than one hour, the data in cache partition data a is updated to the latest data corresponding to partition A. Taking partition B as an example of data caching, a data query for partition B occurs at time t1, and query result b is obtained from partition B. The most recent data update time t0 before time t1 in partition B is obtained, and it is determined whether the time interval between t0 and t1 is less than one hour. If it is not less than one hour, query result b is cached; if it is less than one hour, query result b is not cached. At time t2, a query occurs for partition cache partition data b. If cache partition data b is valid, the cache partition data b can be used directly. A reasonable cache expiration time is set according to the data update frequency of the hot partition to maximize the data query performance while maintaining the consistency of the valid cache partition data.

[0091] In some embodiments, the physical node includes several hard disks, each hard disk is provided with cache space, and the database data is stored in partitions of the several hard disks. Several query requests can be forwarded to different hard disks for processing to reduce the computing pressure of each hard disk process and improve the efficiency of the query. Furthermore, in order to improve the hit rate of the cache partition data in the hard disk, the routing and distribution rules of the query request can be set to assign the same type of query requests to the same hard disk for processing.

[0092] In some specific embodiments, the request type of the current query request can be obtained, and the request type of the corresponding historical query request can be matched in each hard disk. The hard disk assigned to the historical query request is used as the request assigned hard disk, and the current query request is routed and forwarded to the request assigned hard disk for processing. Furthermore, the cache partition data corresponding to the historical query request is cached in the cache space in the process of requesting the hard disk to be assigned, and the data that needs to be cached in the query result of the current query request is cached in the cache space in the process of requesting the hard disk to be assigned.

[0093] In some embodiments, see Figure 4 , Figure 4 4 is a routing forwarding diagram of an embodiment of the data processing method of the present application. The routing forwarding 42 of the current query request 41 and the historical query request is implemented by consistent hashing. Specifically, each hard disk can be hashed to obtain the hash value of each hard disk, and the hash value of each hard disk is mapped to the hash ring 40 to determine the position of each hard disk on the hash ring 40. For example, the hard disk of the current node includes four hard disks. After hashing each hard disk, the positions of the first hard disk 431, the second hard disk 432, the third hard disk 433 and the fourth hard disk 434 distributed in a clockwise direction on the hash ring 40 are determined. When a query request is received, a hash operation is performed on the query request to obtain its hash value, and the hash value of the query request is mapped to the hash ring 40 to determine the position of the query request on the ring. Starting from the position of the query request, the first hard disk on the hash ring 40 is searched clockwise, and the query request is routed to the hard disk for processing. For example, the first query request 411 is located between the fourth hard disk 434 and the first hard disk 431 on the hash ring 40, and the first hard disk in the clockwise direction of the first query request 411 is the first hard disk 431, so the first query request 411 is routed to the first hard disk 431 for processing. Similarly, the target hard disks of the second query request 412, the third query request 413, and the fourth query request 414 are determined according to their locations on the hash ring 40.

[0094] In some application scenarios, the hash operation in the consistent hashing may specifically be an MD5 operation. For example, performing a hash operation on a query request may perform an MD5 signature on the query request. In some specific embodiments, performing a hash operation on a query request may be calculating a type characterization value of the query request. The type characterization value of the query request is mapped to the hash ring 40, and the query request is assigned to the corresponding hard disk process based on the type characterization value of the query request, thereby achieving the assignment of query requests of the same type to the same hard disk process. In some application scenarios, there may also be a situation where a hard disk is offline. If a hard disk on the hash ring 40 is offline, the query requests of the hard disk will be assigned to the next hard disk on the hash ring 40 in the clockwise direction of the hard disk. After the offline hard disk is back online, the hard disk-related data is copied back to the hard disk that is back online. For example, when the second hard disk 432 is offline, the first online hard disk in the clockwise direction on the hash ring 40 for the second query request 412 is the third hard disk 433, and the second query request 412 is routed to the third hard disk 433 for processing. After the second hard disk 432 is back online, the data related to the second hard disk 432 is copied back to the second hard disk 432 that is back online.

[0095] Due to the limitation of cache space of each hard disk, in order to ensure the performance and stability of the hard disk, the target cache space in the hard disk needs to be deleted. In some embodiments, the method for deleting the target cache space includes: in response to the target cache space satisfying the deletion condition, deleting at least one cache partition data of the target cache space. For example, the deletion condition may be that the storage capacity of the target cache space reaches the target upper limit value. For another example, a preliminary upper limit of the storage capacity is set for the target cache space. When the storage capacity of the target cache space reaches the preliminary upper limit value, it is determined whether the system memory of the current node is redundant. If so, the preliminary upper limit of the target cache space is increased by a certain percentage. For example, the storage capacity is increased by 10% on the basis of the original storage capacity to obtain the final upper limit value. If the storage capacity of the target cache space is further increased to the final upper limit value, the target storage space meets the deletion condition.

[0096] In some embodiments, for a target cache space that meets the deletion condition, deleting at least one cache partition data in the target cache space includes: determining the deletion order of each cache partition data in the target cache space according to the weight of each cache partition data in the target cache space. The cache partition data in the target cache space is deleted in the deletion order until the number of deleted cache partition data reaches the target number or the target cache space no longer meets the deletion condition. Specifically, the weight of the cache partition data is determined by the number of occurrences of the historical query request corresponding to the cache partition data within a third time period. Among them, the third time period can be from the moment the cache partition data is cached to the target storage space to the current judgment moment, or the third time period can be a preset time period, the more times it appears, the higher the weight of the corresponding cache partition data, and the cache partition data is deleted according to the order of weight from low to high.

[0097] In a specific embodiment, if Figure 5 As shown, Figure 5 1 is a schematic diagram of cache partition data hits in an embodiment of the data processing method of the present application, wherein the target cache space 50 stores cache partition data A51, cache partition data B52 and cache partition data C53. Since time t1, the target cache partition has experienced a total of five query requests, wherein the first query request, the second query request and the sixth query request all correspond to cache partition data A51, that is, since time t1, the number of hits of cache partition data A51 is 3, and the weight of cache partition data A51 is recorded as 3. The third query request and the fifth query request all correspond to cache partition data B52, that is, since time t1, the number of hits of cache partition data B52 is 2, and the weight of cache partition data B52 is recorded as 2. The fourth query request corresponds to cache partition data C53, and the weight of cache partition data B52 is recorded as 1. Therefore, the deletion order of the cache partition data of the target cache space 50 is, in descending order according to the weight, cache partition data C53, cache partition data B52 and cache partition data A51. The cache partition data with the lowest hit count since the calculation time is eliminated first, even if the cache partition data is the most recent hit data. Dynamic management of the storage capacity in the target cache space 50 is achieved, ensuring the cache hit rate while maintaining the smooth operation of the system.

[0098] In the above scheme, the present application caches each partition data of the database historically queried independently in the target cache space as different cache partition data. When the current query request is received, the matching cache partition data is directly queried from the target cache space as the query result of the current query request. That is, for the partition data that has been historically queried, the corresponding cache partition data can be directly read from the cache space without the need to re-query from the database, thereby reducing the query resource overhead and query time, thereby improving the data query performance.

[0099] See also Figure 6 , Figure 6 It is a schematic diagram of the framework of an embodiment of the data processing device of the present application. The video processing device 60 includes: a receiving module 61 and a query module 62. The receiving module 61 is used to receive a current query request to a database, wherein the database includes multiple partitions, and the current query request includes at least one partition condition, and each partition condition is used to query data from a partition. The query module 62 is used to search for cache partition data that matches the partition condition from the target cache space for each partition condition, as the matching partition data corresponding to the partition condition, wherein each cache partition data is cached separately in the target cache space, and each cache partition data is data in a partition that matches the historical query request. The found matching partition data is used as the query result of the current query request.

[0100] In some embodiments, the query module 62 is also used to, before searching for cache partition data that matches the partition condition from the target cache space for each partition condition as the matching partition data corresponding to the partition condition, also include: splitting the current query request into at least one sub-request, each sub-request containing a partition condition; for each partition condition, searching for cache partition data that matches the partition condition from the target cache space as the matching partition data corresponding to the partition condition, including: for each sub-request, searching for cache partition data that matches the sub-request from the target cache space as the matching partition data corresponding to the partition condition in the sub-request.

[0101] In some embodiments, the target cache space caches historical type characterization values ​​of historical query requests corresponding to each cache partition data; the query module 62 is also used to search the target cache space for cache partition data that matches the sub-request as matching partition data corresponding to the partition condition in the sub-request, including: determining the current type characterization value of the sub-request; searching from each historical type characterization value cached in the target cache space whether there is a matching historical type characterization value consistent with the current type characterization value; in response to the existence of a matching historical type characterization value, searching from each cache partition data corresponding to the matching historical type characterization value for cache partition data that matches the partition condition corresponding to the sub-request as matching partition data corresponding to the partition condition in the sub-request.

[0102] In some embodiments, each partition condition is divided by the query field of the current query request, and the query field includes at least the partition field. The query module 62 is also used to determine the current type characterization value of the sub-request, including: removing the partition field corresponding to the sub-request, and asymmetrically encrypting the removed sub-request to obtain the encryption result of the sub-request as the current type characterization value. In some application scenarios, the cache partition data corresponding to each historical type characterization value is cached in a key-value pair data structure, wherein each historical type characterization value is a first key, and each cache partition data corresponding to the historical type characterization value is used as the first value corresponding to the first key.

[0103] In some embodiments, each cache partition data in the first value is cached in a key-value pair data structure, wherein, in a first value, the partition information corresponding to each partition is the second key, the cache partition data corresponding to the partition is the second value corresponding to the second key, and the partition information includes at least one of the partition name, partition version number, and partition version update time.

[0104] In some embodiments, the method also includes: the query module 62 responds to the partition condition being an unmatched condition, and respectively searches out the original partition data that matches the unmatched condition from multiple partitions of the database as the matching partition data corresponding to the unmatched condition, wherein the unmatched condition is a target partition condition in which there is no matching cache partition data in the target cache space; and caches the matching partition data corresponding to the unmatched condition as new cache partition data in the target cache space.

[0105] In some embodiments, the multiple partitions include at least one cold partition and at least one hot partition. Each cache partition data of the target cache space includes cache partition data corresponding to the cold partition and cache partition data corresponding to the first hot partition. The query module 62 caches the matching partition data that does not match the matching condition as new cache partition data in the target cache space, including: in response to the partition corresponding to the matching partition data being a cold partition or a first hot partition, the matching partition data is cached as new cache partition data in the target cache space, wherein the first hot partition is a hot partition that is not currently in a frequently updated state; in response to the partition condition being an unmatched condition, after searching for matching partition data that does not match the matching condition from multiple partitions of the database, it also includes: in response to the partition corresponding to the matching partition data being a second hot partition, the matching partition data is not cached, and the second hot partition is a hot partition that is currently in a frequently updated state.

[0106] In some embodiments, multiple partitions include at least one cold partition; the method also includes: the query module 62 updates the cache partition data of the cold partition where the data update occurs in response to a data update occurring in a cold partition in the database, and corresponding cache partition data exists in the cold partition; searches for cache partition data that matches the partition condition from the target cache space as matching partition data corresponding to the partition condition, including: searching for cache partition data that matches the partition condition from the target cache space; in response to the existence of cache partition data that matches the partition condition in the target cache space, and the partition condition is the first partition condition corresponding to the cold partition, reads the cache partition data that matches the first partition condition as the matching partition data corresponding to the first partition condition.

[0107] In some application scenarios, multiple partitions include at least one hot partition; the method further includes: the query module 62 is also used to respond to a hot partition in the database having data update, and the hot partition has corresponding cache partition data, and the cache partition data of the hot partition where the data update occurs is not updated; searching for cache partition data matching the partition condition from the target cache space as matching partition data corresponding to the partition condition, including: searching for cache partition data matching the partition condition from the target cache space; the query module 62 is also used to respond to the existence of cache partition data matching the partition condition in the target cache space, and the partition condition is the second partition condition corresponding to the first hot partition, and after updating the cache partition data matching the second partition condition in the target cache space, use it as matching partition data corresponding to the second partition condition, wherein the first hot partition is a hot partition that is not currently in a frequently updated state; in response to the existence of cache partition data matching the partition condition in the target cache space, and the partition condition is the third partition condition corresponding to the second hot partition, searching for original partition data matching the third partition condition from the second hot partition of the database as matching partition data corresponding to the third partition condition, wherein the second hot partition is a hot partition that is currently in a frequently updated state.

[0108] In some application scenarios, multiple partitions include at least one hot partition; the method further includes: the query module 62 is also used to update the cache partition data of the first hot partition where the data is updated in response to a data update of a first hot partition in the database, and the first hot partition has corresponding cache partition data, wherein the first hot partition is a hot partition that is not currently in a frequently updated state. The query module 62 is also used to delete the cache partition data of the second hot partition where the data is updated, or set the cache partition data of the second hot partition where the data is updated to an invalid state in response to a data update of a second hot partition in the database, and the second hot partition has corresponding cache partition data, wherein the cache partition data in the invalid state will no longer be read, and will be restored to a valid state when an update occurs next time, and the second hot partition is a hot partition that is currently in a frequently updated state; searching for cache partition data that matches the partition condition from the target cache space as the matching partition data corresponding to the partition condition, including: searching for cache partition data that matches the partition condition from the target cache space. The query module 62 is also used for responding to the existence of valid cache partition data matching the partition condition in the target cache space, and the partition condition is the fourth partition condition corresponding to the hot partition, reading the cache partition data matching the fourth partition condition as the matching partition data corresponding to the fourth partition condition; or, responding to the existence of invalid cache partition data matching the partition condition in the target cache space, and the partition condition is the fourth partition condition corresponding to the hot partition, finding the original partition data matching the fourth partition condition from the corresponding hot partition of the database as the matching partition data corresponding to the fourth partition condition. The hot partition in the frequently updated state is a hot partition whose update time interval is less than the first time period, or a hot partition that has been updated in the most recent second time period.

[0109] In some embodiments, the method further includes: the query module 62 is also used to delete at least one cache partition data in the target cache space in response to the target cache space satisfying the deletion condition. Wherein, the deletion condition includes that the storage capacity of the target cache space reaches the target upper limit value; and / or, deleting at least one cache partition data in the target cache space, including: determining the deletion order of each cache partition data in the target cache space according to the weight of each cache partition data in the target cache space; deleting the cache partition data in the target cache space according to the deletion order until the number of deleted cache partition data reaches the target number or the target cache space no longer satisfies the deletion condition. In some specific embodiments, the weight of the cache partition data is determined by the number of occurrences of the historical query request corresponding to the cache partition data within the third time period.

[0110] In some embodiments, the data of the database is stored in partitions of several hard disks, and the query module 62 is also used to assign query requests of the same type to the same hard disk for processing; and / or, the data of the database is stored in partitions of several hard disks, wherein the target cache space includes cache space in processes of several hard disks, wherein the hard disk to which the historical query request is assigned is the request assigned hard disk, and the cache partition data corresponding to the historical query request is cached in the cache space in the process of the request assigned hard disk; in some application scenarios, the routing forwarding of the current query request and the historical query request is implemented by consistent hashing; in some application scenarios, each partition condition is divided according to the query field of the current query request, and the query field at least includes the partition field.

[0111] In the above scheme, the present application caches each partition data of the database historically queried independently in the target cache space as different cache partition data. When the current query request is received, the matching cache partition data is directly queried from the target cache space as the query result of the current query request. That is, for the partition data that has been historically queried, the corresponding cache partition data can be directly read from the cache space without the need to re-query from the database, thereby reducing the query resource overhead and query time, thereby improving the data query performance.

[0112] See also Figure 7 , Figure 7 It is a framework diagram of an implementation of an electronic device of the present application. The electronic device 70 includes a memory 72 and a processor 71. The memory 72 is coupled to the processor 71. Program instructions are stored in the memory 72. The processor 71 can be any device with processing capabilities, such as a computer, a tablet computer, a mobile phone, etc.

[0113] The memory 72 is used to store the program data executed by the processor 71 and the data of the processor 71 during the processing process. For example, the amount of data, metadata, timing data, etc. are stored. Among them, the memory 72 includes a non-volatile storage part for storing the above program data. The processor 71 controls the operation of the processing device, and the processor 71 can also be called a CPU (Central Processing Unit). The processor 71 may be an integrated circuit chip with signal processing capabilities. The processor 71 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor 71, etc. In addition, the processor 71 can be implemented by multiple integrated circuit chips. The processor 71 calls the program data stored in the memory 72, and the processor 71 is used to execute program instructions to implement the data processing method of any of the above embodiments.

[0114] See also Figure 8 , Figure 8 1 is a schematic diagram of a computer-readable storage medium of the present application. In this embodiment, the storage medium 80 stores program data 81 executable by a processor, and the program data 81 can be executed to implement the data processing method of any of the above embodiments.

[0115] The storage medium 80 may specifically be a medium that can store program data, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or may be a server that stores the program data. The server may send the stored program data to other devices for execution, or may execute the stored program data itself.

[0116] In some embodiments, the storage medium 80 may also be Figure 7 The memory 72 is shown.

[0117] In the above scheme, the present application caches each partition data of the database that has been historically queried independently in the target cache space as different cache partition data. When the current query request is received, the matching cache partition data is directly queried from the target cache space as the query result of the current query request. That is, for the partition data that has been historically queried, the corresponding cache partition data can be directly read from the cache space without the need to re-query from the database, thereby reducing the query resource overhead and query time, thereby improving the data query performance.

[0118] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Receiving a current query request to a database, wherein the database includes a plurality of partitions, the current query request includes at least one partition condition, and each of the partition conditions is used to query data from a partition; For each of the partition conditions, searching the target cache space for cache partition data that matches the partition condition as matching partition data corresponding to the partition condition, wherein each of the cache partition data is cached separately in the target cache space, and each of the cache partition data is data in a partition that matches a historical query request; The found matching partition data is used as the query result of the current query request; The multiple partitions include at least one hot partition, the at least one hot partition includes a second hot partition, and the second hot partition is a hot partition that is currently in a frequently updated state; the method further includes: In response to a data update of a second hot partition in the database, and the corresponding cache partition data exists in the second hot partition, the cache partition data of the second hot partition where the data update occurs is deleted, or the cache partition data of the second hot partition where the data update occurs is set to an invalid state, wherein the cache partition data in the invalid state will no longer be read, and will be restored to a valid state when an update occurs next time; The step of searching the target cache space for cache partition data matching the partition condition as matching partition data corresponding to the partition condition includes: Searching for cache partition data matching the partition condition from the target cache space; In response to the existence of valid cache partition data matching the partition condition in the target cache space, and the partition condition is the fourth partition condition corresponding to the second hot partition, the cache partition data matching the fourth partition condition is read as the matching partition data corresponding to the fourth partition condition; or, in response to the existence of invalid cache partition data matching the partition condition in the target cache space, and the partition condition is the fourth partition condition corresponding to the second hot partition, the original partition data matching the fourth partition condition is searched out from the corresponding second hot partition of the database as the matching partition data corresponding to the fourth partition condition.

2. The method according to claim 1, characterized in that Before searching the target cache space for cache partition data matching the partition condition for each partition condition as matching partition data corresponding to the partition condition, the method further includes: Splitting the current query request into at least one sub-request, each of the sub-requests including one of the partitioning conditions; For each of the partition conditions, searching the target cache space for cache partition data matching the partition condition as matching partition data corresponding to the partition condition includes: For each of the sub-requests, cache partition data matching the sub-request is searched from the target cache space to serve as matching partition data corresponding to the partition condition in the sub-request.

3. The method according to claim 2, characterized in that The target cache space caches the historical type representation value of the historical query request corresponding to each cache partition data; The step of searching the target cache space for cache partition data matching the sub-request as matching partition data corresponding to the partition condition in the sub-request includes: Determining a current type characterization value of the subrequest; Searching, from each historical type characterization value cached in the target cache space, whether there is a matching historical type characterization value that is consistent with the current type characterization value; In response to the existence of the matching history type characterization value, cache partition data matching the partition condition corresponding to the sub-request is searched from each cache partition data corresponding to the matching history type characterization value as the matching partition data corresponding to the partition condition in the sub-request.

4. The method according to claim 3, characterized in that Each of the partition conditions is divided according to a query field of the current query request, the query field at least includes a partition field, and the determining of the current type characterization value of the sub-request includes: Removing the partition field corresponding to the sub-request, and asymmetrically encrypting the sub-request after the removal, to obtain an encryption result of the sub-request as the current type representation value; And / or, the cache partition data corresponding to each of the historical type characterization values ​​is cached in a key-value pair data structure, wherein each of the historical type characterization values ​​is a first key, and each of the cache partition data corresponding to the historical type characterization value is a first value corresponding to the first key.

5. The method according to claim 4, characterized in that Each cache partition data in the first value is cached in a key-value pair data structure, wherein, in a first value, the partition information corresponding to each partition is the second key, the cache partition data corresponding to the partition is the second value corresponding to the second key, and the partition information includes at least one of a partition name, a partition version number, and an update time of the partition version.

6. The method according to claim 1, characterized in that The method further comprises: In response to the partition condition being an unmatched condition, respectively searching for original partition data matching the unmatched condition from multiple partitions of the database as matching partition data corresponding to the unmatched condition, wherein the unmatched condition is a target partition condition in which no matching cache partition data exists in the target cache space; The matching partition data corresponding to the unmatched conditions are respectively used as new cache partition data and cached in the target cache space.

7. The method according to claim 6, characterized in that The multiple partitions further include at least one cold partition, the at least one hot partition further includes a first hot partition, and the first hot partition is a hot partition that is not currently in a frequently updated state; each cache partition data of the target cache space includes cache partition data corresponding to the cold partition and cache partition data corresponding to the first hot partition; The step of taking the matching partition data corresponding to the unmatched condition as new cache partition data and caching them in the target cache space includes: In response to the partition corresponding to the matching partition data being a cold partition or a first hot partition, the matching partition data is used as new cache partition data and cached in the target cache space; After searching for matching partition data matching the unmatched condition from multiple partitions of the database in response to the partition condition being an unmatched condition, the method further includes: In response to the partition corresponding to the matching partition data being a second hot partition, the matching partition data is not cached.

8. The method according to claim 1, characterized in that The plurality of partitions further comprises at least one cold partition; The method further comprises: In response to a data update occurring in a cold partition in the database, and the cold partition having corresponding cache partition data, updating the cache partition data of the cold partition where the data update occurs; The step of searching the target cache space for cache partition data matching the partition condition as matching partition data corresponding to the partition condition includes: Searching for cache partition data matching the partition condition from the target cache space; In response to the existence of cache partition data matching the partition condition in the target cache space, and the partition condition being a first partition condition corresponding to a cold partition, the cache partition data matching the first partition condition is read as matching partition data corresponding to the first partition condition.

9. The method according to claim 1, characterized in that: The at least one thermal partition further comprises a first thermal partition; The method further comprises: In response to a data update of the first hot partition in the database, and the first hot partition has corresponding cache partition data, performing non-update processing on the cache partition data of the hot partition where the data update occurs; The step of searching the target cache space for cache partition data matching the partition condition as matching partition data corresponding to the partition condition includes: Searching for cache partition data matching the partition condition from the target cache space; In response to the presence of cache partition data matching the partition condition in the target cache space, and the partition condition being the second partition condition corresponding to the first hot partition, after updating the cache partition data matching the second partition condition in the target cache space, the cache partition data is used as the matching partition data corresponding to the second partition condition, wherein the first hot partition is a hot partition that is not currently in a frequently updated state.

10. The method according to claim 1, characterized in that The at least one thermal partition further comprises a first thermal partition; The method further comprises: In response to a data update of a first hot partition in the database, and the first hot partition having corresponding cache partition data, the cache partition data of the first hot partition where the data update occurs is updated, wherein the first hot partition is a hot partition that is not currently in a frequently updated state.

11. The method according to claim 1, characterized in that: The hot partition in the frequently updated state is a hot partition whose update time interval is less than the first time period, or a hot partition that has been updated in the latest second time period.

12. The method according to claim 1, characterized in that The method further comprises: In response to the target cache space satisfying a deletion condition, at least one cache partition data in the target cache space is deleted.

13. The method according to claim 12, characterized in that The deletion condition includes that the storage capacity of the target cache space reaches the target upper limit value; And / or, deleting at least one cache partition data in the target cache space includes: Determining, according to the weights of the cache partition data in the target cache space, a deletion order of the cache partition data in the target cache space; The cache partition data in the target cache space is deleted according to the deletion order until the number of deleted entries of the cache partition data reaches a target number or the target cache space no longer satisfies the deletion condition.

14. The method according to claim 13, characterized in that The weight of the cache partition data is determined by the number of occurrences of the historical query requests corresponding to the cache partition data within the third time period.

15. The method according to claim 1, characterized in that The data of the database is stored in partitions of several hard disks, and query requests of the same type are assigned to the same hard disk for processing; And / or, the data of the database is stored in partitions of several hard disks, wherein the target cache space includes cache space in processes of several hard disks, wherein the hard disk to which the historical query request is allocated is the request allocation hard disk, and the cache partition data corresponding to the historical query request is cached in the cache space in the process of the request allocation hard disk; And / or, the routing and forwarding of the current query request and the historical query request are implemented by using consistent hashing; And / or, each partition condition is divided according to a query field of the current query request, and the query field at least includes a partition field.

16. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the memory is coupled to the processor, and program instructions are stored in the memory to implement the data processing method according to any one of claims 1 to 15.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is processed and executed to implement the data processing method according to any one of claims 1 to 15.

Images