Data query method, device, equipment, computer readable medium and program product

Abstract

Embodiments of the present disclosure disclose a data query method, device, equipment, computer readable medium and program product. A specific embodiment of the method comprises: obtaining a data query time range; determining whether there is at least one primary key group corresponding to the data query time range in a time and primary key relationship table, wherein the time and primary key relationship table is a table generated based on at least one target data in a data table to be queried, and the at least one target data is data corresponding to at least one target time point; in response to determining that there is the at least one primary key group, finding out a required data set from the data table to be queried according to the at least one primary key group. The embodiment is related to big data, and can quickly and efficiently find out the required data from the data table to be queried.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to data query methods, apparatus, devices, computer-readable media, and program products. Background Technology

[0002] Currently, databases are widely used in various fields. The storage and access of massive amounts of data are mostly handled through databases. For data queries targeting massive amounts of data in a database, the typical approach is: first, to shard and partition the database; then, to search for the corresponding data in the partitioned database.

[0003] However, the inventors discovered that when using the above method for data retrieval, the following technical problems often arise:

[0004] Even with database sharding and partitioning, query speeds remain slow when dealing with massive amounts of data. Furthermore, database sharding and partitioning also incurs high deployment and maintenance costs.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide data query methods, apparatuses, devices, computer-readable media, and program products to address the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide a data query method, including: obtaining a data query time range; determining whether there exists at least one primary key group corresponding to the data query time range in a time-primary key relationship table, wherein the time-primary key relationship table is a table generated based on at least one target data in a data table to be queried, and the at least one target data is data corresponding to at least one target time point; in response to determining that there exists at least one primary key group, searching for the required dataset from the data table to be queried based on the at least one primary key group.

[0009] Optionally, the above method further includes storing the required dataset.

[0010] Optionally, the above method further includes: in response to receiving a data query request for the above data query time range, sending the stored required dataset to the data requesting end.

[0011] Optionally, the above-mentioned method of finding the required dataset from the data table to be queried based on the at least one primary key group includes: for each primary key group, performing a primary key processing step: subtracting a first value from the primary key with the smaller value in the primary key group, and adding a second value to the primary key with the larger value in the primary key group, to obtain a processed primary key group; determining a primary key set between two processed primary keys, wherein the two processed primary keys are the two primary keys included in the processed primary key group; finding the dataset corresponding to the primary key set from the data table to be queried, as the search dataset; and determining the required dataset based on the obtained search dataset set.

[0012] Optionally, the at least one target time point is a time point determined based on at least one time point, and the at least one time point is a time point determined based on a preset time granularity; and storing the required dataset includes: for each required data in the required dataset, performing a key-value pair data generation step: determining the time point corresponding to the required data; using the time point as the key and the required data as the value to generate key-value pair data for the required data; and temporarily storing the obtained key-value pair dataset.

[0013] Optionally, the aforementioned time-primary key relationship table is constructed through the following steps: determining at least one time point using a preset time granularity; selecting time points that meet preset time conditions from the time points corresponding to each data in the data table to be queried, based on the at least one time point, as target time points, to obtain at least one target time point, wherein there is a one-to-one correspondence between the at least one time point and the at least one target time point; for each of the at least one target time points, performing the time-primary key relationship data generation step: determining the primary key included in the data corresponding to the target time point; determining the time point among the at least one time points corresponding to the target time point; generating time-primary key relationship data for the data corresponding to the target time point based on the primary key and the time point; and generating the aforementioned time-primary key relationship table based on the obtained time-primary key relationship dataset.

[0014] Optionally, the method further includes: in response to determining that at least one primary key group does not exist, querying the at least one primary key group from the data table to be queried, and writing at least one time and primary key relationship data group for the at least one primary key group into the time and primary key relationship table.

[0015] Optionally, the data table to be queried is a sub-data table after the above full data table has been sharded and partitioned.

[0016] Optionally, the above full data table is sharded and partitioned using the following steps: determining the fields to be sharded and the fields to be partitioned in each field of the above full data table; sharding the above full data table according to the fields to be sharded to obtain a set of sharded data tables; and partitioning each sharded data table in the set of sharded data tables according to the fields to be partitioned to generate a set of partitioned data tables, thus obtaining a group of partitioned data tables.

[0017] Secondly, some embodiments of this disclosure provide a data query apparatus, including: an acquisition unit configured to acquire a data query time range; a determination unit configured to determine whether at least one primary key group corresponding to the data query time range exists in a time-primary key relationship table, wherein the time-primary key relationship table is a table generated based on at least one target data in a data table to be queried, and the at least one target data is data corresponding to at least one target time point; and a search unit configured to, in response to determining the existence of the at least one primary key group, search for the required dataset from the data table to be queried based on the at least one primary key group.

[0018] Optionally, the above-mentioned apparatus further includes: storing the required dataset.

[0019] Optionally, the above apparatus further includes: in response to receiving a data query request for the data query time range, sending the stored required dataset to the data requesting end.

[0020] Optionally, the lookup unit can be further configured to: for each of the at least one primary key group, perform a primary key processing step: subtract a first value from the primary key with the smaller value in the primary key group, and add a second value to the primary key with the larger value in the primary key group, to obtain a processed primary key group; determine a primary key set between two processed primary keys, wherein the two processed primary keys are the two primary keys included in the processed primary key group; find the dataset corresponding to the primary key set from the data table to be queried, as the lookup dataset; and determine the required dataset based on the obtained lookup dataset set.

[0021] Optionally, the at least one target time point is a time point determined based on at least one time point, and the at least one time point is a time point determined based on a preset time granularity; and the storage unit can be further configured to: for each required data in the required dataset, perform a key-value pair data generation step: determine the time point corresponding to the required data; use the time point as the key and the required data as the value to generate key-value pair data for the required data; and temporarily store the obtained key-value pair dataset.

[0022] Optionally, the aforementioned time-primary key relationship table is constructed through the following steps: determining at least one time point using a preset time granularity; selecting time points that meet preset time conditions from the time points corresponding to each data in the data table to be queried, based on the at least one time point, as target time points, to obtain at least one target time point, wherein there is a one-to-one correspondence between the at least one time point and the at least one target time point; for each of the at least one target time points, performing the time-primary key relationship data generation step: determining the primary key included in the data corresponding to the target time point; determining the time point among the at least one time points corresponding to the target time point; generating time-primary key relationship data for the data corresponding to the target time point based on the primary key and the time point; and generating the aforementioned time-primary key relationship table based on the obtained time-primary key relationship dataset.

[0023] Optionally, the query writing unit may be further configured to: in response to determining that at least one primary key group does not exist, query the at least one primary key group from the data table to be queried, and write at least one time and primary key relationship data group for the at least one primary key group into the time and primary key relationship table.

[0024] Optionally, the data table to be queried is a sub-data table after the above full data table has been sharded and partitioned.

[0025] Optionally, the above full data table is sharded and partitioned using the following steps: determining the fields to be sharded and the fields to be partitioned in each field of the above full data table; sharding the above full data table according to the fields to be sharded to obtain a set of sharded data tables; and partitioning each sharded data table in the set of sharded data tables according to the fields to be partitioned to generate a set of partitioned data tables, thus obtaining a group of partitioned data tables.

[0026] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0027] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0028] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0029] The various embodiments of this disclosure have the following beneficial effects: the data query methods of some embodiments of this disclosure can quickly and efficiently retrieve the required data from the data table to be searched. Specifically, the reason for the inefficient retrieval of required data is that, even with database sharding and table partitioning, the query speed remains slow for massive amounts of data. Furthermore, database sharding and table partitioning result in high deployment and maintenance costs. Based on this, the data query methods of some embodiments of this disclosure first obtain the data query time range for subsequent querying of the required dataset. Next, by determining whether at least one primary key group corresponding to the data query time range exists in the time-primary key relationship table, it is determined whether to utilize the time-primary key relationship table to assist in querying the required dataset. Finally, in response to the determination that at least one primary key group exists, the required dataset can be efficiently retrieved from the data table to be queried based on the at least one primary key group. Here, by using at least one primary key group corresponding to the data query time range, the data query range can be effectively shortened from the data range corresponding to the data table to be queried to the query range corresponding to at least one primary key group, thereby greatly improving the query efficiency of the required data. Attached Figure Description

[0030] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0031] Figure 1 This is a schematic diagram illustrating an application scenario of a data query method according to some embodiments of the present disclosure;

[0032] Figure 2 This is a flowchart of some embodiments of the data query method according to this disclosure;

[0033] Figure 3 This is a flowchart of some other embodiments of the data query method according to this disclosure;

[0034] Figure 4These are schematic diagrams illustrating the structure of some embodiments of the data query device according to this disclosure;

[0035] Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0036] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0037] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0038] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0039] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0040] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0041] Before performing any of the operations involving the collection, storage, or use of user personal information (such as user data in the database) disclosed in this disclosure, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, informing personal information subjects, and obtaining prior authorization and consent from personal information subjects.

[0042] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0043] Figure 1 This is a schematic diagram illustrating an application scenario of a data query method according to some embodiments of the present disclosure.

[0044] exist Figure 1In this application scenario, firstly, the electronic device 101 can obtain the data query time range 102. In this application scenario, the data query time range 102 can be "2022-04-01 01:03:05~2022-04-01 01:05:07". Then, the electronic device 101 can determine whether there exists at least one primary key group 104 in the time and primary key relationship table 103 that corresponds to the above data query time range 102. The time and primary key relationship table 103 is a table generated based on at least one target data in the data table 105 to be queried. The at least one target data is data corresponding to at least one target time point. In this application scenario, the at least one primary key group 104 is "(13, 16)". The datasets in the primary key relationship table 103 may include: “2022-04-01 01:01:00:11”, “2022-04-01 01:02:00:12”, “2022-04-01 01:03:00:13”, “2022-04-01 01:04:00:14”, “2022-04-01 01:05:00:15”, “2022-04-01 01:06:00:16”, “2022-04-01 01:07:00:17”, “2022-04-01 01:08:00:18”, and “2022-04-01 01:09:00:19”.The data sets in the data table 105 to be queried may include: "2022-04-01 01:03:06:16 0123** brand", "2022-04-01 01:03:07:17 0124** brand", "2022-04-01 01:03:08:18 0125** brand", "2022-04-01 01:03:09:19 0126** brand"…"2022-04-01 01:03:01:11 0227** brand", "2022-04-01 01:03:02:12 0228** brand", "2022-04-01 01:03:03:13 0229** brand", "2022-04-01 01:03:04:14 0230** brand", "2022-04-01 01:03:05:15 0231** brand", "2022-04-01 01:03:06:16 0232** brand", "2022-04-01 01:03:07:17 0233** brand", "2022-04-01 01:03:08:18 0234** brand", "2022-04-01 01:03:09:19 0235** brand"…"2022-05-03 02:03:01:111 0323** brand", "2022-05-01 02:03:02:112 0324** brand", "2022-04-01 01:03:03:13 0325** brand", "2022-04-01 0I:03:04:14 0326** brand", "2022-04-01 01:03:05:15 0327** brand". At least one target time point may include: "2022-04-01 01:03:00" and "2022-04-01 01:06:00". Finally, in response to determining the existence of the above at least one primary key group 104, the electronic device 101 finds the required data set 106 from the above data table 105 to be queried according to the above at least one primary key group 104. In this application scenario, the required data set 106 may include: "2022-04-01 01:03:03:13 0229** brand", "2022-04-01 01:03:04:14 0230** brand", "2022-04-01 01:03:05:15 0231** brand".

[0045] It should be noted that the aforementioned electronic device 101 can be either hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0046] It should be understood that Figure 1 The number of electronic devices shown is merely illustrative. Any number of electronic devices can be used depending on the implementation requirements.

[0047] Continue to refer to Figure 2 The flowchart 200 illustrates some embodiments of the data query method according to this disclosure. The data query method includes the following steps:

[0048] Step 201: Obtain the data query time range.

[0049] In some embodiments, the execution entity of the above data query method (e.g. Figure 1 The electronic device 101 shown can obtain the data query time range via a wired or wireless connection. The data query time range is the time range corresponding to the required dataset. Each data point in the required dataset has a corresponding data time. In practice, the data time can be the time the corresponding data was added to the database or the time the corresponding data was generated. This is not limited here, and the specific data time can vary depending on different application scenarios.

[0050] It should be noted that the above data query time range can include multiple sub-time ranges, and the time within each sub-time range is continuous. For example, if the data query time range is "April 1, 2011, 12:00 PM - April 2, 2011, 3:00 PM, and April 3, 2011, 12:00 PM - April 10, 2011, 3:00 AM", then the data query time range includes two sub-time ranges. These two sub-time ranges are: "April 1, 2011, 12:00 PM - April 2, 2011, 3:00 PM" and "April 3, 2011, 12:00 PM - April 10, 2011, 3:00 AM".

[0051] Step 202: Determine whether there exists at least one primary key group in the time and primary key relationship table that corresponds to the time range of the data query mentioned above.

[0052] In some embodiments, the execution entity can determine whether there exists at least one primary key group in the time-primary key relationship table corresponding to the data query time range. The time-primary key relationship table is generated based on at least one target data in the data table to be queried, and the at least one target data corresponds to at least one target time point. The time-primary key relationship table represents the correspondence between data time and primary key. The time-primary key relationship table can be a relational table in a database. The primary key can be a primary key. The primary key is one or more fields in the table. Here, the data query time range includes at least one sub-range, and at least one sub-range has a one-to-one correspondence with at least one primary key group. For example, at least one sub-range includes: "April 1, 2011, 12:00 PM - April 2, 2011, 3:00 PM", and "April 3, 2011, 12:00 PM - April 10, 2011, 3:00 AM". The primary key group corresponding to "April 1, 2011, 12:00 PM - April 2, 2011, 3:00 PM" can be (123, 345). The primary key group corresponding to "April 3, 2011, 12:00 PM - April 10, 2011, 3:00 AM" can be (231, 445). At least one target time point can be a time point determined based on a preset time granularity. This preset time granularity can be a time-based indicator for dividing the data table to be queried. For example, the preset time granularity can be hourly, dayly, weekly, monthly, or yearly. For a preset time granularity of hourly, the at least one target time point can be any full hour. For example, November 1, 2011, 12:00 PM, November 1, 2011, 2:00 PM, and November 1, 2011, 3:00 PM. For a preset time granularity of dayly, the at least one target time point can be 24:00 on each day. For example, 24:00 on November 1, 2011; 24:00 on November 1, 2011; 24:00 on November 1, 2011; 24:00 on November 1, 2011. For a preset time granularity of months, at least one of the above target time points can be the last day of each month. For example, November 30, 2011; December 31, 2011; January 31, 2012. For a preset time granularity of years, at least one of the above target time points can be the last day of each year. For example, December 31, 2011; December 31, 2012; December 31, 2013; December 31, 2014.

[0053] It should be noted that the data in the query table can be in various formats, such as image formats. The data in the query table can also be data specific to the target application scenario. For example, the data in the query table could be stored data from historical surveillance videos of autonomous vehicles.

[0054] As an example, the above time-primary key relationship table can be processed through the following steps:

[0055] The first step is to determine the primary key and data time corresponding to each of the at least one set of target data.

[0056] The second step is to store the correspondence between the primary key set and the data time set in an empty table to obtain the time and primary key relationship table.

[0057] In some optional implementations of certain embodiments, the above time-primary key relationship table is constructed through the following steps:

[0058] The first step is to determine at least one point in time by setting a preset time granularity. This preset time granularity can be flexibly set. In practice, the preset time granularity can be one of the following: hourly granularity, dayly granularity, weekly granularity, monthly granularity, or yearly granularity.

[0059] As an example, for a preset time granularity of days, at least one point in time can be a fixed hour of each day within a certain time range. For example, the certain time range is 2011-2012. The corresponding at least one point in time includes: 4:00 AM on January 1, 2011; 4:00 AM on January 2, 2011; 4:00 AM on January 3, 2011… 4:00 AM on December 30, 2012; and 4:00 AM on December 31, 2012.

[0060] The second step involves selecting, based on the aforementioned at least one time point, time points that meet preset time conditions from the time points corresponding to each data point in the aforementioned data table to be queried, as target time points, thus obtaining at least one target time point. There is a one-to-one correspondence between the aforementioned at least one time point and the aforementioned at least one target time point. Each data point has a one-to-one corresponding data time (i.e., time point). The aforementioned preset time condition can be at least one target time point among the aforementioned time points that is closest to at least one time point. Furthermore, at least one time point and at least one target time point are in a one-to-one correspondence. Each target time point is earlier than its corresponding time point.

[0061] For example, at least one time point includes: 4:00 AM on January 1, 2011, 4:00 AM on January 2, 2011, 4:00 AM on January 3, 2011… 4:00 AM on December 30, 2012, and 4:00 AM on December 31, 2012. Other time points include: 3:00 AM on January 1, 2011, 6:00 AM on January 1, 2011, 10:00 AM on January 1, 2011, 2:00 AM on January 2, 2011, 2:00 AM on January 3, 2011, and 6:00 AM on January 3, 2011. Therefore, at least one target time point includes: 3:00 AM on January 1, 2011, and 2:00 AM on January 3, 2011. Among these, 3:00 AM on January 1, 2011 is earlier than 4:00 AM on January 3, 2011, and is closest to 4:00 AM on January 3, 2011. 2:00 AM on January 3, 2011 is earlier than 4:00 AM on January 3, 2011, and is closest to 4:00 AM on January 3, 2011.

[0062] The third step, for each of the at least one target time point mentioned above, is to execute the time-primary key relationship data generation step:

[0063] The first sub-step is to determine the primary key included in the data corresponding to the target time point mentioned above. The primary key can be a pre-defined field within the data.

[0064] As an example, the aforementioned executing entity can determine the primary key included in the data corresponding to the target time point by querying.

[0065] For example, the target time point is 3:00 AM on January 1, 2011. The corresponding primary key is 110.

[0066] The second sub-step is to determine the time point corresponding to the target time point mentioned above, which is one of the at least one of the aforementioned time points.

[0067] As an example, the target time is 3:00 AM on January 1, 2011. The corresponding time for 3:00 AM on January 1, 2011 is 4:00 AM on January 3, 2011.

[0068] The third sub-step involves generating time-primary key relationship data for the data corresponding to the target time point, based on the aforementioned primary key and time point.

[0069] As an example, the aforementioned execution entity can combine the primary key and the time point into a tuple to generate the time-primary key relationship data of the corresponding data.

[0070] For example, if the time point is 4:00 AM on January 3, 2011, and the primary key is 110, then the relationship between time and primary key data can be (4:00 AM on January 3, 2011, 110).

[0071] The fourth step is to generate the aforementioned time and primary key relationship table based on the obtained time and primary key relationship dataset.

[0072] As an example, the aforementioned execution entity can generate the aforementioned time and primary key relationship table by adding the time and primary key relationship dataset to an empty table.

[0073] Step 203: In response to determining that at least one primary key group exists, the required dataset is retrieved from the data table to be queried based on the at least one primary key group.

[0074] In some embodiments, in response to determining the existence of at least one primary key group, the executing entity can retrieve the required dataset from the query table based on the at least one primary key group. The query table stores information about each data field corresponding to each piece of data in the required dataset. Furthermore, the amount of data in the query table is greater than or equal to the amount of data in the required dataset. The query table is a table in a database.

[0075] As an example, firstly, in response to determining the existence of at least one primary key group, for each primary key group, the two primary keys included in the primary key group are determined. Then, the executing entity can determine the two data corresponding to the two primary keys as two target data. Next, the executing entity can determine the two data times corresponding to the two target data as two target data times. Further, the executing entity can filter the dataset within the target time range from the data table to be queried. The target time range consists of two target data times. For example, if the two target time ranges are: April 12, 2011, 12:00 PM and April 12, 2011, 3:00 PM, then the target time range is [April 12, 2011, 12:00 PM, April 12, 2011, 3:00 PM]. Finally, the executing entity can find the required dataset from the dataset.

[0076] In some alternative implementations of certain embodiments, the aforementioned execution entity may store the required dataset.

[0077] As an example, the aforementioned executing entity can store the required dataset in a pre-established target database. This target database is used to store historical required datasets over a certain period of time.

[0078] It should be noted that caching the required database can effectively improve the efficiency of subsequent data retrieval.

[0079] As another example, the aforementioned execution entity can populate the required dataset into a blank table to obtain the populated table. Then, the populated table is stored in a pre-established target database.

[0080] Optionally, in response to receiving a data query request for the aforementioned data query time range, the executing entity may send the stored required dataset to the data requesting end. The data query request can be a request sent by the data requesting end for querying data. In practice, the data requesting end can be an upstream server.

[0081] Optionally, the above-mentioned at least one target time point is a time point determined based on at least one time point, and the above-mentioned at least one time point is a time point determined based on a preset time granularity.

[0082] Storing the required dataset as described above may include the following steps:

[0083] The first step is to perform a key-value pair data generation step for each piece of data in the above-mentioned dataset:

[0084] Sub-step 1: The aforementioned executing entity can determine the time point corresponding to the required data. Specifically, the at least one target time point is a time point determined based on at least one other time point, and the at least one time point is a time point determined based on a preset time granularity. Further details are omitted; see relevant explanations. Figure 2 Corresponding implementation examples.

[0085] As an example, the aforementioned executing entity can use a point-in-time query method to determine the point in time corresponding to the required data.

[0086] Sub-step 2: The executing entity can use the time point as the key and the required data as the value to generate key-value pair data for the required data.

[0087] The second step involves the aforementioned executing entity temporarily storing the obtained key-value pair dataset. This temporary storage can be for a predetermined duration. That is, if the key-value pair data is stored for more than the predetermined duration, it is deleted.

[0088] Here, by storing the required dataset in the form of key-value pairs, the query efficiency of the required dataset can be greatly improved for secondary queries within a certain period of time.

[0089] In some alternative implementations of some embodiments, in response to determining that at least one primary key group does not exist, the execution entity may query the at least one primary key group from the data table to be queried, and write at least one time and primary key relationship data group for the at least one primary key group into the time and primary key relationship table.

[0090] In some optional implementations of certain embodiments, the data table to be queried is a sub-data table resulting from database sharding and table partitioning of the full data table. The full data table can be all data corresponding to the scenario for which the required dataset is located. For example, if the scenario for which the required dataset is located is a face recognition scenario, the corresponding full data table stores all datasets for that face recognition scenario. The data table to be queried is simply a sub-table within the full data table.

[0091] In some optional implementations of certain embodiments, the above-mentioned full data table is processed through database sharding and table partitioning using the following steps:

[0092] The first step is to determine the fields to be sharded in the database and the fields to be sharded in the corresponding fields of the full data table. For example, the fields include: "Camera Location Area ID", "Camera ID", "File Name", "File Size", "Video Start Time", "Video End Time", and "Data Identifier". The field to be sharded in the database could be "Camera Location Area ID". The field to be sharded in the table could be "Camera ID".

[0093] The second step is to perform database sharding on the full data table based on the fields to be sharded, resulting in a sharded data table set.

[0094] The third step is to perform table partitioning on each of the database tables in the database partitioning set according to the fields to be partitioned, so as to generate a partitioned data table set and obtain a partitioned data table set group.

[0095] The various embodiments of this disclosure have the following beneficial effects: the data query methods of some embodiments of this disclosure can quickly and efficiently retrieve the required data from the data table to be searched. Specifically, the reason for the inefficient retrieval of required data is that, even with database sharding and table partitioning, the query speed remains slow for massive amounts of data. Furthermore, database sharding and table partitioning result in high deployment and maintenance costs. Based on this, the data query methods of some embodiments of this disclosure first obtain the data query time range for subsequent querying of the required dataset. Next, by determining whether at least one primary key group corresponding to the data query time range exists in the time-primary key relationship table, it is determined whether to utilize the time-primary key relationship table to assist in querying the required dataset. Finally, in response to the determination that at least one primary key group exists, the required dataset can be efficiently retrieved from the data table to be queried based on the at least one primary key group. Here, by using at least one primary key group corresponding to the data query time range, the data query range can be effectively shortened from the data range corresponding to the data table to be queried to the query range corresponding to at least one primary key group, thereby greatly improving the query efficiency of the required data.

[0096] Further reference Figure 3 The diagram illustrates a flow 300 of another embodiment of the data query method according to this disclosure. This data query method includes the following steps:

[0097] Step 301: Obtain the data query time range.

[0098] In some embodiments, the specific implementation of step 301 and its resulting technical effects can be found in [reference needed]. Figure 2 Step 201 in the corresponding embodiment will not be repeated here.

[0099] Step 302: For each primary key group in at least one of the above primary key groups, perform the primary key processing steps:

[0100] Step 3021: Subtract the first value from the primary key with the smaller value in the above primary key group, and add the second value to the primary key with the larger value in the above primary key group to obtain the processed primary key group.

[0101] In some embodiments, the executing entity (e.g. Figure 1 The electronic device 101 shown can subtract a first value from the smaller primary key in the aforementioned primary key group and add a second value to the larger primary key in the aforementioned primary key group to obtain a processed primary key group. The first and second values ​​can be preset. For example, if the primary key group is (123, 234), the first value is 23, and the second value is 50, then the processed primary key group is (100, 284).

[0102] Step 3022: Determine the set of primary keys that lies between the two processed primary keys.

[0103] In some embodiments, the executing entity may determine a set of primary keys between two processed primary keys. The two processed primary keys are the two primary keys included in the processed primary key group.

[0104] As an example, the aforementioned execution entity can determine the set of primary keys between two processed primary keys by using a primary key query.

[0105] Step 3023: Find the dataset corresponding to the primary key set from the above-mentioned data table to be queried, and use it as the search dataset.

[0106] In some embodiments, the execution entity may retrieve the dataset corresponding to the primary key set from the data table to be queried, and use it as the lookup dataset.

[0107] As an example, the aforementioned execution entity can use data querying to find the dataset corresponding to the aforementioned primary key set from the aforementioned data table to be queried, and use it as the search dataset.

[0108] Step 303: Determine the required dataset based on the obtained query dataset group.

[0109] In some embodiments, the execution entity may determine the required dataset based on the obtained query dataset group.

[0110] As an example, the aforementioned execution entity can query the required dataset from the obtained query dataset group.

[0111] Here, querying the required dataset from the dataset group is much more efficient than querying the dataset directly from the data table to be queried.

[0112] Step 304: In response to determining that at least one primary key group exists, the required dataset is retrieved from the data table to be queried based on the at least one primary key group.

[0113] Step 305: Store the required dataset.

[0114] In some embodiments, the specific implementation of steps 304-305 and the resulting technical effects can be found in [reference needed]. Figure 2 Steps 203-204 in the corresponding embodiments will not be repeated here.

[0115] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3 The data query method flow 300 disclosed in some corresponding embodiments expands the primary key of each primary key group in at least one primary key group, which can avoid data loss.

[0116] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a data query device, which are similar to... Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0117] like Figure 4As shown, a data query device 400 includes: an acquisition unit 401, a determination unit 402, and a search unit 403. The acquisition unit 401 is configured to acquire a data query time range; the determination unit 402 is configured to determine whether at least one primary key group corresponding to the data query time range exists in a time-primary key relationship table, wherein the time-primary key relationship table is a table generated based on at least one target data in the data table to be queried, and the at least one target data is data corresponding to at least one target time point; the search unit 403 is configured to, in response to determining the existence of the at least one primary key group, search for the required dataset from the data table to be queried based on the at least one primary key group.

[0118] In some optional implementations of certain embodiments, the apparatus 400 may further include a storage unit (not shown in the figure). The storage unit may be configured to store the aforementioned required dataset.

[0119] In some optional implementations of certain embodiments, the apparatus 400 may further include a sending unit (not shown in the figure). The sending unit may be configured to send the stored required dataset to the data requesting end in response to receiving a data query request for the aforementioned data query time range.

[0120] In some optional implementations of certain embodiments, the lookup unit 403 in the device 400 may be further configured to: for each of the at least one primary key group, perform a primary key processing step: subtract a first value from the primary key with the smaller value in the primary key group, and add a second value to the primary key with the larger value in the primary key group, to obtain a processed primary key group; determine a primary key set between two processed primary keys, wherein the two processed primary keys are the two primary keys included in the processed primary key group; find the dataset corresponding to the primary key set from the data table to be queried, as the lookup dataset; and determine the required dataset based on the obtained lookup dataset set.

[0121] In some optional implementations of some embodiments, the at least one target time point is a time point determined based on at least one time point, and the at least one time point is a time point determined based on a preset time granularity; and the storage unit can be further configured to: for each required data in the required dataset, perform a key-value pair data generation step: determine the time point corresponding to the required data; use the time point as the key and the required data as the value to generate key-value pair data for the required data; and temporarily store the obtained key-value pair dataset.

[0122] In some optional implementations of certain embodiments, the aforementioned time-primary key relationship table is constructed through the following steps: determining at least one time point by a preset time granularity; selecting time points that meet preset time conditions from the time points corresponding to each data in the data table to be queried, based on the at least one time point, as target time points, to obtain at least one target time point, wherein there is a one-to-one correspondence between the at least one time point and the at least one target time point; for each of the at least one target time points, performing the time-primary key relationship data generation step: determining the primary key included in the data corresponding to the target time point; determining the time point among the at least one time points corresponding to the target time point; generating time-primary key relationship data for the data corresponding to the target time point based on the primary key and the time point; and generating the aforementioned time-primary key relationship table based on the obtained time-primary key relationship dataset.

[0123] In some optional implementations of certain embodiments, the apparatus 400 further includes a query and write unit (not shown in the figure). The query and write unit can be configured to: in response to determining that at least one primary key group does not exist, query the at least one primary key group from the data table to be queried, and write at least one time-primary key relationship data group for the at least one primary key group into the time-primary key relationship table.

[0124] In some optional implementations of certain embodiments, the data table to be queried is a sub-data table after the full data table has been partitioned and sharded.

[0125] In some optional implementations of certain embodiments, the aforementioned full data table is sharded and partitioned through the following steps: determining the fields to be sharded and the fields to be partitioned in each field corresponding to the aforementioned full data table; sharding the aforementioned full data table according to the fields to be sharded to obtain a set of sharded data tables; and partitioning each sharded data table in the aforementioned set of sharded data tables according to the fields to be partitioned to generate a set of partitioned data tables, thus obtaining a group of partitioned data tables.

[0126] It is understandable that the units described in the device 400 are related to the reference. Figure 2 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to device 400 and the units contained therein, and will not be repeated here.

[0127] The following is for reference. Figure 5 It illustrates electronic devices suitable for implementing some embodiments of this disclosure (e.g., Figure 1 A schematic diagram of the structure of electronic device 101)500. Figure 5The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0128] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0129] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.

[0130] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 509, or installed from storage device 508, or installed from ROM 502. When the computer program is executed by processing device 501, it performs the functions defined in the methods of some embodiments of this disclosure.

[0131] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0132] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0133] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: obtain a data query time range; determine whether a time-primary key relationship table exists that corresponds to at least one primary key group of the data query time range, wherein the time-primary key relationship table is a table generated based on at least one target data in the data table to be queried, and the at least one target data is data corresponding to at least one target time point; and, in response to determining the existence of the at least one primary key group, retrieve the required dataset from the data table to be queried based on the at least one primary key group.

[0134] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0136] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor can be described as including: an acquisition unit, a determination unit, and a search unit. The names of these units do not necessarily limit the unit itself; for example, an acquisition unit can also be described as "a unit that acquires a data query time range".

[0137] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0138] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the data query methods described above.

[0139] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A data query method, comprising: Get the time range for data queries; Determine whether there exists at least one primary key group in the time and primary key relationship table that corresponds to the data query time range, wherein the time and primary key relationship table is a table generated based on at least one target data in the data table to be queried, and the at least one target data is data whose corresponding time is at least one target time point; In response to determining that at least one primary key group exists, the required dataset is retrieved from the data table to be queried based on the at least one primary key group; The time-primary key relationship table is constructed through the following steps: based on at least one determined time point, time points that meet preset time conditions are selected from the time points corresponding to each data in the data table to be queried, and these are used as target time points to obtain at least one target time point, wherein there is a one-to-one correspondence between the at least one time point and the at least one target time point; for each of the at least one target time points, the time-primary key relationship data generation step is performed.

2. The method according to claim 1, wherein, The method further includes: Store the required dataset.

3. The method according to claim 2, wherein, The method further includes: In response to receiving a data query request for the specified data query time range, the stored required dataset is sent to the data requesting end.

4. The method according to claim 1, wherein, The step of retrieving the required dataset from the data table to be queried based on the at least one primary key group includes: For each primary key group in the at least one primary key group, perform the primary key processing steps: Subtract the first value from the primary key with the smaller value in the primary key group, and add the second value to the primary key with the larger value in the primary key group to obtain the processed primary key group; Determine the set of primary keys that lies between two processed primary keys, wherein the two processed primary keys are the two primary keys included in the processed primary key group; Find the dataset corresponding to the primary key set from the data table to be queried, and use it as the search dataset; Based on the obtained query dataset group, determine the required dataset.

5. The method according to claim 2, wherein, The at least one target time point is a time point determined based on at least one time point, and the at least one time point is a time point determined based on a preset time granularity; as well as The step of storing the required dataset includes: For each desired data point in the desired dataset, perform the key-value pair data generation step: Determine the time point corresponding to the required data; Using the time point as the key and the required data as the value, generate key-value pair data for the required data; The obtained key-value pairs are temporarily stored in the dataset.

6. The method according to claim 1, wherein, The time-primary key relationship table is also constructed through the following steps: determining at least one time point by setting a preset time granularity; For each of the at least one target time point, the execution time and primary key relationship data generation step is as follows: Determine the primary key included in the data corresponding to the target time point; Determine the time point among the at least one time points corresponding to the target time point; Based on the primary key and the time point, generate time and primary key relationship data for the data corresponding to the target time point; Based on the obtained time and primary key relationship dataset, the time and primary key relationship table is generated.

7. The method according to claim 1, wherein, The method further includes: In response to determining that at least one primary key group does not exist, the at least one primary key group is queried from the data table to be queried, and at least one time and primary key relationship data group for the at least one primary key group is written into the time and primary key relationship table.

8. The method according to claim 1, wherein, The data table to be queried is a sub-data table after the full data table has been sharded and partitioned.

9. The method according to claim 8, wherein, The full data table is processed through database sharding and table partitioning using the following steps: Determine the fields to be partitioned into databases and tables in each field corresponding to the full data table; Based on the fields to be sharded, the full data table is sharded to obtain a sharded data table set; Based on the fields to be sharded, each sharded data table in the sharded data table set is sharded to generate a sharded data table set, resulting in a sharded data table set group.

10. A data query device, comprising: The acquisition unit is configured to retrieve the data query time range; The determining unit is configured to determine whether at least one primary key group exists in the time-primary key relationship table that corresponds to the data query time range. The time-primary key relationship table is a table generated based on at least one target data point in the data table to be queried. The at least one target data point corresponds to at least one target time point. The time-primary key relationship table is constructed through the following steps: based on the determined at least one time point, selecting time points that meet preset time conditions from the time points corresponding to each data point in the data table to be queried, and using these as target time points to obtain at least one target time point. There is a one-to-one correspondence between the at least one target time point. For each target time point, the time-primary key relationship data generation step is executed. The lookup unit is configured to, in response to determining the existence of the at least one primary key group, retrieve the required dataset from the data table to be queried based on the at least one primary key group.

11. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images