Data query method, device, equipment and computer storage medium

Abstract

The present invention relates to the technical field of financial technology (Fintech), and discloses a data query method, which includes: converting target data in a data warehouse into a target file, and importing the target file into multiple target clusters; if a data query instruction is obtained, then based on the data query instruction, simultaneously performing a data query on each of the target clusters to obtain a query result, and when there are multiple query results, detecting whether there is a timed query result in each query result with a query duration greater than or equal to a preset duration; if not, merging each of the query results to obtain a target query result, and outputting the target query result. The present invention also discloses a data query device, equipment, and a computer storage medium. The present invention improves the efficiency of data query.

Description

Technical Field

[0001] The present invention relates to the field of data query technology in financial technology (Fintech), and in particular to data query methods, devices, equipment and computer storage media. Background Art

[0002] With the development of computer technology, more and more technologies (big data, distributed, artificial intelligence, etc.) are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology.

[0003] Currently, data queries are performed using a single-cluster query method. Applications query a single HBase cluster through the HBase (distributed storage system) interface, with the backup HBase cluster activated only when the primary cluster is unavailable. Furthermore, when replication is enabled between the primary and backup HBase clusters, due to HBase's record-by-record insertion approach, frequent data operations and constant modifications to HBase region partitions lead to excessive memory fragmentation, excessive resource usage, and increased data query latency. This makes the current data query method prone to inefficiency. Furthermore, since HBase data is synchronized across clusters, if data loss occurs in the primary cluster, the data in the backup cluster is also affected, reducing the accuracy of query results. Summary of the Invention

[0004] The main purpose of the present invention is to provide a data query method, device, equipment and computer storage medium, aiming to solve the technical problem of how to improve the efficiency and accuracy of data query.

[0005] To achieve the above object, the present invention provides a data query method, which includes the following steps:

[0006] Converting target data in a data warehouse into target files, and importing the target files into multiple target clusters;

[0007] If a data query instruction is obtained, data query is performed on each target cluster simultaneously based on the data query instruction to obtain query results, and when there are multiple query results, whether there is a timed query result with a query duration greater than or equal to a preset duration in each query result;

[0008] If not, each query result is merged to obtain a target query result, and the target query result is output.

[0009] Optionally, the step of performing a data query on each of the target clusters based on the data query instruction to obtain a query result includes:

[0010] Performing data query on each of the target clusters based on the data query instruction, and sequentially detecting whether each of the target clusters has fed back data query information;

[0011] If so, the data query information fed back by the target cluster is used as the query result.

[0012] Optionally, the step of simultaneously performing data query on each of the target clusters based on the data query instruction includes:

[0013] Detecting whether there is a maintenance target cluster in each of the target clusters that is undergoing maintenance;

[0014] If the maintenance target cluster exists, other target clusters except the maintenance target cluster in each target cluster are acquired, and data query is performed on the other target clusters according to the data query instruction.

[0015] Optionally, the step of merging the query results to obtain a target query result includes:

[0016] Traversing each of the query results in sequence, performing similarity detection on the current query result and each of the query results in sequence, counting the number of similarity detection results that are greater than a preset threshold, and if the count is greater than a preset number, taking the current query result as a valid query result, until the traversal of the query results is completed;

[0017] If there are multiple valid query results, each of the valid query results is merged to obtain a target query result.

[0018] Optionally, the step of converting the target data in the data warehouse into a target file includes:

[0019] Segment the target data in the data warehouse according to a preset data conversion method to obtain multiple segmented data;

[0020] The segmented data are assembled into a preset storage format file, and the storage format file having the segmented data is imported into a preset directory file to obtain a target file.

[0021] Optionally, the step of importing the target file into multiple target clusters includes:

[0022] Determine the directory corresponding to each target file, and obtain the target table in each target cluster;

[0023] The target files are simultaneously imported into each of the target tables according to the directory.

[0024] Optionally, after the step of detecting whether there is a timed query result in each query result whose query duration is greater than or equal to a preset duration, the method further includes:

[0025] If so, determine other query results except the timed-out query result in each query result, and output the other query results.

[0026] In addition, to achieve the above-mentioned purpose, the present invention further provides a data query device, the data query device comprising:

[0027] An import module, configured to convert target data in the data warehouse into target files, and import the target files into multiple target clusters;

[0028] a detection module configured to, upon obtaining a data query instruction, simultaneously perform a data query on each of the target clusters based on the data query instruction to obtain query results, and, if multiple query results exist, detect whether any of the query results contains a timed-out query result with a query duration greater than or equal to a preset duration;

[0029] The output module is configured to merge the query results to obtain a target query result if the target query result does not exist, and output the target query result.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also provides a data query device, which includes: a memory, a processor, and a data query program stored on the memory and runnable on the processor, and when the data query program is executed by the processor, the steps of the data query method described above are implemented.

[0031] In addition, to achieve the above-mentioned purpose, the present invention further provides a computer storage medium, on which a data query program is stored. When the data query program is executed by a processor, the steps of the data query method described above are implemented.

[0032] The present invention converts target data in a data warehouse into a target file and imports the target file into multiple target clusters. If a data query instruction is obtained, data queries are performed on each target cluster based on the data query instruction to obtain multiple query results. The method detects whether there is a target query result with a query duration greater than or equal to a preset duration in each query result. If not, the method merges the query results to determine a merged result of the merged process and outputs the merged result. By converting the target data in the data warehouse into a target file and then importing the target file into multiple target clusters, the prior art method of directly importing data into the target cluster word by word, which causes excessive memory fragmentation and excessive resource consumption, is avoided. When a data query instruction is received, data queries are performed on each target cluster simultaneously, which improves the efficiency of data queries compared to the prior art method of single cluster queries. The method also merges the query results and outputs the merged result, avoiding the phenomenon of inaccurate query results due to the failure of a cluster and improving the accuracy of data queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the structure of a data query device in a hardware operating environment according to an embodiment of the present invention;

[0034] Figure 2 This is a flow chart of a first embodiment of a data query method according to the present invention;

[0035] Figure 3 This is a schematic diagram of the device modules of the data query device of the present invention;

[0036] Figure 4 This is a schematic diagram of the process of writing data into a cluster in the data query method of the present invention;

[0037] Figure 5 Schematic diagram of the data query timing in the data query method of the present invention;

[0038] Figure 6 Schematic diagram of the flow of the data query method of the present invention.

[0039] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0040] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] like Figure 1 As shown, Figure 1It is a structural diagram of a data query device in a hardware operating environment involved in an embodiment of the present invention.

[0042] The data query device in the embodiment of the present invention may be a PC or a server device, on which a Java virtual machine runs.

[0043] like Figure 1 As shown, the data query device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0044] Those skilled in the art will understand that Figure 1 The data query device structure shown in the figure does not constitute a limitation to the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0045] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a data query program.

[0046] exist Figure 1 In the data query device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the data query program stored in the memory 1005 and execute the operations in the following data query method.

[0047] Based on the above hardware structure, an embodiment of the data query method of the present invention is proposed.

[0048] Reference Figure 2 , Figure 2 This is a flow chart of a first embodiment of a data query method according to the present invention, wherein the method comprises:

[0049] Step S10: converting target data in the data warehouse into target files, and importing the target files into multiple target clusters;

[0050] In this embodiment, HBase is a distributed columnar storage database. Hive is an offline SQL (Structured Query Language) data warehouse used to store target data. In traditional solutions, when storing data in the Hive data warehouse into HBase, the only way is to use data import to store the data in the Hive data warehouse into a single HBase (the primary HBase). The primary HBase then synchronizes the data with the backup HBase. Since data synchronization between the primary and backup HBases is performed using replication tools, this synchronization method is based on record-by-record insertion. Data operations are frequent, and the region partitions of HBase are constantly modified. This leads to excessive memory fragmentation, excessive resource usage, increased latency in subsequent data queries, and low data query efficiency.

[0051] Based on this, in an embodiment of the present invention, when the data in the hive data warehouse is stored in the hbase database, the target data in the data warehouse is first converted into a target file. In this embodiment, hfile can be used as the storage format of the target file. Moreover, since the data in the data warehouse hive are all stored in data format (such as table form), when the target data in the data warehouse needs to be imported into the hbase cluster (that is, the target cluster), the target data in the table is first segmented according to the preset delimiter, and then each segmented data is assembled into a preset storage format file, wherein the storage format file includes Key Length (keyword length), Value Length (value length), Row Length (row length), Row (row), Column Family Length (column family length), Column Length (column length), Column Length (column length), Column Qualifier (column qualifier), Time Stamp (timestamp), Key Type (keyword type) and Value (value) attributes. The files assembled into the storage format are then imported into a preset directory file (such as a directory file established according to the type of primary key, or a directory file established according to timestamp, etc., which can be defined according to user needs) to obtain multiple target files that conform to the hbase cluster format. Each target file is then uploaded to the target table in each hbase cluster in turn, thereby completing the operation of writing the target data in hive to multiple hbase clusters at the same time, facilitating subsequent data queries in each hbase cluster. In an embodiment of the present invention, after the target data is converted into target files, the target files are imported into multiple target clusters respectively, and the traditional method (importing the target data into a single cluster and then synchronizing the clusters using a replication tool) is no longer used, thus avoiding the defect of excessive memory fragmentation caused by continuous modification of the cluster's region partitions.

[0052] It should be noted that before converting the target data in the data warehouse into the target file, the data is first processed using ETL (Extract-Transform-Load, a data warehouse technology). Specifically, ETL extracts, transforms, and loads data from the source to the destination. The target data is the data associated with the data query instruction. The target cluster is the HBase cluster used for data query (including the primary cluster and the backup cluster), and its number can be determined by the number set in advance by the user. In this embodiment, data queries can be performed on multiple HBase clusters simultaneously.

[0053] In this embodiment, data query is performed by adopting a multi-write and multi-read method. That is, the data in the data warehouse is converted into a target file, and then the target file is imported into multiple target clusters at the same time to complete the multi-write operation. Multi-read is a data query that uses multiple target clusters to query at the same time, and merges the results of each query to obtain the merged result for output. Therefore, the target data in the data warehouse hive is converted into a target file and synchronized to a preset number of target clusters for data query. That is, ETL will batch process the source data corresponding to the data query instruction (such as credit data) into hive data, and convert the hive data into a target file, and then import the target file into each target cluster.

[0054] And more writing can be done as Figure 4 As shown in the example, in the Hive warehouse (data warehouse), the target data is converted into hfiles (target files) and synchronized to HBase cluster-1 (target cluster 1) and HBase cluster-2 (target cluster 2). Specifically, a target table is created in a pre-formatted file, and a directory is created within the target table. The target data is imported into the target table based on this directory, and the file containing the target data is used as the target file, the hfile. After the target data is converted from Hive to hfiles, importing the hfiles into the HBase database reduces HBase data insertion operations, speeding up imports while reducing memory fragmentation and the likelihood of garbage collection triggering, effectively mitigating query latency fluctuations.

[0055] Step S20: If a data query instruction is obtained, data query is performed on each target cluster simultaneously based on the data query instruction to obtain query results, and when there are multiple query results, whether there is a timed query result with a query duration less than a preset duration in each query result;

[0056] The data query instruction can be input by the user or a pre-set instruction used to trigger the data query operation. When the target file exists in each target cluster and the data query instruction is received, multi-read processing can be performed, that is, data query is performed on each target cluster at the same time to obtain multiple query results. And when performing data query on each target cluster, different operations can be performed according to different failure scenarios, and corresponding results can be returned. For example, when data query is performed on cluster-1 and cluster-2 at the same time, if cluster-1 fails, resulting in the loss of customer ID0002 data (i.e., target data), then when a query is initiated for a certain data of customer ID0002 (such as credit record), cluster-1 will fail to return records, and cluster-2 will successfully return data records. At this time, the data records returned by cluster-2 can be directly output. For another example, when data is queried on cluster-1 and cluster-2 at the same time, if cluster-1 needs maintenance and upgrade and cannot perform import and query operations, the data will only be imported into cluster-2, and the credit record of customer ID0002 will be queried through cluster-2. The returned records of cluster-2 will be merged to determine the final result, and cluster-1 will allow query operations only when the service is restored (both import and query operations are available).

[0057] After obtaining the query results, if there is only one query result, this query result will be directly fed back. If there are multiple query results, each query result will be checked to see if there is a timeout query result with a query duration greater than or equal to the preset duration (an arbitrary duration set by the user in advance), and different operations will be performed based on different detection results. That is, timeout detection is performed on each query result. For example, if the system receives a data query instruction and initiates two threads to query cluster-1 and cluster-2 respectively, when thread 2 sends a query to cluster-2, due to the delay of cluster-2, the query result of cluster-2 is returned with a timeout, while the query result of cluster-1 is returned in a timely manner. At this time, the query result of cluster-1 can be output.

[0058] For example, Figure 5As shown, the system includes a client, a main query thread, an HBase cluster-1 query thread, and an HBase cluster-2 query thread. When a client initiates a query, the main query thread queries both clusters 1 and 2, merges and returns the query results from both the HBase cluster-1 and HBase cluster-2 query threads. The returned results from each thread are then checked to determine if a timeout has occurred. If a timeout has occurred, a timeout result is returned to the client. If not, the query result is returned to the client. Specifically, when the main query thread receives a data query instruction, it proactively launches at least two query threads to simultaneously query HBase clusters 1 and 2. The main thread monitors the returned results (i.e., query results) from both clusters. Any query result received within a preset timeframe (arbitrary, user-defined timeframe) serves as the merged result for this query. This reduces the latency fluctuations associated with a single cluster, reduces latency, and improves effective service response and reliability.

[0059] Step S30: If the query result does not exist, the query results are merged to obtain a target query result, and the target query result is output.

[0060] If it is determined that no query results have timed out with a query duration greater than or equal to the preset duration, it can be determined that none of the query results have timed out. At this point, the query results can be merged to obtain a target query result, which is then output. The merging process involves combining similar items in the obtained query results to prevent duplicate data from being fed back. Furthermore, when merging the query results, if a query result has a low similarity to other query results, this query result can be ignored, and the query results with higher similarity can be merged to obtain the target query result.

[0061] In addition, to assist in understanding the principle of data query in this embodiment, an example is given below.

[0062] For example, Figure 6As shown, data is first imported. In the Hive warehouse (data warehouse), the target data is converted into hfiles (target files) and synchronized to HBase cluster 1 (target cluster 1) and HBase cluster 2 (target cluster 2). When data queries begin, query scheduling is performed based on the main query thread, creating threads for query clusters 1 and 2. Query cluster 1 queries HBase cluster 1, obtains the returned results, and sends these results to the merge result section. Query cluster 2 queries HBase cluster 2, obtains the returned results, and sends these results to the merge result section. Timeout checks are performed on each returned result. If the timeout is within the specified range, the returned result is output in the merge result section. If the timeout is exceeded, the timeout result is returned, and the process continues until the query completes.

[0063] In this embodiment, target data in a data warehouse is converted into target files and the target files are imported into multiple target clusters. If a data query instruction is received, data queries are simultaneously performed on each target cluster based on the data query instruction to obtain query results. If multiple query results exist, the query results are detected to determine whether there is a timed-out query result with a query duration greater than or equal to a preset duration. If not, the query results are merged to obtain a target query result, which is then output. By converting the target data in a data warehouse into target files and then importing the target files into multiple target clusters, the prior art practice of directly importing data into the target clusters word by word, which results in excessive memory fragmentation and excessive resource usage, is avoided. When a data query instruction is received, data queries are simultaneously performed on each target cluster, improving data query efficiency compared to the prior art single-cluster query method. Furthermore, the query results are merged and the merged result is output, avoiding inaccurate query results due to a failure in a cluster and improving data query accuracy.

[0064] Furthermore, based on the first embodiment of the data query method of the present invention, a second embodiment of the data query method of the present invention is proposed. This embodiment is a refinement of step S20 of the first embodiment of the present invention, which performs a data query on each target cluster based on the data query instruction to obtain a query result, including:

[0065] Step a: performing data query on each target cluster based on the data query instruction, and sequentially detecting whether each target cluster feeds back data query information;

[0066] In this embodiment, when performing data queries on each target cluster according to a data query instruction, each target cluster needs to be tested, that is, each target cluster is tested in turn to see if it has responded with data query information, and different operations are performed based on different test results. The data query information can be the result of the HBase cluster responding to the data query instruction.

[0067] Step b: If yes, the data query information fed back by the target cluster is used as the query result.

[0068] If it is determined that there is a target cluster that has responded to the query, the response from this target cluster can be used as the query result. However, if there is a target cluster that has not responded to the query, it is determined that there is an abnormality in this target cluster.

[0069] In this embodiment, when performing data query on each target cluster, if it is detected that each target cluster can feed back data query information, the data query information is used as the query result, thereby ensuring the accuracy of the query result obtained.

[0070] Furthermore, the step of simultaneously performing data query on each of the target clusters based on the data query instruction includes:

[0071] Step c, detecting whether there is a maintenance target cluster that is undergoing maintenance in each of the target clusters;

[0072] In this embodiment, before performing a data query on each target cluster according to a data query instruction, each target cluster needs to be checked to determine whether any target cluster is undergoing a version upgrade and is unable to perform data query operations. If so, this target cluster is treated as a maintenance target cluster, and the number of target clusters to be queried is determined based on the different inspection results. The data query interface and data import interface of the maintenance target cluster are unavailable.

[0073] Step d: If the maintenance target cluster exists, then obtaining other target clusters except the maintenance target cluster from among the target clusters, and performing data query on the other target clusters according to the data query instruction.

[0074] If a maintenance target cluster is found among the target clusters, the system determines other target clusters in the target clusters except for the maintenance target cluster and sends a data query instruction to the other target clusters to perform data query on the other target clusters. However, if the maintenance target cluster is not found among the target clusters, the system performs data query on all target clusters according to the data query instruction.

[0075] In this embodiment, when it is determined that a maintenance target cluster exists among the target clusters, data query is performed on other target clusters, thereby ensuring effective execution of the data query.

[0076] Furthermore, the step of merging the query results to obtain the target query result includes:

[0077] Step e, traversing each of the query results in sequence, performing similarity detection on the current query result and each of the query results in sequence, counting the number of similarity detection results that are greater than a preset threshold, and if the statistical number is greater than a preset number, taking the current query result as a valid query result, until the traversal of the query results is completed;

[0078] In this embodiment, when multiple query results are obtained, each query result is traversed in turn, and the current query result being traversed is tested for similarity with each query result obtained in turn, that is, the similarity between the current query result and each query result is calculated, and each similarity is judged in turn whether it is greater than a preset threshold (an arbitrary proportion value set in advance by the user), and then the number of similarities greater than the preset threshold is obtained, that is, the statistical number, and the statistical number is compared with the preset number (an arbitrary number set in advance by the user). If it is greater, it can be determined that the current query result is valid and can be used as a valid query result until the traversal of each query result is completed, that is, all query results are judged in the same way to obtain a valid query result.

[0079] Step x: If there are multiple valid query results, merge the valid query results to obtain the target query result.

[0080] After obtaining a valid query result, determine whether there are multiple valid query results. If there is only one valid query result, output this valid query result as the target query result. If there are multiple valid query results, merge the valid query results to obtain the merged query result, which is the target query result.

[0081] In this embodiment, valid query results are obtained from each query result, and when there are multiple valid query results, each valid query result is merged to obtain a target query result, thereby ensuring the accuracy of the data query result.

[0082] Furthermore, the step of converting the target data in the data warehouse into a target file includes:

[0083] Step g, segmenting the target data in the data warehouse according to a preset data conversion method to obtain a plurality of segmented data;

[0084] In this embodiment, when converting target data in a data warehouse into a target file using a preset data conversion method, it is necessary to first determine the table location of the target data in the data warehouse, and then segment the Hive table containing the target data according to a preset delimiter to obtain multiple segmented data. The preset delimiter can be any user-defined symbol, such as ",". In this example, this data conversion method is pre-set. For convenience, the importTsv method can also be used to convert the Hive table format into the Hfile format.

[0085] Step h: assembling each of the segmented data into a preset storage format file, and importing the storage format file with each of the segmented data into a preset directory file to obtain a target file.

[0086] After obtaining each segmented data, each segmented data can be assembled into a preset storage format file (a file set in advance by the user that is applicable to the HBase cluster, such as RowKey, which is used to represent a unique row of records). The storage format file is then imported into the directory specified by HDFS in the format of an HFile (HFile is the data format file of HBase), that is, the preset directory file. The preset directory file with the storage format file is used as the target file and imported into multiple target clusters at the same time. The way to assemble the segmented data into the storage format file can be to classify and merge the segmented data according to the attribute definition to obtain the storage format file. Among them, the storage format file includes Key Length (keyword length), Value Length (value length), Row Length (row length), Row (row), Column Family Length (column family length), Column Length (column length), Column Qualifier (column qualifier), TimeStamp (timestamp), Key Type (keyword type) and Value (value).

[0087] In this embodiment, the target data is divided into multiple segmented data according to a preset data conversion method, assembled into a storage format file, and imported into a preset directory file to obtain the target file, thereby ensuring the accuracy of the obtained target file.

[0088] Furthermore, the step of importing the target file into multiple target clusters includes:

[0089] Step k, determining the directory corresponding to the target file, and obtaining the target table in each target cluster;

[0090] When importing a target file into multiple target clusters, you need to first determine the existing directories in the target file, such as "key Length, Value", and then determine the target clusters to which the file needs to be imported and obtain the tables (such as HBase tables) in these target clusters, i.e., the target tables.

[0091] Step m: importing the target files into each of the target tables simultaneously according to the directory.

[0092] After obtaining the target tables in each target cluster, the hfile files in the target file can be imported into the target tables in each target cluster in sequence according to the directory in the target file to complete the import of data in the data warehouse into the hbase cluster. In addition, the target files can also be imported into each of the target tables through bulkload.

[0093] In this embodiment, by first determining the directory corresponding to the target file, and then importing the target file into the target table in each target cluster at the same time according to the directory, the target file is effectively imported into multiple target clusters.

[0094] Furthermore, after the step of detecting whether there is a target query result with a query duration shorter than a preset duration in each query result, the method includes:

[0095] Step n: if it exists, determine other query results in each of the query results except the target query result, and output the other query results.

[0096] If a timeout query result is found in each query result, where the query duration is greater than or equal to the preset duration, it can be determined that a data query on the target cluster has timed out. The system then determines the query results other than the timeout query result and checks whether there are multiple other query results. If there is only one other query result, it can be directly output. If there are multiple other query results, they can be merged and output.

[0097] In this embodiment, when it is determined that there is a timed query result with a query duration greater than or equal to a preset duration, other query results are determined and output, thereby ensuring the accuracy of data query.

[0098] The present invention also provides a data query device, referring to Figure 3 , the data query device includes:

[0099] An import module A10 is used to convert target data in the data warehouse into target files and import the target files into multiple target clusters;

[0100] The detection module A20 is configured to, upon receiving a data query instruction, simultaneously perform a data query on each of the target clusters based on the data query instruction to obtain query results, and, if multiple query results exist, detect whether any of the query results contains a timed-out query result with a query duration greater than or equal to a preset duration;

[0101] The output module A30 is configured to merge the query results to obtain a target query result if the target query result does not exist, and output the target query result.

[0102] Optionally, the detection module A20 is further configured to:

[0103] Performing data query on each of the target clusters based on the data query instruction, and sequentially detecting whether each of the target clusters has fed back data query information;

[0104] If so, the data query information fed back by the target cluster is used as the query result.

[0105] Optionally, the detection module A20 is further configured to:

[0106] Detecting whether there is a maintenance target cluster in each of the target clusters that is undergoing maintenance;

[0107] If the maintenance target cluster exists, other target clusters except the maintenance target cluster in each target cluster are acquired, and data query is performed on the other target clusters according to the data query instruction.

[0108] Optionally, the output module A30 is further configured to:

[0109] Traversing each of the query results in sequence, performing similarity detection on the current query result and each of the query results in sequence, counting the number of similarity detection results that are greater than a preset threshold, and if the count is greater than a preset number, taking the current query result as a valid query result, until the traversal of the query results is completed;

[0110] If there are multiple valid query results, each of the valid query results is merged to obtain a target query result.

[0111] Optionally, the import module A10 is further configured to:

[0112] Segment the target data in the data warehouse according to a preset data conversion method to obtain multiple segmented data;

[0113] The segmented data are assembled into a preset storage format file, and the storage format file having the segmented data is imported into a preset directory file to obtain a target file.

[0114] Optionally, the import module A10 is further configured to:

[0115] Determine the directory corresponding to each target file, and obtain the target table in each target cluster;

[0116] The target files are simultaneously imported into each of the target tables according to the directory.

[0117] Optionally, the detection module A20 is further configured to:

[0118] If so, determine other query results except the timed-out query result in each query result, and output the other query results.

[0119] The methods executed by the above-mentioned program units can refer to the various embodiments of the data query method of the present invention, and will not be described in detail here.

[0120] The present invention also provides a computer storage medium.

[0121] The computer storage medium of the present invention stores a data query program, which implements the steps of the data query method described above when executed by a processor.

[0122] The method implemented when the data query program running on the processor is executed can refer to the various embodiments of the data query method of the present invention, and will not be described in detail here.

[0123] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0124] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0125] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0126] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data query method, characterized in that: The data query method comprises the following steps: Converting target data in the data warehouse into target files, and importing the target files into multiple target clusters, wherein the target data is data associated with the data query instruction; If the data query instruction is obtained, data query is performed on each of the target clusters simultaneously based on the data query instruction to obtain query results, and when there are multiple query results, whether there is a timed query result with a query duration greater than or equal to a preset duration in each of the query results; If not, each query result is merged to obtain a target query result, and the target query result is output.

2. The data query method according to claim 1, wherein: The step of performing a data query on each of the target clusters based on the data query instruction to obtain a query result includes: Performing data query on each of the target clusters based on the data query instruction, and sequentially detecting whether each of the target clusters has fed back data query information; If so, the data query information fed back by the target cluster is used as the query result.

3. The data query method according to claim 2, wherein: The step of simultaneously querying data on each of the target clusters based on the data query instruction includes: Detecting whether there is a maintenance target cluster in each of the target clusters that is undergoing maintenance; If the maintenance target cluster exists, other target clusters except the maintenance target cluster in each target cluster are acquired, and data query is performed on the other target clusters according to the data query instruction.

4. The data query method according to claim 1, wherein: The step of merging the query results to obtain the target query result includes: Traversing each of the query results in sequence, performing similarity detection on the current query result and each of the query results in sequence, counting the number of similarity detection results that are greater than a preset threshold, and if the count is greater than a preset number, taking the current query result as a valid query result, until the traversal of the query results is completed; If there are multiple valid query results, each of the valid query results is merged to obtain a target query result.

5. The data query method according to claim 1, wherein: The step of converting the target data in the data warehouse into a target file includes: Segment the target data in the data warehouse according to a preset data conversion method to obtain multiple segmented data; The segmented data are assembled into a preset storage format file, and the storage format file having the segmented data is imported into a preset directory file to obtain a target file.

6. The data query method according to claim 1, wherein: The step of importing the target file into multiple target clusters includes: Determine the directory corresponding to each target file, and obtain the target table in each target cluster; The target files are simultaneously imported into each of the target tables according to the directory.

7. The data query method according to any one of claims 1 to 6, characterized in that: After the step of detecting whether there is a timed query result with a query duration greater than or equal to a preset duration in each query result, the following steps are included: If so, determine other query results except the timed-out query result in each query result, and output the other query results.

8. A data query device, characterized in that: The data query device comprises: An import module, configured to convert target data in a data warehouse into target files, and import the target files into a plurality of target clusters, wherein the target data is data associated with a data query instruction; a detection module configured to, upon obtaining the data query instruction, simultaneously perform data queries on each of the target clusters based on the data query instruction to obtain query results, and, when there are multiple query results, detect whether there is a timed query result in each of the query results whose query duration is greater than or equal to a preset duration; The output module is configured to merge the query results to obtain a target query result if the target query result does not exist, and output the target query result.

9. A data query device, characterized in that: The data query device includes: a memory, a processor, and a data query program stored in the memory and executable on the processor. When the data query program is executed by the processor, the steps of the data query method according to any one of claims 1 to 7 are implemented.

10. A computer storage medium, characterized in that The computer storage medium stores a data query program, which, when executed by a processor, implements the steps of the data query method according to any one of claims 1 to 7.

Images