Data query method, device, equipment and storage medium

Abstract

This application discloses a data query method, apparatus, device, and storage medium, relating to the field of computer technology, to address the slow query speed of table content in current databases. The method includes: obtaining a table to be queried from the database and obtaining target information of the table; determining whether to perform a preset table partitioning operation on the table based on the target information; if the preset table partitioning operation is performed, obtaining all data blocks corresponding to the table through the preset table partitioning operation; and sending the data blocks to a target thread in a thread pool to execute a preset information query operation. This application determines whether the current table needs to be partitioned based on the target information of the table. If so, the table is partitioned into a preset number of data blocks, and then the data blocks are queried synchronously in the thread pool, improving the query speed of table content and simplifying the operation steps.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data query method, apparatus, device, and storage medium. Background Technology

[0002] Currently, with the development of databases and the trend of localization of basic software, Oracle databases are gradually being migrated to other databases in China. During the static data migration process, the querying of database table contents directly affects the overall migration performance of the database.

[0003] Currently, most methods for querying database table content involve using concurrency or thread pools. This involves migrating multiple tables simultaneously or performing sharded queries on tables with large amounts of data. This often requires manually setting fields for large tables to confirm sharding, which is complex and cannot handle situations with a large number of tables, resulting in slow query speeds. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a data query method, apparatus, device, and storage medium that can improve the query speed of table content and simplify operation steps. The specific solution is as follows:

[0005] Firstly, this application discloses a data query method, including:

[0006] Obtain the table to be queried from the database, and obtain the target information of the table to be queried;

[0007] Based on the target information, determine whether to perform a preset table splitting operation on the table to be queried;

[0008] If the preset table splitting operation is performed on the table to be queried, then all data blocks corresponding to the table to be queried are obtained through the preset table splitting operation;

[0009] The data block is sent to the target thread in the thread pool to perform a preset information query operation.

[0010] Optionally, determining whether to perform a preset table splitting operation on the table to be queried based on the target information includes:

[0011] Data is randomly extracted from the table to be queried based on the parameters in the preset function, and the value of the extracted row corresponding to the extracted data is determined.

[0012] The total number of rows in the table to be queried is calculated based on the extracted row values ​​and the parameters in the preset function.

[0013] Determine whether the total number of rows of data is within the preset number of rows;

[0014] If the total number of data rows is within the preset number of rows, then it is determined that the preset table splitting operation will be performed on the table to be queried;

[0015] If the total number of data rows is lower than the minimum value of the preset row range, then it is determined that the preset table splitting operation will not be performed on the table to be queried.

[0016] Optionally, obtaining the target information of the table to be queried includes:

[0017] Obtain the field types of the table to be queried and the large object query results.

[0018] Optionally, obtaining all data blocks corresponding to the table to be queried through the preset table partitioning operation includes:

[0019] If the table to be queried contains a primary key, then obtain the field type corresponding to the primary key to get the target field type;

[0020] When the target field type is an integer, the table to be queried is divided based on the primary key and the extracted row values ​​to obtain all the corresponding data blocks;

[0021] Alternatively, when the target field type is numeric, the table to be queried is divided based on the primary key, the total number of data rows, and the value of a custom preset data block to obtain all the corresponding data blocks.

[0022] Optionally, obtaining all data blocks corresponding to the table to be queried through the preset table partitioning operation includes:

[0023] If the primary key is not contained in the table to be queried, then determine whether the numeric field exists in the table to be queried;

[0024] If the table to be queried contains a field of the numeric type, then randomly select one field from all the numeric type fields to obtain the segmentation reference field;

[0025] The table to be queried is segmented based on the segmentation reference field, the total number of data rows, and the custom preset data block value to obtain all the corresponding data blocks;

[0026] If the table to be queried does not contain the primary key and does not have the field of the numeric type, then the table to be queried is divided based on the total number of data rows and the value of the custom preset data block to obtain all the corresponding data blocks.

[0027] Optionally, sending the data block to the target thread in the thread pool includes:

[0028] Determine whether there is a data block containing a large object in the current batch of data blocks received by the thread pool;

[0029] If the data block containing the large object does not exist in the current batch of data blocks, then the data block is sent to the target thread in the thread pool;

[0030] If the data block containing the large object exists in the current batch of data blocks, then the data block containing the large object is sent to the reserved thread in the thread pool; the reserved thread is the only thread in the thread pool used to process the data block containing the large object.

[0031] The data blocks in the current batch that do not contain large objects are sent to the remaining threads in the thread pool that do not contain the reserved threads.

[0032] Optionally, after sending the data block to the target thread in the thread pool, the method further includes:

[0033] If there are idle threads in the thread pool, then determine whether there is a next batch of the data blocks;

[0034] If the next batch of data blocks exists, then the next batch of data blocks is obtained, and the next batch of data blocks is determined as the data blocks of the current batch;

[0035] Re-enter the step of determining whether there is a data block containing a large object in the current batch of data blocks received by the thread pool;

[0036] If the next batch of the data block does not exist, the process ends.

[0037] Secondly, this application discloses a data query device, comprising:

[0038] The information acquisition module is used to acquire the table to be queried in the database and to acquire the target information of the table to be queried.

[0039] The segmentation judgment module is used to determine whether to perform a preset table segmentation operation on the table to be queried based on the target information;

[0040] The segmentation module is used to obtain all data blocks corresponding to the table to be queried through the preset table segmentation operation if the preset table segmentation operation is performed on the table to be queried.

[0041] The data block sending module is used to send the data block to the target thread in the thread pool so as to execute a preset information query operation.

[0042] Thirdly, this application discloses an electronic device, comprising:

[0043] Memory, used to store computer programs;

[0044] A processor for executing the computer program to implement the steps of the data query method disclosed above.

[0045] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the data query method disclosed above.

[0046] As can be seen, this application provides a data query method, including: obtaining a table to be queried in a database and obtaining target information of the table to be queried; determining whether to perform a preset table partitioning operation on the table to be queried based on the target information; if the preset table partitioning operation is performed on the table to be queried, obtaining all data blocks corresponding to the table to be queried through the preset table partitioning operation; and sending the data blocks to a target thread in a thread pool to execute a preset information query operation. Therefore, this application can obtain data information for determining whether the current table needs to be partitioned based on the target information of the table. If the data information indicates that the current table needs to be partitioned, the table is partitioned into a preset number of data blocks, and then the data blocks are queried synchronously in the thread pool. The size of the data blocks is much smaller than the original table size, which improves the query speed of table content and simplifies the operation steps under concurrent processing. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 This is a flowchart of a data query method disclosed in this application;

[0049] Figure 2 This is a flowchart of a specific data query method disclosed in this application;

[0050] Figure 3 This is a flowchart of a specific data query method disclosed in this application;

[0051] Figure 4 This is a schematic diagram of the total query time disclosed in this application;

[0052] Figure 5 This is a schematic diagram of the total query time disclosed in this application;

[0053] Figure 6 A schematic diagram of the data query device provided in this application;

[0054] Figure 7 This application provides a structural diagram of an electronic device. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Currently, with the development of databases and the trend of localization of basic software, Oracle databases are gradually being migrated to other databases in China. During the static data migration process, the querying of database table content directly affects the overall migration performance of the database. Current methods for querying database table content mostly use concurrency or thread pools, migrating multiple tables simultaneously or performing sharded queries on tables with large amounts of data. This often requires manually setting fields for large tables to confirm sharding, which is complex and cannot handle a large number of tables, resulting in slow query speeds. Therefore, this application provides a data querying method that can improve the query speed of table content and simplify the operation steps.

[0057] This invention discloses a data query method, see [link to relevant documentation]. Figure 1 As shown, the method includes:

[0058] Step S11: Obtain the table to be queried from the database, and obtain the target information of the table to be queried.

[0059] In this embodiment, the tables to be queried in the database are obtained, along with their target information. It is understood that after obtaining the tables to be queried in the database, information such as the field types and large object query results for each table is acquired. This allows for fuzzy analysis of the corresponding tables based on the target information, such as fuzzy calculation of the table size, field data, number of rows, field types, and the presence of large objects. Then, based on this information, it is determined whether to split the table. Pre-analyzing the database allows for optimization of query methods in subsequent queries. However, database analysis itself takes time; therefore, fuzzy analysis can be used to reduce processing time.

[0060] Step S12: Based on the target information, determine whether to perform a preset table splitting operation on the table to be queried.

[0061] In this embodiment, after obtaining the target information of the table to be queried, it is determined whether to perform a preset table partitioning operation on the table to be queried based on the target information. It can be understood that after obtaining the target information of each table, the size of the table, the number of rows, and the presence of large objects obtained through fuzzy analysis of the target information can be used to determine whether the table will require a long query time. If so, it is determined whether to perform a preset table partitioning operation on the table to be queried.

[0062] Step S13: If the preset table splitting operation is performed on the table to be queried, then all data blocks corresponding to the table to be queried are obtained through the preset table splitting operation.

[0063] In this embodiment, after determining whether to perform a preset table splitting operation on the table to be queried based on the target information, if the preset table splitting operation is performed on the table to be queried, then all data blocks corresponding to the table to be queried are obtained through the preset table splitting operation. It is understood that dividing the table content into data blocks, each data block may be a whole table or a part of a table, and the database table content does not need to be split into completely uniform sizes. For example, if the number of rows in the table is greater than a preset threshold, the data block obtained after performing the preset table splitting operation will be a portion of the data in the table. If the number of rows in the table is less than the minimum preset table splitting threshold, the preset table splitting operation will not be performed on the table; that is, the table is considered a single data block, meaning the data block represents a single table.

[0064] Furthermore, after performing fuzzy analysis on the database, the table content in the database is divided into multiple data blocks, and it is determined whether there is a primary key, numeric field, index, etc. in the table. Then, the best method is selected to divide the table content based on the judgment results.

[0065] Step S14: Send the data block to the target thread in the thread pool to perform a preset information query operation.

[0066] In this embodiment, after obtaining all data blocks corresponding to the table to be queried through the preset table partitioning operation, the data blocks are sent to the target thread in the thread pool to execute the preset information query operation. It is understood that concurrent queries are performed on the data blocks during the process of sending them to the target thread in the thread pool. Database query speed is limited by database performance and machine performance. By performing fuzzy analysis on the database table beforehand, the database table content is partitioned into multiple data blocks, and the preset information query operation is executed concurrently to maximize the I / O (Input / Output) of the database and machine, reduce database query idle time, and optimize the query SQL (Structured Query Language) to improve overall performance.

[0067] As can be seen, this application provides a data query method, including: obtaining a table to be queried in a database and obtaining target information of the table to be queried; determining whether to perform a preset table partitioning operation on the table to be queried based on the target information; if the preset table partitioning operation is performed on the table to be queried, obtaining all data blocks corresponding to the table to be queried through the preset table partitioning operation; and sending the data blocks to a target thread in a thread pool to execute a preset information query operation. Therefore, this application can obtain data information for determining whether the current table needs to be partitioned based on the target information of the table. If the data information indicates that the current table needs to be partitioned, the table is partitioned into a preset number of data blocks, and then the data blocks are queried synchronously in the thread pool. The size of the data blocks is much smaller than the original table size, which improves the query speed of table content and simplifies the operation steps under concurrent processing.

[0068] See Figure 2 As shown, this embodiment of the invention discloses a data query method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0069] Step S21: Obtain the table to be queried from the database, and obtain the target information of the table to be queried.

[0070] Step S22: Randomly extract data from the table to be queried based on the parameters in the preset function, and determine the value of the extracted row corresponding to the extracted data.

[0071] In this embodiment, after obtaining the target information of the table to be queried, data is randomly extracted from the table based on the parameters in a preset function, and the value of the extracted row corresponding to the extracted data is determined. Specifically, for example, in an Oracle database, the sample() function can be used to sample data, and then the approximate range of the number of rows in the table can be determined through multiple samplings. That is, after randomly extracting data from the table to be queried, the value of the row corresponding to the extracted data in the table to be queried can be directly read. It should be noted that the parameter in the sample() function must be greater than 0.000001 and less than 100; for example, data is randomly extracted from the table at a rate of 0.001%, and the number of rows of randomly extracted data is determined. The final value of the parameter in the sample() function is a value that meets the current requirements after multiple gradient changes.

[0072] Step S23: Calculate the total number of data rows in the table to be queried based on the extracted row values ​​and the parameters in the preset function.

[0073] In this embodiment, after determining the number of extracted rows corresponding to the extracted data, the total number of rows in the table to be queried is calculated based on the extracted row value and the parameters in the preset function. Specifically, when the extracted function value is controlled within a preset range, for example, when the number of rows is controlled within 10 to 100, the estimated number of table rows (i.e., the total number of data rows) can be obtained by dividing the number of rows by the parameters. Since a random result set is obtained, even executing the same SQL may result in inconsistent estimated table row counts. This method improves query speed by orders of magnitude compared to actually querying the number of table rows.

[0074] Step S24: Determine whether the total number of data rows is within the preset number of rows. If the total number of data rows is within the preset number of rows, then determine to perform the preset table splitting operation on the table to be queried.

[0075] In this embodiment, after calculating the total number of rows in the table to be queried based on the extracted row count and the parameters in the preset function, it is determined whether the total number of rows is within a preset row count range. If the total number of rows is within the preset row count range, it is determined that the preset table splitting operation will be performed on the table to be queried. It can be understood that if the total number of rows is within the preset row count range, it indicates that the table is too large and needs to be split into data blocks to improve query speed. If the total number of rows is lower than the minimum value of the preset row count range, it is determined that the preset table splitting operation will not be performed on the table to be queried. For example, if the sampling result is not within 10-100 and is lower than the minimum value of 10 in that range, it is considered that the number of rows in the table is too small, and the table will not be split.

[0076] Step S25: Obtain all data blocks corresponding to the table to be queried through the preset table splitting operation.

[0077] Step S26: Send the data block to the target thread in the thread pool to perform a preset information query operation.

[0078] For details regarding steps S21, S25, and S26, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0079] As can be seen, this embodiment of the application obtains a table to be queried from the database and acquires the target information of the table to be queried; randomly extracts data from the table to be queried based on parameters in a preset function and determines the number of rows corresponding to the extracted data; calculates the total number of data rows in the table to be queried based on the extracted row numbers and the parameters in the preset function; determines whether the value of the total number of data rows is within a preset number of rows; if the value of the total number of data rows is within the preset number of rows, determines to perform the preset table splitting operation on the table to be queried; obtains all data blocks corresponding to the table to be queried through the preset table splitting operation; and sends the data blocks to the target thread in the thread pool to execute the preset information query operation, thereby improving the query speed of table content and simplifying the operation steps.

[0080] See Figure 3 As shown, this embodiment of the invention discloses a data query method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0081] Step S31: Obtain the table to be queried from the database, and obtain the target information of the table to be queried.

[0082] Step S32: Based on the target information, determine whether to perform a preset table splitting operation on the table to be queried.

[0083] Step S33: If the preset table splitting operation is performed on the table to be queried, then all data blocks corresponding to the table to be queried are obtained through the preset table splitting operation.

[0084] In this embodiment, if the preset table splitting operation is performed on the table to be queried, all data blocks corresponding to the table to be queried are obtained through the preset table splitting operation. It can be understood that, based on the queried table field types, a target table splitting method is selected from multiple table splitting methods. Specifically, it is first determined whether the table to be queried contains a primary key. If the table to be queried contains a primary key, the field type corresponding to the primary key is obtained to obtain the target field type. When the target field type of the primary key is an integer type, the table to be queried is split based on the primary key and the extracted row values; or, when the target field type of the primary key is a numeric type, the table to be queried is split based on the primary key, the total number of data rows, and a custom preset data block value. That is, if the table has a primary key, the primary key will be used as the splitting field. For an integer primary key, the primary key information corresponding to the data obtained by quickly sampling the data obtained in the step of randomly extracting data from the table to be queried based on the parameters in the preset function and determining the extracted row values ​​corresponding to the extracted data can be used to split the table into multiple data blocks. If the primary key is numeric, the MOD() function is used to split the table into multiple data blocks. If the primary key is not present in the table to be queried, it is determined whether a numeric field exists in the table. If a numeric field exists, a field is randomly selected from all numeric fields to obtain a splitting reference field. The table is then split based on the splitting reference field, the total number of rows, and the custom preset data block value. It is understood that when a numeric field exists, the MOD() function can be used to split the table into multiple data blocks. If the primary key is not present in the table and the numeric field does not exist, the table is split based on the total number of rows and the custom preset data block value, for example, using ROWNUM rows to split the table into multiple data blocks. This method is slower than the previous two methods.

[0085] Step S34: Determine whether there is a data block containing a large object in the current batch of data blocks received by the thread pool.

[0086] In this embodiment, after obtaining all data blocks corresponding to the first batch of tables to be queried through the preset table partitioning operation, the first batch of data blocks is sent to the thread pool for concurrent querying. It is determined whether any data blocks containing large objects exist in the current batch of data blocks received by the thread pool. If a data block containing a large object exists, it is isolated and a unique query channel is reserved for it. It can be understood that by filtering for large objects and reserving a separate query channel for them, the overall query speed is improved by mixing large objects with other queries, as table queries with large objects are relatively slow.

[0087] It should be noted that while sending the first batch of data blocks to the thread pool for concurrent querying, the information of the next set of tables (i.e., the second batch of tables) is obtained. The table splitting steps described above are repeated to split the second batch of tables into multiple data blocks. When the thread pool is idle, the data blocks of the second batch are added to the thread pool for processing.

[0088] Step S35: If the data block containing the large object does not exist in the data block of the current batch, the data block is sent to the target thread in the thread pool to perform a preset information query operation.

[0089] In this embodiment, it is determined whether there is a data block containing a large object among the data blocks in the current batch received by the thread pool. If there is no data block containing a large object in the current batch, the data block is sent to the target thread in the thread pool to perform a preset information query operation. For example, if there are n threads in the thread pool, and there is no data block containing a large object among the data blocks in the current batch received by the thread pool, then n data blocks are selected from all the currently received data blocks and placed into the corresponding thread for querying.

[0090] Step S36: If there is a data block containing a large object in the data blocks of the current batch, then the data block containing the large object is sent to a reserved thread in the thread pool to perform a preset information query operation.

[0091] In this embodiment, after determining whether a data block containing a large object exists in the current batch of data blocks received by the thread pool, if such a data block exists, it is sent to a reserved thread in the thread pool to perform a preset information query operation. Specifically, the reserved thread is the only thread in the thread pool dedicated to processing the data block containing the large object; data blocks in the current batch that do not contain large objects are sent to the remaining threads in the thread pool that do not contain the reserved thread. For example, if there are n threads in the thread pool, and a data block containing a large object exists in the current batch of data blocks received by the thread pool, the data block containing the large object is sent to the only reserved thread in the thread pool, and n-1 data blocks that do not contain large objects are selected from the current batch and sent to the remaining threads in the thread pool. If the current thread completes its query, a corresponding number of data blocks are selected from the waiting data blocks in the current batch and sent to the aforementioned threads to continue performing the preset information query operation. If the reserved thread containing the data block containing the large object completes its execution, the data block not containing the large object will be sent to the reserved thread for querying.

[0092] It can be understood that after testing, the query speed for large objects is slow, and a certain amount of performance is not fully utilized. To address the problem of slow large-object queries, a dedicated query channel can be reserved and mixed with other queries to improve the overall query speed.

[0093] For example, assume that the query time for a table with large objects is X, and the query time for a table without large objects is Y. As Figure 4 shown, under normal circumstances, the total query time can be calculated as X + Y. Assume that after using a dedicated reserved query channel, the query times become MX and NY respectively. In this case, there are two situations. If both tables are executed and completed simultaneously, the maximum time is MX or NY; if one table is executed and completed first, the maximum elapsed time Z can be NY <= Z <= MX or as Figure 5 shown, MX <= Z <= NY. When Z < X + Y, taking the maximum value of Z gives MX < X + Y or NY < X + Y, which leads to X < Y / (M - 1) or Y < X(N - 1). When the large-object table, the table without large objects, and their execution speeds satisfy the above formula, the overall query speed is improved.

[0094] Step S37: If there are idle threads in the thread pool, determine whether there is a next batch of the data blocks.

[0095] In this embodiment, if there are idle threads in the thread pool, determine whether there is a next batch of the data blocks. It can be understood that if there are idle threads in the thread pool and there are no unprocessed data blocks in the current batch in the thread pool, determine whether there is a next batch of the data blocks.

[0096] Step S38: If there is a next batch of the data blocks, obtain the next batch of the data blocks and determine the next batch of the data blocks as the current batch of the data blocks.

[0097] In this embodiment, after determining whether there is a next batch of the data blocks, if there is a next batch of the data blocks, obtain the next batch of the data blocks and determine the next batch of the data blocks as the current batch of the data blocks. If there is no next batch of the data blocks, end.

[0098] Understandably, this solution improves overall query speed by using fuzzy analysis of the Oracle database to query table fields and quickly sample and estimate the number of table rows; by grouping database tables and quickly partitioning and querying tables with integer and numeric fields; and by employing algorithms for mixed queries on tables containing large objects and other tables. Specifically, it obtains table analysis information by querying table fields and sampling to estimate the number of table rows; based on this analysis, it groups the tables; it quickly partitions tables with integer and numeric fields; it performs concurrent multi-threaded queries on the partitioned data blocks; while querying, it begins analyzing the next group of tables, repeating the above steps until all table contents are retrieved, thus completing the query process. For databases containing multiple large tables, such as those with a large number of rows or large objects, the solution leverages fuzzy analysis to filter tables, quickly partitioning table content into multiple data blocks by processing integer and numeric fields, and effectively improving query speed by handling large object tables and other tables in combination.

[0099] Step S39: Re-enter the step of determining whether there is a data block containing a large object in the current batch of data blocks received by the thread pool.

[0100] For details regarding steps S31, S32, and S39, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0101] As can be seen, this application embodiment obtains a query table from the database and acquires target information of the query table; based on the target information, it determines whether to perform a preset table splitting operation on the query table; if the preset table splitting operation is performed on the query table, then all data blocks corresponding to the query table are obtained through the preset table splitting operation; it determines whether there is a data block containing a large object in the current batch of data blocks received by the thread pool; if there is no data block containing a large object in the current batch of data blocks, then the data block is sent to the target thread in the thread pool to perform a preset information query operation; if the current batch of data blocks does not contain a large object, then the data block is sent to the target thread in the thread pool to perform a preset information query operation; if the current batch of data blocks does not contain a large object, then the data block is sent to the target thread in the thread pool to perform a preset information query operation. If a data block containing a large object exists in the batch of data blocks, the data block containing the large object is sent to a reserved thread in the thread pool to perform a preset information query operation. If there is an idle thread in the thread pool, it is determined whether there is a next batch of data blocks. If there is a next batch of data blocks, the next batch of data blocks is obtained and identified as the data blocks of the current batch. The process then re-enters the step of determining whether there is a data block containing a large object in the current batch of data blocks received by the thread pool. This improves the query speed of table content and simplifies the operation steps.

[0102] See Figure 6 As shown in the figure, this application also discloses a data query device, including:

[0103] The information acquisition module 11 is used to acquire the table to be queried in the database and acquire the target information of the table to be queried.

[0104] The segmentation judgment module 12 is used to determine whether to perform a preset table segmentation operation on the table to be queried based on the target information;

[0105] The segmentation module 13 is used to obtain all data blocks corresponding to the table to be queried through the preset table segmentation operation if the preset table segmentation operation is performed on the table to be queried.

[0106] The data block sending module 14 is used to send the data block to the target thread in the thread pool so as to perform a preset information query operation.

[0107] As can be seen, this application includes: obtaining a table to be queried from a database and obtaining target information of the table to be queried; determining whether to perform a preset table partitioning operation on the table to be queried based on the target information; if the preset table partitioning operation is performed on the table to be queried, obtaining all data blocks corresponding to the table to be queried through the preset table partitioning operation; and sending the data blocks to a target thread in a thread pool to execute a preset information query operation. Therefore, this application can obtain data information for determining whether the current table needs to be partitioned based on the target information of the table. If the data information indicates that the current table needs to be partitioned, the table is partitioned into a preset number of data blocks, and then the data blocks are queried synchronously in the thread pool. The size of the data blocks is much smaller than the original table size, which improves the query speed of table content and simplifies the operation steps under concurrent processing.

[0108] In some specific embodiments, the information acquisition module 11 specifically includes:

[0109] The query table reading unit is used to retrieve the query table from the database;

[0110] The target information acquisition unit is used to acquire the field types of the table to be queried and the large object query results.

[0111] In some specific embodiments, the segmentation judgment module 12 specifically includes:

[0112] A data random extraction unit is used to randomly extract data from the query table based on parameters in a preset function.

[0113] The extracted row value determination unit is used to determine the extracted row value corresponding to the extracted data;

[0114] The total number of data rows calculation unit is used to calculate the total number of data rows in the table to be queried based on the extracted row values ​​and the parameters in the preset function.

[0115] The total number of data rows determination unit is used to determine whether the value of the total number of data rows is within a preset number of rows;

[0116] The first segmentation determination unit is used to determine that if the total number of data rows is within the preset number of rows, the preset table segmentation operation will be performed on the table to be queried.

[0117] The second segmentation determination unit is used to determine that if the total number of data rows is lower than the minimum value of the preset number of rows range, the preset table segmentation operation will not be performed on the table to be queried.

[0118] In some specific embodiments, the segmentation module 13 specifically includes:

[0119] The primary key determination unit is used to determine whether the table to be queried contains a primary key if the preset table splitting operation is performed on the table to be queried.

[0120] The target field type acquisition unit is used to obtain the field type corresponding to the primary key if the table to be queried contains a primary key, so as to obtain the target field type;

[0121] The first table segmentation unit is used to segment the table to be queried based on the primary key and the extracted row values ​​when the target field type is an integer type, so as to obtain all the corresponding data blocks;

[0122] The second table segmentation unit is used to segment the table to be queried based on the primary key, the total number of data rows, and the value of a custom preset data block when the target field type is numeric, so as to obtain all the corresponding data blocks.

[0123] A numeric type determination unit is used to determine whether a field of the numeric type exists in the table to be queried if the primary key is not contained in the table to be queried.

[0124] The segmentation reference field determination unit is used to randomly select a field from all the numeric type fields in the query table if the numeric type field exists, so as to obtain the segmentation reference field.

[0125] The third table segmentation unit is used to segment the table to be queried based on the segmentation reference field, the total number of data rows, and the custom preset data block value, so as to obtain all the corresponding data blocks;

[0126] The fourth table segmentation unit is used to segment the table to be queried based on the total number of data rows and the custom preset data block value if the table to be queried does not contain the primary key and does not have the field of the numeric type, so as to obtain all the corresponding data blocks.

[0127] In some specific embodiments, the data block sending module 14 specifically includes:

[0128] The large object determination unit is used to determine whether there is a data block containing a large object in the current batch of data blocks received by the thread pool;

[0129] The first data block sending unit is configured to send the data block to the target thread in the thread pool if there is no data block containing a large object in the data blocks of the current batch, so as to perform a preset information query operation.

[0130] The second data block sending unit is configured to send the data block containing the large object to a reserved thread in the thread pool if there is a data block containing the large object in the current batch of data blocks, so as to perform a preset information query operation; the reserved thread is the only thread in the thread pool used to process the data block containing the large object;

[0131] The third data block sending unit is used to send data blocks that do not contain large objects in the current batch to the remaining threads in the thread pool that do not contain the reserved threads, so as to perform a preset information query operation;

[0132] The data block determination unit is used to determine whether there is a next batch of the data blocks if there are idle threads in the thread pool.

[0133] The next batch of data block acquisition unit is used to acquire the data block of the next batch if the data block of the next batch exists, and determine the data block of the next batch as the data block of the current batch;

[0134] The large object re-judgment unit is used to re-enter the step of judging whether there is a data block containing a large object in the current batch of data blocks received by the thread pool;

[0135] The termination unit is used to terminate the process if the next batch of the data blocks does not exist.

[0136] Furthermore, embodiments of this application also provide an electronic device. Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0137] Figure 7 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data query method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0138] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0139] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0140] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the data query method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0141] Furthermore, this application also discloses a storage medium storing a computer program, which, when loaded and executed by a processor, implements the data query method steps disclosed in any of the foregoing embodiments.

[0142] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0143] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0144] The above provides a detailed description of the data query method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data query method, characterized in that, include: Obtain the table to be queried from the database, and obtain the target information of the table to be queried; The target information includes field types and large object query results; Based on the target information obtained by fuzzy analysis, it is determined whether the query execution time of the table to be queried meets the preset conditions, and if so, it is determined whether to perform a preset table splitting operation on the table to be queried. If the preset table splitting operation is performed on the table to be queried, then all data blocks corresponding to the table to be queried are obtained through the preset table splitting operation; If the table to be queried contains a primary key, then the preset table splitting operation is an operation of splitting the table to be queried based on the field type corresponding to the primary key and the primary key; The data block is sent to the target thread in the thread pool to perform a preset information query operation; The step of sending the data block to the target thread in the thread pool includes: Determine whether any data blocks containing large objects exist in the current batch of data blocks received by the thread pool; If the data block containing the large object does not exist in the current batch of data blocks, then the data block is sent to the target thread in the thread pool; If the data block containing the large object exists in the current batch of data blocks, then the data block containing the large object is sent to the reserved thread in the thread pool; the reserved thread is the only thread in the thread pool used to process the data block containing the large object. Send data blocks that do not contain large objects in the current batch to the remaining threads in the thread pool that do not contain the reserved threads; Determining whether to perform a preset table partitioning operation on the table to be queried based on the target information includes: Data is randomly extracted from the table to be queried based on the parameters in the preset function, and the value of the extracted row corresponding to the extracted data is determined; the final value of the parameters in the preset function is the random extraction ratio value that meets the current requirements after multiple gradient changes. The total number of rows in the table to be queried is calculated based on the extracted row values ​​and the parameters in the preset function. Determine whether the total number of rows of data is within the preset number of rows; If the total number of data rows is within the preset number of rows, then it is determined that the preset table splitting operation will be performed on the table to be queried; If the total number of rows is lower than the minimum value of the preset row range, then it is determined that the preset table splitting operation will not be performed on the table to be queried. The step of obtaining all data blocks corresponding to the table to be queried through the preset table splitting operation includes: If the table to be queried contains a primary key, then obtain the field type corresponding to the primary key to get the target field type; When the target field type is an integer, the table to be queried is divided based on the primary key and the extracted row values ​​to obtain all the corresponding data blocks; Alternatively, when the target field type is numeric, the table to be queried is divided based on the primary key, the total number of data rows, and the value of a custom preset data block to obtain all the corresponding data blocks; The step of obtaining all data blocks corresponding to the table to be queried through the preset table splitting operation includes: If the primary key is not contained in the table to be queried, then determine whether the numeric field exists in the table to be queried; If the table to be queried contains a field of the numeric type, then randomly select one field from all the numeric type fields to obtain the segmentation reference field; The table to be queried is segmented based on the segmentation reference field, the total number of data rows, and the custom preset data block value to obtain all the corresponding data blocks; If the table to be queried does not contain the primary key and does not have the field of the numeric type, then the table to be queried is divided based on the total number of data rows and the value of the custom preset data block to obtain all the corresponding data blocks.

2. The data query method according to claim 1, characterized in that, After sending the data block to the target thread in the thread pool, the method further includes: If there are idle threads in the thread pool, then determine whether there is a next batch of the data blocks; If the next batch of data blocks exists, then the next batch of data blocks is obtained, and the next batch of data blocks is determined as the data blocks of the current batch; Re-enter the step of determining whether there is a data block containing a large object in the current batch of data blocks received by the thread pool; If the next batch of the data block does not exist, the process ends.

3. A data query device, characterized in that, include: The information acquisition module is used to acquire the table to be queried in the database and to acquire the target information of the table to be queried. The target information includes field types and large object query results; The segmentation judgment module is used to determine whether the query execution time of the table to be queried meets the preset conditions based on the target information through fuzzy analysis, and when it does, to determine whether to perform a preset table segmentation operation on the table to be queried. If the table to be queried contains a primary key, then the preset table splitting operation is an operation of splitting the table to be queried based on the field type corresponding to the primary key and the primary key; The segmentation module is used to obtain all data blocks corresponding to the table to be queried through the preset table segmentation operation if the preset table segmentation operation is performed on the table to be queried. The data block sending module is used to send the data block to the target thread in the thread pool so as to execute a preset information query operation; The data block sending module includes: The large object determination unit is used to determine whether there is a data block containing a large object in the current batch of data blocks received by the thread pool; The first data block sending unit is configured to send the data block to the target thread in the thread pool if there is no data block containing a large object in the data blocks of the current batch, so as to perform a preset information query operation. The second data block sending unit is configured to send the data block containing the large object to a reserved thread in the thread pool if there is a data block containing the large object in the current batch of data blocks, so as to perform a preset information query operation; the reserved thread is the only thread in the thread pool used to process the data block containing the large object; The third data block sending unit is used to send data blocks that do not contain large objects in the current batch to the remaining threads in the thread pool that do not contain the reserved threads; The segmentation judgment module includes: A data random sampling unit is used to randomly sample data from the query table based on parameters in a preset function; the final value of the parameters in the preset function is a random sampling ratio value that meets the current requirements after multiple gradient changes. The extracted row value determination unit is used to determine the extracted row value corresponding to the extracted data; The total number of data rows calculation unit is used to calculate the total number of data rows in the table to be queried based on the extracted row values ​​and the parameters in the preset function. The total number of data rows determination unit is used to determine whether the value of the total number of data rows is within a preset number of rows; The first segmentation determination unit is used to determine that if the total number of data rows is within the preset number of rows, the preset table segmentation operation will be performed on the table to be queried. The second segmentation determination unit is used to determine that if the total number of data rows is lower than the minimum value of the preset number of rows range, the preset table segmentation operation will not be performed on the table to be queried. The segmentation module includes: The target field type acquisition unit is used to obtain the field type corresponding to the primary key if the table to be queried contains a primary key, so as to obtain the target field type; The first table segmentation unit is used to segment the table to be queried based on the primary key and the extracted row values ​​when the target field type is an integer type, so as to obtain all the corresponding data blocks; The second table segmentation unit is used to segment the table to be queried based on the primary key, the total number of data rows, and the value of a custom preset data block when the target field type is numeric, so as to obtain all the corresponding data blocks. A numeric type determination unit is used to determine whether a field of the numeric type exists in the table to be queried if the primary key is not contained in the table to be queried. The segmentation reference field determination unit is used to randomly select a field from all the numeric type fields in the query table if the numeric type field exists, so as to obtain the segmentation reference field. The third table segmentation unit is used to segment the table to be queried based on the segmentation reference field, the total number of data rows, and the custom preset data block value, so as to obtain all the corresponding data blocks; The fourth table segmentation unit is used to segment the table to be queried based on the total number of data rows and the custom preset data block value if the table to be queried does not contain the primary key and does not have the field of the numeric type, so as to obtain all the corresponding data blocks.

4. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data query method as described in claim 1 or 2.

5. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the data query method as described in claim 1 or 2.

Images