Data query method, device, computer equipment, storage medium and program product
By calculating the redundancy of each column in a MySQL database table and selecting index columns that meet the criteria, the problem of index failure and slow query caused by redundancy in the data table is solved, and efficient and accurate data query is achieved.
Patent Information
- Application Number
- CN202410377131.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-29
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-03-29
AI Technical Summary
In existing technologies, MySQL database tables contain a large amount of duplicate and redundant data, which leads to problems such as index failure, slow queries, and high master-slave synchronization latency. Furthermore, index settings rely on the experience of developers, resulting in low accuracy and efficiency.
By calculating the repetition rate of each column in the data table to be analyzed, and combining the task identifier to be analyzed, columns that meet the preset conditions for repetition are selected as index columns, thus excluding data with high repetition rate and little relationship, thereby improving the accuracy and efficiency of queries.
It improves the accuracy and efficiency of data retrieval, reduces the interference of duplicate data on the query process, and increases query speed.
Smart Images

Figure CN118296001B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data analysis technology, and in particular to a data query method, device, computer equipment, storage medium and program product. Background Art
[0002] MySQL is a relational database commonly used in software development. It consists of libraries and data tables. Software developers can store the data they need to store in MySQL by designing data tables.
[0003] As time goes by and business volume increases, software developers constantly update newly acquired data in data tables, resulting in more and more data in a single data table, and there may be a large amount of duplicate and redundant data in the data table. High duplication of data in each column of the table will cause index failure, resulting in full table scan and slow query. A large amount of duplicate and redundant data will also cause problems such as high master-slave synchronization delay and slow data table backup. Many problems can only be discovered during query, which greatly affects the query speed. The related technology relies on the experience of software developers to judge how to set indexes and how to query data more effectively, which has low accuracy and efficiency. Summary of the invention
[0004] In view of this, the present invention provides a data query method, apparatus, computer equipment, storage medium and program product to solve the problems of low accuracy and low efficiency when performing data query.
[0005] In a first aspect, the present invention provides a data query method, which includes: obtaining a data table to be analyzed; calculating the repetition of each column of data in the data table to be analyzed respectively; the repetition is used to indicate the proportion of repeated data in each column of data to the total number of data items; based on the repetition of each column of data and a preset task identifier to be analyzed, selecting a column corresponding to the task identifier to be analyzed and whose repetition meets preset conditions as an index column to perform data query.
[0006] The above scheme first calculates the repetition of each column of data in the data table to be analyzed, obtains the repetition of each column of data, and then combines the task identifier to be analyzed and the repetition of each column of data to select the column corresponding to the task identifier to be analyzed and the repetition that meets the preset conditions as the index column, so that when subsequent data queries are performed, the task identifier to be analyzed can be targeted and the repetition of the index column can be guaranteed to be reasonable and accurate, and the interference of highly repetitive data and data that are not closely related to the task to be analyzed on the query process is eliminated, thereby improving the query speed and efficiency.
[0007] In an optional embodiment, the respectively calculating the degree of duplication of each column of data in the data table to be analyzed includes: obtaining the total number of data items in the data table to be analyzed; performing a deduplication operation on each column of data respectively, and recording the amount of deduplicated data in each column of data respectively; and determining the degree of duplication of each column of data based on the quotient of the amount of deduplicated data in each column of data and the total number of data items.
[0008] The above scheme defines how to calculate the repetition of each column of data in the data table to be analyzed and refines the scheme.
[0009] In an optional embodiment, based on the repetition of each column of data and a preset task identifier to be analyzed, a column corresponding to the task identifier to be analyzed and with a repetition meeting preset conditions is selected as an index column, including: sorting the repetition of each column of data from small to large; selecting a first target number of columns corresponding to the task identifier to be analyzed and with the smallest repetition as index columns.
[0010] The above scheme limits how to select index columns and refines the scheme.
[0011] In an optional embodiment, the method also includes: detecting whether the data table to be analyzed contains a preset index column; if it contains, calculating the repetition of the preset index column; comparing the repetition of the preset index column with a preset repetition threshold, if the repetition of the preset index column exceeds the preset repetition threshold, then based on the repetition of each column of data and the task identifier to be analyzed, selecting a column corresponding to the task identifier to be analyzed and whose repetition meets the preset conditions as the recommended index column.
[0012] The above scheme first detects whether the data table to be analyzed has a preset index column. If so, it determines whether the preset index column is reasonable. If not, it gives a recommended index column to ensure the accuracy and efficiency of data query.
[0013] In an optional embodiment, the method also includes: detecting the number of columns of the preset index column; comparing the number of columns of the preset index column with a preset column number threshold, if the number of columns of the preset index column exceeds the preset column number threshold, sorting the repetition of each column of the preset index column from small to large, and selecting the second target number of columns with the smallest repetition as the combined index column.
[0014] The above scheme determines whether the number of preset index columns is reasonable. If the number is too large, preset index columns with small duplication are selected for combination as combined index columns to ensure the efficiency of data query.
[0015] In an optional embodiment, the method further includes: comparing the total number of data items in the data table to be analyzed with a preset data volume threshold; if the total number of data items in the data table to be analyzed exceeds the preset data volume threshold, displaying a warning control in the interface of the data table to be analyzed; the warning control is used to prompt a user that the total number of data items in the data table to be analyzed is excessive.
[0016] The above scheme determines whether the total number of data in the data table to be analyzed is reasonable. If it is excessive, the user is prompted that the total number of data is excessive, so as to warn the user to optimize and ensure the efficiency of data query.
[0017] In a second aspect, the present invention provides a data query device, the device comprising:
[0018] A data acquisition module is used to obtain the data table to be analyzed;
[0019] A repetition calculation module is used to calculate the repetition of each column of data in the data table to be analyzed; the repetition is used to indicate the proportion of repeated data in each column of data to the total number of data;
[0020] The index column determination module is used to select a column corresponding to the task identifier to be analyzed and having a repetition degree that meets preset conditions as an index column based on the repetition degree of each column of data and the preset task identifier to be analyzed, so as to perform data query.
[0021] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data query method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0022] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data query method of the first aspect or any corresponding embodiment thereof.
[0023] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the data query method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0025] Figure 1 is a flow chart of a data query method according to an embodiment of the present invention;
[0026] Figure 2 is a flow chart of another data query method according to an embodiment of the present invention;
[0027] Figure 3 is a flow chart of another data query method according to an embodiment of the present invention;
[0028] Figure 4 is a structural block diagram of a data query device according to an embodiment of the present invention;
[0029] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0031] MySQL is a relational database commonly used in software development. It consists of libraries and data tables. Software developers can store the data they need to store in MySQL by designing data tables.
[0032] As time goes by and business volume increases, software developers constantly update newly acquired data in data tables, resulting in more and more data in a single data table. In addition, since there may be repeated steps or modules in the software development process, there may be a large amount of duplicate and redundant data in the data table, which takes up storage space. High repetition of data in each column of the table will cause index failure, resulting in full table scans and slow queries. A large amount of duplicate and redundant data will also cause problems such as high master-slave synchronization delays and slow data table backups. Many problems can only be discovered during specific query operations, which greatly affects query speed. Related technologies rely on the experience of software developers to judge how to set indexes and how to query data more effectively, which has low accuracy and efficiency.
[0033] Therefore, an embodiment of the present invention provides a data query method, which selects index columns by analyzing the repetition in a data table to improve the query accuracy and efficiency.
[0034] According to an embodiment of the present invention, a data query method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0035] In this embodiment, a data query method is provided, which can be used in the above-mentioned mobile terminal, such as a laptop computer, a desktop computer, etc. Figure 1 is a flow chart of a data query method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:
[0036] Step S101, obtaining a data table to be analyzed.
[0037] The data table to be analyzed can be a data table pre-recorded and stored by the user. The data table to be analyzed includes rows and columns, and the specific data recorded in each row and column is set according to the needs. For example, if the data table records the grades of middle school students in a certain school, each column can be the school, school year, class, student number, student, subject, grade, etc., and each row is for various information of a single student.
[0038] Step S102, respectively calculating the repetition degree of each column of data in the data table to be analyzed.
[0039] The repetition rate is used to indicate the proportion of repeated data in each column of data to the total number of data. After obtaining the repetition rate of each column of data in the data table to be analyzed, it is possible to analyze whether the column of data is suitable as an index for data query based on the repetition rate of each column of data. For example, if the repetition rate of a column of data is 100%, it means that the data in this column is the same and cannot be used as an index.
[0040] Step S103 , based on the repetition of each column of data and the preset task identifier to be analyzed, a column corresponding to the task identifier to be analyzed and having a repetition that meets a preset condition is selected as an index column to perform data query.
[0041] The task identifier to be analyzed contains characteristic information of the task to be analyzed, which is related to the data in the data table to be analyzed. It can be set by the user according to the task requirements. For example, if the user needs to analyze the grades of a student in a class in a school in a certain academic year, the pre-stored data table to be analyzed records at least the school, academic year, class, student, and grades. The task identifier to be analyzed includes school, academic year, class, student, and grades. It should be noted that the data table to be analyzed may contain data other than the task identifier to be analyzed, such as the age of the student.
[0042] When the user performs a data query, the user can select the task identifier to be analyzed according to the task requirements, and the computer device can analyze the task identifier to be analyzed and the repetition of each column of data, and select the column corresponding to the task identifier to be analyzed and the repetition that meets the preset conditions as the index column. The preset conditions can be set according to the requirements, for example, the repetition is set to be less than 20%.
[0043] The data query method provided in this embodiment first calculates the repetition of each column of data in the data table to be analyzed, obtains the repetition of each column of data, and then combines the task identifier to be analyzed and the repetition of each column of data to select a column corresponding to the task identifier to be analyzed and whose repetition meets the preset conditions as an index column, so that subsequent data queries can be targeted at the task identifier to be analyzed and the repetition of the index column can be guaranteed to be reasonable and accurate, and interference with the query process by data with high repetition and data that is not closely related to the task to be analyzed is eliminated, thereby improving the query speed and efficiency.
[0044] In this embodiment, a data query method is provided, which can be used in the above-mentioned mobile terminal, such as a laptop computer, a desktop computer, etc. Figure 2 is a flow chart of a data query method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0045] Step S201, obtaining a data table to be analyzed.
[0046] Optionally, the computer device can also pre-acquire the task identifier to be analyzed input by the user. The user can input the task identifier to be analyzed through the interactive interface, for example, input a description statement corresponding to the task to be analyzed or directly input the keywords of the task to be analyzed. The computer device responds to the description statement input by the user, splits the description statement by keywords, and compares the split keywords with the type information in the database to filter out the task identifier to be analyzed, or the computer device responds to the keywords input by the user, compares the keywords with the type information in the database, and filters out the task identifier to be analyzed. Exemplarily, if the task to be analyzed is to query the grades of all students in the Y class of X school, then the description statement "query the grades of all students in the Y class of X school" can be input, and the computer device splits out the keywords "X school", "Y class", "all students" and "grades", or the user directly inputs the keywords "X school", "Y class", "all students" and "grades".
[0047] For example, a client logs in to a MySQL database, which stores at least data related to the task to be analyzed. After logging in, the user can select, export or download the required data table to be analyzed in the interface. First, the table names of all data tables are displayed in the interface, and the user can select the data table to be analyzed by the table name. After selecting the data table to be analyzed, the DDL (Data Definition Language, used to define database objects, such as libraries, tables, columns, etc.) information of the data table to be analyzed can be obtained to obtain all column data of the data table to be analyzed.
[0048] Optionally, analyze whether the column data can be null (empty) to determine the validity of the column data. For example, if column A must have a value, but the table creation statement for column A defines that the data in column A can be null, then null is determined to be a status value, the data in column A is valid, or there is a problem with the table creation statement and needs to be repaired.
[0049] Step S202, respectively calculating the repetition degree of each column of data in the data table to be analyzed.
[0050] The repetition degree is used to indicate the proportion of repeated data in each column of data to the total number of data entries.
[0051] Specifically, the above step S202 includes:
[0052] Step S2021, obtaining the total number of data items in the data table to be analyzed.
[0053] The data table to be analyzed includes rows and columns. The total number of data items indicates how many rows there are in total. After obtaining the data table to be analyzed, this solution will first align the data to ensure that each row of data corresponds to the information of the same statistical object (such as the school, class, student number, etc. of the same student), and each column of data corresponds to the same type of statistical information (for example, all student numbers in the same column), and then generate and select at least one column of data as the index column.
[0054] Step S2022, perform deduplication operation on each column of data respectively, and record the deduplication data volume of each column of data respectively.
[0055] The deduplicated data volume is the data volume after deduplication, that is, the amount of non-duplicate data remaining after removing duplicate data. The smaller the deduplicated data volume of the column data, the higher the duplication of the column data, and the more difficult it is for the column to reflect the difference between each row of data, and the less suitable the column is as an index.
[0056] Step S2023, determining the duplication of each column of data based on the quotient of the amount of deduplicated data in each column of data and the total number of data items.
[0057] Optionally, the duplication of each column of data is calculated by the formula "duplication of column data = 1 - amount of deduplicated data in the column of data / total number of data items".
[0058] Step S203 , based on the repetition of each column of data and the identification of the task to be analyzed, a column corresponding to the identification of the task to be analyzed and having a repetition that meets a preset condition is selected as an index column to perform data query.
[0059] Optionally, when the task to be analyzed is identified as preset by the user or output in advance, step S203 executed by the computer includes:
[0060] Step S2031, sorting the repetition degree of each column of data from small to large.
[0061] Optionally, the first n pieces of data with the highest degree of repetition in each column and the number of times each of the first n pieces of data appears are recorded to facilitate analysis of the specific circumstances of each column of data.
[0062] Step S2032 : Selecting a first target number of columns corresponding to the task identifier to be analyzed and having the smallest degree of repetition as index columns.
[0063] The first target number can be set according to actual needs. For example, if it is set to 4, it means that 4 columns corresponding to the task identifiers to be analyzed and with the smallest duplication are selected as index columns for subsequent data query.
[0064] In another possible implementation of the embodiment of the present application, the computer device first selects the columns whose repetition meets the preset conditions as candidate index columns based on the repetition of each column of data, and then displays the keywords corresponding to the index columns on the interface; when receiving the user's selection operation for the keyword (for example, clicking the keyword or entering the corresponding keyword in the input box), the candidate index column corresponding to the keyword is selected as the index column that the user needs to use. Optionally, since the data table to be analyzed may have pre-contained a default index column or an index column has been set, when the data table to be analyzed is obtained, it is also possible to first detect whether the data table to be analyzed contains a preset index column, and if it contains a preset index column, the rationality of the preset index column is judged. Specifically, first detect whether the data table to be analyzed contains a preset index column. If it does, calculate the repetition of the preset index column, and then compare the repetition of the preset index column with the preset repetition threshold. If the repetition of the preset index column exceeds the preset repetition threshold, based on the repetition of each column of data and the task identifier to be analyzed, select the column corresponding to the task identifier to be analyzed and the repetition meets the preset conditions as the recommended index column; if it does not contain, display the index prompt control in the interface of the data table to be analyzed, and the index prompt control is used to prompt the user that the data table to be analyzed does not contain an index column and suggest the user to add an index column to improve the query efficiency. That is to say, in the case of containing a preset index column, if the repetition of the preset index column exceeds the preset repetition threshold, that is, when the repetition of the preset index column is high, the preset index column is not suitable as an index for data query, so a column with low repetition can be selected and recommended to the user, so that the user can replace a more reasonable index column according to the recommendation. The preset repetition threshold can be set according to actual needs, for example, set to 20%. If no index column is included, a reasonable index column can be recommended to the user by using the method of selecting index columns in this solution (ie, selecting the first target number of columns corresponding to the task identifier to be analyzed and with the smallest repetition as index columns).
[0065] For example, if the data table to be analyzed does not contain a preset index column, a prompt "There is currently no index. It is recommended to add an index according to the index duplication to speed up the query" is given in the index prompt control.
[0066] Furthermore, since the data table to be analyzed may contain multiple preset index columns, if the number of preset index columns is too large, it will cause large space occupation, increase maintenance difficulty, and affect the efficiency of data query. Therefore, the number of preset index columns can be judged, and when the number is too large, the preset index columns can be optimized, such as screening or combining, to improve the query efficiency. Specifically, the number of columns of the preset index column is first detected, and then the number of columns of the preset index column is compared with the preset column number threshold. If the number of columns of the preset index column exceeds the preset column number threshold, the repetition of each column of the preset index column is sorted from small to large, and the second target number of columns with the smallest repetition is selected as the index column. The preset column number threshold can be set according to actual needs, for example, it is set to 4, indicating that when the number of preset index columns exceeds 4, it is judged that the number is too large and needs to be optimized. The second target number can also be set according to actual needs, for example, it is set to 1, indicating that the column with the smallest repetition is used as the index column, or it is set to 2, indicating that the two columns with the smallest repetition are used as index columns. After selecting the second target number of columns with the smallest repetition as the index column, cancel the other preset index columns. Furthermore, the second target number of columns with the smallest repetition degree can be combined to form a combined index column to facilitate data query. For example, if the columns with the smallest repetition degree are the subject column and the score column, the name column and the score column are combined into a json format. In this way, the subject column and the score column will be combined into one column, reducing the total number of data items and improving the query speed.
[0067] Optionally, when the total number of data items in the data table to be analyzed is excessive, the user can also be prompted to optimize the data table to be analyzed. Specifically, the total number of data items in the data table to be analyzed is compared with the preset data volume threshold. If the total number of data items in the data table to be analyzed exceeds the preset data volume threshold, a warning control is displayed in the interface of the data table to be analyzed. The warning control is used to prompt the user that the total number of data items in the data table to be analyzed is excessive. The warning control can also display prompt information, such as "The current data volume is too large, and the query performance will decrease. It is recommended to use methods such as read-write separation and sub-library and table sharding to improve query performance." The preset data volume threshold depends on factors such as hardware equipment, system environment, and usage scenarios. Exceeding the preset data volume threshold will slow down the speed of querying, adding, deleting, and modifying the MySQL database. The preset data volume threshold can be set according to actual needs, for example, set to 20 million.
[0068] The data query method provided in this embodiment first calculates the repetition of each column of data in the data table to be analyzed, obtains the repetition of each column of data, and then combines the task identifier to be analyzed and the repetition of each column of data to select a column corresponding to the task identifier to be analyzed and whose repetition meets the preset conditions as an index column, so that subsequent data queries can be targeted at the task identifier to be analyzed and the repetition of the index column can be guaranteed to be reasonable and accurate, and interference with the query process by data with high repetition and data that is not closely related to the task to be analyzed is eliminated, thereby improving the query speed and efficiency.
[0069] As one or more specific application embodiments of the embodiments of the present invention, the optimal implementation scheme or the scheme that the inventor most wants to embody is described below in combination with specific application scenarios.
[0070] Figure 3 is a flow chart of a data query method according to an embodiment of the present invention. Figure 3As shown, first connect to the MySQL database through the account password IP address, select the table to be analyzed (the data table to be analyzed), obtain the DDL information of the table, obtain the names of each column, and obtain the data volume of the table (the total number of data items dataCount). Exemplarily, the total number of data items in the data table to be analyzed is obtained through the instruction "(dataCount)select count(*)from t)". Then calculate the repetition (repeat) of each column of data in the table, and calculate the corresponding number of occurrences of the top 10 data items with the highest repetition in each column. Exemplarily, perform a deduplication operation on the column data through the instruction "select count(distinct coloumName)from t" to obtain the deduplication data volume of the column data. Obtain the top 10 data items with the highest repetition in the column data and the number of occurrences of each of the top 10 data items through the instruction "SELECT column_name,COUNT(*)as count FROM table_nameGROUP BY column_name ORDER BY count DESC LIMIT 10". Then query the index information of the current table. If the current table has no index, it will give a prompt "There is no index currently. It is recommended to add indexes according to the index repetition" to speed up the query and give optimization suggestions. If the current table has an index, first query the number of indexes. If it is greater than 4, it will give a prompt "There are too many indexes in the current table, which can be optimized to a combined index" to reduce space usage and reduce index maintenance costs, and give optimization suggestions; if it is less than 4, determine whether the index is suitable. If it is suitable, determine that the current index does not need to be modified and end the process. If it is not suitable, it will give a prompt "The index repetition is greater than 20%, which is easy to fail. Please consider using other columns as index columns" and give optimization suggestions. This optimization suggestion is to arrange the previously calculated repetition of each column from low to high and select a suitable index. Finally, determine whether the amount of table data is appropriate. If it is excessive, it is necessary to give optimization suggestions such as read-write separation, sub-library and sub-table.
[0071] In the present embodiment, a data query device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.
[0072] This embodiment provides a data query device, such as Figure 4 As shown, including:
[0073] The data acquisition module 401 is used to acquire the data table to be analyzed;
[0074] The repetition calculation module 402 is used to calculate the repetition of each column of data in the data table to be analyzed; the repetition is used to indicate the proportion of repeated data in each column of data to the total number of data;
[0075] The index column determination module 403 is used to select a column corresponding to the task identifier to be analyzed and having a repetition degree that meets a preset condition as an index column based on the repetition degree of each column of data and the preset task identifier to be analyzed, so as to perform data query.
[0076] In an optional embodiment, the duplication calculation module is also used to: obtain the total number of data items in the data table to be analyzed; perform deduplication operations on each column of data respectively, and record the amount of deduplicated data in each column of data respectively; determine the duplication of each column of data based on the quotient of the amount of deduplicated data in each column of data and the total number of data items.
[0077] In an optional implementation, the index column determination module is further used to: sort the repetition of each column of data from small to large; select a first target number of columns corresponding to the task identifier to be analyzed and with the smallest repetition as index columns.
[0078] In an optional embodiment, it also includes a preset index detection module, which is used to detect whether the data table to be analyzed contains a preset index column; if it does, calculate the repetition of the preset index column; compare the repetition of the preset index column with a preset repetition threshold value, if the repetition of the preset index column exceeds the preset repetition threshold value, then based on the repetition of each column of data and the task identifier to be analyzed, select a column corresponding to the task identifier to be analyzed and whose repetition meets the preset conditions as the recommended index column.
[0079] In an optional embodiment, the preset index detection module is also used to: detect the number of columns of the preset index column; compare the number of columns of the preset index column with a preset column number threshold, if the number of columns of the preset index column exceeds the preset column number threshold, sort the repetition of each column of the preset index column from small to large, and select the second target number of columns with the smallest repetition as the combined index column.
[0080] In an optional embodiment, it also includes a data volume comparison module, which is used to compare the total number of data items in the data table to be analyzed with a preset data volume threshold; if the total number of data items in the data table to be analyzed exceeds the preset data volume threshold, a warning control is displayed in the interface of the data table to be analyzed; the warning control is used to prompt the user that the total number of data items in the data table to be analyzed is excessive.
[0081] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0082] The data query device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0083] The embodiment of the present invention also provides a computer device having the above Figure 4 The data query device shown.
[0084] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.
[0085] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0086] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0087] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0088] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0089] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.
[0090] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.
[0091] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0092] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0093] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A data query method, characterized in that: The method comprises: Get the data table to be analyzed; Calculate the repetition of each column of data in the data table to be analyzed respectively; the repetition is used to indicate the proportion of repeated data in each column of data to the total number of data; Based on the repetition of each column of data and the preset task identifier to be analyzed, a column corresponding to the task identifier to be analyzed and having a repetition that meets the preset conditions is selected as an index column to perform data query; The method further comprises: In response to the description sentence input by the user, the description sentence is split by keywords, and the split keywords are compared with the type information in the database to filter out the task identifier to be analyzed; or, in response to the keyword input by the user, the keyword is compared with the type information in the database to filter out the task identifier to be analyzed; The selecting, based on the repetition of each column of data and a preset identification of the task to be analyzed, a column corresponding to the identification of the task to be analyzed and having a repetition that meets a preset condition as an index column includes: Sort the repetition of each column of data from small to large; Selecting a first target number of columns corresponding to the task identifier to be analyzed and having the smallest degree of repetition as index columns; The method further comprises: Check whether the data table to be analyzed contains a preset index column; If included, calculate the repetition degree of the preset index column; The repetition of the preset index column is compared with a preset repetition threshold. If the repetition of the preset index column exceeds the preset repetition threshold, based on the repetition of each column of data and the task identifier to be analyzed, a column corresponding to the task identifier to be analyzed and whose repetition meets the preset conditions is selected as the recommended index column.
2. The method according to claim 1, characterized in that The step of respectively calculating the repetitiveness of each column of data in the data table to be analyzed comprises: Get the total number of data in the data table to be analyzed; Perform deduplication operations on each column of data respectively, and record the amount of deduplication data in each column of data respectively; The duplication of each column of data is determined based on the quotient of the amount of deduplicated data in each column of data and the total number of data entries.
3. The method according to claim 1, characterized in that The method further comprises: Detecting the number of columns of the preset index columns; The number of columns of the preset index column is compared with a preset column number threshold. If the number of columns of the preset index column exceeds the preset column number threshold, the repetition of each column of the preset index column is sorted from small to large, and the second target number of columns with the smallest repetition is selected as the combined index column.
4. The method according to claim 2, characterized in that: The method further comprises: Compare the total number of data items in the data table to be analyzed with a preset data volume threshold; If the total number of data items in the data table to be analyzed exceeds a preset data volume threshold, a warning control is displayed in the interface of the data table to be analyzed; the warning control is used to prompt the user that the total number of data items in the data table to be analyzed is excessive.
5. A data query device, characterized in that: The device comprises: A data acquisition module is used to obtain the data table to be analyzed; A repetition calculation module is used to calculate the repetition of each column of data in the data table to be analyzed; the repetition is used to indicate the proportion of repeated data in each column of data to the total number of data; An index column determination module is used to select a column corresponding to the task identifier to be analyzed and having a repetition degree that meets a preset condition as an index column based on the repetition degree of each column of data and a preset task identifier to be analyzed, so as to perform data query; The index column determination module is also used for: In response to the description sentence input by the user, the description sentence is split by keywords, and the split keywords are compared with the type information in the database to filter out the task identifier to be analyzed; or, in response to the keyword input by the user, the keyword is compared with the type information in the database to filter out the task identifier to be analyzed; The index column determination module is also used for: Sort the repetition of each column of data from small to large; Selecting a first target number of columns corresponding to the task identifier to be analyzed and having the smallest degree of repetition as index columns; The index column determination module is also used for: Check whether the data table to be analyzed contains a preset index column; If included, calculate the repetition degree of the preset index column; The repetition of the preset index column is compared with a preset repetition threshold. If the repetition of the preset index column exceeds the preset repetition threshold, based on the repetition of each column of data and the task identifier to be analyzed, a column corresponding to the task identifier to be analyzed and whose repetition meets the preset conditions is selected as the recommended index column.
6. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data query method according to any one of claims 1 to 4 by executing the computer instructions.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data query method according to any one of claims 1 to 4.
8. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the data query method according to any one of claims 1 to 4.
