Joint unique index traversal method and device, equipment and storage medium

Abstract

The invention discloses a united unique index traversal method and device, equipment and a storage medium, and the method comprises the steps: firstly obtaining all field distinction degrees of a database data table, then traversing corresponding fields of a preset data table data range according to the field distinction degrees, determining a field lower limit condition of next traversal, and obtaining a united unique index corresponding to the field lower limit condition; and finally, corresponding fields in the data range of the preset data table are traversed in sequence until a field lower limit condition of next traversal is not formed any more, and the problems that in the database comparison process at present, united unique index traversal is carried out, multiple fields exist, and each field has an ascending order, a descending order and other possible scenes are solved, so that when data are traversed in batches, the efficiency is high, and the efficiency is high are solved. The technical problem that united unique index traversal is inaccurate due to the fact that the upper limit and the lower limit of each batch of acquisition records may not complete a result set closed loop of a first field (or multiple fields) due to different UK field recognition degrees is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a combined unique index traversal method, device, equipment and storage medium. Background Art

[0002] Database table record comparison is an important task in database management, which involves checking whether the data between two database tables (which may be different tables in the same database, or tables in different databases) is consistent.

[0003] In database design, a joint unique index (UK for short) is a special type of index that allows indexes to be created on two or more columns to ensure that the combined values ​​of these columns are unique. This means that the database does not allow duplicate rows in the combination of indexed columns. It is a type of index used in the database to ensure that the combined values ​​of two or more columns are unique in the entire table. This type of index is very important for maintaining data integrity and improving query performance. Its advantages: Improve query efficiency: By creating indexes on multiple columns, query operations involving these columns can be accelerated, especially when these columns are often used together in query conditions.

[0004] There are several important reasons for traversing the joint unique index, especially during the database comparison process: Ensure the uniqueness of the data: The joint unique index is an index composed of two or more columns, which ensures that the combination of these columns is unique in the table. During the comparison process, traversing these indexes can ensure that the compared data rows are uniquely identifiable, thereby avoiding data duplication or inconsistency. Improve comparison efficiency: The use of joint unique indexes can quickly locate and retrieve data rows, which is especially important in large databases. Through index traversal, unnecessary data scanning can be reduced and the efficiency of comparison operations can be improved. Data consistency verification: When comparing two databases, you can verify data consistency by checking whether the data rows corresponding to the joint unique index exist in both databases and whether their values ​​are consistent.

[0005] However, during the current database comparison process, a joint unique index traversal is performed. There are multiple fields and each field has possible scenarios such as ascending and descending. As a result, when traversing the data in batches, due to the different recognition of the UK field, the upper and lower limits of the records obtained in each batch may not complete the result set closed loop of the first field (or multiple fields), making the joint unique index traversal inaccurate. Summary of the invention

[0006] The present application provides a combined unique index traversal method, apparatus, device and storage medium, which first obtains the discrimination of all fields in a database data table, then traverses the corresponding fields of a preset data table data range according to the field discrimination, determines the field lower limit condition for the next traversal, and finally traverses the corresponding fields of the preset data table data range in sequence until the field lower limit condition for the next traversal is no longer formed, thereby solving the technical problem that in the current database comparison process, a combined unique index traversal is performed, and there are multiple fields and each field has an ascending order, a descending order and other possible scenarios, so that when traversing data in batches, due to the different recognition of UK fields, the upper and lower limits of each batch of records may not complete the result set closed loop of the first field (or multiple fields), resulting in inaccurate combined unique index traversal.

[0007] In view of this, the first aspect of the present application provides a combined unique index traversal method, the method comprising:

[0008] Get the discrimination of all fields in the database table;

[0009] According to the field discrimination, the corresponding fields of the preset data table data range are traversed to determine the lower limit condition of the field for the next traversal;

[0010] The corresponding fields of the preset data table data range are traversed in sequence until the field lower limit condition for the next traversal is no longer formed.

[0011] Optionally, obtaining the discrimination of all fields in a database data table specifically includes:

[0012] Get the discrimination of all fields in the database table;

[0013] The discrimination degree of the first field is determined. If the discrimination degree is lower than a preset value, the first field is combined with the remaining fields except the first field to generate corresponding at least two discrimination subtasks.

[0014] Optionally, after determining the discrimination of the first field, if the discrimination is lower than a preset value, combining the first field with the remaining fields except the first field, and generating corresponding at least two discrimination subtasks, the following further includes:

[0015] For at least two of the discrimination subtasks, corresponding identifications are provided by comparing conditions;

[0016] The discrimination level of the discrimination subtask is determined by the recognitionLevel data item.

[0017] Optionally, traversing fields corresponding to a preset data table row number range according to the field discrimination, and determining a field lower limit condition for the next traversal specifically includes:

[0018] Traversing the discrimination subtasks corresponding to the data range of the preset data table;

[0019] Identifying the discrimination level of the discrimination subtask, and identifying the corresponding comparison condition as a result set closed loop;

[0020] Determine the lower limit condition of the field for the next traversal.

[0021] Optionally, traversing the preset data table data range in sequence until the field lower limit condition for the next traversal is no longer formed specifically includes:

[0022] Traversing the discrimination subtasks corresponding to the data range of the preset data table in sequence, and determining that the comparison condition corresponding to each traversal identifier is a result set closed loop;

[0023] Until all the comparison conditions are a closed loop of the result set and no longer form the field lower limit condition for the next traversal.

[0024] Optionally, before obtaining the discrimination of all fields in the database data table, the following steps are also included:

[0025] Calculate the discrimination of all fields by counting the number and proportion of different values ​​in the database data table;

[0026] De-duplicate and merge fields whose discrimination is lower than the threshold.

[0027] Optionally, traversing the corresponding fields of the preset data table data range in sequence until the field lower limit condition for the next traversal is no longer formed further includes:

[0028] Generate a field traversal result of traversing the database data table;

[0029] Generate and display a corresponding notification based on the field traversal result.

[0030] A second aspect of the present application provides a joint unique index traversal device, the device comprising:

[0031] An acquisition unit is used to obtain the discrimination of all fields in a database table;

[0032] A first traversal unit is used to traverse the corresponding fields of the preset data table data range according to the field discrimination degree to determine the lower limit condition of the field for the next traversal;

[0033] The second traversal unit is used to traverse the corresponding fields of the data range of the preset data table in sequence until the field lower limit condition for the next traversal is no longer formed.

[0034] Optionally, the acquisition unit specifically includes:

[0035] Get subunit, used to get the discrimination of all fields in the database table;

[0036] The discrimination subtask subunit is used to determine the discrimination of the first field. If the discrimination is lower than a preset value, the first field is combined with the remaining fields except the first field to generate corresponding at least two discrimination subtasks.

[0037] Optionally, the acquiring unit further includes:

[0038] A comparison condition subunit, used for correspondingly marking at least two of the discrimination subtasks through comparison conditions;

[0039] The discrimination level subunit is used to determine the discrimination level of the discrimination subtask through the recognitionLevel data item.

[0040] Optionally, the first traversal unit specifically includes:

[0041] A first traversal unit sub-unit is used to traverse the discrimination sub-task corresponding to the data range of the preset data table;

[0042] an identification discrimination level subunit, used to identify the discrimination level of the discrimination subtask and identify the corresponding comparison condition as a result set closed loop;

[0043] The next traversal subunit is used to determine the lower limit condition of the field for the next traversal.

[0044] Optionally, the second traversal unit specifically includes:

[0045] A second traversal sub-unit is used to traverse the discrimination sub-tasks corresponding to the data range of the preset data table in sequence, and determine that the comparison condition corresponding to each traversal identifier is a result set closed loop;

[0046] The result set closed loop sub-unit is used to complete the result set closed loop until all the comparison conditions are completed and no longer form the field lower limit condition for the next traversal.

[0047] Optionally, it also includes:

[0048] A statistical unit, used to calculate the discrimination of all fields by counting the number and proportion of different values ​​in the database data table;

[0049] The deduplication and merging processing unit is used for deduplication and merging of fields whose discrimination is lower than a threshold.

[0050] Optionally, it also includes:

[0051] A field traversal result unit, used to generate a field traversal result of traversing the database data table;

[0052] A notification unit is used to generate and display a corresponding notification according to the field traversal result.

[0053] A third aspect of the present application provides a joint unique index traversal device, the device including a processor and a memory:

[0054] The memory is used to store the program code and transmit the program code to the processor;

[0055] The processor is used to execute the steps of the method of the combined unique index traversal method of the first aspect according to the instructions in the program code.

[0056] A fourth aspect of the present application provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the method of the first aspect above.

[0057] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0058] In the present application, a combined unique index traversal method, apparatus, device and storage medium are provided, which first obtain the discrimination of all fields in a database data table, then traverse the corresponding fields of a preset data table data range according to the field discrimination, determine the field lower limit condition for the next traversal, and finally traverse the corresponding fields of the preset data table data range in sequence until the field lower limit condition for the next traversal is no longer formed. This solves the current technical problem of inaccurate combined unique index traversal in the current database comparison process, where there are multiple fields and each field has possible scenarios such as ascending and descending order, so that when traversing data in batches, due to different UK field recognition, the upper and lower limits of each batch of records may not complete the result set closed loop of the first field (or multiple fields), resulting in inaccurate combined unique index traversal.

[0059] At the same time, in this application, the discrimination of the first field is determined. If the discrimination is low, the first field is combined with the remaining fields except the first field to generate corresponding at least 2 discrimination subtasks, which greatly improves the traversal efficiency.

[0060] In addition, at least two of the discrimination subtasks are identified by comparing the corresponding conditions, and the discrimination level of the discrimination subtask is determined by the recognitionLevel data item, so as to calculate the query conditions for the next batch and complete the continuous traversal of batches, thus solving the problem that the joint unique index scenario cannot be traversed in batches. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A method flow chart of a combined unique index traversal method in an embodiment of the present application;

[0062] Figure 2 This is a structural diagram of a combined unique index traversal device in an embodiment of the present application;

[0063] Figure 3 This is a structural diagram of a joint unique index traversal device in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] This application designs a joint unique index traversal method, firstly, the extracted samples of the first time period and the samples of the second time period are subjected to sample filling operation through the filling variable algorithm to obtain new samples of the second time period, then the model score average error of the first time period samples and the second time period samples is obtained by the sample filling operation through the filling variable algorithm, and finally the model score average error is subjected to the corresponding model variable replacement operation to calculate the average error of the individual variables, which solves the problem that the current model is applied to samples of different time periods, and the expected score is stable. If a large offset occurs, it is necessary to analyze the cause of the offset, but the current mainstream machine learning algorithms themselves have a weak explanation of the cause of the offset, and each algorithm has different principles. For example, the tree model and logistic regression have different definitions of feature importance, and it is necessary to perform offset attribution processing for different models, which leads to technical problems in the low efficiency of mathematical model calculation.

[0066] For easier understanding, see Figure 1 , Figure 1 This is a method flow chart of the combined unique index traversal method in the embodiment of the present application, such as Figure 1 As shown, specifically:

[0067] S101, obtaining the discrimination of all fields in a database data table;

[0068] When it is necessary to calculate the last UK field value and sorting rules of the current batch result set according to a certain algorithm to generate the query conditions for the next batch, obtain the discrimination of all fields in the database data table.

[0069] It should be noted that obtaining the discrimination of all fields in a database table specifically includes:

[0070] Get the discrimination of all fields in the database table, and how many fields can be used to identify a unique record among all fields in the combined unique index;

[0071] Determine the discrimination degree of the first field. If the discrimination degree is lower than a preset value, combine the first field with the remaining fields except the first field to generate at least two corresponding discrimination subtasks.

[0072] For at least two of the discrimination subtasks, corresponding identifications are provided by comparing conditions;

[0073] The discrimination level of the discrimination subtask is determined by the recognitionLevel data item, where the discrimination level means <how many fields can identify a record within the current result set>. The default value is 1, that is, one field can identify a record.

[0074] As the discrimination level increases from high to low, the discrimination level increases. For example: a joint index of three fields, the data is: {1,2,3} / {1,2,4}, in the result set of the current two records, three fields are required to identify the records, so the discrimination level is 3.

[0075] It is understandable that the discrimination of the first field is judged. If the discrimination is low, the subtasks are divided according to the first field + the second field; if the discrimination is high, the subtasks are divided by the first field; if the first field crosses subtasks, the subtasks are divided by the first field + the second field, and the first field is used to judge the subtask division and subtask data acquisition by <equal>, and the data acquisition discrimination is judged by the same unique set of logic. Count-distinct the entire table once to filter out the extra ones. If the discrimination is low, you can consider distinguishing according to the first field, and you need to consider the judgment basis of <high ​​discrimination>, which can be measured. For example, if there are 10 million data and each is 100,000, then for 100 tasks, if the distinction of the first field of the 10 million data is about 100, the subtasks can be divided directly according to the first field (Note: there are 99 first fields that only account for 1 million and the 100th first field accounts for 9 million, which needs to be processed once through count-distinct). When there are many first fields with a large span, they cannot be spliced ​​in the way of notexes, but must be spliced ​​into clear conditions such as greater than, less than, equal to greater than or equal to, and less than or equal to through comparison.

[0076] In databases and data analysis, COUNT(DISTINCT...) is a commonly used SQL function that counts the number of different (unique) values ​​in a specified column. When you see an expression like COUNT(DISTINCT whole table), it probably means counting all the different values ​​of one or some columns in the entire table.

[0077] COUNT(): This function is usually used to count the number of rows in a table.

[0078] DISTINCT: This is a keyword that modifies the behavior of the COUNT() function so that it only counts distinct values ​​and ignores duplicate values.

[0079] COUNT(DISTINCT all tables) counts different combinations of all columns: In some database systems, it may be possible to count all different row combinations of all columns in a table. For example, if there is a table containing three columns A, B, and C, COUNT(DISTINCT A, B, C) will count the number of different rows in all (A, B, C) combinations in the table.

[0080] Before obtaining the distinction of all fields in the database data table, it also includes:

[0081] Calculate the discrimination of all fields by counting the number and proportion of different values ​​in the database data table;

[0082] De-duplicate and merge fields whose discrimination is lower than the threshold.

[0083] It is understood that in a database, the discrimination of a field (also called selectivity) refers to the distribution of different values ​​of a field in a table. A field with high discrimination means that the values ​​of the field are widely distributed in the table, and different records have a greater probability of having different field values. Such fields are usually more suitable as indexes because they can more effectively reduce the amount of data that needs to be scanned when querying. Determine the discrimination of a field: Statistical analysis: The discrimination of a field can be analyzed by counting the number and proportion of different values ​​in the table. For example, if a field has 90% unique values ​​in a table, then its discrimination is high. Use database tools: Most database management systems provide tools or commands to analyze the discrimination of a field. For example, in MySQL, you can use the SHOW INDEX command to view index information, or use the EXPLAIN command to analyze query plans. Observe data patterns: Observing data patterns and business logic can also help determine the discrimination of a field. For example, a field containing user IDs usually has high discrimination because each user has a unique ID. Features of highly discriminative fields: Unique or almost unique: such as user IDs, order numbers, etc. Widely distributed values: For example, if there are data records every day in the date field, the date field has a high degree of discrimination. Infrequent repetition: Field values ​​are rarely repeated, such as randomly generated serial numbers. Characteristics of fields with low discrimination: A large number of repeated values: For example, gender, country code, etc., the values ​​of these fields may have a high repetition rate in the table. Fixed values ​​or limited range: For example, if there are only a few status values ​​(such as "new", "in progress", "completed"), then the discrimination of this field is low. Frequent updates: If a field is frequently updated, it may affect the efficiency of the index because each update may require an update to the index. Ways to improve the discrimination of fields: Use composite indexes: When the discrimination of a single field is not enough, you can consider combining multiple fields into a composite index. Data normalization: Reduce data redundancy and improve the discrimination of fields through data normalization. Data preprocessing: Preprocess the data before it enters the database, such as merging similar values ​​or deleting duplicate data.

[0084] S102, traversing the corresponding fields of the preset data table data range according to the field discrimination, and determining the lower limit condition of the field for the next traversal;

[0085] It should be noted that the discrimination subtask corresponding to the preset data table data range is traversed; the discrimination level of the discrimination subtask is identified, and the corresponding comparison condition is marked as a result set closed loop; and the field lower limit condition for the next traversal is determined.

[0086] As shown in Table 1, for the first time, the first 4 records of the sequence number are taken out, and the discrimination recognition is: f1=1&f2=9&f3<4. Row 4 is not processed, forming the lower limit condition for the next fetch operation: f1=1&f2=9&f3>=4;

[0087] Second time. Take out the records with sequence numbers 4 to 7, and identify the discrimination: f1=1&f2>7. Do not process row 7, forming the lower limit condition for the next fetch operation: f1=1&f2<=7;

[0088] The third time. Take out row 7, discrimination recognition: f1>1. Process row 7. Because the number of result sets is less than the specified number of 4 records, it means that all records that cause discrimination problems have been processed. Therefore, there is no need to leave redundant records for the next fetch operation.

[0089] Serial number First field asc Two fields desc Three fields asc Four fields desc 1 1 9 2 5 2 1 9 2 1 3 1 9 3 7 4 1 9 4 3 5 1 9 4 1 6 1 8 9 9 7 1 7 8 9 8 2 7 1 1

[0090] Table 1

[0091] Complete all records corresponding to a certain discrimination level. Refer to Table 1, and the following is an example:

[0092] 1. Sequence number 2, the result set closed loop is completed with recognition level 4, first field = 1, second field = 9, third field = 2;

[0093] 2. Sequence number 5, the result set closed loop is completed with recognition level 3, first field = 1, second field = 9;

[0094] 3. Sequence number 7 completes the result set closure with recognition level 2 and first field = 1.

[0095] Subtasks are divided by the first field + the second field, and the first field is judged by <equal to>. Note: This should support the judgment of multiple fields such as 3, 4, and 5, and the method is universal. The front is the discrimination judgment method to confirm which field can be distinguished. The fields before the discrimination are all judged by equal to, based on when it was changed to greater than or equal to the previous time, and the completion of the discrimination.

[0096] S103: traverse the corresponding fields of the preset data table data range in sequence until the field lower limit condition for the next traversal is no longer formed.

[0097] Traversing the discrimination subtasks corresponding to the data range of the preset data table in sequence, and determining that the comparison condition corresponding to each traversal identifier is a result set closed loop;

[0098] Until all the comparison conditions are a closed loop of the result set and no longer form the field lower limit condition for the next traversal.

[0099] In database comparison or synchronization tasks, lowLimitValues ​​(lower limit values) are often used to define the starting point of data acquisition or comparison operations. These values ​​can ensure that when processing large amounts of data in batches or pages, each operation starts from a known, consistent state. Construct lowLimitValues ​​as follows:

[0100] Determine the unique index: First, you need to determine the combination of fields in the table that uniquely identifies each row of data. This is usually the table's primary key or a joint unique index.

[0101] Get the current minimum value: For each unique index field, you need to get the minimum value in the current data set. These values ​​will be used as the starting point for the next data acquisition operation. Build lowLimitValues: Combine these minimum values ​​into a structure (such as a dictionary or mapping) where the key is the index field and the value is the corresponding minimum value.

[0102] Apply to queries: In subsequent data retrieval or comparison queries, use these lowLimitValues ​​as the starting point of the query to ensure that no data is missed.

[0103] Ensuring data consistency and accuracy when constructing lowLimitValues ​​is an important part of database synchronization, migration, or comparison tasks. Use transactions: Ensure that data is not modified by other operations when obtaining lowLimitValues. You can use database transactions to ensure the atomicity of operations. Lock data: When obtaining lowLimitValues, you can use the database's locking mechanism (such as SELECT...FOR UPDATE) to prevent other transactions from modifying data. Consistent view: In some database systems, snapshots or consistent views can be used to obtain the consistent state of data. Idempotence: Ensure that the operation of obtaining lowLimitValues ​​is idempotent, that is, the results of multiple executions are the same and no new problems are introduced. Validate data: After obtaining lowLimitValues, these values ​​should be validated to ensure that they are reasonable and meet the expected range or conditions. Sequential acquisition: In a distributed system, ensure that lowLimitValues ​​are obtained sequentially on different nodes or instances to avoid data inconsistency. Use timestamps: If data version control relies on timestamps, ensure the accuracy of timestamps and consider timestamps when obtaining lowLimitValues. Avoid concurrent modifications: During the construction of lowLimitValues, avoid other processes or threads from modifying related data. Record logs: Record detailed operation logs, including the time when lowLimitValues ​​was obtained, the operator, the operation details, etc., so that problems can be tracked and located. Data backup: Before performing operations that may affect data consistency, create a data backup so that it can be restored when problems occur. Use consistency tools provided by the database: Some databases provide tools for data consistency checking and synchronization, such as MySQL's binlog, Oracle's Data Guard, etc. Testing and verification: Before deployment in the production environment, the process of building lowLimitValues ​​is fully tested and verified through a test environment. Monitoring and alarm: Implement a monitoring mechanism to detect and alarm in a timely manner when data consistency problems occur.

[0104] The corresponding fields of the preset data table data range are traversed sequentially until the field lower limit condition for the next traversal is no longer formed, and further includes:

[0105] Generate a field traversal result of traversing the database data table;

[0106] Generate and display a corresponding notification based on the field traversal result.

[0107] A result set is a collection of data returned by a database query (such as an SQL query). In programming, a result set can be viewed as a temporary table that contains all the rows and columns returned by the query.

[0108] Closed Loop: Refers to a feedback mechanism in a system or process, so that the output of the system can affect its future input, thereby achieving self-regulation and control. In software engineering, a closed loop may refer to a process that can be self-sustaining and self-improving.

[0109] It can be understood that: a) When calculating the lower limit of the source table, the lower limit and upper limit of the target table need to be calculated. It is necessary to include three conditions: greater than or equal to and greater than and equal to. Equal to is required for closed loop in the discrimination processing process;

[0110] b) Only when the processing discrimination is low, the query condition is equal to and has multiple fields. In other scenarios, it is a single field, otherwise the condition is problematic;

[0111] c) In the recognition scenario, every time the number of result sets is less than the standard number, it is an opportunity to close the loop;

[0112] d) The recognitionLevel data item is only involved in the calculation of the upper and lower limits and the deduplication of the comparison result set, and does not participate in the comparison. Note: This data item, called the field discrimination level, corresponds to the discrimination ability of a field in the joint unique index for the data range. For example: For a joint unique index composed of two fields, f1 and f2, when a certain record can be uniquely identified by the f1 field, the field recognition level of the current record corresponding to the f1 field is 1; when a certain day record needs to be uniquely identified by two fields, f1 and f2, the discrimination level of the two fields f1 and f2 of the record is 2;

[0113] e) Principle: The new first field must start traversal with greater than or greater than or equal to.

[0114] This embodiment is mainly applicable to the comparison of the source table and the target table of the database. The comparison of database table records is an important task in database management, which involves checking whether the data between two database tables (which may be different tables in the same database, or tables in different databases) are consistent. Determine the comparison target and scope, data column selection: determine which columns need to be compared. This usually includes primary keys, foreign keys, business-critical data, etc. Data row selection: determine whether to compare the entire table or the rows filtered out based on certain conditions. Select a suitable comparison strategy: Full comparison: compare all data in the two tables. Incremental comparison: only compare the data that changes in the two tables after a specific time point. Key field comparison: only compare key fields, such as primary keys, important business data, etc. Process the comparison results: Difference records: record all inconsistent data points, including missing data, inconsistent data, etc. Report generation: Generate a detailed comparison report, including a detailed description and statistical information of the difference data. Data synchronization and repair: Data synchronization: synchronize the data of the two tables according to the comparison results to ensure data consistency. Data repair: repair the problems found, which may include update, delete or insert operations. Performance and optimization: Index optimization: Ensure that the fields involved in the comparison have appropriate indexes to speed up the query. Batch processing: For large tables, batch processing is used to avoid locking the table for a long time or consuming too many resources. Regular comparison: Automation: Set up regular comparison tasks to automate the comparison process. Monitoring: Monitor the comparison process and results to find and solve problems in a timely manner.

[0115] A combined unique index traversal method provided in the present embodiment first obtains the discrimination of all fields in a database data table, then traverses the corresponding fields of a preset data table data range according to the field discrimination, determines the field lower limit condition for the next traversal, and finally traverses the corresponding fields of the preset data table data range in sequence until the field lower limit condition for the next traversal is no longer formed. This solves the current technical problem of inaccurate combined unique index traversal in the current database comparison process, where there are multiple fields and each field has ascending or descending order, so that when traversing data in batches, due to the different recognition of the UK field, the upper and lower limits of the records obtained in each batch may not complete the result set closed loop of the first field (or multiple fields), resulting in inaccurate combined unique index traversal.

[0116] At the same time, in this embodiment, the discrimination degree of the first field is determined. If the discrimination degree is low, the first field is combined with the remaining fields except the first field to generate corresponding at least 2 discrimination subtasks, which greatly improves the traversal efficiency.

[0117] In addition, at least two of the discrimination subtasks are identified by comparing the corresponding conditions, and the discrimination level of the discrimination subtask is determined by the recognitionLevel data item, so as to calculate the query conditions for the next batch and complete the continuous traversal of batches, thus solving the problem that the joint unique index scenario cannot be traversed in batches.

[0118] See also Figure 2 , Figure 2 Schematic diagram of the structure of the combined unique index traversal device in the embodiment of the present application. Figure 2 As shown, specifically:

[0119] An acquisition unit 201 is used to acquire the discrimination of all fields in a database data table;

[0120] The first traversal unit 202 is used to traverse the corresponding fields of the preset data table data range according to the field discrimination degree to determine the field lower limit condition for the next traversal;

[0121] The second traversal unit 203 is used to traverse the corresponding fields of the preset data table data range in sequence until the field lower limit condition for the next traversal is no longer formed.

[0122] In the present embodiment, a combined unique index traversal device is provided, which first obtains the discrimination of all fields in the database data table, then traverses the corresponding fields of the preset data table data range according to the field discrimination, determines the field lower limit condition for the next traversal, and finally traverses the corresponding fields of the preset data table data range in sequence until the field lower limit condition for the next traversal is no longer formed. This solves the current technical problem of inaccurate combined unique index traversal in the database comparison process, where there are multiple fields and each field has ascending or descending order, so that when traversing data in batches, due to the different recognition of the UK field, the upper and lower limits of the records obtained in each batch may not complete the result set closed loop of the first field (or multiple fields), resulting in inaccurate combined unique index traversal.

[0123] At the same time, in this embodiment, the discrimination degree of the first field is determined. If the discrimination degree is low, the first field is combined with the remaining fields except the first field to generate corresponding at least 2 discrimination subtasks, which greatly improves the traversal efficiency.

[0124] In addition, at least two of the discrimination subtasks are identified by comparing the corresponding conditions, and the discrimination level of the discrimination subtask is determined by the recognitionLevel data item, so as to calculate the query conditions for the next batch and complete the continuous traversal of batches, thus solving the problem that the joint unique index scenario cannot be traversed in batches.

[0125] The embodiment of the present application also provides another joint unique index traversal device, such as Figure 3 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:

[0126] Figure 3 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 3 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0127] Combine the following Figure 3 A detailed introduction to the various components of the mobile phone:

[0128] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0129] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0130] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0131] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0132] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0133] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0134] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0135] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0136] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0137] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0138] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:

[0139] Get the data analysis request sent by the front end;

[0140] Parsing the data analysis request and verifying it with metadata information;

[0141] The data analysis request that passes the verification is sent to the parallel processing database for processing.

[0142] An embodiment of the present application also provides a computer-readable storage medium for storing program code, where the program code is used to execute any one of the implementations of the combined unique index traversal method described in the aforementioned embodiments.

[0143] In an embodiment of the present application, a combined unique index traversal method, apparatus, device and storage medium are provided, which first obtains the discrimination of all fields in a database data table, then traverses the corresponding fields of a preset data table data range according to the field discrimination, determines the field lower limit condition for the next traversal, and finally traverses the corresponding fields of the preset data table data range in sequence until the field lower limit condition for the next traversal is no longer formed. This solves the current technical problem of inaccurate combined unique index traversal in the current database comparison process, where there are multiple fields and each field has an ascending or descending order, so that when traversing data in batches, due to the different recognition of the UK field, the upper and lower limits of the records obtained in each batch may not complete the result set closed loop of the first field (or multiple fields), resulting in inaccurate combined unique index traversal.

[0144] At the same time, in this embodiment, the discrimination degree of the first field is determined. If the discrimination degree is low, the first field is combined with the remaining fields except the first field to generate corresponding at least 2 discrimination subtasks, which greatly improves the traversal efficiency.

[0145] In addition, at least two of the discrimination subtasks are identified by comparing the corresponding conditions, and the discrimination level of the discrimination subtask is determined by the recognitionLevel data item, so as to calculate the query conditions for the next batch and complete the continuous traversal of batches, thus solving the problem that the joint unique index scenario cannot be traversed in batches.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0147] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0148] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0149] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0153] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A joint unique index traversal method, characterized in that: include: Get the discrimination of all fields in the database table; According to the field discrimination, the corresponding fields of the preset data table data range are traversed to determine the lower limit condition of the field for the next traversal; The corresponding fields of the preset data table data range are traversed in sequence until the field lower limit condition for the next traversal is no longer formed.

2. The combined unique index traversal method according to claim 1, characterized in that: Obtaining the distinction of all fields in the database data table specifically includes: Get the discrimination of all fields in the database table; The discrimination degree of the first field is determined. If the discrimination degree is lower than a preset value, the first field is combined with the remaining fields except the first field to generate corresponding at least two discrimination subtasks.

3. The combined unique index traversal method according to claim 2, characterized in that: Determine the discrimination of the first field. If the discrimination is lower than a preset value, combine the first field with the remaining fields except the first field to generate at least two corresponding discrimination subtasks, and further include: For at least two of the discrimination subtasks, corresponding identifications are provided by comparing conditions; The discrimination level of the discrimination subtask is determined by the recognitionLevel data item.

4. The combined unique index traversal method according to claim 3, characterized in that: Traversing the fields corresponding to the preset data table row number range according to the field discrimination, and determining the field lower limit condition for the next traversal specifically includes: Traversing the discrimination subtasks corresponding to the data range of the preset data table; Identifying the discrimination level of the discrimination subtask, and identifying the corresponding comparison condition as a result set closed loop; Determine the lower limit condition of the field for the next traversal.

5. The combined unique index traversal method according to claim 4, characterized in that: Sequentially traversing the preset data table data range until the field lower limit condition for the next traversal is no longer formed specifically includes: Traversing the discrimination subtasks corresponding to the data range of the preset data table in sequence, and determining that the comparison condition corresponding to each traversal identifier is a result set closed loop; Until all the comparison conditions are a closed loop of the result set and no longer form the field lower limit condition for the next traversal.

6. The combined unique index traversal method according to claim 1, characterized in that: Before obtaining the distinction of all fields in the database data table, it also includes: Calculate the discrimination of all fields by counting the number and proportion of different values ​​in the database data table; De-duplicate and merge fields whose discrimination is lower than the threshold.

7. The combined unique index traversal method according to claim 1, characterized in that: The corresponding fields of the preset data table data range are traversed sequentially until the field lower limit condition for the next traversal is no longer formed, and further includes: Generate a field traversal result of traversing the database data table; Generate and display a corresponding notification based on the field traversal result.

8. A combined unique index traversal device, characterized in that: include: An acquisition unit is used to obtain the discrimination of all fields in a database table; A first traversal unit is used to traverse the corresponding fields of the preset data table data range according to the field discrimination degree to determine the lower limit condition of the field for the next traversal; The second traversal unit is used to traverse the corresponding fields of the data range of the preset data table in sequence until the field lower limit condition for the next traversal is no longer formed.

9. A joint unique index traversal device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the joint unique index traversal method described in any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the joint unique index traversal method described in any one of claims 1-7.

Images