A method and apparatus for full-amount data comparison of personnel information

Abstract

The application provides a method and device for full-amount data comparison of personnel information, and belongs to the technical field of data processing. The full-amount data is compared through the following steps: 1) traversing data tables to obtain table names, table record numbers and data field names of each data table, and saving in a probe result full-amount partition table; 2) constructing a personnel characteristic data check function according to a generation rule of personnel characteristic data; 3) using the check function to check the value of each field in the probe result full-amount partition table, and obtaining fields with personnel characteristic data; 4) using the check function to traverse the fields with personnel characteristic data, and obtaining a personnel characteristic data set; and 5) searching the personnel characteristic data set with personnel characteristic data to be searched, and obtaining a data set to be associated with the personnel to be searched in the data warehouse. The method can identify personnel characteristic data in all fields, and realizes quick and accurate searching of associated data of the personnel to be searched.

Description

Technical Field

[0001] This invention relates to a method and apparatus for comparing personnel information across a full dataset, belonging to the field of data processing technology. Background Technology

[0002] Full data refers to all data in a data warehouse or big data platform, including both existing and incremental data. Full data is typically scattered across different channels or systems, covering a wide range of business areas, including heterogeneous business systems or documents related to communication itineraries, border control, air travel, rail travel, and city travel. Each business is highly correlated with individual identity attributes, which are both universal and unique, playing a crucial role in identifying individuals, such as an ID card number or driver's license number. By aggregating this heterogeneous full data from multiple sources into a unified data warehouse and establishing data associations based on individual identity attributes, it becomes possible to retrieve and compare various business data related to individuals.

[0003] For example, as the pandemic enters a new phase of normalized prevention and control, utilizing big data technology to uncover potential patterns and experiences hidden within the full dataset can help improve the accuracy of epidemic prevention and control, and contain the spread of the epidemic to the smallest possible extent. Therefore, it is necessary to establish data tables corresponding to personnel data in the epidemic data warehouse to fully leverage the role of the full personnel data within the warehouse. The full dataset will gradually grow over time; therefore, after multiple rounds of epidemics, the data warehouse is prone to issues such as large data table accumulation, outdated data, data redundancy, or duplicate data, which can hinder the data's effectiveness. Simultaneously, offline data sources are mostly spreadsheet files, whose format and content lack unified verification standards, easily leading to mismatches in some data content and column names (i.e., misplaced field names). Because this erroneous content is inconsistent with the data dictionary description, it is difficult to discover and utilize the data after it is entered into the database. Furthermore, the data tables provided by the online system may have missing dictionary comments, inconsistencies between field descriptions and actual content, causing the location and content of this data to be ignored during retrieval and comparison. Traditionally, data is described using a data dictionary and manually selected to query data from some tables in the entire database. This approach only utilizes a portion of the tables and relies solely on post-event experience. If, beforehand, one manually queries every field of every data table to retrieve all records, the efficiency is low, the workload is large, and the data is constantly changing. The resulting location of all personnel characteristic data in the data warehouse is inaccurate, making it difficult to pinpoint the location of at-risk individuals in the entire database. This affects the completeness, accuracy, and timeliness of searching and comparing individuals at risk of infection. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for comparing personnel information across a full dataset, thereby solving the problem of inaccurate results in searching and filtering full datasets in the prior art.

[0005] To achieve the above objectives, the present invention provides a method for comparing personnel information across a full dataset, comprising the following steps:

[0006] 1) Traverse the metadata of the data tables in the data warehouse to obtain the table name, number of records in each table, and field names of each table, and save the results as a full partition table of the exploration results; 2) Construct a personnel feature data verification function based on the generation rules of personnel feature data, whereby the personnel feature data is used to indicate the uniqueness of personnel identity; 3) Use the personnel feature data verification function to verify the data records of each field in each table stored in the full partition table of the exploration results, and obtain the fields that conform to the personnel feature data verification function, i.e., the records containing personnel feature data in the fields, thereby locating the fields containing the personnel feature data; 4) Traverse all records in the located fields, and use the personnel feature data verification function to verify the data in the traversed records, extracting the personnel feature data to obtain the full personnel feature data set of the data warehouse; 5) Search and compare the personnel feature data to be searched in the full personnel feature data set of the data warehouse, find the position of the data in the data table that matches the personnel feature data to be searched, and use the record corresponding to each position as the dataset that needs to be associated with the personnel to be searched in the data warehouse.

[0007] This invention can identify personnel feature data in all fields of the full partition table of the exploration results generated from the full personnel data of the data warehouse. In this way, even if the personnel feature data is in an incorrect field or the description and data content are inconsistent during the information filling or collection and summarization process, as long as the personnel feature data is correct and compliant, it can pass the personnel feature data verification function and be included in the full personnel feature data set. This ensures that the full data is complete when searching and filtering, thereby improving the accuracy of the search and filtering results.

[0008] Furthermore, in the above method for comparing personnel information across full data, the method also includes determining the number of days each table has not been updated based on the number of data records in each table in the full partition table of the daily exploration results. If the number of days not updated exceeds the set number of days, then step 5) will no longer perform a full data comparison on that table.

[0009] If the number of days a data table has not been updated exceeds a set number of days, it indicates that the data table was updated a long time ago. Therefore, the timeliness of the data table is greatly reduced, and it can be considered unsuitable as a source for data analysis. Thus, in step 5), a full data analysis and comparison will not be performed on this data table. This eliminates data tables with poor timeliness, thereby improving the efficiency of the full data analysis and comparison.

[0010] Furthermore, in the above method for comparing personnel information across full data sets, the method further includes determining the similarity between the first type of data tables and the second type of data tables in the data warehouse, filtering out suspected discontinued tables based on the similarity judgment, filtering out discontinued tables from the suspected discontinued tables, adding the filtered discontinued tables to the data model discontinued set, and in step 1), no longer traversing the data tables in the discontinued set; wherein the first type of data table refers to a data table with 0 days without update, and the second type of data table refers to a data table with a non-zero number of days without update; the process of determining the similarity between data tables is as follows: the field names under the same table name and the same table comment in the full partition table of the exploration results are sequentially concatenated to form a field name concatenation string; the similarity between the field name concatenation strings of the two tables is calculated based on the string similarity calculation formula.

[0011] Data warehouses may contain duplicate records for the same data table. Therefore, each data table is judged based on similarity. If a data table is found to be highly similar to a previous data table, it is marked as a suspected discontinued data table. The user then manually determines whether the suspected discontinued data table should be added to the data model discontinued set and is no longer included in the traversal process in step 1). The data model similarity algorithm assists in manually removing duplicates or redundant models, eliminating redundancy in the entire database of personnel feature values ​​across models, and improving the efficiency of retrieving and comparing the location of personnel feature data.

[0012] Furthermore, in the above method for comparing personnel information across full data, step 3) involves concatenating the table name and field name from the read data records using SQL syntax, and inputting the concatenated result into the personnel feature data verification function to obtain an SQL statement that determines whether the field values ​​in the data table pass the verification, thereby determining the fields in the data table where the personnel feature data is located.

[0013] Furthermore, in the above method for comparing personnel information across a full dataset, step 3) involves filtering based on the data type of the field names in the data table before using the personnel feature data verification function, excluding fields whose data type does not conform to the feature value data type.

[0014] Before starting the verification in step 3), the data type of the field containing the personnel feature data is filtered out to remove fields that do not match the data type of the personnel feature data, thereby reducing the amount of data to be traversed in the subsequent steps and improving the efficiency of data processing.

[0015] Furthermore, in the above method for comparing personnel information across all data, step 4) involves using SQL syntax to concatenate the table name, field name, and partition type of the location field, and inputting the concatenated result into the personnel feature data verification function to obtain an SQL statement that verifies the data content based on the table's partition type, thereby obtaining the personnel feature data.

[0016] Furthermore, in the above method for comparing personnel information across full data, steps 3) and 4) both use UNION ALL to merge multiple SQL statements into a single execution statement.

[0017] By using UNION ALL, multiple SQL statements can be merged into a single execution statement, allowing the search process of multiple SQL statements to be processed in parallel, which greatly improves the efficiency of data processing.

[0018] Furthermore, in the above method for comparing personnel information across a full dataset, the personnel feature data is the ID card number, and the corresponding personnel feature data verification function is related to the verification code generation rule of the last digit of the ID card number.

[0019] Furthermore, in the above method for comparing personnel information across full data, the number of records in the data table is related to the partition type of the data table. If it is a full partition type, the number of records is counted based on the latest full partition; if it is an incremental partition type, the number of records is counted based on all partitions.

[0020] When the data table's partition type is full partition, new data will be stored in the existing partitions, so the latest full partition statistics of the data table are taken as the latest record count; when the data table's partition type is incremental partition, new data will be stored in the newly created partitions, so the total partition statistics of the data table are taken as the latest record count; when the data table's partition type is non-partitioned, the total content statistics of the data table are taken as the latest record count.

[0021] In addition, the present invention also provides an apparatus for comparing personnel information across a full dataset, which employs the aforementioned method for comparing personnel information across a full dataset. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the process of the present invention;

[0023] Figure 2 This is a schematic diagram illustrating the process of exploring metadata in a data warehouse according to the present invention;

[0024] Figure 3 This is a schematic diagram illustrating the process of extracting the location of all personnel feature data from the data warehouse according to the present invention;

[0025] Figure 4 This is a flowchart illustrating the process of extracting record values ​​from the full set of personnel feature data in the data warehouse according to the present invention.

[0026] Figure 5 A flowchart illustrating the process of calculating the number of days a data warehouse table record has not been updated;

[0027] Figure 6 This is a schematic diagram of a device used for comparing personnel information across a full dataset in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0029] Method Implementation Examples:

[0030] like Figure 1 As shown, the method of the present invention for comparing personnel information across a full dataset includes the following steps:

[0031] Step S1: As Figure 2 As shown, the metadata of the data tables in the data warehouse is traversed, and the traversal results are saved as a full partition table of exploration results.

[0032] First, the metadata of the data tables in the data warehouse is probed daily, and the metadata information of all data tables is read and retrieved to obtain the table name, table comments, and partition type information of each data table. The latest record count of the data table is calculated based on different partition types. When the partition type of the data table is full partition, new data will be stored in the existing partitions, so the latest full partition statistics are taken as the latest record count. When the partition type of the data table is incremental partition, new data will be stored in the newly created partitions, so the latest record count is taken as the total partition statistics. When the partition type of the data table is a non-partitioned table, the total content statistics of the data table are taken as the latest record count. Second, the data field names, field comments, and field data types of the data table are obtained from the metadata and merged with the table name, table comments, partition type, and the latest record count of the table into a single record stored in the probe results full partition table. Third, the traversal of all data tables is completed, and the resulting probe results full partition table is shown in Table 1. The partition value is the date at the time of the traversal.

[0033] As another implementation, to avoid traversing disabled tables, during the traversal process, it is determined whether the currently read data table is in the data model disabled set. If it is in the data model disabled set, the table is not traversed again. The generation of the data model disabled set is described in detail in step S6 and will not be described here.

[0034] Table 1: Full Partition Table of Exploration Results

[0035]

[0036]

[0037] Step S2: Create a personnel feature data verification function according to the personnel feature data generation rules. In this embodiment, the citizen identification number is used as the personnel feature data. In other embodiments, passport numbers, mobile phone numbers, email addresses, etc., can also be used as personnel feature data. Personnel feature data has the characteristics of being machine-readable and unique. According to the provisions of the People's Republic of China National Standard GB 11643-1999 regarding citizen identification numbers, the citizen identification number is a feature combination code, consisting of a seventeen-digit body code and a one-digit check code, where the last digit is the check code. The calculation method for the check code is as follows:

[0038] ∑(A i *W i (mod11)

[0039] Where i represents the position number of the number character from right to left, including the check digit; Ai represents the value of the number character at position i; and Wi represents the weighting factor at position i. The formula for calculating the weighting factor Wi is:

[0040] W i =2^(i-1)(mod11)

[0041] Based on the above formula, the weighting factors from the second rightmost weight W2 to the eighteenth rightmost weighting factor W are calculated. 18 The sequence is 7, 9, 10, 5, 8, 4, 2, 1, 6, 3, 7, 9, 10, 5, 8, 4, 2. Substituting the first seventeen digits of the body code and the corresponding weighting factor values ​​into the formula for calculating the check code yields the check code value. The created personnel feature data verification function then compares the last digit of the eighteen digits to be verified with the check code value obtained using the above method. If they match, the verification passes, indicating that the eighteen digits conform to the identity number generation rules and constitute an identity number; otherwise, it indicates that it is not a valid identity card number. When the data to be verified passes the personnel feature verification function, the verification data content is returned; otherwise, a failure message "fail" is returned, as shown in Table 2.

[0042] Since the data itself may contain spaces, line breaks, or noisy characters, in order to avoid these interferences from damaging the feature data, interference information filtering is performed on the data to be verified before it passes through the personnel feature data verification function. Non-capitalic x characters or non-numeric characters are replaced with empty spaces to improve the fault tolerance of interference information.

[0043] Table 2: Verification Results of Feature Data of ID Numbers to be Verified

[0044] Personnel ID Number Feature Data Value Feature data verification function returns results 410328198802288548 fail 41832719930214002X fail 61196270038 fail 410329198109072542 fail 411524160120160088 fail 410224201801060033 fail 412927197311102114 fail 421724195604073711 fail 110102********2390 110102********2390 654221********403x 654221********403X

[0045] To protect personal information privacy, this embodiment de-identifies legitimate ID numbers, displaying only the first six and last four digits. This does not affect the explanation of the technical solution of the present invention. Illegal ID numbers do not involve personal privacy and therefore are not de-identified.

[0046] Step S3: As Figure 3 As shown, the purpose of this step is to obtain the location information of personnel feature data, that is, to obtain the data table and field information where the personnel feature data is located. Therefore, it is only necessary to determine whether the field value contains personnel feature data. If it contains personnel feature data, the table name and field name are recorded to determine the location information of the personnel feature data.

[0047] First, iterate through the records of the latest partition in the full partition table of the exploration results, reading each record sequentially. Based on the characteristics of the data type of the feature value, filter out records whose data type does not conform to the feature value's data type. For example, in this embodiment, ID card numbers are used as personnel feature data, and the data type of ID card numbers cannot be date, integer, or floating-point. Therefore, records with field data types of date, integer, and floating-point are filtered out. Second, according to SQL syntax, concatenate the data table name, table comment, field name, and field comment in the read records. Input the concatenated result into the personnel feature data verification function constructed in step S2 to generate an SQL statement to determine whether the field values ​​of the data table pass the verification. Third, based on the data calculation engine's support for the upper limit of the number of datasets that can be merged using the SQL keyword UNION ALL, merge the SQL statements corresponding to N field names into one execution statement using UNION ALL, where N is 100. Fourth, the calculation engine executes the statement, generating information representing the field value verification result. If the verification passes, the result information is [table name.field name]; if it fails, the result information is [NULL]. Fifth, write the verification result information into the feature data location information table to complete the traversal.

[0048] As shown in Table 3 below, the location of the feature data containing the information [ods_data_fcrydjxx_di.name] indicates that the personnel feature data, 412825********5326, appears in the name field of the registration information for returning personnel. It can be seen that this method can identify personnel feature data in all fields. Therefore, even if errors occur during the information filling or collection process, such as entering an ID number into the name field, it will not be considered erroneous data and excluded.

[0049] Table 3 Feature Data Location Information Table

[0050]

[0051] Step S4: As Figure 4 As shown, the fields containing personnel feature data that were identified in step S3 are traversed to find and extract personnel feature data, and the record values ​​of all personnel feature data in the data warehouse are extracted.

[0052] First, iterate through the records in the feature data location information table, sequentially reading each record, parsing the location information of the feature data in each record (i.e., table name and field name), and associating it with the corresponding record in the latest partition of the full partition table of the exploration results to obtain the table name, table comment, field name, field comment, number of records, and partition type information. Second, according to SQL syntax, concatenate the table name, table comment, field name, field comment, and partition type, and input the concatenated result into the personnel feature data verification function constructed in step S2 to generate an SQL statement that combines the partition type of the data table to verify the data content of the data table. For example, for a data table with feature data location [ods_jkdj_ryxx_di.sfzhm], and the partition type of this data table is incremental partition, the corresponding SQL statement is as follows:

[0053]

[0054]

[0055] For a data table with feature data located at [ods_data_fxxxb_df.zjhm], and whose partition type is full partition, the corresponding SQL statement is as follows:

[0056]

[0057] Third, based on the data calculation engine's support for the maximum number of datasets that can be merged using the SQL keyword UNION ALL, the SQL statements corresponding to N fields are merged into a single execution statement using UNION ALL, where N is 100. Fourth, the calculation engine executes the statement, generating record values ​​that pass validation at the feature data locations. These validated record values ​​are then merged with the table name, table comments, field names, and field comments, saved as a single record, and written to the personnel feature data set table. An example of the personnel feature data set table is shown in Table 4. In the second and third example records, double and single quotes (noise characters) are added to the ID card number data. Due to the validation's fault tolerance capability, valid ID card number information containing noise characters can still pass validation.

[0058] Table 4: Collection of Personnel Characteristics Data

[0059]

[0060]

[0061] Step S5: As Figure 5 As shown, the purpose of this step is to calculate the number of days the data table has not been updated. If the number of days the data table has not been updated exceeds the set number of days, then the full information comparison of the table will no longer be performed.

[0062] First, read the full partition table of the exploration results. During the initial calculation, the exploration results table is sorted in reverse order by date partition value. Read all partitions of the exploration results table. The partition is represented by pt. A partition pair is formed by the date of each partition and the date preceding the date of each partition. Assume that the maximum partition date is max(pt) and the minimum partition date is min(pt). The date partition pairs are arranged to form a set S{(i,i-1)}, where the value of i decreases sequentially from max(pt) to min(dt). In subsequent calculations, the probe results table is sorted in descending order by date partition value. The largest partition value (i.e., the latest partition) is read daily. A partition pair is formed using the date of the newly added partition and the date of the previous day's partition. Second, the date partition pair is read, and the probe results set of the larger partition date is left-joined (LEFT JOIN) with the probe results set of the smaller partition date by table name. The difference in the number of records in the two partitions for each data table is calculated. If the number of records in the smaller partition date is NULL, it means the larger partition date is the table's creation date. NULL values ​​are converted to 0, and the difference is calculated. If the difference is 0, the number of records remains unchanged. If the difference is not 0, the number of records changes. The difference is used as the record count change value, and the corresponding larger partition date is used as the record count change date. The record count change value, record count change date, and table name are merged into a single record and written to the record count change details table. Third, read the record count change details table daily, group by table name, and sort within each group by the date of record count change. Obtain the latest record count change date from the data table. Associate the latest partition in the exploration results table by table name, calculate the difference between the latest partition date and the latest record count change date in the exploration results table, and use this difference as the number of days the record count in the data table has not been updated. Combine the latest partition date, table name, latest record count change date, and number of days the record count has not been updated from the exploration results table into a single record and write it to the record count change statistics table, as shown in Table 5. When the number of days the record count has not been updated is 0, it means that the record count in the data table has changed from the previous day on the exploration date.

[0063] Table 5: Statistics on Changes in the Number of Records

[0064]

[0065] Step S6: The purpose of this step is to construct a data model similarity algorithm to assist manual model deduplication and add the deduplicated models to the data model deprecation set.

[0066] First, the table structure of a data model consists of data fields. Calculating the similarity of strings composed of these data fields can help manual judgment of whether the data model is redundant or repetitive. String similarity is determined using Jaro-Winkler Distance text edit distance calculation. The formula for Jaro-Winkler Distance similarity calculation is:

[0067] Sim ω =Sim j +(lp(1-Sim j ))

[0068] Where: l is the length of the common prefix of the strings, with a maximum value of 4; P is a constant factor, with a default value of 0.1 and a maximum value of 0.25; Sim j It is the Jaro similarity between strings S1 and S2, given strings S1, S2, and Sim. j The formula for calculating the similarity between the two is as follows:

[0069]

[0070] Where: |S i | is string S i The length of the string is given by m, the number of matched characters is given by t, and the string consists of characters that are identical in S1 and S2 and are no more than 1 meter apart. When the two strings are considered to match, they are considered to be a match.

[0071] Second, read the latest partition content of the exploration results table, group it by table name and table comments, and concatenate the field names and field comments under the same table name and the same table comment using the GROUP_CONCAT() function, separating each field name and each field comment with a comma, generating field name concatenation strings and field comment concatenation strings. Obtain the table name, table comments, field name concatenation strings, and field comment concatenation strings. Simultaneously, read the latest partition content of the record count change statistics table, retrieve the number of days the record count has not been updated by associating it with the table name, and write the table name, table comments, field name concatenation strings, field comment concatenation strings, and record count not updated days into the data model reference table, as shown in Table 6 below.

[0072] Table 6 Data Model Reference Table

[0073]

[0074]

[0075] Third, the data model reference table is divided into two sets based on the number of days since the last update. For example, records with a non-zero number of days since the last update are assigned to set A, and records with a zero number of days since the last update are assigned to set B. The field name concatenation strings in set A are used as the primary set, and the field name concatenation strings in set B are used as the secondary set. The similarity is calculated using the Jaro-Winkler Distance formula to determine the field name concatenation strings in set B that are most similar to the field name concatenation strings in set A. This yields the model in B that best matches the model in A and its similarity, and a model similarity matrix is ​​then established.

[0076] For example, the table name of the data model is tmp_hsjc_result_copy, and its field name concatenation string is "xm,sfzh,jg,cysj,jcjg,cbgsj,jcpc,cydmc,zz,sjh,sgbh,dt". The field name concatenation string "xm,sfzh,jg,cysj,jcjg,cbgsj,jcpc,cydmc,zz,sjh,sgbh" in the data model ods_hsjc_result_df has the highest similarity to it. In the matrix, the similarity value between the two models, calculated using the Jaro-Winkler Distance similarity method, is 0.98846. Since the number of records in the model `tmp_hsjc_result_copy` has not been updated for 193 days since the exploration results table began recording, while the number of records in `ods_hsjc_result_df` is continuously updated, this matrix assists in manually determining that model `tmp_hsjc_result_copy` is a temporary backup table of model `ods_hsjc_result_df`, and model `tmp_hsjc_result_copy` is added to the data model discontinued set. From the time the data table is explored in step S1, this model does not participate in the iteration of the full set of personnel feature data.

[0077] Through the above process, all personnel characteristic data is extracted from the data warehouse (excluding data that has not been updated for more than a set number of days or data in deactivated models), resulting in a complete set of personnel characteristic data in the data warehouse. The personnel characteristic data to be searched is then searched and compared within this complete set to find the location of data in the corresponding data table. The record corresponding to each location is then used as the dataset to be associated with that person in the data warehouse. For example, to find the relevant epidemic prevention data for a person with ID number α, α can be compared with the data in the complete set of personnel characteristic data in the data warehouse to find relevant records for α, which are then associated with the data records in the relevant tables to quickly find the relevant epidemic prevention data for α.

[0078] Device Example:

[0079] This invention provides a device for comparing personnel information across a full dataset, such as... Figure 6 As shown, it includes a memory, a processor, and an internal bus. The processor and memory interact with each other via the internal bus. The processor can be a microprocessor (MCU), a programmable logic device (FPGA), or other processing devices. The memory can be various types of memory that store information using electrical energy, such as RAM and ROM; it can also be various types of memory that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memory, bubble memory, and USB flash drives; it can also be various types of memory that store information using optical methods, such as CDs and DVDs; and of course, it can also be other types of memory, such as quantum memory and graphene memory.

Claims

1. A method for comparing personnel information across a full dataset, characterized in that, The full dataset was compared using the following steps: 1) Traverse the metadata of the data tables in the data warehouse to obtain the table name, the number of records in each table, and the names of the data fields contained in each table, and save the results as a full partition table of exploration results; 2) Based on the generation rules of personnel feature data, construct a personnel feature data verification function, wherein the personnel feature data is used to indicate the uniqueness of personnel identity; 3) Filter the data records of each field in each table stored in the full partition table of the exploration results by data type to filter out records that do not conform to the data type of the characteristic value, and use the personnel characteristic data verification function to verify the filtered data records to obtain the fields that conform to the personnel characteristic data verification function, that is, the records that contain personnel characteristic data in the fields, so as to locate the field where the personnel characteristic data is located. 4) Traverse all records in the located field, use the personnel feature data verification function to verify the data in the traversed records, extract personnel feature data, and obtain the full set of personnel feature data in the data warehouse. 5) Search and compare the personnel feature data to be searched in the full set of personnel feature data in the data warehouse, find the position of the data table where the data that matches the personnel feature data to be searched is located, and take the record corresponding to each position as the dataset that needs to be associated with the personnel to be searched in the data warehouse.

2. The method for comparing personnel information across a full dataset according to claim 1, characterized in that, The method also includes determining the number of days each table has not been updated based on the number of data records in each table in the full partition table obtained from the daily exploration results. If the number of days not updated exceeds the set number of days, then step 5) will no longer perform a full information comparison for that table.

3. The method for comparing personnel information across a full dataset according to claim 1 or 2, characterized in that, The method also includes determining the similarity between the first type of data table and the second type of data table in the data warehouse, filtering out suspected discontinued tables based on the similarity, filtering out discontinued tables from the suspected discontinued tables, adding the filtered discontinued tables to the data model discontinued set, and in step 1), no longer traversing the data tables in the discontinued set; The first type of data table refers to a data table with 0 days not updated, and the second type of data table refers to a data table with non-zero days not updated. The process of determining the similarity between data tables is as follows: the field names with the same table name and under the same table comments in the full partition table of the exploration results are sequentially connected to form a field name concatenation string; This calculation uses a string similarity formula to determine the similarity between concatenated strings containing field names from two tables.

4. The method for comparing personnel information across a full dataset according to claim 1, characterized in that, Step 3) involves concatenating the table name and field name from the read data record using SQL syntax, and inputting the concatenated result into the personnel feature data verification function to obtain an SQL statement that determines whether the field values ​​in the data table pass the verification, thereby determining the fields in the data table where the personnel feature data is located.

5. The method for comparing personnel information across a full dataset according to claim 1, characterized in that, In step 4), the table name, field name, and partition type of the location field are concatenated using SQL syntax. The concatenated result is then input into the personnel feature data verification function to obtain an SQL statement that verifies the data content based on the table's partition type, thereby obtaining the personnel feature data.

6. The method for comparing personnel information across a full dataset according to claim 1, characterized in that, In both steps 3) and 4), UNION ALL is used to merge multiple SQL statements into a single execution statement.

7. The method for comparing personnel information across a full dataset according to claim 1, characterized in that, The personnel feature data is the ID card number, and the corresponding personnel feature data verification function is related to the verification code generation rule of the last digit of the ID card number.

8. The method for comparing personnel information across a full dataset according to claim 1, characterized in that, The number of records in the data table is related to the partition type of the data table. If it is a full partition type, the number of records is counted based on the latest full partition; if it is an incremental partition type, the number of records is counted based on all partitions.

9. A device for comparing personnel information across a full dataset, characterized in that, The method for comparing personnel information across a full dataset, as described in any one of claims 1-8, was adopted.

Images