Table data proofreading method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a table data proofreading method and device, electronic equipment and a storage medium, and belongs to the artificial intelligence technical field, and is suitable for the financial technology field and the medical field. The method comprises the following steps: obtaining a reference table and a target table; wherein the reference table comprises reference data, the reference data is provided with a reference field identifier, the target table comprises intermediate data, and the intermediate data is provided with an intermediate field identifier; the preset field identifier mapping relationship is queried according to the intermediate field identifier of the intermediate data, and a reference field identifier is obtained; candidate data is screened out from the reference data according to the reference field identifier; a target data mapping relationship is generated according to the language type of the candidate data, the language type of the intermediate data, the candidate data and the intermediate data, the verification data is screened out from the reference data according to the target data mapping relationship, and the data of the intermediate data is proofread according to the verification data. The embodiment of the application can improve the proofreading efficiency of the table data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to the fields of financial technology and medical technology, and particularly to a method and apparatus for tabular data verification, electronic equipment and storage medium. Background Technology

[0002] Currently, in the fintech field (such as bank data reconciliation) and the healthcare technology field (such as the aggregation of electronic medical records and regional health platforms), there is a need to perform consistency checks on large amounts of data. Such checks typically rely on manually configured rules. However, traditional checks have significant drawbacks: firstly, field mapping requires manual configuration item by item, which is inefficient when dealing with hundreds of tables and thousands of fields. Secondly, the mapping relationships lack automatic verification, failing to detect omissions, mismatches, or type incompatibilities. For example, mismapping "annual interest rate" to "fee percentage" or "patient ID" to "medical record number" leads to unreliable verification results, requiring extensive manual review. Therefore, improving the efficiency of cross-tabulation has become a pressing technical problem. Summary of the Invention

[0003] The main objective of this application is to provide a method, apparatus, electronic device, and storage medium for verifying tabular data, aiming to improve the efficiency of verifying tabular data.

[0004] To achieve the above objectives, a first aspect of this application proposes a method for verifying tabular data, the method comprising: Obtain a reference table and a target table; wherein, the reference table includes reference data, the reference data has a preset reference field identifier, and the target table includes intermediate data, the intermediate data has a preset intermediate field identifier; For each piece of intermediate data, a preset field identifier mapping relationship is queried based on the intermediate field identifier of the intermediate data to obtain a reference field identifier; wherein, the field identifier mapping relationship is used to represent the matching relationship between the reference field identifier and the intermediate field identifier; Candidate data is selected from the reference data based on the benchmark field identifier; Based on the language type of the candidate data, the language type of the intermediate data, the candidate data, and the intermediate data, a data mapping relationship is generated to obtain the target data mapping relationship; For each of the intermediate data, verification data is selected from the reference data according to the target data mapping relationship, and the intermediate data is verified based on the verification data.

[0005] In some embodiments, generating a target data mapping relationship based on the language type of the candidate data, the language type of the intermediate data, the candidate data, and the intermediate data includes: The candidate data and the intermediate data of the same language type are paired to obtain data pairs; Based on the language type, a target matching rule is selected from the preset matching rules, and the data pairs are evaluated for matching based on the target matching rule to obtain a matching score. The target data mapping relationship is determined based on the matching score and the preset score threshold.

[0006] In some embodiments, determining the target data mapping relationship based on the matching score and a preset score threshold includes: If the matching score is greater than or equal to the score threshold, a first data mapping relationship is determined based on the data pair; If the matching score is less than the score threshold, then the data type of the intermediate data is obtained; A second data mapping relationship is obtained by determining the mapping relationship based on the data type, the intermediate data, and the candidate data. The target data mapping relationship is determined based on the first data mapping relationship and the second data mapping relationship.

[0007] In some embodiments, the data type includes a numeric type, and the step of determining a second data mapping relationship based on the data type, the intermediate data, and the candidate data includes: If the data type is a numeric type, the reference numeric error is obtained by querying the preset numeric error mapping relationship based on the intermediate field identifier of the intermediate data. A first reference value and a second reference value are obtained by calculating based on the reference value error and the candidate data; wherein the first reference value is less than the second reference value. If the intermediate data is greater than or equal to the first reference value, and the intermediate data is less than or equal to the second reference value, then the second data mapping relationship is determined based on the intermediate data and the candidate data.

[0008] In some embodiments, the data type includes a text type, and the step of determining a second data mapping relationship based on the data type, the intermediate data, and the candidate data includes: If the data type is text, perform text semantic recognition on the candidate data to obtain candidate semantic features, and perform text semantic recognition on the intermediate data to obtain intermediate semantic features; Calculate the text similarity between the intermediate semantic features and the candidate semantic features, and determine the baseline data from the candidate data based on the text similarity; The second data mapping relationship is determined based on the baseline data and the intermediate data.

[0009] In some embodiments, the field identifier mapping relationship is obtained based on the following steps: Semantic extraction is performed on the reference field identifier to obtain the reference field semantics, and semantic extraction is performed on the intermediate field identifier to obtain the intermediate field semantics; Calculate the similarity between the semantics of the reference field and the semantics of the intermediate field to obtain a semantic similarity score; The field identifier mapping relationship is determined based on the semantic similarity score.

[0010] In some embodiments, obtaining the reference table and the target table includes: Get the first and second tables; Sensitive data identification is performed on the first table to obtain the first target data, and sensitive data identification is performed on the second table to obtain the second target data; The reference table is obtained by replacing the first target data in the first table with a preset data mask. The second target data in the second table is replaced according to the data mask to obtain the target table.

[0011] To achieve the above objectives, a second aspect of this application provides a table data proofreading apparatus, the apparatus comprising: A table acquisition module is used to acquire a reference table and a target table; wherein, the reference table includes reference data, the reference data is preset with reference field identifiers, and the target table includes intermediate data, the intermediate data is preset with intermediate field identifiers; A field identifier filtering module is used to query a preset field identifier mapping relationship for each intermediate data based on the intermediate field identifier of the intermediate data to obtain a reference field identifier; wherein, the field identifier mapping relationship is used to represent the matching relationship between the reference field identifier and the intermediate field identifier; A data filtering module is used to filter candidate data from the reference data based on the benchmark field identifier; A data mapping relationship generation module is used to generate a target data mapping relationship based on the language type of the candidate data, the language type of the intermediate data, the candidate data, and the intermediate data. A data verification module is used to filter verification data from the reference data according to the target data mapping relationship for each intermediate data, and to perform data verification on the intermediate data based on the verification data.

[0012] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0013] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0014] The table data verification method, apparatus, electronic device, and storage medium proposed in this application achieve preliminary matching between reference field identifiers and intermediate field identifiers through a preset field identifier mapping relationship. Based on this, the reference data is filtered through the benchmark field identifier to narrow down the range of candidate data. Furthermore, a target data mapping relationship is generated by combining the language type of the candidate data, the language type of the intermediate data, and the candidate data and intermediate data, thereby achieving dynamic optimization and verification of the mapping relationship. Finally, the intermediate data is accurately verified based on the target data mapping relationship. The table data verification method of this application improves the accuracy of field matching and reduces the risk of omissions and mismatches without requiring a large amount of manual configuration of field mappings item by item, significantly improving the efficiency and reliability of table data verification results. Attached Figure Description

[0015] Figure 1 This is a flowchart of the tabular data verification method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S101 in the text; Figure 3 yes Figure 1 The flowchart of step S102 in the document; Figure 4 yes Figure 1 The flowchart of step S104 in the process; Figure 5 yes Figure 4 The flowchart of step S403 in the process; Figure 6 yes Figure 5 The flowchart of step S503 in the process; Figure 7 yes Figure 5 Another flowchart of step S503 in the process; Figure 8 This is a schematic diagram of the structure of the tabular data verification device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0017] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0019] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0020] Currently, in the fintech field (such as bank data reconciliation) and the healthcare technology field (such as the aggregation of electronic medical records and regional health platforms), there is often a need to perform consistency checks on large amounts of data. Such checks typically rely on manually configured rules. However, traditional checks have significant drawbacks: firstly, field mapping requires manual configuration item by item, which is inefficient when dealing with hundreds of tables and thousands of fields. Secondly, the mapping relationships lack automatic verification, failing to detect omissions, mismatches, or type incompatibilities. For example, mismapping "annual interest rate" to "fee percentage" or "patient ID" to "medical record number" leads to unreliable verification results, requiring extensive manual review. Therefore, improving the efficiency of cross-tabulation has become a pressing technical problem.

[0021] Based on this, embodiments of this application provide a method and apparatus for verifying tabular data, an electronic device, and a storage medium, aiming to improve the efficiency of verifying tabular data.

[0022] The table data verification method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the table data verification method in this application is described.

[0023] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0024] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0025] The table data verification method provided in this application relates to the field of artificial intelligence technology. This table data verification method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the table data verification method, but is not limited to the above forms.

[0026] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0027] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0028] Figure 1 This is an optional flowchart of the tabular data verification method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0029] Step S101: Obtain a reference table and a target table; wherein, the reference table includes reference data, which is pre-defined with reference field identifiers, and the target table includes intermediate data, which is pre-defined with intermediate field identifiers. Step S102: For each piece of intermediate data, query the preset field identifier mapping relationship based on the intermediate field identifier of the intermediate data to obtain the reference field identifier; wherein, the field identifier mapping relationship is used to represent the matching relationship between the reference field identifier and the intermediate field identifier. Step S103: Select candidate data from the reference data based on the baseline field identifier; Step S104: Generate the target data mapping relationship based on the language type of the candidate data, the language type of the intermediate data, the candidate data, and the intermediate data; Step S105: For each intermediate data, select verification data from the reference data according to the target data mapping relationship, and perform data verification on the intermediate data based on the verification data.

[0030] Steps S101 to S105 of this application embodiment achieve preliminary matching between reference field identifiers and intermediate field identifiers through a preset field identifier mapping relationship. Based on this, the reference data is filtered through the benchmark field identifier to narrow down the candidate data range. Furthermore, the target data mapping relationship is generated by combining the language type of the candidate data, the language type of the intermediate data, the candidate data, and the intermediate data, thereby achieving dynamic optimization and verification of the mapping relationship. Finally, the intermediate data is accurately verified based on the target data mapping relationship. The table data verification method of this application improves the accuracy of field matching and reduces the risk of omissions and mismatches without requiring a large amount of manual configuration of field mappings item by item, significantly improving the efficiency and reliability of table data verification results.

[0031] In step S101 of some embodiments, the reference table is a structured data table used to provide a standard data source, and the target table is a structured data table to be validated. Reference data refers to the data content corresponding to each field in the reference table, and intermediate data refers to the data content corresponding to each field in the target table. Reference field identifiers are identifiers used to identify the meaning of each field in the reference table, and intermediate field identifiers are identifiers used to identify the meaning of each field in the target table.

[0032] Furthermore, structured data tables are obtained through database reading, file import, or API access, i.e., reference tables and target tables are acquired. Reference field identifiers and intermediate field identifiers can be obtained by parsing the table header information, the English names of the fields, or the Chinese comments of the fields. For example, the reference table can be an account transaction information table in a banking system, and the target table can be a transaction summary table in a reconciliation system. The reference table includes two columns of data, with headers for "Account ID" and "Annual Interest Rate," respectively. Therefore, the reference field identifiers include "Account ID" and "Annual Interest Rate," and the reference data is the specific numerical value or text content under the corresponding field. Similarly, the intermediate table includes two columns of data, with headers for "Account ID" and "interest_rate," respectively. Therefore, the intermediate field identifiers include "Account ID" and "interest_rate," and the intermediate data is the specific numerical value or text content under the corresponding field.

[0033] Please see Figure 2 In some embodiments, the reference table and the target table may store user privacy information. To ensure user information security, it is necessary to de-identify some information in the tables. Step S101 may include, but is not limited to, steps S201 to S204: Step S201: Obtain the first table and the second table; Step S202: Perform sensitive data identification on the first table to obtain the first target data, and perform sensitive data identification on the second table to obtain the second target data; Step S203: Replace the first target data in the first table with a preset data mask to obtain a reference table; Step S204: Replace the second target data in the second table with the data mask to obtain the target table.

[0034] In step S201 of some embodiments, the first table is a structured data table used to provide the original data source, and the second table is a structured data table to be processed used for table data validation. Taking a medical scenario as an example, the first table can be a patient medical record information table in a regional medical information platform, and the second table can be a patient medical record information table in a hospital information system.

[0035] In step S202 of some embodiments, the first target data is a set of data in a first table that are identified as sensitive attributes, and the second target data is a set of data in a second table that are identified as sensitive attributes. Sensitive attributes include ID card number, home address, name, etc.

[0036] Furthermore, field identifiers can be matched based on preset sensitive field rules, and verification and identification can be performed in conjunction with field value characteristics. Field identifier matching includes rule comparison of the English and Chinese names of the fields to determine the specific scope of sensitive data. For example, fields identified as "ID card number" and "id_number," as well as data conforming to the ID card number encoding rules, are identified as sensitive data. Assuming a reference field identifier "ID card number" exists in the first table, the corresponding reference data is identified as the first target data. Assuming an intermediate field identifier "id_number" exists in the second table, the corresponding intermediate data is identified as the second target data. In other embodiments, field names, field annotations, or data content can also be encoded, and the aforementioned encoding can be classified using a pre-trained deep learning classification model (e.g., a pre-trained language model based on the Transformer structure, or a support vector machine model) to determine whether the corresponding data is private information.

[0037] In step S203 of some embodiments, the data mask is a set of rules for replacing sensitive data, used to hide the original sensitive information while preserving the data structure and some features.

[0038] Furthermore, character replacement processing can be performed on the first target data based on preset masking rules. These masking rules are manually pre-configured to ensure that the replaced data still meets the format constraints of the original field. For example, for the ID number "123456789", the first four and last two digits are retained according to the masking rules, and the middle part is replaced with "*", resulting in "1234**89". After all the first target data in the first table has undergone data masking processing, the first table after de-identification processing is determined as the reference table.

[0039] In step S204 of some embodiments, the implementation method of this embodiment is the same as that of the embodiment of step S203, and will not be described again here. After all the second target data in the second table has completed the data masking process, the second table after the de-identification process is determined as the target table.

[0040] Steps S201 to S204 as illustrated in this application embodiment identify sensitive data in the first and second tables before data verification, and replace sensitive fields based on a unified data mask. This ensures that the reference table and the target table maintain data structure consistency without exposing the original sensitive information. In this way, the risk of sensitive information leakage is reduced while ensuring the executability and reliability of the data verification process. This achieves a balance between sensitive data protection and data verification requirements, thereby improving the security of table data verification.

[0041] In step S102 of some embodiments, the field identifier mapping relationship is a set of rules used to represent the correspondence between reference field identifiers and intermediate field identifiers, and the baseline field identifier is a reference field identifier obtained by matching the intermediate field identifier.

[0042] Furthermore, the field identifier mapping relationship can be constructed as follows: If both the reference field identifier and the intermediate field identifier are in Chinese, the Chinese characters are directly compared; if the Chinese text is completely identical, a mapping relationship is established. If both the reference field identifier and the intermediate field identifier are in English, case sensitivity, underscores, and spaces are ignored, and character matching is performed; if the letter types of both are completely identical, a mapping relationship is established.

[0043] Please see Figure 3 In some embodiments, when the above method cannot match the reference field identifier and the intermediate field identifier (for example, one field identifier is in Chinese and the other field identifier is in English), the process of establishing the field identifier mapping relationship in step S102 may also include, but is not limited to, steps S301 to S303: Step S301: Semantic extraction is performed on the reference field identifier to obtain the reference field semantics, and semantic extraction is performed on the intermediate field identifier to obtain the intermediate field semantics; Step S302: Calculate the similarity between the semantics of the reference field and the semantics of the intermediate field to obtain the semantic similarity score; Step S303: Determine the field identifier mapping relationship based on the semantic similarity score.

[0044] In step S301 of some embodiments, the reference field semantics is the semantic feature of the reference field identifier, and the intermediate field semantics is the semantic feature of the intermediate field identifier. Semantic extraction is used to convert the reference field identifier and the intermediate field identifier from their surface character form into semantic representations that can reflect business meaning, thereby eliminating the impact of field naming differences on field matching. Further, a preset semantic representation model can be used to encode the reference field identifier and the intermediate field identifier, and then map them into semantic vectors or semantic feature representations. Specifically, the semantic representation model can employ a word vector model, a recurrent neural network-based encoding model, or a Transformer-based encoding model.

[0045] In step S302 of some embodiments, the semantic similarity score is a numerical result used to measure the degree of similarity between the semantics of the reference field and the semantics of the intermediate field. The semantic similarity score can be the cosine similarity between the semantics of the reference field and the semantics of the intermediate field.

[0046] In step S303 of some embodiments, the semantic similarity score is compared with a preset similarity threshold. When the semantic similarity score is higher than or equal to the preset similarity threshold, a mapping relationship is established between the reference field identifier and the intermediate field identifier. When the semantic similarity score is lower than the preset similarity threshold, no mapping relationship is established, thereby filtering out semantically consistent field correspondences. In this embodiment, the numerical range of the semantic similarity score is [0,1], and the preset similarity threshold can be 0.8.

[0047] Steps S301 to S303 as shown in the embodiments of this application extract semantic features from the reference field identifier and the intermediate field identifier to obtain the semantic information of the field identifier, and establish a field identifier mapping relationship based on the semantic similarity score. This enables effective matching between fields even when the field languages ​​are inconsistent, avoids mismatch problems caused by simply relying on character matching, improves the accuracy of field matching, and thus improves the overall data verification efficiency.

[0048] For example, the reference field identifiers include "account_id", "attending physician", and "disease type", while the intermediate field identifiers include "AccountID", "name of doctor", and "disease type".

[0049] After the above steps, the field identifier mapping relationships are obtained as follows: reference field identifier "account_id" and intermediate field identifier "AccountID", reference field identifier "attending physician" and intermediate field identifier "name of doctor", and reference field identifier "disease type" and intermediate field identifier "disease type". For the intermediate field identifier "AccountID", the base field identifier is "account_id".

[0050] In step S103 of some embodiments, candidate data refers to the set of data corresponding to intermediate data that is filtered from the reference data based on the benchmark field identifier. For example, when the benchmark field identifier is "account_id", all data in the reference table with the header "account_id" will be used as the candidate data set.

[0051] In step S104 of some embodiments, the target data mapping relationship refers to the matching relationship between each intermediate data and the reference data in the reference table, which is used to clarify the specific correspondence between the intermediate data and the reference data.

[0052] Further, please refer to Figure 4 In some embodiments, step S104 may include, but is not limited to, steps S401 to S403: Step S401: Pair candidate data and intermediate data of the same language type to obtain data pairs; Step S402: Select target matching rules from preset matching rules according to language type, and evaluate the data pairs according to the target matching rules to obtain matching scores; Step S403: Determine the target data mapping relationship based on the matching score and the preset score threshold.

[0053] In step S401 of some embodiments, a data pair refers to a one-to-one data combination formed by candidate data and intermediate data according to the same language type. Candidate data and intermediate data with the same language type are paired item by item according to the field identifier mapping relationship. For example, when the candidate data are the English strings "AccountID123", "AccountID456", and "User321", and the field value in the intermediate data is the English string "account_id123", then data pairs "account_id123" and "AccountID123" and "account_id123" and "AccountID456" are formed.

[0054] In step S402 of some embodiments, the preset pairing rule is a set of data matching rules pre-set for different language types, the target pairing rule is a specific matching rule obtained by filtering the preset pairing rules according to the language type of the current data pair, and the matching score is a numerical result used to quantify the degree of matching of the data pair.

[0055] Furthermore, the preset matching rules may include: 1. When the language type corresponding to the data pair is English, ignore the differences in capitalization, underscores and spaces, and compare the letters in turn. The matching score is obtained by calculating the percentage of the number of identical letters out of the total number of letters in the intermediate data; 2. When the language type corresponding to the data pair is Chinese, compare the text characters in turn. The matching score is obtained by calculating the percentage of the number of identical characters out of the total number of characters in the intermediate data.

[0056] In step S403 of some embodiments, further please refer to Figure 5 Step S403 may also include, but is not limited to, steps S501 to S504: Step S501: If the matching score is greater than or equal to the score threshold, determine the first data mapping relationship based on the data pair; Step S502: If the matching score is less than the score threshold, then obtain the data type of the intermediate data; Step S503: Determine the mapping relationship based on the data type, intermediate data, and candidate data to obtain the second data mapping relationship; Step S504: Determine the target data mapping relationship based on the first data mapping relationship and the second data mapping relationship.

[0057] In step S501 of some embodiments, the score threshold is a numerical standard used to determine whether the matching degree of data pairs meets the conditions for establishing a mapping relationship. The first data mapping relationship is the correspondence between candidate data and intermediate data determined based on the matching score meeting the score threshold condition. In this embodiment, the score threshold value is 80%.

[0058] In step S502 of some embodiments, the data type of the intermediate data is determined by field type definition or by format parsing of the data content, which may specifically include numeric type and text type.

[0059] In step S503 of some embodiments, please refer to Figure 6 When the data type of the intermediate data is numeric, step S503 includes, but is not limited to, steps S601 to S603: Step S601: If the data type is numeric, query the preset numeric error mapping relationship according to the intermediate field identifier of the intermediate data to obtain the reference numeric error. Step S602: Calculate the first reference value and the second reference value based on the reference value error and the candidate data; wherein the first reference value is less than the second reference value. Step S603: If the intermediate data is greater than or equal to the first reference value and the intermediate data is less than or equal to the second reference value, then determine the second data mapping relationship based on the intermediate data and the candidate data.

[0060] In step S601 of some embodiments, the numerical error mapping relationship is a set of rules used to describe the allowable numerical deviation range corresponding to different intermediate field identifiers, and the reference numerical error is the numerical deviation range obtained from the numerical error mapping relationship based on the intermediate field identifier. For example, when the intermediate field identifier is "interest_rate", the corresponding reference numerical error in the numerical error mapping relationship is ±0.002.

[0061] In step S602 of some embodiments, the first reference value and the second reference value are numerical interval boundaries calculated based on candidate data and reference value error. The first reference value represents the minimum value within the allowable error range, and the second reference value represents the maximum value within the allowable error range. Specifically, using candidate data as a reference, interval calculation is performed based on the reference value error. The first reference value is obtained by subtracting the reference value error from the candidate data, and the second reference value is obtained by adding the reference value error to the candidate data. For example, when the candidate data is 1 and the reference value error is ±0.2, the first reference value is 0.8, and the second reference value is 1.2.

[0062] In step S603 of some embodiments, if the intermediate data is less than the first reference value or greater than the second reference value, it indicates that the intermediate data does not fall within the allowable error range and does not meet the numerical matching condition. A second data mapping relationship is not established based on the intermediate data and the candidate data, and the intermediate data is marked as data to be further verified or mismatched data.

[0063] Steps S601 to S603 of this application embodiment introduce a numerical error mapping relationship for numerical data, obtain the corresponding reference numerical error based on the intermediate field identifier, and construct a numerical range based on the candidate data. When the intermediate data falls into the numerical range, it is determined that there is a mapping relationship between the data, thereby avoiding misjudgment problems caused by precision differences or calculation errors. The method of this application embodiment can improve the fault tolerance and accuracy of numerical data matching, reduce the false alarm rate, and improve the reliability and applicability of the table data verification results.

[0064] Please see Figure 7 In other embodiments, the data type includes text, and step S503 may include, but is not limited to, steps S701 to S703: Step S701: If the data type is text, perform text semantic recognition on the candidate data to obtain candidate semantic features, and perform text semantic recognition on the intermediate data to obtain intermediate semantic features. Step S702: Calculate the text similarity between the intermediate semantic features and the candidate semantic features, and determine the baseline data from the candidate data based on the text similarity. Step S703: Determine the second data mapping relationship based on the baseline data and intermediate data.

[0065] In step S701 of some embodiments, candidate semantic features are vectorized or structured representations obtained by semantically abstracting the content of candidate data text, and intermediate semantic features are vectorized or structured representations obtained by semantically abstracting the content of intermediate data text. Text semantic recognition can be performed using word vector models, recurrent neural network models, or encoding models based on the Transformer structure.

[0066] In step S702 of some embodiments, text similarity is used to measure the semantic closeness between candidate data and intermediate data. Text similarity can be obtained using vector inner product, cosine similarity, or a similarity evaluation model based on a deep matching network. The similarity scores corresponding to all candidate data are sorted, and the candidate data with the highest score that meets preset conditions is selected as the benchmark data. For example, among multiple candidate data, "abnormal blood sugar," "chronic hyperglycemia," and "abnormal insulin metabolism" are compared with the intermediate data "diabetes" to obtain different similarity scores, and "chronic hyperglycemia," with the highest similarity, is selected as the benchmark data.

[0067] In step S703 of some embodiments, a data correspondence relationship is established based on the semantic matching result between the reference data and the intermediate data, which is the second data mapping relationship.

[0068] Steps S701 to S703 of this application embodiment extract semantic features from candidate data and intermediate data, transforming text information into quantifiable semantic feature representations. By calculating the text similarity between semantic features, the benchmark data that is semantically closest to the intermediate data is selected. Based on this, a second data mapping relationship is established, thereby achieving effective matching of text data that are semantically consistent but express different meanings. The method of this application embodiment can improve the accuracy and adaptability of text type data matching, reduce matching deviations caused by differences in text expression, and improve the automation level and reliability of table data verification results.

[0069] In step S504 of some embodiments, the target data mapping relationship is the final data correspondence formed after integrating the first data mapping relationship and the second data mapping relationship.

[0070] Steps S501 to S504 of this application embodiment directly confirm the data pairs whose matching scores meet the score threshold as the first data mapping relationship. At the same time, for the data pairs whose matching scores do not reach the score threshold, a second data mapping relationship is further constructed according to the data type of the intermediate data. The first data mapping relationship and the second data mapping relationship are then integrated. In order to ensure the comprehensiveness of the mapping results, the method of this application embodiment can improve the coverage and completeness of the data mapping relationship, reduce the omission of effective data, and ultimately improve the reliability of table data verification.

[0071] Steps S401 to S403 of this embodiment involve pairing candidate data and intermediate data according to language type, prioritizing direct comparison of data in the same language type, and selecting corresponding pairing rules based on language type to evaluate the matching of data pairs, thereby reducing the difficulty of matching. Subsequently, a matching score reflecting the degree of data consistency is obtained. Furthermore, the target data mapping relationship is determined based on the matching score and a preset score threshold, realizing unified matching processing for data with different expression forms. The method of this embodiment improves the accuracy and adaptability of data matching and reduces the reduction in matching efficiency caused by language differences or format differences.

[0072] In step S105 of some embodiments, the verification data refers to the data determined from the reference data according to the target data mapping relationship and used to compare the intermediate data. The intermediate data is compared with the corresponding verification data for consistency, including numerical comparison, format comparison, and logical relationship verification. Data is deemed abnormal when the data logic seriously deviates from the corresponding pattern or the numerical difference exceeds a preset threshold. For example, if the "interest_rate" in the intermediate data is 0.35, while the corresponding reference data in the verification data is 0.035, a numerical comparison indicates an order-of-magnitude error. Similarly, if the verification data corresponding to the "patient ID" in the intermediate data is an inconsistent unique identifier, a mapping error is determined. This completes the automated data verification process for the intermediate data, and intermediate data that fails verification is highlighted.

[0073] Please see Figure 8 This application also provides a table data proofreading device that can implement the above-described table data proofreading method. The device includes: The table acquisition module 801 is used to acquire a reference table and a target table; wherein, the reference table includes reference data, which is preset with reference field identifiers, and the target table includes intermediate data, which is preset with intermediate field identifiers. The field identifier filtering module 802 is used to query the preset field identifier mapping relationship based on the intermediate field identifier of each intermediate data to obtain the reference field identifier; wherein, the field identifier mapping relationship is used to represent the matching relationship between the reference field identifier and the intermediate field identifier. The data filtering module 803 is used to filter candidate data from the reference data based on the baseline field identifier; The data mapping relationship generation module 804 is used to generate a data mapping relationship based on the language type of the candidate data, the language type of the intermediate data, the candidate data, and the intermediate data, so as to obtain the target data mapping relationship. The data verification module 805 is used to select verification data from the reference data according to the target data mapping relationship for each intermediate data, and to perform data verification on the intermediate data based on the verification data.

[0074] The specific implementation of the table data verification device is basically the same as the specific implementation of the table data verification method described above, and will not be repeated here.

[0075] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described table data verification method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0076] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the table data verification method of the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0077] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described table data verification method.

[0078] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0079] The table data verification method, device, electronic device, and storage medium provided in this application embodiment achieve preliminary matching between reference field identifiers and intermediate field identifiers through a preset field identifier mapping relationship. Based on this, the reference data is filtered through the benchmark field identifier to narrow down the range of candidate data. Furthermore, the target data mapping relationship is generated by combining the language type of the candidate data, the language type of the intermediate data, and the candidate data and intermediate data, thereby achieving dynamic optimization and verification of the mapping relationship. Finally, the intermediate data is accurately verified based on the target data mapping relationship. The table data verification method of this application improves the accuracy of field matching and reduces the risk of omissions and mismatches without requiring a large amount of manual configuration of field mappings item by item, significantly improving the efficiency and reliability of table data verification results.

[0080] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0081] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0082] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0084] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0085] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0087] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0089] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method of proofreading tabular data, characterized by, The method includes: Obtain a reference table and a target table; wherein, the reference table includes reference data, the reference data has a preset reference field identifier, and the target table includes intermediate data, the intermediate data has a preset intermediate field identifier; For each piece of intermediate data, a preset field identifier mapping relationship is queried based on the intermediate field identifier of the intermediate data to obtain a reference field identifier; wherein, the field identifier mapping relationship is used to represent the matching relationship between the reference field identifier and the intermediate field identifier; Candidate data is selected from the reference data based on the benchmark field identifier; Based on the language type of the candidate data, the language type of the intermediate data, the candidate data, and the intermediate data, a data mapping relationship is generated to obtain the target data mapping relationship; For each of the intermediate data, verification data is selected from the reference data according to the target data mapping relationship, and the intermediate data is verified based on the verification data.

2. The method of claim 1, wherein, The step of generating a data mapping relationship based on the language type of the candidate data, the language type of the intermediate data, the candidate data, and the intermediate data to obtain the target data mapping relationship includes: The candidate data and the intermediate data of the same language type are paired to obtain data pairs; Based on the language type, a target matching rule is selected from the preset matching rules, and the data pairs are evaluated for matching based on the target matching rule to obtain a matching score. The target data mapping relationship is determined based on the matching score and the preset score threshold.

3. The method of claim 2, wherein, Determining the target data mapping relationship based on the matching score and a preset score threshold includes: If the matching score is greater than or equal to the score threshold, a first data mapping relationship is determined based on the data pair; If the matching score is less than the score threshold, then the data type of the intermediate data is obtained; A second data mapping relationship is obtained by determining the mapping relationship based on the data type, the intermediate data, and the candidate data. The target data mapping relationship is determined based on the first data mapping relationship and the second data mapping relationship.

4. The method according to claim 3, characterized in that, The data type includes a numeric type. The step of determining a second data mapping relationship based on the data type, the intermediate data, and the candidate data includes: If the data type is a numeric type, the reference numeric error is obtained by querying the preset numeric error mapping relationship based on the intermediate field identifier of the intermediate data. A first reference value and a second reference value are obtained by calculating based on the reference value error and the candidate data; wherein the first reference value is less than the second reference value. If the intermediate data is greater than or equal to the first reference value, and the intermediate data is less than or equal to the second reference value, then the second data mapping relationship is determined based on the intermediate data and the candidate data.

5. The method according to claim 3, characterized in that, The data type includes text type. The step of determining a second data mapping relationship based on the data type, the intermediate data, and the candidate data includes: If the data type is text, perform text semantic recognition on the candidate data to obtain candidate semantic features, and perform text semantic recognition on the intermediate data to obtain intermediate semantic features; Calculate the text similarity between the intermediate semantic features and the candidate semantic features, and determine the baseline data from the candidate data based on the text similarity; The second data mapping relationship is determined based on the baseline data and the intermediate data.

6. The method according to any one of claims 1 to 5, characterized in that, The field identifier mapping relationship is obtained based on the following steps: Semantic extraction is performed on the reference field identifier to obtain the reference field semantics, and semantic extraction is performed on the intermediate field identifier to obtain the intermediate field semantics; Calculate the similarity between the semantics of the reference field and the semantics of the intermediate field to obtain a semantic similarity score; The field identifier mapping relationship is determined based on the semantic similarity score.

7. The method according to any one of claims 1 to 5, characterized in that, The process of obtaining the reference table and the target table includes: Get the first and second tables; Sensitive data identification is performed on the first table to obtain the first target data, and sensitive data identification is performed on the second table to obtain the second target data; The reference table is obtained by replacing the first target data in the first table with a preset data mask. The second target data in the second table is replaced according to the data mask to obtain the target table.

8. A table data verification device, characterized in that, The device includes: A table acquisition module is used to acquire a reference table and a target table; wherein, the reference table includes reference data, the reference data is preset with reference field identifiers, and the target table includes intermediate data, the intermediate data is preset with intermediate field identifiers; A field identifier filtering module is used to query a preset field identifier mapping relationship for each intermediate data based on the intermediate field identifier of the intermediate data to obtain a reference field identifier; wherein, the field identifier mapping relationship is used to represent the matching relationship between the reference field identifier and the intermediate field identifier; A data filtering module is used to filter candidate data from the reference data based on the benchmark field identifier; A data mapping relationship generation module is used to generate a target data mapping relationship based on the language type of the candidate data, the language type of the intermediate data, the candidate data, and the intermediate data. A data verification module is used to filter verification data from the reference data according to the target data mapping relationship for each intermediate data, and to perform data verification on the intermediate data based on the verification data.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images