Data processing method and apparatus

Abstract

Embodiments of the present specification provide a data processing method and device, wherein a data processing method comprises: obtaining at least two files to be checked according to file identifiers in a data checking template; the data checking template records checking configuration information determined according to file configuration information corresponding to the files to be checked; performing data extraction on the at least two files to be checked according to the checking configuration information, to obtain checking data of the at least two files to be checked; performing data pairing processing on the checking data based on data attributes corresponding to the checking data; and performing data checking processing on a data pair set obtained through the data pairing processing. The embodiments of the present application can improve data checking efficiency.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to a data processing method and apparatus. Background Technology

[0002] With the continuous development of internet finance, more and more technologies (such as big data, distributed systems, blockchain, and artificial intelligence) are being applied in the financial field, and financial data is growing exponentially. Consumer finance institutions connect with numerous funding and asset providers, and need to regularly verify data, information flow, and capital flow with these providers. At the same time, these institutions also regularly verify internal settlement and payment information, resulting in a very strong demand for data verification. Summary of the Invention

[0003] This application provides a data processing method and apparatus to improve the efficiency of reading files to be verified and verifying data during the data verification process, while also improving the accuracy of data verification.

[0004] In a first aspect, embodiments of this application provide a data processing method, including:

[0005] At least two files to be verified are obtained based on the file identifiers in the data verification template; the data verification template contains verification configuration information determined based on the file configuration information corresponding to the files to be verified.

[0006] Data is extracted from the at least two files to be verified according to the verification configuration information to obtain the verification data of the at least two files to be verified.

[0007] Based on the data attributes corresponding to the data to be verified, the data to be verified is subjected to data pairing processing;

[0008] The data set obtained from the data pairing process is then subjected to data verification processing.

[0009] Secondly, embodiments of this application provide a data processing apparatus, including:

[0010] The file to be verified module is used to obtain at least two files to be verified based on the file identifiers in the data verification template; the data verification template records verification configuration information determined based on the file configuration information corresponding to the files to be verified.

[0011] The data extraction module is used to extract data from the at least two files to be verified according to the verification configuration information, and obtain the data to be verified from the at least two files to be verified.

[0012] The data pairing processing module is used to perform data pairing processing on the data to be cored based on the data attributes corresponding to the data to be cored.

[0013] The data verification processing module is used to perform data verification processing on the data pair set obtained from the data pairing processing.

[0014] Thirdly, embodiments of this application provide a data processing apparatus, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the data processing method described in the first aspect.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the data processing method as described in the first aspect.

[0016] As can be seen, in this embodiment, at least two files to be verified are first obtained according to the file identifier in the data verification template, saving the time required for manual reading of the files to be verified. Then, the files to be verified are extracted according to the verification configuration information in the data verification template to obtain the data to be verified for each file. Then, based on the data attributes corresponding to the data to be verified, the data to be verified for the files to be verified is paired, saving the verification time consumed by manual verification due to the large amount of data, and improving the efficiency of data verification. Finally, the data verification is performed on the data pair set of the files to be verified obtained by the data pairing process. Since manual verification has a certain degree of omission, the data verification based on the data verification template also improves the accuracy of data verification. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A data processing method flowchart provided in this application embodiment;

[0019] Figure 2 A schematic diagram illustrating a data verification process provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a data verification template provided in an embodiment of this application;

[0021] Figure 4 A flowchart illustrating a data processing method for a two-party data verification scenario provided in this application embodiment;

[0022] Figure 5A schematic diagram of a data processing apparatus provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the protection scope of this document.

[0025] In real-world data verification scenarios, the differences between files from different data verification systems are identified by manually comparing their preset formats. This process is time-consuming because it requires manually exporting the files from different data verification systems, and manual file comparison has a high error rate.

[0026] To address this, and in order to save data verification time, improve data verification efficiency, and increase the accuracy of data verification, this embodiment provides a data processing method. This method sets up different data verification tasks and configures a corresponding data verification template for each task. When the execution time for the target verification task arrives, at least two files to be verified are read according to the file identifier in the corresponding data verification template. Based on the verification configuration information recorded in the data verification template, the data to be verified in each file is extracted. The data to be verified in each file is processed into a set of data pairs, and data verification is performed based on the set of data pairs in each file. Specifically, starting from the data verification template, the method realizes the acquisition of files to be verified, the extraction of data to be verified, and the data comparison during the data verification process, thereby improving data verification efficiency and the accuracy of the verification results. Furthermore, corresponding data verification templates are configured for different data verification types or different files to be verified, improving adaptability to different data verification types or different files to be verified.

[0027] Reference Figure 1 It shows a flowchart of a data processing method provided in an embodiment of this application, with reference to Figure 2 It shows a schematic diagram of a data verification process provided in an embodiment of this application, with reference to Figure 3 It shows a schematic diagram of a data verification template provided in an embodiment of this application, with reference to Figure 4 The diagram illustrates a data processing method for a two-party data verification scenario provided by an embodiment of this application.

[0028] Reference Figure 1 The data processing method provided in this embodiment is applied to electronic devices and specifically includes steps S102 to S108.

[0029] Step S102: Obtain at least two files to be verified based on the file identifiers in the data verification template.

[0030] The data verification template described in this embodiment includes a template created specifically for data verification tasks; the data verification template records verification configuration information determined based on the file configuration information corresponding to each file to be verified; such as... Figure 3 As shown, the template records information such as the task name, execution time, fund identifier, file identifier, and file source of the data verification task. The file obtained from the data provider using the file identifier recorded in the data verification template is the file to be verified in this embodiment.

[0031] In this embodiment, the configuration process for three data verification templates is provided. The process for each of these three data verification templates will be described in detail below.

[0032] (1) In specific implementation, for the data verification task, a corresponding data verification template is configured to read and parse the file to be verified, that is, data verification processing is performed based on the data verification template to improve data verification efficiency; in order to improve the configuration efficiency of the data verification template, during the configuration of the data verification template, the initial verification template is configured according to the file configuration information of the file to be verified of one of the multiple participants in the data verification task, and the file configuration information of other participants is written into the initial verification template; in an optional implementation provided in this embodiment, the data verification template is configured in the following way:

[0033] Read the data identifier and identifier sequence number from the file configuration information of the first data party, and write the data identifier into the preset verification template according to the identifier sequence number to obtain the initial verification template;

[0034] Determine the mapping sequence number of the data identifier in the file configuration information of the second data party, and write the mapping sequence number into the initial verification template to obtain the intermediate verification template;

[0035] Configure data verification for the data identifier in the intermediate verification template, determine the data attributes corresponding to the data identifier, and obtain the data verification template.

[0036] Specifically, upon obtaining two file identifiers, the information configuration module in the data verification template is split into a baseline section and a correction section. The data identifiers and identifier serial numbers from the file configuration information of the first data party are read, and the data identifiers are written into the identifier items of the baseline section and the correction section according to the identifier serial numbers. Then, the mapping serial numbers of the data identifiers from the file configuration information of the second data party are read, and the mapping serial numbers corresponding to the data identifiers are written into the serial number item of the correction section. Data verification configuration is then performed in the correction section. The data verification configuration includes configuring the data identifiers for primary key attributes and the data identifiers for verification execution attributes.

[0037] It should be noted that the file configuration information provided by the data provider can be the file configuration information of the format sample file submitted before data verification, the file configuration information of the file to be verified during the first data verification, or other information that can indicate the file configuration order. The configuration of the data verification template mainly involves configuring the verification configuration information, which primarily involves configuring the position of the data identifier in the file to be verified, the primary key, and the key value used to perform the data verification.

[0038] like Figure 3 As shown, the user creates a new data verification template for a target verification task, selecting the data verification type as a verification of the fund settlement status between the first and second data parties. The file identifiers of the files to be verified, provided by the first and second data parties, are written into the data verification template. If only two file identifiers are detected, the verification configuration information module is divided into a baseline section and a correction section. Then, the data identifier and identifier sequence number from the first data party's file configuration information are read, and the data identifiers are written into the second-side data identifier field in both the baseline and correction sections according to the identifier sequence number. Finally, the mapping sequence number corresponding to the data identifier in the second data party's file configuration information is read, and the mapping sequence number is written into the second-side sequence number field in the correction section. The baseline section corresponds to the file identifier of the file to be verified by the first data party, and the correction section corresponds to the file identifier of the file to be verified by the second data party.

[0039] In the fund settlement data reconciliation type, the settlement voucher numbers of the first data party and the second data party are the same, but the monthly balances may be different. Therefore, the settlement voucher number in the data reconciliation template is configured as the primary key attribute, and the monthly balance is configured as the reconciliation execution attribute. It should be noted that... Figure 3 The data verification template shown is merely illustrative; other forms of data verification templates can also be configured, but this embodiment does not limit them.

[0040] It should be noted that if there are multiple (more than two) file identifiers entered for the data verification template, the verification configuration information module will be divided into multiple parts. One part is the baseline part, and the other parts are the correction parts that correspond one-to-one with the file identifiers other than those in the baseline part.

[0041] For example, four file identifiers are entered for the data verification template: the first file identifier, the second file identifier, the third file identifier, and the fourth file identifier. The verification configuration information module of the data verification template is divided into a baseline part and three correction parts. The baseline part corresponds to the first file identifier, the first correction part corresponds to the second file identifier, the second correction part corresponds to the third file identifier, and the third correction part corresponds to the fourth file identifier. To ensure the effectiveness of the data verification, data verification configuration can be performed in one of the three correction parts. Alternatively, data verification configuration can be performed in each correction part, but the data verification configuration for each correction part must be consistent.

[0042] (2) In one optional implementation of this embodiment, the data verification template is further configured in the following manner:

[0043] Read the data identifier configured in the initial verification template corresponding to the target verification task;

[0044] Determine the first-party mapping sequence number corresponding to the data identifier in the file configuration information of the first data party, and the second-party mapping sequence number corresponding to the data identifier in the file configuration information of the second data party;

[0045] Based on the data identifier, the first party mapping number and the second party mapping number are written into the initial verification template to obtain the intermediate verification template;

[0046] Configure data verification for the data identifier in the intermediate verification template, determine the data attributes corresponding to the data identifier, and obtain the data verification template.

[0047] Specifically, firstly, based on the task type of the target verification task selected by the user, the corresponding initial verification template is loaded and the data identifiers configured in the initial verification template are read. Then, the mapping sequence number corresponding to the data identifier in the file configuration information of at least two data parties is determined, and the mapping sequence number is written into the corresponding sequence number item in the initial verification template to obtain the intermediate verification template. Next, the data identifiers in the intermediate verification template are configured for data verification, and the data attributes corresponding to each data identifier are determined to obtain the data verification template. It should be noted that here, the target verification task corresponds to one data verification template, which records the configuration information of at least two files to be verified participating in the data verification.

[0048] (3) Since the data parties involved in the data verification are different, in order to enhance the trust level of the data parties and ensure the data security of different data parties, multiple data verification templates are configured for multiple data parties when configuring the data verification template. Optionally, the data verification template and the file to be verified have a one-to-one correspondence. The file to be verified obtained according to the file identifier in the data verification template corresponds to the data verification template. That is, one file identifier corresponds to one data verification template, and the file to be verified read according to the data verification template has a one-to-one correspondence with the data verification template.

[0049] In response to this, in an optional implementation of this embodiment, if the target verification task corresponds to two file identifiers, the following steps are performed during the configuration of the data verification template for the target verification task:

[0050] Load the first initial verification template and the second initial verification template; the first initial verification template and the second initial verification template are templates that correspond to the task type of the target verification task and have the same configuration information;

[0051] The first initial verification template is updated to the first intermediate verification template based on the file configuration information of the first data party; and the second initial verification template is updated to the second intermediate verification template based on the file configuration information of the second data party.

[0052] Data verification configuration is performed on the first intermediate verification template and the second intermediate verification template to obtain the first verification template and the second verification template;

[0053] The first verification template and the second verification template are determined as the data verification templates corresponding to the target verification task.

[0054] Furthermore, in this embodiment, during the process of updating the first initial verification template to a first intermediate verification template based on the file configuration information of the first data party, the first party mapping sequence number of the data identifier in the first initial verification template in the file configuration information of the first data party is first determined; then, the first party mapping sequence number is written into the first initial verification template to obtain the first intermediate verification template. During the process of updating the second initial verification template to a second intermediate verification template based on the file configuration information of the second data party, the second party mapping sequence number of the data identifier in the second initial verification template in the file configuration information of the second data party is first determined; then, the second party mapping sequence number is written into the second initial verification template to obtain the second intermediate verification template.

[0055] Specifically, firstly, the initial verification templates corresponding to the two target verification tasks are read, and the configuration information in these two initial verification templates is consistent. Then, the identifier sequence numbers corresponding to the data identifiers in the file configuration information of the first data party and the file configuration information of the second data party are determined, and these identifier sequence numbers are written into the two initial verification templates to obtain two intermediate verification templates. Next, consistent data verification configurations are applied to the two intermediate verification templates to obtain data verification templates corresponding to the file identifiers of the two files to be verified. It should be noted that the number of data verification templates corresponds to the number of data parties involved in the target verification task, and the data verification templates corresponding to the target verification task are consistent with each other except for information related to the data parties (such as file identifier, identifier sequence number, and file source).

[0056] To ensure efficient reading of data verification templates and improve storage quality, after creating the data verification templates corresponding to the target verification task, the data verification templates are stored in the database so that the corresponding data verification templates can be read from the database when the target verification task is executed subsequently.

[0057] It should be noted that the above three processes for configuring data verification templates can also be configured through user interaction. Taking the third configuration method as an example, the user determines the mapping sequence number corresponding to the data identifier in the first data party's file configuration information and the mapping sequence number corresponding to the data identifier in the second data party's file configuration information by viewing the file configuration information of the first data party and the file configuration information of the second data party, and enters them into their respective data verification templates. The user then configures the primary key attribute and verification execution attribute for the data identifier in the two data verification templates.

[0058] In practice, different data verification templates are configured for different data verification tasks, and the data verification tasks are placed in a task queue. In order to ensure file consistency, the task queue is managed in a distributed manner. Therefore, when executing the target verification task, it is first determined whether there is a data verification task using a distributed lock in the data verification queue. If not, the target verification task is executed.

[0059] In one optional implementation of this embodiment, the process first checks whether the data verification time for the target verification task has arrived. If the data verification time for the target verification task has arrived, the process queries whether there is a data verification task using a distributed lock in the task queue to which the target verification task belongs. The data verification template corresponds to the target verification task. If yes, the process returns to the step of checking whether the data verification time for the target verification task has arrived. If no, a distributed lock is created for the target verification task, and the data verification template is read from the database.

[0060] After reading the data verification template, at least two files to be verified are obtained according to the file identifiers in the data verification template. It should be noted that if a data verification template corresponding to the target verification task contains several file identifiers, then the corresponding number of files to be verified are obtained; or, if the target verification task corresponds to several data verification templates, then the files to be verified that have a one-to-one correspondence with each data verification template are read according to the file identifiers in each data verification template.

[0061] To ensure the authenticity and validity of the files to be verified and to avoid data verification errors caused by reading other files with the same file identifier as the files to be verified, this embodiment obtains the corresponding files to be verified from the data gateway provided by the data provider. This ensures the validity of the file source of the files to be verified and further guarantees the validity of the files to be verified. In one optional implementation of this embodiment, taking two files to be verified as an example, the process of obtaining at least two files to be verified based on the file identifier in the data verification template is explained as follows:

[0062] Based on the file identifier of the first file to be verified in the data verification template, the first file to be verified is obtained from the data gateway of the first data party to which the first file to be verified belongs; and based on the file identifier of the second file to be verified in the data verification template, the second file to be verified is obtained from the data gateway of the second data party to which the second file to be verified belongs.

[0063] In practical applications, to ensure data privacy, the data provider does not return files to all received requests, and the returned files vary depending on the request parameters. Furthermore, to improve the scalability and compatibility of data verification processing, in one optional implementation of this embodiment, the following steps are performed during the process of obtaining the first file to be verified from the data gateway of the first data provider to which the first file to be verified belongs, based on the file identifier of the first file to be verified in the data verification template:

[0064] Based on the file identifier of the first file to be verified, the data party identifier of the first data party is determined;

[0065] Based on the data party identifier, query the first gateway protocol signed by the first data party;

[0066] Send a first file request to the data gateway of the first data party. The first file request is generated based on the first protocol parameters recorded in the first gateway protocol.

[0067] Obtain the first file to be verified returned by the data gateway of the first data party in response to the first file request.

[0068] Accordingly, in the process of obtaining the second file to be verified from the data gateway of the second data party to which the second file to be verified belongs, based on the file identifier of the second file to be verified in the data verification template, firstly, the data party identifier of the second data party is determined based on the file identifier of the second file to be verified, then, according to the data party identifier, the second gateway protocol signed by the second data party is queried, then a second file request generated based on the second protocol parameters recorded in the second gateway protocol is sent to the data gateway of the second data party, and finally, the second file to be verified returned by the data gateway of the second data party in response to the second file request is obtained.

[0069] Specifically, firstly, based on the file identifier of the first file to be verified in the data verification template, the data party identifier of the first data party is determined, and based on the file identifier of the second file to be verified in the data verification template, the data party identifier of the second data party is determined. Then, based on the data party identifier of the first data party, the first gateway agreement signed by the first data party is queried, and based on the data party identifier of the second data party, the second gateway agreement signed by the second data party is queried. Next, a first file request is sent to the data gateway of the first data party, and a second file request is sent to the data gateway of the second data party. The first file request is generated based on the first protocol parameters recorded in the first gateway agreement, and the second file request is generated based on the second protocol parameters recorded in the second gateway agreement. Finally, the first file to be verified returned by the data gateway of the first data party in response to the first file request and the second file to be verified returned by the data gateway of the second data party in response to the second file request are obtained.

[0070] In addition, the corresponding first gateway protocol can be queried based on the file identifier of the first file to be verified in the data verification template, and the corresponding second gateway protocol can be queried based on the file identifier of the second file to be verified in the data verification template.

[0071] For example, the system queries the corresponding first gateway protocol based on the first-side file identifier in the data verification template, and queries the corresponding second gateway protocol based on the file identifier of the second file to be verified in the data verification template. When the system finds that the first gateway protocol requires sending a file identifier to the first-side data gateway when reading the file to be verified needed for the target verification task, and the second gateway protocol requires sending a file identifier to the second-side data gateway when reading the file to be verified needed for the target verification task, the system generates a file request containing the first-side file identifier and sends it to the first-side data gateway, and generates a file request containing the second-side file identifier and sends it to the second-side data gateway. The system then obtains the first file to be verified returned by the first-side data gateway and the second file to be verified returned by the second-side data gateway.

[0072] Step S104: Extract data from the at least two files to be verified according to the verification configuration information to obtain the verification data of the at least two files to be verified.

[0073] The data to be verified includes the data corresponding to the primary key identifier and the data corresponding to the second data identifier configured in the data verification template contained in the file to be verified.

[0074] After obtaining at least two files to be verified, data is extracted from each file according to the verification configuration information in the data verification template. During the data extraction process, the file content of each line in the file to be verified is read first. For each line of file content read, the data to be verified in that line of file content is read out according to the first data identifier and the second data identifier configured in the data verification template.

[0075] In practice, to improve data verification efficiency and avoid wasting a lot of time comparing all the data in the file to be verified, after obtaining the file to be verified, the data to be verified in the file is read out according to the data verification template, and then the read data to be verified is compared. To improve the efficiency of extracting the data to be verified, the verification configuration information recorded in the data verification template includes a first data identifier whose data attribute is the primary key attribute, and a second data identifier whose data attribute is the verification execution attribute. In an optional implementation provided in this embodiment, the data to be verified in the file to be verified is read in the following manner:

[0076] Read the first data identifier's first mapping sequence number in the at least two kernel files, and the second data identifier's second mapping sequence number in the at least two kernel files;

[0077] Based on the first mapping sequence number, first kernel data is read from the at least two kernel files; and second kernel data is read from the at least two kernel files based on the second mapping sequence number.

[0078] Specifically, for each of the at least two files to be verified, the first mapping sequence number corresponding to the first data identifier with the primary key attribute in the data verification template, and the second mapping sequence number corresponding to the second data identifier with the verification execution attribute, are read. Based on the first mapping sequence number and the second mapping sequence number, the first and second data to be verified are read in the file to be verified.

[0079] Continue with Figure 3Taking the data verification template shown as an example, the settlement voucher number of the primary key attribute is number 1 in the first file to be verified and number 2 in the second file to be verified. The current month balance of the verification execution attribute is number 5 in both the first and second files to be verified. Then, the data in the first and fifth columns of each row of the first file to be verified is read as the data to be verified in the first file to be verified. The data in the second and fifth columns of each row of the second file to be verified is read as the data to be verified in the second file to be verified.

[0080] Step S106: Based on the data attributes corresponding to the data to be cored, perform data pairing processing on the data to be cored.

[0081] The data attributes include primary key attributes and verification execution attributes; the data pairing process includes pairing the data corresponding to the primary key attributes and verification execution attributes to obtain key-value pairs with the data corresponding to the primary key attributes as the primary key and the data corresponding to the verification execution attributes as the key value.

[0082] In practical implementation, to improve the convenience and efficiency of data verification processing, during the reading of the file to be verified, the data corresponding to the primary key attribute in that row is used as the primary key, and the data corresponding to the verification execution attribute is used as the key value to generate key-value pairs for the file to be verified. In an optional implementation provided in this embodiment, taking two files to be verified as an example, the process of pairing the data based on the data attributes corresponding to the data to be verified is implemented in the following way:

[0083] A first set of key-value pairs is obtained by pairing the first and second data extracted from the first file to be verified; and a second set of key-value pairs is obtained by pairing the first and second data extracted from the second file to be verified.

[0084] Following the previous example, after reading the data in the first and fifth columns of each row of the first file to be verified, and the data in the second and fifth columns of each row of the second file to be verified, key-value pairs are created for each row, using the data in the first row of the first file as the primary key and the data in the fifth column as the key value. Based on the created key-value pairs, a first set of key-value pairs for the first file to be verified is generated. Similarly, key-value pairs are created for each row of the second file to be verified, using the data in the second row of the second file as the primary key and the data in the fifth column as the key value. Based on the created key-value pairs, a second set of key-value pairs for the second file to be verified is generated.

[0085] Step S108: Perform data verification processing on the data set obtained from the data pairing process.

[0086] In step S108, the data pair set is obtained by performing data pairing processing on the data to be kerneled. The data pair set includes at least one of the intersection set, one-sided set, and two-sided difference set mentioned below.

[0087] In practice, during the data verification process, it is necessary to identify inconsistencies among the participants in the data verification. To improve the efficiency of data verification processing, avoid oversights caused by manual comparison, and enhance the accuracy of data verification, this embodiment provides an optional implementation method. Taking two files to be verified as an example, after obtaining the first key-value pair set and the second key-value pair set, the data verification process is performed in the following manner:

[0088] The intersection of the first set of key-value pairs and the second set of key-value pairs is obtained.

[0089] Calculate the difference between the first key-value pair set and / or the second key-value pair set and the intersection set to obtain a one-sided set;

[0090] The intersection set consists of key-value pairs with the same primary key and the same key value in the first key-value pair set and the second key-value pair set.

[0091] In addition, in an optional implementation of this embodiment, to improve the comprehensiveness of the data verification results, the following steps are also performed during the data verification process:

[0092] Identify the primary key data of key-value pairs in the first and second key-value pair sets that have the same primary key but different key values;

[0093] Read the first key-value data and the second key-value data corresponding to the primary key data from the first key-value pair set and the second key-value pair set;

[0094] Based on the primary key data, the first key value data, and the second key value data, construct differential key-value pairs to obtain a bilateral differential set consisting of at least one differential key-value pair.

[0095] Specifically, after obtaining the first key-value pair set and the second key-value pair set, firstly, the intersection set of the first key-value pair set and the second key-value pair set is taken. The intersection set contains key-value pairs in the first key-value pair set and the second key-value pair set that have the same primary key and the same key value. Then, the difference set between the first key-value pair set and the intersection set is calculated to obtain the first one-sided set. The first one-sided set contains key-value pairs that exist in the first key-value pair set but not in the second key-value pair set. And the difference set between the second key-value pair set and the intersection set is calculated to obtain the second one-sided set. The second one-sided set contains key-value pairs that do not exist in the first key-value pair set but exist in the second key-value pair set.

[0096] Simultaneously, the primary key data of key-value pairs in the first key-value pair set and the second key-value pair set that have the same primary key but different key values ​​are identified. The first key value data corresponding to the primary key data is read from the first key-value pair set, and the second key value data corresponding to the primary key data is read from the second key-value pair set. A difference key-value pair is created with the primary key data as the primary key and the first key value data and the second key value data as the key values. A two-sided difference set is obtained based on the difference key-value pair.

[0097] For example, after obtaining the first key-value pair set and the second key-value pair set, the set operation method in the utility class is used. First, the intersection of the first key-value pair set and the second key-value pair set is taken to obtain the intersection set composed of key-value pairs that are consistent on both sides. Then, the bilateral difference data in the first key-value pair set and the second key-value pair set that have the same primary key but inconsistent key values ​​are identified. Then, the difference set of the first key-value pair set and the intersection set is performed to obtain the first one-sided set. Finally, the difference set of the second key-value pair set and the intersection set is performed to obtain the second one-sided set.

[0098] After obtaining the intersection set, the bilateral difference set, the first unilateral set, and the second unilateral set, store the intersection set, the bilateral difference set, the first unilateral set, and the second unilateral set into the database, and improve storage efficiency by batch inserting into the database.

[0099] In practice, the intersection set, bilateral difference set, first unilateral set, and second unilateral set obtained from the data verification process can be viewed subsequently. To improve the user's understanding of the intersection set, bilateral difference set, first unilateral set, and second unilateral set, and to improve the efficiency of the user in viewing the data verification results, in an optional implementation of this embodiment, after obtaining the intersection set, bilateral difference set, first unilateral set, and second unilateral set, the following operations are also performed:

[0100] Store the intersection set, one-sided set, and two-sided difference set obtained from data verification processing into the database;

[0101] Configure the access interfaces corresponding to the intersection set, the one-sided set, and the two-sided difference set in the data verification template.

[0102] Specifically, the intersection set, bilateral difference set, first unilateral set, and second unilateral set are stored in the database for later viewing. In order to improve the user's awareness of the intersection set, bilateral difference set, first unilateral set, and second unilateral set, access interfaces corresponding to the intersection set, bilateral difference set, first unilateral set, and second unilateral set are configured in the data verification template so that users can view the obtained sets through these access interfaces.

[0103] In addition, to improve the integrity of the files to be verified, data can be supplemented by writing the data corresponding to the key-value pairs in the second one-sided set into the first file to be verified, and writing the data corresponding to the key-value pairs in the first one-sided set into the second file to be verified. This will improve the integrity of the first and second files to be verified. The first and second supplementary files obtained after data supplementation are also stored in the database for later viewing of the data supplementation results.

[0104] The following description uses the application of a data processing method provided in this embodiment in a two-party data verification scenario as an example to further illustrate the data processing method provided in this embodiment. (See also...) Figure 4 The data processing method applied to the scenario of two-party data verification specifically includes steps S402 to S420.

[0105] Step S402: Detect the data verification time of the target verification task.

[0106] Step S404: If the data verification time arrives, query the task list to which the target verification task belongs to whether there is a data verification task using a distributed lock;

[0107] If so, then no action will be taken;

[0108] If not, proceed to steps S406 to S420.

[0109] Step S406: Create a distributed lock for the target verification task and read the data verification template corresponding to the target verification task from the database.

[0110] Step S408: Read the left-side file to be verified from the left data gateway based on the left-side file identifier recorded in the data verification template, and read the right-side file to be verified from the right data gateway based on the right-side file identifier recorded in the data verification template.

[0111] Step S410: Extract the left-side data to be verified from the left-side file to be verified based on the left-side configuration information configured in the data verification template, and extract the right-side data to be verified from the right-side file to be verified based on the right-side configuration information configured in the data verification template.

[0112] The left-side configuration information includes the mapping sequence number of the first data identifier configured as the primary key attribute in the left-side file to be verified, and the mapping sequence number of the second data identifier configured as the verification execution attribute in the left-side file; the right-side configuration information includes the mapping sequence number of the first data identifier configured as the primary key attribute in the right-side file to be verified, and the mapping sequence number of the second data identifier configured as the verification execution attribute in the right-side file.

[0113] Step S412: Create a left-side key-value pair set according to the data attributes of the data to be verified on the left, and create a right-side key-value pair set according to the data attributes of the data to be verified on the right.

[0114] Specifically, in the process of creating the left key-value pair set, the first data to be checked for the primary key attribute and the second data to be checked for the execution attribute in each row of the file to be checked on the left are constructed into a key-value pair, thus obtaining the left key-value pair set composed of the key-value pairs in each row; in the process of creating the right key-value pair set, the first data to be checked for the primary key attribute and the second data to be checked for the execution attribute in each row of the file to be checked on the right are constructed into a key-value pair, thus obtaining the right key-value pair set composed of the key-value pairs in each row.

[0115] Step S414: Determine the intersection set of key-value pairs with the same primary key and the same key value in the left key-value pair set and the right key-value pair set.

[0116] Step S416: Determine the bilateral difference set consisting of key-value pairs with the same primary key but different key values ​​in the left key-value pair set and the right key-value pair set.

[0117] Step S418: Calculate the difference between the left key-value pair set and the intersection set to obtain the left one-sided set; and calculate the difference between the right key-value pair set and the intersection set to obtain the right one-sided set.

[0118] Step S420: Store the intersection set, bilateral difference set, left-side unilateral set, and right-side unilateral set into the database.

[0119] The following is an embodiment of a data processing device provided in this specification:

[0120] In the above embodiments, a data processing method is provided, and correspondingly, a data processing device is also provided, which will be described below with reference to the accompanying drawings.

[0121] Reference Figure 5 The diagram shows a data processing apparatus provided in this embodiment.

[0122] Since the apparatus embodiments correspond to the method embodiments, the descriptions are relatively simple. For relevant parts, please refer to the corresponding descriptions of the method embodiments provided above. The apparatus embodiments described below are merely illustrative.

[0123] This embodiment provides a data processing apparatus, including:

[0124] The file to be verified module 502 is used to obtain at least two files to be verified according to the file identifier in the data verification template; the data verification template records verification configuration information determined according to the file configuration information corresponding to the files to be verified.

[0125] The data extraction module 504 is used to extract data from the at least two files to be verified according to the verification configuration information, and obtain the data to be verified from the at least two files to be verified.

[0126] The data pairing processing module 506 is used to perform data pairing processing on the data to be cored based on the data attributes corresponding to the data to be cored.

[0127] The data verification processing module 508 is used to perform data verification processing on the data pair set obtained from the data pairing processing.

[0128] The following is an embodiment of a data processing device provided in this specification:

[0129] Corresponding to the data processing method described above, based on the same technical concept, this application also provides a data processing device for executing the data processing method provided above. Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application.

[0130] This embodiment provides a data processing device, including:

[0131] like Figure 6 As shown, data processing devices can vary significantly due to differences in configuration or performance. They may include one or more processors 601 and memory 602, with memory 602 storing one or more application programs or data. Memory 602 can be temporary or persistent storage. The application programs stored in memory 602 may include one or more modules (not shown), each module including a series of computer-executable instructions from the data processing device. Furthermore, processor 601 may be configured to communicate with memory 602, executing the series of computer-executable instructions stored in memory 602 on the data processing device. The data processing device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input / output interfaces 605, one or more keyboards 606, etc.

[0132] In one specific embodiment, the data processing device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the data processing device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0133] At least two files to be verified are obtained based on the file identifiers in the data verification template; the data verification template contains verification configuration information determined based on the file configuration information corresponding to the files to be verified.

[0134] Data is extracted from the at least two files to be verified according to the verification configuration information to obtain the verification data of the at least two files to be verified.

[0135] Based on the data attributes corresponding to the data to be verified, the data to be verified is subjected to data pairing processing;

[0136] The data set obtained from the data pairing process is then subjected to data verification processing.

[0137] This specification provides an embodiment of a computer-readable storage medium as follows:

[0138] Corresponding to the data processing method described above, and based on the same technical concept, embodiments of this application also provide a computer-readable storage medium.

[0139] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed by a processor, implement the following process:

[0140] At least two files to be verified are obtained based on the file identifiers in the data verification template; the data verification template contains verification configuration information determined based on the file configuration information corresponding to the files to be verified.

[0141] Data is extracted from the at least two files to be verified according to the verification configuration information to obtain the verification data of the at least two files to be verified.

[0142] Based on the data attributes corresponding to the data to be verified, the data to be verified is subjected to data pairing processing;

[0143] The data set obtained from the data pairing process is then subjected to data verification processing.

[0144] It should be noted that the embodiments of computer-readable storage media in this specification and the embodiments of data processing methods in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0145] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0146] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, this specification can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0152] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The embodiments of this application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0155] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0156] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A data processing method, characterized in that, The method includes: At least two files to be verified are obtained based on the file identifiers in the data verification template; the data verification template contains verification configuration information determined based on the file configuration information corresponding to the files to be verified. Data is extracted from the at least two files to be verified according to the verification configuration information to obtain the verification data of the at least two files to be verified; the verification configuration information includes: a first data identifier whose data attribute is a primary key attribute and a second data identifier whose data attribute is a verification execution attribute; the verification data is obtained by: reading the verification data of the primary key attribute in the at least two files to be verified based on the first data identifier, and reading the verification data of the verification execution attribute in the at least two files to be verified based on the second data identifier; Based on the data attributes corresponding to the data to be verified, data pairing processing is performed on the data to be verified; the data pairing processing includes pairing the data corresponding to the primary key attribute and the verification execution attribute to obtain key-value pairs with the data corresponding to the primary key attribute as the primary key and the data corresponding to the verification execution attribute as the key value. The data pair set obtained from the data pairing process is subjected to data verification processing, wherein the data pair set includes at least one of the following: intersection set, one-sided set, and two-sided difference set.

2. The method according to claim 1, characterized in that, The step of extracting data from the at least two files to be verified according to the verification configuration information to obtain the verification data of the at least two files to be verified includes: Read the first data identifier's first mapping sequence number in the at least two kernel files, and the second data identifier's second mapping sequence number in the at least two kernel files; Based on the first mapping sequence number, first kernel data is read from the at least two kernel files; and second kernel data is read from the at least two kernel files based on the second mapping sequence number.

3. The method according to claim 1, characterized in that, The data verification template is configured as follows: Read the data identifier and identifier sequence number from the file configuration information of the first data party, and write the data identifier into the preset verification template according to the identifier sequence number to obtain the initial verification template; Determine the mapping sequence number of the data identifier in the file configuration information of the second data party, and write the mapping sequence number into the initial verification template to obtain the intermediate verification template; Configure data verification for the data identifier in the intermediate verification template, determine the data attributes corresponding to the data identifier, and obtain the data verification template.

4. The method according to claim 1, characterized in that, The step of obtaining at least two files to be verified based on the file identifiers in the data verification template includes: Based on the file identifier of the first file to be verified in the data verification template, the first file to be verified is obtained from the data gateway of the first data party to which the first file to be verified belongs; and based on the file identifier of the second file to be verified in the data verification template, the second file to be verified is obtained from the data gateway of the second data party to which the second file to be verified belongs.

5. The method according to claim 4, characterized in that, The step of obtaining the first file to be verified from the data gateway of the first data party to which the first file to be verified belongs, based on the file identifier of the first file to be verified in the data verification template, includes: Based on the file identifier of the first file to be verified, the data party identifier of the first data party is determined; Based on the data party identifier, query the first gateway protocol signed by the first data party; Send a first file request to the data gateway of the first data party. The first file request is generated based on the first protocol parameters recorded in the first gateway protocol. Obtain the first file to be verified returned by the data gateway of the first data party in response to the first file request.

6. The method according to claim 5, characterized in that, The step of performing data pairing processing on the data to be verified based on the data attributes corresponding to the data to be verified includes: A first set of key-value pairs is obtained by pairing the first and second data extracted from the first file to be verified; and a second set of key-value pairs is obtained by pairing the first and second data extracted from the second file to be verified.

7. The method according to claim 5, characterized in that, The data verification process for the data set obtained from the data pairing process includes: The intersection of the first set of key-value pairs and the second set of key-value pairs is obtained. Calculate the difference between the first key-value pair set and / or the second key-value pair set and the intersection set to obtain a one-sided set.

8. The method according to claim 5, characterized in that, The method further includes: Identify the primary key data of key-value pairs in the first and second key-value pair sets that have the same primary key but different key values; Read the first key-value data and the second key-value data corresponding to the primary key data from the first key-value pair set and the second key-value pair set; Based on the primary key data, the first key value data, and the second key value data, construct differential key-value pairs to obtain a bilateral differential set consisting of at least one differential key-value pair.

9. The method according to claim 1, characterized in that, The method further includes: When the data verification time for the target verification task arrives, query whether there is a data verification task using a distributed lock in the task queue to which the target verification task belongs; the data verification template corresponds to the target verification task; If not, create a distributed lock for the target verification task and read the data verification template from the database.

10. The method according to claim 1, characterized in that, The data verification template is created in the following manner: Read the data identifier configured in the initial verification template corresponding to the target verification task; Determine the first-party mapping sequence number corresponding to the data identifier in the file configuration information of the first data party, and the second-party mapping sequence number corresponding to the data identifier in the file configuration information of the second data party; Based on the data identifier, the first party mapping number and the second party mapping number are written into the initial verification template to obtain the intermediate verification template; Configure data verification for the data identifier in the intermediate verification template, determine the data attributes corresponding to the data identifier, and obtain the data verification template.

11. The method according to claim 1, characterized in that, The data verification template and the file to be verified have a one-to-one correspondence. The file to be verified obtained according to the file identifier in the data verification template corresponds to the data verification template.

12. The method according to claim 11, characterized in that, The data verification template is configured as follows: Load the first initial verification template and the second initial verification template; the first initial verification template and the second initial verification template are templates that correspond to the task type of the target verification task and have the same configuration information; The first initial verification template is updated to the first intermediate verification template based on the file configuration information of the first data party; And, based on the file configuration information of the second data party, update the second initial verification template to the second intermediate verification template; Data verification configuration is performed on the first intermediate verification template and the second intermediate verification template to obtain the first verification template and the second verification template; The first verification template and the second verification template are determined as the data verification templates corresponding to the target verification task.

13. The method according to claim 12, characterized in that, The step of updating the first initial verification template to a first intermediate verification template based on the file configuration information of the first data party includes: Determine the first-party mapping sequence number of the data identifier in the first initial verification template in the first data party's file configuration information; Write the first party mapping sequence number into the first initial verification template to obtain the first intermediate verification template.

14. The method according to claim 1, characterized in that, After performing data verification processing on the data set obtained from the data pairing process, the method further includes: Store the intersection set, one-sided set, and two-sided difference set obtained from data verification processing into the database; Configure the access interfaces corresponding to the intersection set, the one-sided set, and the two-sided difference set in the data verification template.

15. A data processing apparatus, characterized in that, The device includes: The file to be verified module is used to obtain at least two files to be verified based on the file identifiers in the data verification template; the data verification template records verification configuration information determined based on the file configuration information corresponding to the files to be verified. The data extraction module is used to extract data from the at least two files to be verified according to the verification configuration information to obtain the data to be verified from the at least two files to be verified. The verification configuration information includes a first data identifier whose data attribute is a primary key attribute and a second data identifier whose data attribute is a verification execution attribute. The data to be verified is obtained by reading the data to be verified based on the primary key attribute in the at least two files to be verified based on the first data identifier, and reading the data to be verified based on the verification execution attribute in the at least two files to be verified based on the second data identifier. The data pairing processing module is used to perform data pairing processing on the data to be verified based on the data attributes corresponding to the data to be verified; the data pairing processing includes pairing the data corresponding to the primary key attribute and the verification execution attribute to obtain key-value pairs with the data corresponding to the primary key attribute as the primary key and the data corresponding to the verification execution attribute as the key value. The data verification processing module is used to perform data verification processing on the data pair set obtained from the data pairing processing. The data pair set includes at least one of the intersection set, the one-sided set, and the two-sided difference set.

16. A data processing device, characterized in that, The device includes: A processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the data processing method as described in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store computer-executable instructions that, when executed by a processor, implement the data processing method as described in any one of claims 1-14.

Images