Data processing method, device, equipment, medium and computer program product

Abstract

The present application discloses a data processing method, apparatus, device, medium and computer program product. The method includes: obtaining a first data set and a second data set, and splitting the first data set and the second data set into M first data subsets and M second data subsets respectively according to a preset data identifier; obtaining a first total hash value of each first data subset and a second total hash value of each second data subset; corresponding to each preset data identifier, searching for a first data subset and a second data subset with different first total hash values ​​and second total hash values; when the amount of business data included in the first data subset or the second data subset is greater than a preset amount, returning to perform iterative calculation until the amount of data is less than or equal to the preset amount, and obtaining inconsistent business data. According to an embodiment of the present application, inconsistent business data in different business systems can be determined in a timely manner, effectively improving the efficiency of data comparison.

Description

Technical Field

[0001] The present application relates to the field of data analysis, and in particular to a data processing method, apparatus, device, medium, and computer program product. Background Art

[0002] With the development of electronic information technology, the supporting services provided by operators in the communications industry are becoming increasingly diverse, and the interactions between the business systems supporting these services are becoming increasingly complex. For example, in application scenarios where multiple business systems interact with each other, the business processes are long, there is information overlap between systems, the interface connection is a mesh network, and the business processes have both synchronous and asynchronous modes. Therefore, application scenarios where multiple business systems interact with each other can easily lead to inconsistent data generated by different business terminals.

[0003] Currently, data comparison solutions typically involve performing consistency checks on existing data within business systems. This involves regularly aggregating data into a centralized system, performing calculations based on agreed-upon data comparison rules, and providing the results to maintenance and business personnel for processing. Due to the sheer volume of data, each comparison requires extensive data comparison, consuming excessive computing and storage resources, resulting in lengthy comparison times and low data comparison efficiency. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device, medium, and computer program product, which can promptly determine inconsistent business data in different business systems and effectively improve data comparison efficiency.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a smart contract management platform and includes:

[0006] Acquire a first data set and a second data set, where the first data set includes business data of the same amount as the second data set, and there is inconsistent business data in the first data set and the second data set, and each business data includes a preset data identifier;

[0007] According to the preset data identifier, the first data set and the second data set are respectively split into M first data subsets and M second data subsets, where M is a positive integer;

[0008] Obtaining a first total hash value for each first data subset and a second total hash value for each second data subset;

[0009] For each preset data identifier, searching for a first data subset and a second data subset having different first total hash values ​​and second total hash values;

[0010] When the amount of business data included in the first data subset or the second data subset is greater than a preset number, the first data subset and the second data subset with different first total hash values ​​are respectively used as the first data set and the second data set, and the step is returned to split the first data set and the second data set into M first data subsets and M second data subsets respectively according to the preset data identifier, until the amount of business data included in the first data subset or the second data subset is less than or equal to the preset number, and a comparison result is obtained, and the comparison result includes inconsistent business data.

[0011] In some implementations of the first aspect, obtaining a first total hash value for each first data subset and a second total hash value for each second data subset includes:

[0012] Obtain a first hash value for each piece of business data in the first data subset and a second hash value for each piece of business data in the second data subset;

[0013] Sum the first hash value to obtain a first total hash value;

[0014] The second hash values ​​are summed to obtain a second total hash value.

[0015] In some implementations of the first aspect, obtaining the first data set and the second data set includes:

[0016] Performing a hash operation on each piece of business data in the first data set to obtain a plurality of first hash values;

[0017] Summing the multiple first hash values ​​to obtain a third total hash value;

[0018] Performing a hash operation on each piece of business data in the second data set to obtain a plurality of second hash values;

[0019] Summing the plurality of second hash values ​​to obtain a fourth total hash value;

[0020] When the third total hash value and the fourth total hash value are different, it is determined that inconsistent business data exists in the first data set and the second data set.

[0021] In some implementations of the first aspect, performing a hash operation on each piece of business data in the first data set to obtain multiple first hash values ​​includes:

[0022] Obtaining first character length information of each business data in the first data;

[0023] Performing character padding on each business data in the first data according to the first character length information of each business data in the first data, and adding the first length identification character of the first character length information to obtain fifth business data, where the character length of the fifth business data is the target length;

[0024] Perform a hash operation on the fifth service data to obtain a first hash value.

[0025] In some implementations of the first aspect, the smart contract management platform includes a database; and before obtaining the first data set and the second data set, the method further includes:

[0026] Respectively receiving first business data sent in real time by the first business system and second business data sent in real time by the second business system, the first business data including a first business information identifier and first operation information corresponding to the first business information, and the second business data including a second business information identifier and second operation information corresponding to the second business information;

[0027] Performing a first operation on the first business information stored in the database according to the first business information identifier and the first operation information corresponding to the first business information identifier to obtain business data included in the first data set;

[0028] According to the second business information identifier and the second operation information corresponding to the second business information identifier, a second operation is performed on the second business information stored in the database to obtain business data included in the second data set.

[0029] In some implementations of the first aspect, the business data included in the first data set and the business data included in the second data set are respectively stored in a database.

[0030] In a second aspect, an embodiment of the present application provides a data processing device, which is applied to a smart contract management platform and includes:

[0031] an acquisition module, configured to acquire a first data set and a second data set, wherein the first data set and the second data set include business data of the same amount, and there is inconsistent business data in the first data set and the second data set, and each business data includes a preset data identifier;

[0032] A processing module, configured to split the first data set and the second data set into M first data subsets and M second data subsets respectively according to a preset data identifier, where M is a positive integer;

[0033] The acquisition module is further configured to acquire a first total hash value of each first data subset and a second total hash value of each second data subset;

[0034] A search module, configured to search, for each preset data identifier, a first data subset and a second data subset having different first and second total hash values;

[0035] The processing module is also used to, when the amount of business data included in the first data subset or the second data subset is greater than a preset amount, use the first data subset and the second data subset with different first total hash values ​​as the first data set and the second data set, respectively, and return to the step to split the first data set and the second data set into M first data subsets and M second data subsets, respectively, according to the preset data identifier, until the amount of business data included in the first data subset or the second data subset is less than or equal to the preset amount, to obtain a comparison result, and the comparison result includes inconsistent business data.

[0036] In a third aspect, the present application provides a data processing device comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the data processing method described in the first aspect or any implementable manner of the first aspect.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the data processing method described in the first aspect or any implementable manner of the first aspect is implemented.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data processing method as described in the first aspect or any implementable embodiment of the first aspect.

[0039] The data processing method, apparatus, device, medium, and computer program product of the embodiments of the present application are applied to a smart contract service platform. For inconsistent business data, a first data set and a second data set with the same data volume of the business data are obtained through the smart contract service platform. The first data set and the second data set are respectively split into M first data subsets and M second data subsets according to preset data identifiers. Then, a first total hash value of each first data subset and a second total hash value of each second data subset are obtained. For each preset data identifier, a first data subset and a second data subset with different first total hash values ​​are searched. When the amount of business data included in the first data subset or the second data subset is greater than a preset amount, the first data subset and the second data subset with different first total hash values ​​are used as the first data set and the second data set, respectively. The first data subset and the second data subset are re-determined until the amount of business data included in the first data subset or the second data subset is less than or equal to the preset amount, thereby obtaining a comparison result, wherein the comparison result includes inconsistent business data. Through the above iterative calculation, the inconsistent business data can be directly located, the data comparison efficiency can be effectively improved, and the cost and difficulty of subsequent business data repair can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 This is a flow chart of a data processing method provided in an embodiment of the present application;

[0042] Figure 2 This is a schematic diagram of the architecture of a data processing flow provided in an embodiment of the present application;

[0043] Figure 3 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0044] Figure 4 It is a structural diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0046] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0047] With the development of electronic information technology, the supporting services provided by operators in the communications industry are becoming increasingly diverse, and the interactions between the business systems supporting these services are becoming increasingly complex. For example, in application scenarios where multiple business systems interact with each other, the business processes are long, there is information overlap between systems, the interface connection is a mesh network, and the business processes have both synchronous and asynchronous modes. Therefore, application scenarios where multiple business systems interact with each other can easily lead to inconsistent data generated by different business terminals.

[0048] If there is inconsistency in the data generated by different business terminals, it may lead to inconsistency between the services provided by the operator and the services subscribed by the user. This will not only affect the development of the operator itself, but also reduce user satisfaction and easily cause user churn.

[0049] Currently, data comparison solutions typically involve performing consistency comparisons on existing data within business systems. This involves regularly aggregating data into a centralized system, performing calculations according to agreed-upon data comparison rules, and providing the output results to maintenance and business personnel for processing. Due to the sheer volume of data, each comparison requires a massive amount of data, consuming excessive computing and storage resources and taking a long time. This results in delayed discovery of discrepancies between business data and inefficient business data comparisons. Furthermore, due to the delay in discovering discrepancies in business data, new problematic data may continue to be generated during the discovery process, further complicating the repair process.

[0050] To this end, the embodiments of the present application provide a data processing method, apparatus, device, medium, and computer program product, which are applied to a smart contract service platform. For inconsistent business data, a first data set and a second data set with the same data volume of the business data are obtained through the smart contract service platform, and the first data set and the second data set are respectively split into M first data subsets and M second data subsets according to a preset data identifier. After that, a first total hash value of each first data subset and a second total hash value of each second data subset are obtained, and corresponding to each preset data identifier, the first data subset and the second data subset with different first total hash values ​​are searched. Subset; when the amount of business data included in the first data subset or the second data subset is greater than a preset number, the first data subset and the second data subset with different first total hash values ​​are used as the first data set and the second data set respectively, and the first data subset and the second data subset are returned to be re-determined until the amount of business data included in the first data subset or the second data subset is less than or equal to the preset number, and a comparison result is obtained, wherein the comparison result includes inconsistent business data. Through the above-mentioned iterative calculation, the inconsistent business data can be directly located, the data comparison efficiency can be effectively improved, and the cost and difficulty of subsequent business data repair can be reduced.

[0051] Figure 1 This is a flow chart of a data processing method provided by an embodiment of the present application, combined with Figure 1 As shown, the method is applied to the smart contract management platform, and the method may include the following steps:

[0052] Step 110: Acquire a first data set and a second data set.

[0053] Specifically, the first data set includes business data with the same data volume as the second data set, and there is inconsistent business data in the first data set and the second data set, and each business data includes a preset data identifier.

[0054] Step 120 : Split the first data set and the second data set into M first data subsets and M second data subsets respectively according to preset data identifiers, where M is a positive integer.

[0055] Step 130: Obtain a first total hash value of each first data subset and a second total hash value of each second data subset.

[0056] Step 140 : corresponding to each preset data identifier, searching for a first data subset and a second data subset having different first total Hash values ​​and second total Hash values.

[0057] Step 150, when the amount of business data included in the first data subset or the second data subset is greater than a preset amount, the first data subset and the second data subset with different first total hash values ​​are used as the first data set and the second data set, respectively, and return to step 120 until the amount of business data included in the first data subset or the second data subset is less than or equal to the preset amount, and a comparison result is obtained, and the comparison result includes inconsistent business data.

[0058] The specific implementation of each of the above steps will be described in detail below.

[0059] According to an embodiment of the present application, for inconsistent business data, a first data set and a second data set with the same data volume of business data are obtained through the smart contract service platform, and the first data set and the second data set are respectively split into M first data subsets and M second data subsets according to a preset data identifier. Then, a first total hash value of each first data subset and a second total hash value of each second data subset are obtained, and corresponding to each preset data identifier, a first data subset and a second data subset with different first total hash values ​​and second total hash values ​​are searched; when the data volume of the business data included in the first data subset or the second data subset is greater than a preset number, the first data subset and the second data subset with different first total hash values ​​and second total hash values ​​are used as the first data set and the second data set, respectively, and the first data subset and the second data subset are returned to be re-determined until the data volume of the business data included in the first data subset or the second data subset is less than or equal to the preset number, and a comparison result is obtained, wherein the comparison result includes inconsistent business data. Through the above-mentioned iterative calculation, the inconsistent business data can be directly located, the data comparison efficiency can be effectively improved, and the cost and difficulty of later business data repair can be reduced.

[0060] The specific implementation methods of the above steps are introduced below.

[0061] In step 110 , a first data set and a second data set are obtained.

[0062] Specifically, the business data in the first data set and the business data in the second data set come from different business systems. For example, the business system may be a supply chain management (SCM) system, a customer relationship management (CRM) system, a decision support system (DSS) system, etc. The business system may also be a business support system (BSS), an operations support system (OSS), a business and operations support system (BOSS), etc. The business system is not specifically limited herein.

[0063] The smart contract service platform can include a blockchain storage unit, allowing business data generated by business systems to be stored on the blockchain. This prevents data tampering and facilitates tracing back to the data source. Because smart contracts are digitally stored on the blockchain and can enforce agreements using encrypted code, they can be implemented as a decentralized blockchain platform, enabling asynchronous autonomy of business data generated by participating business systems. Hash digest data is uploaded to the blockchain, leveraging the distributed ledger nature of the blockchain to ensure that on-chain data cannot be tampered with.

[0064] In order to improve the reliability of the smart contract service platform, it is still necessary to conduct consistency comparison of business data between different business systems so as to timely discover inconsistent data and reduce the cost of repairing business data.

[0065] In some embodiments, the smart contract management platform includes a database. In order to reduce the amount of data processing, before obtaining the first data set and the second data set, the data processing method may also include the following steps: respectively receiving the first business data sent in real time by the first business system and the second business data sent in real time by the second business system, the first business data including the first business information identifier and the first operation information corresponding to the first business information, and the second business data including the second business information identifier and the second operation information corresponding to the second business information; according to the first business information identifier and the first operation information corresponding to the first business information identifier, performing a first operation on the first business information stored in the database to obtain the business data included in the first data set; and, according to the second business information identifier and the second operation information corresponding to the second business information identifier, performing a second operation on the second business information stored in the database to obtain the business data included in the second data set.

[0066] The blockchain unit in the smart contract management platform can serve as the above-mentioned database, in which the historical business data generated by each business system can be stored.

[0067] To reduce the amount of data processing, the smart contract management platform can obtain incremental data from each business system. Specifically, taking the first business system as an example, the smart contract management platform can receive the first business data sent in real time by the first business system. The first business data includes a first business information identifier and first operation information corresponding to the first business information. In this way, the smart contract management platform can find the historical business data of the first business based on the first business information identifier, that is, the first business information in the first business system, and perform corresponding operations on the first business information based on the first operation information, thereby accurately locating the incremental data in the first business system.

[0068] In the embodiment of the present application, the same data processing flow as above can be executed on the second business system, which will not be described in detail here.

[0069] In some embodiments, the business data included in the first data set and the business data included in the second data set are respectively stored in a database.

[0070] As a specific example, take the first business system as a CRM system and the second business system as a BOSS system. Figure 2 The following is a schematic diagram showing an architecture of a data processing flow provided by an embodiment of the present application. Figure 2 As shown, the CRM system and the BOSS system can each receive information about business data operations. The Kafka middleware can obtain this information. This information can include operations such as adding, deleting, or modifying data, i.e., Database Schema Definition Language (DDL) operations. As a specific example, this information can include creating a user, granting permissions, revoking permissions, changing passwords, or deleting users, among other operations. These operations are not listed here.

[0071] For example, a CRM system can include business for group customers and business for family customers. Taking group customers (referred to as "Ji Ke") as an example, the CRM system can include Ji Ke user information and Ji Ke attribute tables. The BOSS system can include CB user tables and CB attribute tables.

[0072] The Kafka middleware creates topics and consumer groups corresponding to DDL operations and provides them to FLINK for data parsing. FLINK is a real-time computing engine for big data. FLINK can parse the fields updated by DDL operations from the topics and consumer groups corresponding to the Kafka middleware DDL operations.

[0073] For example, the business data in the CRM system that FLINK can parse from the DDL operations of the KAFKA middleware can be shown in Tables 1 and 2:

[0074] Table 1

[0075]

[0076] Table 2

[0077]

[0078] The business data in the BOSS system that FLINK can parse from the DDL operations of the KAFKA middleware can be shown in Tables 3 and 4:

[0079] Table 3

[0080]

[0081] Table 4

[0082]

[0083] After obtaining the field information updated by the DDL operation, FLINK can associate it with the existing data in the HBAE database. That is, according to the field information updated by the DDL operation, the historical business data is updated to obtain the updated business data.

[0084] Exemplarily, the updated business data stored in the HBAE database may include a customer acquisition user table, as shown in Table 5.

[0085] Table 5

[0086]

[0087] The updated business data stored in the HBAE database may include a customer acquisition user attribute association table, as shown in Table 6.

[0088] Table 6

[0089]

[0090] The updated business data stored in the HBAE database may include a CB user & attribute association table, as shown in Table 7.

[0091] Table 7

[0092]

[0093] Next, FLINK can integrate the updated data. For example, if the updated business data needs to be linked, multiple tables can be linked as needed. FLINK can also convert the processed data into structured messages and write them to the Kafka message middleware to facilitate subsequent comparative analysis of business data. This allows accurate access to incremental data in each business system. This allows for comparative analysis of newly added business data, improving data comparison efficiency.

[0094] For example, FLINK can integrate the updated data and establish a collection user attribute association table. For example, it can establish an association relationship between USER_ID and PROVINCE_USER_ID, where USER_ID can be associated with the "USER_ID" field in the collection user table (Table 5), and PROVINCE_USER_ID can be associated with the "USER_ID" field in the CB user & attribute association table (Table 7). FLINK can write the above association relationship to the KAFKA message middleware to facilitate the subsequent comparative analysis of business data.

[0095] According to the embodiment of the present application, the incremental data in each business system can be accurately obtained. Therefore, when comparing and analyzing business data, the newly added business data can be compared and analyzed, thereby improving the efficiency of data comparison.

[0096] In some embodiments, obtaining a first data set and a second data set can specifically refer to the following steps: first, performing a hash operation on each business data in the first data set to obtain multiple first hash values; summing the multiple first hash values ​​to obtain a third total hash value; performing a hash operation on each business data in the second data set to obtain multiple second hash values; summing the multiple second hash values ​​to obtain a fourth total hash value; when the third total hash value and the fourth total hash value are different, determining that there is inconsistent business data in the first data set and the second data set.

[0097] Because the same business data will generate the same hash value after hashing, and the first and second data sets contain the same amount of business data, if the business data in the two data sets is identical, a unique hash value can be obtained regardless of whether the business data in the two sets is ordered. This eliminates the need to sort the business data, saving computing resources. In particular, if the third and fourth hash values ​​are identical, it can be directly concluded that the business data in the first and second data sets are consistent, thereby rapidly completing the data comparison process.

[0098] In the embodiment of the present application, when the third total hash value and the fourth total hash value are different, it is determined that inconsistent business data exists in the first data set and the second data set.

[0099] In some embodiments, when performing a hash operation on each piece of business data in the first data set to obtain multiple first hash values, each piece of business data in the first data set may be preprocessed. Specifically, the preprocessing may include the following steps: obtaining first character length information for each piece of business data in the first data set; performing character padding on each piece of business data in the first data based on the first character length information for each piece of business data in the first data, and adding a first length identifier character from the first character length information to obtain fifth business data, where the character length of the fifth business data is the target length. After the above preprocessing is completed, a hash operation is performed on the fifth business data to obtain a first hash value.

[0100] Specifically, the fifth business data obtained after preprocessing of each business data can meet the target length. Exemplarily, the target length is set to meet 512 modulo 0, and a preset length is reserved in the target length for adding the first character length information of the business data. For example, the reserved preset length can be 64 bits. In order to make the fifth business data meet the target length, it can be determined that the remainder of the business data after 512 modulo is 448. Since 448+64=512, if the business data does not meet the remainder of 448 after 512 modulo, the first bit can be padded with 1, and then all bits can be padded with 0 until the length meets the remainder of 448 after 512 modulo.

[0101] According to an embodiment of the present application, by performing preprocessing on the business data in different business systems, the obtained fifth business data can meet the preset structure, so as to improve the reliability of business data comparison in different business systems.

[0102] As a specific example, a hash operation is performed on the fifth business data to obtain a first hash value. Specifically, the hash operation is performed, for example: the fifth business data is first decomposed into 512-bit blocks. Assuming that the fifth business data M can be decomposed into n blocks, then the entire algorithm only needs to complete n iterations. The result of n iterations is the final hash value, that is, the 256-bit first hash value. An initial value H0 of the 256-bit first hash value is obtained after the first iteration operation, H1. H1 is obtained after the second iteration operation, H2, ..., and so on. Finally, Hn is obtained, which is the final 256-bit fifth business data. The mapping performed in each iteration can be represented by $Map(H_{i-1})=H_{i}$.

[0103] In some examples, since the minimum operation unit in the SHA256 algorithm is a "word", and a word is 32 bits, the 256-bit Hi can be described by 8 small blocks.

[0104] In addition, in the first iteration, the initial value of the mapping is set to 8 hash values. The content of each iteration is as follows: STEP 1: Construct 64 words; for each block, decompose the block into 16 32-bit big-endian words, denoted as w[0],…,w

[15] . In other words, the first 16 words are directly obtained by decomposing the i-th block of the fifth business data, and the remaining words are calculated using the iterative formula. STEP 2: Perform 64 loops; mapping $Map(H_{i-1})=H_{i}$ includes 64 loop calculations, that is, 64 loop calculations are performed to complete one iteration.

[0105] According to an embodiment of the present application, based on the same inventive concept, when a hash operation is performed on each business data in the second data set to obtain multiple second hash values, the calculation steps are the same as those of performing a hash operation on each business data in the second data set to obtain the first hash value, and will not be repeated here.

[0106] In some embodiments, the hash value calculated for the business data in each business system can be saved so that it can be directly used when locating inconsistent business data in different business systems, thereby avoiding waste of computing resources.

[0107] In the embodiment of the present application, when the third total hash value and the fourth total hash value are different, it is determined that inconsistent business data exists in the first data set and the second data set.

[0108] In step 120 , according to a preset data identifier, the first data set and the second data set are respectively split into M first data subsets and M second data subsets, where M is a positive integer.

[0109] For example, data sets in two business systems can be cached. When the cached data reaches a target amount, for example, between 10 million and 1 million, the two databases are respectively formed into multiple data subsets based on preset data identifiers, and the data volume of each data subset is consistent.

[0110] Step 130: Obtain a first total hash value of each first data subset and a second total hash value of each second data subset.

[0111] Specifically, the first hash value of each business data in the first data subset and the second hash value of each business data in the second data subset can be obtained; the first hash values ​​are summed to obtain a first total hash value; the second hash values ​​are summed to obtain a second total hash value.

[0112] Among them, the hash value of each business data can be directly obtained from the database or recalculated, and there is no specific limitation.

[0113] Step 140 : corresponding to each preset data identifier, searching for a first data subset and a second data subset having different first total Hash values ​​and second total Hash values.

[0114] Step 150, when the amount of business data included in the first data subset or the second data subset is greater than a preset amount, the first data subset and the second data subset with different first total hash values ​​are used as the first data set and the second data set, respectively, and return to step 120 until the amount of business data included in the first data subset or the second data subset is less than or equal to the preset amount, and a comparison result is obtained, and the comparison result includes inconsistent business data.

[0115] According to an embodiment of the present application, data sets with differences are piled up, that is, the data subsets are divided to obtain several smaller subsets, and then the hash values ​​of the subsets are compared until there is only a single business data in the set. Then, the comparison is performed directly to finally obtain the difference result. The data consistency comparison algorithm based on hash summary improves the comparison efficiency, can quickly locate the difference data, and facilitates data repair.

[0116] According to the embodiment of the present application, the received business data is automatically compared for differences through the smart contract service platform, and data inconsistencies are discovered in real time, solving the resource occupation problem of data aggregation and full comparison of historical data.

[0117] It can be understood that when the business data of three or more business systems need to be compared, all the business systems can be combined in pairs to obtain two different business systems, and then the business data of the two business systems can be compared in combination with the data processing method provided in the embodiment of the present application to find inconsistent business data. The specific data processing method can refer to the data processing method described in the above embodiment of the present application, and will not be repeated here.

[0118] Based on the same inventive concept, the present application also provides a data processing device 300 corresponding to the above data processing method. Figure 3 Provide detailed explanation.

[0119] Figure 3 This is a structural diagram of a data processing device provided in an embodiment of the present application, which is applied to a smart contract management platform, such as Figure 3 As shown, the data processing device 300 may include:

[0120] An acquisition module 310 is configured to acquire a first data set and a second data set, wherein the first data set and the second data set include business data of the same amount, and there is inconsistent business data in the first data set and the second data set, and each business data includes a preset data identifier;

[0121] A processing module 320 is configured to split the first data set and the second data set into M first data subsets and M second data subsets respectively according to a preset data identifier, where M is a positive integer;

[0122] The acquisition module 310 is further configured to acquire a first total hash value of each first data subset and a second total hash value of each second data subset;

[0123] A search module 330 is configured to search, for each preset data identifier, a first data subset and a second data subset having different first and second total hash values;

[0124] The processing module 320 is also used to, when the amount of business data included in the first data subset or the second data subset is greater than a preset amount, use the first data subset and the second data subset with different first total hash values ​​as the first data set and the second data set, respectively, and return to the step to split the first data set and the second data set into M first data subsets and M second data subsets, respectively, according to the preset data identifier, until the amount of business data included in the first data subset or the second data subset is less than or equal to the preset amount, to obtain a comparison result, and the comparison result includes inconsistent business data.

[0125] In some embodiments, the first hash value of the data and the second hash value of each business data in the second subset of data;

[0126] The processing module 320 is further configured to sum the first hash values ​​to obtain a first total hash value;

[0127] The processing module 320 is further configured to sum the second hash values ​​to obtain a second total hash value.

[0128] In some embodiments, the processing module 320 is further configured to perform a hash operation on each business data in the first data set to obtain a plurality of first hash values;

[0129] The processing module 320 is further configured to sum the multiple first hash values ​​to obtain a third total hash value;

[0130] The processing module 320 is further configured to perform a hash operation on each business data in the second data set to obtain a plurality of second hash values;

[0131] The processing module 320 is further configured to sum the multiple second hash values ​​to obtain a fourth total hash value;

[0132] The processing module 320 is further configured to determine that inconsistent business data exists between the first data set and the second data set when the third total hash value and the fourth total hash value are different.

[0133] In some embodiments, the acquisition module 310 is further configured to acquire first character length information of each business data in the first data;

[0134] The processing module 320 is further configured to perform character padding on each business data in the first data according to the first character length information of each business data in the first data, and add the first length identification character of the first character length information to obtain fifth business data, where the character length of the fifth business data is the target length;

[0135] The processing module 320 is further configured to perform a hash operation on the fifth service data to obtain a first hash value.

[0136] In some embodiments, the apparatus further comprises:

[0137] A receiving module, configured to respectively receive first business data sent in real time by the first business system and second business data sent in real time by the second business system, wherein the first business data includes a first business information identifier and first operation information corresponding to the first business information, and the second business data includes a second business information identifier and second operation information corresponding to the second business information;

[0138] The processing module 320 is further configured to perform a first operation on the first business information stored in the database according to the first business information identifier and the first operation information corresponding to the first business information identifier, to obtain business data included in the first data set;

[0139] The processing module 320 is further configured to perform a second operation on the second business information stored in the database according to the second business information identifier and the second operation information corresponding to the second business information identifier, to obtain the business data included in the second data set.

[0140] In some embodiments, the processing module 320 is further configured to store the business data included in the first data set and the business data included in the second data set into a database respectively.

[0141] It is understood that the data processing device 200 of the embodiment of the present application may correspond to the execution subject of the data processing method provided in the embodiment of the present application. The specific details of the operation and / or function of each module / unit of the data processing device 200 can be found in the above embodiment of the present application. Figure 1 For the sake of brevity, the description of the corresponding parts of the data processing method will not be repeated here.

[0142] The data processing device of an embodiment of the present application obtains, for inconsistent business data, a first data set and a second data set with the same data volume of business data through a smart contract service platform, and splits the first data set and the second data set into M first data subsets and M second data subsets respectively according to preset data identifiers. Thereafter, a first total hash value of each first data subset and a second total hash value of each second data subset are obtained, and corresponding to each preset data identifier, a first data subset and a second data subset with different first total hash values ​​are searched; when the data volume of the business data included in the first data subset or the second data subset is greater than a preset number, the first data subset and the second data subset with different first total hash values ​​and the second total hash values ​​are used as the first data set and the second data set, respectively, and the first data subset and the second data subset are redetermined until the data volume of the business data included in the first data subset or the second data subset is less than or equal to the preset number, thereby obtaining a comparison result, wherein the comparison result includes inconsistent business data. Through the above iterative calculation, the inconsistent business data can be directly located, the data comparison efficiency can be effectively improved, and the cost and difficulty of subsequent business data repair can be reduced.

[0143] Figure 4 FIG. 1 shows a schematic diagram of the structure of a data processing device provided by an embodiment of the present application. Figure 4 As shown, the device may include a processor 401 and a memory 402 storing computer program instructions.

[0144] Specifically, the processor 401 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0145] Memory 402 may include a large capacity memory for information or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. In one example, memory 402 may include removable or non-removable (or fixed) media, or memory 402 may be a non-volatile solid-state memory. Memory 402 may be internal or external to the data processing device.

[0146] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0147] The processor 401 implements the method described in the embodiment of the present application by reading and executing the computer program instructions stored in the memory 402, and achieves the corresponding technical effect achieved by executing the method in the embodiment of the present application. For the sake of brevity, it will not be repeated here.

[0148] In one example, the data processing device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401 , the memory 402 , and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0149] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0150] Bus 410 includes hardware, software or both, and couples the components of the online information flow metering device to each other. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus or other suitable buses or a combination of two or more of these. Where appropriate, bus 410 may include one or more buses. Although the present application describes and illustrates a specific bus, the present application considers any suitable bus or interconnect.

[0151] The data processing device can execute the data processing method in the embodiment of the present application, thereby achieving the corresponding technical effects of the data processing method described in the embodiment of the present application.

[0152] In addition, in combination with the data processing method in the above embodiment, the embodiment of the present application may provide a readable storage medium for implementation. The readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the data processing methods in the above embodiment is implemented. Examples of readable storage media may be non-transitory machine-readable media, such as electronic circuits, semiconductor memory devices, read-only memories (ROMs), floppy disks, compact discs (CD-ROMs), optical discs, hard disks, etc.

[0153] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0154] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The programs or code segments can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link via a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memories (ROMs), flash memories, erasable read-only memories (EROMs), floppy disks, compact disc read-only memories (CD-ROMs), optical discs, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via computer networks such as the Internet and intranets.

[0155] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0156] In addition, in combination with the data processing methods, devices, and readable storage media in the above embodiments, embodiments of the present application may be implemented by providing a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device performs any one of the data processing methods in the above embodiments.

[0157] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0158] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method is applied to a smart contract management platform, and the method includes: Acquire a first data set and a second data set, where the first data set includes the same amount of business data as the second data set, and there is inconsistent business data in the first data set and the second data set, each piece of business data includes a preset data identifier, and the business data in the first data set and the business data in the second data set come from different business systems; According to a preset data identifier, split the first data set and the second data set into M first data subsets and M second data subsets, respectively, where M is a positive integer; Obtaining a first total hash value of each of the first data subsets and a second total hash value of each of the second data subsets; For each of the preset data identifiers, searching for a first data subset and a second data subset having different first total hash values ​​and second total hash values; When the amount of business data included in the first data subset or the second data subset is greater than a preset amount, the first data subset and the second data subset having different first total hash values ​​and second total hash values ​​are used as the first data set and the second data set, respectively, and the process returns to step 10 to split the first data set and the second data set into M first data subsets and M second data subsets, respectively, according to the preset data identifier, until the amount of business data included in the first data subset or the second data subset is less than or equal to the preset amount, thereby obtaining a comparison result, wherein the comparison result includes inconsistent business data; The obtaining of the first data set and the second data set includes: Performing a hash operation on each piece of the business data in the first data set to obtain a plurality of first hash values; Summing the multiple first hash values ​​to obtain a third total hash value; Performing a hash operation on each piece of the business data in the second data set to obtain a plurality of second hash values; Summing the multiple second Hash values ​​to obtain a fourth total Hash value; When the third total Hash value and the fourth total Hash value are different, it is determined that inconsistent business data exists in the first data set and the second data set.

2. The method according to claim 1, characterized in that The obtaining of a first total hash value of each first data subset and a second total hash value of each second data subset includes: Obtain a first hash value for each piece of business data in the first data subset and a second hash value for each piece of business data in the second data subset; Summing the first Hash values ​​to obtain the first total Hash value; The second hash values ​​are summed to obtain the second total hash value.

3. The method according to claim 1, characterized in that The performing a hash operation on each of the business data in the first data set to obtain a plurality of first hash values ​​includes: Obtaining first character length information of each of the business data in the first data; performing character padding on each of the business data in the first data according to the first character length information of each of the business data in the first data, and adding the first length identification character of the first character length information to obtain fifth business data, where the character length of the fifth business data is the target length; Perform a hash operation on the fifth service data to obtain the first hash value.

4. The method according to claim 1, wherein The smart contract management platform includes a database; before obtaining the first data set and the second data set, the method further includes: Respectively receiving first business data sent in real time by a first business system and second business data sent in real time by a second business system, the first business data including a first business information identifier and first operation information corresponding to the first business information, and the second business data including a second business information identifier and second operation information corresponding to the second business information; performing a first operation on the first business information stored in the database according to the first business information identifier and first operation information corresponding to the first business information identifier to obtain business data included in the first data set; According to the second service information identifier and the second operation information corresponding to the second service information identifier, a second operation is performed on the second service information stored in the database to obtain the service data included in the second data set.

5. The method according to claim 4, characterized in that The method further comprises: The business data included in the first data set and the business data included in the second data set are respectively stored in the database.

6. A data processing device, characterized in that: The device is applied to a smart contract management platform, and includes: an acquisition module, configured to acquire a first data set and a second data set, wherein the first data set includes business data of the same amount as the second data set, and there is inconsistent business data in the first data set and the second data set, each business data includes a preset data identifier, and the business data in the first data set and the business data in the second data set are from different business systems; a processing module, configured to split the first data set and the second data set into M first data subsets and M second data subsets respectively according to preset data identifiers, where M is a positive integer; The acquisition module is further configured to acquire a first total hash value of each first data subset and a second total hash value of each second data subset; A search module, configured to search for a first data subset and a second data subset having different first total Hash values ​​and second total Hash values ​​for each of the preset data identifiers; The processing module is further configured to, when the amount of business data included in the first data subset or the second data subset is greater than a preset amount, use the first data subset and the second data subset having different first total hash values ​​and second total hash values ​​as the first data set and the second data set, respectively, and return to the step of splitting the first data set and the second data set into M first data subsets and M second data subsets, respectively, according to the preset data identifier, until the amount of business data included in the first data subset or the second data subset is less than or equal to the preset amount, thereby obtaining a comparison result, wherein the comparison result includes inconsistent business data; The acquisition module is further configured to acquire the first data set and the second data set, including: Performing a hash operation on each piece of the business data in the first data set to obtain a plurality of first hash values; Summing the multiple first hash values ​​to obtain a third total hash value; Performing a hash operation on each piece of the business data in the second data set to obtain a plurality of second hash values; Summing the multiple second Hash values ​​to obtain a fourth total Hash value; When the third total Hash value and the fourth total Hash value are different, it is determined that inconsistent business data exists in the first data set and the second data set.

7. A data processing device, characterized in that: The device includes: a processor, and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the data processing method according to any one of claims 1 to 5.

8. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the data processing method according to any one of claims 1 to 5 is implemented.

9. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data processing method according to any one of claims 1 to 5.

Images