Data comparison method, apparatus, device, medium, and program product

Abstract

The application provides a data comparison method and device, equipment, medium and program product, relates to the technical field of data processing, and is used for improving the efficiency of data processing. The specific technical scheme is as follows: obtaining a to-be-compared data group; in the case that the data state of to-be-compared data in the to-be-compared data group is a compressed state, reading the to-be-compared data in the to-be-compared data group into a memory buffer area, and decompressing in the memory buffer area to obtain a decompressed to-be-compared data group; in the memory buffer area, based on the business type identifier corresponding to the to-be-compared data in the decompressed to-be-compared data group, performing data extraction on the to-be-compared data in the decompressed to-be-compared data group to obtain a first data set and a second data set; in the memory buffer area, performing data comparison on the first data set and the second data set to obtain a comparison result. The application is applied to a data consistency checking scene.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data comparison method, apparatus, device, medium, and program product. Background Technology

[0002] With the continuous evolution and iteration of distributed architectures, microservice systems, and large-scale enterprise-level IT infrastructures, system version upgrades and architecture refactoring have become high-frequency operational scenarios for maintaining business continuity. These operational scenarios are usually accompanied by structural changes to a large number of configuration parameters, interface definitions, and business rules, posing a serious challenge to verifying the accuracy and consistency of the output data from the old and new systems.

[0003] To ensure the accuracy and consistency of files, relevant technologies often employ Linux difference comparison tools, bit comparison visualization tools, or big data comparison schemes based on traditional extraction, transformation, and loading engines to compare the consistency of data between old and new systems. Specifically, at each key stage of data processing, it is necessary to first complete the full data loading and fixed field normalization, then calculate the hash value for each specified field and perform temporary storage, and finally rely on hash value comparison to determine whether the data between the old and new systems has been changed or damaged during the flow or migration, thereby identifying inconsistencies in the output data of the old and new systems.

[0004] However, when faced with a massive amount of files to be compared, loading all the data, performing field-by-field hash calculations, and temporarily storing the data will generate a large number of read and write operations and computational overhead, resulting in long data comparison and data processing times and low data processing efficiency. Summary of the Invention

[0005] This application provides a data comparison method, apparatus, device, medium, and program product for improving the efficiency of data processing.

[0006] In a first aspect, embodiments of this application provide a data comparison method, the method comprising: acquiring a data set to be compared, the data set including first data to be compared generated by a first system and second data to be compared generated by a second system; if the data state of the data to be compared in the data set is compressed, reading the data to be compared in the data set into a memory buffer and decompressing it in the memory buffer to obtain a decompressed data set to be compared; in the memory buffer, based on the service type identifier corresponding to the data to be compared in the decompressed data set, extracting data from the data to be compared in the decompressed data set to obtain a first data set and a second data set, the first data set being the set corresponding to the first data to be compared and the second data set being the set corresponding to the second data to be compared; and comparing the first data set and the second data set in the memory buffer to obtain a comparison result.

[0007] The technical solution provided in this application brings at least the following benefits: By adopting streaming memory decompression, massive amounts of data are decompressed in memory, eliminating the disk input / output throughput bottleneck in the process of massive data comparison from the root, significantly improving data processing speed. Furthermore, by extracting data based on the business type identifier corresponding to the data to be compared, it is possible to match the field organization rules of different business data, compare only a portion of the data, abandon full comparison, and selectively extract effective comparison data, thereby improving the efficiency of data processing.

[0008] One possible implementation involves extracting the data to be compared from the decompressed data set based on the business type identifier corresponding to the data to be compared in the decompressed data set, to obtain a first data set and a second data set. This includes: determining at least one first data to be extracted from the decompressed first data set based on the first business type identifier corresponding to the first data set; determining at least one second data to be extracted from the decompressed second data set based on the second business type identifier corresponding to the second data set; obtaining the first data set based on the at least one first data to be extracted; and obtaining the second data set based on the at least one second data to be extracted.

[0009] Another possible implementation, in which a first data set is obtained based on at least one first data to be extracted, and a second data set is obtained based on at least one second data to be extracted, includes: performing hash processing on each first data to be extracted to obtain a first hash value corresponding to each first data to be extracted, and performing hash processing on each second data to be extracted to obtain a second hash value corresponding to each second data to be extracted; constructing a first data set based on the first hash value corresponding to each first data to be extracted, and constructing a second data set based on the first hash value corresponding to each second data to be extracted.

[0010] Another possible implementation, as described above, involves comparing the first and second data sets in the memory buffer to obtain a comparison result, including performing a bidirectional difference operation on the first and second data sets in the memory buffer to obtain a comparison result.

[0011] Another possible implementation involves performing bidirectional difference processing on the first and second data sets to obtain the comparison results. The method further includes: if the comparison results include difference data, obtaining the metadata corresponding to the difference data, wherein the metadata includes at least one of the following: original file name, line number, system identifier, and business type; and generating difference analysis information based on the metadata corresponding to the difference data, wherein the difference analysis information is used to indicate the data error nodes corresponding to the difference data.

[0012] Secondly, embodiments of this application provide a data comparison device, comprising: an acquisition module, a processing module, and a comparison module. The acquisition module is used to acquire a data set to be compared, including first data to be compared generated by a first system and second data to be compared generated by a second system. The processing module is used to, when the data state of the data to be compared in the data set is compressed, read the data to be compared in the data set into a memory buffer, and decompress it in the memory buffer to obtain a decompressed data set; and in the memory buffer, based on the service type identifier corresponding to the data to be compared in the decompressed data set, extract data from the data to be compared in the decompressed data set to obtain a first data set and a second data set, wherein the first data set is the set corresponding to the first data to be compared, and the second data set is the set corresponding to the second data to be compared. The comparison module is used to compare the first data set and the second data set in the memory buffer to obtain a comparison result.

[0013] In one possible implementation, the aforementioned processing module is specifically used to determine at least one first data to be extracted from the decompressed first data to be compared based on the first business type identifier corresponding to the decompressed first data to be compared, and to determine at least one second data to be extracted from the decompressed second data to be compared based on the second business type identifier corresponding to the decompressed second data to be compared; and to obtain a first data set based on at least one first data to be extracted, and to obtain a second data set based on at least one second data to be extracted.

[0014] Another possible implementation is that the above processing module is specifically used to perform hash processing on each first piece of data to be extracted to obtain a first hash value corresponding to each piece of data to be extracted, and to perform hash processing on each second piece of data to be extracted to obtain a second hash value corresponding to each piece of data to be extracted; and to construct a first data set based on the first hash value corresponding to each piece of data to be extracted, and to construct a second data set based on the first hash value corresponding to each piece of data to be extracted.

[0015] Another possible implementation is that the aforementioned processing module is specifically used to perform bidirectional difference processing on the first data set and the second data set in the memory buffer to obtain the comparison result.

[0016] Another possible implementation is that the data comparison device further includes: a generation module; an acquisition module, which is used to acquire metadata corresponding to the difference data when the comparison result includes difference data, the metadata including at least one of the following: original file name, line number, system identifier and business type, etc.; and a generation module, which is used to generate difference analysis information based on the metadata corresponding to the difference data, the difference analysis information being used to indicate the data error node corresponding to the difference data.

[0017] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory stores a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the method of the first aspect described above.

[0018] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a computer, implement the method of the first aspect described above.

[0019] Fifthly, this application provides a computer program product stored in a storage medium, which, when executed by a computer, implements the method described in the first aspect.

[0020] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0021] The beneficial effects of the second to sixth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description

[0022] Figure 1 A schematic diagram of the network architecture for a data comparison method provided in this application embodiment;

[0023] Figure 2 A flowchart illustrating a data comparison method provided in an embodiment of this application;

[0024] Figure 3 A flowchart illustrating another data comparison method provided in an embodiment of this application;

[0025] Figure 4 A flowchart illustrating another data comparison method provided in an embodiment of this application;

[0026] Figure 5 A flowchart illustrating another data comparison method provided in an embodiment of this application;

[0027] Figure 6 A flowchart illustrating yet another data comparison method provided in an embodiment of this application;

[0028] Figure 7 This is a schematic diagram of the structure of a data comparison system provided in an embodiment of this application;

[0029] Figure 8 This is a schematic diagram of the structure of a data comparison device provided in an embodiment of this application;

[0030] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0032] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0033] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0034] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0035] The data comparison method, apparatus, device, medium, and program product provided in this application embodiment can be applied in data processing scenarios.

[0036] Currently, with the continuous evolution of distributed architectures, microservice systems, and large-scale enterprise-level IT infrastructure, business systems such as telecommunications billing and data collection systems, financial clearing and reconciliation platforms, and network security audit centers frequently undergo version upgrades, architecture restructuring, and migration to domestically developed cloud-native architectures. Consistency verification of massive heterogeneous call detail record (CDR) data is a crucial means to ensure a smooth business transition and maintain business continuity. As the scale of digital business continues to expand, data verification scenarios face challenges such as terabytes to petabytes of massive data, highly compressed archive storage, and heterogeneous multi-source data formats. Consequently, various data consistency verification solutions have been developed within the industry.

[0037] To address the aforementioned issues, a consistency verification scheme based on distributed database temporary storage combined with fixed-field hashing has been widely adopted in existing technologies. This scheme first decompresses massive amounts of highly compressed call detail record (CDR) archive files to a disk, filters the data to be compared according to preset fixed-field rules, imports the processed CDR data into a database for temporary storage, and finally reads the data from the database to perform hash operations and set comparisons, outputting the data difference results between the old and new CDRs.

[0038] However, the above-mentioned solution relies on a processing mode of disk decompression, fixed field filtering, full data import, and memory set comparison to carry out the verification work. It cannot adapt to multi-source heterogeneous call detail record (CDR) formats and is prone to false alarms. The decompression of massive files will bring huge disk read and write overhead, and the full data participation in the calculation is very likely to cause memory overflow. The output difference results also lack complete traceability information, resulting in insufficient stability, low verification accuracy and low running efficiency of data consistency verification in the scenario of massive heterogeneous compressed CDRs.

[0039] To address the aforementioned technical issues, embodiments of this application provide a data comparison method, apparatus, device, medium, and program product. It employs streaming memory decompression, enabling massive amounts of data to be decompressed in memory, eliminating disk input / output throughput bottlenecks in the massive data comparison process at its source, significantly improving data processing speed. Furthermore, based on the business type identifier corresponding to the data to be compared, data extraction can match the field organization rules of different business data, comparing only a portion of the data, abandoning full comparison, and selectively extracting effective comparison data. This improves the processing speed and overall operating efficiency of data consistency verification between old and new systems in the scenario of comparing files.

[0040] The data comparison methods, apparatus, devices, media, and program products provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0041] Figure 1 The network architecture for a data comparison method provided in an embodiment of this application is illustrated. For example... Figure 1As shown, the network architecture includes a data comparison device 101 and a terminal device 102. The data comparison device 101 and the terminal device 102 are interconnected.

[0042] In some embodiments, the data comparison device 101 may be a server, a computer, or a processor or processing unit within a server or computer. The server may be a single server or a server cluster comprising multiple servers. It should be noted that the embodiments of this application do not limit the specific device form of the data comparison device 101. Figure 1 The data comparison device 101 is shown as a single server as an example.

[0043] In some embodiments, the terminal device may be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, personal computer (PC), ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., and the embodiments of this application do not specifically limit it. Figure 1 The example shown is a mobile phone, with terminal device 102 as an example.

[0044] In some embodiments, terminal device 102 sends a data set to be compared to data comparison device 101. Data comparison device 101 receives the data set to be compared from terminal device 102. Then, if the data state of the data to be compared in the data set is compressed, data comparison device 101 reads the data to be compared into a memory buffer and decompresses it in the memory buffer to obtain a decompressed data set to be compared. In the memory buffer, data comparison device 101 extracts data from the data to be compared based on the service type identifier corresponding to the data to be compared in the decompressed data set to obtain a first data set and a second data set. Data comparison device 101 compares the first data set and the second data set in the memory buffer to obtain a comparison result. Finally, data comparison device 101 sends the comparison result to terminal device 102.

[0045] It should be noted that the network architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As network architectures evolve, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0046] See Figure 2 This is a flowchart illustrating a data comparison method provided in an embodiment of this application. Figure 2 As shown, the data comparison method provided in this application embodiment can be implemented by the above-mentioned data comparison device, specifically including the following steps 201 to 204.

[0047] Step 201: The data comparison device acquires the data set to be compared.

[0048] In some embodiments, the data set to be compared may include at least a first set of data to be compared generated by a first system and a second set of data to be compared generated by a second system.

[0049] In some embodiments, the first system described above can be a clearing system, a centralized data collection system, or a traditional business auditing system. The specific system can be determined based on actual usage requirements, and this application does not impose any limitations.

[0050] In some embodiments, the second system described above can be a reconciliation system, a business data collection system, or a security audit system. The specific system can be determined based on actual usage requirements, and this application does not impose any limitations.

[0051] In some embodiments, the first data to be compared may include at least the original business records, business field information, and field location identifiers of the first data object.

[0052] In some embodiments, the second data to be compared may include at least the original business records, business field information, and field location identifiers of the second data object.

[0053] For example, the data object is a business data packet to be verified for consistency.

[0054] In some embodiments, the aforementioned business field information may include at least one of the following: business primary key identifier, business attribute field, field location information, and field format definition. The specific details can be determined based on actual usage requirements, and this application embodiment does not impose any limitations.

[0055] In some embodiments, the aforementioned identifier may include at least one of the following: text identifier, numerical identifier, or special symbol identifier, etc. The specific identifier can be determined according to actual usage requirements, and this application embodiment does not impose any limitations.

[0056] In some embodiments, the terminal may send a data packet to be compared to the data comparison device. The data comparison device receives the data packet to be compared sent by the service terminal and integrates the data packets corresponding to the first system and the second system to form a data group to be compared.

[0057] Step 202: When the data state of the data to be compared in the data group to be compared is compressed, the data comparison device reads the data to be compared in the data group to be compared into the memory buffer and decompresses it in the memory buffer to obtain the decompressed data group to be compared.

[0058] In some embodiments, the decompressed comparison data set may include at least the first comparison data generated by the first decompressed system and the second comparison data generated by the second decompressed system.

[0059] In some embodiments, the data comparison device can determine whether the data to be compared is in a compressed state.

[0060] For example, the data comparison device can read the data status identifier and header feature information of the data to be compared, and match the header feature information with the feature codes of various compressed files; if the match is successful, the data status identifier is verified to be valid, and the data to be compared is determined to be in a compressed state based on the data status identifier; if the match fails, the data status identifier is determined to be abnormal, and the data to be compared is determined to be in an original uncompressed state.

[0061] In some embodiments, the data comparison device can invoke a memory streaming decompression engine to read data blocks directly into a memory buffer in batches and complete the decompression in memory.

[0062] For example, the data comparison device can divide the original data to be compared into continuous data blocks according to a preset fixed size; read each data block into a memory buffer in sequence, clear the buffer after processing, and then load the next set of data blocks to perform the cycle.

[0063] In some embodiments, without the need for decompression, the original data blocks to be compared can be directly stored in a memory buffer.

[0064] Step 203: The data comparison device extracts data from the data to be compared in the decompressed data to be compared group based on the business type identifier corresponding to the data to be compared in the decompressed data to be compared group in the memory buffer, and obtains the first data set and the second data set.

[0065] In some embodiments, the data comparison device can obtain the business type identifier of the current data to be processed and determine the offset position of the target business field and the corresponding parsing rule from the extraction rule mapping table.

[0066] In some embodiments, the data comparison device can determine the mapping rule of the current service from a preset extraction rule mapping table based on the service type identifier of the data to be processed.

[0067] For example, the data comparison device can use the business type identifier as the retrieval primary key to retrieve the extraction rule mapping table and match the mapping rule data of the corresponding business. The mapping rule data includes at least parsing information such as the target business field, field offset position, and field length.

[0068] In some embodiments, the extraction rule mapping relationship contained in the extraction rule mapping table can be configured in real time, that is, the mapping relationship in the extraction rule mapping table can be dynamically updated.

[0069] In some embodiments, the above-mentioned business type identifier is used to distinguish the character encoding, numeric number, or string label of different business scenarios and data source systems.

[0070] In some embodiments, the first data set is the set corresponding to the first data to be compared, and the second data set is the set corresponding to the second data to be compared.

[0071] In some embodiments, combined with Figure 2 ,like Figure 3 As shown, step 203 above can be implemented through steps 203a and 203b.

[0072] Step 203a: Based on the first service type identifier corresponding to the first decompressed data to be compared, the data comparison device determines at least one first data to be extracted from the first data to be compared after decompression, and based on the second service type identifier corresponding to the second data to be compared after decompression, it determines at least one second data to be extracted from the second data to be compared after decompression, in the extraction rule mapping table.

[0073] In some embodiments, the data comparison device can select and load the extraction rule mapping table corresponding to the service type identifier in real time.

[0074] In some embodiments, the extraction rule mapping table may include at least the field positions, field order, and concatenation rules that need to be extracted for the business type.

[0075] In some embodiments, the data comparison device can determine the target business field and the corresponding mapping rule from the extraction rule mapping table based on the business type identifier of the data to be processed, and match the storage location information of the target field according to the mapping rule, thereby obtaining the text content of the target field based on the storage location information of the target field.

[0076] In some embodiments, the data comparison device can find the storage location information of the target field inside the data to be compared based on the above mapping rules, thereby obtaining the text content of the target field.

[0077] In some embodiments, the data comparison device may use a preset delimiter to concatenate and combine the text content of the target field into a target feature string.

[0078] In some embodiments, the preset separators mentioned above include at least: newline characters, commas, tab characters, and vertical line separators.

[0079] Step 203b: The data comparison device obtains a first data set based on at least one first data to be extracted, and obtains a second data set based on at least one second data to be extracted.

[0080] In some embodiments, the first data to be extracted may be the first target feature string concatenated and combined according to the preset delimiter, and the second data to be extracted may be the second target feature string concatenated and combined according to the preset delimiter.

[0081] In this way, by dynamically reading specified rows or fields from the extraction rule mapping table through business type identifiers, the full comparison is abandoned, the differences in format are masked, and heterogeneous data is converted into homogeneous core feature strings, thereby improving the flexibility and versatility of the comparison system, adapting to multi-source heterogeneous data scenarios, and reducing the false alarm rate.

[0082] In some embodiments, combined with Figure 3 ,like Figure 4 As shown, step 203b above can be implemented through steps 203b1 and 203b2.

[0083] Step 203b1: The data comparison device performs hash processing on each first piece of data to be extracted to obtain a first hash value corresponding to each piece of data to be extracted, and performs hash processing on each second piece of data to be extracted to obtain a second hash value corresponding to each piece of data to be extracted.

[0084] In some embodiments, the data comparison device can perform one-way hashing discretization processing on at least one first data to be extracted to calculate at least one hash digest value of a first fixed length, i.e., the first hash value.

[0085] In some embodiments, the data comparison device can perform one-way hashing discretization processing on at least one second data to be extracted to calculate at least one second fixed-length hash digest value, i.e., the second hash value.

[0086] Step 203b2: The data comparison device constructs a first data set based on the first hash value corresponding to each first data to be extracted, and constructs a second data set based on the first hash value corresponding to each second data to be extracted.

[0087] In some embodiments, the data comparison device may perform a modulo operation on the hash digest value.

[0088] In some embodiments, the data comparison device can construct logical bucket storage units with the same modulus identifier based on the above modulus values.

[0089] In some embodiments, the data comparison device can stream-append a formatted data stream to a logical bucket storage unit with the same name corresponding to the data modulus.

[0090] In some embodiments, the formatted data stream may include at least a target feature string, a hash digest value, and raw metadata including the original filename and line number.

[0091] In some embodiments, the data comparison device may use the hash digest value of each piece of data as a unique element of the set to construct a corresponding first system first data set and a second system second data set in memory.

[0092] In this way, by performing one-way hashing discretization on the target feature string, a fixed-length hash digest value is calculated, which realizes the dimensionality reduction of data features. At the same time, the hash digest value is transformed and moduloed, and the data is bucketed according to the calculated modulo value, which realizes the balanced and distributed processing of the global comparison operation pressure. Thus, full coverage comparison of all data is achieved under the condition of limited memory resources, significantly reducing hardware resource consumption.

[0093] Step 204: The data comparison device compares the first data set and the second data set in the memory buffer to obtain the comparison result.

[0094] In some embodiments, the comparison results described above can characterize business records that do not exist or differ between the first data set and the second data set.

[0095] In some embodiments, combined with Figure 2 ,like Figure 5 As shown, step 204 above can be implemented through step 204a as follows.

[0096] Step 204a: The data comparison device performs bidirectional difference processing on the first data set and the second data set in the memory buffer to obtain the comparison result.

[0097] In some embodiments, the comparison results described above may include difference data.

[0098] For example, the data comparison device can perform bidirectional difference processing on the first data set and the second data set to extract data that exists only in the first data set and data that exists only in the second data set, thereby determining inconsistent business data between the two sets.

[0099] In the task execution method provided in this application embodiment, the data comparison device acquires a data group to be compared, which includes first data to be compared generated by a first system and second data to be compared generated by a second system. When the data state of the data to be compared in the data group is compressed, the data comparison device reads the data to be compared into a memory buffer and decompresses it in the memory buffer to obtain a decompressed data group. In the memory buffer, based on the service type identifier corresponding to the data to be compared in the decompressed data group, the data comparison device extracts data from the data to be compared in the decompressed data group to obtain a first data set and a second data set. The first data set is the set corresponding to the first data to be compared, and the second data set is the set corresponding to the second data to be compared. In the memory buffer, the data comparison device compares the first data set and the second data set to obtain a comparison result. This solution employs streaming memory decompression, enabling massive amounts of data to be decompressed in memory. This eliminates the disk input / output throughput bottleneck in the massive data comparison process at its source, significantly improving data processing speed. Furthermore, data extraction is performed based on the business type identifier corresponding to the data to be compared. This allows for matching the field organization rules of different business data, comparing only a portion of the data, abandoning full comparison, and selectively extracting effective comparison data, thereby improving data processing efficiency.

[0100] In some embodiments, combined with Figure 5 ,like Figure 6 As shown, after step 204a above, the data comparison method provided in this application embodiment further includes the following steps 301 and 302.

[0101] Step 301: If the comparison results include difference data, the data comparison device acquires the metadata corresponding to the difference data.

[0102] In some embodiments, the data comparison device may invoke set operation operators to perform bidirectional difference operations on the first data set and the second data set to obtain temporary files of content differences with the same modulus corresponding to all modulus values.

[0103] In some embodiments, the aforementioned temporary difference file may store business records that indicate whether the first data set and the second data set are non-existent or differ from each other.

[0104] In some embodiments, the aforementioned temporary difference file includes at least a difference hash digest value, a corresponding target feature string, original metadata, and a difference direction identifier.

[0105] In some embodiments, the data comparison device may employ a multi-way merge sort algorithm to globally and asynchronously merge all temporary files with the same modulus content differences corresponding to the modulus values, eliminate duplicates, and reorder them to obtain the final difference data.

[0106] In some embodiments, the aforementioned original file metadata may include at least one of the following: original file name, line number, system identifier, and business type.

[0107] Step 302: The data comparison device generates difference analysis information based on the metadata corresponding to the difference data.

[0108] In some embodiments, the data comparison device can automatically backtrack and align the original data error points of both systems based on the original file metadata carried in the final difference element, and finally generate a structured heterogeneous data stream consistency verification difference analysis report.

[0109] In some embodiments, heterogeneous analysis information can be used to indicate data error nodes corresponding to discrepancies in the data.

[0110] In this way, by extracting the original file metadata carried in the final difference elements, the system can automatically reverse map and locate the specific error points in the original data of the old and new systems, and automatically generate a structured heterogeneous data flow consistency verification difference analysis report containing "difference characteristics, source system location, target system location, and error type". This solves the problems of difficulty in tracing the comparison results and difficulty in manual investigation in the existing technology, and significantly improves the efficiency of operation and maintenance.

[0111] The data comparison method of this application will be described below through specific embodiments.

[0112] The implementation process of the data comparison method provided in this application includes the following steps S1 to S3.

[0113] S1, Data comparison device for dynamic feature extraction and flow-through binning.

[0114] For example, S1 can be implemented by S11, S12, S13 and S14.

[0115] S11. The data comparison device determines whether the first archived data object generated by the first system and the second archived data object generated by the second system are compressed. If they are compressed, the memory streaming decompression engine is called to read the data blocks into the memory buffer in batches and decompress them in memory. It is strictly forbidden to write temporary decompression files to the physical disk to avoid the loss of disk secondary write I / O.

[0116] S12. The data comparison device obtains the business type identifier of the current data to be processed, and loads the preset dynamic extraction rule mapping table accordingly; using specific data lines as the basic unit, it dynamically extracts the text content of the specified topological position in the first and second archived data objects, and concatenates and combines them into the target feature string according to the preset delimiter.

[0117] S13. The data comparison device performs one-way hashing discretization on the target feature string and calculates its fixed-length hash digest value to achieve dimensionality reduction of the data features.

[0118] S14. The data comparison device converts the hash digest value into numerical data and then performs modulo binning calculation; based on the calculated modulo value, the formatted data stream is appended to the logical bin storage unit with the same name corresponding to the modulo value.

[0119] S2, the data comparison device performs parallel set difference operation of modular buckets.

[0120] For example, S2 can be implemented through S21 and S22.

[0121] S21. The data comparison device traverses and retrieves a pair of logical bucket storage units with the same modulus identifier generated by the first system and the second system in step S1.

[0122] S22. The data comparison device reads the formatted data from any set of logical bucket storage units with the same modulus in a streaming manner, extracts the hash digest value as the unique element of the set, and constructs the corresponding first feature set A and second feature set B in memory respectively.

[0123] S23. The data comparison device calls the set operation operator to perform a symmetric bidirectional difference operation on the first feature set A and the second feature set B.

[0124] S3. The data comparison device performs multi-path merging of different results and backtracks the results.

[0125] For example, S3 can be implemented through S31 and S32.

[0126] S31. The data comparison device uses a multi-way merge sort algorithm to globally and asynchronously merge all temporary files with the same modulus content differences, eliminate duplicates, and reorder them.

[0127] S32. The data comparison device extracts the original file metadata carried in the final difference elements, automatically reverses and aligns the original data error points of the two systems, and finally generates a structured heterogeneous data stream consistency verification difference analysis report.

[0128] Thus, by employing streaming memory decompression, massive amounts of data are decompressed in memory, eliminating disk I / O throughput bottlenecks in the massive data comparison process at its source and significantly improving data processing speed. Furthermore, this solution abandons full-scale comparison, dynamically extracting comparison information from the decompressed data based on business type, converting heterogeneous data into homogeneous core feature strings, and masking data format differences. This enhances the flexibility and versatility of the comparison system, adapting to multi-source heterogeneous data scenarios and reducing false alarm rates. Simultaneously, the homogeneous core feature strings undergo hash dimensionality reduction and modulo binning, evenly dispersing the comparison pressure of massive global data, achieving a space-for-time tradeoff, completely eliminating memory overflow, and thus improving the accuracy of data consistency verification results between old and new systems in comparison file scenarios.

[0129] It should be noted that the descriptions of each step S1 to S3 in this embodiment can be found in the descriptions in the above embodiments, and will not be repeated here.

[0130] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there is no conflict, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0131] Figure 7 This is a schematic diagram of the structure of a data comparison system provided in an embodiment of this application. Figure 7 As shown, the data comparison system 800 may include: a memory streaming decompression and high compression ratio data conversion module 801, a heterogeneous data dynamic adaptive extraction module 802, a hash modulo binning and data dimensionality reduction module 803, and a same-modulus bidirectional difference set operation and accurate source tracing module 804.

[0132] Among them, the memory streaming decompression and high compression ratio data conversion module 801 is used for massive storage of historical archived data objects with high compression ratio characteristics. The system directly opens a streaming data block buffer in memory and introduces a streaming block decompression mechanism, which is applied to the relevant scheme of step 202 above.

[0133] The heterogeneous data dynamic adaptive extraction module 802 is used in the solution's built-in unstructured or semi-structured data dynamic reduction engine. When faced with heterogeneous data streams from different data sources with different formats, the system does not blindly compare all fields. Instead, it dynamically locates and extracts the core key feature elements that determine the correctness of the business logic based on the business type attributes of the current data, and applies them to the relevant solutions in steps 203 and 203a above.

[0134] The hash modulo binning and data dimensionality reduction module 803 is used to convert dynamically extracted fixed-length or variable-length feature strings into fixed-length low-dimensional hash digest values ​​through a one-way hash algorithm, and to perform mathematical modulo operations on the hash digest values ​​to evenly and deterministically distribute massive amounts of data records to logical data buckets or storage units corresponding to the modulo value. This is applied to the relevant schemes in steps 203, 203b1, and 203b2 above.

[0135] The same-modulus bidirectional difference operation and precise traceability module 804 is used to simultaneously load a pair of logical data buckets with the same modulus value from two data sources to be verified in a parallel or polling manner, perform bidirectional difference operation using a high-concurrency memory collection structure, and finally perform multi-way data merging to generate the final audit difference report, which is applied to the relevant schemes of steps 204, 204a, 301 and 302 above.

[0136] It should be noted that for a detailed explanation of the steps performed by each module and their beneficial effects, please refer to the description in the above embodiments, which will not be repeated here.

[0137] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] This application embodiment can divide the data comparison device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0139] In some embodiments, this application also provides a data comparison apparatus. The data comparison apparatus may include one or more functional modules for implementing the data comparison method of the above method embodiments.

[0140] For example, Figure 8 This is a schematic diagram of a data comparison device provided in an embodiment of this application. Figure 8 As shown, the data comparison device 900 includes: an acquisition module 901, a processing module 902, and a comparison module 903.

[0141] The system includes an acquisition module 901, which acquires a data set to be compared, comprising first comparison data generated by a first system and second comparison data generated by a second system. A processing module 902, when the comparison data in the data set is in a compressed state, reads the comparison data into a memory buffer and decompresses it within the memory buffer to obtain a decompressed comparison data set. Then, based on the business type identifier corresponding to the comparison data in the decompressed comparison data set, the module extracts data from the decompressed comparison data set within the memory buffer to obtain a first data set and a second data set. The first data set corresponds to the first comparison data set, and the second data set corresponds to the second comparison data set. A comparison module 903 compares the first and second data sets within the memory buffer to obtain a comparison result.

[0142] The data comparison device provided in this application employs streaming memory decompression, enabling massive amounts of data to be decompressed in memory. This eliminates the disk input / output throughput bottleneck in the massive data comparison process from the root, significantly improving data processing speed. Furthermore, data extraction is performed based on the business type identifier corresponding to the data to be compared. This allows for matching the field organization rules of different business data, comparing only a portion of the data, abandoning full comparison, and selectively extracting effective comparison data, thereby improving data processing efficiency.

[0143] In some embodiments, the processing module 902 is specifically configured to determine at least one first data to be extracted from the decompressed first data to be compared based on the first business type identifier corresponding to the decompressed first data to be compared, and to determine at least one second data to be extracted from the decompressed second data to be compared based on the second business type identifier corresponding to the decompressed second data to be compared; and to obtain a first data set based on at least one first data to be extracted, and to obtain a second data set based on at least one second data to be extracted.

[0144] In other embodiments, the processing module 902 is specifically used to perform hash processing on each first piece of data to be extracted to obtain a first hash value corresponding to each piece of data to be extracted, and to perform hash processing on each second piece of data to be extracted to obtain a second hash value corresponding to each piece of data to be extracted; and to construct a first data set based on the first hash value corresponding to each piece of data to be extracted, and to construct a second data set based on the first hash value corresponding to each piece of data to be extracted.

[0145] In other embodiments, the processing module 902 is specifically used to perform bidirectional difference processing on the first data set and the second data set in the memory buffer to obtain a comparison result.

[0146] In some other embodiments, the data comparison device further includes: a generation module; an acquisition module 901, which is further configured to acquire metadata corresponding to the difference data when the comparison result includes difference data, the metadata including at least one of the following: original file name, line number, system identifier and business type, etc.; and a generation module, which is configured to generate difference analysis information based on the metadata corresponding to the difference data, the difference analysis information being used to indicate the data error node corresponding to the difference data.

[0147] It should be noted that the data comparison device can implement all the processes implemented in the above method embodiments and achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0148] In the case where the functions of the integrated modules described above are implemented in hardware, this application provides a possible structural schematic diagram of the electronic device involved in the above embodiments. For example... Figure 9 As shown, the electronic device 90 includes: a processor 92, a communication interface 93, and a bus 94. Optionally, the electronic device 90 may also include a memory 91.

[0149] Processor 92 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0150] Communication interface 93 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0151] The memory 91 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0152] As one possible implementation, the memory 91 can exist independently of the processor 92. The memory 91 can be connected to the processor 92 via a bus 94 and is used to store instructions or program code. When the processor 92 calls and executes the instructions or program code stored in the memory 91, it can implement the data comparison method provided in the embodiments of this application.

[0153] In another possible implementation, memory 91 can also be integrated with processor 92.

[0154] Bus 94 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 94 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0155] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.

[0156] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above data comparison method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0157] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0158] This application also provides a readable storage medium storing a program or instructions that, when executed by a computer, implement the data comparison method provided in the above embodiments. It is understood that all or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware; the readable storage medium can be any of the foregoing embodiments or memory; the readable storage medium can also be an external storage device of the service invocation device, such as a plug-in hard drive, Smart MediaCard (SMC), Secure Digital (SD) card, flash card, etc., equipped on the service invocation device. Further, the readable storage medium can include both internal storage units of the service invocation device and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the service invocation device. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0159] This application also provides a computer program product, which is stored in a storage medium and implements the data comparison method provided in the above embodiments when the computer program product is executed by a computer.

[0160] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0162] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data comparison method, characterized in that, include: Obtain a data set to be compared, wherein the data set to be compared includes a first data set to be compared generated by a first system and a second data set to be compared generated by a second system; When the data state of the data to be compared in the data group to be compared is compressed, the data to be compared in the data group to be compared is read into the memory buffer and decompressed in the memory buffer to obtain the decompressed data group to be compared. In the memory buffer, based on the service type identifier corresponding to the data to be compared in the decompressed data to be compared group, data extraction is performed on the data to be compared in the decompressed data to be compared group to obtain a first data set and a second data set. The first data set is the set corresponding to the first data to be compared, and the second data set is the set corresponding to the second data to be compared. In the memory buffer, the first data set and the second data set are compared to obtain the comparison result.

2. The data comparison method according to claim 1, characterized in that, Based on the service type identifier corresponding to the data to be compared in the decompressed comparison data group, the comparison data to be compared in the decompressed comparison data group is extracted to obtain a first data set and a second data set, including: Based on the first business type identifier corresponding to the first decompressed comparison data, at least one first data to be extracted is determined from the extraction rule mapping table; and based on the second business type identifier corresponding to the second decompressed comparison data, at least one second data to be extracted is determined from the extraction rule mapping table. Based on the at least one first data to be extracted, the first data set is obtained, and based on the at least one second data to be extracted, the second data set is obtained.

3. The data comparison method according to claim 2, characterized in that, The step of obtaining the first data set based on the at least one first data to be extracted, and obtaining the second data set based on the at least one second data to be extracted, includes: Each first piece of data to be extracted is hashed to obtain a first hash value corresponding to each piece of data to be extracted; and each second piece of data to be extracted is hashed to obtain a second hash value corresponding to each piece of data to be extracted. Based on the first hash value corresponding to each first piece of data to be extracted, a first data set is constructed, and based on the first hash value corresponding to each second piece of data to be extracted, a second data set is constructed.

4. The data comparison method according to claim 1, characterized in that, The step of comparing the first data set and the second data set in the memory buffer to obtain the comparison result includes: In the memory buffer, a bidirectional difference process is performed on the first data set and the second data set to obtain the comparison result.

5. The data comparison method according to claim 4, characterized in that, After performing bidirectional difference processing on the first data set and the second data set to obtain the comparison result, the method further includes: If the comparison results include difference data, the metadata corresponding to the difference data is obtained. The metadata includes at least one of the following: original file name, line number, system identifier, and business type, etc. Based on the metadata corresponding to the difference data, difference analysis information is generated, which is used to indicate the data error nodes corresponding to the difference data.

6. A data comparison device, characterized in that, The data comparison device includes: an acquisition module, a processing module, and a comparison module; The acquisition module is used to acquire a data set to be compared, which includes a first data set to be compared generated by a first system and a second data set to be compared generated by a second system. The processing module is configured to, when the data state of the data to be compared in the data group to be compared is compressed, read the data to be compared in the data group to be compared into a memory buffer, and decompress it in the memory buffer to obtain a decompressed data group to be compared; and, in the memory buffer, extract data from the data to be compared in the decompressed data group to be compared based on the service type identifier corresponding to the data to be compared in the data group to be compared, to obtain a first data set and a second data set, wherein the first data set is the set corresponding to the first data to be compared, and the second data set is the set corresponding to the second data to be compared; The comparison module is used to compare the first data set and the second data set in the memory buffer to obtain a comparison result.

7. The data comparison device according to claim 6, characterized in that, The processing module is specifically used to determine at least one first data to be extracted from the first data to be compared after decompression based on the first business type identifier corresponding to the first data to be compared after decompression, and to determine at least one second data to be extracted from the second data to be compared after decompression based on the second business type identifier corresponding to the second data to be compared after decompression. Based on the at least one first data to be extracted, the first data set is obtained, and based on the at least one second data to be extracted, the second data set is obtained.

8. The data comparison device according to claim 7, characterized in that, The processing module is specifically used to perform hash processing on each first piece of data to be extracted to obtain a first hash value corresponding to each piece of data to be extracted, and to perform hash processing on each second piece of data to be extracted to obtain a second hash value corresponding to each piece of data to be extracted; Based on the first hash value corresponding to each first piece of data to be extracted, a first data set is constructed, and based on the first hash value corresponding to each second piece of data to be extracted, a second data set is constructed.

9. The data comparison device according to claim 6, characterized in that, The processing module is specifically used to perform bidirectional difference processing on the first data set and the second data set in the memory buffer to obtain the comparison result.

10. The data comparison device according to claim 9, characterized in that, The data comparison device further includes: a generation module; The acquisition module is further configured to acquire metadata corresponding to the difference data when the comparison result includes difference data, wherein the metadata includes at least one of the following: original file name, line number, system identifier, and business type, etc. The generation module is used to generate difference analysis information based on the metadata corresponding to the difference data, and the difference analysis information is used to indicate the data error nodes corresponding to the difference data.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the data comparison method as described in any one of claims 1-5.

12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a computer, implement the data comparison method as described in any one of claims 1-5.

13. A computer program product, characterized in that, The computer program product is stored in a storage medium, and when executed by a computer, the computer program product implements the data comparison method as described in any one of claims 1-5.

Images