Data set comparison method and device, electronic equipment and storage medium

Abstract

The invention discloses a data set comparison method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a plurality of data sets; grouping the plurality of data sets based on the target database connection pool to obtain a plurality of data sets; the multiple data sets are distributed to multiple target threads based on the target thread pool, and the multiple data sets and the multiple target threads are in one-to-one correspondence; different data sets contained in the data set corresponding to each target thread in the multiple target threads are compared, a comparison result is obtained, and the comparison result is used for data migration of the multiple data sets. The technical problem of low efficiency of large-scale data comparison in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for comparing data sets. Background Art

[0002] In the civil aviation sector, with the cloud-based upgrade of business systems, a large amount of historical data and real-time transaction data need to be migrated from local databases to the cloud to build a more flexible and scalable IT architecture. However, data migration is not a simple copy and paste, but a complex and challenging process, especially when the data volume reaches GB level and the number of records exceeds 100 million, the limitations of traditional data comparison methods become apparent.

[0003] When the amount of data is small, manual comparison or simple data comparison tools are still competent, but when the data scale increases dramatically, manual comparison becomes impractical, inefficient, and prone to human errors. Existing basic data comparison tools, such as SQL query-based comparisons, are efficient when processing structured data, but their performance drops significantly when faced with unstructured data and complex data query logic. At the same time, such tools have high requirements for data real-time performance and have poor adaptability to dynamically changing data sets. In addition, although existing big data comparison technologies can process large-scale data, they have obvious shortcomings in processing speed, computing resource consumption, result interpretability, and robustness to noise and outliers. With the diversification of data types and the explosive growth of data volume, existing technologies are difficult to meet the needs of efficient and accurate comparisons.

[0004] Currently, no corresponding solution has been proposed for the above problems. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, electronic device and storage medium for comparing data sets, so as to at least solve the technical problem of low efficiency of large-scale data comparison in the prior art.

[0006] According to one aspect of an embodiment of the present invention, a method for comparing data sets is provided, comprising: acquiring multiple data sets; grouping the multiple data sets based on a target database connection pool to obtain multiple data sets; allocating the multiple data sets to multiple target threads based on a target thread pool, wherein the multiple data sets correspond one-to-one to the multiple target threads; comparing different data sets contained in the data set corresponding to each target thread in the multiple target threads to obtain a comparison result, wherein the comparison result is used for data migration of the multiple data sets.

[0007] Optionally, acquiring multiple data sets includes: acquiring multiple initial data sets; performing data preprocessing operations on the multiple initial data sets to obtain multiple data sets, wherein the data preprocessing operations include data cleaning, data deduplication and data formatting.

[0008] Optionally, grouping multiple data sets based on the target database connection pool to obtain multiple data sets includes: obtaining the initial database connection pool and the data set logical relationship between the multiple data sets; performing an initialization operation on the initial database connection pool to obtain the target database connection pool; grouping the multiple data sets based on the target database connection pool and the data set logical relationship to obtain multiple data sets.

[0009] Optionally, allocating multiple data sets to multiple target threads based on the target thread pool includes: obtaining an initial thread pool; performing an initialization operation on the initial thread pool to obtain a target thread pool; generating multiple target threads based on the target thread pool; and allocating multiple data sets to multiple target threads based on the target thread pool.

[0010] Optionally, different data sets contained in a data set corresponding to each target thread in multiple target threads are compared to obtain a comparison result, including: for any target thread, constructing a target query statement; processing a first data set and a second data set in the target thread according to the target query statement to determine a first difference set and a second difference set, wherein the first difference set is a data set contained in the first data set but not contained in the second data set, and the second difference set is a data set contained in the second data set but not contained in the first data set; and determining a first target difference set based on the first difference set and the second difference set.

[0011] Optionally, the data set comparison method further includes: performing a data analysis operation on the target difference set to generate a difference report, wherein the data analysis operation includes data verification, data correction and data synchronization.

[0012] Optionally, the data set comparison method also includes: acquiring multiple data sets at the current time point; comparing the multiple data sets at the current time point with multiple data sets at a preset time point to obtain a second target difference set; and generating a target comparison report based on the first target difference set and the second target difference set.

[0013] According to another aspect of an embodiment of the present invention, a data set comparison device is also provided, including: an acquisition module, used to acquire multiple data sets; a grouping module, used to group the multiple data sets based on a target database connection pool to obtain multiple data sets; an allocation module, used to allocate the multiple data sets to multiple target threads based on a target thread pool, wherein the multiple data sets correspond one-to-one to the multiple target threads; a comparison module, used to compare different data sets contained in the data set corresponding to each target thread in the multiple target threads to obtain a comparison result, wherein the comparison result is used for data migration of the multiple data sets.

[0014] According to another aspect of an embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute any of the above data set comparison methods.

[0015] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute any of the above data set comparison methods when running.

[0016] In an embodiment of the present invention, multiple data sets are obtained; the multiple data sets are grouped based on the target database connection pool to obtain multiple data sets; the multiple data sets are allocated to multiple target threads based on the target thread pool, wherein the multiple data sets correspond one to one to the multiple target threads; different data sets contained in the data set corresponding to each target thread in the multiple target threads are compared to obtain a comparison result, wherein the comparison result is used for data migration of the multiple data sets. The present invention achieves the technical effect of improving the efficiency and accuracy of large-scale data comparison by combining multi-threaded concurrency and set comparison strategy, thereby solving the technical problem of low efficiency of large-scale data comparison in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of a method for comparing data sets according to one embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of a method for comparing data sets according to one embodiment of the present invention;

[0020] Figure 3 is a structural block diagram of a data set comparison device according to one embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] According to an embodiment of the present invention, an embodiment of a method for comparing data sets is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0024] The method embodiment may also be performed in an electronic device, a similar control device or an electronic device including a memory and a processor. Taking an electronic device as an example, the electronic device may include one or more processors and a memory for storing data. Optionally, the electronic device may also include a communication device and a display device for communication functions. It will be understood by those skilled in the art that the above structural description is only illustrative and does not limit the structure of the above electronic device. For example, the electronic device may also include more or fewer components than the above structural description, or have a configuration different from the above structural description.

[0025] The processor may include one or more processing units. For example, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP) chip, a microcontroller unit (MCU), a field-programmable gate array (FPGA), a neural-network processing unit (NPU), a tensor processing unit (TPU), an artificial intelligence (AI) type processor, and other processing devices. Among them, different processing units may be independent components or integrated into one or more processors. In some instances, the electronic device may also include one or more processors.

[0026] The memory can be used to store computer programs, such as storing computer programs corresponding to the data set comparison method in the embodiment of the present invention. The processor implements the above-mentioned data set comparison method by running the computer program stored in the memory. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the electronic device via a grid. Examples of the above-mentioned grid include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0027] Figure 1 is a flow chart of a method for comparing data sets according to one embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0028] Step S101, obtaining multiple data sets.

[0029] Specifically, clearly identify the data sets that need to be compared, such as data from two databases, two files, or two systems. Data sets can be structured (such as database tables) or unstructured (such as text files). Based on the comparison requirements, obtain the data sets that need to be compared.

[0030] Optionally, after the data set is acquired, preprocessing operations are required, such as data cleaning, data format conversion, and removal of redundant data, to ensure data consistency and accuracy of comparison.

[0031] Step S102: grouping multiple data sets based on the target database connection pool to obtain multiple data sets.

[0032] Specifically, the target database connection pool is used to uniformly manage and allocate database connections, provide database access resources for each thread, and ensure that database connections can be efficiently and fairly distributed to each thread when multiple threads process data sets in parallel. The target database connection pool pre-creates a certain number of database connections, which remain active and ready to serve comparison operations at any time, reducing the overhead of each new connection.

[0033] Specifically, according to the number of connections of the target database connection pool and the system resource situation, multiple data sets are grouped to form multiple data sets. For example, if the target database connection pool has N connections, the data sets are grouped into N data sets, and each set will be processed by a separate database connection. This grouping strategy ensures that each database connection has a corresponding data set to process, achieving balanced allocation of resources.

[0034] Specifically, when grouping multiple data sets to form multiple data sets, the logical relationship between the data sets needs to be considered, and closely related data sets should be grouped together to optimize data comparison efficiency.

[0035] Step S103: allocating the multiple data sets to the multiple target threads based on the target thread pool, wherein the multiple data sets correspond to the multiple target threads in a one-to-one manner.

[0036] Specifically, create a target thread pool to manage multiple threads that execute data comparison tasks in parallel. The size of the target thread pool should be set based on the system resource situation and the number of connections in the target database connection pool evaluated in advance to ensure that each target thread can obtain sufficient resources to execute tasks.

[0037] Specifically, a one-to-one correspondence is established between multiple target threads in the thread pool and multiple data sets, that is, each target thread is responsible for processing one data set. This correspondence ensures that data comparison tasks can be executed in parallel, and each thread independently processes the data set assigned to it, avoiding resource competition and conflicts during data processing.

[0038] Specifically, the system uses a task scheduling mechanism to assign multiple data sets to threads in the target thread pool. The scheduling strategy can be a simple round-robin allocation, or an intelligent allocation based on thread execution efficiency and data set size to achieve the best parallel processing effect.

[0039] Step S104 , comparing different data sets contained in the data set corresponding to each target thread in the multiple target threads to obtain a comparison result, wherein the comparison result is used for data migration of the multiple data sets.

[0040] Specifically, each target thread independently compares different data sets contained in the data set assigned to it. The comparison process involves executing SQL (Structured Query Language) query statements and using set operation principles to identify differences between data sets.

[0041] Specifically, after completing the comparison of the data sets, the target thread records the comparison results, including but not limited to difference information such as addition, deletion, and modification of data, as well as possible data inconsistencies or abnormalities.

[0042] Specifically, after all target threads complete the comparison of data sets, the comparison results of each thread are summarized, which involves integrating the scattered comparison results into a unified data difference report.

[0043] It should be noted that the comparison results are an important basis for decision-making in data migration projects. Based on the differences found in the comparison, the data migration strategy can be adjusted, such as adding data verification steps, optimizing the data transmission mechanism, or fixing data consistency issues, to improve the accuracy and efficiency of data migration.

[0044] In an embodiment of the present invention, multiple data sets are obtained; the multiple data sets are grouped based on the target database connection pool to obtain multiple data sets; the multiple data sets are allocated to multiple target threads based on the target thread pool, wherein the multiple data sets correspond one to one to the multiple target threads; different data sets contained in the data set corresponding to each target thread in the multiple target threads are compared to obtain a comparison result, wherein the comparison result is used for data migration of the multiple data sets. The present invention achieves the technical effect of improving the efficiency and accuracy of large-scale data comparison by combining multi-threaded concurrency and set comparison strategy, thereby solving the technical problem of low efficiency of large-scale data comparison in the prior art.

[0045] Optionally, in step S101, obtaining multiple data sets may include the following steps:

[0046] S201, obtaining multiple initial data sets;

[0047] Specifically, in the field of civil aviation, data sets exist from multiple sources, such as local databases, cloud databases, or other data storage systems. Identify all relevant data sources and ensure that all data sets that need to be compared can be accessed and extracted. Extract data from the above data sources to form multiple initial data sets, each of which represents a database table or a collection of specific business data. Data extraction should cover all data that needs to be compared to ensure the comprehensiveness and integrity of the data.

[0048] S202, performing data preprocessing operations on the multiple initial data sets to obtain multiple data sets, wherein the data preprocessing operations include data cleaning, data deduplication and data formatting.

[0049] Specifically, data cleaning operations are performed on multiple initial data sets, including: performing anomaly detection on each initial data set, identifying and processing erroneous data, null values, inconsistent formats, etc., to improve the accuracy of subsequent data comparison. Abnormal data includes but is not limited to incorrectly formatted data items, values ​​outside the normal range, duplicate records, or other non-compliant data; detected abnormal data is repaired or deleted to ensure the cleanliness of the data set. Repair involves filling in gaps with default values, correcting incorrect formats, or performing logical checks.

[0050] Specifically, data deduplication operations are performed on multiple initial data sets, including: selecting an appropriate deduplication algorithm to process duplicate records in the data set, which includes precise deduplication based on data key values ​​or approximate deduplication using fuzzy matching technology; performing deduplication operations on each data set to ensure the uniqueness of each record and avoid inaccurate results during the data comparison process.

[0051] Specifically, data formatting operations are performed on multiple initial data sets, including: converting the data in the data set into a unified format, including data type conversion, date and time format specifications, text encoding unification, etc., to avoid data comparison errors caused by inconsistent formats; maintaining the metadata of the data set, such as data type, field name, and data format, to ensure that the data can be correctly parsed and processed during data preprocessing and comparison.

[0052] Optionally, in step S102, grouping multiple data sets based on the target database connection pool to obtain multiple data sets may include the following steps:

[0053] S301, obtaining an initial database connection pool and a data set logical relationship between multiple data sets;

[0054] Specifically, before data comparison, you first need to design an initial database connection pool, which is used to manage connection resources with the database. The size, management strategy, and performance optimization of the connection pool are determined based on the expected data processing concurrency, system resource limitations, and database performance.

[0055] Specifically, the logical relationship between multiple data sets is analyzed, including the relevance of data tables, the hierarchical structure of data, or the order of business processes. This analysis helps maintain the logical consistency of data sets in the subsequent grouping process and avoids data comparison errors in parallel processing.

[0056] S302, performing an initialization operation on the initial database connection pool to obtain a target database connection pool;

[0057] Specifically, initialization operations are performed on the initial database connection pool, including: setting connection parameters such as database address, port, user name, password, and connection pool size, timeout period, maximum number of active connections and other properties; creating and pre-allocating a certain number of database connections, which will be reused, avoiding the overhead of creating a new connection each time data is compared, and improving database access efficiency.

[0058] S303, grouping multiple data sets based on the target database connection pool and the logical relationship between the data sets to obtain multiple data sets.

[0059] Specifically, according to the number of connections in the target database connection pool and the results of the data set logical relationship analysis, multiple data sets are grouped to form multiple data sets. The size and content of each data set should fully consider the processing capacity of the database connection and the logical correlation between data sets.

[0060] Optionally, in step S103, allocating multiple data sets to multiple target threads based on the target thread pool may include the following steps:

[0061] S401, obtaining an initial thread pool;

[0062] Specifically, the structure and parameters of the initial thread pool are designed, including the maximum number of threads in the thread pool, the idle thread retention time, the thread creation and destruction strategy, etc. When designing the initial thread pool, the system's resource limitations, the complexity of the data set, and the expected concurrency of the comparison tasks must also be considered. At the same time, the number of CPU (Central Processing Unit) cores, memory capacity, and I / O (Input / Output) performance available in the system are evaluated to ensure that the size of the thread pool can match the system resources to avoid excessive consumption or waste of resources.

[0063] S402, performing an initialization operation on the initial thread pool to obtain a target thread pool;

[0064] Specifically, initialization operations are performed on the initial thread pool, including: setting parameters such as thread pool size, thread priority and thread factory to ensure that the thread pool can meet the needs of large-scale data comparison; creating and pre-allocating a certain number of thread resources, which will be used to process data sets in parallel. The number of pre-allocated threads should match the number of connections in the target database connection pool to ensure that each data set has sufficient thread resources for comparison.

[0065] S403, generating multiple target threads based on the target thread pool;

[0066] Specifically, multiple target threads are created from the initialized target thread pool, and each thread will independently execute a data comparison task.

[0067] S404: Allocate multiple data sets to multiple target threads based on the target thread pool.

[0068] Specifically, according to the resource status of the threads in the thread pool and the size of the data set, a fair or weighted task allocation strategy is adopted to allocate each data set to a target thread for processing. The allocation strategy should ensure the balanced use of thread resources to avoid excessive thread load or idle resources. Each target thread independently uses the database connection allocated from the target database connection pool to perform the comparison task of the data set. Through concurrent processing, the system can process multiple data sets at the same time, significantly improving the efficiency of data comparison.

[0069] Optionally, in step S104, comparing different data sets contained in a data set corresponding to each target thread in the multiple target threads to obtain a comparison result may include the following steps:

[0070] S501, for any target thread, construct a target query statement;

[0071] Specifically, in each target thread, a target query statement is designed and constructed according to the structure and relationship between the first data set and the second data set in the corresponding data set. These query statements use the set operation principle to efficiently identify the difference between the two data sets.

[0072] For example, a SQL query statement is constructed to perform set operations. The design of the query statement is based on the mathematical principles of difference and union, as shown below:

[0073] (SELECT*FROM tableA WHERE NOT EXISTS(SELECT 1FROM tableB WHEREtableB.keyColumn=tableA.keyColumn))

[0074] UNION

[0075] (SELECT*FROM tableB WHERE NOT EXISTS(SELECT 1FROM tableA WHEREtableA.keyColumn=tableB.keyColumn));

[0076] This query statement first finds the records that exist in table A but not in table B, then finds the records that exist in table B but not in table A, and merges these two parts of the results to obtain the union and difference of the two data sets.

[0077] S502, processing the first data set and the second data set in the target thread according to the target query statement to determine a first difference set and a second difference set, wherein the first difference set is a data set included in the first data set but not included in the second data set, and the second difference set is a data set included in the second data set but not included in the first data set;

[0078] Specifically, in the target thread, the database connection allocated by the target database connection pool is used to execute the constructed target query statement to compare the first data set with the second data set in the data set. After the query is executed, the first difference set and the second difference set are determined by comparing the results. The first difference set is a set of data records that exist in the first data set but not in the second data set, and the second difference set is a set of data records that exist in the second data set but not in the first data set.

[0079] S503: Determine a first target difference set according to the first difference set and the second difference set.

[0080] Specifically, the first difference set is merged with the second difference set to obtain a first target difference set, that is, a set of all difference data between the two data sets. This merger can be achieved through a set operation UNION (an operator in a structured query statement) to ensure that the data records in the difference set are not repeated.

[0081] Optionally, the details of each data difference in the first target difference set, including the difference type (new addition, deletion or modification), the key value of the data record, the specific content of the difference data, etc., are recorded as part of the comparison result.

[0082] Optionally, the data set comparison method further includes: performing a data analysis operation on the target difference set to generate a difference report, wherein the data analysis operation includes data verification, data correction and data synchronization.

[0083] Specifically, data verification operations are performed on the target difference set, including: first, performing an integrity check on the data in the target difference set to confirm whether the data is complete and correct, including the integrity of the fields, the correctness of the data type, and whether the data range is reasonable; secondly, verifying the logical relationship between the data, such as foreign key constraints, data dependencies, and business rules, to ensure that the data in the difference set conforms to the expected business logic.

[0084] Specifically, data correction operations are performed on the target difference set, including: correcting or deleting erroneous data found in data verification, such as data with incorrect format, out-of-range or logical conflicts, to restore the consistency of the data set; converting the data in the target difference set into a unified format to facilitate subsequent data analysis and processing.

[0085] Specifically, performing data synchronization operations on the target difference set includes: using automated tools to perform data synchronization operations, synchronizing the corrected data to the target system, and recording key information during the synchronization process, such as synchronization time, synchronized data volume, and synchronization operation results.

[0086] Specifically, a difference report is generated based on key information in the data verification, data correction and data synchronization process. The report content should include but is not limited to: the name of the data set, the time window of the comparison, the type and number of data differences, the operation and results of data correction, the execution status and results of data synchronization, etc.

[0087] Optionally, data analysis and interpretation are provided in the difference report to help users understand the results of the data comparison, identify potential problem points, and provide suggestions for resolving problems.

[0088] Optionally, the data set comparison method may further include the following steps:

[0089] S601, obtaining multiple data sets at the current time point;

[0090] S602, comparing multiple data sets at the current time point with multiple data sets at a preset time point to obtain a second target difference set;

[0091] Exemplarily, at the current time point, a predefined SQL query statement or data extraction logic is used to obtain the latest multiple data sets from a local database or other data source. These data sets cover various aspects of civil aviation business, such as flight information, passenger data, financial records, etc. Use set operations, such as EXCEPT (an operator in a structured query statement), to determine the difference between the data set at the current time point and the data set at a preset time point. For any two data sets, the EXCEPT operation returns a set of records that exist in the first data set but not in the second data set, and so on, to obtain the second difference set of each data set. The second difference sets of multiple data sets are merged into the second target difference set, that is, the summary of the changes in all data sets between the current time point and the preset time point.

[0092] S603: Generate a target comparison report according to the first target difference set and the second target difference set.

[0093] Specifically, a target comparison report is generated based on the first target difference set and the second target difference set, and the report records an overview of data changes between the first target difference set and the second target difference set, including the amount of changed data, the type of change, and the time point of the change, etc., to help users understand the specific differences.

[0094] Optionally, when comparing data sets, an incremental comparison strategy can be used for comparison. The incremental comparison strategy is a method for optimizing the comparison process. It only focuses on the changes between the two data sets, thereby reducing unnecessary calculations. The specific comparison process is as follows:

[0095] Change detection: First, we need to detect changes between the two datasets since the last comparison, which can be achieved through timestamps, version numbers, or other change identifiers;

[0096] Extraction of changed data: Based on the results of change detection, the changed data records are extracted, and these records will be used as input for incremental comparison;

[0097] Execution of incremental comparison: comparison operation is performed on the extracted data records that have changed. Since the amount of data is small, the comparison process can be completed faster;

[0098] Integration of results: Integrate the results of the incremental alignment with the results of the full alignment to obtain a complete alignment report.

[0099] Through incremental comparison, the speed of big data comparison can be significantly improved and computing resources and time costs can be reduced.

[0100] Optionally, automated data governance is an important part of ensuring data consistency and integrity. Automated tools can be used to reduce manual intervention and improve the accuracy and efficiency of comparison. The implementation process of automated data governance is as follows:

[0101] Development of automated tools, development of automated tools for data collection, analysis and comparison, these tools can be scripts, software programs or services; regular collection of data, automated tools regularly collect data from MySQL database (an open source relational database), including full data and incremental data; automatic comparison of data, the collected data is automatically fed into the comparison process without human intervention; automatic reporting of results, after the comparison is completed, the automated tool generates a comparison report and presents it in an appropriate format, such as text, table or graphic.

[0102] Through the above implementation process, the problem of massive data comparison can be handled efficiently, and through automated optimization, the efficiency and accuracy of data governance can be significantly improved.

[0103] For example, Figure 2 is a schematic diagram of a method for comparing data sets according to one embodiment of the present invention, such as Figure 2 As shown, the specific process of big data comparison includes: starting the pre-developed automated comparison tool; creating a connection pool (PoolDB) to allocate database connections to each thread; creating a thread pool (ThreadPool) to allocate thread resources; starting the comparison algorithm, executing the comparison task, and obtaining the comparison result; disconnecting the database connection to complete the comparison.

[0104] In the embodiments of the present invention, a data set comparison device is also provided to implement the above embodiments and preferred implementation modes, which have been described and will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0105] Figure 3 is a structural block diagram of a data set comparison device 300 according to one embodiment of the present invention. Figure 3 As shown, the device includes: an acquisition module 301, used for acquiring multiple data sets; a grouping module 302, used for grouping the multiple data sets based on the target database connection pool to obtain multiple data sets; an allocation module 303, used for allocating the multiple data sets to multiple target threads based on the target thread pool, wherein the multiple data sets correspond one to one with the multiple target threads; a comparison module 304, used for comparing different data sets contained in the data set corresponding to each target thread in the multiple target threads to obtain a comparison result, wherein the comparison result is used for data migration of the multiple data sets.

[0106] Optionally, the acquisition module 301 is further used to: acquire multiple initial data sets; perform data preprocessing operations on the multiple initial data sets to obtain multiple data sets, wherein the data preprocessing operations include data cleaning, data deduplication and data formatting.

[0107] Optionally, the grouping module 302 is further used to: obtain the initial database connection pool and the data set logical relationship between multiple data sets; perform initialization operations on the initial database connection pool to obtain a target database connection pool; and group multiple data sets based on the target database connection pool and the data set logical relationship to obtain multiple data sets.

[0108] Optionally, the allocation module 303 is further used to: obtain an initial thread pool; perform an initialization operation on the initial thread pool to obtain a target thread pool; generate multiple target threads based on the target thread pool; and allocate multiple data sets to multiple target threads based on the target thread pool.

[0109] Optionally, the comparison module 304 is also used to: construct a target query statement for any target thread; process the first data set and the second data set in the target thread according to the target query statement to determine a first difference set and a second difference set, wherein the first difference set is a data set included in the first data set but not included in the second data set, and the second difference set is a data set included in the second data set but not included in the first data set; determine a first target difference set based on the first difference set and the second difference set.

[0110] Optionally, the comparison module 304 is further used to: perform a data analysis operation on the target difference set to generate a difference report, wherein the data analysis operation includes data verification, data correction and data synchronization.

[0111] Optionally, the comparison module 304 is also used to: obtain multiple data sets at the current time point; compare the multiple data sets at the current time point with multiple data sets at preset time points to obtain a second target difference set; and generate a target comparison report based on the first target difference set and the second target difference set.

[0112] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the data set comparison method described in any of the above embodiments.

[0113] Optionally, in this embodiment, the processor in the above electronic device may be configured to run a computer program to perform the following steps:

[0114] Step S101, obtaining multiple data sets.

[0115] Step S102: grouping multiple data sets based on the target database connection pool to obtain multiple data sets.

[0116] Step S103: allocating the multiple data sets to the multiple target threads based on the target thread pool, wherein the multiple data sets correspond to the multiple target threads in a one-to-one manner.

[0117] Step S104 , comparing different data sets contained in the data set corresponding to each target thread in the multiple target threads to obtain a comparison result, wherein the comparison result is used for data migration of the multiple data sets.

[0118] Optionally, the processor in the above-mentioned electronic device can be configured to run a computer program to perform the following steps: obtain multiple initial data sets; perform data preprocessing operations on the multiple initial data sets to obtain multiple data sets, wherein the data preprocessing operations include data cleaning, data deduplication and data formatting.

[0119] Optionally, the processor in the above-mentioned electronic device can be configured to run a computer program to perform the following steps: obtain an initial database connection pool and a data set logical relationship between multiple data sets; perform an initialization operation on the initial database connection pool to obtain a target database connection pool; and group multiple data sets based on the target database connection pool and the data set logical relationship to obtain multiple data sets.

[0120] Optionally, the processor in the above electronic device can be configured to run a computer program to perform the following steps: obtain an initial thread pool; perform initialization operations on the initial thread pool to obtain a target thread pool; generate multiple target threads based on the target thread pool; and allocate multiple data sets to multiple target threads based on the target thread pool.

[0121] Optionally, the processor in the above-mentioned electronic device can be configured to run a computer program to perform the following steps: for any target thread, construct a target query statement; process the first data set and the second data set in the target thread according to the target query statement to determine a first difference set and a second difference set, wherein the first difference set is a data set included in the first data set but not included in the second data set, and the second difference set is a data set included in the second data set but not included in the first data set; determine a first target difference set based on the first difference set and the second difference set.

[0122] Optionally, the processor in the above electronic device can be configured to run a computer program to perform the following steps: perform a data analysis operation on the target difference set to generate a difference report, wherein the data analysis operation includes data verification, data correction and data synchronization.

[0123] Optionally, the processor in the above-mentioned electronic device can be configured to run a computer program to perform the following steps: obtain multiple data sets at the current time point; compare the multiple data sets at the current time point with multiple data sets at a preset time point to obtain a second target difference set; and generate a target comparison report based on the first target difference set and the second target difference set.

[0124] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the data set comparison method described in any of the above embodiments when running.

[0125] Optionally, in this embodiment, the above-mentioned computer program may be configured to store a computer program for performing the following steps:

[0126] Step S101, obtaining multiple data sets.

[0127] Step S102: grouping multiple data sets based on the target database connection pool to obtain multiple data sets.

[0128] Step S103: allocating the multiple data sets to the multiple target threads based on the target thread pool, wherein the multiple data sets correspond to the multiple target threads in a one-to-one manner.

[0129] Step S104 , comparing different data sets contained in the data set corresponding to each target thread in the multiple target threads to obtain a comparison result, wherein the comparison result is used for data migration of the multiple data sets.

[0130] Optionally, the above-mentioned computer program can be configured to store a computer program for performing the following steps: obtaining multiple initial data sets; performing data preprocessing operations on the multiple initial data sets to obtain multiple data sets, wherein the data preprocessing operations include data cleaning, data deduplication and data formatting.

[0131] Optionally, the above-mentioned computer program can be configured to store a computer program for executing the following steps: obtaining an initial database connection pool and a data set logical relationship between multiple data sets; performing an initialization operation on the initial database connection pool to obtain a target database connection pool; and grouping multiple data sets based on the target database connection pool and the data set logical relationship to obtain multiple data sets.

[0132] Optionally, the above-mentioned computer program can be configured to store a computer program for performing the following steps: obtaining an initial thread pool; performing an initialization operation on the initial thread pool to obtain a target thread pool; generating multiple target threads based on the target thread pool; and allocating multiple data sets to multiple target threads based on the target thread pool.

[0133] Optionally, the above-mentioned computer program can be configured to store a computer program for performing the following steps: for any target thread, construct a target query statement; process the first data set and the second data set in the target thread according to the target query statement to determine a first difference set and a second difference set, wherein the first difference set is a data set included in the first data set but not included in the second data set, and the second difference set is a data set included in the second data set but not included in the first data set; determine a first target difference set based on the first difference set and the second difference set.

[0134] Optionally, the above-mentioned computer program may be configured to store a computer program for executing the following steps: performing a data analysis operation on a target difference set and generating a difference report, wherein the data analysis operation includes data verification, data correction and data synchronization.

[0135] Optionally, the above-mentioned computer program can be configured to store a computer program for performing the following steps: obtaining multiple data sets at the current time point; comparing the multiple data sets at the current time point with multiple data sets at a preset time point to obtain a second target difference set; and generating a target comparison report based on the first target difference set and the second target difference set.

[0136] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0137] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] In some embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0140] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0141] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a grid device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc., which can store program codes.

[0142] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for comparing data sets, characterized in that: include: Get multiple datasets; Grouping the multiple data sets based on the target database connection pool to obtain multiple data sets; Allocating the multiple data sets to multiple target threads based on a target thread pool, wherein the multiple data sets correspond to the multiple target threads in a one-to-one manner; Different data sets contained in the data set corresponding to each target thread in the multiple target threads are compared to obtain a comparison result, wherein the comparison result is used for data migration of the multiple data sets.

2. The data set comparison method according to claim 1, characterized in that: Acquiring the multiple data sets includes: Obtain multiple initial data sets; A data preprocessing operation is performed on the multiple initial data sets to obtain the multiple data sets, wherein the data preprocessing operation includes data cleaning, data deduplication and data formatting.

3. The data set comparison method according to claim 1, characterized in that: The multiple data sets are grouped based on the target database connection pool to obtain the multiple data sets including: Acquire the initial database connection pool and the data set logical relationship between the multiple data sets; Performing an initialization operation on the initial database connection pool to obtain a target database connection pool; The multiple data sets are grouped based on the target database connection pool and the logical relationship between the data sets to obtain the multiple data sets.

4. The method for comparing data sets according to claim 1, characterized in that: Allocating the multiple data sets to the multiple target threads based on the target thread pool includes: Get the initial thread pool; Performing an initialization operation on the initial thread pool to obtain a target thread pool; Generate the multiple target threads based on the target thread pool; The plurality of data sets are distributed to the plurality of target threads based on the target thread pool.

5. The method for comparing data sets according to claim 1, characterized in that: Comparing the different data sets contained in the data set corresponding to each target thread in the multiple target threads to obtain the comparison result includes: For any target thread, construct a target query statement; Processing the first data set and the second data set in the target thread according to the target query statement to determine a first difference set and a second difference set, wherein the first difference set is a data set included in the first data set but not included in the second data set, and the second difference set is a data set included in the second data set but not included in the first data set; A first target difference set is determined according to the first difference set and the second difference set.

6. The method for comparing data sets according to claim 5, characterized in that: The method further comprises: A data analysis operation is performed on the target difference set to generate a difference report, wherein the data analysis operation includes data verification, data correction and data synchronization.

7. The method for comparing data sets according to claim 5, characterized in that: The method further comprises: Get multiple data sets at the current time point; Comparing the multiple data sets at the current time point with the multiple data sets at the preset time point to obtain a second target difference set; A target comparison report is generated based on the first target difference set and the second target difference set.

8. A data set comparison device, characterized in that: include: An acquisition module, used to acquire multiple data sets; A grouping module, used for grouping the multiple data sets based on the target database connection pool to obtain multiple data sets; An allocating module, configured to allocate the plurality of data sets to a plurality of target threads based on a target thread pool, wherein the plurality of data sets correspond to the plurality of target threads in a one-to-one manner; The comparison module is used to compare different data sets contained in the data set corresponding to each target thread in the multiple target threads to obtain a comparison result, wherein the comparison result is used for data migration of the multiple data sets.

9. An electronic device, comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the data set comparison method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the data set comparison method described in any one of claims 1 to 7 when running on a computer or a processor.

Images