Data comparison method and device, equipment and storage medium

Abstract

The invention discloses a data comparison method and device, equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: obtaining source data and target data, and determining a key classification based on the source data; grouping the source data according to the key classification to obtain at least one source data group; creating at least one sub-process according to the key classification, and reading at least one target data group from the target data according to the key classification through the sub-process; performing parallel calculation on each source data group and each target data group to obtain a difference set; and combining the difference sets to obtain a comparison result, so that the comparison efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data comparison method, apparatus, device, and storage medium. Background Technology

[0002] With the development of technology, most companies need to migrate data from old platforms to new platforms to meet business needs. Since the two platforms typically organize and store data with different data structures, data comparison is required after the migration to ensure the integrity and accuracy of the migrated data.

[0003] Currently, manual comparison is generally used. However, manual comparison is inefficient.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a data comparison method, apparatus, device, and storage medium, aiming to solve the technical problem of low efficiency in current comparison methods.

[0006] To achieve the above objectives, this application proposes a data comparison method, the method comprising:

[0007] Acquire source data and target data, and determine key categories based on the source data;

[0008] The source data is grouped according to the key classification to obtain at least one source data group;

[0009] At least one subprocess is created based on the key classification, and at least one target data group is read from the target data according to the key classification through the subprocess;

[0010] The difference set is obtained by parallel computation of each of the source data groups and each of the target data groups;

[0011] The difference sets are merged to obtain the comparison results.

[0012] In one embodiment, the step of determining the key classification based on the source data includes:

[0013] Get the total number of source data and the total number of target data;

[0014] Compare the total number of source data with the total number of target data;

[0015] If the total number of source data and the total number of target data are inconsistent, a mark will be made in the data record and / or comparison report.

[0016] In one embodiment, the steps of obtaining the total number of source data and the total number of target data are preceded by:

[0017] Obtain source data samples and target data samples;

[0018] The source data sample and the target data sample are input into the optimized general large language model to obtain the target data mapping relationship and the target script;

[0019] The file type selection parser reads different types of files from the source data to obtain the read data.

[0020] A source data list is constructed based on the read data;

[0021] The target data is obtained by migrating the source data list using the target data mapping relationship and the target script.

[0022] In one embodiment, the step of constructing a source data list based on the read data includes:

[0023] Based on the read data, a source data list is constructed using a generator and / or lazy computation.

[0024] In one embodiment, the step of inputting the source data sample and the target data sample into the optimized general large language model to obtain the target data mapping relationship and the target script includes:

[0025] Generate language model prompt words based on the source data samples and the target data samples;

[0026] Input the language model prompts into a preset general large language model to obtain the initial data mapping relationship and the initial script;

[0027] Based on the source data sample, the target data sample, the initial data mapping relationship, and the initial script, the general large model is optimized in multiple rounds until the target data mapping relationship and the target script are generated through the optimized general large model.

[0028] In one embodiment, the step of performing multiple rounds of optimization on the general large model based on the source data sample, the target data sample, the initial data mapping relationship, and the initial script until the target data mapping relationship and the target script are generated through the optimized general large model includes:

[0029] The initial data mapping relationship and the initial script are reviewed to obtain the review results;

[0030] The source data sample, the target data sample, and the review result are input into the general large model to obtain a new primary data mapping relationship and a new primary script. The following steps are performed: the initial data mapping relationship and the initial script are reviewed to obtain the review result, until the target data mapping relationship and the target script are generated through the optimized general large model.

[0031] In one embodiment, the data comparison system includes a data platform, and the acquisition of source data includes:

[0032] Install a pre-developed script management plugin on the data platform;

[0033] In response to the source data acquisition instruction, obtain the application name of the application to which the source data belongs;

[0034] Based on the application name, the valid configuration file of the application is exported through the script management plugin, and the valid configuration file is used as the source data.

[0035] Furthermore, to achieve the above objectives, this application also proposes a data comparison device, which includes:

[0036] The data acquisition module is used to acquire source data and target data, and determine key categories based on the source data;

[0037] The source data grouping module is used to group the source data according to the key classification to obtain at least one source data group;

[0038] The target data grouping module is used to create at least one sub-process according to the key classification and read at least one target data group from the target data according to the key classification through the sub-process;

[0039] The calculation module is used to perform parallel calculations on each of the source data groups and each of the target data groups to obtain the difference set;

[0040] The merging module is used to merge the difference sets to obtain the comparison results.

[0041] In addition, to achieve the above objectives, this application also proposes a data comparison device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data comparison method as described above.

[0042] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the data comparison method described above.

[0043] One or more technical solutions proposed in this application have at least the following technical effects:

[0044] This application groups the source data according to key categories, making each group relatively small in size and similar. When calculating the difference set, only a local comparison needs to be performed on each group, reducing the computational load of a single comparison and further improving processing speed. Furthermore, this application creates at least one subprocess based on the key categories and uses these subprocesses to read the target data in parallel to obtain target data groups. The difference set is then calculated in parallel for the source data groups and the target data groups, improving data comparison efficiency. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a system architecture diagram of the data comparison system for this application;

[0048] Figure 2 This is a flowchart illustrating the data comparison method of this application in Embodiment 1.

[0049] Figure 3 This is a flowchart illustrating the process of data comparison in parallel for multiple subtasks in this application.

[0050] Figure 4 This is a flowchart illustrating the process of obtaining the target data mapping relationship and target script in this application;

[0051] Figure 5 This is a schematic diagram of the overall process of the data comparison method in this application.

[0052] Figure 6 This is a schematic diagram of the module structure of the data comparison device in an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the data comparison method in the embodiments of this application.

[0054] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0056] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The main solution of this application embodiment is as follows: acquire source data and target data, and determine key classifications based on the source data; group the source data according to the key classifications to obtain at least one source data group; create at least one subprocess according to the key classifications and read at least one target data group from the target data according to the key classifications through the subprocesses; perform parallel calculations on each source data group and each target data group to obtain a difference set; merge the difference sets to obtain a comparison result.

[0058] In this embodiment, for ease of description, the following description will focus on the data comparison system as the main execution entity.

[0059] Due to the low efficiency of data comparison using existing technologies.

[0060] This application provides a solution that groups source data according to key categories, making each group relatively small in size and similar. When calculating the difference set, only a local comparison needs to be performed on each group, reducing the computational load of a single comparison and further improving processing speed. Furthermore, this application creates at least one sub-process based on the key categories and uses these sub-processes to read the target data in parallel to obtain target data groups. The difference set is then calculated in parallel for both the source data groups and the target data groups, improving data comparison efficiency.

[0061] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone; or an electronic device or data comparison device capable of performing the above functions; or an electronic system or data comparison system capable of performing the above functions. The following description uses a data comparison system as an example to illustrate this embodiment and the subsequent embodiments.

[0062] This application adopts a layered architecture design, referring to... Figure 1 , Figure 1 This is a system architecture diagram of the data comparison system for this application, which mainly includes a data acquisition layer, a core processing layer, and a result display layer.

[0063] The data acquisition layer includes a multi-source adapter, a file collector, and a file parser.

[0064] Among them, the multi-source adapter supports data collection from various data sources, including direct database connection, API interface, file import, etc.; the file collector is used to develop front-end plugins to help users export application data files with one click; the file parser includes EXCEL file parser, CSV file parser, JSON / XML parser, custom format parser, etc.

[0065] The core processing layer includes an intelligent mapping engine, a data standardization engine, and a multi-dimensional comparison engine.

[0066] Among them, the intelligent mapping engine is used to identify the mapping rules between source data and target data using a large language model, and outputs the rule transformation (data mapping) logic, which can be manually adjusted; the data standardization engine processes the source data and target data according to the mapping rules and standardizes the comparison data structure; the multi-dimensional comparison engine is used for summary comparison, detailed comparison, and difference analysis.

[0067] The results display layer includes a report generator, API services, and command-line interfaces.

[0068] The report generator is used to generate visual reports based on the comparison results, and the report format supports Excel, JSON and other formats; the API service is used to provide key technology APIs such as rule parsing and data comparison; the command line interface can be executed and called directly from the command line.

[0069] If the existing data platform does not meet compliance requirements, the data from the original data platform needs to be migrated to the new data platform.

[0070] Business system data is typically organized and stored using different data structures depending on its type. The storage methods of the original data platform and the new data platform differ significantly, requiring customized data processing rules based on different data types. These rules include, but are not limited to, complex operations such as data format conversion, data value mapping, and data classification and reorganization.

[0071] After data migration, data comparison testing is required to ensure the integrity and accuracy of the data.

[0072] The current mainstream data migration comparison methods mainly include methods (1) to (3):

[0073] (1) Manual comparison: The data list is manually checked by the testers;

[0074] (2) Custom SQL script comparison: Use the basic script to compare values;

[0075] (3) Specialized tools for comparison: database comparison tools, such as Redgate SQL Compare; file comparison tools, such as Beyond Compare.

[0076] However, the existing technology has some limitations, including those listed in (a) to (e) below:

[0077] (a) Inefficiency: Manual comparison is time-consuming when processing large-scale data; inconsistent processing rules for different data types lead to high costs in establishing data mapping rules. Source data is directly read from the database, resulting in unclear data structures and lengthy parsing times. Database read / write times increase rapidly with data volume, and connections are prone to timeouts.

[0078] (b) Insufficient accuracy: Manual comparison is prone to errors due to fatigue or negligence; existing tools can only compare the amount of data when the data structures are inconsistent, and cannot clearly compare information such as data structure and data values. The development and migration process uses SQL scripts for processing, and if the testing uses the same scripts, it will not achieve the desired comparison and testing effect.

[0079] (c) Poor versatility: Existing tools are usually designed for specific systems and are difficult to adapt to the changing needs of different application scenarios, and cannot support data comparison of multiple database types. Database comparison tools only support comparison of structural consistency.

[0080] (d) Difficult to maintain: Data mapping relationships are difficult to maintain and lack flexible configuration mechanisms.

[0081] (e) Poor scalability: Adding new data types requires code modification; comparison rules are fixed and difficult to adjust; performance drops sharply as the amount of data increases.

[0082] Addressing technical challenges such as achieving cross-platform and cross-system universal parameter comparison, improving the efficiency and accuracy of large-scale parameter comparison, designing flexible configuration mechanisms to adapt to different business scenarios, reducing the technical threshold of the parameter comparison process, and achieving intelligent parameter difference analysis and problem localization, this application provides a data comparison method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the data comparison method of this application in Embodiment 1.

[0083] In this embodiment, the data comparison method includes steps S120 to S160:

[0084] Step S120: Obtain source data and target data, and determine key classifications based on the source data;

[0085] Key classification refers to the core features or attributes used when grouping source data. It divides the source data into subsets with similar business logic, structural characteristics, or data features. The selection of key classifications can be based on principles such as business relevance, data distinguishability, and stability.

[0086] The source data can come from old systems, databases, files, etc., while the target data is the new data obtained after migrating the source data.

[0087] Step S130: Group the source data according to the key classification to obtain at least one source data group;

[0088] Among them, reference Figure 3 , Figure 3 This is a flowchart illustrating the parallel data comparison process of multiple subtasks in this application. Source data with the same key classification are grouped together. By grouping, large-scale source data can be decomposed into multiple smaller subsets with relatively simple structures, which facilitates subsequent processing of each group and improves the efficiency and targeting of data processing.

[0089] Step S140: Create at least one subprocess according to the key classification and read at least one target data group from the target data according to the key classification through the subprocess;

[0090] Among them, target data grouping is obtained by grouping the target data.

[0091] The purpose of creating subprocesses based on key categories is to achieve parallel processing, fully utilize the computing power of multi-core processors, and improve the efficiency of data reading and processing. For example, if there are three key categories, three subprocesses can be created, each responsible for processing the target data group corresponding to one key category. Since the subprocesses run in parallel, they can access the target data simultaneously, avoiding the waiting time of serial reading and greatly shortening the total data reading time.

[0092] Step S150: The difference set is obtained by parallel calculation of each of the source data groups and each of the target data groups;

[0093] Specifically, for each corresponding source data group and target data group, the difference between them is calculated. Due to the parallel processing approach, multiple difference calculation tasks can be executed simultaneously in different subprocesses or threads, further improving computational efficiency.

[0094] The specific difference calculation method can be selected according to the data type and storage format.

[0095] As one implementation method, for simple list data, a nested loop can be used to traverse the two groups to find elements that exist in the source data group but not in the target data group.

[0096] As another implementation method, the difference set can be calculated using the EXCEPT operation in the SQL statement for data in the database.

[0097] Step S160: Merge the difference sets to obtain the comparison results.

[0098] After calculating the difference between each source data group and the target data group, these differences need to be merged into a final comparison result.

[0099] The merging process can be either a simple summary or a more complex process, depending on the actual needs.

[0100] One approach is to merge the elements from all the difference sets into a single set to determine which data differences exist overall.

[0101] As another implementation method, each difference set can be associated with its corresponding key classification information to generate a structured result containing classification information and difference data, so as to statistically analyze the differences according to the key classification.

[0102] The embodiments of this application are applicable to large amounts of data and are more compatible with multi-core CPUs.

[0103] In addition, multi-dimensional data comparisons (such as name, value, type, etc.) can be performed on the source data and target data, and the details of the comparison differences can be recorded to generate a difference score.

[0104] As one implementation method, target data can be obtained from the target database, database technology fields such as update time and version number can be removed from the target data, set operations can be used to calculate the difference between the source data and the target data, and a comparison algorithm can be used to calculate the similarity score between the two. This implementation method is suitable for lightweight data.

[0105] The specific difference calculation method can be selected according to the data type and storage format.

[0106] As one implementation method, for simple list data, a nested loop can be used to traverse the two groups to find elements that exist in the source data group but not in the target data group.

[0107] As another implementation method, the difference set can be calculated using the EXCEPT operation in the SQL statement for data in the database.

[0108] The comparison algorithm can include string similarity algorithms, such as edit distance algorithm and cosine similarity algorithm. The comparison algorithm can also include numerical similarity algorithms, such as Euclidean distance algorithm and Pearson correlation coefficient algorithm.

[0109] In addition, users can choose the output method of the results through development configuration before starting a user task.

[0110] As one implementation method, analysis reports can be printed or generated via command line. Data comparison results include summary results, detailed results, and difference analysis results.

[0111] Optionally, the original file can be deleted after the data comparison is completed.

[0112] This embodiment provides a data comparison method. By grouping source data according to key categories, each group has a relatively small data size and high similarity. When calculating the difference set, only a local comparison needs to be performed on each group, reducing the computational load of a single comparison and further improving processing speed. Furthermore, this application creates at least one sub-process based on the key categories and reads the target data in parallel through these sub-processes to obtain target data groups. The difference set is then calculated in parallel for both the source data groups and the target data groups, further improving data comparison efficiency.

[0113] This application embodiment also employs multi-process parallel processing technology to intelligently select data acquisition and processing methods based on the data volume, enabling rapid comparison of batch parameters and reducing the time and workload of manual comparison. This application embodiment uses automation tools to reduce misjudgments and omissions during manual comparison, and accurately locates discrepancies through summary comparison and detailed comparison. This application embodiment also provides rich comparison result displays, intuitive problem location functions, and supports multiple report output formats.

[0114] Based on Embodiment 1 of this application, in Embodiment 2 of this application, the content that is the same as or similar to that in Embodiment 1 can be referred to the above description, and will not be repeated hereafter. Furthermore, before determining the key classification based on the source data, the data comparison method further includes steps S90 to S110:

[0115] Step S90: Obtain the total number of source data and the total number of target data;

[0116] Step S100: Compare the total number of source data with the total number of target data;

[0117] Step S110: If the total number of source data and the total number of target data are inconsistent, a mark is made in the data record and / or comparison report.

[0118] It should be noted that the embodiments of this application are used for summarizing and comparing source data and target data.

[0119] When a discrepancy is found between the total number of source data and the total number of target data, it can be marked for subsequent processing and analysis. For example, a specific flag field can be added to the data record, and the value of this field can be set to an identifier indicating inconsistency; or special colors, symbols, or text can be used to mark discrepancies in quantity in the generated comparison report; or inconsistent data records can be extracted and stored separately in a specific file or database table.

[0120] This application embodiment, by performing a total count comparison during the data comparison process, can quickly discover the difference in quantity between the source data and the target data. This allows potential problems to be identified in the early stages of data processing, avoiding the waste of a lot of time and computing resources when conducting detailed content comparisons later.

[0121] Based on Embodiment 2 of this application, in Embodiment 3 of this application, the content that is the same as or similar to Embodiment 2 can be referred to the above description, and will not be repeated hereafter. Based on this, before step S90, which obtains the total number of source data and the total number of target data, the data comparison method further includes steps S10, S50 to S80:

[0122] Step S10: Obtain source data samples and target data samples;

[0123] Among them, a small number of data from different data categories can be selected from the source data as source data samples for data mapping and recognition;

[0124] The selected source data sample can be used to retrieve the corresponding migrated data from the target database as the target data sample for data mapping identification.

[0125] Step S50: Input the source data sample and the target data sample into the optimized general large language model to obtain the target data mapping relationship and the target script;

[0126] As one implementation method, an optimized general large language model can be used to identify the mapping relationship between source data samples and target data samples, and automatically generate Python scripts for data processing.

[0127] The mapping relationship defines the correspondence rules between source data and target data. Since the source data and target data may differ in structure, field names, data types, etc., the mapping relationship tells the system how to convert each field of the source data into the corresponding field of the target data.

[0128] The target script is an executable code or set of instructions that contains the specific logic and operational steps for data migration. It utilizes the target data mapping relationship to process each record in the source data list, converting it into a format that conforms to the target data structure requirements, and ultimately storing the converted data in the target location. The script can be written in various programming languages ​​(such as Python, Java, SQL, etc.), depending on the data migration environment and requirements.

[0129] Step S60: The file type selection parser reads different types of files in the source data to obtain the read data;

[0130] Different types of files (such as CSV, JSON, Excel, XML, etc.) have their own unique structures and data storage methods, thus requiring different parsers to accurately read the information within them.

[0131] For example, CSV files are typically parsed using specialized CSV parsing libraries, such as the csv module in Python, which provides methods for reading CSV files and can correctly handle data rows and columns separated by specific delimiters (such as commas).

[0132] For example, Python's built-in json module is used to parse JSON files, which can easily convert JSON-formatted strings into Python objects (such as dictionaries, lists, etc.) for convenient subsequent processing.

[0133] For example, for Excel files (.xlsx or .xls format), you can use Python libraries such as openpyxl or pandas. openpyxl focuses on reading and writing Excel files, while pandas provides more powerful data processing capabilities, making it easy to read Excel files and convert them into DataFrame structures.

[0134] For example, when processing XML files, commonly used parsers include xml.etree.ElementTree (a module in the Python standard library) or the lxml library, which can parse XML documents into a tree structure, making it easy to traverse and extract the required data.

[0135] After determining the file type by checking the file extension or using the file header information, the appropriate parser is selected to read the file content.

[0136] Step S70: Construct a source data list based on the read data;

[0137] After reading the file using the selected parser, the read data is organized according to certain rules and structures to form a source data list.

[0138] As one implementation, the source data list can be a collection containing multiple data items, each representing a record or a data unit in the file. For example, for a CSV file, each row of data can be added to the source data list as a data item; for a JSON file, the parsed dictionary or list can be directly used as an element of the source data list.

[0139] Step S80: The source data list is migrated using the target data mapping relationship and the target script to obtain the target data.

[0140] The target script can utilize the target data mapping relationship to process each record in the source data list, convert it into a format that meets the requirements of the target data structure, and finally store the converted data in the target location.

[0141] Assume the data platform can export four data type files: bizConfig.xlsx; dictCommConfig.xlsx; bizCommConfig.xlsx; and commConfig.xlsx.

[0142] The format of the target database after migration can be simply described as shown in Table 1:

[0143] Table 1

[0144]

[0145] The generated mapping relationships are shown in the following examples, depending on the file:

[0146] Input: bizConfig.xlsx; dictCommConfig.xlsx; bizCommConfig.xlsx; commmConfig.xlsx.

[0147] Output: A list of dictionaries {PARAM_GRUOP, PARAM_KEY, PARAM_VALUE}.

[0148] Processing logic:

[0149] Processing commConfig: PARAM_GROUP = commConfig, PARAM_KEY = data type.condition1.condition2..., PARAM_VALUE = configuration value;

[0150] Process bizCommConfig.tree: Read the bizCommConfig.tree file of the corresponding application, PARAM_GROUP = 'bizCommConfig.tree', PARAM_KEY = data type.condition1.condition2..., PARAM_VALUE = configuration value;

[0151] dictCommConfig.tree: Reads the corresponding application's dictCommConfig.tree file, PARAM_GROUP = 'dictCommConfig.tree', PARAM_KEY = data type.condition1.condition2..., PARAM_VALUE = configuration value;

[0152] `bizConfig.oto`: Reads the `bizConfig` file for the corresponding application and parses each sheet page sequentially. Assuming the data class is A, and `PARAM_GROUP = A`. It retrieves all `KEYCOL` values ​​and their configuration values ​​for A from `commConfig.tree`, constructing `PARAM_KEY = {"bizno":"A", KEYCOL: the column value corresponding to the `KEYCOL` configuration value on the A tab of the `bizConfig` file}` (there may be multiple `KEYCOL` values; the header of the `bizConfig` file contains the configuration values ​​of `KEYCOL` and `VALUECOL`). It also retrieves all `VALUECOL` values ​​and their configuration values ​​for A from `commConfig.tree`, constructing `PARAM_VALUE = {VALUECOL1: the column value corresponding to the matching value of `VALUECOL1`, `VALUECOL2: the column value corresponding to the configuration value of `VALUECOL2`.

[0153] As one implementation method, when performing data migration and comparison between different systems on the same original platform, the target data mapping relationship and / or target script can be directly reused. Users can choose to use the generated target script for comparison testing of new applications, or perform optimization after selection. This achieves universal comparison within the same platform.

[0154] This application embodiment, through target data mapping relationships, can transform source data from one structure to another, enabling the data to adapt to the requirements of the target system. Furthermore, this application embodiment has the ability to support multiple database types, can adapt to different system application scenarios, and employs a flexibly extensible text type parser.

[0155] Based on Embodiment 3 of this application, in Embodiment 4 of this application, the content that is the same as or similar to Embodiment 3 can be referred to the above description, and will not be repeated hereafter. Based on this, step S70, which constructs the source data list based on the read data, further includes step S71:

[0156] Step S71: Based on the read data, construct a source data list through a generator and / or lazy computation.

[0157] A generator is a special type of iterator that uses the `yield` keyword to produce values ​​one by one, rather than returning all the results at once. Using generators avoids loading all the data into memory at once when reading file data and constructing a list of source data.

[0158] Lazy computation is a technique of deferred computation, meaning that computation is performed only when the result is actually needed. In the context of data processing, it means that when constructing a list of source data, complex calculations or transformations are not immediately performed on all the data, but rather these operations are postponed until the data is actually needed.

[0159] When processing large datasets or complex computational tasks, loading all content into memory at once often leads to a sharp increase in memory usage, reaching the system's memory limit, and even triggering memory overflow errors. However, when using generators and lazy computation, data is generated and loaded incrementally on demand, with only the data needed for the current processing always remaining in memory, effectively reducing peak memory usage. This application, by utilizing generators and lazy computation, avoids loading all content into memory at once, thus preventing unnecessary memory reservations and waste.

[0160] Based on Embodiment 3 of this application, in Embodiment 5 of this application, the content that is the same as or similar to Embodiment 3 can be referred to the above description, and will not be repeated hereafter. Based on this, step S50, which involves inputting the source data sample and the target data sample into the optimized general large language model to obtain the target data mapping relationship and the target script, includes steps S20 to S40:

[0161] Step S20: Generate language model prompt words based on the source data sample and the target data sample;

[0162] Among them, language model prompts can provide necessary background information, such as the structure of the source data file and the target database, and the general meaning of the data.

[0163] For example, language model prompts could be:

[0164] Input: source data, target data.

[0165] Output: Mapping relationship description, data mapping processing script.

[0166] Task: Analyze the input source data and target data, and identify their mapping relationship. Implement the data mapping processing logic using a Python script.

[0167] Require:

[0168] 1. The logic for mapping relationships between output data.

[0169] 2. The Python script conforms to the logic of the Python language.

[0170] Step S30: Input the language model prompt words into a preset general large language model to obtain the initial data mapping relationship and the initial script;

[0171] Step S40: Based on the source data sample, the target data sample, the initial data mapping relationship, and the initial script, perform multiple rounds of optimization on the general large model until the target data mapping relationship and the target script are generated through the optimized general large model.

[0172] You can refer to Figure 4 , Figure 4 This is a flowchart illustrating the process of obtaining the target data mapping relationship and target script in this application.

[0173] As one implementation method, the initial mapping relationship and initial script generated by the general large model can be manually modified or supplemented, and then the general large model can be optimized in multiple rounds based on the modified or supplemented mapping relationship and script, source data samples, and target data samples.

[0174] This application's embodiments employ multiple rounds of optimization based on source data samples, target data samples, initial data mapping relationships, and initial scripts. This allows the model to continuously correct errors and deficiencies in the initial results. Each round of optimization can be viewed as a training and fine-tuning of the model, enabling it to gradually learn more accurate data mapping rules and script generation strategies. After multiple rounds of optimization, the final generated target data mapping relationship and target script more accurately reflect the correspondence between the source and target data, meeting the needs of practical applications. This application simplifies the script writing process by introducing a general large-scale model to identify data mapping relationships.

[0175] Based on Embodiment 5 of this application, in Embodiment 6 of this application, the content that is the same as or similar to Embodiment 5 can be referred to the above description, and will not be repeated hereafter. Based on this, step S40, which optimizes the general large model multiple times based on the source data sample, the target data sample, the initial data mapping relationship, and the initial script, until the target data mapping relationship and the target script are generated through the optimized general large model, includes steps S41 and S42:

[0176] Step S41: Review the initial data mapping relationship and the initial script to obtain the review result;

[0177] Step S42: Input the source data sample, the target data sample, and the review result into the general large model to obtain a new primary data mapping relationship and a new primary script, and perform the following steps: review the initial data mapping relationship and the initial script to obtain the review result, until the target data mapping relationship and the target script are generated through the optimized general large model.

[0178] The language model prompts mentioned above can be input into a general large model to obtain the initial mapping relationship and initial script generated by the model.

[0179] Next, carefully review the mapping relationships and scripts generated by the general model to identify potential problems, such as inaccurate mappings, script logic errors, or inability to handle specific situations. Provide clear and specific feedback to the model, along with more example data or relevant information to help the model better understand the task. Repeat this feedback process, analyzing and providing feedback on the problems again based on each new mapping relationship and script generated by the general model, until the generated mapping relationships and scripts are accurate and usable.

[0180] As one implementation method, detailed records can be kept of all modifications and adjustments made, including the reasons for the modifications, the differences before and after the modifications, and the effects of the modifications. These records will serve as feedback data to optimize the language model.

[0181] This application's embodiments employ multiple rounds of optimization based on source data samples, target data samples, initial data mapping relationships, and initial scripts. This allows the model to continuously correct errors and deficiencies in the initial results. Each round of optimization can be viewed as a training and fine-tuning of the model, enabling it to gradually learn more accurate data mapping rules and script generation strategies. After multiple rounds of optimization, the final generated target data mapping relationship and target script more accurately reflect the correspondence between the source and target data, meeting the needs of practical applications.

[0182] Based on Embodiment 1 of this application, in Embodiment 7 of this application, the content that is the same as or similar to Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, it is applied to a data comparison system, the data comparison system including a data platform, and the acquisition of source data includes steps S121 to S123:

[0183] Step S121: Install the pre-developed script management plugin on the data platform;

[0184] The script management plugin is essentially a container that allows users to install and manage various user scripts. These user scripts are programs written in JavaScript that can run automatically when a user browses a webpage, modifying, enhancing, or customizing the content, style, or behavior of the webpage, thereby providing the user with a personalized web browsing experience.

[0185] The script management plugin can include the Tampermonkey plugin.

[0186] It should be noted that the embodiments of this application implement data acquisition operations.

[0187] As one implementation method, users can download data from the data platform and save it to the server's file directory. The data platform provides a function to export configurations according to data type.

[0188] As another implementation method, users can also use a pre-developed Tampermonkey plugin. After installing the Tampermonkey plugin, entering the application name will allow users to download configuration files for all data categories with a single click.

[0189] Step S122: In response to the source data acquisition instruction, obtain the application name of the application to which the source data belongs;

[0190] Step S123: Based on the application name, export the application's valid configuration file through the script management plugin, and use the valid configuration file as the source data.

[0191] For example, after installing the Tampermonkey plugin, log in to the data platform, and the interface will display the "One-click Export Data Configuration" button. Click the button, and an application information input pop-up window will appear. After entering the application name, click OK to export all valid configuration files of the application and upload them to the server file directory.

[0192] This application embodiment exports source data through a script management plugin, ensuring that the directly exported data file format is complete and has good readability and usability. However, if data is read directly from the source database, data parsing becomes difficult and readability is poor without knowledge of data storage rules; if the data volume is too large, database reading becomes slow. This application embodiment utilizes a configurable plugin to assist in one-click export of source data files, providing reusable configuration options and lowering the technical barrier to entry.

[0193] For example, to help understand the implementation process of the data comparison method in the above embodiments, please refer to... Figure 5 , Figure 5 This is a schematic diagram of the overall process of the data comparison method in this application. Specifically, the process includes the following steps A1 to A11:

[0194] A1 allows you to export multiple parameter files / data files from the original platform with a single click.

[0195] A2 performs preliminary parsing of the parameter file / data file to obtain source data samples.

[0196] A3 collects training data from the target database according to parameter type / data type.

[0197] A4, obtain target data samples based on training data.

[0198] A5 inputs the source data samples and target data samples into the optimized general large language model to obtain the initial data mapping relationship and the initial script.

[0199] A6, Script Optimization: Based on the source data sample, the target data sample, the initial data mapping relationship, and the initial script, perform multiple rounds of optimization on the general large model until the target data mapping relationship and the target script are generated through the optimized general large model.

[0200] A7, Source File Processing: Data migration is performed on the source data through the target data mapping relationship and the target script to obtain the target data.

[0201] A8 reads target data from the target database.

[0202] A9 summarizes and verifies the source and target data.

[0203] A10, perform detailed verification of source data and target data.

[0204] A11 displays the summary verification results and detailed verification results.

[0205] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data comparison method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0206] This application also provides a data comparison device, please refer to... Figure 6 The data comparison device includes:

[0207] The data acquisition module 10 is used to acquire source data and target data, and determine key classifications based on the source data;

[0208] The source data grouping module 20 is used to group the source data according to the key classification to obtain at least one source data group;

[0209] The target data grouping module 30 is used to create at least one subprocess according to the key classification and read at least one target data group from the target data according to the key classification through the subprocess;

[0210] Calculation module 40 is used to perform parallel calculations on each of the source data groups and each of the target data groups to obtain the difference set;

[0211] The merging module 50 is used to merge the difference sets to obtain the comparison results.

[0212] The data comparison device provided in this application, employing the data comparison method described in the above embodiments, can solve the technical problem of low efficiency in current comparison methods. Compared with the prior art, the beneficial effects of the data comparison device provided in this application are the same as those of the data comparison method provided in the above embodiments, and other technical features in the data comparison device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0213] This application provides a data comparison device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data comparison method in Embodiment 1 above.

[0214] Reference below Figure 7The diagram illustrates a structural schematic of a data comparison device suitable for implementing embodiments of this application. The data comparison device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The data comparison device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0215] like Figure 7 As shown, the data comparison device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the data comparison device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the data comparison device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show data comparison devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0216] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0217] The data comparison device provided in this application, employing the data comparison method described in the above embodiments, can solve the technical problem of low efficiency in current comparison methods. Compared with the prior art, the beneficial effects of the data comparison device provided in this application are the same as those of the data comparison method provided in the above embodiments, and other technical features of this data comparison device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0218] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0219] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0220] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data comparison method in the above embodiments.

[0221] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0222] The aforementioned computer-readable storage medium may be included in the data comparison device; or it may exist independently and not assembled into the data comparison device.

[0223] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a data comparison device, cause the data comparison device to: acquire source data and target data, and determine a key classification based on the source data; group the source data according to the key classification to obtain at least one source data group; create at least one subprocess according to the key classification and read at least one target data group from the target data according to the key classification through the subprocess; perform parallel computation on each source data group and each target data group to obtain a difference set; and merge the difference sets to obtain a comparison result.

[0224] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0225] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0226] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0227] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data comparison method, thereby solving the technical problem of low efficiency in current comparison methods. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data comparison method provided in the above embodiments, and will not be repeated here.

[0228] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A data comparison method, characterized in that, The method includes: Acquire source data and target data, and determine key classifications based on the source data; The source data is grouped according to the key classification to obtain at least one source data group; At least one subprocess is created based on the key classification, and at least one target data group is read from the target data according to the key classification through the subprocess; The difference set is obtained by parallel computation of each of the source data groups and each of the target data groups; The difference sets are merged to obtain the comparison results.

2. The method as described in claim 1, characterized in that, Prior to the step of determining the key classification based on the source data, the following is included: Get the total number of source data and the total number of target data; Compare the total number of source data with the total number of target data; If the total number of source data and the total number of target data are inconsistent, a mark will be made in the data record and / or comparison report.

3. The method as described in claim 1, characterized in that, Prior to the steps of acquiring source data and target data, the following are included: Obtain source data samples and target data samples; The source data sample and the target data sample are input into the optimized general large language model to obtain the target data mapping relationship and the target script; The file type selection parser reads different types of files from the source data to obtain the read data. A source data list is constructed based on the read data; The target data is obtained by migrating the source data list using the target data mapping relationship and the target script.

4. The method as described in claim 3, characterized in that, The step of constructing the source data list based on the read data includes: Based on the read data, a source data list is constructed using a generator and / or lazy computation.

5. The method as described in claim 3, characterized in that, Before inputting the source data sample and the target data sample into the optimized general large language model to obtain the target data mapping relationship and the target script, the following steps are included: Generate language model prompt words based on the source data samples and the target data samples; Input the language model prompts into a preset general large language model to obtain the initial data mapping relationship and the initial script; Based on the source data sample, the target data sample, the initial data mapping relationship, and the initial script, the general large model is optimized in multiple rounds until the target data mapping relationship and the target script are generated through the optimized general large model.

6. The method as described in claim 5, characterized in that, The step of performing multiple rounds of optimization on the general large model based on the source data sample, the target data sample, the initial data mapping relationship, and the initial script until the target data mapping relationship and the target script are generated through the optimized general large model includes: The initial data mapping relationship and the initial script are reviewed to obtain the review results; The source data sample, the target data sample, and the review result are input into the general large model to obtain a new primary data mapping relationship and a new primary script. The following steps are performed: the initial data mapping relationship and the initial script are reviewed to obtain the review result, until the target data mapping relationship and the target script are generated through the optimized general large model.

7. The method as described in claim 1, characterized in that, The source data to be acquired includes: Install a pre-developed script management plugin on the data platform; In response to the source data acquisition instruction, obtain the application name of the application to which the source data belongs; Based on the application name, the valid configuration file of the application is exported through the script management plugin, and the valid configuration file is used as the source data.

8. A data comparison device, characterized in that, The device includes: The data acquisition module is used to acquire source data and target data, and determine key categories based on the source data; The source data grouping module is used to group the source data according to the key classification to obtain at least one source data group; The target data grouping module is used to create at least one sub-process according to the key classification and read at least one target data group from the target data according to the key classification through the sub-process; The calculation module is used to perform parallel calculations on each of the source data groups and each of the target data groups to obtain the difference set; The merging module is used to merge the difference sets to obtain the comparison results.

9. A data comparison device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data comparison method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data comparison method as described in any one of claims 1 to 7.

Images