Data comparison method and device for database, equipment and medium

Abstract

The invention relates to the technical field of databases, and discloses a database data comparison method which comprises the following steps: analyzing a data comparison task to obtain a comparison white list, and determining a to-be-compared data range and related limiting conditions based on the comparison white list; collecting metadata features according to a to-be-compared data range and related limiting conditions, obtaining a resource condition snapshot of a current server system at the same time, and processing the metadata features and the resource condition snapshot to generate feature vectors; inputting the feature vector into a lightweight machine learning model for reasoning operation, and outputting an optimal concurrent parameter set and an initial optimal resource configuration scheme; executing a data comparison task according to the optimal concurrent parameter set and the initial optimal resource configuration scheme; during task execution, monitoring task execution efficiency and a system resource utilization rate, and when it is judged that an abnormal condition exists according to the task execution efficiency and the system resource utilization rate, adjusting the concurrent parameter set and the resource configuration scheme; and the data comparison task is executed again.

Description

A method, apparatus, device, and medium for comparing data in a database. Technical Field

[0001] This invention relates to the field of database technology, and in particular to a data comparison method, apparatus, device, and storage medium for databases. Background Technology

[0002] In existing data comparison technology practices, parameter configuration and resource allocation rely heavily on manual intervention for core operations. At the initial stage of data comparison task creation, users must rely on their accumulated professional experience to manually analyze and configure task-related parameters. Throughout the task execution process, continuous manual monitoring is required to track the task's operational status in real time. When resource bottlenecks or performance failures occur during task execution, staff must spend additional time analyzing the root causes and readjusting parameter settings and resource allocation schemes. This entire process not only demands high levels of technical experience and practical skills from users but also requires significant manpower, resulting in limited overall work efficiency and difficulty in quickly responding to dynamic changes during task execution. Summary of the Invention

[0003] The main objective of this invention is to provide a database data comparison method, apparatus, device, and storage medium, aiming to solve the problems of low task execution efficiency and high manpower consumption when encountering resource bottlenecks in the existing technology for database data comparison.

[0004] To achieve the above objectives, this invention provides a database data comparison method, comprising: parsing a user-submitted data comparison task to obtain a comparison whitelist; determining the range of data to be compared and related limiting conditions based on the comparison whitelist; collecting metadata features in parallel according to the range of data to be compared and related limiting conditions, while simultaneously acquiring a snapshot of the current server system's resource status, and standardizing the metadata features and resource status snapshot to generate feature vectors; inputting the feature vectors into a lightweight machine learning model for inference operations, and outputting an optimal set of concurrency parameters and an initial optimal resource configuration scheme; performing a data comparison task on target data in the source database and the target database according to the optimal set of concurrency parameters and the initial optimal resource configuration scheme; continuously monitoring task execution efficiency and system resource utilization during task execution, determining whether there are any abnormalities based on the task execution efficiency and system resource utilization, and adjusting the concurrency parameter set and resource configuration scheme if abnormalities are found; and performing the data comparison task according to the adjusted set of concurrency parameters and resource configuration scheme.

[0005] Furthermore, to achieve the above objectives, the present invention provides a database data comparison device, comprising: a task acquisition module, used to parse user-submitted data comparison tasks, obtain a comparison whitelist, and determine the range of data to be compared and related limiting conditions based on the comparison whitelist; a feature vector module, used to collect metadata features in parallel according to the range of data to be compared and related limiting conditions, and simultaneously acquire a snapshot of the current server system's resource status, and perform standardization processing on the metadata features and resource status snapshot to generate a feature vector; and a resource and parameter configuration module, used to input the feature vector into a lightweight machine learning model. The system performs inference operations and outputs the optimal concurrency parameter set and the initial optimal resource configuration scheme. A data comparison module performs a data comparison task on the target data in the source and target databases based on the optimal concurrency parameter set and the initial optimal resource configuration scheme. A resource and parameter adjustment module continuously monitors the task execution efficiency and system resource utilization during task execution, determines whether there are any abnormalities based on the task execution efficiency and system resource utilization, and adjusts the concurrency parameter set and resource configuration scheme if abnormalities are found. A task execution module performs the data comparison task based on the adjusted concurrency parameter set and resource configuration scheme.

[0006] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a database data comparison program stored in the memory and executable on the processor, wherein when the database data comparison program is executed by the processor, it implements the steps of the database data comparison method as described above.

[0007] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a database data comparison program, wherein when the database data comparison program is executed by a processor, it implements the steps of the database data comparison method described above.

[0008] Beneficial Effects: This invention relates to the field of database technology and discloses a database data comparison method, comprising: parsing a user-submitted data comparison task to obtain a comparison whitelist; determining the range of data to be compared and related limiting conditions based on the comparison whitelist; collecting metadata features in parallel according to the range of data to be compared and related limiting conditions, while simultaneously acquiring a snapshot of the current server system's resource status, and standardizing the metadata features and resource status snapshot to generate feature vectors; inputting the feature vectors into a lightweight machine learning model for inference operations, and outputting an optimal set of concurrency parameters and an initial optimal resource configuration scheme; performing a data comparison task on target data in the source database and the target database according to the optimal set of concurrency parameters and the initial optimal resource configuration scheme; continuously monitoring task execution efficiency and system resource utilization during task execution, determining whether there are any abnormalities based on the task execution efficiency and system resource utilization, and adjusting the concurrency parameter set and resource configuration scheme if abnormalities are found; and performing the data comparison task according to the adjusted set of concurrency parameters and resource configuration scheme. This invention obtains a comparison whitelist by parsing data comparison tasks, clarifies the data range and limiting conditions, then collects metadata features in parallel, obtains server resource snapshots and standardizes them to generate feature vectors, and outputs optimal parameters and resource configurations through lightweight machine learning models. It monitors efficiency and resource utilization in real time during the execution of comparison tasks, and dynamically adjusts them in case of anomalies, without the need for manual intervention, saving manpower and time costs, achieving optimal resource allocation, significantly improving data comparison efficiency and stability, and exhibiting strong adaptability and scalability. Attached Figure Description

[0009] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the drawings: Figure 1 is a schematic diagram of an application environment of the database data comparison method in an embodiment of the present invention; Figure 2 is a schematic flowchart of an embodiment of the database data comparison method of the present invention; Figure 3 is a schematic diagram of the functional modules of a preferred embodiment of the database data comparison device of the present invention; Figure 4 is a schematic diagram of the structure of a computer device in an embodiment of the present invention; Figure 5 is another schematic diagram of the structure of a computer device in an embodiment of the present invention. Detailed Implementation

[0010] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0011] The database data comparison method provided in this embodiment of the invention can be applied in the application environment shown in Figure 1, where the user terminal communicates with the server terminal via a network. The server terminal can parse the data comparison task submitted by the user terminal to obtain a comparison whitelist, and determine the range of data to be compared and related limiting conditions based on the comparison whitelist; collect metadata features in parallel according to the range of data to be compared and related limiting conditions, and simultaneously obtain a snapshot of the current server system's resource status, and perform standardization processing on the metadata features and resource status snapshot to generate feature vectors; input the feature vectors into a lightweight machine learning model for inference operations, and output the optimal concurrency parameter set and the initial optimal resource configuration scheme; perform a data comparison task on the target data of the source database and the target database according to the optimal concurrency parameter set and the initial optimal resource configuration scheme; during task execution, continuously monitor the task execution efficiency and system resource utilization rate, and determine whether there are any abnormalities based on the task execution efficiency and system resource utilization rate. If there are any abnormalities, adjust the concurrency parameter set and resource configuration scheme; and perform the data comparison task according to the adjusted concurrency parameter set and resource configuration scheme. This invention obtains a comparison whitelist by parsing a data comparison task, clarifying the data range and limiting conditions. It then collects metadata features in parallel, obtains server resource snapshots, and standardizes them to generate feature vectors. A lightweight machine learning model infers and outputs optimal parameters and resource configurations. During the comparison task, efficiency and resource utilization are monitored in real time, and adjustments are made dynamically in case of anomalies. No manual intervention is required, saving manpower and time costs, achieving optimal resource allocation, and significantly improving data comparison efficiency and stability. It also exhibits strong adaptability and scalability. The user end can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server end can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0012] Please refer to Figure 2, which is a flowchart illustrating an embodiment of the database data comparison method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0013] As shown in Figure 2, the database data comparison method proposed in this invention includes the following steps: S100, parsing the data comparison task submitted by the user to obtain a comparison whitelist, and determining the range of data to be compared and related limiting conditions based on the comparison whitelist; S200, collecting metadata features in parallel according to the range of data to be compared and related limiting conditions, and simultaneously obtaining a snapshot of the current server system's resource status, and standardizing the metadata features and resource status snapshot to generate feature vectors; S300, inputting the feature vectors into a lightweight machine learning model for inference operations, and outputting an optimal concurrency parameter set and an initial optimal resource configuration scheme; S400, performing a data comparison task on the target data of the source database and the target database according to the optimal concurrency parameter set and the initial optimal resource configuration scheme; S500, continuously monitoring the task execution efficiency and system resource utilization during task execution, judging whether there are any abnormalities based on the task execution efficiency and system resource utilization, and adjusting the concurrency parameter set and resource configuration scheme if there are any abnormalities; S600, performing the data comparison task according to the adjusted concurrency parameter set and resource configuration scheme.

[0014] In this embodiment, after the user submits a data comparison task, the system first parses the comparison whitelist in the task. The comparison whitelist refers to the specific data range specified by the user that needs to be verified for data consistency (i.e., data comparison). By parsing this whitelist, the system can clarify the specific boundaries of the data to be compared, such as the database table names and data row ranges involved, and clarify the relevant limiting conditions, thus defining a clear execution scope for subsequent data comparison work.

[0015] Next, based on the parsed range and constraints of the comparison data, the system initiates the metadata collection process. The metadata collection module collects metadata features corresponding to the comparison whitelist in parallel processing. These features include key data attributes such as table name, table size, number of rows, and index information. Simultaneously, the resource monitoring module captures real-time snapshots of the current server system's resource status, covering core resource indicators such as CPU utilization, memory usage, I / O performance, and network bandwidth usage. Afterward, the system standardizes the collected metadata features and resource status snapshots, converting data of different formats and magnitudes into feature vectors that meet the model input requirements, eliminating interference from data differences, and providing a standardized data foundation for subsequent model inference.

[0016] Subsequently, this standardized feature vector is input into a lightweight machine learning model for inference operations. This model is trained using metadata from historical tasks and corresponding resource usage data. Its core employs the random forest algorithm, which combines the specific data features of the task to be compared with the current system resource status to quickly calculate the optimal set of concurrency parameters (such as thread pool size, number of task shards, etc.) and the initial optimal resource configuration scheme (such as the number of CPU cores and memory quota allocated to the task), ensuring that the task receives appropriate parameter and resource support upon startup.

[0017] After obtaining the optimal set of concurrency parameters and the initial optimal resource configuration, the system will create the corresponding thread pool based on these configurations, adjust the relevant parameter settings, and officially start the target data comparison task between the source database and the target database. The core purpose of data comparison is to verify whether the data in the source database and the target database are consistent, ensuring the accuracy of the data in scenarios such as migration.

[0018] Throughout the entire data comparison task execution process, the system continuously monitors task execution efficiency (such as the data comparison rate and completion progress) and system resource utilization in real time. Using this real-time collected data, the system determines if there are any anomalies in the current task execution. These anomalies primarily include resource bottlenecks (such as high CPU usage causing task lag, or insufficient memory affecting data processing) and poor performance (such as a comparison rate far below expectations). Once an anomaly is detected, the system immediately triggers an adjustment mechanism, dynamically modifying the concurrency parameter set and resource configuration scheme. For example, when CPU resources are strained, the number of concurrent threads is appropriately reduced, or more idle memory resources are allocated to the task to ensure its continued efficient progress.

[0019] Finally, the system continues to execute the data comparison task according to the adjusted concurrency parameter set and resource configuration scheme until the task is completed. The entire process requires no manual intervention in parameter configuration and resource scheduling. Through automated parsing, collection, inference, monitoring, and adjustment, it achieves adaptive optimal allocation of parameters and resources during data comparison, saving manpower and time costs while significantly improving the efficiency and stability of data comparison. After the task is completed, the parameter configuration, execution performance, and other relevant data are fully recorded for subsequent retraining and optimization of the lightweight machine learning model, enabling the model to provide more accurate parameter and resource configuration suggestions in future tasks.

[0020] In one embodiment, S100 includes: S101, obtaining a data comparison task created and submitted by the user through the user interface of the application layer; S102, parsing the data comparison task through the task management module of the application layer to obtain a comparison whitelist; S103, analyzing the comparison whitelist to obtain the range of data to be compared and related limiting conditions.

[0021] In this embodiment, the user first initiates an operation through the user interface of the application layer (data comparison system), creating and submitting a data comparison task. The core of this task is to verify the data consistency between the source database and the target database. After submission, the task is passed to the task management module of the application layer, which is specifically responsible for parsing and processing the data comparison task. The core objective is to extract the comparison whitelist contained in the task. The comparison whitelist is a set of specific data identifiers that the user has pre-specified and that explicitly require data comparison. All subsequent data comparison operations will revolve around this whitelist to prevent irrelevant data from being included in the comparison process.

[0022] After obtaining the comparison whitelist, the system conducts further in-depth analysis. By parsing key information in the whitelist, the specific scope of the data to be compared is clarified, such as determining the database table names involved, the range of rows within the tables, and the time range to which the data belongs. Simultaneously, relevant limiting conditions are extracted. These conditions may include data indexing rules and field filtering criteria, used to precisely define the comparison objects and ensure that data comparison is conducted only within the specific scope required by the user. This guarantees the accuracy of the comparison results while avoiding unnecessary resource consumption, laying a clear foundation for subsequent metadata collection, resource configuration, and task execution. The entire process relies on the core functional modules of the application layer, achieving an orderly connection from task submission to a clearly defined comparison scope, and is a crucial pre-processing step for starting a data comparison task.

[0023] In one embodiment, step S200 includes: S201, the metadata acquisition module collects metadata features from the database in parallel according to the range of data to be compared and related limiting conditions; S202, the resource monitoring module obtains a snapshot of the current server system's resource status in real time; and S203, the metadata features and the resource status snapshot are standardized to generate a feature vector.

[0024] In this embodiment, after clarifying the scope of the data to be compared and the relevant limiting conditions, the metadata acquisition module will collect metadata features from the corresponding database in a parallel processing manner based on these defining information. The metadata features cover the key attribute information of the data to be compared, such as the specific table name, table storage size, total number of data rows, index configuration, etc. This information is an important basis for subsequent model inference and resource configuration. The parallel acquisition method can improve data acquisition efficiency and avoid process delays caused by collecting data one by one.

[0025] Meanwhile, the resource monitoring module deployed on the server captures real-time snapshots of the server system's resource status. These snapshots include key operational metrics of the server's core hardware and network, such as CPU utilization, memory usage, I / O read / write performance, and network bandwidth usage. By obtaining these snapshots, the system can accurately grasp the current resource availability of the server, providing real-time reference for subsequent resource allocation.

[0026] After completing metadata feature collection and resource status snapshot acquisition, the system standardizes these two types of data. Since metadata features and resource status data belong to different types, their data formats and numerical magnitudes differ. Directly using them for model inference would affect the accuracy of the results. Therefore, it is necessary to convert these data into standardized data with consistent format and balanced magnitude through unified standardization rules. Finally, this standardized data is integrated to generate feature vectors that meet the input requirements of lightweight machine learning models, providing standardized and effective data support for subsequent optimal concurrency parameters and initial resource configuration inference. Throughout the process, metadata collection and resource status snapshot acquisition are performed simultaneously, ensuring data comprehensiveness and improving process efficiency, laying the foundation for the efficient initiation of data comparison tasks.

[0027] In one embodiment, step S300 includes: S301, loading a trained lightweight machine learning model through the model training and model inference service of the service layer; S302, inputting the feature vector into the lightweight machine learning model for real-time inference operations and outputting the inference result; S303, combining historical data training model experience and the inference result to generate an optimal set of concurrent parameters and an initial optimal resource configuration scheme.

[0028] In this embodiment, after generating the feature vectors, the system loads the trained lightweight machine learning model through the model training and inference services of the service layer. This model, based on the random forest algorithm, is trained using metadata (such as table name, table size, number of rows, indexes, etc.) and corresponding task resource usage data (such as CPU utilization, memory consumption, task execution time, etc.) from a large amount of historical data. The model file is stored in the model repository of the data layer and infrastructure layer, supporting flexible loading according to different scenarios and versions to ensure adaptation to the needs of the current data comparison task.

[0029] Subsequently, the standardized feature vectors are input into the lightweight machine learning model, which then initiates real-time inference operations. During inference, the model combines the metadata features of the data to be compared contained in the feature vectors with the current server system resource status (such as CPU, memory, I / O, network bandwidth, etc.), fully utilizing the patterns and experience accumulated through training on historical data to accurately analyze and judge the resource requirements and optimal execution method of the current task, and finally outputs preliminary inference results.

[0030] Finally, the system will combine historical data training experience accumulated during model training (including feedback on the effect of parameter configuration in different scenarios, optimization cases of resource allocation, etc.) to further optimize and adjust the initial inference results output by the model, ultimately generating the optimal set of concurrent parameters and the initial optimal resource configuration scheme adapted to the current data comparison task. The optimal set of concurrent parameters covers key execution parameters such as thread pool size and the number of task shards, while the initial optimal resource configuration scheme includes resource allocation standards such as the number of CPU cores allocated to the task, memory quota, and I / O priority. Both together provide core support for the efficient and stable execution of the data comparison task, ensuring that the task obtains the most reasonable parameters and resource guarantees from the start-up phase.

[0031] In one embodiment, step S400 includes: S401, the application layer schedules a dynamic scheduling module to create a thread pool based on the optimal concurrency parameter set and the initial optimal resource configuration scheme; S402, the application layer schedules a dynamic scheduling module to update relevant system parameters based on the optimal concurrency parameter set and the initial optimal resource configuration scheme; and S403, the application layer schedules a data comparison task according to the thread pool and relevant system parameters to perform data consistency verification on the target data in the source database and the target database.

[0032] In this embodiment, the application layer's scheduling engine core invokes the dynamic scheduling module. This module, a key component for intelligent resource and parameter allocation, operates strictly according to the optimal concurrency parameter set and initial optimal resource configuration scheme output by the lightweight machine learning model. First, the dynamic scheduling module creates a thread pool adapted to the current data comparison task, based on key information such as the thread pool size and task sharding rules specified in the optimal concurrency parameter set. The proper construction of the thread pool enables efficient parallel processing of tasks, avoids resource waste caused by frequent thread creation and destruction, and provides stable thread support for task execution.

[0033] Meanwhile, the dynamic scheduling module updates and adjusts relevant system parameters based on the optimal set of concurrency parameters and the initial optimal resource configuration scheme. These system parameters that need to be updated cover configuration items closely related to task execution, such as the CPU core utilization ratio allocated to the data comparison task, memory usage threshold, I / O read / write priority, and network bandwidth allocation quota. By updating parameters, the system resources are ensured to accurately match the task requirements, creating the optimal operating environment for task execution.

[0034] After completing thread pool creation and system parameter updates, the system will officially launch the data comparison task according to the constructed thread pool architecture and updated system parameters. Once the task starts, it will extract the corresponding target data from the source and target databases based on the pre-defined range and constraints of the data to be compared. A comprehensive consistency check will be performed on both sets of data, verifying the integrity, accuracy, and consistency of each piece of data to ensure complete matching between the target data in the source and target databases. This provides reliable data consistency assurance for scenarios such as database migration. The entire process relies on the precise allocation of the dynamic scheduling module, achieving seamless integration of parameter configuration, resource allocation, and task execution, ensuring the efficient startup and stable operation of the data comparison task.

[0035] In one embodiment, step S500 includes: S501, during task execution, the application layer scheduling feedback adjustment module continuously monitors task execution efficiency and system resource utilization; S502, based on the task execution efficiency and system resource utilization, the task running status and resource usage changes are captured in real time; S503, based on the task running status and resource usage changes, it is determined whether there are any abnormal situations; S504, if there are any abnormal situations, the feedback adjustment module sends an intervention command to the dynamic scheduling module, and the dynamic scheduling module adjusts the concurrent parameter set and resource configuration scheme in real time according to the intervention command.

[0036] In this embodiment, throughout the entire data comparison task execution process, the application layer scheduling engine core will activate the feedback adjustment module. This module will continuously monitor and track task execution efficiency and system resource utilization in real time. Task execution efficiency is mainly reflected by indicators such as data comparison rate, task completion progress, and data processing accuracy. System resource utilization focuses on core resource operation data such as server CPU utilization, memory usage, input / output (I / O) read / write rate, and network bandwidth usage, ensuring comprehensive awareness of task execution and system operation status.

[0037] The feedback adjustment module dynamically captures the real-time running status of tasks (such as normal progress, progress delays, data verification errors, etc.) and dynamic changes in resource usage (such as sudden spikes in CPU usage, memory shortages, network transmission lags, etc.) based on real-time collected task execution efficiency and system resource utilization data. Through continuous analysis and judgment of this data, the system can accurately determine whether there are any abnormalities in the current task execution. These abnormalities mainly include two categories: first, resource bottlenecks, i.e., excessive occupation of certain system resources causing task execution to be hindered; and second, poor performance, i.e., situations that affect the efficient progress of tasks, such as task execution speed being far lower than expected and progress being severely delayed.

[0038] Once the feedback adjustment module determines that the above-mentioned anomaly exists, it will immediately send a real-time intervention command to the application-layer dynamic scheduling module. The command will specify the anomaly type, the scope of impact, and the direction of adjustment required. After receiving the intervention command, the dynamic scheduling module will quickly respond and make targeted adjustments based on the current task running status and resource usage: for concurrency parameter sets, it may increase or decrease the number of threads in the thread pool, adjust the task fragment size, etc., according to the actual situation; for resource configuration schemes, it may reallocate CPU core quotas, adjust memory usage thresholds, optimize I / O read / write priorities, or network bandwidth allocation, etc. Through real-time and flexible parameter and resource adjustments, the anomaly can be resolved in a timely manner, ensuring that the data comparison task can always proceed stably and efficiently under optimal resource configuration and parameter settings, avoiding task interruption or significant efficiency loss due to anomalies.

[0039] In one embodiment, after step S600, the method further includes: after the data comparison task is completed, collecting key data of the data comparison task and storing the key data in the monitoring module; retraining the lightweight machine learning model using the historical key data and the current key data stored in the monitoring module, and storing the trained model file in the model repository.

[0040] In this embodiment, after the data comparison task is completed according to the adjusted thread pool parameters and resource configuration scheme, the system will initiate a key data collection process. The collected key data covers two core aspects: first, parameter configuration data during task execution, including the set of concurrency parameters used throughout the task's lifecycle (such as thread pool size, number of task shards, etc.) and dynamically adjusted resource configuration parameters (such as CPU core quota, memory usage threshold, etc.); second, task performance data, such as total task execution time, data comparison rate, data consistency verification results, peak and average resource utilization rates at each stage, etc. This data comprehensively records the execution status and resource usage characteristics of this task. After collection, the system will uniformly store this key data in the monitoring modules of the data layer and infrastructure layer, and summarize it with the key data of historical tasks to form a complete task execution and resource usage database.

[0041] Subsequently, the service layer's model training and inference services will utilize historical key data stored in the monitoring module and the key data collected this time as the dataset for model retraining. The retraining process uses the random forest algorithm as its core, combining newly supplemented task data to optimize and iterate the original lightweight machine learning model. By learning the correlation between parameter configuration, resource allocation, and task performance in the new task, the model's prediction logic is corrected, allowing it to better adapt to data comparison tasks in different scenarios and improving the accuracy of predictions for optimal concurrent parameter sets and initial resource configuration schemes. After the model retraining is completed and validated, the system will store the updated model file in the model repository of the data layer and infrastructure layer, classifying and managing it according to different scenarios and versions of model files. This ensures that when processing similar data comparison tasks, the optimized model can be loaded for inference operations, enabling continuous upgrades to model capabilities and ultimately driving continuous improvement in the efficiency and stability of the entire data comparison system.

[0042] In one embodiment, a database data comparison device is provided, which corresponds one-to-one with the database data comparison method in the above embodiments. Referring to Figure 3, Figure 3 is a functional module diagram of a preferred embodiment of the database data comparison device of the present invention. The device comprises a task acquisition module 10, a feature vector module 20, a resource and parameter configuration module 30, a data comparison module 40, a resource and parameter adjustment module 50, and a task execution module 60. The detailed descriptions of each functional module are as follows: The task acquisition module 10 is used to parse the data comparison task submitted by the user, obtain a comparison whitelist, and determine the range of data to be compared and related limiting conditions based on the comparison whitelist; the feature vector module 20 is used to collect metadata features in parallel according to the range of data to be compared and related limiting conditions, and simultaneously obtain a snapshot of the current server system's resource status, and perform standardization processing on the metadata features and resource status snapshot to generate a feature vector; the resource and parameter configuration module 30 is used to input the feature vector into a lightweight machine learning model for inference operations and output the optimal... The system includes: a concurrency parameter set and an initial optimal resource configuration scheme; a data comparison module 40, used to perform a data comparison task on the target data of the source database and the target database according to the optimal concurrency parameter set and the initial optimal resource configuration scheme; a resource and parameter adjustment module 50, used to continuously monitor the task execution efficiency and system resource utilization during task execution, and determine whether there are any abnormalities based on the task execution efficiency and system resource utilization. If there are any abnormalities, the concurrency parameter set and resource configuration scheme are adjusted; and a task execution module 60, used to perform the data comparison task according to the adjusted concurrency parameter set and resource configuration scheme.

[0043] In one embodiment, the task acquisition module 10 includes: acquiring data comparison tasks created and submitted by users through the user interface of the application layer; parsing the data comparison tasks through the task management module of the application layer to obtain a comparison whitelist; and analyzing the comparison whitelist to obtain the range of data to be compared and related limiting conditions.

[0044] In one embodiment, the feature vector module 20 includes: a metadata acquisition module that collects metadata features in parallel from a database based on the range of data to be compared and relevant limiting conditions; a resource monitoring module that obtains a real-time snapshot of the current server system's resource status; and a standardization process that generates a feature vector from the metadata features and the resource status snapshot.

[0045] In one embodiment, the resource and parameter configuration module 30 includes: loading a trained lightweight machine learning model through the model training and model inference service of the service layer; inputting the feature vector into the lightweight machine learning model for real-time inference operations and outputting the inference result; and generating an optimal concurrent parameter set and an initial optimal resource configuration scheme by combining historical data training model experience and the inference result.

[0046] In one embodiment, the data comparison module 40 includes: an application layer scheduling dynamic scheduling module, which creates a thread pool based on the optimal concurrency parameter set and the initial optimal resource configuration scheme; updates relevant system parameters based on the optimal concurrency parameter set and the initial optimal resource configuration scheme; and starts a data comparison task according to the thread pool and relevant system parameters to perform data consistency verification on the target data in the source database and the target database.

[0047] In one embodiment, the resource and parameter adjustment module 50 includes: during task execution, an application layer scheduling feedback adjustment module continuously monitors task execution efficiency and system resource utilization; based on the task execution efficiency and system resource utilization, it captures the task running status and resource usage changes in real time; based on the task running status and resource usage changes, it determines whether there are any abnormal situations; if there are any abnormal situations, it sends an intervention command to the dynamic scheduling module through the feedback adjustment module, and the dynamic scheduling module adjusts the concurrent parameter set and resource configuration scheme in real time according to the intervention command.

[0048] In one embodiment, the task execution module 60 further includes: after the data comparison task is completed, collecting key data of the data comparison task and storing the key data in the monitoring module; retraining the lightweight machine learning model using the historical key data and the current key data stored in the monitoring module, and storing the trained model file in the model repository.

[0049] In one embodiment, a computer device is provided, which can be a server, and its internal structure diagram is shown in Figure 4. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a database data comparison method on the server side.

[0050] In one embodiment, a computer device is provided, which can be a user terminal, and its internal structure diagram is shown in Figure 5. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communication with an external server via a network connection. When executed by a processor, the computer program implements the user-side functions or steps of a database data comparison method. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: parsing a user-submitted data comparison task to obtain a comparison whitelist; determining the range of data to be compared and related limiting conditions based on the comparison whitelist; collecting metadata features in parallel according to the range of data to be compared and related limiting conditions, while simultaneously acquiring a snapshot of the current server system's resource status, and standardizing the metadata features and resource status snapshot to generate a feature vector; inputting the feature vector into a lightweight machine learning model for inference operations, and outputting an optimal concurrency parameter set and an initial optimal resource configuration scheme; performing a data comparison task on target data in the source database and the target database according to the optimal concurrency parameter set and the initial optimal resource configuration scheme; continuously monitoring task execution efficiency and system resource utilization during task execution, judging whether there are any abnormalities based on the task execution efficiency and system resource utilization, and adjusting the concurrency parameter set and resource configuration scheme if there are any abnormalities; and performing the data comparison task according to the adjusted concurrency parameter set and resource configuration scheme.

[0051] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: parsing a data comparison task submitted by a user to obtain a comparison whitelist; determining the range of data to be compared and related limiting conditions based on the comparison whitelist; collecting metadata features in parallel according to the range of data to be compared and related limiting conditions, while simultaneously acquiring a snapshot of the current server system's resource status, and standardizing the metadata features and resource status snapshot to generate a feature vector; inputting the feature vector into a lightweight machine learning model for inference operations, and outputting an optimal set of concurrency parameters and an initial optimal resource configuration scheme; performing a data comparison task on target data in the source database and the target database according to the optimal set of concurrency parameters and the initial optimal resource configuration scheme; continuously monitoring task execution efficiency and system resource utilization during task execution, determining whether there are any abnormalities based on the task execution efficiency and system resource utilization, and adjusting the concurrency parameter set and resource configuration scheme if abnormalities are found; and performing the data comparison task according to the adjusted set of concurrency parameters and resource configuration scheme.

[0052] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0053] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0054] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0055] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data comparison method for a database, characterized in that, The process includes the following steps: parsing the data comparison task submitted by the user to obtain a comparison whitelist, and determining the range of data to be compared and related limiting conditions based on the comparison whitelist; Metadata features are collected in parallel according to the range of data to be compared and relevant constraints. Simultaneously, a snapshot of the current server system's resource status is obtained, and the metadata features and resource status snapshot are standardized to generate feature vectors. These feature vectors are then input into a lightweight machine learning model for inference operations, outputting an optimal set of concurrency parameters and an initial optimal resource configuration scheme. A data comparison task is performed on the target data in the source and target databases based on the optimal set of concurrency parameters and the initial optimal resource configuration scheme. During task execution, task execution efficiency and system resource utilization are continuously monitored. Anomalies are assessed based on these metrics; if anomalies are found, the concurrency parameter set and resource configuration scheme are adjusted. The data comparison task is then performed based on the adjusted concurrency parameter set and resource configuration scheme.

2. The database data comparison method as described in claim 1, characterized in that, The process of parsing the data comparison task submitted by the user to obtain a comparison whitelist, and determining the range of data to be compared and related limiting conditions based on the comparison whitelist, includes: obtaining the data comparison task created and submitted by the user through the user interface of the application layer; parsing the data comparison task through the task management module of the application layer to obtain the comparison whitelist; and analyzing the comparison whitelist to obtain the range of data to be compared and related limiting conditions.

3. The database data comparison method as described in claim 1, characterized in that, The step of collecting metadata features in parallel according to the range of data to be compared and related limiting conditions, while simultaneously obtaining a snapshot of the current server system's resource status, and standardizing the metadata features and resource status snapshot to generate a feature vector, includes: the metadata collection module collecting metadata features in parallel from the database according to the range of data to be compared and related limiting conditions; obtaining a snapshot of the current server system's resource status in real time through the resource monitoring module; and standardizing the metadata features and resource status snapshot to generate a feature vector.

4. The database data comparison method as described in claim 1, characterized in that, The step of inputting the feature vector into a lightweight machine learning model for inference operations and outputting an optimal set of concurrent parameters and an initial optimal resource configuration scheme includes: loading a trained lightweight machine learning model through the model training and model inference services of the service layer; inputting the feature vector into the lightweight machine learning model for real-time inference operations and outputting inference results; and generating an optimal set of concurrent parameters and an initial optimal resource configuration scheme by combining historical data training experience and the inference results.

5. The database data comparison method as described in claim 1, characterized in that, The step of performing a data comparison task on the target data of the source database and the target database according to the optimal concurrency parameter set and the initial optimal resource configuration scheme includes: the application layer scheduling a dynamic scheduling module to create a thread pool according to the optimal concurrency parameter set and the initial optimal resource configuration scheme; updating relevant system parameters based on the optimal concurrency parameter set and the initial optimal resource configuration scheme; and starting the data comparison task according to the thread pool and relevant system parameters to perform data consistency verification on the target data of the source database and the target database.

6. The database data comparison method as described in claim 1, characterized in that, During task execution, the system continuously monitors task execution efficiency and system resource utilization. Based on these metrics, it determines whether any abnormalities exist. If abnormalities are found, it adjusts the concurrency parameter set and resource configuration scheme. This includes: during task execution, the application layer schedules a feedback adjustment module to continuously monitor task execution efficiency and system resource utilization; real-time capture of task running status and resource usage changes based on these metrics; determining whether any abnormalities exist based on these metrics; and if abnormalities are found, sending an intervention command to the dynamic scheduling module through the feedback adjustment module. The dynamic scheduling module then adjusts the concurrency parameter set and resource configuration scheme in real-time based on the intervention command.

7. The database data comparison method as described in claim 1, characterized in that, After executing the data comparison task according to the adjusted concurrency parameter set and resource configuration scheme, the process also includes: after the data comparison task is completed, collecting key data of the data comparison task and storing the key data in the monitoring module; retraining the lightweight machine learning model using the historical key data and the current key data stored in the monitoring module, and storing the trained model file in the model repository.

8. A data comparison device for a database, characterized in that, The database data comparison device includes: a task acquisition module, used to parse the data comparison task submitted by the user, obtain a comparison whitelist, and determine the range of data to be compared and related limiting conditions based on the comparison whitelist; a feature vector module, used to collect metadata features in parallel according to the range of data to be compared and related limiting conditions, and simultaneously obtain a snapshot of the current server system's resource status, and perform standardization processing on the metadata features and resource status snapshot to generate feature vectors; a resource and parameter configuration module, used to input the feature vectors into a lightweight machine learning model for inference operations, and output the optimal concurrency parameter set and the initial optimal resource configuration scheme; a data comparison module, used to perform a data comparison task on the target data of the source database and the target database according to the optimal concurrency parameter set and the initial optimal resource configuration scheme; a resource and parameter adjustment module, used to continuously monitor the task execution efficiency and system resource utilization during task execution, determine whether there are any abnormalities based on the task execution efficiency and system resource utilization, and adjust the concurrency parameter set and resource configuration scheme if there are any abnormalities; and a task execution module, used to execute the data comparison task according to the adjusted concurrency parameter set and resource configuration scheme.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a database comparison program stored in the memory and executable on the processor. When executed by the processor, the database comparison program implements the steps of the database comparison method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a database data comparison program, which, when executed by a processor, implements the steps of the database data comparison method as described in any one of claims 1-7.

Images