Data acquisition method, device and equipment based on CDC and medium

Abstract

The embodiment of the invention discloses a data acquisition method and device based on CDC, equipment and a medium, and relates to the technical field of low-intrusion information acquisition. The method comprises the following steps: acquiring real-time performance data of a target system; processing the real-time performance data through a pre-trained system load prediction model to obtain a predicted load of the target system; and when data acquisition is carried out on the target system through the CDC at the prediction time, adjusting CDC task parameters according to the prediction load so as to reduce the data acquisition amount of the CDC when the load of the target system meets a high load condition. According to the technical scheme, the predicted load of the target system is obtained through the system load prediction model, and then the CDC task parameters are adjusted, so that the target system can maintain stable operation, and normal business processing is not affected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of low-invasive information collection, and in particular to a CDC-based data collection method, device, equipment and medium. BACKGROUND

[0002] CDC (Change Data Capture) is a technology for capturing data changes (such as addition, modification, and deletion operations) in a database in real time or near real time. When collecting data, it collects data through logs and has the characteristics of low invasiveness.

[0003] However, CDC data collection still has an impact on the performance of the collection target, affecting the normal business processing of the collection target. SUMMARY

[0004] The present application provides a CDC-based data collection method, device, equipment and medium, which can collect data of a target system by predicting the load of the target system to optimize the collection scheme and reduce the impact on the performance of the target system.

[0005] According to an aspect of the present application, a CDC-based data collection method is provided, the method comprising:

[0006] obtaining real-time performance data of a target system;

[0007] processing the real-time performance data through a pre-trained system load prediction model to obtain a predicted load of the target system;

[0008] when collecting data of the target system through CDC at a predicted time, adjusting CDC task parameters according to the predicted load to reduce the amount of CDC data collection when the load of the target system meets a high load condition.

[0009] According to another aspect of the present application, a CDC-based data collection device is provided, comprising:

[0010] a real-time performance data acquisition module for acquiring real-time performance data of a target system;

[0011] a predicted load determination module for processing the real-time performance data through a pre-trained system load prediction model to obtain a predicted load of the target system;

[0012] a data collection module for adjusting CDC task parameters according to the predicted load when collecting data of the target system through CDC at a predicted time, to reduce the amount of CDC data collection when the load of the target system meets a high load condition.

[0013] According to another aspect of the present application, there is provided an electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein

[0016] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the CDC-based data collection method according to any one of the embodiments of the present application.

[0017] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for enabling a processor to implement the CDC-based data collection method according to any one of the embodiments of the present application when executed by the processor.

[0018] The technical solution of the embodiments of the present application comprises: obtaining real-time performance data of a target system; processing the real-time performance data by a pre-trained system load prediction model to obtain a predicted load of the target system; when performing data collection on the target system by CDC at a prediction time, adjusting CDC task parameters according to the predicted load to reduce the amount of CDC data collection when the load of the target system meets a high load condition. The technical solution obtains the predicted load of the target system by the system load prediction model, and then adjusts the CDC task parameters, so that the target system can maintain stable operation and its normal business processing is not affected.

[0019] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is a flowchart of a CDC-based data collection method according to Embodiment One of the present application;

[0022] Figure 2 is a flowchart of a CDC-based data collection method according to Embodiment Two of the present application;

[0023] Figure 3 FIG. 1 is a structural schematic diagram of a low-invasion information collection system based on CDC and performance monitoring prediction according to Embodiment Two of the present application;

[0024] Figure 4 FIG. 2 is an architecture diagram of a low-invasion information collection system based on CDC and performance monitoring prediction according to Embodiment Two of the present application;

[0025] Figure 5 FIG. 3 is a flowchart of a low-invasion information collection system based on CDC and performance monitoring prediction according to Embodiment Two of the present application;

[0026] Figure 6 FIG. 4 is a structural schematic diagram of a data collection device based on CDC according to Embodiment Three of the present application;

[0027] Figure 7 FIG. 5 is a structural schematic diagram of an electronic device implementing a data collection method based on CDC according to Embodiment Three of the present application. DETAILED DESCRIPTION

[0028] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", "target" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment One

[0031] Figure 1A flowchart of a CDC-based data collection method is provided for Embodiment One of the present application. The present application can be applied to the case of hierarchical merging of time-series data content. The method can be executed by a CDC-based data collection device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device with data processing capability. As shown in FIG. 10, the method comprises the following steps. Figure 1

[0032] S110, obtaining real-time performance data of a target system.

[0033] The target system is the target of data collection, i.e., the present application collects data of the target system. The target system can be a server provided with a database, etc. The performance data can reflect the data processing performance of the target system, including but not limited to CPU usage, memory usage, etc.

[0034] S120, processing the real-time performance data by a pre-trained system load prediction model to obtain a predicted load of the target system.

[0035] The system load prediction model can predict the load of the target system. Since the performance data of the target system is real-time and is continuously updated, the real-time performance data is essentially time-series data. The system load prediction model of the present application can be an LSTM (Long Short-Term Memory) model to process time-series real-time performance data and obtain more accurate prediction results.

[0036] Specifically, after obtaining the real-time performance data of the target system, it is input into the pre-trained system load prediction model to obtain the predicted load of the target system output by the model. The predicted load refers to the load of the target system at a future time.

[0037] For example, if the CPU occupancy rate and other data of the target system continuously increase, it indicates that the target system needs to perform a large amount of data processing. In this case, the predicted load can be high.

[0038] S130, when collecting data of the target system by CDC at a prediction time, adjusting the CDC task parameters according to the predicted load to reduce the amount of CDC data collection when the load of the target system meets the high-load condition.

[0039] The prediction time is the time corresponding to the predicted load. For example, the predicted load obtained by step S120 is the prediction result of the load of the target system at 8:00 am the next day. The prediction time corresponding to the predicted load is 8:00 am the next day.

[0040] ​Specifically, when data collection is performed on the target system at the prediction time, the data collection amount can be adjusted adaptively according to the predicted load, so that the load of the target system is kept balanced, and the normal business processing of the target system is not affected.

[0041] For example, if the predicted load is small, the remaining computing resources of the target system can support the CDC when the CDC performs data collection, and the CDC task parameters can not be changed. If the predicted load is large, the remaining computing resources of the target system are small when the CDC performs data collection, and the normal business processing of the target system can be affected after the computing resources required by the CDC are allocated. In this case, the CDC task parameters can be changed to reduce the data collection amount of the CDC, so as to reduce the impact on the target system.

[0042] The technical scheme of the embodiment of the present application comprises: obtaining real-time performance data of a target system; processing the real-time performance data by a pre-trained system load prediction model to obtain a predicted load of the target system; when data collection is performed on the target system by a CDC at a prediction time, adjusting CDC task parameters according to the predicted load to reduce the data collection amount of the CDC when the load of the target system meets a high load condition. The technical scheme obtains the predicted load of the target system by the system load prediction model, and then adjusts the CDC task parameters, so that the target system can keep stable operation, and the normal business processing is not affected.

[0043] Embodiment Two

[0044] Figure 2 A flowchart of a CDC-based data collection method provided by Embodiment Two of the present application is provided, and the embodiment of the present application is optimized based on the above-mentioned embodiments.

[0045] As shown in Figure 2 The method of the embodiment of the present application specifically comprises the following steps:

[0046] S210, obtaining a configuration file of a CDC tool located in a database and SSH connection information.

[0047] The CDC tool is used to implement the CDC function, which can be set according to actual conditions. For example, the CDC tool is Debezium.

[0048] S220, creating a CDC task based on the configuration file of the CDC tool; the CDC task is used to collect data from a target system.

[0049] S230, establishing a connection with the target system based on the SSH connection information to obtain real-time performance data of the target system through SSH.

[0050] In the embodiments of the present application, the SSH connection corresponds to the CDC task one by one. In this way, when a CDC task is enabled, only the corresponding SSH connection needs to be enabled, thereby realizing fine control of resources.

[0051] In the embodiments of the present application, the configuration file and the SSH connection information of the CDC tool are placed in the database to realize centralized management of configuration.

[0052] S240, real-time performance data of a target system is acquired.

[0053] S250, the real-time performance data is processed by a pre-trained system load prediction model to obtain a predicted load of the target system.

[0054] In the embodiments of the present application, optionally, the training process of the system load prediction model comprises: determining the preprocessed system performance data as training data; initializing an input sequence length, a number of layers, a number of hidden units of each layer, a learning rate and a batch size of an LSTM model; training the initialized LSTM model based on the training data to obtain the system load prediction model.

[0055] In this way, the long-term dependence of the load can be captured by the LSTM model, thereby improving the prediction accuracy. Through continuous updating and training, the LSTM model can continuously adapt to the changes of the system load and enhance the stability of the prediction.

[0056] In the embodiments of the present application, optionally, the method further comprises: updating the system load prediction model based on incremental learning; and an update formula of a weight of the system load prediction model is:

[0057] ;

[0058] wherein, is the weight at the t th moment, is the learning rate, is the gradient of the loss function.

[0059] The technical solution introduces the incremental learning method to fine-tune the LSTM model, so that it can quickly adapt to new data distribution. Thus, without retraining the entire model, the adaptability and training efficiency of the model can be improved by adjusting the existing model.

[0060] In the embodiments of the present application, optionally, the prediction strategy of the system load prediction model combined with the multi-head attention mechanism is updated based on a reinforcement learning algorithm.

[0061] The technical solution dynamically adjusts the prediction strategy of the LSTM model and the allocation of computing resources through adaptive algorithms such as reinforcement learning, so as to adaptively adjust the allocation of computing resources according to the load change in the model application process, and improve the efficiency of resource scheduling and the prediction accuracy.

[0062] The technical solution combines the multi-head attention mechanism with the LSTM, which can effectively improve the memory capacity and modeling capacity for long-term dependencies of the model.

[0063] S260, when collecting data of the target system through the CDC at the prediction time, adjusting the CDC task parameters according to the predicted load to reduce the CDC data collection amount when the load of the target system meets the high load condition.

[0064] In the embodiments of the application, adjusting the CDC task parameters according to the predicted load comprises: when the predicted load meets the high load condition, adjusting the CDC task parameters according to the confidence of the predicted load and a preset adjustment coefficient to obtain adjusted parameters; suspending the CDC data collection; and after determining that the queue processing is completed, updating the CDC task parameters and resuming the CDC data collection.

[0065] For example, if the CPU usage of the target system exceeds 80% and / or the memory usage exceeds 90% in the predicted load, the predicted load meets the high load condition.

[0066] In another possible solution, if the CPU usage of the target system continues to rise in a future preset time period (for example, 30 seconds) in the predicted load, it is determined that the predicted load meets the high load condition at the current time, so that parameter adjustment is performed in advance.

[0067] In this way, the resource consumption of the CDC task can be dynamically adjusted according to the actual load condition, so as to maximize the reduction of the impact on the target system while ensuring the data capture efficiency. The combination of the front-end control and the database configuration makes the parameter adjustment more flexible and controllable.

[0068] In the embodiments of the application, the method further comprises: if the amount of data of the target system collected meets the full update condition, forwarding the collected data based on the lock-free concurrency mechanism of the target framework, and merging a plurality of change events into one batch message for sending when sending.

[0069] The target framework can be the Disruptor framework. In this way, the data forwarding capability in the full update data forwarding scenario is greatly improved, and a series of problems caused by the delay in data forwarding are avoided.

[0070] The technical scheme of the embodiment of the application can reduce the influence on the target system to the greatest extent while ensuring the data acquisition efficiency, and realize low-invasive information acquisition.

[0071] In one specific example, the method described in the embodiment of the application can be executed by a low-invasive information acquisition system based on CDC and performance monitoring prediction, as shown in the figure, which mainly consists of the following modules: Figure 3

[0072] Control module (Control Module): responsible for system initialization, CDC task configuration and management, and dynamic adjustment of CDC task parameters according to the prediction results, which is the core control unit of the system.

[0073] Monitoring module (Monitoring Module): responsible for real-time collection of performance index data of the target system, such as CPU usage, memory usage, etc., providing data basis for load prediction.

[0074] Prediction module (Prediction Module): based on LSTM model to predict the future load of the target system, providing decision basis for CDC task parameter adjustment.

[0075] CDC task management unit (CDC Task Management Unit): responsible for creating, managing and configuring CDC task instances, ensuring the stable operation of CDC tasks.

[0076] Data change capture component (CDC Component): Debezium, responsible for capturing data change information in the target system, which is the core component of data acquisition.

[0077] Figure 4 The architecture diagram of the low-invasive information acquisition system based on CDC and performance monitoring prediction. Figure 5 The low-invasive flowchart based on CDC and performance monitoring prediction.

[0078] The system is started based on the Spring Boot framework. The control module is responsible for loading global configuration information from the database, as well as Debezium task configuration information and SSH connection information.

[0079] Configuration storage: Debezium task configuration and SSH connection information are stored in the tb_cdc_config and tb_ssh_config tables of the database. Through centralized storage, configuration management is simplified.

[0080] ​Dynamic loading: When the system starts, the control module loads all Debezium task configurations and SSH connection information from the database. This ensures that the system can immediately resume the previous task state after startup.

[0081] Innovation: Storing Debezium configuration files and SSH connection information in the database enables centralized management of configurations.

[0082] Dynamic management of SSH connections:

[0083] SSH connections are associated with each CDC task, enabling fine-grained control of resources. The control module dynamically creates and destroys SSH connections based on the start and stop state of CDC tasks. SSH connection information is read from the tb_ssh_config table in the database, ensuring the accuracy of connection information.

[0084] In this solution, since each CDC task only establishes an SSH connection when needed, it avoids the resource waste and security risks caused by global SSH connections.

[0085] CDC task instantiation and dynamic control:

[0086] The control module calls the CDC task management unit (DebeziumTaskManager.java) to load Debezium task configurations and SSH connection configurations from the database, dynamically creating and managing different CDC task instances. The start, stop, and update operations of the task are triggered by the front-end sending requests, and the control module executes the corresponding operations after receiving the requests.

[0087] Key processes:

[0088] Task discovery: When the system starts, DebeziumTaskManager.java reads all Debezium task configurations from the database.

[0089] Task creation: Create DebeziumTask instances based on configuration information, each instance has an independent thread pool and SSH connection. The size of the thread pool and SSH connection parameters are obtained from the read configuration.

[0090] Task control: The front-end sends requests to the control module, which can include starting tasks, stopping tasks, updating task configurations, and other operations.

[0091] Configuration update: After receiving the request, the control module updates the Debezium configuration information in the database and restarts the corresponding task to make the new configuration take effect.

[0092] Dynamic task management: The control module dynamically manages the life cycle of tasks, including start, stop, update, and other operations.

[0093] This solution adopts a task isolation mechanism to ensure the stability and reliability of CDC tasks. The front-end sends requests to control the start and stop of tasks and updates, achieving centralized management of tasks and greatly improving the maintainability of the system. Through interaction with the front-end, users can easily manage CDC tasks without directly operating the database or configuration files.

[0094] Real-time system load monitoring:

[0095] The monitoring module (MonitorModule) is responsible for collecting real-time performance data of the target system, providing data support for LSTM model training and dynamic parameter adjustment.

[0096] Data collection: The monitoring module connects to the target system through SSH and executes predefined commands to collect performance indicators, including CPU usage, memory usage, etc.

[0097] Data storage: The collected performance data is stored in a local txt file, making it easy to read data for prediction calculation when predicting.

[0098] This solution dynamically obtains SSH session information related to each CDC task through the monitoring module, ensuring the accuracy and security of monitoring. At the same time, it is stored in a local file to facilitate subsequent queries and analysis, providing high-quality data for load prediction.

[0099] When collecting performance data based on SSH, in a system configuration environment that is relatively simple, the performance data of the target system can be obtained directly through the SSH protocol. This method is suitable for most standard Linux / Unix systems and has the advantage of simple deployment by executing standard system commands to collect data.

[0100] This technical solution can also be based on the data collection of third-party monitoring tools. In the case of high requirements for host resource occupation, it can work collaboratively with the LSTM model by combining third-party monitoring tools (such as Prometheus, Zabbix). This way uses existing monitoring tools to collect performance data, reducing the additional overhead of the system, and providing more comprehensive monitoring indicators for the LSTM model to analyze and predict, improving the prediction accuracy.

[0101] LSTM model training and load prediction:

[0102] The Prediction Module uses a Long Short-Term Memory (LSTM) model to make real-time predictions of system load, aiming to dynamically adjust resource allocation and optimize the execution efficiency of computational tasks such as CDC tasks. LSTM is a neural network model suitable for processing and predicting time series data, capable of effectively capturing long-term dependencies in data, especially when faced with complex and volatile system loads.

[0103] Data Preprocessing:

[0104] Data preprocessing is a crucial step in building an efficient LSTM model. The Prediction Module reads historical performance data from local files, including CPU usage, memory usage, disk I / O, and other key system indicators. The data is processed through data cleaning, data smoothing, and data standardization to ensure that the model can be trained on high-quality data. Data cleaning involves analyzing the distribution of data, removing obvious outliers to prevent adverse effects on model training, and using data interpolation or nearest neighbor-based data filling to ensure data integrity. Data smoothing uses moving averages or other smoothing algorithms (such as exponential weighted smoothing) to smooth data with high noise, reducing short-term fluctuations that interfere with the model. Data standardization standardizes each feature to have a mean of 0 and a variance of 1, which helps improve model convergence speed and compresses data to the [0, 1] interval, suitable for training neural networks that use activation functions, avoiding activation function saturation.

[0105] Model Training: The Prediction Module uses preprocessed data to train the LSTM model. The parameters of the LSTM model include:

[0106] Input Sequence Length = 3, which means the model will predict the next time step based on the past 3 time steps. Number of LSTM Layers = 2, the first layer of LSTM has 100 hidden units, and the second layer of LSTM has 50 hidden units. Learning Rate = 0.0001, controlled by the Adam optimizer. Batch Size = 64, meaning 64 samples will be used for each gradient update during model training. Epochs = 100, meaning the entire dataset will be trained for 100 rounds, and the model will traverse the data once per round.

[0107] Real-time Prediction:

[0108] After training, the prediction module uses the trained LSTM model to predict the system load for a future period. Based on real-time collected system performance data, the prediction module can dynamically adjust the prediction window, real-time load prediction, and provide efficient and accurate load management strategies. The advantage of LSTM is that it can capture the long-term dependence of load through historical data, thereby improving the prediction accuracy. Through continuous updating and training, the LSTM model can continuously adapt to the changes of system load, and enhance the stability of prediction.

[0109] The LSTM model of the technical solution can effectively capture the long-term dependence in time series through its gating structure, which makes it have obvious advantages in predicting system load (especially when the load fluctuates greatly). Compared with traditional regression models, LSTM can better capture complex patterns and trends in data, providing high-precision prediction results. These prediction results will directly affect the decision to dynamically adjust the CDC task parameters, helping the system to reasonably allocate computing resources during load peaks and optimize task scheduling during load valleys, thereby improving system efficiency and stability.

[0110] The technical solution introduces an incremental learning method to fine-tune the LSTM model, enabling it to quickly adapt to new data distributions. This method does not require retraining the entire model, but adjusts the existing model, thereby improving the adaptability and training efficiency of the model. Specifically, we update the model's weights during training to enable real-time learning of new input data, avoiding the high computational overhead of starting from scratch. Assuming that in incremental learning, the weight update rule of the model can be expressed as:

[0111] ;

[0112] where, is the weight at time t, is the learning rate, is the gradient of the loss function.

[0113] Further, in real-time prediction, combined with the fluctuation of the load, through reinforcement learning and other adaptive algorithms, the prediction strategy of the LSTM model and the allocation of computing resources are dynamically adjusted. Through dynamic optimization, the model can adaptively adjust the allocation of computing resources according to the changes in load, improving the efficiency of resource scheduling and the accuracy of prediction. In reinforcement learning, the state transition can be represented by the following formula:

[0114] ;

[0115] where, is the current state, is the discount factor, It provides immediate rewards. Through this method, LSTM models can make smarter predictions and schedule resources under fluctuating loads, achieving efficient resource management.

[0116] Introducing a multi-head attention mechanism and combining it with LSTM can effectively improve the model's memory capacity and ability to model long-term dependencies. In a multi-head attention LSTM model, the input and output of an LSTM unit are influenced not only by the previous state and the current input, but also by attention information from different subspaces. This mechanism can be described by the following steps: The input data is first processed through multiple different attention heads. Each head independently computes a weighted sum of the query, key, and value. The final output of the LSTM is the combined result of the multi-head attention mechanism. The update formula for the LSTM unit is as follows:

[0117] ;

[0118] ;

[0119] ;

[0120] ;

[0121] ;

[0122] ;

[0123] in, It is the input at the current moment. It was the hidden state from a previous moment. , , , These are the forget gate, input gate, candidate cell state, and output gate. Combining multi-head attention LSTM with the powerful sequence modeling capabilities of LSTM and the local information focusing ability of multi-head attention allows for better handling of complex load prediction tasks.

[0124] Dynamic adjustment and management of CDC mission parameters:

[0125] The ControlModule dynamically adjusts the parameters of the CDC task based on the prediction results of the LSTM model, so as to reduce the impact on the target system while ensuring data capture efficiency.

[0126] Prediction result analysis: The control module analyzes the prediction results of the LSTM model and determines whether to adjust the CDC task parameters according to the predefined strategy. Adjustment strategies include: CPU usage threshold: if the predicted future CPU usage exceeds 80%, trigger parameter adjustment. Memory usage threshold: if the predicted future memory usage exceeds 90%, trigger parameter adjustment. CPU usage rising trend: if the predicted CPU usage continues to rise within the next 30 seconds, adjust the parameters in advance.

[0127] Parameter adjustment: If the parameters need to be adjusted, the control module will modify the values of parameters such as max.batch.size and max.queue.size. The adjustment range of the parameters is adjusted according to the confidence of the prediction results and the preset adjustment coefficient. The adjustment of the parameters is actually achieved by modifying the values of the corresponding debezium configuration map in memory.

[0128] Task restart: The control module triggers the graceful restart of the CDC task to make the new parameter configuration take effect, while avoiding interrupting the ongoing data capture process. The specific implementation of graceful restart includes: pause data collection -> wait for queue processing to complete -> update configuration -> resume data collection.

[0129] This technical solution can dynamically adjust the resource consumption of the CDC task according to the actual load situation through intelligent parameter adjustment, so as to maximize the reduction of the impact on the target system while ensuring the data capture efficiency. The combination of front-end control and database configuration makes parameter adjustment more flexible and controllable.

[0130] Full update high-throughput data forwarding problem solving (based on Disruptor framework):

[0131] In the full update scenario, the system needs to quickly forward a large amount of data. In order to solve this problem, the system uses the Disruptor framework.

[0132] RingBuffer: Use pre-allocated RingBuffer to store change events, avoiding frequent memory allocation and recycling. The size of RingBuffer is set to 4, and in actual testing and application, it is found that setting it to 4 can better balance the demand for memory occupation and high-speed data forwarding.

[0133] Lock-free concurrency: Use the lock-free concurrency mechanism of the Disruptor framework to improve the efficiency of data processing. The coordination between producers and consumers is realized through Sequence Barrier, avoiding lock contention.

[0134] Batch sending: Multiple change events are combined into one batch message for sending, reducing the overhead of network transmission. The size of batch sending is set to 2024, which is the default value. Generally, this value is appropriate, and in practice, it will be dynamically adjusted according to the load state.

[0135] Eventual consistency guarantee: RecordCommitter is used to record data that has been successfully sent, ensuring the eventual consistency of data even if TCP sending fails.

[0136] TCP retry mechanism: When TCP sending fails, the system will perform exponential backoff retry to ensure that the data can be successfully sent eventually.

[0137] This technical solution applies the Disruptor framework to the full update data forwarding scenario, solving the performance bottleneck of large data forwarding. Combined with the batch sending mechanism, it further improves the forwarding efficiency. Through RecordCommitter, it guarantees the eventual consistency, even if TCP sending fails, it can guarantee the eventual consistency of data. And join the TCP retry mechanism, further guarantee the reliable transmission of data.

[0138] Solution to improve the generalization ability of load prediction model (based on Dropout framework):

[0139] In deep learning, especially in LSTM and other RNN variants, the problem of overfitting often occurs. Overfitting refers to the model performing very well on the training set, but performing poorly on the test set or new data. To solve this problem, Dropout is a commonly used regularization technique. Dropout randomly discards a portion of neurons in the network, so that the model does not rely too much on specific neurons, thereby reducing the risk of overfitting.

[0140] Basic principle: During training, Dropout randomly "discards" a portion of neurons in the neural network with a certain probability (usually between 0.2 and 0.5). This means that the network structure will be different each time it is trained, so the model trained each time will be different. After training is complete, the network will "restore" these discarded neurons and integrate them. For LSTM networks, Dropout is usually applied to the connections between LSTM layers and fully connected layers, as well as the recursive connections within LSTM layers. This can prevent overfitting and improve generalization ability.

[0141] The technical solution can also be based on resource management of containerization technology. For scenarios that require high control of resource occupation, containerization technology can be used to deploy CDC task instances in independent Docker containers, realize dynamic allocation and management of resources, and dynamically adjust resource limits inside each container according to the prediction results of the LSTM model, thereby realizing optimal utilization of resources and minimizing system resource occupation.

[0142] The technical solution combines the LSTM load prediction model with CDC task dynamic adjustment to significantly optimize the performance of the data acquisition system. In particular, when the system load is high, this method can dynamically adjust task parameters according to real-time load prediction, thereby effectively reducing system resource occupation and interference. This method is different from the simple application of existing technologies, and its innovation and advantages are embodied in the following aspects:

[0143] Load prediction combined with LSTM model: Current technologies usually rely on distributed processing frameworks and log stream synchronization to implement data change capture. Although these methods can efficiently synchronize data, they often cannot monitor system load in real time and are difficult to adjust effectively when the load fluctuates. In contrast, this proposal uses an LSTM model to predict the load of the target system in real time, which can accurately predict future load trends and dynamically adjust the execution frequency of CDC tasks accordingly. Through this method, excessive occupation of system resources in high-load situations can be effectively avoided, reducing the risk of system crashes or task interruptions. The LSTM model can capture long-term load changes and provide more accurate load predictions, thereby optimizing task scheduling strategies and significantly improving system stability and performance.

[0144] Low-invasive data acquisition: Traditional CDC technologies mostly rely on log stream synchronization for data acquisition, which may cause high system resource consumption when data throughput is large, and even affect the normal operation of the system. In order to reduce the impact on the system, this proposal combines real-time load prediction and dynamic adjustment of CDC tasks to reduce the invasiveness of the data acquisition process. By dynamically adjusting the processing frequency and resource allocation of tasks, the system can reduce resource occupation during high load and increase task execution frequency during low load. Such adjustment can improve the efficiency of data acquisition while ensuring the stable operation of the system.

[0145] Task dynamic adjustment combined with incremental learning: Existing technologies usually use fixed parameters or static configurations to handle CDC tasks, lacking the ability to dynamically adapt to environmental changes. This proposal innovatively introduces incremental learning, multi-head attention mechanism, and fine-tuning with LSTM models, enabling the system to continuously adapt to new data distributions. Through real-time learning and prediction, the system can dynamically adjust the processing frequency and resource allocation of tasks, optimizing the execution efficiency and accuracy of data collection tasks. This method effectively avoids the inefficiency and lack of adaptability that may exist in traditional static configurations.

[0146] Enhanced system stability and fault tolerance: Although the distributed framework and log synchronization method in current technologies can capture data changes, they often struggle to ensure task stability and system fault tolerance under high load conditions. By combining the load prediction mechanism of the LSTM model with dynamic resource scheduling, this proposal can predict system load trends in advance and adjust CDC task resource allocation and processing frequency based on real-time prediction results. This mechanism enables the system to flexibly adjust task execution under large load fluctuations, avoiding task crashes or data loss due to excessive load, significantly improving system stability and fault tolerance.

[0147] Embodiment Three

[0148] Figure 6 A structure diagram of a CDC-based data collection device is provided for Embodiment Three of the present application. This device can execute the CDC-based data collection method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. As shown in Figure 6 The device includes:

[0149] A real-time performance data acquisition module 310 is configured to acquire real-time performance data of a target system.

[0150] A predicted load determination module 320 is configured to process the real-time performance data through a pre-trained system load prediction model to obtain a predicted load of the target system.

[0151] A data collection module 330 is configured to adjust CDC task parameters according to the predicted load when performing data collection on the target system through CDC at a predicted time, so as to reduce the amount of CDC data collection when the load of the target system meets the high load condition.

[0152] The technical scheme of the embodiment of the application comprises: a real-time performance data acquisition module 310, configured to acquire target system real-time performance data; a predicted load determination module 320, configured to process the real-time performance data by using a pre-trained system load prediction model to obtain a predicted load of the target system; and a data acquisition module 330, configured to adjust a CDC task parameter according to the predicted load when data of the target system is acquired by using the CDC at a predicted time, so as to reduce the amount of CDC data acquisition when the load of the target system meets a high-load condition. The technical scheme obtains the predicted load of the target system by using the system load prediction model, and then adjusts the CDC task parameter, so that the target system can be kept in stable operation, and normal business processing of the target system is not affected.

[0153] In the embodiment of the application, the device further comprises:

[0154] The configuration information acquisition module is configured to acquire a configuration file and SSH connection information of the CDC tool located in the database;

[0155] The CDC task creation module is configured to create a CDC task based on the configuration file of the CDC tool; the CDC task is configured to acquire data from the target system;

[0156] The SSH connection module is configured to establish a connection with the target system based on the SSH connection information, so as to acquire target system real-time performance data by using SSH.

[0157] In the embodiment of the application, optionally, the device further comprises a system load prediction model training module, comprising:

[0158] The training data determination unit is configured to determine the preprocessed system performance data as training data;

[0159] The model initialization unit is configured to initialize an input sequence length, a number of layers, a number of hidden units of each layer, a learning rate, and a batch size of the LSTM model;

[0160] The model training unit is configured to train the initialized LSTM model based on the training data, to obtain the system load prediction model.

[0161] In the embodiment of the application, optionally, the device further comprises:

[0162] The model updating module is configured to update the system load prediction model based on incremental learning; and an update formula of a weight of the system load prediction model is as follows:

[0163] ;

[0164] wherein, is the weight at the t th time point, is a learning rate, is a gradient of a loss function.

[0165] In the embodiments of the present application, optionally, the prediction strategy of the system load prediction model combined with the multi-head attention mechanism is updated based on the reinforcement learning algorithm.

[0166] In the embodiments of the present application, optionally, the data acquisition module 330 comprises:

[0167] The parameter adjustment unit is configured to adjust the CDC task parameter according to the confidence of the predicted load and a preset adjustment coefficient to obtain an adjusted parameter when the predicted load meets the high load condition.

[0168] The pause acquisition unit is configured to pause the CDC data acquisition.

[0169] The recovery acquisition unit is configured to update the CDC task parameter and recover the CDC data acquisition after the queue processing is completed.

[0170] In the embodiments of the present application, optionally, the device further comprises:

[0171] The data generation module is configured to forward the collected data based on the lock-free concurrent mechanism of the target framework if the data volume of the target system meets the full volume update condition, and to send a plurality of change events in a batch message when sending.

[0172] The CDC-based data acquisition device provided in the embodiments of the present application can execute the CDC-based data acquisition method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0173] Embodiment Four

[0174] Figure 7 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0175] As Figure 7As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0176] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0177] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the CDC-based data acquisition method.

[0178] In some embodiments, the CDC-based data acquisition method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the CDC-based data acquisition method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the CDC-based data acquisition method by any other appropriate means, such as by means of firmware.

[0179] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0180] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program

[0181] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0182] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0183] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0184] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0185] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0186] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Accordingly, the disclosure is not limited to the specific embodiments described above, but only by the scope of the appended claims.

Claims

1. A CDC-based data collection method, characterized by, The method comprises the following steps: obtaining real-time performance data of a target system; processing the real-time performance data through a pre-trained system load prediction model to obtain a predicted load of the target system; adjusting CDC task parameters according to the predicted load when data of the target system is collected by CDC at a predicted time, so as to reduce the amount of CDC data collection when the load of the target system meets a high-load condition.

2. The method of claim 1, wherein, Before the step of obtaining real-time performance data of a target system, the method further comprises the following steps: obtaining a configuration file of a CDC tool located in a database and SSH connection information; creating a CDC task based on the configuration file of the CDC tool; the CDC task is used to collect data from the target system; establishing a connection with the target system based on the SSH connection information to obtain real-time performance data of the target system through SSH.

3. The method of claim 1, wherein, The training process of the system load prediction model comprises the following steps: determining preprocessed system performance data as training data; initializing the input sequence length, the number of layers, the number of hidden units in each layer, the learning rate and the batch size of the LSTM model; training the initialized LSTM model based on the training data to obtain the system load prediction model.

4. The method of claim 1, wherein, The method further comprises the following steps: updating the system load prediction model based on incremental learning; the weight update formula of the system load prediction model is: ； wherein, is the weight at the t-th time, is the learning rate, is the gradient of the loss function.

5. The method of claim 1, wherein, updating the prediction strategy of the system load prediction model combined with the multi-head attention mechanism based on the reinforcement learning algorithm.

6. The method of claim 1, wherein, The step of adjusting CDC task parameters according to the predicted load comprises the following steps: when the predicted load meets the high-load condition, adjusting the CDC task parameters according to the confidence of the predicted load and a preset adjustment coefficient to obtain adjusted parameters; suspending CDC data collection; after determining that the queue processing is completed, updating the CDC task parameters and resuming CDC data collection.

7. The method of claim 1, wherein, The method further comprises the following steps: if the amount of data collected from the target system meets the full update condition, when the collected data is forwarded, the data is forwarded based on the lock-free concurrency mechanism of the target framework, and multiple change events are combined into one batch message for sending when sending.

8. A CDC-based data acquisition device, characterized by The method comprises the following steps: a real-time performance data obtaining module, configured to obtain real-time performance data of a target system; a predicted load determining module, configured to process the real-time performance data through a pre-trained system load prediction model to obtain a predicted load of the target system; a data collection module, configured to adjust CDC task parameters according to the predicted load when data of the target system is collected by CDC at a predicted time, so as to reduce the amount of CDC data collection when the load of the target system meets a high-load condition.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the CDC-based data collection method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the CDC-based data collection method of any one of claims 1-7 when executed.

Images