Real-time data synchronization system, method and equipment based on knowledge base and storage medium

Abstract

The invention discloses a real-time data synchronization system, method and device based on a knowledge base and a storage medium, and the system comprises a self-adaptive data synchronization module which carries out the preprocessing of original change data in a knowledge base source system, and obtains preset format change data, transmitting the preset format change data to a distributed cache optimization module based on a data synchronization strategy; the distributed cache optimization module determines a data cache strategy corresponding to the preset format change data, splits the preset format change data according to the data cache strategy and stores the data to nodes in each fragment unit; and the fault node processing module monitors nodes in each fragment unit in real time, and when a node fault exists, the fault node is switched to the corresponding standby node through the load balancer for data processing. By integrating a self-adaptive data synchronization strategy, intelligent distributed cache optimization and high-availability design, the complex requirement of large-scale data synchronization is met, and the data processing efficiency and the overall system performance are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing and synchronization, and in particular, relates to a real-time data synchronization system, method, device and storage medium based on a knowledge base. Background Art

[0002] The development of big data technology has driven the demand for massive real-time data processing and synchronization. In particular, in distributed database systems, real-time data synchronization plays a vital role in ensuring data consistency. Current technologies face multiple challenges when dealing with large-scale data synchronization and high real-time requirements, including performance bottlenecks, increased latency, and reduced system reliability.

[0003] Traditional data synchronization solutions usually use a fixed frequency and batch synchronization method. This method works well when data changes are infrequent or the system load is low, but it has serious limitations in an environment where the amount of data is surging. First, the fixed synchronization frequency and batch strategy cannot adapt to the dynamic changes in data traffic and system load, resulting in system delays and even data loss when the amount of data increases dramatically. In addition, when the load is low, the fixed strategy may lead to unnecessary waste of resources and inefficient system operation.

[0004] On the other hand, existing distributed cache systems usually rely on static rules for configuration, which makes it difficult to cope with dynamic changes in data access patterns. This inflexible cache strategy performs poorly when processing hot data, resulting in low cache hit rates, which seriously affects the overall performance of the system, especially when dealing with sudden data access needs.

[0005] Existing systems also have deficiencies in high availability design. Current high availability designs mainly rely on redundant configurations and failover mechanisms, but these systems are slow to respond to fault detection and failover, making it difficult to restore normal operations in a short period of time. This lag causes the system to be down for too long when hardware failures or network problems occur, making it impossible to effectively guarantee data consistency and business continuity, further affecting system reliability.

[0006] As digital transformation deepens, enterprises and organizations are increasingly in need of real-time processing and synchronization of large data sets. However, existing technologies have significant deficiencies in large-scale, real-time synchronization requirements, as shown by the difficulty of traditional synchronization strategies to maintain efficient and stable performance, the difficulty of static configuration of distributed cache systems to adapt to dynamic access patterns, and the failure handling speed of high-availability designs failing to meet expectations.

[0007] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0008] The main purpose of the present invention is to provide a real-time data synchronization system, method, device and storage medium based on a knowledge base, aiming to solve the technical problem of how to improve the efficiency of data processing and the overall system performance.

[0009] To achieve the above object, the present invention provides a real-time data synchronization system based on a knowledge base, the real-time data synchronization system based on a knowledge base includes an adaptive data synchronization module, a distributed cache optimization module and a fault node processing module:

[0010] The adaptive data synchronization module is used to pre-process the original change data in the knowledge base source system to obtain the preset format change data, and transmit the preset format change data to the distributed cache optimization module based on the data synchronization strategy;

[0011] The distributed cache optimization module is used to determine the data cache strategy corresponding to the preset format change data, and split the preset format change data into multiple nodes in each shard unit according to the data cache strategy;

[0012] The fault node processing module is used to monitor each node in multiple sharding units in real time. If there is a node failure, the faulty node is switched to the corresponding standby node for data processing through the load balancer.

[0013] Optionally, the adaptive data synchronization module is further used to determine the data change frequency corresponding to the current network transmission load and the original change data;

[0014] The adaptive data synchronization module is further used to determine the data synchronization strategy corresponding to the preset format change data through the MAB algorithm based on the current network transmission load and the data change frequency;

[0015] The MAB algorithm is:

[0016]

[0017] In the formula, Q(a t ) is the data synchronization strategy, μ(a t ) is for strategy a t The historical average return is N, the total number of synchronizations, n t (a) is the number of times strategy a is selected, and c is the adjustment parameter.

[0018] Optionally, the distributed cache optimization module is further used to predict cache hotspot information through a cache hit rate prediction model according to the preset format change data;

[0019] The distributed cache optimization module is further used to determine the data cache strategy corresponding to the preset format change data based on the cache hotspot information.

[0020] Optionally, the cache hit rate prediction model is:

[0021] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0022] In the formula, f t is the cache hotspot information output by the forget gate at the current time step, W f is the weight matrix of the forget gate, h t-1 is the hidden state of the previous time step, x t For preset format change data, b f is the bias term, and σ is the Sigmoid activation function.

[0023] Optionally, the fault node processing module is used to monitor multiple nodes in each shard unit in real time, and the multiple nodes include a master node and at least one slave node;

[0024] The fault node processing module is further used to obtain monitoring indicator information corresponding to the master node and the at least one slave node respectively;

[0025] The fault node processing module is further configured to determine, when the monitoring indicator information meets the preset indicator condition, that the node corresponding to the preset indicator condition is a fault node;

[0026] The fault node processing module is further used to determine the corresponding standby node according to the fault node through the Raft consensus algorithm;

[0027] The Raft consensus algorithm is:

[0028] term current =term candidate +1

[0029] vote granted =trueifterm candidate ≥term current

[0030] In the formula, term current is the term of the failed node. candidate is the term of the standby node, vote granted Vote for standby node.

[0031] Optionally, the faulty node processing module is further configured to switch the faulty node to a corresponding standby node for data processing through a load balancer based on the total node switching duration.

[0032] Optionally, the fault node processing module is further used to determine the total node switching time according to the fault detection time, the leader selection time, the data recovery time and the task switching time.

[0033] In addition, to achieve the above purpose, the present invention also proposes a real-time data synchronization method based on a knowledge base, and the real-time data synchronization method based on a knowledge base comprises the following steps:

[0034] The adaptive data synchronization module pre-processes the original change data in the knowledge base source system to obtain the preset format change data, and transmits the preset format change data to the distributed cache optimization module based on the data synchronization strategy;

[0035] The distributed cache optimization module determines a data cache strategy corresponding to the preset format change data, and splits and stores the preset format change data into multiple nodes in each sharding unit according to the data cache strategy;

[0036] The fault node processing module monitors each node in multiple shard units in real time. If there is a node failure, the faulty node is switched to the corresponding backup node for data processing through the load balancer.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also proposes a real-time data synchronization device based on a knowledge base, the device comprising: a memory, a processor, and a real-time data synchronization program based on a knowledge base stored on the memory and executable on the processor, the real-time data synchronization program based on the knowledge base being configured to implement the steps of the real-time data synchronization method based on the knowledge base as described above.

[0038] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a real-time data synchronization program based on a knowledge base is stored. When the real-time data synchronization program based on the knowledge base is executed by a processor, the steps of the real-time data synchronization method based on the knowledge base as described above are implemented.

[0039] The present invention firstly pre-processes the original change data in the knowledge base source system by an adaptive data synchronization module to obtain the preset format change data, and transmits the preset format change data to the distributed cache optimization module based on the data synchronization strategy, and then the distributed cache optimization module determines the data caching strategy corresponding to the preset format change data, and splits and stores the preset format change data to multiple nodes in each sharding unit according to the data caching strategy, and then the fault node processing module monitors each node in multiple sharding units in real time, and if there is a node failure, the faulty node is switched to the corresponding standby node for data processing through the load balancer. The present invention solves the challenges of performance bottlenecks, delay problems and reduced system reliability faced by the prior art when processing large-scale data synchronization by integrating adaptive data synchronization strategies, intelligent distributed cache optimization and high availability design, thereby meeting the complex needs of large-scale data synchronization, thereby improving the efficiency of data processing and the overall system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a structural diagram of a real-time data synchronization device based on a knowledge base in a hardware operating environment involved in an embodiment of the present invention;

[0041] Figure 2 It is a structural block diagram of a first embodiment of a real-time data synchronization system based on a knowledge base of the present invention;

[0042] Figure 3 This is a system architecture diagram of a first embodiment of a real-time data synchronization system based on a knowledge base of the present invention;

[0043] Figure 4 It is a flow chart of adaptive data synchronization of the first embodiment of the real-time data synchronization system based on knowledge base of the present invention;

[0044] Figure 5 It is a flow chart of intelligent cache optimization of the first embodiment of the real-time data synchronization system based on the knowledge base of the present invention;

[0045] Figure 6 A schematic diagram of high availability design of a first embodiment of a real-time data synchronization system based on a knowledge base of the present invention;

[0046] Figure 7 It is a flowchart of the first embodiment of the real-time data synchronization method based on the knowledge base of the present invention.

[0047] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0049] Reference Figure 1 , Figure 1 The present invention is a schematic diagram of the structure of a real-time data synchronization device based on a knowledge base in a hardware operating environment according to an embodiment of the present invention.

[0050] like Figure 1 As shown, the real-time data synchronization device based on the knowledge base may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage system independent of the aforementioned processor 1001.

[0051] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the knowledge base-based real-time data synchronization device, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0052] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a real-time data synchronization program based on a knowledge base.

[0053] exist Figure 1 In the knowledge base-based real-time data synchronization device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the knowledge base-based real-time data synchronization device of the present invention can be set in the knowledge base-based real-time data synchronization device, and the knowledge base-based real-time data synchronization device calls the knowledge base-based real-time data synchronization program stored in the memory 1005 through the processor 1001, and executes the knowledge base-based real-time data synchronization method provided by the embodiment of the present invention.

[0054] The embodiment of the present invention provides a real-time data synchronization system based on a knowledge base, referring to Figure 2 , Figure 2 It is a structural block diagram of the first embodiment of the real-time data synchronization system based on the knowledge base of the present invention.

[0055] The adaptive data synchronization module 2001 is used to pre-process the original change data in the knowledge base source system, obtain the preset format change data, and transmit the preset format change data to the distributed cache optimization module 2002 based on the data synchronization strategy.

[0056] In this embodiment, reference Figure 3 , Figure 3 The system architecture diagram of the first embodiment of the real-time data synchronization system based on the knowledge base of the present invention is as follows. The real-time data synchronization system based on the knowledge base includes an adaptive data synchronization module, a distributed cache optimization module and a fault node processing module (i.e., a high availability module). Figure 3 The target system is also included in the system, which is the business system. These modules work together to solve the limitations of existing technologies and significantly improve the performance and reliability of the system.

[0057] Furthermore, the knowledge base may be a large model knowledge base or a large model RAG (Retrieval-Augmented Generation) or the like.

[0058] It should be understood that the source system refers to the storage system of the original data, which is a storage system corresponding to the large model knowledge base or the large model RAG (Retrieval-Augmented Generation), such as an enterprise database, file system, cloud storage or large-scale data processing system. The large model knowledge base can store structured, semi-structured and unstructured data, covering various forms of data resources such as text, images, and videos.

[0059] In this embodiment, reference Figure 4 , Figure 4 This is an adaptive data synchronization flow chart of the first embodiment of the real-time data synchronization system based on the knowledge base of the present invention. When the system is initially deployed, full data synchronization is performed to transfer all necessary data from the source system to the target system. During subsequent data synchronization, the system needs to monitor and analyze through the data acquisition module to obtain the changed data (i.e., the original changed data) from these complex data sources.

[0060] The original change data may be structured data, semi-structured data, unstructured data, etc. In order to ensure data consistency and integrity, the original change data needs to be preprocessed.

[0061] In the specific implementation, the processing methods for preprocessing the original change data in the knowledge base source system are: deduplication, format standardization, missing value processing, redundant information deletion, etc. The cleaned data needs to be converted from various formats of the source system (such as JSON, XML, CSV, relational database table structure, etc.) to standard formats suitable for the target system, such as Protobuf, JSON-LD, etc., to obtain preset format change data, which can ensure data consistency and integrity and meet RAG input standards, thereby providing support for subsequent intelligent recommendation and generation services.

[0062] Further, the adaptive data synchronization module determines the current network transmission load and the data change frequency corresponding to the original change data; based on the current network transmission load and the data change frequency, the data synchronization strategy corresponding to the preset format change data is determined by the MAB algorithm;

[0063] The MAB algorithm is:

[0064]

[0065] In the formula, Q(a t ) is the data synchronization strategy, μ(a t ) is for strategy a t The historical average return is N, the total number of synchronizations, n t (a) is the number of times strategy a is selected, and c is the adjustment parameter (used to control the balance between exploration and exploitation, reflecting the upper confidence bound (UCB)).

[0066] It should also be noted that the adjustment parameter c in the MAB algorithm is not fixed, but is dynamically adjusted according to the system load, data change frequency, and historical revenue. This mechanism ensures that the exploration frequency is reduced when network resources are tight, and more strategy exploration is performed when resources are abundant to find the best combination of synchronization frequency and batch size. This dynamic adjustment mechanism significantly improves the algorithm's adaptability in complex network environments, especially in scenarios where network conditions and data synchronization requirements change rapidly, avoiding performance bottlenecks caused by fixed parameters.

[0067] Adjusting parameter c can be achieved through the following aspects:

[0068] Load condition: When the system load is high, resources may be tight. At this time, the exploration frequency can be reduced to reduce the occupation of system resources. When the load is low, the system has more resources to use, and the exploration frequency can be increased to find a better strategy.

[0069] Data change frequency: If the data change frequency is high, it means that the environment is changing rapidly and more exploration may be needed to adapt to the new situation in a timely manner. Therefore, in this case, the value of c can be increased to encourage the algorithm to explore more. On the contrary, if the data change frequency is low, c can be reduced to make the algorithm more inclined to use existing information.

[0070] Historical returns: By analyzing historical returns, you can evaluate the effectiveness of the current strategy. If the returns over the past period have been consistently good, it means that the current strategy may be effective. In this case, you can lower the value of c and reduce exploration. If the historical returns fluctuate greatly, it may mean that you need to re-explore a better strategy, in which case you can increase c.

[0071] In this embodiment, the value of c is dynamically adjusted by real-time monitoring of the system load, data changes and revenue. For example, a threshold can be set, and when the load exceeds a certain value, c is automatically reduced; when the revenue is lower than a certain threshold, c is increased.

[0072] Implement a feedback system to adjust c based on real-time data. For example, set rules: Load exceeds 80%: c is reduced to 0.5. Data change frequency is high: c is increased to 1.5. Historical revenue is below a certain level: c is increased to 2.0, etc. This embodiment is not limited.

[0073] It should be understood that the data synchronization strategy includes data synchronization frequency and data batch size. Synchronization frequency refers to the cycle of data synchronization, that is, how often data synchronization is performed. Batch size refers to the amount of data transmitted during each synchronization.

[0074] The MAB algorithm dynamically selects the optimal data synchronization strategy (synchronization frequency and batch size) through the following steps:

[0075] 1) Benefit evaluation:

[0076] After each synchronization, the system will evaluate the benefits of the current strategy based on the current network conditions and data transmission results. The benefits value takes into account the following factors:

[0077] Transmission success rate: whether the transmitted data is complete and error-free.

[0078] Transmission latency: The time it takes for data to be transmitted from the source system to the target system.

[0079] Transmission latency: The time it takes for data to be transmitted from the source system to the target system.

[0080] System resource consumption: including consumption of CPU, memory and other resources.

[0081] It should also be noted that the data transmission result refers to the effect and status of the data successfully transmitted and received by the system during the synchronization process. The MAB algorithm can adaptively adjust the synchronization strategy according to the changes in the actual operating environment to ensure that the system maintains optimal performance in different scenarios.

[0082] (1) High data change scenario:

[0083] When the system detects that data changes frequently and the amount of change is large (for example, during business peak hours), the MAB algorithm will give priority to higher synchronization frequencies and larger batch sizes to quickly synchronize large amounts of data. In this case, the algorithm will prefer strategies with higher historical returns to ensure timely synchronization of data and avoid data lags.

[0084] (2) Scenarios with high network load:

[0085] When the network load is high (for example, bandwidth is limited or concurrent requests surge), the MAB algorithm will dynamically reduce the synchronization frequency and select a smaller batch size to avoid network congestion caused by excessive data transmission. The algorithm will explore more and evaluate new synchronization strategies to find the optimal parameter combination under the current network environment.

[0086] (3) Low data change scenario:

[0087] In scenarios where data changes are not frequent (such as business downturns), the system does not need to synchronize data frequently. At this time, the MAB algorithm will choose a lower synchronization frequency and a smaller batch size to reduce unnecessary data synchronization and resource consumption. At the same time, the system ensures that high resource utilization can be maintained even under low load through benefit evaluation.

[0088] In this embodiment, it is assumed that the system currently has three candidate strategies:

[0089] Strategy A: high synchronization frequency (synchronize every 5 seconds) and large batch size (100MB of data is synchronized each time).

[0090] Strategy B: medium synchronization frequency (synchronize every 10 seconds) and medium batch size (50MB of data each time).

[0091] Strategy C: low synchronization frequency (synchronize every 30 seconds) and small batch size (10MB of data each time).

[0092] In high-load, high-variability scenarios, the MAB algorithm may select Strategy A multiple times because it can quickly synchronize large amounts of data and has a higher historical return value.

[0093] When the load increases but the data changes decrease, the system may tend to choose strategy B to maintain synchronization efficiency while reducing the use of network resources.

[0094] When the network load is extremely high and data changes rarely, strategy C may be more often chosen because it minimizes the impact of synchronization on the system load.

[0095] The MAB algorithm is traditionally used to select the optimal strategy. This invention deeply optimizes the data synchronization scenario by introducing dynamic network load perception and the frequency of time series data changes. The synchronization frequency and batch size are dynamically adjusted according to network conditions to ensure that the system remains stable under high load and responds quickly when data changes frequently.

[0096] It should also be noted that for data that changes frequently, the algorithm will give priority to a higher synchronization frequency to ensure timely data transmission; for data that changes less frequently, a low-frequency synchronization will be selected to avoid unnecessary waste of resources.

[0097] It should also be noted that after data transmission, a hash check should be performed to compare whether the data in the source system and the target system are consistent.

[0098] The distributed cache optimization module 2002 is used to determine the data cache strategy corresponding to the preset format change data, and split the preset format change data into multiple nodes in each shard unit according to the data cache strategy.

[0099] Furthermore, the distributed cache optimization module predicts cache hotspot information according to the preset format change data through a cache hit rate prediction model; and determines a data cache strategy corresponding to the preset format change data based on the cache hotspot information.

[0100] The cache hit rate prediction model is:

[0101] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0102] In the formula, f t is the cache hotspot information output by the forget gate at the current time step, W f is the weight matrix of the forget gate, h t-1 is the hidden state of the previous time step, x t For preset format change data, b f is the bias term, and σ is the Sigmoid activation function.

[0103] In the specific implementation, refer to Figure 5 , Figure 5This is a flow chart of intelligent cache optimization of the first embodiment of the real-time data synchronization system based on the knowledge base of the present invention. In order to improve the cache utilization efficiency of the system, the system uses a machine learning model (i.e., long short-term memory network (LSTM)) to build a cache hit rate prediction model by analyzing historical access data. The model predicts data that may be frequently accessed within a specific time period (i.e., cache hotspot information) by capturing long-term trends and short-term fluctuations in data. The model can also effectively capture the timing dependency of data access, predict future hotspot data, optimize cache utilization, reduce resource waste, and improve data access efficiency.

[0104] It is also necessary to explain the construction process of the cache hit rate prediction model: (1) Preparation of training data: First, collect a large amount of cache history data, including the access frequency, access time interval, and data block size of each data block. These training data are provided by the actual cache access logs generated during system operation, which record the time series of data access. (2) Training process: The LSTM model is trained through supervised learning. The cache access history data is divided into multiple training samples according to the time series, and each sample contains the access information of a specific time step. The goal is to input a series of historical data and predict the cache hit situation in the next time step or several future time steps. (3) Loss function and optimizer: The mean square error (MSE) is used as the loss function to measure the gap between the predicted result and the actual cache hit situation. Through the back propagation algorithm, the error is gradually minimized to optimize the prediction performance of the model. In order to improve the convergence speed of the model, the Adam optimizer is used for gradient update. The Adam optimizer combines the advantages of momentum and adaptive learning rate adjustment, which can effectively improve the efficiency of large-scale data training. (4) Model initialization: Define the LSTM model structure, including the number of nodes in the input layer, hidden layer, and output layer. A multi-layer LSTM structure is usually used to improve the ability to capture complex time series patterns. Training set and validation set division: historical access data is divided into training set (80%) and validation set (20%). During the training process, the overfitting of the model is monitored in real time through the validation set to ensure the generalization ability of the model. Model training: The model parameters are gradually optimized through multiple iterations (Epochs), and the prediction effect of the model is evaluated through the validation set data after each training. Model evaluation and tuning: According to the performance indicators on the validation set (such as MSE, accuracy, etc.), the model's hyperparameters, such as the number of LSTM layers, the number of hidden units, and the learning rate, are adjusted.

[0105] It should also be noted that in order to ensure that the model can accurately predict cache hotspot information, the input features are carefully selected and processed, mainly including: Historical access frequency: the number of times each data block has been accessed in the past few time steps. By capturing the access frequency of data blocks, the model can identify data that may become cache hotspots in the future.

[0106] Data access time interval: The time interval between consecutive accesses to each data block reflects the usage pattern of the data block and helps the model predict the possibility of future access. Data block size: Larger data blocks occupy more cache resources, and their access frequency may have a significant impact on the cache strategy. This feature helps the model balance the cache management of large data blocks and small data blocks. System load and cache status: including the current system's CPU utilization, memory occupancy, network load, etc., to help the model predict cache hotspots based on the system status and reasonably adjust the cache strategy. Historical hit rate: The cache hit situation in the past several time steps. The changing trend of the hit rate provides valuable reference information for the model.

[0107] It should be understood that the time step is an important parameter of the LSTM model, which defines the length of the model's observation window in time series data and is crucial to the prediction of cache hit rate.

[0108] The optimal time step is determined through experiments. In scenarios where the cache is frequently accessed, using a short time step (such as the past 10 access cycles) helps capture fast-changing cache hot spots. In relatively stable access scenarios, using a longer time step (such as the past 30 access cycles) can obtain longer-term access trends.

[0109] The present invention also designs a dynamic time step adjustment mechanism based on the system status. When the system detects that the access pattern fluctuates frequently, the time step is appropriately shortened to quickly respond to changes in cache hot spots; when the access pattern is relatively stable, the time step is extended to obtain more comprehensive historical access information.

[0110] To improve the accuracy of the forecast, we took the following measures:

[0111] (1) Model evaluation

[0112] During the training process, the prediction effect of the model is measured by cross-validation and the MSE of the validation set. A comparison curve between the predicted value and the true value is drawn to observe whether the model can accurately capture the changing trend of the data.

[0113] (2) Overfitting Prevention

[0114] The Dropout regularization method is introduced to randomly discard some neuron connections during the training process of the LSTM layer to prevent the model from overfitting and improve the generalization ability of the model.

[0115] (3) Model update

[0116] Since cache access patterns may change over time, the model is designed with online learning capabilities. The model is regularly retrained and updated based on new cache access data to ensure that it can adapt to changes in data access patterns and maintain a high level of prediction accuracy.

[0117] (4) Adaptive caching strategy

[0118] According to the prediction results of the LSTM model, the system adjusts the cache strategy in real time, giving priority to caching data blocks with high prediction hit rates and removing data blocks with low prediction hit rates. This dynamic cache management ensures efficient use of cache resources and improves the overall performance of the system.

[0119] In the present invention, an LSTM optimization method with adaptive timing window is proposed to address the limitations of the traditional long short-term memory (LSTM) model. The time window of the traditional LSTM model is usually fixed and cannot flexibly respond to different cache access patterns. To this end, the system dynamically adjusts the time window length of the LSTM model according to the changes in the cache access pattern. When the access frequency of hot data is detected to increase rapidly, the timing window will be shortened to capture more detailed time changes; conversely, when the access pattern is stable, the timing window will be appropriately extended to obtain more historical data for prediction.

[0120] It should also be understood that cache access patterns refer to the behavioral characteristics of the system when accessing cached data, including the access frequency, access order, time interval, and burst access of the data. Specifically, hot data tends to be frequently accessed, while other data is relatively unpopular. The access order may show a certain regularity, resulting in specific data being accessed intensively at certain times. In addition, the time intervals for access may be uneven, and some data may be accessed burstily in a short period of time. Changes in these patterns directly affect the effectiveness of the cache and the predictive ability of the LSTM model. Therefore, in response to these dynamic changes, the system can flexibly adjust the length of the timing window to adapt to different cache access patterns, thereby improving the accuracy of the prediction.

[0121] The adaptive time window mechanism significantly enhances the practicality of LSTM in cache prediction and solves the problem that fixed window length cannot adapt to different access patterns. This dynamic adjustment method is particularly suitable for distributed systems with frequently changing cache hotspots, improving the prediction accuracy and response speed of the system in a high-concurrency environment.

[0122] The system dynamically adjusts the data cache strategy based on the prediction results, giving priority to storing data with high access frequency in the cache. This strategy can effectively improve the cache hit rate, thereby increasing the speed of data access and reducing system latency.

[0123] Furthermore, the preset format change data is split and stored in multiple nodes (ie, a master node and at least one slave node) in each shard unit according to the data caching strategy.

[0124] It should be noted that the concurrent processing capability of the system is enhanced through read-write separation and data sharding technology. Data read-write separation ensures that data reading operations will not be affected by write operations, avoiding performance problems caused by read-write conflicts. At the same time, data sharding technology allows data to be distributed to multiple cache nodes, with each node processing a different subset of data, thereby achieving load balancing. The cache hit rate prediction module is closely integrated with the distributed cache architecture: by predicting hot data and caching it first, the system can distribute frequently accessed data to multiple nodes, reducing the load pressure on a single node. In addition, when the amount of data increases, the system can add new cache nodes through horizontal expansion to ensure that the system can continue to provide stable responses in a high-concurrency environment.

[0125] It is also necessary to explain that the master node is a database node in the shard, which is responsible for handling all write operations (such as creating and updating orders, etc.). Whenever there is a write operation, the system will write the data to the master node of the shard. Whenever there is a read operation, the system will write the data to at least one slave node of the shard.

[0126] The faulty node processing module 2003 is used to monitor each node in a plurality of sharding units in real time. If a node failure occurs, the faulty node is switched to a corresponding standby node through a load balancer for data processing.

[0127] In this embodiment, reference Figure 6 , Figure 6 This is a schematic diagram of the high availability design of the first embodiment of the real-time data synchronization system based on the knowledge base of the present invention. The high availability design (i.e., the fault node processing module) is based on the principle of distributed systems and adopts a fault automatic transfer mechanism. When a system node failure is detected, the system can quickly transfer the task to a healthy node for continued execution, thereby minimizing downtime. The present invention uses the Raft consensus algorithm in the high availability design of the system to ensure that when a node fails or the network is abnormal, the system can quickly perform leadership elections and maintain data consistency.

[0128] Furthermore, the fault node processing module monitors multiple nodes in each shard unit in real time, the multiple nodes include a master node and at least one slave node, and obtains monitoring indicator information corresponding to each node; when the monitoring indicator information meets the preset indicator conditions, the node corresponding to the preset indicator conditions is determined to be a faulty node; and the corresponding backup node is determined according to the faulty node through the Raft consensus algorithm.

[0129] The Raft consensus algorithm is:

[0130] term current =term candidate +1

[0131] vote granted=true if term candidate ≥term current

[0132] In the formula, term current is the term of the failed node. candidate is the term of the standby node, vote granted Vote for standby node.

[0133] It should be understood that only when the term of the candidate node candidate Greater than or equal to the term of the current node current This mechanism ensures the consistency and stability of the system during the leadership election process.

[0134] It should be noted that, through the Raft algorithm, the present invention can quickly respond when a node failure is detected, initiate a new leadership election, and ensure the continuity of task transfer and data synchronization through the consistency mechanism in the distributed system. This is particularly important in large-scale real-time data synchronization tasks, which can significantly improve the fault tolerance and high availability of the system, and ensure data consistency and reliable operation of the system.

[0135] It should be understood that the monitoring indicator information includes heartbeat detection, node response time, CPU and memory utilization, data transmission delay time and accuracy. The preset indicator conditions include that the system will periodically send a "heartbeat" signal to each node, and the node needs to return a response within a specified time. If a node does not respond or times out within multiple consecutive cycles (assuming the heartbeat interval is 1 second, the system is set to determine that the node is faulty if it does not receive a heartbeat signal for three consecutive times, so the fault detection time is 3 seconds); or the response time of a node exceeds the set threshold; or the CPU and memory utilization exceeds the preset threshold; or the data transmission delay time exceeds the time threshold; or the data transmission accuracy is lower than the preset threshold.

[0136] It should also be noted that both the master node and the slave node have corresponding backup nodes. The master node of the shard is responsible for processing all write operations, and the backup node monitors the status of the master node and synchronizes data in real time. If the master node fails, the backup node can automatically switch to the new master node within the total node switching time and continue to undertake the task of write operations. This method ensures that even if the master node of a shard fails, the system can still maintain high availability through the backup node.

[0137] When a slave node fails, the standby node can take over the failed slave node within the total node switching time to ensure that the read operation within the shard is not affected. The goal of this design is to ensure that the high concurrent read capability within the shard is not affected by the failure of a slave node.

[0138] The failover time mainly consists of the following stages:

[0139] (1) Fault detection time: Fault detection time is the time required for the system to detect node abnormalities and confirm faults. Taking the heartbeat signal timeout as an example, the heartbeat interval is 1 second. If the signal is not received for 3 consecutive times, the fault is confirmed. The fault detection time is 3 seconds. In a high-concurrency environment, network latency may increase and the reception of heartbeat signals may be affected. To this end, the system adopts a fast heartbeat detection mechanism, shortens the heartbeat interval and increases the detection frequency, shortens the heartbeat interval to 500 milliseconds, reduces the number of consecutive signal failures to 2 times, and shortens the fault detection time to 1 second.

[0140] (2) Leadership election time

[0141] The time it takes for the Raft consensus algorithm to start the leader election process after a node failure is detected. The election timeout is usually between 150 and 300 milliseconds, randomly distributed to avoid election conflicts. Assuming a network communication delay of 50 milliseconds, the election process is generally completed within 150 to 300 milliseconds when the network is in good condition. In a high-concurrency environment, network delays may increase. By using high-performance network equipment and optimizing data transmission paths to improve communication efficiency, the election time can be controlled within 100 to 150 milliseconds.

[0142] (3) Data recovery time

[0143] Data recovery time is the time it takes for the new leader to synchronize uncommitted log entries with other nodes. Assuming the amount of uncommitted log data is 10MB and the network bandwidth is 100Mbps, the data recovery time is about 0.8 seconds (10MB divided by 100Mbps). By receiving the master node log in real time through the standby node, maintaining data consistency, and adopting an incremental update mechanism, the amount of unsynchronized data can be greatly reduced and the data recovery time can be shortened to 50 to 100 milliseconds.

[0144] (4) Task switching time

[0145] Task switching time is the time it takes for a new leader to start processing client requests. Task switching involves state migration and task scheduling, and the processing time is usually between 10 and 50 milliseconds. The number of concurrent tasks, system load, and resource scheduling efficiency all affect this time. By introducing an intelligent load balancing strategy, real-time monitoring of node health status, and fast traffic switching, the task switching time can be shortened to less than 10 milliseconds.

[0146] In a high-concurrency environment, the system needs to handle a large number of concurrent requests, which may lead to increased network latency and intensified resource competition. To solve these problems, the system adopts a distributed architecture and uses microservices and clustering technology to disperse processing pressure. Resource isolation is achieved by using containerization technology to prevent mutual interference. At the same time, message queues and asynchronous programming are used to improve task processing efficiency and further reduce delays when switching tasks.

[0147] Total node switching time

[0148] Based on the above four stages, the total node switching time of the system in a high-concurrency environment can be expressed as: total node switching time = fault detection time + leader election time + data recovery time + task switching time.

[0149] In the best case scenario, the total failover time is:

[0150] 3 seconds (fault detection) + 10 milliseconds (leader election) + 50 milliseconds (data recovery) + 1 millisecond (task switching) = 3.061 seconds

[0151] Under high data volume and network load conditions, the total failover time may increase to:

[0152] 5 seconds (fault detection) + 50 milliseconds (leader election) + 200 milliseconds (data recovery) + 10 milliseconds (task switching) = 5.26 seconds

[0153] Therefore, the system's failover time can usually be controlled within 3-5 seconds, ensuring business continuity in a high-concurrency environment.

[0154] As an independent redundant node within the shard, the standby node can not only back up the master node, but also back up the slave node at the same time, ensuring that the master-slave architecture within the shard has sufficient redundancy. In this case, the standby node can take over the work when the master node or slave node fails through the redundancy strategy without manual intervention.

[0155] Furthermore, the fault node processing module also includes data backup and disaster recovery. The system regularly performs data backup and disaster recovery operations to ensure that data can be quickly restored and business continuity can be maintained in extreme cases. The system also uses the Raft consensus algorithm to ensure data consistency, and uses redundant design and load balancing technology to enhance the system's resilience to hardware failures or network problems.

[0156] In this embodiment, first, the adaptive data synchronization module pre-processes the original change data in the knowledge base source system to obtain the preset format change data, and transmits the preset format change data to the distributed cache optimization module based on the data synchronization strategy. Then the distributed cache optimization module determines the data cache strategy corresponding to the preset format change data, and splits and stores the preset format change data to multiple nodes in each shard unit according to the data cache strategy. After that, the fault node processing module monitors each node in multiple shard units in real time. If there is a node failure, the faulty node is switched to the corresponding backup node for data processing through the load balancer. This embodiment solves the challenges of performance bottlenecks, delay problems and reduced system reliability faced by the prior art when processing large-scale data synchronization by integrating adaptive data synchronization strategies, intelligent distributed cache optimization and high availability design, thereby meeting the complex needs of large-scale data synchronization, thereby improving the efficiency of data processing and the overall system performance.

[0157] Reference Figure 7 , Figure 7 It is a flowchart of the first embodiment of the real-time data synchronization method based on the knowledge base of the present invention.

[0158] like Figure 7 As shown, the real-time data synchronization method based on the knowledge base proposed in the embodiment of the present invention includes:

[0159] Step S10: the adaptive data synchronization module pre-processes the original change data in the knowledge base source system to obtain the preset format change data, and transmits the preset format change data to the distributed cache optimization module based on the data synchronization strategy;

[0160] Step S20: the distributed cache optimization module determines the data cache strategy corresponding to the preset format change data, and splits and stores the preset format change data into multiple nodes in each sharding unit according to the data cache strategy;

[0161] Step S30: The faulty node processing module monitors each node in the multiple sharding units in real time. If there is a node failure, the faulty node is switched to the corresponding standby node for data processing through the load balancer.

[0162] Other embodiments or specific implementations of the knowledge base-based real-time data synchronization method of the present invention can refer to the above-mentioned system embodiments and will not be described in detail here.

[0163] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0164] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0166] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A real-time data synchronization system based on a knowledge base, characterized in that: The real-time data synchronization system based on the knowledge base includes an adaptive data synchronization module, a distributed cache optimization module and a fault node processing module: The adaptive data synchronization module is used to pre-process the original change data in the knowledge base source system to obtain the preset format change data, and transmit the preset format change data to the distributed cache optimization module based on the data synchronization strategy; The distributed cache optimization module is used to determine the data cache strategy corresponding to the preset format change data, and split the preset format change data into multiple nodes in each shard unit according to the data cache strategy; The fault node processing module is used to monitor each node in multiple sharding units in real time. If there is a node failure, the faulty node is switched to the corresponding standby node for data processing through the load balancer.

2. The system according to claim 1, characterized in that The adaptive data synchronization module is further used to determine the data change frequency corresponding to the current network transmission load and the original change data; The adaptive data synchronization module is further used to determine the data synchronization strategy corresponding to the preset format change data through the MAB algorithm based on the current network transmission load and the data change frequency; The MAB algorithm is: In the formula, Q(a t ) is the data synchronization strategy, μ(a t ) is for strategy a t The historical average return is N, the total number of synchronizations, n t (a) is the number of times strategy a is selected, and c is the adjustment parameter.

3. The system according to claim 1, characterized in that The distributed cache optimization module is further used to predict cache hotspot information through a cache hit rate prediction model according to the preset format change data; The distributed cache optimization module is further used to determine the data cache strategy corresponding to the preset format change data based on the cache hotspot information.

4. The system according to claim 3, wherein the cache hit rate prediction model is: ft=σ(Wf·[ht-1,xt]+bf) Where ft is the cache hotspot information output by the forget gate at the current time step, Wf is the weight matrix of the forget gate, ht-1 is the hidden state of the previous time step, xt is the preset format change data, bf is the bias term, and σ is the Sigmoid activation function.

5. The system according to claim 1, wherein: The fault node processing module is used to monitor multiple nodes in each shard unit in real time, and the multiple nodes include a master node and at least one slave node; The fault node processing module is further used to obtain monitoring indicator information corresponding to the master node and the at least one slave node respectively; The fault node processing module is further configured to determine, when the monitoring indicator information meets the preset indicator condition, that the node corresponding to the preset indicator condition is a fault node; The fault node processing module is further used to determine the corresponding standby node according to the fault node through the Raft consensus algorithm; The Raft consensus algorithm is: term current =term candidate +1 vote granted =true if term candidate ≥term current In the formula, term current is the term of the failed node, term candidate is the term of the standby node, vote granted Vote for standby node.

6. The system according to claim 5, characterized in that The fault node processing module is also used to switch the fault node to the corresponding standby node for data processing through the load balancer based on the total node switching time.

7. The system according to claim 6, characterized in that The fault node processing module is also used to determine the total node switching time according to the fault detection time, the leader selection time, the data recovery time and the task switching time.

8. A real-time data synchronization method based on a knowledge base, characterized in that: The real-time data synchronization method based on the knowledge base comprises the following steps: The adaptive data synchronization module pre-processes the original change data in the knowledge base source system to obtain the preset format change data, and transmits the preset format change data to the distributed cache optimization module based on the data synchronization strategy; The distributed cache optimization module determines a data cache strategy corresponding to the preset format change data, and splits and stores the preset format change data into multiple nodes in each sharding unit according to the data cache strategy; The fault node processing module monitors each node in multiple shard units in real time. If there is a node failure, the faulty node is switched to the corresponding backup node for data processing through the load balancer.

9. A real-time data synchronization device based on a knowledge base, characterized in that: The device includes: a memory, a processor, and a knowledge base-based real-time data synchronization program stored in the memory and executable on the processor, wherein the knowledge base-based real-time data synchronization program is configured to implement the steps of the knowledge base-based real-time data synchronization method as described in claim 8.

10. A storage medium, characterized in that: The storage medium stores a real-time data synchronization program based on a knowledge base, and when the real-time data synchronization program based on a knowledge base is executed by a processor, the steps of the real-time data synchronization method based on a knowledge base as claimed in claim 8 are implemented.

Images