Data synchronization method for heterogeneous databases based on reinforcement learning

Abstract

The present application belongs to the technical field of data synchronization, and specifically relates to a data synchronization method for heterogeneous databases based on reinforcement learning, aiming to solve the problem that the existing data synchronization method is difficult to adapt to the dynamic environment, leading to synchronization delay or data inconsistency, and unable to realize efficient and accurate data synchronization. The method comprises: obtaining the state information of the current data to be synchronized as the current state; combining the confidence and the current state to calculate the dynamic exploration probability; combining the dynamic exploration probability to select the action of the current data to be synchronized; executing the selected action to update the parameters of the Q network and the target Q network; and synchronizing the data to be synchronized through the trained Q network. The present application introduces an adaptive exploration strategy based on the dynamic characteristics of the environment to adapt to different environmental changes, solves the problem of synchronization delay or data inconsistency, and realizes efficient and accurate data synchronization.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data synchronization, and particularly relates to a data synchronization method and system for heterogeneous databases based on reinforcement learning, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] In modern information systems, data synchronization of heterogeneous databases is a crucial issue. Traditional data synchronization methods often rely on static configurations, making it difficult to adapt to dynamic changes in data sources. These methods perform well in static environments, but in dynamic environments, such as changes in network conditions, fluctuations in data update frequency, etc., fixed rules and preset strategies often cannot be adjusted in time, leading to synchronization delays or data inconsistencies. For example, in a material management system, the update frequency of financial data and personnel data is different, and fixed rule synchronization may result in frequent synchronization of financial data and insufficient synchronization of personnel data, or vice versa. In addition, preset synchronization strategies may not be able to respond to sudden emergencies, resulting in delayed data synchronization and affecting decision-making efficiency.

[0003] In recent years, reinforcement learning, as a machine learning method, has shown great potential in data synchronization by interacting with the environment to learn the optimal strategy. However, existing reinforcement learning methods usually focus on a single objective, such as maximizing synchronization speed or minimizing resource consumption, while in practical applications, data synchronization often involves multiple objectives, such as synchronization speed, resource utilization, and data consistency. Existing multi-objective reinforcement learning methods usually use fixed weights to balance different objectives, which may no longer be applicable in different environments, leading to performance degradation. In addition, existing reinforcement learning methods lack adaptive ability when dealing with dynamic environmental changes, and cannot dynamically adjust the weights of the objective function according to the current environmental changes, thus failing to achieve efficient and accurate data synchronization.

[0004] Based on this, the present application proposes a data synchronization method for heterogeneous databases based on reinforcement learning, which introduces an adaptive exploration strategy based on environmental dynamic characteristics, enabling the algorithm to dynamically adjust the weights of the objective function to adapt to different environmental changes. SUMMARY

[0005] To solve the above problems in the prior art, i.e. to solve the problem that existing data synchronization methods are difficult to adapt in dynamic environments, leading to synchronization delays or data inconsistencies, and failing to achieve efficient and accurate data synchronization, the present application proposes, in a first aspect, a data synchronization method for heterogeneous databases based on reinforcement learning, which comprises:

[0006] S10, obtaining state information of the current data to be synchronized as the current state;

[0007] S20, calculate the arithmetic mean of the difference between the action value function value and the average action value function value in the current state as the confidence based on the current state, time step; calculate the dynamic exploration probability in combination with the confidence and the current state:

[0008] confidence(s t , t) = e0 x exp(-y x (confidence(s t , t) + a d x f d (s t ) + a n x f n (s t ));

[0009] wherein, e(s t , t) represents the dynamic exploration probability, s t represents the current state, t represents the time step, e0 represents the initial dynamic exploration probability, y represents the decay coefficient, confidence(s t , t) represents the confidence, f d (s t ) represents the update frequency of the data in the database within a set time unit, f n (s t ) represents the network fluctuation, i.e. the stability of the network connection within a set time unit, a d , a n both represent weight parameters;

[0010] S30, select the action of the data to be synchronized in the current in combination with the dynamic exploration probability;

[0011] S40, execute the selected action, calculate the immediate reward and observe the state of the next time step; put the current state, the selected action, the immediate reward, the state of the next time step and the importance of experience into the experience replay pool as experience; collect samples from the experience replay pool according to the importance of experience, update the parameters of the Q network and the target Q network;

[0012] S50, cyclically execute S10-S40 until the data synchronization reaches the expected performance indicators, then synchronize the data to be synchronized through the trained Q network.

[0013] In some preferred embodiments, the action of the data to be synchronized in the current is selected in combination with the dynamic exploration probability, and the method is:

[0014] generate a random number between 0 and 1;

[0015] If the difference between the random number and the dynamic exploration probability is less than 1, then an action is randomly selected from the action set as the action of the data to be synchronized. Otherwise, the action with the highest Q value in the current state is selected as the action of the data to be synchronized.

[0016] In some preferred embodiments, the action with the highest Q value in the current state is obtained by:

[0017]

[0018] Where, α t Indicates the selected action, Q(s) t , a; θ) represents in s t Let θ represent the expected reward of taking action a, θ represent the parameters of the action value network, random_score(a) be the random score, cur(a) be the curiosity reward score, h(a) be the historical score, and β1, β2, β3, and β4 be weights.

[0019] In some preferred embodiments, the instant reward is calculated as follows:

[0020]

[0021] Where R represents the immediate reward, and I(t) represents the number of data items identified at time step t. max T(t) represents the maximum number of data items that can be identified, and T(t) represents the amount of data successfully transmitted at time step t. max It is the maximum amount of data that can be transmitted, ρ(s) t a t ) represents the prediction error, i.e., the error in the prediction of the agent in state s. t Execute a t The error between the predicted and actual reward, d(s) t a t ) indicates that the agent is in state s t Perform action a t Similarity to expert demonstration, ω 识别 ω 传输 All represent weighting parameters, δ, k, λ, and μ all represent attenuation coefficients, and η, ψ1 and ψ2 are both positive constants used to adjust the size of the corresponding reward.

[0022] In some preferred embodiments, the importance of the experience is based on the TD error calculation of the experience.

[0023] In some preferred embodiments, the method for updating the parameters of the action value network includes gradient descent.

[0024] The second aspect of the application provides a data synchronization system of a heterogeneous database based on reinforcement learning, which comprises:

[0025] a state acquisition module configured to acquire state information of current data to be synchronized as a current state;

[0026] a probability calculation module configured to calculate an arithmetic mean of a difference between a value of an action value function and a value of an average action value function in the current state as a confidence degree based on the current state and a time step, and calculate a dynamic exploration probability in combination with the confidence degree and the current state;

[0027] ∈(s t ,t)=∈0×exp(-γ×(confidence(s t ,t)+α d ×f d (s t )+α n ×f n (s t ));

[0028] wherein ∈(s t ,t) represents the dynamic exploration probability, s t represents the current state, t represents the time step, ∈0 represents an initial dynamic exploration probability, γ represents a decay coefficient, confidence(s t ,t) represents the confidence degree, f d (s t ) represents an update frequency of data in the database within a set time unit, f n (s t ) represents network fluctuation, i.e., stability of network connection within the set time unit, and α d and α n both represent weight parameters;

[0029] an action selection module configured to select an action of the current data to be synchronized in combination with the dynamic exploration probability;

[0030] a network update module configured to execute the selected action, calculate an immediate reward and observe a state of a next time step, put the current state, the executed selected action, the immediate reward, the state of the next time step and importance of experience into an experience replay pool as experience, collect samples from the experience replay pool according to the importance of experience, and update parameters of a Q network and a target Q network;

[0031] a data synchronization module configured to cyclically execute the state acquisition module and the network update module until data synchronization reaches an expected performance index, and then synchronize the data to be synchronized through the trained Q network.

[0032] In a third aspect, the present invention provides an electronic device, the device comprising:

[0033] At least one processor, and a memory communicatively connected to at least one of the processors;

[0034] The memory stores instructions that can be executed by the processor to implement the aforementioned data synchronization method for heterogeneous databases based on reinforcement learning.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions for execution by a computer to implement the above-described data synchronization method for heterogeneous databases based on reinforcement learning.

[0036] The beneficial effects of this invention are:

[0037] This invention introduces an adaptive exploration strategy based on the dynamic characteristics of the environment to adapt to different environmental changes, solving the problems of synchronization delay or data inconsistency, and achieving efficient and accurate data synchronization.

[0038] 1) This invention introduces an adaptive exploration strategy based on the dynamic characteristics of the environment. It not only considers the confidence level of the current state, but also combines the dynamic characteristics of the environment, such as data update frequency and network fluctuations. It can dynamically adjust the weight of the objective function to adapt to different environmental changes, thereby achieving efficient and accurate data synchronization and ensuring that the data in each database remains consistent.

[0039] 2) This invention solves the problems of lack of immediate feedback on rewards, low search efficiency, and credit allocation in existing reinforcement learning by introducing reward shaping, curiosity-driven learning, and demonstration learning;

[0040] 3) This invention is applicable to synchronization tasks of various heterogeneous databases and has wide applicability and scalability. Attached Figure Description

[0041] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0042] Figure 1 This is a flowchart illustrating a data synchronization method for heterogeneous databases based on reinforcement learning, according to one embodiment of the present invention. Detailed Implementation

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0044] The present application will be described in further detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, but not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings.

[0045] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0046] A data synchronization method of a heterogeneous database based on reinforcement learning according to a first embodiment of the present application, as shown in Figure 1 includes the following steps:

[0047] S10, obtaining state information of current data to be synchronized as a current state;

[0048] S20, calculating an arithmetic mean of a difference between an action value function value and an average action value function value in the current state based on the current state and a time step as a confidence; combining the confidence and the current state to calculate a dynamic exploration probability:

[0049] ∈(s t ,t)=∈0×exp(-γ×(confidence(s t ,t)+α d ×f d (s t )+α n ×f n (s t ));

[0050] Wherein, ∈(s t ,t) represents a dynamic exploration probability, s t represents a current state, t represents a time step, ∈0 represents an initial dynamic exploration probability, γ represents a decay coefficient, confidence(s t ,t) represents a confidence, f d (s t ) represents the update frequency of data in the database within a set time unit, f n (s t ) represents network fluctuation, i.e. the stability of network connection within a set time unit, and αd , a n all represent weight parameters;

[0051] S30, in combination with the dynamic exploration probability, selecting an action of the current data to be synchronized;

[0052] S40, performing the selected action, calculating the immediate reward and observing the state of the next time step; putting the current state, the selected action, the immediate reward, the state of the next time step and the importance of experience into the experience replay pool as experience; collecting samples from the experience replay pool according to the importance of experience, updating the parameters of the Q network and the target Q network;

[0053] S50, cyclically performing S10-S40 until the data synchronization reaches the expected performance indicators, then synchronizing the data to be synchronized through the trained Q network.

[0054] In order to more clearly illustrate the present application, the steps of an embodiment of the method of the present application are described in detail below with reference to the accompanying drawings.

[0055] The first embodiment of the present application provides a data synchronization method for heterogeneous databases based on reinforcement learning. The data synchronization is based on the ASRL (Adaptive State Representation Learning) algorithm. By introducing an adaptive exploration strategy based on environmental dynamic characteristics, the algorithm can dynamically adjust the weights of the objective function to adapt to different environmental changes. The ASRL algorithm not only considers the confidence of the current state, but also combines the dynamic characteristics of the environment, such as data update frequency and network fluctuation, to finely adjust the exploration probability, thereby achieving efficient and accurate data synchronization and ensuring that the data in each database remains consistent. In addition, the present application introduces reward shaping, curiosity-driven learning, hierarchical reinforcement learning and demonstration learning to solve the problems of lack of immediate feedback in existing reinforcement learning, low search efficiency, credit allocation, and local optimal solution.

[0056] Specifically as follows:

[0057] S10, obtaining state information of the current data to be synchronized as the current state;

[0058] In this embodiment, the Q network and the target Q network in reinforcement learning are first initialized, including initializing the experience replay pool, the initial state, initializing the dynamic exploration probability parameters, the decay coefficient, the data update frequency weight, the network fluctuation weight, etc.

[0059] Then state awareness is performed to obtain the current state. The state is a description of the environment, representing the current environmental situation. In the context of data synchronization, the state includes the state of financial data, the state of personnel data, the network connection state, etc. For example, in a material management system, the current state includes financial data (such as budget, expenditure, and income) and personnel data (such as basic information, training records, and health status). For example, the financial data is: budget 1 million, expenditure 500,000, and income 300,000; the personnel data is: total number of 100, 95 in good health, and 5 with illness.

[0060] S20, based on the current state, the time step, calculate the arithmetic mean of the difference between the action value function value and the average action value function value in the current state as the confidence; combine the confidence and the current state to calculate the dynamic exploration probability;

[0061] Traditional multi-objective optimization methods usually use fixed weights to balance different objectives. In this embodiment, an adaptive exploration strategy based on the dynamic characteristics of the environment is introduced. This strategy not only considers the confidence of the current state, but also combines the dynamic characteristics of the environment, such as data update frequency and network fluctuations, to adjust the exploration probability more finely, so that the algorithm can dynamically adjust the weights of the objective function to adapt to different environmental changes. Specifically as follows:

[0062] Calculate the arithmetic mean of the difference between the Q value (i.e. the Q value of a certain action in the current state, in reinforcement learning, the Q value is usually estimated by a Q network (a neural network). The action value function value is the Q value, which represents the expected value of the cumulative reward from the current time step to all future time steps after taking an action in a state. The higher the Q value, the better the long-term return of taking that action. Actions include synchronizing financial data, synchronizing personnel data, and pausing synchronization; input the current state and each action into the Q network to obtain the Q value) and the average Q value (i.e. the average Q value of all actions in the current state) in the current state as the confidence (the smaller the deviation, the higher the confidence, indicating that the algorithm has a higher certainty about the current state); combine the confidence and the current state to calculate the dynamically adjusted exploration probability, as shown in the following formula:

[0063] ∈(s t ,t)=∈0×exp(-γ×(confidence(s t ,t)+α d ×f d (s t )+α n ×f n (s t ));

[0064] Where ∈(s t ,t) represents the dynamic exploration probability, s trepresents the current state, t represents the time step, represents the initial dynamic exploration probability, the initial exploration probability is the set exploration probability at the beginning of the algorithm, which is usually set to a higher value (preferably 0.5 or 0.8 in the present application) to ensure that the algorithm can fully explore different actions in the early stage of learning, represents the decay coefficient, confidence(s t represents the confidence, f d (s t ) represents the update frequency of data in the database within a set time unit, that is, the ratio of the number of data updates to time, reflecting the activity level of the data, the higher the data update frequency, the more exploration the algorithm needs to adapt to new data, f n (s t ) represents network fluctuation, that is, the stability of network connection within a set time unit (the ratio of the standard deviation of network delay within a set time unit to the average value of network delay within a set time unit), reflecting the stability of network connection, the greater the network fluctuation, the more exploration the algorithm needs to cope with the impact of network instability, and d , and n all represent weight parameters.

[0065] Dynamic adjustment of exploration probability is an important part of reinforcement learning, especially in the ASRL (Adaptive State Representation Learning) algorithm for adaptive multi-objective optimization, which helps the ASRL algorithm to balance between exploring new actions and using known best actions.

[0066] S30, in combination with the dynamic exploration probability, selects an action for the current data to be synchronized;

[0067] In this embodiment, the specific process of selecting an action for the current data to be synchronized is as follows:

[0068] A random number between 0 and 1 is generated;

[0069] If the random number is less than the difference between 1 and the dynamic exploration probability (i.e. 1-dynamic exploration probability), a random action is selected from the action set as the action for the current data to be synchronized, otherwise, the action with the highest Q value in the current state is selected as the action for the current data to be synchronized.

[0070] The action with the highest Q value in the current state is obtained by:

[0071]

[0072] wherein a t represents the selected action, Q(s t , a; theta) represents the Q value of the action a in the state s tThe expected return of the action a is taken, θ represents the parameters of the action value network (i.e., the Q network), random_score(a) is a random score used to increase the randomness of exploration, which can help the algorithm avoid falling into a local optimal solution and ensure sufficient exploration in the early stage, cur(a) is a curiosity reward score used to encourage the algorithm to explore unknown states and actions, which can help the algorithm discover new and valuable strategies, h(a) represents the historical score reflecting the performance of the action in the past, which can help the algorithm use past successful experience to avoid repeating mistakes, and β1, β2, β3, β4 are weights.

[0073] S40, performing the selected action, calculating the immediate reward and observing the state of the next time step; putting the current state, the selected action, the immediate reward, the state of the next time step and the importance of experience into the experience replay pool as experience; collecting samples from the experience replay pool according to the importance of experience, updating the parameters of the Q network and the target Q network;

[0074] In reinforcement learning, the sparsity of reward signals is a common problem, especially in complex tasks. When the reward signal is sparse (i.e., in most cases, the actions performed by the agent will not produce any positive or negative reward immediately. Only at a certain key moment or when a certain goal state is reached, the agent will receive a reward signal. For example, in a maze navigation task, the agent may need to move several times to reach the end point, and only when it reaches the end point will it receive a positive reward), the agent may need to perform many actions to obtain useful feedback, which greatly increases the difficulty of learning. This is manifested in the following aspects:

[0075] 1) Lack of immediate feedback: In a sparse reward environment, it is difficult for the agent to evaluate the good or bad of its actions through immediate feedback. Since most actions do not produce rewards immediately, it is difficult for the agent to know which actions are beneficial and which are harmful; this lack of immediate feedback makes it difficult for the agent to quickly learn effective strategies through trial and error;

[0076] 2) Low exploration efficiency: In order to find an effective strategy, the agent needs to conduct a large amount of exploration. In a sparse reward environment, the exploration process can be very inefficient, because the agent needs to try many different action combinations to accidentally discover a path that can obtain a reward;

[0077] 3) The Credit Assignment Problem: Credit assignment refers to attributing the final reward to a series of previous actions. This problem is particularly prominent in sparse reward environments because agents need to trace long-term rewards back to actions from a long time ago; for example, in data synchronization tasks, an agent may need to perform a series of complex synchronization operations to successfully complete a synchronization. If the final reward is only provided after synchronization is complete, it is difficult for the agent to determine which specific operations played a key role in successful synchronization;

[0078] 4) Local Optimum: In sparse reward environments, agents are prone to getting trapped in local optima. Due to the lack of immediate feedback, the agent may find a relatively good solution early on and continue to linger around this local optimum, unable to further explore better global solutions. This situation may cause the agent's learning to stagnate and prevent it from finding the optimal synchronization strategy.

[0079] To address the aforementioned problems, this embodiment introduces two main methods. First, it introduces intermediate rewards to guide the agent's learning faster. These intermediate rewards provide positive feedback as the agent approaches the target state, thus accelerating the learning process. Second, it encourages the agent to explore unknown states to improve learning efficiency. Intrinsic rewards are generated by calculating the agent's prediction error of the environment, thereby motivating the agent to explore new actions and states. Finally, expert demonstrations guide the agent's learning. Experts can provide successful synchronization examples to help the agent understand and learn effective synchronization strategies more quickly. Specifically, as shown in the following formula:

[0080]

[0081] Where R represents the immediate reward, and I(t) represents the number of data items identified at time step t. max T(t) represents the maximum number of data items that can be identified, and T(t) represents the amount of data successfully transmitted at time step t. max It is the maximum amount of data that can be transmitted, ρ(s) t a t ) represents the prediction error, i.e., the error in the prediction of the agent in state s. t Execute a t The error between the predicted and actual reward, d(s) t a t ) indicates that the agent is in state s t Perform action a t Similarity to expert demonstration, ω 识别 ω 传输 All represent weighting parameters, δ, k, λ, and μ all represent attenuation coefficients, and η, ψ1 and ψ2 are both positive constants used to adjust the size of the corresponding reward.

[0082] Then, the current state, the selected action, the immediate reward, the next time step state and the importance of the experience are put into the experience replay pool as an experience;

[0083] The importance of the experience is calculated based on the TD error of the experience, and the TD error reflects the difference between the current experience and the model prediction. A larger TD error means that the experience has a greater potential contribution to the update of the model. The calculation process is as follows: calculate the difference between the maximum Q value in the next state and the Q value after taking the operation in the current state, take the absolute value of the TD error, sum it with a very small positive number (to prevent the importance of the experience from being 0), and then normalize it.

[0084] Samples are collected from the experience replay pool according to the importance of the experience (the sampling probability is calculated according to the importance of the experience, and the samples are collected according to the sampling probability, for example, 1000 experiences, P represents the importance of the experience, is equal to 0.6, 1000 experiences are collected according to the sampling probability P1), the parameters of the Q network and the target Q network are updated, including calculating the target Q value, updating the Q network and the target Q network, which are as follows:

[0085] For each collected sample, the target Q value is calculated as the discount factor multiplied by the maximum Q value of all actions predicted by the target Q network in the next state + the sum of the immediate reward corresponding to the experience replay pool sample; for example, a batch of experiences (s1, a1, 10, s2, p1) are sampled from the experience replay pool, which are the current state, the selected action, the immediate reward, the next time step state and the importance of the experience, and the target Q value y1 is calculated: y1 = 10 + 0.9 multiplied by the maximum Q value of all actions predicted by the target Q network in the next state.

[0086] For each sampled sample, a loss function is calculated between the predicted value of the Q network and the target Q value (the loss function of the present application is preferably a mean square error loss function, and the loss function in the traditional Q learning has the following two problems: 1) the target Q value is calculated by maximizing the operation, which will cause the overestimation bias of the Q value; 2) the action values of adjacent states may have implicit correlation, but the traditional method does not explicitly model it, so the loss function in this embodiment is Wherein, L represents the loss function, N represents the number of samples, confidence represents the confidence, respectively represent the predicted value of the Q network and the target Q value, and L represents a similarity adjustment parameter, sim(s i , s j ) represents state similarity, represents the state s iThe gradient of the loss function on the parameters of the Q network is calculated by back propagation, and the parameters of the Q network are updated using the gradient descent method.

[0087] Using a soft update method, the parameters of the target network are gradually updated based on the updated parameters of the Q network: θ' <- τθ + (1-τ)θ', τ is a smoothing coefficient.

[0088] S50, S10-S40 are executed in a loop until the data synchronization reaches the expected performance indicators, then the data to be synchronized is synchronized by the trained Q network.

[0089] In this embodiment, when the Q network and the target Q network are trained, the trained Q network is loaded, the state information of the current data to be synchronized is obtained, the dynamic exploration probability is calculated, and the action of the current data to be synchronized is selected in combination with the dynamic exploration probability. In addition, the next state and the reward can also be observed, and the effect of the action is evaluated. If it is found that the environment changes greatly or the synchronization effect is not good, these situations can be recorded for further optimization and adjustment in the future.

[0090] In summary, the present application can find the best balance between exploration and utilization by dynamically adjusting the exploration probability, reduce unnecessary resource consumption, and improve the overall performance of the system; and dynamically adjust the weight of the target function according to the change of the current environment, to ensure that the optimal synchronization strategy can be found in different scenarios, suitable for synchronization tasks of multiple heterogeneous databases, and has wide applicability and expansibility.

[0091] The second embodiment of the present application is a data synchronization system for heterogeneous databases based on reinforcement learning, which comprises:

[0092] The state acquisition module is configured to obtain the state information of the current data to be synchronized as the current state;

[0093] The probability calculation module is configured to calculate the arithmetic mean of the difference between the action value function value and the average action value function value under the current state based on the current state and the time step as the confidence; and calculate the dynamic exploration probability in combination with the confidence and the current state:

[0094] ∈(s t , t) = ∈0 x exp(-γ x (confidence(s t , t) + α d x f d (s t ) + α n x f n (s t ));

[0095] Wherein, the dynamic exploration probability is represented by e(st, t), the current state is represented by st, the time step is represented by t, the initial dynamic exploration probability is represented by e0, the decay coefficient is represented by g, the confidence is represented by confidence(st, t), the update frequency of the data in the database within a set time unit is represented by fd(st), the network fluctuation, that is, the stability of the network connection within a set time unit is represented by fn(st), and the weight parameters are represented by ad and an.

[0096] The action selection module is configured to select an action of the data to be synchronized in combination with the dynamic exploration probability.

[0097] The network updating module is configured to execute the selected action, calculate an immediate reward, and observe a state of a next time step; put the current state, the executed selected action, the immediate reward, the state of the next time step, and importance of experience into an experience replay pool as experience; collect samples from the experience replay pool according to the importance of experience, and update parameters of the Q network and the target Q network.

[0098] The data synchronization module is configured to cyclically execute the state acquisition module and the network updating module until data synchronization reaches an expected performance index, and then synchronize the data to be synchronized through the trained Q network.

[0099] The skilled in the art of the technical field can clearly understand that, for the convenience and brevity of description, the specific working process and related description of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0100] It should be noted that the data synchronization system of the heterogeneous database based on reinforcement learning provided in the foregoing embodiments is only exemplified by the division of the foregoing functional modules, and in actual application, the foregoing functions can be completed by different functional modules according to needs, that is, the modules or steps in the embodiments of the present application are further decomposed or combined, for example, the modules in the foregoing embodiments can be combined into one module, or can be further split into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present application are only for distinguishing the modules and steps, and should not be considered as an improper limitation of the present application.

[0101] The electronic device of the third embodiment of the present application comprises at least one processor, and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the data synchronization method of the heterogeneous database based on reinforcement learning.

[0102] The fourth embodiment of the application is a computer readable storage medium storing computer instructions for being executed by a computer to implement the above-mentioned method for data synchronization of heterogeneous databases based on reinforcement learning.

[0103] Those skilled in the art of the technical field can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic device, the readable storage medium and the related descriptions described above can refer to the corresponding processes in the foregoing method examples, which will not be described here.

[0104] Those skilled in the art should be able to realize that the modules and method steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. The programs corresponding to the software modules and method steps can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been generally described in the foregoing description. Whether the functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0105] The terms "first", "second", "third" and the like are used to distinguish similar objects, not to describe or indicate a particular order or sequence.

[0106] So far, the technical solution of the present application has been described in combination with the preferred embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical solutions after such changes or replacements will fall within the protection scope of the present application.

Claims

1. A data synchronization method for a heterogeneous database based on reinforcement learning, characterized in that, The method comprises: S10, obtaining state information of current data to be synchronized as a current state; S20, calculating an arithmetic mean of a difference between an action value function value and an average action value function value in the current state based on the current state and a time step as a confidence level; and calculating a dynamic exploration probability based on the confidence level and the current state; e(s t , t) = e0 x exp(-y x (confidence(s t , t) + a d x f d (s t ) + a n x f n (s t )); wherein ∈ (s t , t) represents a dynamic exploration probability, s t represents a current state, t represents a time step, ∈0represents an initial dynamic exploration probability, γ represents a decay coefficient, confidence (s t , t) represents a confidence degree, f d (s t ) represents an update frequency of data in the database within a set time unit, f n (s t ) represents a network fluctuation, i.e. stability of the network connection within a set time unit, and α d and α n both represent weight parameters; S30, selecting an action of the current data to be synchronized based on the dynamic exploration probability; S40, executing the selected action, calculating an immediate reward and observing a next state; putting the current state, the executed selected action, the immediate reward, the next state and importance of experience into an experience replay pool as experience; collecting samples from the experience replay pool according to the importance of experience; and updating parameters of a Q network and a target Q network; S50, repeatedly executing S10-S40 until data synchronization reaches an expected performance index, and then synchronizing the data to be synchronized through the trained Q network.

2. The data synchronization method for heterogeneous databases based on reinforcement learning according to claim 1, characterized in that, The method for selecting the action of the current data to be synchronized based on the dynamic exploration probability comprises: generating a random number between 0 and 1; if the random number is less than a difference between 1 and the dynamic exploration probability, randomly selecting an action from an action set as the action of the current data to be synchronized, otherwise, selecting an action with the highest Q value in the current state as the action of the current data to be synchronized. 3.The method of Claim 2, wherein, The method for obtaining the action with the highest Q value in the current state comprises: where a t represents the selected action, Q(s t , a; θ) represents the expected return of taking action a at s t , and θ represents the parameters of the action value network. random_score(a) is the random score, cur(a) is the curiosity reward score, h(a) represents the history score, and β1, β2, β3, and β4 are weights.

4. The data synchronization method for heterogeneous databases based on reinforcement learning according to claim 3, characterized in that, The method for calculating the immediate reward comprises: where R represents the immediate reward, I(t) represents the number of data items identified at time step t, I max represents the maximum number of identified data items, T(t) represents the amount of data successfully transmitted at time step t, T max is the maximum amount of transmitted data, p(s t , a t ) represents the prediction error, that is, the error between the predicted value of the reward after the agent performs a t in state s t and the actual reward, d(s t , a t ) represents the similarity between the agent performing action a t in state s t and the expert demonstration, w 识别、 w 传输 both represent weight parameters, d, k, l, m all represent decay coefficients, h, p1, p2 are both positive constants for adjusting the size of the corresponding reward. 5.The data synchronization method of the heterogeneous database based on reinforcement learning according to claim 4, characterized in that, The importance of the experience is calculated based on a TD error of the experience.

6. The data synchronization method for heterogeneous databases based on reinforcement learning according to claim 5, characterized in that, The method for updating the parameters of the action value network comprises a gradient descent method.

7. A data synchronization system of heterogeneous databases based on reinforcement learning, characterized in that, The system comprises: a state obtaining module configured to obtain state information of current data to be synchronized as a current state; a probability calculating module configured to calculate an arithmetic mean of a difference between an action value function value and an average action value function value in the current state based on the current state and a time step as a confidence level; and calculate a dynamic exploration probability based on the confidence level and the current state; e(s t , t) = e0 x exp(-y x (confidence(s t , t) + a d x f d (s t ) + a n x f n (s t )); wherein ∈ (s t , t) represents a dynamic exploration probability, s t represents a current state, t represents a time step, ∈0represents an initial dynamic exploration probability, γ represents a decay coefficient, confidence (s t , t) represents a confidence degree, f d (st) represents an update frequency of data in the database within a set time unit, f n (s t ) represents a network fluctuation, i.e. stability of the network connection within a set time unit, and α d and α n both represent weight parameters. an action selecting module configured to select an action of the current data to be synchronized based on the dynamic exploration probability; a network updating module configured to execute the selected action, calculate an immediate reward and observe a next state; put the current state, the executed selected action, the immediate reward, the next state and importance of experience into an experience replay pool as experience; collect samples from the experience replay pool according to the importance of experience; and update parameters of a Q network and a target Q network; a data synchronization module configured to repeatedly execute the state obtaining module-the network updating module until data synchronization reaches an expected performance index, and then synchronize the data to be synchronized through the trained Q network.

8. An electronic device, comprising: The electronic device comprises: at least one processor, and a memory in communication connection with the at least one processor; The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the data synchronization method of the heterogeneous database based on the reinforcement learning in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the data synchronization method of the heterogeneous database based on the reinforcement learning in any one of claims 1-6.

Images