A method and device for adjusting polling strategy through Markov chain

Abstract

The present invention relates to the technical field of API data synchronization between systems, and provides a method and device for adjusting polling strategy through Markov chain, wherein the present invention records the random data state sequence of the source system through Markov chain, uses the state sequence data as a sample to calculate the data state transition probability, and then predicts whether the source system has data state at a specific moment, and adopts state record compensation and polling compensation methods to improve the accuracy of state transition probability calculation. The present invention can automatically adjust the polling frequency without manual intervention, can adapt to the data distribution state, and can achieve zero intrusion into the original code, and does not rely on the specific implementation of the source system and the target system.

Description

Technical Field

[0001] The present invention relates to the technical field of API data synchronization between systems, and specifically to a method and device for adjusting a polling strategy through a Markov chain. Background Art

[0002] When synchronizing data between two systems, the target system usually polls data from the source system. However, since the target system does not know when the source system has data, it can only poll the source system periodically. If the polling frequency is too fast, it will waste the hardware resources of the source and target systems. If the polling frequency is too slow, it will greatly affect the timeliness of data synchronization.

[0003] In the prior art, the Chinese patent application number 202311197235.X, a data synchronization method and device for group execution, discloses that by introducing multi-version group management, on the basis of ensuring the consistency of synchronized data and the granularity of split transaction execution, it supports adjusting the grouping rules at any time during the data synchronization process, improves the convenience and fault tolerance of the synchronization program, and improves the data synchronization efficiency. However, this patent is limited to improving the concurrent efficiency in the transaction execution of the database.

[0004] The Chinese patent with application number 202011180641.1, a Markov switching complex network synchronization method with event-driven technology, discloses that through the pinning strategy algorithm, the pinning nodes can be quickly determined in the synchronization process of randomly switched complex networks; according to the pinning feedback gain strategy, the coupling strength and internal coupling matrix can be reasonably selected when the network control efficiency is limited and reliable pinning feedback gain cannot be obtained; the event-driven network synchronization strategy can effectively realize the synchronization of complex networks, while maintaining satisfactory performance under the premise of reducing information exchange and computing load in the network system. However, this patent requires the selection of appropriate links in complex networks and the determination of traction nodes as a condition. When there is an error with the prediction, there is no mechanism for timely feedback and correction of the error, and the accuracy of data synchronization needs to be improved. Summary of the invention

[0005] To this end, the present invention provides a method and device for adjusting the polling strategy through a Markov chain to solve the above-mentioned problems. When polling data across systems, the number of idle times is reduced through prediction, thereby improving the utilization rate of hardware resources. The specific scheme is as follows:

[0006] A method for adjusting a polling strategy through a Markov chain comprises the following steps:

[0007] Step 1: Sample and record the polling data of the source system every minute every day, confirm the data status of the polling data, and obtain the Markov chain random state sequence Xt of the polling data.

[0008] Step 2: Set the sliding window width M in days for the Markov chain random state sequence Xt of the polling data in step 1, and determine the input parameter data to be calculated.

[0009] Step 3: Based on the input data, the probability of data state transition is calculated every minute to obtain the probability of data being generated every minute.

[0010] Step 4: Start data synchronization. The target system sends a polling request to poll the source system every minute. Before polling every minute, check the probability of generating data at the moment. Set the probability threshold to S. If the probability of generating data at the moment is lower than the probability threshold S, skip the polling. Otherwise, perform the polling.

[0011] Step 5: In the Markov chain random state sequence Xt formed by polling data, if data is polled in the current minute, and polling was skipped in the previous minute, the state of the previous minute is also set to have data, that is, state record compensation; state record compensation is used to make up for the error caused by inaccurate probability speculation in the previous minute, and reduce the probability of skipping polling in subsequent calculations.

[0012] Step 6: Poll until the last minute of each day, and perform a fixed polling as polling compensation to read the data that was not polled on that day to correct the data missed due to probability errors.

[0013] Specifically, in step 1, the state of the polling data is recorded every minute of the 1440 minutes of a day. The polling data state of the tth minute is Xt=j, t∈[1,2,…,1440], j∈[0,1], j=0 means no data, j=1 means data; the Markov chain {Xt,t=1,2,…,1440} random state sequence is used to represent the data state every minute.

[0014] Specifically, in step 1, the initial state sequence is defined as {Xt=1,t=1,2,…,1440}, which means that there is data for 1440 minutes.

[0015] Specifically, in step 2, the width of the sliding window is defined as M days, and M×1440 data in the most recent sliding window are collected as input data to be calculated and participate in the next step of data state transition probability calculation.

[0016] Specifically, in step 3, the input data is used as a sample to calculate the state transition probability in the Markov chain random state sequence, and then the probability of generating data every minute on the day of the polling data is calculated.

[0017] The state transition probability Pij=P(Xt+1=j|Xt=i) represents the probability of being in state i at time t and in state j at time t+1. There is a state transition probability matrix every minute. The result of the calculation is to output 1440 state transition probability matrices P.

[0018] Specifically, in step 4, whether or not polling is skipped, the status of whether or not there is data in the current minute must be updated. When polling is skipped, the status of the current minute is no data.

[0019] Specifically, in step 6, the presence or absence of data from 1 to 1439 minutes is represented by a random state sequence of a Markov chain {Xt,t=1,2,…,1439}, and the data state X1440 at 1440 minutes is fixed to 1.

[0020] A device for adjusting a polling strategy through a Markov chain, used to implement the method for adjusting a polling strategy through a Markov chain, is composed of the following functional modules:

[0021] Sampling module: samples and records the polling data of the source system every minute every day, confirms the data status of the polling data, and obtains the Markov chain random state sequence Xt of the polling data; sets the sliding window width M in days for the Markov chain random state sequence Xt of the polling data, and determines the input parameter data to be calculated.

[0022] After data synchronization starts, in the Markov chain random state sequence Xt formed by polling data, if data is polled in the current minute, and polling was skipped in the previous minute, the state of the previous minute is also set to have data, that is, state record compensation; state record compensation is used to make up for the error caused by inaccurate probability speculation in the previous minute, reducing the probability of skipping polling in subsequent calculations.

[0023] Calculation module: Based on the input data, the probability of data state transition is calculated every minute to obtain the probability of data being generated every minute.

[0024] Polling module: accepts polling requests from the target system, polls the source system every minute, and starts data synchronization; before polling every minute, checks the probability of generating data at the current time; sets the probability threshold to S, and if the probability of generating data at the current minute is lower than the probability threshold S, skips the polling, otherwise it performs the polling.

[0025] After data synchronization starts, in the Markov chain random state sequence Xt formed by polling data, a fixed polling is performed until the last minute of each day as polling compensation to read the data that was not polled on that day to correct the data missed by probability errors.

[0026] Furthermore, the device for adjusting the polling strategy through a Markov chain includes at least one processor and a memory, the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the method for adjusting the polling strategy through a Markov chain.

[0027] Beneficial effects:

[0028] 1. The present invention adopts the Markov chain model to calculate the data state transition probability and adopts the statistical method to optimize the polling strategy, which can achieve zero intrusion into the original code and is independent of the specific implementation of the source system and the target system.

[0029] 2. The present invention can automatically adjust the polling frequency without manual intervention, can adapt to any data distribution state, and maintain the timeliness similar to conventional polling.

[0030] 3. The present invention adopts state compensation and polling compensation schemes to effectively optimize and correct the probability prediction error, thereby greatly improving the accuracy of the Markov chain model in calculating the data state transition probability. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 A schematic diagram of the working of functional modules of the device for adjusting the polling strategy through a Markov chain provided by an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of sliding window sampling provided by an embodiment of the present invention;

[0034] Figure 3 This is a polling flow chart provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. The principles and features of the present invention are described below in combination with the accompanying drawings. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict. The embodiments are only used to explain the present invention and are not used to limit the scope of the present invention.

[0036] Each company has different businesses, and the distribution patterns of data over time are also different. For example, documents manually entered into the CRM system generally do not generate data in the middle of the night; finance closes accounts at the end of the month, so generally no data is generated at other times of the month; and there is generally no manually entered data for holidays such as the Mid-Autumn Festival and National Day.

[0037] From the statistical data, there is no strict rule for the distribution of enterprise data on the time axis. Every day has 1440 minutes, which can be represented by a random state sequence such as Markov chain {Xt, t=1,2,…,1440}, where Xt=j means the state at the tth minute is j, j=0 means no data, and j=1 means data. For example, X6=1 means there is data at the 6th minute. The following is the method and device of adjusting the polling strategy through the Markov chain of the present invention.

[0038] A method for adjusting a polling strategy through a Markov chain comprises the following steps:

[0039] Step 1: Sample and record the polling data of the source system every minute every day, confirm the data status of the polling data, and obtain the Markov chain random state sequence Xt of the polling data.

[0040] Step 2: Set the sliding window width M in days for the Markov chain random state sequence Xt of the polling data in step 1, and determine the input parameter data to be calculated.

[0041] Step 3: Based on the input data, the probability of data state transition is calculated every minute to obtain the probability of data being generated every minute.

[0042] Step 4: Start data synchronization. The target system sends a polling request to poll the source system every minute. Before polling every minute, check the probability of generating data at the moment. Set the probability threshold to S. If the probability of generating data at the moment is lower than the probability threshold S, skip the polling. Otherwise, perform the polling.

[0043] Step 5: In the Markov chain random state sequence Xt formed by polling data, if data is polled in the current minute, and polling was skipped in the previous minute, the state of the previous minute is also set to have data, that is, state record compensation; state record compensation is used to make up for the error caused by inaccurate probability speculation in the previous minute, and reduce the probability of skipping polling in subsequent calculations.

[0044] Step 6: Poll until the last minute of each day, and perform a fixed polling as polling compensation to read the data that was not polled on that day to correct the data missed due to probability errors.

[0045] Specifically, in step 1, the state of the polling data is recorded every minute of the 1440 minutes of a day. The polling data state of the tth minute is Xt=j, t∈[1,2,…,1440], j∈[0,1], j=0 means no data, j=1 means data; the Markov chain {Xt,t=1,2,…,1440} random state sequence is used to represent the data state every minute.

[0046] Specifically, in step 1, the initial state sequence is defined as {Xt=1,t=1,2,…,1440}, which means that there is data for 1440 minutes.

[0047] Specifically, in step 2, the width of the sliding window is defined as M days, and M×1440 data in the most recent sliding window are collected as input data to be calculated and participate in the next step of data state transition probability calculation.

[0048] Specifically, in step 3, the input data is used as a sample to calculate the state transition probability in the Markov chain random state sequence, and then the probability of generating data every minute on the day of the polling data is calculated.

[0049] The state transition probability Pij=P(Xt+1=j|Xt=i) represents the probability of being in state i at time t and in state j at time t+1. There is a state transition probability matrix every minute. The result of the calculation is to output 1440 state transition probability matrices P.

[0050] Specifically, in step 4, whether or not polling is skipped, the status of whether or not there is data in the current minute must be updated. When polling is skipped, the status of the current minute is no data.

[0051] Specifically, in step 6, the presence or absence of data from 1 to 1439 minutes is represented by a random state sequence of a Markov chain {Xt,t=1,2,…,1439}, and the data state X1440 at 1440 minutes is fixed to 1.

[0052] A device for adjusting polling strategy through Markov chain, its functional modules are as follows Figure 1 As shown, the method for implementing the method of adjusting the polling strategy through the Markov chain is composed of the following functional modules:

[0053] Sampling module: samples and records the polling data of the source system every minute every day, confirms the data status of the polling data, and obtains the Markov chain random state sequence Xt of the polling data; sets the sliding window width M in days for the Markov chain random state sequence Xt of the polling data, and determines the input parameter data to be calculated.

[0054] After data synchronization starts, in the Markov chain random state sequence Xt formed by polling data, if data is polled in the current minute, and polling was skipped in the previous minute, the state of the previous minute is also set to have data, that is, state record compensation; state record compensation is used to make up for the error caused by inaccurate probability speculation in the previous minute, reducing the probability of skipping polling in subsequent calculations.

[0055] Calculation module: Based on the input data, the probability of data state transition is calculated every minute to obtain the probability of data being generated every minute.

[0056] Polling module: accepts polling requests from the target system, polls the source system every minute, and starts data synchronization; before polling every minute, checks the probability of generating data at the current time; sets the probability threshold to S, and if the probability of generating data at the current minute is lower than the probability threshold S, skips the polling, otherwise it performs the polling.

[0057] After data synchronization starts, in the Markov chain random state sequence Xt formed by polling data, a fixed polling is performed until the last minute of each day as polling compensation to read the data that was not polled on that day to correct the data missed by probability errors.

[0058] Furthermore, the device for adjusting the polling strategy through a Markov chain includes at least one processor and a memory, the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the method for adjusting the polling strategy through a Markov chain. Example

[0059] First, the transition probability in the Markov chain random state sequence is calculated. The sampling module records the state of the source system polling data every minute every day, Xt=j, t∈[1,2,…,1440], j∈[0,1], j=0 means no data, j=1 means data. Define the width of the sliding window M as 7 days, such as Figure 2 , collect the data in the sliding window of the most recent N days as the input parameter of the calculation module, and participate in the next state transition probability calculation. The content of the data is the state value Xt every 1 minute. There are 1440 minutes in a day, and there are 7×1440=10080 values ​​in a sliding window as the input parameter of the calculation module.

[0060] The calculation module uses the data from the sampling module as samples to calculate the transition probability in the Markov chain, and then calculates the probability of generating data every minute on the day of the polling data. The state transition conditional probability Pij=P(Xt+1=j|Xt=i) represents the probability of being in state i at time t and in state j at time t+1. There is a state transition probability matrix every 1 minute. The result of the calculation module is to output 1440 state transition probability matrices P.

[0061] Taking P(X3=j|X2=i) as an example, calculate the probability that the state is i at the 2nd minute and j at the 3rd minute. For example, the state sequence at the 2nd minute and the 3rd minute of the 7 days in the sliding window is as follows, 1 indicates data, and 0 indicates no data.

[0062] Minute 2: {1,1,1,0,0,0,0};

[0063] 3rd minute: {1,0,0,1,0,0,0};

[0064] In the second minute, there are 3 values ​​and 4 no values. In the 3 days with values, there is 1 value and 2 no values ​​in the third minute.

[0065] P(X3=1|X2=1)=1 / 3; P(X3=0|X2=1)=2 / 3;

[0066] P(X3=0|X2=0)=3 / 4; P(X3=1|X2=0)=1 / 4;

[0067] Then the current transition probability from the 2nd minute to the 3rd minute is ;

[0068] Then, the target system initiates a polling request to the polling module and starts polling the source system, such as Figure 3 This is the data polling process. Before polling every minute, check the probability of generating data. Set the probability threshold S to 1 / 20. If the probability of generating data in the current minute is lower than this threshold, skip this polling. Poll once every minute. If data is polled in the second minute, the state is 1. When we decide whether to poll in the third minute, we use the state 1 of the second minute and the transition probability P11=1 / 3 from the second minute to the third minute. The probability of having data in the third minute is 1 / 3, which is higher than the set threshold 1 / 20, so the polling is not skipped.

[0069] Regardless of whether the polling is skipped or not, the sampling module must update the status of whether there is data in the current minute. When the polling is skipped, the status of the current minute is no data, and the status of the current minute is marked as 0. If the data Xt=1 is polled in the current minute, and the polling was skipped in the previous minute, the status of the previous minute is also set to have a value Xt-1=1. This is to compensate for the error caused by the inaccurate probability speculation in the previous minute. This compensated value will reduce the probability of skipping polling in subsequent calculations, that is, sampling status record compensation.

[0070] In addition, at 23:59, the last minute of each day, a fixed poll is performed to read the data that was not polled that day to correct the inaccuracy of the model in prediction, which is called polling compensation.

[0071] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0072] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0073] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention. It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, those skilled in the art can select part or all of them according to actual needs to achieve the purpose of the solution of this embodiment, and there is no limitation here.

Claims

1. A method for adjusting a polling strategy by a Markov chain, comprising the following steps: Step 1: Sample and record the polling data of the source system every minute every day, confirm the data status of the polling data, and obtain the Markov chain random state sequence Xt of the polling data; Step 2: Set the sliding window width M in days for the Markov chain random state sequence Xt of the polling data in step 1, and determine the input parameter data to be calculated; Step 3: Calculate the probability of data state transition every minute based on the input data to obtain the probability of data generation every minute; Step 4: Start data synchronization. The target system sends a polling request to poll the source system every minute. Before polling every minute, check the probability of generating data at the current time. Set the probability threshold to S. If the probability of generating data in the current minute is lower than the probability threshold S, then skip this polling. Otherwise, proceed with the polling. Step 5: In the Markov chain random state sequence Xt formed by polling data, if data is polled in the current minute, and the polling was skipped in the previous minute, the state of the previous minute is also set to have data, that is, state record compensation; the error caused by inaccurate probability speculation in the previous minute is compensated by state record compensation, and the probability of skipping polling in subsequent calculations is reduced; Step 6: Poll until the last minute of each day, and perform a fixed poll as polling compensation to read the data that was not polled on that day to correct the data missed by probability error; In step 1, the state of the polling data is recorded every minute of the 1440 minutes of the day, and the state of the polling data at the tth minute is Xt=j, t [1,2,…,1440],j [0,1], j=0 means no data, j=1 means data; the random state sequence of Markov chain {Xt,t=1,2,…,1440} is used to represent the data state of each minute; In step 1, the initial state sequence is defined as {Xt=1, t=1,2,…,1440}, which means that there is data for 1440 minutes; In step 2, the width of the sliding window is defined as M days, and M*1440 data in the most recent sliding window are collected as input data to be calculated, and participate in the next step of data state transition probability calculation; In step 3, the input data is used as a sample to calculate the state transition probability in the Markov chain random state sequence, and then the probability of generating data every minute on the day of the polling data is calculated; The state transition probability Pij = P(Xt + 1 = j | Xt = i) represents the probability of being in state i at time t and in state j at time t+1. Every minute there is a state transition probability matrix P= The result of the calculation is to output 1440 state transition probability matrices P; In step 6, the presence or absence of data from 1 to 1439 minutes is represented by a random state sequence of a Markov chain {Xt, t=1,2,…,1439}, and the data state X1440 at 1440 minutes is fixed to 1.

2. The method for adjusting the polling strategy by using a Markov chain according to claim 1, characterized in that: In step 4, whether or not polling is skipped, the status of whether or not there is data in the current minute must be updated. When polling is skipped, the status of the current minute is no data.

3. A device for adjusting the polling strategy through a Markov chain, used to implement the method for adjusting the polling strategy through a Markov chain according to any one of claims 1-2, comprising the following functional modules: Sampling module: samples and records the polling data of the source system every minute every day, confirms the data status of the polling data, and obtains the Markov chain random state sequence Xt of the polling data; sets the sliding window width M in days for the Markov chain random state sequence Xt of the polling data, and determines the input parameter data to be calculated; After data synchronization starts, in the Markov chain random state sequence Xt formed by polling data, if data is polled in the current minute, and polling was skipped in the previous minute, the state of the previous minute is also set to have data, that is, state record compensation; state record compensation is used to make up for the error caused by inaccurate probability speculation in the previous minute, and reduce the probability of skipping polling in subsequent calculations; Calculation module: Calculate the probability of data state transition every minute based on the input data to obtain the probability of data generation every minute; Polling module: accepts polling requests from the target system, polls the source system every minute, and starts data synchronization; before polling every minute, checks the probability of generating data at the current time; Set the probability threshold to S. If the probability of generating data in the current minute is lower than the probability threshold S, then skip this polling. Otherwise, proceed with the polling. After data synchronization starts, in the Markov chain random state sequence Xt formed by polling data, a fixed polling is performed until the last minute of each day as polling compensation to read the data that was not polled on that day to correct the data missed by probability errors.

4. The device for adjusting the polling strategy by using a Markov chain according to claim 3, characterized in that: It includes at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after the instructions are executed by the processor, they are used to complete the method for adjusting the polling strategy through a Markov chain as described in any one of claims 1-2.

Images