Data repetition checking method

Abstract

The invention relates to the technical field of data repetition checking, aims to solve the problems that in an existing anti-re-checking technology, due to the fact that the requirements for the system performance, the real-time performance and the accuracy are high, the system processing complexity is increased, and the system performance is reduced, and provides a data repetition checking method. According to the method, the repetition probability is evaluated by simulating the time acceleration attenuation function, so that the repetition probability decreases in an accelerated manner along with the increase of time, repeated instructions or data are effectively prevented, the safety can be improved, the risk is reduced, the efficiency is improved, and the method is easy to implement.

Description

Technical Field

[0001] The present invention relates to the technical field of data duplication verification, and in particular to a data duplication verification method. Background Art

[0002] In modern information systems, request duplication is a common and important problem that needs to be solved. Due to network delays or repeated submission of requests caused by frequent user operations, the system may generate a large number of unnecessary operations, bringing risks and economic losses to the enterprise. For example, in a financial transaction system, a user may click the submit button multiple times due to network response delays, resulting in the same instruction being sent repeatedly. If the system cannot identify and process such duplications, multiple unnecessary transactions may be generated, bringing significant risks and economic losses to users and systems. Similar situations may also occur in various scenarios such as order systems and ticketing systems. Generally speaking, the shorter the time interval between receiving two identical requests, the higher the probability that it is a duplicate request, and as the interval time gradually increases, the probability of duplication will show a rapid downward trend.

[0003] At present, there are several anti-duplicate verification technologies, such as idempotency identification, timestamp and sequence number, token bucket algorithm, digital signature and check code, etc. Each has its own unique technical features and implementation methods, but also comes with some technical challenges and defects. Among them, idempotency identification uses a unique identification key to prevent request duplication. The client generates the key, and the server stores the key status to determine whether the request has been processed, but it requires additional storage space, and key management may become a performance bottleneck; timestamp and sequence number identify the uniqueness and order of each request, assign a sequence number to each user or session, and the sequence number increases with each operation and is used in combination with the timestamp, but strict time synchronization is required, and time error or sequence number overflow may lead to misjudgment; token bucket or funnel algorithm controls the request rate to avoid repeated submission caused by frequent operations, uses containers to store tokens, and adds tokens at a set rate, but it is not flexible in handling burst traffic, and token management may become a performance bottleneck; digital signature and check code generates a unique signature or check code through encryption to verify data integrity and consistency, generates a check code or signature before data is sent, and the receiver verifies it, but the encryption and decryption process consumes resources and may affect the system processing speed. These methods have some limitations and shortcomings in practical applications, and have high requirements for system performance, real-time and accuracy, which may lead to problems such as increased system complexity and performance degradation. Therefore, further research and development of new technical solutions are needed to address the shortcomings of existing technologies and improve business security and efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a data duplication verification method to solve the problem that the existing anti-duplication verification technology has high requirements for system performance, real-time and accuracy, resulting in increased system processing complexity and decreased system performance. The verification method can improve the security and efficiency of the business.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A data duplication verification method evaluates the duplication probability by simulating a time accelerated decay function, so that the duplication probability decreases faster with the increase of time, thereby achieving effective prevention of duplicate instructions or data.

[0007] In some embodiments, the data duplication verification method specifically includes the following steps:

[0008] Step 1: Extract key feature fields of request data;

[0009] Step 2: Calculate the Hash value of the requested data;

[0010] Step 3: Store the Hash value in the historical Hash value database to record the time when the payment instruction is received;

[0011] Step 4: Calculate the hash value of the new request and compare it with the hash value in the historical record. If the hash value is repeated, record the time and calculate the interval t between the two identical instructions.

[0012] Step 5: Select a decay function and use the time fit to estimate the probability of repetition;

[0013] Step 6: The user sets the warning level corresponding to different repetition probabilities;

[0014] Step 7: Calculate the probability of request repetition at interval t and output the risk level.

[0015] In step 1, the key features of the requested data include number, name, and amount.

[0016] In step 1, a hash function is used to perform a hash operation on the request data to generate a unique hash value.

[0017] In step 2, the calculation formula for calculating the Hash value of the request data is:

[0018] hash_output = H(hash_input)

[0019] In the formula, H represents the hash function, Hash_input represents the key features such as number, name, amount, etc., and Hash_output represents the hash value of the request generated by the hash algorithm.

[0020] In step 4, you need to select the time unit including seconds, minutes, and hours.

[0021] In step 5, the time fitting degree N(t) is calculated as follows:

[0022]

[0023] In the formula, N0 represents the initial value of N at time 0, N(t) is the value of N at time t, N0=N(0) is the initial value of N at time 0, N finish It means that after m days, it decays to N finish , N init Indicates that the decay starts from a non-N0 position, and this non-N0 position is defined as N init , m represents the number of days;

[0024] The time fitting degree N(t) is used to estimate the probability of repetition. The longer the time interval, the lower the corresponding probability of repetition.

[0025] In step 5, the decay function includes an exponential decay function, a power law decay function and a logistic decay function.

[0026] In step 6, different warning levels are customized according to the calculated repetition probability.

[0027] In step 7, the Logistic decay model is selected to fit the process of Logistic decay of the probability of repeated payment over time. The time interval t is input into the Logistic decay model to calculate the probability of request repetition at interval time t.

[0028] Compared with the prior art, the data duplication verification method provided by the present invention has the following beneficial effects:

[0029] The present invention can improve security. By using the Hash algorithm and the decay function to perform anti-duplication verification, the occurrence of duplication can be effectively prevented, thereby improving the security of data transmission.

[0030] The present invention can reduce the risk. According to the calculated repetition probability, different warning levels can be set, thereby timely reminding users and administrators to pay attention to possible repetition risks.

[0031] The present invention can improve efficiency. By performing Hash calculation on data, it can be quickly determined whether there is a suspicion of duplication, thereby improving the efficiency of processing data requests.

[0032] The present invention is easy to implement. This method is based on the existing Hash algorithm and mathematical functions, does not require additional hardware equipment or complex software changes, and is therefore easy to implement in an actual system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for the technical description.

[0034] Figure 1 A flow chart of the data duplication verification method provided by the present invention;

[0035] Figure 2 This is a function effect diagram of the data duplication verification method provided by the present invention. DETAILED DESCRIPTION

[0036] The following is further explained in detail through specific implementation methods.

[0037] like Figure 1 As shown, the present invention provides a data duplication verification method, which evaluates the duplication probability by simulating a time accelerated decay function, so that the duplication probability decreases rapidly with the increase of time, thereby achieving effective prevention of repeated instructions or data, which is in line with the actual situation. The verification method includes:

[0038] Step 1: Extract key feature fields of request data;

[0039] Step 2: Calculate the Hash value of the requested data;

[0040] Step 3: Store the Hash value in the historical Hash value database to record the time when the payment instruction is received;

[0041] Step 4: Every time the treasury management system receives a new payment instruction, it calculates the Hash value of the new request and compares it with the Hash value in the historical record. If the Hash value is repeated, the time is recorded, and the interval t between the two identical instructions is calculated. The treasury management system is a system used to record instructions and time. The receipt time is recorded when an instruction is received for the first time, and the receipt time is also recorded when the same instruction is received again, so the time is recorded twice.

[0042] Step 5: Select a decay function and use the time fit to estimate the probability of repetition;

[0043] Step 6: The user sets the warning level corresponding to different repetition probabilities;

[0044] Step 7: Calculate the probability of request repetition at interval t and output the risk level.

[0045] Therefore, the present invention designs an effective anti-duplicate verification technology, which verifies and manages the received requests by combining unique identifiers with time trend models and other mechanisms, thereby ensuring the uniqueness of requests, preventing data conflicts and maintaining business stability.

[0046] In step 1, the treasury management system extracts key features from the request, such as number, name, amount, etc. (or information such as the payment unit, payment account, amount, etc.), uses the hash function to perform hash operations on the request data, and generates a unique hash value.

[0047] It should be noted that, regarding the generation of unique identifiers, the hash algorithm, as an algorithm that converts input data of any length into output data of fixed length through a hash function, is a suitable choice. This output data is called a hash value, which is used to uniquely represent the original input data. Hash algorithms are widely used in fields such as data integrity verification and digital signatures. Data integrity verification is to ensure that it has not been tampered with during transmission or storage by calculating the hash value of a file or data block. Digital signatures use private keys to sign the hash value of data to prove the integrity and authenticity of the data. In addition, hash algorithms are also used in scenarios such as hash tables and duplicate data detection to improve query efficiency and reduce data redundancy by uniquely identifying data. The determinism of the hash algorithm means that the same output can always be generated for the same input, and the efficiency ensures that the hash function can quickly calculate the hash value of the input data. Irreversibility means that the original input data cannot be reversed based on the hash value. Uniformity requires that the hash values ​​of different input data should be evenly distributed in the output range to avoid conflicts. Anti-collision requires that the probability of finding two different input data with the same hash value is extremely small.

[0048] Common hash algorithms include MD5, SHA-1, SHA-2 series, and SHA-3 series. MD5 is a hash algorithm with an output length of 128 bits. It is fast and easy to implement, but it is no longer suitable for security-related scenarios due to serious collision problems. SHA-1 has an output length of 160 bits and is more secure than MD5, but it has also been found to have security vulnerabilities and a higher risk of collision. The SHA-2 series includes SHA-224, SHA-256, SHA-384, and SHA-512. The output length ranges from 224 bits to 512 bits. It is currently a more commonly used secure hash algorithm.

[0049] Exemplarily, the hash function adopted by the embodiment of the present invention selects the SHA-256 algorithm, and one of the existing SHA-224, SHA-256, SHA-384 and SHA-512 can be selected according to corresponding requirements.

[0050] In step 2, the Hash value of the request data is calculated using the following formula:

[0051] hash_output = H(hash_input)

[0052] In the formula, H represents the hash function, Hash_input represents key features such as number, name, amount, etc., and Hash_output represents the hash value of the request generated by the hash algorithm.

[0053] In step 3, a historical hash value database is established to store the hash values ​​of all processed requests in a database, and the time when the payment instruction is received is recorded to facilitate the subsequent calculation of the time interval between the last time the same instruction is received and the next time.

[0054] In step 4, the hash value of the new request is calculated and compared with the hash value in the historical record to check whether there is a duplicate hash value. If the hash value is repeated, the interval t between the two requests is recorded.

[0055] You need to select the time unit. The default is minutes. You can also select seconds, minutes, or hours as the unit time based on the different time sensitivities of different systems.

[0056] In step 5, select a decay function. For example, you can select an exponential decay function, or you can select power law decay or logistic decay with inconsistent decay rates according to the business scenario.

[0057] Regarding the time trend model, the decay function is used to simulate the changing trend of the probability of duplication over time. Different decay functions have different decay rates and characteristics, such as exponential decay function, power law decay, and logistic decay. Different business scenarios can choose different decay functions to simulate the change of the probability of duplication over time to improve the accuracy of request duplication verification. The exponential decay function is suitable for constant rate decay scenarios, the power law decay function is suitable for scenarios with slow initial speed and accelerated later, and the logistic decay function is suitable for the decay process of the S-shaped curve. By selecting a suitable decay function, the probability of duplication of requests can be predicted more accurately, improving the efficiency of the system's anti-duplicate verification.

[0058] The present invention uses an exponential decay function as an example to fit the process in which the probability of repetition decays uniformly with time in practice. The mathematical formula is as follows:

[0059]

[0060] Among them, α>0 is called the exponential decay constant. By solving the differential equation, we can get:

[0061] N(t)=N0e -αt (2)

[0062] Where N(t) is the value of N at time t, N0=N(0) is the initial value of N at the initial time, and is also the maximum value when t≥0.

[0063] However, formula (2) only indicates that the value can decay in an exponential form, but a more precise control of the decay process is required. The value decays from 100% probability, and after a set decay time, it drops to the lowest probability. The calculation formula is modified as follows:

[0064] N(t)=N0e -α(t+l) (3)

[0065] Among them, l represents the translation to the left, which is used to change the attenuation from the non-N0 position, and this non-N0 position is defined as N init , decays to N after m days finish , the equation is:

[0066] No -αl =N init (4)

[0067] No -α(m+l) =N finish (5)

[0068] Solving the above equation gives:

[0069]

[0070] Substituting into formula (3) we can get the final calculation formula:

[0071]

[0072] In the formula, N0 represents the initial value of N at time 0, N(t) is the value of N at time t, N0=N(0) is the initial value of N at time 0, N finish It means that after m days, it decays to N finish , N init Indicates that the decay starts from a non-N0 position, and the non-N0 position is defined as N init , m represents the number of days;

[0073] The time fitting degree N(t) is used to estimate the probability that the request is repeated. The longer the time interval, the lower the corresponding probability of repetition.

[0074] In step 6, set the warning level according to the probability. According to the calculated repetition probability, you can customize different warning levels. Default example: when the probability is higher than 90%, the system issues a red warning; when the probability is between 70%-90%, the system issues a yellow warning; when the probability is between 50%-70%, the system only issues a risk warning.

[0075] In step 7, the time interval t is input into the selected attenuation function model, the request repetition probability of the interval time is calculated, and the risk level is output.

[0076] Exemplarily, a Logistic decay model is selected to fit the process of Logistic decay of the probability of repeated payment over time, the time interval t is input into the Logistic decay model, the probability of request repetition at interval time t is calculated, and the risk level is output.

[0077] like Figure 2 As shown in the figure, the three curves all represent the process in which the probability of instruction repetition decreases as the time interval t between instructions increases. The green curve represents the Logistic decay model, the orange curve represents the power law decay model, and the blue curve represents the exponential decay model. The corresponding points on each curve represent the repetition probability corresponding to the three decay function models when the interval t is 1000.

[0078] The verification method of the present invention is a time-related data duplication verification method, which can improve the efficiency and accuracy of anti-duplicate verification. The present invention can meet the high requirements for real-time and security in modern information systems. By accurately evaluating and processing the risk of duplication, it can reduce false positives and missed positives, optimize the use of system resources, and improve the smoothness of operations; by introducing different warning levels, the flexibility of system management is enhanced, so that it can more effectively deal with complex situations, thereby improving the efficiency of the overall business.

[0079] Anti-duplicate verification technology plays an important role and significance in modern enterprise management. The present invention provides an effective anti-duplicate verification technology that can prevent repeated and erroneous operations, ensure the security and accuracy of data, and avoid operational confusion and waste of resources. At the same time, the present invention can also improve the efficiency of business processing, reduce the need for manual intervention and adjustment, and improve the accuracy and real-time performance of operations. In addition, the use of effective anti-duplicate verification technology can also enhance the credibility and reputation of the enterprise, reflect the improvement of the internal management level of the enterprise, and thus lay a solid foundation for the steady and long-term development of the enterprise.

[0080] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data duplication verification method, characterized in that: The repetition probability is evaluated by simulating the time acceleration decay function, so that the repetition probability decreases faster with the increase of time, thereby achieving effective prevention of repeated instructions or data.

2. The data duplication checking method according to claim 1, characterized in that: The specific steps include: Step 1: Extract key feature fields of request data; Step 2: Calculate the Hash value of the requested data; Step 3: Store the Hash value in the historical Hash value database to record the time when the payment instruction is received; Step 4: Calculate the hash value of the new request and compare it with the hash value in the historical record. If the hash value is repeated, record the time and calculate the interval t between the two identical instructions. Step 5: Select a decay function and use the time fit to estimate the probability of repetition; Step 6: Set the warning level corresponding to different repetition probabilities; Step 7: Calculate the probability of request repetition at interval t and output the risk level.

3. The data duplication checking method according to claim 2, characterized in that: In step 1, the key features include number, name, and amount.

4. The data duplication checking method according to claim 2, characterized in that: In step 1, a hash function is used to perform a hash operation on the request data to generate a unique hash value.

5. The data duplication checking method according to claim 2, characterized in that: In step 2, the calculation formula for calculating the Hash value of the request data is: hash_output = H(hash_input) In the formula, H represents the hash function, Hash_input represents the key features such as number, name, amount, etc., and Hash_output represents the hash value of the request generated by the hash algorithm.

6. The data duplication checking method according to claim 2, characterized in that: In step 4, you need to select the time unit including seconds, minutes, and hours.

7. The data duplication checking method according to claim 2, characterized in that: In step 5, the time fitting degree N(t) is calculated as follows: In the formula, N0 represents the initial value of N at time 0, N(t) is the value of N at time t, N0=N(0) is the initial value of N at time 0, N finish It means that after m days, it decays to N finish , N init Indicates that the decay starts from a non-N0 position, and the non-N0 position is defined as N init , m represents the number of days; The time fitting degree N(t) is used to estimate the probability of repetition. The longer the time interval, the lower the corresponding probability of repetition.

8. The data duplication checking method according to claim 2, characterized in that: In step 5, the decay function includes an exponential decay function, a power law decay function and a logistic decay function.

9. The data duplication checking method according to claim 2, characterized in that: In step 6, different warning levels are customized according to the calculated repetition probability.

10. The data duplication checking method according to claim 2, characterized in that: In step 7, the Logistic decay model is selected to fit the process of Logistic decay of the probability of repeated payment over time. The time interval t is input into the Logistic decay model to calculate the probability of request repetition at interval time t.

Images