Online transaction message deduplication method, device, medium, product and equipment

Abstract

The present disclosure relates to a method, apparatus, medium, product and equipment for deduplicating online transaction messages. A message deduplication configuration list is cached in a key-value pair database, and the first bit vector table in the first vector table filter is used to store the hash value corresponding to the specified primary key value of the received online transaction message. When a new online transaction message is received, the message deduplication configuration list is obtained from the key-value pair database to determine the corresponding message deduplication index primary key. Determine whether the first hash value corresponding to the primary key value has been stored. If it is determined that it has been stored, indicate not to process the new online transaction message, thereby achieving message deduplication. Since the key-value pair database can provide a message deduplication configuration list with lower latency and faster response speed, and the vector table filter can achieve fast lookup of the bit vector table, the timeliness of message deduplication can be effectively guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, apparatus, medium, product and device for deduplicating online transaction messages. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the present disclosure that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] With the development of information systems, message exchange between online transaction systems has become increasingly heterogeneous. While the introduction of technologies like blockchain and Kafka distributed messaging has greatly facilitated the construction of online transaction systems, it has also introduced the problem of message duplication. For example, suppose a blockchain has two nodes, A and B. When an online transaction message is sent to node A, the blockchain will synchronize this message to node B through a consensus mechanism. In clustered mode, both nodes A and B will send the message downstream, resulting in message duplication.

[0004] For online transaction systems, duplicate messages can lead to unnecessary misunderstandings. Especially for account transactions, duplicate messages can reduce transaction success rates or even lead to incorrect transactions, causing significant trouble.

[0005] Currently, message deduplication mainly includes the following two solutions:

[0006] The first option:

[0007] Using database unique constraint technology, the system stores each online transaction message received in the database, using one or more attributes of the message to create a primary key for the message. If a message is inserted into the database repeatedly, an exception will be generated.

[0008] If database unique constraint technology is used without adding duplicate verification logic to the system, database system-level error logs will be thrown. If the system does not handle these exceptions, the error will be directly returned to the user, which is not user-friendly. If duplicate verification logic is added to the system, the primary keys of different online transactions in the same system may be different. This not only significantly increases the amount of code and the developer's workload, but also affects the maintainability of the system due to the fragmented code. Furthermore, frequent interaction with the database cannot guarantee the timeliness of deduplication of online transaction messages.

[0009] Second option:

[0010] In the system, an interceptor is added to the uniform resource locator (URL) of all requests. Each time the requested URL is stored in the session control (Session), the next request is made to verify whether the URL data is the same. If it is the same, access is denied.

[0011] However, adding interceptors based on sessions has certain limitations. Servers can host several or even dozens of services simultaneously. High traffic demands high server performance, making it difficult to guarantee the timeliness of deduplication of online transaction messages. Furthermore, since server memory cannot be shared across multiple servers, this solution's scalability is affected.

[0012] Therefore, there is an urgent need to provide an online transaction message deduplication mechanism that can ensure timeliness when deduplicating messages. Summary of the Invention

[0013] The embodiments of the present disclosure provide a method, apparatus, medium, product, and device for deduplicating online transaction messages, which are used to solve the problem of poor timeliness in deduplicating online transaction messages.

[0014] In a first aspect, the present disclosure provides a method for deduplicating online transaction messages, the method comprising:

[0015] If an online transaction message is received, obtaining a message deduplication configuration list cached in a key-value pair database, wherein the message deduplication configuration list includes at least one pre-configured online transaction type number and at least one message deduplication index primary key corresponding to each online transaction type number;

[0016] According to the online transaction type corresponding to the online transaction message, searching the message deduplication configuration list for at least one message deduplication index primary key corresponding to the corresponding online transaction type number;

[0017] De-indexing the primary key of the at least one message found, and obtaining each primary key value corresponding to the online transaction message;

[0018] Determine a first hash value based on each primary key value obtained, search a first bit vector table from a first vector table filter, and determine whether a bit corresponding to the first hash value is a specified value;

[0019] If it is determined that the bit corresponding to the first hash value is a specified value, the online transaction message is determined to be a duplicate message, and an instruction is given to not process the online transaction message.

[0020] Optionally, the method further includes:

[0021] If it is determined that the bit corresponding to the first hash value is not the specified value, determine a second hash value based on each primary key value obtained, search a second bit vector table from a second vector table filter, and determine whether the bit corresponding to the second hash value is the specified value;

[0022] If it is determined that the bit corresponding to the second hash value is a specified value, the online transaction message is determined to be a duplicate message, and an instruction is given to not process the online transaction message.

[0023] Optionally, the method further includes:

[0024] If it is determined that the bit corresponding to the second hash value is not a specified value, then the online transaction message is determined not to be a duplicate message, and the online transaction message is instructed to be processed; and

[0025] The bit corresponding to the first hash value in the first bit vector table is updated to a specified value, and the bit corresponding to the second hash value in the second bit vector table is updated to a specified value.

[0026] Optionally, the first vector table filter and the second vector table filter are reconstructed asynchronously at regular intervals according to a set duration;

[0027] Searching the first bit vector table from the first vector table filter includes:

[0028] Searching the first bit vector table from the first vector table filter corresponding to the current moment;

[0029] Searching the second bit vector table from the second vector table filter includes:

[0030] The second bit vector table is searched from the second vector table filter corresponding to the current moment.

[0031] Optionally, determining a first hash value based on each obtained primary key value includes:

[0032] Encrypt and compress each obtained primary key value using an information digest algorithm to obtain a first character string;

[0033] Perform a hash operation on the first character string to obtain a first hash value.

[0034] Optionally, determining a second hash value based on each obtained primary key value includes:

[0035] Multiply the specified code of each character of each primary key value obtained by the specified prime number and sum them to obtain a second character string;

[0036] Perform a hash operation on the second character string to obtain a second hash value.

[0037] In a second aspect, the present disclosure further provides an online transaction message deduplication device, the device comprising:

[0038] an acquisition module, configured to, upon receiving an online transaction message, acquire a message deduplication configuration list cached in a key-value pair database, wherein the message deduplication configuration list includes at least one pre-configured online transaction type number and at least one message deduplication index primary key corresponding to each online transaction type number;

[0039] a primary key search module, configured to search, according to the online transaction type corresponding to the online transaction message, for at least one message deduplication index primary key corresponding to the corresponding online transaction type number from the message deduplication configuration list;

[0040] A primary key value obtaining module, configured to de-index the primary key of the at least one message found, and obtain each primary key value corresponding to the online transaction message;

[0041] a vector table lookup module, configured to determine a first hash value based on each obtained primary key value, search a first bit vector table from a first vector table filter, and determine whether a bit corresponding to the first hash value is a specified value;

[0042] The processing module is configured to determine that the online transaction message is a duplicate message and instruct not to process the online transaction message if it is determined that the bit corresponding to the first hash value is a specified value.

[0043] Optionally, the vector table search module is further configured to, if it is determined that the bit corresponding to the first hash value is not a specified value, determine a second hash value based on each obtained primary key value, search a second bit vector table from a second vector table filter, and determine whether the bit corresponding to the second hash value is a specified value;

[0044] The processing module is further configured to determine that the online transaction message is a duplicate message and instruct not to process the online transaction message if it is determined that the bit corresponding to the second hash value is a specified value.

[0045] Optionally, the processing module is further used to determine that the online transaction message is not a duplicate message if it is determined that the bit corresponding to the second hash value is not a specified value, and instruct to process the online transaction message; and update the bit corresponding to the first hash value in the first bit vector table to the specified value, and update the bit corresponding to the second hash value in the second bit vector table to the specified value.

[0046] Optionally, the first vector table filter and the second vector table filter are reconstructed asynchronously at regular intervals according to a set duration;

[0047] The vector table search module searches for the first bit vector table from the first vector table filter, including: searching for the first bit vector table from the first vector table filter corresponding to the current moment; and searches for the second bit vector table from the second vector table filter, including: searching for the second bit vector table from the second vector table filter corresponding to the current moment.

[0048] Optionally, the vector table lookup module determines the first hash value based on each obtained primary key value, including: encrypting and compressing each obtained primary key value through an information digest algorithm to obtain a first character string; and performing a hash operation on the first character string to obtain a first hash value.

[0049] Optionally, the vector table lookup module determines a second hash value based on each primary key value obtained, including: multiplying a specified code of each character of each primary key value obtained by a specified prime number and summing the results to obtain a second character string; performing a hash operation on the second character string to obtain a second hash value.

[0050] In a third aspect, the present disclosure further provides a computer program product, which includes an executable program, and the executable program is executed by a processor to implement the method described above.

[0051] In a fourth aspect, the present disclosure further provides a non-volatile computer storage medium, wherein the computer storage medium stores an executable program, and the executable program is executed by a processor to implement the method described above.

[0052] In a fifth aspect, the present disclosure further provides an online transaction message deduplication device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0053] The memory is used to store computer programs;

[0054] The processor is configured to implement the above-described method steps when executing the program stored in the memory.

[0055] According to the solution provided by the embodiments of the present disclosure, a configurable message deduplication configuration list can be cached in a key-value pair database, and the first bit vector table in the first vector table filter can be used to store the hash value corresponding to the specified primary key value of the received online transaction message. Thus, when an online transaction message is received, the message deduplication configuration list can be obtained from the key-value pair database, and the corresponding message deduplication index primary key can be determined based on the type of the online transaction message. Furthermore, based on the first hash value corresponding to the primary key value, the first bit vector table in the first vector table filter can be searched to determine whether the first hash value has already been stored. If it is determined that the first hash value has already been stored, the received online transaction message can be determined to be a duplicate message, and the online transaction message can be instructed not to be processed, thereby achieving deduplication of the online transaction message. Because the key-value pair database can provide the message deduplication configuration list with low latency and fast response speed, and the vector table filter can quickly search the bit vector table, the timeliness of deduplication of online transaction messages can be effectively guaranteed.

[0056] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0058] Figure 1 A flow chart of a method for deduplicating online transaction messages provided by an embodiment of the present disclosure;

[0059] Figure 2 A schematic diagram of the structure of an online transaction message deduplication device provided by an embodiment of the present disclosure;

[0060] Figure 3 A schematic diagram of the structure of an online transaction message deduplication device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the present disclosure will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only a portion of the embodiments of the present disclosure, rather than all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure without creative effort are intended to fall within the scope of protection of the present disclosure.

[0062] It should be noted that the term "multiple or several" in this document refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0063] The terms "first," "second," and the like in the specification and claims of the present disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present disclosure described herein can be practiced in orders other than those illustrated or described herein.

[0064] In addition, the terms "comprises" and "having" and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, system, product or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product or apparatus.

[0065] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0066] In order to ensure the timeliness of online transaction message deduplication, this application proposes that a key-value pair database and a vector table filter can be used to jointly implement message deduplication.

[0067] The key-value database mentioned in this application can be understood as a database that stores data as a set of key-value pairs, where the key is a unique identifier. The key-value database mentioned in this application can be, but is not limited to, a remote dictionary service (redis) database or a memecache database.

[0068] The vector table filter mentioned in this application can be understood as a filter that saves data by setting the corresponding bit in the bit vector table to a specified value (such as 1) based on the hash value of the data, and can filter the saved data based on the bit vector table. The vector table filter mentioned in this application can be, but is not limited to, a Bloom filter or a cuckoo filter.

[0069] Based on the above description, the embodiment of the present disclosure provides a method for deduplicating online transaction messages. The steps of the method can be as follows: Figure 1 As shown, including:

[0070] Step 101: Receive online transaction message.

[0071] In this step, an online transaction message may be received. If received, step 102 may be continued.

[0072] Step 102: Obtain a message de-reconfiguration list cached in the key-value pair database.

[0073] Considering the advantages of low latency and fast response times of key-value databases, such as Redis, in this embodiment, the message deduplication configuration list can be stored in the key-value database to ensure high system availability. In this step, the cached message deduplication configuration list can be retrieved from the key-value database. The message deduplication configuration list in the key-value database can be configured as needed.

[0074] Considering that a system may have multiple online transaction types, such as Alipay payment transactions, WeChat payment transactions, and bank card payment transactions, in this embodiment, at least one online transaction type number can be configured in the message deduplication configuration list, and at least one corresponding message deduplication index primary key can be configured for each online transaction type number.

[0075] It can be understood that, in this embodiment, a message deduplication configuration list can be used to support message deduplication for various types of online transactions, thereby achieving reusable, configurable, efficient and stable message deduplication.

[0076] Step 103: According to the online transaction type corresponding to the online transaction message, search the message deduplication configuration list for at least one message deduplication index primary key corresponding to the corresponding online transaction type number.

[0077] In this step, the received online transaction message may be parsed, and according to the corresponding online transaction type, the corresponding message deduplication index primary key may be searched from the obtained message deduplication configuration list.

[0078] Step 104: De-index the primary key of the at least one message found, and obtain each primary key value corresponding to the online transaction message.

[0079] In this step, the primary key value corresponding to the found primary key can be obtained by parsing the received online transaction message. In this way, the uniqueness and repeatability of online transactions can be determined based on the obtained primary key value, avoiding system confusion caused by duplicate transactions.

[0080] Step 105: Determine a first Hash value based on each obtained primary key value.

[0081] Considering that when using a key-value database to store large amounts of data, the demand for memory space will increase linearly, resulting in high costs, the data storage of the vector table filter saves space. There is no need to store the entire data. Instead, data storage can be achieved by simply determining the hash value of the data and then setting the corresponding bit in the bit vector table to a specified value (e.g., 1). Furthermore, the vector table filter has a faster data search speed. It only needs to determine the hash value of the data to be searched and then determine whether the corresponding bit in the bit vector table is set to a specified value (e.g., 1) to achieve data search. Therefore, in this embodiment, the bit vector table of the vector table filter can be used to store the hash value corresponding to the specified primary key value of the received online transaction message.

[0082] Since the bit vector table of the vector table filter is used to store the hash values ​​corresponding to the designated primary key values ​​of the received online transaction messages, in this step, the first hash value can be determined for each newly received online transaction message based on each obtained primary key value.

[0083] The first hash value may be determined in any manner as long as it is determined in the same manner as the hash value stored in the bit vector table of the vector table filter.

[0084] Step 106: Search the first bit vector table from the first vector table filter to determine whether the bit corresponding to the first hash value is a specified value.

[0085] In this step, the first bit vector table of the first vector table filter for storing the hash value corresponding to the specified primary key value of the received online transaction message can be searched to determine whether the bit corresponding to the first hash value determined in step 105 is a specified value, such as 1.

[0086] If it is determined to be the specified value, the received online transaction message can be determined to be a duplicate message, and step 107 is executed. After step 107, the present process ends. Otherwise, the received online transaction message can be determined to be not a duplicate message, and step 108 is executed. After step 108, the present process ends.

[0087] Step 107: Instruct not to process the online transaction message.

[0088] In this step, it can be indicated that the online transaction message received this time will not be processed, so as to achieve message deduplication.

[0089] Step 108: Instruct to process the online transaction message and update the bit corresponding to the first hash value in the first bit vector table to a specified value.

[0090] In this step, an instruction can be given to process the online transaction message received this time, and use the first hash value corresponding to the online transaction message received this time to update the first bit vector table, so that the updated first bit vector table can be used subsequently to deduplicate messages that are identical to the online transaction message received this time.

[0091] In one possible implementation, considering that the vector table filter may have a certain probability of misjudgment, and its misjudgment characteristic is that what exists must exist, and what does not exist may not not exist, that is, if it is determined that the bit corresponding to the first hash value is not the specified value, this result may be a misjudgment. Therefore, in this embodiment, after determining that the bit corresponding to the first hash value is not the specified value, step 108' can be continued after step 106 to further perform message deduplication using the second vector table filter:

[0092] Step 108 ′: Determine a second hash value based on each obtained primary key value.

[0093] The second hash value may be determined in any manner, as long as the manner of determining the hash value stored in the second bit vector table of the second vector table filter is the same.

[0094] In one possible implementation, in order to further improve the accuracy of message deduplication, the method for determining the hash value stored in the second bit vector table of the second vector table filter may be different from the method for determining the hash value stored in the second bit vector table of the first vector table filter. In this case, the method for determining the second hash value is also different from the method for determining the first hash value.

[0095] For example, in one possible implementation, the first hash value may be obtained in, but is not limited to, the following manner:

[0096] Each obtained primary key value is encrypted and compressed using an information digest algorithm (eg, MD5 algorithm) to obtain a first character string; and a hash operation is performed on the first character string to obtain a first hash value.

[0097] The second hash value can be obtained by, but is not limited to, the following methods:

[0098] A second character string is obtained by multiplying a specified code of each character of each primary key value obtained by a specified prime number and summing the results to obtain a second character string; and a hash operation is performed on the second character string to obtain a second hash value.

[0099] Step 109 ′: search the second bit vector table from the second vector table filter to determine whether the bit corresponding to the second hash value is a specified value.

[0100] Similar to step 106, in this step, the second bit vector table of the second vector table filter for storing the hash value corresponding to the specified primary key value of the received online transaction message can be searched to determine whether the bit corresponding to the second hash value determined in step 108' is a specified value, such as 1.

[0101] If the value is determined to be the specified value, the received online transaction message is determined to be a duplicate message, and step 110' is executed. After step 110', the present process ends. Otherwise, the received online transaction message is determined to be not a duplicate message, and step 111' is executed. After step 111', the present process ends.

[0102] Step 110': instruct not to process the online transaction message.

[0103] In this step, it is possible to indicate that the currently received online transaction message will not be processed, thereby achieving message deduplication. In other words, by setting two vector table filters, if the first vector table filter does not determine that a message is duplicated, the second vector table filter can be used to determine that a message is duplicated.

[0104] Step 111 ′: instruct to process the online transaction message, and update the bit corresponding to the first hash value in the first bit vector table to a specified value, and update the bit corresponding to the second hash value in the second bit vector table to a specified value.

[0105] In this embodiment, in order to ensure the timeliness of message deduplication, two vector table filters are set as an example for explanation. Of course, more vector table filters can also be set for message deduplication, which will not be further explained in this embodiment.

[0106] If the two vector table filters do not determine that there is message duplication, in this step, an instruction can be given to process the online transaction message received this time, and the first bit vector table can be updated using the first hash value corresponding to the online transaction message received this time, and the second bit vector table can be updated using the second hash value, so that the updated first bit vector table and second bit vector table can be further used to deduplicate messages that are identical to the online transaction message received this time.

[0107] Furthermore, considering that the probability of misjudgment of a vector table filter is related to the amount of data it stores, the probability of misjudgment increases when the amount of data stored approaches the estimated maximum capacity. Therefore, it is also possible to perform timed asynchronous reconstruction on vector table filters, such as Bloom filters, and replace the original vector table filter with the latest one to avoid excessive data in the vector table filter.

[0108] In one possible implementation, both the first vector table filter and the second vector table filter can be asynchronously rebuilt according to a set duration. In this case, searching the first bit vector table from the first vector table filter can be understood as including searching the first bit vector table from the first vector table filter corresponding to the current moment. Searching the first bit vector table from the first vector table filter can be understood as including searching the second bit vector table from the second vector table filter corresponding to the current moment. This not only improves the accuracy of message deduplication, but also enables message deduplication within a specified time range.

[0109] Furthermore, considering that the probability of misjudgment can be reduced when the capacity of the vector table filter is a specified multiple of the amount of stored data, such as 5 to 10 times, in this embodiment, the capacity of the vector table filter can also be determined based on the set duration of asynchronous reconstruction of the vector table filter. That is, the capacity of the vector table filter can be determined based on the online transaction volume within the scheduled duration (e.g., one day or one week) of asynchronous reconstruction of the vector table filter. This ensures that the capacity of the vector table filter can meet the specified multiple of the amount of stored data within the scheduled duration, further reducing the probability of misjudgment of the vector table filter and improving the accuracy of information deduplication.

[0110] It should be pointed out that if two vector table filters are set for message deduplication, by controlling the misjudgment probability of the two vector table filters, for example, if the misjudgment probability of the first vector table filter and the second vector table filter can be controlled at 0.01, the accuracy of the entire message deduplication process can be greatly improved.

[0111] The solution provided by the present invention can provide reusable, configurable, efficient, and stable online transaction message deduplication when multiple types of online transactions exist within the same system, without requiring modifications to the application logic layer. The high performance of the key-value database ensures high system availability, and the key-value database can also be implemented using a cluster to further enhance system reliability. Configuration enhances the flexibility of the system's uniqueness determination for multiple types of online transaction messages, effectively integrating support for multiple message modes and uniqueness determination for multiple online transaction messages into the system. This system also offers strong scalability and maintainability. As online transaction messages from more sources are subsequently added, message deduplication and message uniqueness can be achieved simply through configuration.

[0112] By determining the uniqueness and repetitiveness of online transaction messages, business disruptions caused by duplicate transactions can be avoided. Given the high real-time nature of online transactions, the present invention provides a message deduplication solution with a fast response time. Furthermore, as online transaction volumes increase, only a minimal increase in hardware costs is required to meet the increased demand. By controlling the error rate of the message deduplication process, the stability of the online transaction system can also be effectively improved.

[0113] Corresponding to the provided method, the following device is further provided.

[0114] The embodiment of the present disclosure provides an online transaction message deduplication device, the structure of which can be as follows: Figure 2 As shown, including:

[0115] The acquisition module 11 is configured to, upon receiving an online transaction message, acquire a message deduplication configuration list cached in a key-value pair database, wherein the message deduplication configuration list includes at least one pre-configured online transaction type number and at least one message deduplication index primary key corresponding to each online transaction type number;

[0116] The primary key search module 12 is configured to search, according to the online transaction type corresponding to the online transaction message, for at least one message deduplication index primary key corresponding to the corresponding online transaction type number from the message deduplication configuration list;

[0117] The primary key value obtaining module 13 is used to de-index the primary key according to the at least one message found, and obtain each primary key value corresponding to the online transaction message;

[0118] The vector table lookup module 14 is configured to determine a first hash value based on each primary key value obtained, and search a first bit vector table from a first vector table filter to determine whether a bit corresponding to the first hash value is a specified value;

[0119] The processing module 15 is configured to determine that the online transaction message is a duplicate message and instruct not to process the online transaction message if it is determined that the bit corresponding to the first hash value is a specified value.

[0120] Optionally, the vector table lookup module 14 is further configured to, if it is determined that the bit corresponding to the first hash value is not a specified value, determine a second hash value based on each obtained primary key value, and search a second bit vector table from a second vector table filter to determine whether the bit corresponding to the second hash value is a specified value;

[0121] The processing module 15 is further configured to determine that the online transaction message is a duplicate message if it is determined that the bit corresponding to the second hash value is a specified value, and instruct not to process the online transaction message.

[0122] Optionally, the processing module 15 is further used to determine that the online transaction message is not a duplicate message if it is determined that the bit corresponding to the second hash value is not a specified value, and instruct to process the online transaction message; and update the bit corresponding to the first hash value in the first bit vector table to the specified value, and update the bit corresponding to the second hash value in the second bit vector table to the specified value.

[0123] Optionally, the first vector table filter and the second vector table filter are reconstructed asynchronously at regular intervals according to a set duration;

[0124] The vector table search module 14 searches for the first bit vector table from the first vector table filter, including: searching for the first bit vector table from the first vector table filter corresponding to the current moment; and searches for the second bit vector table from the second vector table filter, including: searching for the second bit vector table from the second vector table filter corresponding to the current moment.

[0125] Optionally, the vector table lookup module 14 determines the first hash value based on each obtained primary key value, including: encrypting and compressing each obtained primary key value through an information digest algorithm to obtain a first character string; and performing a hash operation on the first character string to obtain a first hash value.

[0126] Optionally, the vector table lookup module 14 determines a second hash value based on each primary key value obtained, including: multiplying a specified code of each character of each primary key value obtained by a specified prime number and summing the results to obtain a second string; performing a hash operation on the second string to obtain a second hash value.

[0127] The functions of each functional unit of each device provided in the above embodiments of the present disclosure can be implemented through the steps of the above corresponding methods. Therefore, the possible working processes and beneficial effects of each functional unit in each device provided in the embodiments of the present disclosure are not repeated here.

[0128] Based on the same inventive concept, the embodiments of the present disclosure provide the following devices and media.

[0129] The embodiment of the present disclosure provides an online transaction message deduplication device, the structure of which can be as follows: Figure 3 As shown, it includes a processor 21, a communication interface 22, a memory 23 and a communication bus 24, wherein the processor 21, the communication interface 22, and the memory 23 communicate with each other through the communication bus 24;

[0130] The memory 23 is used to store computer programs;

[0131] The processor 21 is configured to implement the steps described in the above method embodiment of the present disclosure when executing the program stored in the memory.

[0132] Optionally, the processor 21 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), may be one or more integrated circuits for controlling program execution, may be a hardware circuit developed using a field programmable gate array (FPGA), or may be a baseband processor.

[0133] Optionally, the processor 21 may include at least one processing core.

[0134] Optionally, the memory 23 may include a read-only memory (ROM), a random access memory (RAM), and a disk storage. The memory 23 is used to store data required by at least one processor 21 during operation. The number of memories 23 may be one or more.

[0135] An embodiment of the present disclosure further provides a non-volatile computer storage medium, wherein the computer storage medium stores an executable program. When the executable program is executed by a processor, the method provided in the above method embodiment of the present disclosure is implemented.

[0136] In a possible implementation process, computer storage media may include: Universal Serial Bus Flash Drive (USB), mobile hard disk, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0137] An embodiment of the present disclosure further provides a computer program product, which includes an executable program. The executable program is executed by a processor to implement the method provided by the above method embodiment of the present disclosure.

[0138] In the embodiments of the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical or other forms.

[0139] Each functional unit in the embodiment of the present disclosure may be integrated into one processing unit, or each unit may be an independent physical module.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device, such as a personal computer, a server, or a network device, or a processor to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: a Universal Serial Bus Flash Drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0141] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0145] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present disclosure.

[0146] Obviously, those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A method for deduplicating online transaction messages, characterized in that: The method comprises: If an online transaction message is received, obtaining a message deduplication configuration list cached in a key-value pair database, wherein the message deduplication configuration list includes at least one pre-configured online transaction type number and at least one message deduplication index primary key corresponding to each online transaction type number; According to the online transaction type corresponding to the online transaction message, searching the message deduplication configuration list for at least one message deduplication index primary key corresponding to the corresponding online transaction type number; De-indexing the primary key of the at least one message found, and obtaining each primary key value corresponding to the online transaction message; Determine a first hash value based on each primary key value obtained, search a first bit vector table from a first vector table filter, and determine whether a bit corresponding to the first hash value is a specified value; If it is determined that the bit corresponding to the first hash value is a specified value, determining that the online transaction message is a duplicate message and instructing not to process the online transaction message; If it is determined that the bit corresponding to the first hash value is not the specified value, then it is determined that the online transaction message is not a duplicate message, and an instruction is given to process the online transaction message; Taking into account the possibility of misjudgment, when it is determined that the bit corresponding to the first hash value is not the specified value, the method further includes: multiplying the specified code of each character of each primary key value obtained by a specified prime number and summing the results to obtain a second character string, performing a hash operation on the second character string to obtain a second hash value, and determining whether the online transaction message is a duplicate message based on the second hash value.

2. The method according to claim 1, wherein The determining whether the online transaction message is a duplicate message based on the second hash value includes: Searching a second bit vector table from a second vector table filter to determine whether a bit corresponding to the second hash value is a specified value; If it is determined that the bit corresponding to the second hash value is a specified value, determining that the online transaction message is a duplicate message and instructing not to process the online transaction message; If it is determined that the bit corresponding to the second hash value is not the specified value, it is determined that the online transaction message is not a duplicate message, and an instruction is given to process the online transaction message.

3. The method according to claim 2, wherein When it is determined that the online transaction message is not a duplicate message, the method further includes: If it is determined based on the bit corresponding to the first hash value that the online transaction message is not a duplicate message, updating the bit corresponding to the first hash value in the first bit vector table to a specified value; If it is determined based on the bit corresponding to the second hash value that the online transaction message is not a duplicate message, the bit corresponding to the first hash value in the first bit vector table is updated to a specified value, and the bit corresponding to the second hash value in the second bit vector table is updated to a specified value.

4. The method according to claim 2, wherein The first vector table filter and the second vector table filter are reconstructed asynchronously at regular intervals according to a set duration; Searching the first bit vector table from the first vector table filter includes: Searching the first bit vector table from the first vector table filter corresponding to the current moment; Searching the second bit vector table from the second vector table filter includes: The second bit vector table is searched from the second vector table filter corresponding to the current moment.

5. The method according to any one of claims 1 to 4, characterized in that: Determine a first hash value based on each obtained primary key value, including: Encrypt and compress each obtained primary key value using an information digest algorithm to obtain a first character string; Perform a hash operation on the first character string to obtain a first hash value.

6. An online transaction message deduplication device, characterized in that: The device comprises: an acquisition module, configured to, upon receiving an online transaction message, acquire a message deduplication configuration list cached in a key-value pair database, wherein the message deduplication configuration list includes at least one pre-configured online transaction type number and at least one message deduplication index primary key corresponding to each online transaction type number; a primary key search module, configured to search, according to the online transaction type corresponding to the online transaction message, for at least one message deduplication index primary key corresponding to the corresponding online transaction type number from the message deduplication configuration list; A primary key value obtaining module, configured to de-index the primary key of the at least one message found, and obtain each primary key value corresponding to the online transaction message; a vector table lookup module, configured to determine a first hash value based on each obtained primary key value, search a first bit vector table from a first vector table filter, and determine whether a bit corresponding to the first hash value is a specified value; a processing module, configured to, if it is determined that the bit corresponding to the first hash value is a specified value, determine that the online transaction message is a duplicate message and instruct not to process the online transaction message; If it is determined that the bit corresponding to the first hash value is not the specified value, then it is determined that the online transaction message is not a duplicate message, and an instruction is given to process the online transaction message; Taking into account the possibility of misjudgment, if it is determined that the bit corresponding to the first hash value is not the specified value, then it is determined that the online transaction message is not a duplicate message, and the vector table lookup module is further used to: multiply the specified code of each character of each primary key value obtained by a specified prime number and sum them to obtain a second string, and perform a hash operation on the second string to obtain a second hash value; the processing module is also used to determine whether the online transaction message is a duplicate message based on the second hash value.

7. The device according to claim 6, characterized in that The vector table search module is further configured to: search a second bit vector table from a second vector table filter to determine whether a bit corresponding to the second hash value is a specified value; The processing module is further configured to, if it is determined that the bit corresponding to the second hash value is a specified value, determine that the online transaction message is a duplicate message and instruct not to process the online transaction message; If it is determined that the bit corresponding to the second hash value is not the specified value, it is determined that the online transaction message is not a duplicate message, and an instruction is given to process the online transaction message.

8. The device according to claim 7, wherein The processing module is further configured to, when determining that the online transaction message is not a duplicate message, update the bit corresponding to the first hash value in the first bit vector table to a specified value if the online transaction message is determined not to be a duplicate message based on the bit corresponding to the first hash value; and update the bit corresponding to the first hash value in the first bit vector table to a specified value, and update the bit corresponding to the second hash value in the second bit vector table to a specified value if the online transaction message is determined not to be a duplicate message based on the bit corresponding to the second hash value.

9. The device according to claim 7, wherein The first vector table filter and the second vector table filter are reconstructed asynchronously at regular intervals according to a set duration; The vector table search module searches for the first bit vector table from the first vector table filter, including: searching for the first bit vector table from the first vector table filter corresponding to the current moment; and searches for the second bit vector table from the second vector table filter, including: searching for the second bit vector table from the second vector table filter corresponding to the current moment.

10. The device according to any one of claims 6 to 9, characterized in that: The vector table lookup module determines the first hash value according to each obtained primary key value, including: encrypting and compressing each obtained primary key value through an information digest algorithm to obtain a first character string; and performing a hash operation on the first character string to obtain the first hash value.

11. A non-volatile computer storage medium, characterized in that: The computer storage medium stores an executable program, and the executable program is executed by a processor to implement the method according to any one of claims 1 to 5.

12. A computer program product, characterized in that The computer program product includes an executable program, which is executed by a processor to implement the method according to any one of claims 1 to 5.

13. An online transaction message deduplication device, characterized in that: The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is configured to implement the method steps described in any one of claims 1 to 5 when executing the program stored in the memory.

Images