Data duplication checking method, device, equipment and computer storage medium

Abstract

The present invention relates to the technical field of financial technology (Fintech), and discloses a data duplication checking method, which includes: when receiving target data, detecting whether the target data is subjected to duplication checking processing; if so, detecting whether the data transaction corresponding to the target data satisfies a preset ignore condition; if not, calculating the current summary corresponding to the target data, and determining the target time range corresponding to the current summary, inserting the target time range and the current summary into the summary data table corresponding to the target data to obtain the target summary data table, and detecting whether the target summary data table has a primary key uniqueness conflict; if the target summary data table has a primary key uniqueness conflict, determining that the target data is duplicate data. The present invention also discloses a data duplication checking device, equipment, and a computer storage medium. The present invention improves the availability of data duplication checking processing.

Description

Technical Field

[0001] The present invention relates to the field of financial technology (Fintech), and in particular to a data duplication checking method, device, equipment and computer storage medium. Background Art

[0002] With the development of computer technology, more and more technologies (big data, distributed, blockchain, artificial intelligence, etc.) are being applied in the financial field. The traditional financial industry is gradually transforming to financial technology (Fintech), but due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology. At present, when conducting data transactions, duplicate data often occurs due to network factors, technical factors, customer factors, or compensation measures used to ensure distributed consistency, which seriously affects the normal progress of data transactions. However, the existing data duplication detection methods for duplicate data have various limitations (such as requiring a large amount of memory, or being invasive to business codes, etc.), and low availability. Therefore, how to improve the availability of data duplication detection processing has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the present invention is to propose a data duplication checking method, device, equipment and computer storage medium, aiming to solve the technical problem of how to improve the usability of data duplication checking processing.

[0004] To achieve the above object, the present invention provides a data duplication checking method, which comprises the following steps:

[0005] When receiving the target data, detecting whether to perform duplicate checking on the target data;

[0006] If so, detecting whether the data transaction corresponding to the target data satisfies a preset ignoring condition;

[0007] If not, the current summary corresponding to the target data is calculated, and the target time range corresponding to the current summary is determined, the target time range and the current summary are inserted into the summary data table corresponding to the target data to obtain the target summary data table, and the target summary data table is detected whether a primary key uniqueness conflict occurs;

[0008] If a primary key uniqueness conflict occurs in the target summary data table, the target data is determined to be duplicate data.

[0009] Optionally, the step of determining a target time range corresponding to the current summary includes:

[0010] Obtaining a current time range corresponding to the current summary, and determining a previous time range of the current time range;

[0011] The target time range is determined according to the current time range and the previous time range.

[0012] Optionally, the step of determining the target time range according to the current time range and the previous time range includes:

[0013] An initial time node of the previous time range is determined, an end time node of the current time range is determined, and a time range from the initial time node to the end time node is used as a target time range.

[0014] Optionally, the step of calculating the current summary corresponding to the target data includes:

[0015] Detecting whether the configuration item corresponding to the target data meets a preset configuration condition;

[0016] If the configuration item meets the preset configuration condition, the business serial number corresponding to the data transaction and the message corresponding to the target data are obtained, and the business serial number and the message are converted to obtain the current summary.

[0017] Optionally, after the step of detecting whether the configuration item corresponding to the target data satisfies a preset configuration condition, the following steps are included:

[0018] If the configuration item does not satisfy the preset configuration condition, a message corresponding to the target data is obtained, and the message is converted to obtain a current summary.

[0019] Optionally, the step of detecting whether to perform duplicate checking on the target data includes:

[0020] Acquire a plurality of preset target parameters, and detect whether the data transaction corresponding to the target data matches each of the target parameters;

[0021] If the data transaction matches each of the target parameters, it is determined to perform data duplication checking on the data transaction.

[0022] Optionally, the step of detecting whether the data transaction corresponding to the target data satisfies a preset ignoring condition includes:

[0023] Obtaining a preset configuration anti-duplicate ignore table, and detecting whether the transaction type of the data transaction matches the ignore configuration item in the configuration anti-duplicate ignore table;

[0024] If the transaction type and the ignore configuration item do not match, it is determined that the data transaction does not meet the preset ignore condition.

[0025] In addition, to achieve the above-mentioned purpose, the present invention also provides a data duplication checking device, the data duplication checking device comprising:

[0026] A first detection module is used to detect whether to perform duplicate checking on the target data when receiving the target data;

[0027] A second detection module, configured to detect whether the data transaction corresponding to the target data satisfies a preset ignoring condition;

[0028] a calculation module, configured to calculate the current summary corresponding to the target data if the conditions are not met, determine the target time range corresponding to the current summary, insert the target time range and the current summary into the summary data table corresponding to the target data to obtain a target summary data table, and detect whether a primary key uniqueness conflict occurs in the target summary data table;

[0029] The determination module is used to determine that the target data is duplicate data if a primary key uniqueness conflict occurs in the target summary data table.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also provides a data duplication checking device, which includes: a memory, a processor, and a data duplication checking program stored on the memory and executable on the processor, and the data duplication checking program, when executed by the processor, implements the steps of the data duplication checking method described above.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a data duplication checking program is stored, and when the data duplication checking program is executed by a processor, the steps of the data duplication checking method as described above are implemented.

[0032] The present invention detects whether duplicate checking is performed on the target data when the target data is received; if so, detects whether the data transaction corresponding to the target data satisfies a preset ignore condition; if not, calculates the current summary corresponding to the target data, determines the target time range corresponding to the current summary, inserts the target time range and the current summary into a summary data table corresponding to the target data to obtain a target summary data table, and detects whether a primary key uniqueness conflict occurs in the target summary data table; if a primary key uniqueness conflict occurs in the target summary data table, determines that the target data is duplicate data. By first calculating the current summary and determining the target time range corresponding to the current summary when it is detected that the target data is being checked for duplicates and the target data does not meet the preset ignore condition, and inserting the target time range and the current summary into the summary data table to obtain the target summary data table, when a uniqueness conflict occurs in the target summary data table, the target data is determined to be duplicate data, thereby avoiding the occurrence of business code intrusion. Data duplicate checking can be achieved by only detecting the current summary, thereby improving the efficiency of data duplicate checking. Moreover, only the current summary within the target time range needs to be detected, instead of performing a one-time overall detection on the target data, saving the memory resources required for data duplicate checking, improving the availability of data duplicate checking, and performing primary key uniqueness conflict detection on the target summary data table with the target time range and the current summary, thereby improving the accuracy of data duplicate checking. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the structure of a data duplication checking device in a hardware operating environment involved in an embodiment of the present invention;

[0034] Figure 2 This is a flow chart of a first embodiment of a data duplication checking method of the present invention;

[0035] Figure 3 A schematic diagram of a device module of a data duplication checking device according to the present invention;

[0036] Figure 4 It is a flowchart of the data duplication checking method of the present invention.

[0037] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0038] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0039] like Figure 1 As shown, Figure 1 It is a schematic diagram of the structure of a data duplication checking device in a hardware operating environment involved in an embodiment of the present invention.

[0040] The data duplication checking device in the embodiment of the present invention may be a PC or a server device, on which a Java virtual machine runs.

[0041] like Figure 1 As shown, the data duplication checking device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0042] Those skilled in the art will understand that Figure 1 The data duplication checking device structure shown in the figure does not constitute a limitation of the device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0043] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and a data duplication checking program.

[0044] exist Figure 1 In the data duplication checking device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the data duplication checking program stored in the memory 1005 and execute the operations in the following data duplication checking method.

[0045] Based on the above hardware structure, an embodiment of the data duplication checking method of the present invention is proposed.

[0046] Reference Figure 2 , Figure 2 This is a flow chart of a first embodiment of a data duplication checking method of the present invention, the method comprising:

[0047] Step S10, when receiving the target data, detecting whether to perform duplicate checking on the target data;

[0048] In this embodiment, the distributed node is the computing node that constitutes the entire distributed system. The application instance is an application service deployed on a single distributed node. The primary key constraint is the unique characteristic of the primary key of the database table, and the primary key field in the database table is unique. The primary key field can be a single-field primary key or a multi-field combination field. The timing problem is a problem caused by the difference in the natural time of each application instance. When the time difference of the same message arriving at different application instances is not large, the natural time obtained by the message with a later timing may be earlier than the natural time obtained by the message with an earlier timing. Redis is an open source cache middleware. MD5 (MD5 Message-Digest Algorithm) is a cryptographic hash function that can generate a 128-bit (16-byte) hash value, which can ensure that the MD5 code generated by the same characters is the same. In this embodiment, the MD5 algorithm is used as an example to calculate the current summary, but it is not limited to the MD5 algorithm.

[0049] In the present embodiment, after the system is started, before the data is checked for duplicate, it is necessary to set various parameters first, such as setting the global anti-duplicate strategy configuration table, the transaction MD5 flow table, the ignore check configuration table and the duplicate transaction special processing strategy table, etc. And these parameters are added to the jvm level cache to prompt the subsequent access performance. Among them, JVM (Java Virtual Machine, Java Virtual Machine), JVM uses Mybatis or Hibernate and other Java programming language-based data persistence frameworks to operate the database. For example, in the financial field, it is often necessary to use JVM to access financial database tables, such as account information tables, account opening information tables, financial transaction information tables, etc., and cache technology will be used in this process to meet the concurrency and access speed requirements of financial services.

[0050] In this embodiment, when the target data is received, it is necessary to first check the transaction type of the data transaction corresponding to the target data, and determine whether the transaction type supports entering the anti-duplicate logic (i.e., data duplication check processing). Then, different operations are performed according to different judgment results. For example, if it is set that the financial transaction type can enter the anti-duplicate logic (i.e., duplication check processing), the query transaction type and the maintenance transaction type cannot enter the anti-duplicate logic, when the transaction type of the data transaction is a financial transaction type, it can be determined that the target data is to be checked for duplication.

[0051] Step S20: If yes, then detecting whether the data transaction corresponding to the target data satisfies a preset ignore condition;

[0052] When it is found through judgment that the target data is subjected to data duplicate checking, it is also necessary to detect whether the data transaction corresponding to the target data meets the preset ignore condition. If the data transaction meets the preset ignore condition, it can be determined that the target data does not need to be subjected to data duplicate checking. However, if the data transaction does not meet the preset ignore condition, it is necessary to perform data duplicate checking on the target data. That is, in the present embodiment, anti-duplicate configurability can be realized, and according to actual transaction requirements, under normal anti-duplicate management and control, the data transaction configuration ignore item that does not need anti-duplicate can be configured through the preset configuration anti-duplicate ignore table, so as to skip the anti-duplicate control, that is, data duplicate checking does not need to be performed. However, if the data transaction cannot be subjected to data duplicate checking, the target data is not subjected to data duplicate checking. Among them, the ignore condition can be that the transaction type of the data transaction matches the configuration anti-duplicate ignore table, and the configuration anti-duplicate ignore table stores multiple arbitrary transaction types set by the user in advance.

[0053] Step S30: if not satisfied, then calculate the current summary corresponding to the target data, determine the target time range corresponding to the current summary, insert the target time range and the current summary into the summary data table corresponding to the target data to obtain the target summary data table, and detect whether the target summary data table has a primary key uniqueness conflict;

[0054] When it is found through judgment that the data transaction does not meet the preset ignoring condition, it is determined to perform data duplication check on the target data. Then the configuration item corresponding to this data transaction can be checked first, that is, the data transaction is detected according to the anti-duplicate configuration item in the global anti-duplicate policy configuration table. If the business serial number is ignored, the business serial number can be directly converted to generate the current summary. Then the data duplication check time range corresponding to the current summary is determined according to the parameter information corresponding to the data transaction, that is, the target time range. And the current summary generated within the target time range is inserted into the summary data table to obtain the target summary data table, and since the current summary is preferably calculated by the MD5 algorithm in this embodiment, the summary data table can also be a transaction MD5 flow table. That is, the current summary and the target time range are inserted into the summary data table in the database as the only constraint to obtain the target summary data table. It should be noted that in this embodiment, the target time range includes at least two records, namely the current time range and the message summary information (i.e., the current summary), and the previous time range and the message summary information. And both records are inserted into the summary data table. Then check whether the target summary data table has a primary key uniqueness conflict, that is, use the primary key constraint to detect whether there are duplicate MD5 codes within the target time period (that is, if there are two identical MD5 codes, then determine that the two identical MD5 codes are duplicate MD5 codes), that is, at least two records within the target time range are detected. If one of the records reports a duplicate KEY exception, it is determined that a primary key uniqueness conflict has occurred in the target data table.

[0055] That is, in this embodiment, a time range factor is added on the basis of the MD5 code to avoid the phenomenon that the system time obtained by the message with a later time sequence is earlier than the message with an earlier time sequence. That is, when the application instance receives a message, it is necessary to calculate the message summary information, i.e., the MD5 code, and it is also necessary to determine which time range the message belongs to and the previous time range obtained according to the anti-re-configuration check time, and use these two time ranges as the target time range. For example, if the data transaction is implemented based on the time range parameter set to 10 minutes, the target time range is 20 minutes. And in this embodiment, the MD5 code and the target time range will be inserted into the database as the only constraint, and at least two records (MD5 code, time range) and (MD5 code, previous time range) will be generated each time, and these two records will be checked. If one of the records reports a duplicate key (key) exception, it is determined that the target data is repeated. That is, in this embodiment, when the target data is processed for anti-re-duplication, at least two MD5 codes will be generated. It also processes message duplication based on message digest information (MD5), and can support multiple message formats, such as JOSN, XML, and other message formats, and can support serialization and other scenarios. It does not need to parse the message, and directly obtains the specific field meaning for message duplication processing. It can also support cutting into business logic based on AOP (aspect-oriented programming), without modifying the original business code, and can be applied through AOP.

[0056] For example, when the current transaction date of the data transaction corresponding to the target data is set as date1, the current transaction natural time is A1, and the anti-duplicate check time range is time_scope, the data transaction corresponding to the current target data occurs at 0:00 on the same day, then an MD5 record with a transaction date of date1 and a transaction time of 0:00 (A1) is generated, as well as an MD5 record with a transaction date of the previous day, i.e. date2, and a transaction time of time_scope minutes before 24:00 (A2). If the transaction occurs at a time other than 0:00 on the same day, then an MD5 record with a transaction date of date1 and a transaction time of natural time A1 is generated, as well as an MD5 record with a transaction date of data2 that is the same as data1 and 10 minutes before the transaction time (A2). The MD5 record of A1 and the MD5 record of A2 are inserted into the transaction MD5 flow table (i.e., the summary data table), and the primary key constraint is used for anti-duplicate processing to determine whether there are repeated MD5 codes.

[0057] Step S40: If a primary key uniqueness conflict occurs in the target summary data table, it is determined that the target data is duplicate data.

[0058] When the MD5 code within the target time range is inserted into the transaction MD5 flow table, and it is found that there is no abnormality in the insertion, it can be determined that there is no primary key uniqueness conflict in the target summary data table, and the target data can be processed normally. However, if a primary key uniqueness conflict occurs during the insertion, that is, a primary key uniqueness conflict occurs in the target summary data table, the data transaction corresponding to the target data is determined to be a duplicate transaction, and it is necessary to query the duplicate transaction special processing strategy table and configuration items for corresponding strategy processing, such as allowing continued execution, alarm processing, and error processing.

[0059] In addition, in this embodiment, the anti-duplicate processing is based on data transactions (such as financial transactions). In order to prevent duplicate accounting, anti-duplicate control is also performed for the same transaction message. In actual use, only the unique primary key transaction flow can be used for anti-duplicate. In addition, in this embodiment, clustering or distributed deployment can be performed, and cluster nodes or distributed nodes access the same anti-duplicate processing module. Different nodes in the cluster or different distributed nodes receive the same message, and anti-duplicate processing can be performed.

[0060] In addition, to assist in understanding the principle of data duplication checking in this embodiment, an example is given below.

[0061] For example, Figure 4 As shown, when the anti-duplicate logic starts to run, the data transaction corresponding to the acquired target data is first detected, that is, the transaction type is checked. If it is a financial transaction, the financial transaction is checked, that is, whether the anti-duplicate configuration item is required. If not, the normal transaction logic is maintained. If so, it is also necessary to check whether the ignore check configuration table verification meets the ignore condition to determine whether anti-duplicate processing is required. If the ignore condition is met, no anti-duplicate processing is required. If the ignore condition is not met, the message MD5 (i.e. MD5 code) of the financial transaction is calculated, and the range configuration item is obtained according to MD5, such as the target time range. The transaction MD5 flow table record is generated according to the MD5 code within the target time range and inserted into the database to detect whether a duplicate KEY exception occurs. If not, it is determined that the financial transaction has not been repeated and there is no duplicate data. If so, anti-duplicate processing is performed according to the configuration strategy. Until the anti-duplicate logic ends.

[0062] In this embodiment, when the target data is received, it is detected whether the target data is checked for duplicates; if so, it is detected whether the data transaction corresponding to the target data meets the preset ignore condition; if not, the current summary corresponding to the target data is calculated, and the target time range corresponding to the current summary is determined, the target time range and the current summary are inserted into the summary data table corresponding to the target data to obtain the target summary data table, and it is detected whether the target summary data table has a primary key uniqueness conflict; if the target summary data table has a primary key uniqueness conflict, it is determined that the target data is duplicate data. By first calculating the current summary and determining the target time range corresponding to the current summary when it is detected that the target data is being checked for duplicates and the target data does not meet the preset ignore condition, and inserting the target time range and the current summary into the summary data table to obtain the target summary data table, when a uniqueness conflict occurs in the target summary data table, the target data is determined to be duplicate data, thereby avoiding the occurrence of business code intrusion. Data duplicate checking can be achieved by only detecting the current summary, thereby improving the efficiency of data duplicate checking. Moreover, only the current summary within the target time range needs to be detected, instead of performing a one-time overall detection on the target data, saving the memory resources required for data duplicate checking, improving the availability of data duplicate checking, and performing primary key uniqueness conflict detection on the target summary data table with the target time range and the current summary, thereby improving the accuracy of data duplicate checking.

[0063] Further, based on the first embodiment of the data duplication checking method of the present invention, a second embodiment of the data duplication checking method of the present invention is proposed. This embodiment is a refinement of step S30 of the first embodiment of the present invention, the step of determining the target time range corresponding to the current summary, including:

[0064] Step a, obtaining a current time range corresponding to the current summary, and determining a previous time range of the current time range;

[0065] In this embodiment, when determining the target time range corresponding to the current summary, it is necessary to first obtain the current time range for generating the current summary, and determine the previous time range based on the current time range. For example, when the current time range is determined to be 10 minutes, the previous time range is 10 minutes before the current time range. Among them, the current time range is the time range where the time node of the target data generates the current summary, and the time range can be set and adjusted by the user, such as setting it to 0 minutes, 5 minutes and 10 minutes. The previous time range is the time range before the current time range and the value of the current time range is equal.

[0066] Step b: determining the target time range according to the current time range and the previous time range.

[0067] After obtaining the current time range and the previous time range, you need to first determine whether the current time range and the previous time range are on the same day. If so, directly add the current time range and the previous time range, and use the sum as the target time range. For example, if the current time range is 10 minutes, and the time node for generating the current summary is 5:40-5:50 on the data trading day, then the previous time range is 5:30-5:40. At this time, the target time range can be determined to be 5:30-5:40, that is, the target time range is 20 minutes. If the current time range and the previous time range are not on the same day, the target time range is also determined based on the current time range and the previous time range.

[0068] In this embodiment, the current time range corresponding to the current summary is obtained, the previous time range is determined, and the target time range is determined according to the current time range and the previous time range, thereby ensuring the accuracy of the obtained target time range.

[0069] Specifically, the step of determining the target time range according to the current time range and the previous time range includes:

[0070] Step c, determining the initial time node of the previous time range, determining the end time node of the current time range, and taking the time range from the initial time node to the end time node as the target time range.

[0071] After obtaining the previous time range and the current time range, you can first determine the initial time node of the previous time range (i.e. the time node at which the previous time range begins), and then determine the end time node of the current time range (i.e. the last time node of the current time range). Then directly use the time range from the initial time node to the end time node as the target time range.

[0072] In this embodiment, the time range between the initial time node of the previous time range and the end time node of the current time range is used as the target time range, thereby ensuring the accuracy of the acquired target time range.

[0073] Furthermore, the step of calculating the current summary corresponding to the target data includes:

[0074] Step m, detecting whether the configuration item corresponding to the target data meets the preset configuration condition;

[0075] In this embodiment, before calculating the current summary, it is necessary to first detect whether the configuration item corresponding to the target data meets the preset configuration conditions, so as to generate different MD5 codes (i.e., summary information) according to different detection conditions. The preset configuration conditions can be parameters set in the anti-duplication configuration item in advance by the user according to different data transactions.

[0076] Step d: if the configuration item meets the preset configuration condition, obtain the business serial number corresponding to the data transaction and the message corresponding to the target data, convert the business serial number and the message to obtain the current summary.

[0077] When it is determined that the configuration item meets the preset configuration conditions, that is, the parameters in the data transaction and the anti-duplication configuration item match, and it is determined that the business serial number is not ignored, the business serial number corresponding to the data transaction (that is, the number of this data transaction) and the message generated according to the target data can be obtained first, and the business serial number and the message are converted to obtain the current summary. For example, after performing MD5 conversion and generating an MD5 code, it is used as the current MD5 code, and the current MD5 code is used as the current summary.

[0078] In this embodiment, when the configuration item corresponding to the target data meets the preset configuration condition, the business serial number corresponding to the data transaction and the message corresponding to the target data are converted to obtain the current summary, thereby ensuring the accuracy of the obtained current summary.

[0079] Specifically, after the step of detecting whether the configuration item corresponding to the target data meets the preset configuration condition, the following steps are included:

[0080] Step e: if the configuration item does not satisfy the preset configuration condition, a message corresponding to the target data is obtained, and the message is converted to obtain a current summary.

[0081] When it is determined that the configuration item does not meet the preset configuration condition, it can be determined that the business serial number needs to be ignored. At this time, the message generated according to the target data can be directly converted to generate the current summary.

[0082] In this embodiment, when it is determined that the configuration item does not satisfy the preset configuration condition, the message corresponding to the target data is converted to obtain the current summary, thereby ensuring the accuracy of the obtained current summary.

[0083] Furthermore, the step of detecting whether to perform duplicate checking on the target data includes:

[0084] Step f, obtaining a plurality of preset target parameters, and detecting whether the data transaction corresponding to the target data matches each of the target parameters;

[0085] In this embodiment, after the system is started, before checking for duplicate data, it is necessary to set various parameters (i.e., target parameters), such as setting the global anti-duplicate policy configuration table, the transaction MD5 flow table, the ignore check configuration table, and the duplicate transaction special processing policy table, etc. Then, it is detected whether the data transaction corresponding to the target data matches the various target parameters, and it is determined whether to check for duplicate data on the target data according to the detection results.

[0086] Step g: If the data transaction matches each of the target parameters, determine to perform data duplication check on the target data.

[0087] When it is determined that the data transaction matches each target parameter, it can be determined that the target data will be checked for duplicates. If there is a mismatch in the target parameters, or the configuration items in the anti-duplicate configuration item record table corresponding to the target data do not allow duplicate transaction checking and processing to be enabled, it is determined that the target data will not be checked for duplicates.

[0088] In this embodiment, when it is determined that the data transaction matches each target parameter, it is determined that the data transaction is to be subjected to data duplication checking, thereby ensuring that the data duplication checking is carried out normally.

[0089] Furthermore, the step of detecting whether the data transaction corresponding to the target data satisfies a preset ignoring condition includes:

[0090] Step h, obtaining a preset configuration anti-duplicate ignore table, and detecting whether the transaction type of the data transaction matches the ignore configuration item in the configuration anti-duplicate ignore table;

[0091] In this embodiment, it is necessary to first obtain the configuration anti-duplicate ignore table (i.e., ignore check configuration table) set in advance by the user, and detect whether the transaction type of the data transaction matches the ignore configuration item in the configuration anti-duplicate ignore table, and perform different operations according to different detection results. Among them, the configuration anti-duplicate ignore table stores multiple arbitrary transaction types (i.e., ignore configuration items) set in advance by the user.

[0092] Step k: if the transaction type and the ignore configuration item do not match, determine that the data transaction does not meet the preset ignore condition.

[0093] When it is determined that the transaction type and the ignored configuration item do not match, it is determined that the data transaction does not meet the preset ignored condition, that is, a data duplicate check operation is required. However, if the transaction type and the ignored configuration item match, it is determined that the data transaction meets the ignored condition, and no data duplicate check operation is required.

[0094] In this embodiment, when the transaction type of the data transaction does not match the ignore configuration item in the configuration anti-duplication ignore table, it is determined that the data transaction does not meet the ignore condition, thereby ensuring the normal progress of subsequent data duplication checking operations.

[0095] In this embodiment, the settings of various parameters can be set as shown below: The table structure of the global anti-duplicate policy configuration table can be set as shown in Table 1 below.

[0096]

[0097]

[0098] Table 1

[0099] The anti-duplicate configuration item record of the global anti-duplicate policy configuration table may be shown in Table 2 below.

[0100]

[0101] Table 2

[0102] The table structure of the transaction MD5 flow table can be shown in Table 3 below.

[0103]

[0104] Table 3

[0105] The table structure of the ignored check configuration table may be as shown in Table 4 below.

[0106]

[0107]

[0108] Table 4

[0109] The table structure of the repeated transaction special processing strategy table can be shown in Table 5 below.

[0110]

[0111] Table 5

[0112] The present invention also provides a data duplication checking device, referring to Figure 3 , the data duplication checking device comprises:

[0113] The first detection module A10 is used to detect whether to perform duplicate checking on the target data when receiving the target data;

[0114] A second detection module A20, configured to detect whether the data transaction corresponding to the target data satisfies a preset ignoring condition;

[0115] The calculation module A30 is used for calculating the current summary corresponding to the target data if it is not satisfied, and determining the target time range corresponding to the current summary, inserting the target time range and the current summary into the summary data table corresponding to the target data to obtain the target summary data table, and detecting whether the target summary data table has a primary key uniqueness conflict;

[0116] The determination module A40 is configured to determine that the target data is duplicate data if a primary key uniqueness conflict occurs in the target summary data table.

[0117] Optionally, the computing module A30 is further used for:

[0118] Obtaining a current time range corresponding to the current summary, and determining a previous time range of the current time range;

[0119] The target time range is determined according to the current time range and the previous time range.

[0120] Optionally, the computing module A30 is further used for:

[0121] An initial time node of the previous time range is determined, an end time node of the current time range is determined, and a time range from the initial time node to the end time node is used as a target time range.

[0122] Optionally, the computing module A30 is further used for:

[0123] Detecting whether the configuration item corresponding to the target data meets a preset configuration condition;

[0124] If the configuration item meets the preset configuration condition, the business serial number corresponding to the data transaction and the message corresponding to the target data are obtained, and the business serial number and the message are converted to obtain the current summary.

[0125] Optionally, the computing module A30 is further used for:

[0126] If the configuration item does not satisfy the preset configuration condition, a message corresponding to the target data is obtained, and the message is converted to obtain a current summary.

[0127] Optionally, the first detection module A10 is further used to:

[0128] Acquire a plurality of preset target parameters, and detect whether the data transaction corresponding to the target data matches each of the target parameters;

[0129] If the data transaction matches each of the target parameters, it is determined to perform data duplication checking on the data transaction.

[0130] Optionally, the second detection module A20 is further used to:

[0131] Obtaining a preset configuration anti-duplicate ignore table, and detecting whether the transaction type of the data transaction matches the ignore configuration item in the configuration anti-duplicate ignore table;

[0132] If the transaction type and the ignore configuration item do not match, it is determined that the data transaction does not meet the preset ignore condition.

[0133] The methods executed by the above-mentioned program units can refer to the various embodiments of the data duplication checking method of the present invention, and will not be described in detail here.

[0134] The invention also provides a computer storage medium.

[0135] The computer storage medium of the present invention stores a data duplication checking program, and when the data duplication checking program is executed by a processor, the steps of the data duplication checking method described above are implemented.

[0136] Among them, the method implemented when the data duplication checking program running on the processor is executed can refer to the various embodiments of the data duplication checking method of the present invention, and will not be repeated here.

[0137] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0138] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0140] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data duplication checking method, characterized in that: The data duplication checking method comprises the following steps: When receiving the target data, obtaining a plurality of preset target parameters, and detecting whether the data transaction corresponding to the target data matches each of the target parameters; If the data transaction matches each of the target parameters, determining to perform data duplication check on the data transaction; If it is determined that the data transaction is to be subjected to data duplication check processing, a preset configuration anti-duplication ignore table is obtained, and it is detected whether the transaction type of the data transaction matches the ignore configuration item in the configuration anti-duplication ignore table; If the transaction type and the ignore configuration item do not match, determining that the data transaction does not meet the preset ignore condition; If it is determined that the data transaction does not meet the preset ignore condition, the current summary corresponding to the target data is calculated, and the target time range corresponding to the current summary is determined, the target time range and the current summary are inserted into the summary data table corresponding to the target data to obtain a target summary data table, and the target summary data table is detected whether a primary key uniqueness conflict occurs; If a primary key uniqueness conflict occurs in the target summary data table, the target data is determined to be duplicate data.

2. The data duplication checking method according to claim 1, characterized in that: The step of determining the target time range corresponding to the current summary includes: Obtaining a current time range corresponding to the current summary, and determining a previous time range of the current time range; The target time range is determined according to the current time range and the previous time range.

3. The data duplication checking method according to claim 2, characterized in that: The step of determining the target time range according to the current time range and the previous time range includes: An initial time node of the previous time range is determined, an end time node of the current time range is determined, and a time range from the initial time node to the end time node is used as a target time range.

4. The data duplication checking method according to claim 1, characterized in that: The step of calculating the current summary corresponding to the target data includes: Detecting whether the configuration item corresponding to the target data meets a preset configuration condition; If the configuration item meets the preset configuration condition, the business serial number corresponding to the data transaction and the message corresponding to the target data are obtained, and the business serial number and the message are converted to obtain the current summary.

5. The data duplication checking method according to claim 4, characterized in that: After the step of detecting whether the configuration item corresponding to the target data meets the preset configuration condition, the method further includes: If the configuration item does not satisfy the preset configuration condition, a message corresponding to the target data is obtained, and the message is converted to obtain a current summary.

6. A data duplication checking device, characterized in that: The data duplication checking device comprises: A first detection module is used to detect whether to perform duplicate checking on the target data when receiving the target data; A second detection module, configured to detect whether the data transaction corresponding to the target data satisfies a preset ignoring condition; a calculation module, configured to calculate the current summary corresponding to the target data if the conditions are not met, determine the target time range corresponding to the current summary, insert the target time range and the current summary into the summary data table corresponding to the target data to obtain a target summary data table, and detect whether a primary key uniqueness conflict occurs in the target summary data table; A determination module, configured to determine that the target data is duplicate data if a primary key uniqueness conflict occurs in the target summary data table; Wherein, the first detection module is further used for: Acquire a plurality of preset target parameters, and detect whether the data transaction corresponding to the target data matches each of the target parameters; If the data transaction matches each of the target parameters, determining to perform data duplication check on the data transaction; The second detection module is further used for: Obtaining a preset configuration anti-duplicate ignore table, and detecting whether the transaction type of the data transaction matches the ignore configuration item in the configuration anti-duplicate ignore table; If the transaction type and the ignore configuration item do not match, it is determined that the data transaction does not meet the preset ignore condition.

7. A data duplication checking device, characterized in that: The data duplication checking device comprises: a memory, a processor, and a data duplication checking program stored in the memory and executable on the processor. When the data duplication checking program is executed by the processor, the steps of the data duplication checking method as described in any one of claims 1 to 5 are implemented.

8. A computer storage medium, characterized in that: The computer storage medium stores a data duplication checking program, and when the data duplication checking program is executed by the processor, the steps of the data duplication checking method as described in any one of claims 1 to 5 are implemented.

Images