Data anti-duplicate checking method and device

Abstract

The application provides a data anti-duplication checking method and device. The method comprises the following steps: constructing an anti-duplication table for data anti-duplication checking; obtaining data to be checked; calculating an anti-duplication value of the data to be checked according to a preset anti-duplication algorithm; judging whether the anti-duplication value can be inserted into the anti-duplication table; and if yes, determining that the data to be checked passes the anti-duplication verification. It can be seen that the method can solve the problems of expansibility and flexibility in the existing anti-duplication method, thereby improving the database read-write performance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data anti-duplicate verification method and apparatus. Background Technology

[0002] Currently, with the increasingly complex regulatory environment in the financial industry, various financial institutions have successively developed systems for collecting and reporting customer information, transaction information, and account information to meet operational compliance requirements. To avoid data redundancy, it is necessary to filter duplicate data. Existing methods typically use unique identifiers (such as ID card number columns, order number columns, etc.) for deduplication prevention. However, in practice, it has been found that existing methods require the use of unique indexes of different identifier types for deduplication prevention. This approach has limitations; when multiple deduplication prevention conditions are simultaneously applied, multiple indexes need to be created, thus impacting database read / write performance. Therefore, existing data deduplication prevention methods are not flexible enough and are full of limitations, thereby affecting database read / write performance. Summary of the Invention

[0003] The purpose of this application is to provide a data anti-duplicate verification method and apparatus, which can solve the problems of scalability and flexibility in existing anti-duplicate methods, thereby improving database read and write performance.

[0004] The first aspect of this application provides a data deduplication prevention method, including:

[0005] Construct a deduplication prevention table for data deduplication verification;

[0006] Obtain the data to be checked for plagiarism;

[0007] The anti-duplicate value of the data to be checked is calculated according to the preset anti-duplicate algorithm;

[0008] Determine whether the anti-overload value can be inserted into the anti-overload table;

[0009] If so, then the data to be checked for plagiarism is confirmed to have passed the anti-plagiarism verification.

[0010] In the above implementation process, this method can first construct an anti-duplicate table for data anti-duplicate verification, and then obtain the data to be checked for duplication. When the data to be checked for duplication is obtained, the anti-duplicate value of the data to be checked is calculated according to a preset anti-duplicate algorithm; and it is determined whether the anti-duplicate value can be inserted into the anti-duplicate table; when the anti-duplicate value can be inserted into the anti-duplicate table, it is determined that the data to be checked has passed the anti-duplicate verification. It can be seen that implementing this method can solve the scalability and flexibility problems of existing anti-duplicate methods, thereby improving database read and write performance.

[0011] Furthermore, the construction of the anti-duplicate table for data anti-duplicate verification includes:

[0012] Define a duplicate prevention table and set the general columns of the duplicate prevention table; wherein the type of the duplicate prevention table is text;

[0013] Set the anti-duplicate table as a unique index.

[0014] Further, the step of calculating the anti-duplicate value of the data to be checked according to the preset anti-duplicate algorithm includes:

[0015] The unique identifier information of the data to be deduplicated is obtained based on the data to be deduplicated; wherein, the unique identifier information of the data includes customer number, order number, product identifier number and date;

[0016] The first anti-duplicate verification code and the second anti-duplicate verification code are obtained based on the unique identification information;

[0017] The first anti-duplicate value is calculated based on the preset anti-duplicate algorithm and the first anti-duplicate check code, and the second anti-duplicate value is calculated based on the anti-duplicate algorithm and the second anti-duplicate check code;

[0018] The weight protection value is obtained by summing the first weight protection value and the second weight protection value.

[0019] Further, obtaining the first anti-duplicate verification code and the second anti-duplicate verification code based on the unique identifier information includes:

[0020] Multiple character identifiers are selected from the unique identifier information and concatenated to obtain the first anti-duplicate check code;

[0021] Multiple character identifiers that are different from the first anti-duplicate check code are selected from the unique identifier information and concatenated to obtain the second anti-duplicate check code.

[0022] Further, determining whether the anti-overlap value can be inserted into the anti-overlap table includes:

[0023] Determine whether both the first anti-duplicate check code and the second anti-duplicate check code can be inserted into the anti-duplicate table;

[0024] If both the first anti-duplicate verification code and the second anti-duplicate verification code can be inserted into the anti-duplicate table, then the process of determining that the data to be checked for duplicates has passed the anti-duplicate verification is executed.

[0025] A second aspect of this application provides a data anti-duplicate verification device, the data anti-duplicate verification device comprising:

[0026] The building unit is used to build the anti-duplicate table for data anti-duplicate verification;

[0027] The acquisition unit is used to acquire the data to be checked for plagiarism.

[0028] The calculation unit is used to calculate the anti-duplicate value of the data to be checked according to a preset anti-duplicate algorithm;

[0029] The judgment unit is used to determine whether the anti-overlap value can be inserted into the anti-overlap table;

[0030] The determining unit determines that the data to be checked has passed the anti-duplicate verification when it determines that the anti-duplicate value can be inserted into the anti-duplicate table.

[0031] In the above implementation process, the device can construct an anti-duplicate table for data anti-duplicate verification through a construction unit; acquire the data to be checked through an acquisition unit; calculate the anti-duplicate value of the data to be checked according to a preset anti-duplicate algorithm through a calculation unit; determine whether the anti-duplicate value can be inserted into the anti-duplicate table through a judgment unit; and determine that the data to be checked passes the anti-duplicate verification when the determination unit determines that the anti-duplicate value can be inserted into the anti-duplicate table. It is evident that implementing this method can solve the scalability and flexibility problems of existing anti-duplicate methods, thereby improving database read and write performance.

[0032] Furthermore, the construction unit is specifically used to define an anti-duplicate table and set the general columns of the anti-duplicate table; wherein the type of the anti-duplicate table is text; and the anti-duplicate table is set as a unique index.

[0033] Furthermore, the computing unit includes:

[0034] The first acquisition subunit is used to acquire unique data identification information based on the data to be deduplicated; wherein, the unique data identification information includes customer number, order number, product identification number, and date;

[0035] The second acquisition subunit is used to acquire the first anti-duplicate verification code and the second anti-duplicate verification code based on the unique identification information.

[0036] The calculation subunit is used to calculate a first anti-duplicate value according to a preset anti-duplicate algorithm and the first anti-duplicate check code, and to calculate a second anti-duplicate value according to the anti-duplicate algorithm and the second anti-duplicate check code;

[0037] The summarization subunit is used to summarize the first anti-weight value and the second anti-weight value to obtain the anti-weight value.

[0038] Furthermore, the second acquisition subunit includes:

[0039] The first parallel module is used to select multiple character identifiers from the unique identifier information and perform character parallel concatenation to obtain the first anti-duplicate check code.

[0040] The second concatenation module is used to select multiple character identifiers that are different from the first anti-duplicate check code from the unique identifier information and perform character concatenation processing to obtain the second anti-duplicate check code.

[0041] Furthermore, the judgment unit is specifically used to determine whether both the first anti-duplicate check code and the second anti-duplicate check code can be inserted into the anti-duplicate table;

[0042] The determining unit is specifically used to determine that the data to be checked passes the anti-duplicate verification when both the first anti-duplicate verification code and the second anti-duplicate verification code can be inserted into the anti-duplicate table.

[0043] A third aspect of this application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to cause the electronic device to perform the data anti-duplicate verification method described in any one of the first aspects of this application.

[0044] The fourth aspect of this application provides a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the data anti-duplicate verification method described in any one of the first aspects of this application. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating a data anti-duplicate verification method provided in an embodiment of this application;

[0047] Figure 2 A flowchart illustrating another data anti-duplicate verification method provided in this application embodiment;

[0048] Figure 3 This is a schematic diagram of the structure of a data anti-duplicate verification device provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of another data anti-duplicate verification device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0051] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0052] Example 1

[0053] Please refer to Figure 1 , Figure 1 This application provides a flowchart illustrating a data deduplication prevention verification method. The method includes:

[0054] S101. Construct a data anti-duplicate table for data anti-duplicate verification.

[0055] S102. Obtain the data to be checked for plagiarism.

[0056] S103. Calculate the anti-duplicate value of the data to be checked according to the preset anti-duplicate algorithm.

[0057] S104. Determine whether the anti-duplicate value can be inserted into the anti-duplicate table. If yes, proceed to step S105; otherwise, end the process.

[0058] S105. Confirm that the data to be checked for plagiarism has passed the anti-plagiarism verification.

[0059] Currently, traditional programming uses unique indexes in databases for deduplication prevention. This involves adding a unique index to one or more columns in a database table (e.g., ID number column, order number column, etc.) to prevent data duplication. However, this method has limitations. For example, if a customer can only purchase the same product once on the same day, the traditional approach to the order's goods table would require the following processing:

[0060] Create a unique index for the order number and product ID to restrict the same product to appear only once in a customer's order;

[0061] Create a unique index for date, customer number, and product ID to restrict the same product from appearing only once for the same customer on the same day.

[0062] As can be seen from the above examples, using the traditional database unique index restriction method to restrict multiple conditions simultaneously requires creating multiple indexes. However, multiple indexes will affect the database read and write performance. This traditional method is not flexible enough and is full of limitations.

[0063] However, this method can solve the scalability and flexibility problems in the traditional database anti-duplicate methods by using a programming approach that splits into independent anti-duplicate tables, a single unique index, and a strategy pattern, thereby improving database read and write performance.

[0064] In this embodiment, the subject executing the method can be a computing device such as a computer or server, and no limitation is made in this embodiment.

[0065] In this embodiment, the subject executing the method can also be a smart device such as a smartphone or tablet, and no limitation is made in this embodiment.

[0066] As can be seen, the data deduplication prevention verification method described in this embodiment can solve the problems of scalability and flexibility in deduplication programming by utilizing the strategy pattern programming approach. Since the strategy is dynamic, different strategies can be used for deduplication prevention in different situations. Furthermore, for scenarios with multiple deduplication prevention conditions, this method can complete data deduplication prevention with a single unique index, thus requiring only one unique index in the database table and eliminating the need for multiple indexes, thereby avoiding impacting the read and write performance of the data table.

[0067] Example 2

[0068] Please refer to Figure 2 , Figure 2 This application provides a flowchart illustrating a data deduplication prevention verification method. The method includes:

[0069] S201. Define a deduplication prevention table and set the general columns of the deduplication prevention table; wherein, the type of the deduplication prevention table is text type.

[0070] S202. Set the anti-duplicate table as a unique index.

[0071] In this embodiment, the method can first define an independent anti-duplicate table, set a general column of type text, and set it as a unique index.

[0072] S203. Obtain the data to be checked for plagiarism.

[0073] S204. Obtain the unique identifier information of the data based on the data to be checked for plagiarism; the unique identifier information includes the customer number, order number, product identifier number, and date.

[0074] S205. Select multiple character identifiers from the unique identifier information and concatenate them to obtain the first anti-duplicate check code.

[0075] S206. Select multiple character identifiers that are different from the first anti-duplicate check code from the unique identifier information and perform character concatenation processing to obtain the second anti-duplicate check code.

[0076] S207. Calculate the first anti-duplicate value according to the preset anti-duplicate algorithm and the first anti-duplicate check code, and calculate the second anti-duplicate value according to the anti-duplicate algorithm and the second anti-duplicate check code.

[0077] S208, sum up the first and second anti-weight values ​​to obtain the anti-weight value.

[0078] S209. Determine whether both the first anti-duplicate check code and the second anti-duplicate check code can be inserted into the anti-duplicate table. If yes, proceed to step S210; otherwise, end the process.

[0079] S210. Confirm that the data to be checked for plagiarism has passed the anti-plagiarism verification.

[0080] In this embodiment, the method can predefine a policy interface. Here, genCheckKey is the method for calculating the anti-duplicate value, CommonDto is the common data DTO, and the getName method returns the policy service name.

[0081] The code for calculating the anti-weight value is shown below:

[0082]

[0083]

[0084] In this embodiment, the method can further implement the GenKeyService interface as needed. When implementing genCheckKey, it can return the anti-duplicate value as a string concatenation as required. For example...

[0085]

[0086]

[0087] In this embodiment, the method can also utilize Spring functionality to create a factory at container startup, allowing business logic applications to retrieve the anti-duplicate value calculation service based on the name. The specific code is as follows:

[0088]

[0089]

[0090] In this embodiment, when writing to the database, the method can calculate the anti-duplicate value key as needed. For example, the customer number plus the order number is used as the anti-duplicate key to prevent the same customer from appearing repeatedly in the same order. Example code is as follows:

[0091]

[0092]

[0093] In this embodiment, the method can write the calculated key into the general anti-duplicate unique index column of the database at this point, but does not commit it. Then, according to the anti-duplicate business logic, the two steps of calculating the anti-duplicate value and writing the calculated key into the general anti-duplicate unique index column of the database are repeated until all anti-duplicate scenarios are processed, and then a single commit is made. Specifically, the first time, the order number is combined with the product ID to calculate the key value and write it; the second time, the date, customer number, and product ID are combined to calculate the key value and write it.

[0094] In this embodiment, if the above steps can be successfully implemented, the submission is considered successful, that is, the data deduplication check is passed.

[0095] In this embodiment, the overall idea of ​​the method is as follows: Utilize the database to analyze the independent anti-duplicate table, set a general column of type text, set it as a unique index, and calculate the anti-duplicate value according to the required anti-duplicate needs through a dynamic strategy mode. Write the calculation result into this column. If the insertion and submission are successful, it means that the anti-duplicate verification has been passed; otherwise, the data is duplicated.

[0096] In this embodiment, the subject executing the method can be a computing device such as a computer or server, and no limitation is made in this embodiment.

[0097] In this embodiment, the subject executing the method can also be a smart device such as a smartphone or tablet, and no limitation is made in this embodiment.

[0098] As can be seen, the data deduplication prevention verification method described in this embodiment can solve the problems of scalability and flexibility in deduplication programming by utilizing the strategy pattern programming approach. Since the strategy is dynamic, different strategies can be used for deduplication prevention in different situations. Furthermore, for scenarios with multiple deduplication prevention conditions, this method can complete data deduplication prevention with a single unique index, thus requiring only one unique index in the database table and eliminating the need for multiple indexes, thereby avoiding impacting the read and write performance of the data table.

[0099] Example 3

[0100] Please refer to Figure 3 , Figure 3 This is a schematic diagram of a data anti-duplicate verification device provided in an embodiment of this application. Figure 3 As shown, the data anti-duplicate verification device includes:

[0101] Construction unit 310 is used to construct an anti-duplicate table for data anti-duplicate verification;

[0102] Acquisition unit 320 is used to acquire the data to be checked for plagiarism.

[0103] The calculation unit 330 is used to calculate the anti-duplicate value of the data to be checked according to the preset anti-duplicate algorithm;

[0104] The judgment unit 340 is used to determine whether the anti-overlap value can be inserted into the anti-overlap table;

[0105] Unit 350 determines that when it is determined that the anti-duplicate value can be inserted into the anti-duplicate table, the data to be checked is confirmed to have passed the anti-duplicate verification.

[0106] In this embodiment, the explanation of the data anti-duplicate verification device can be referred to the description in Embodiment 1 or Embodiment 2, and will not be repeated here.

[0107] As can be seen, the data deduplication prevention and verification device described in this embodiment can solve the problems of scalability and flexibility in deduplication programming by utilizing the strategy pattern programming method. Since the strategy is dynamic, different strategies can be used for deduplication prevention in different situations. Furthermore, for scenarios with multiple deduplication conditions, this method can complete data deduplication prevention with a single unique index, thus requiring only one unique index in the database table and eliminating the need for multiple indexes, thereby avoiding impacting the read and write performance of the data table.

[0108] Example 4

[0109] Please refer to Figure 4 , Figure 4 This is a schematic diagram of a data anti-duplicate verification device provided in an embodiment of this application. Figure 4 As shown, the data anti-duplicate verification device includes:

[0110] Construction unit 310 is used to construct an anti-duplicate table for data anti-duplicate verification;

[0111] Acquisition unit 320 is used to acquire the data to be checked for plagiarism.

[0112] The calculation unit 330 is used to calculate the anti-duplicate value of the data to be checked according to the preset anti-duplicate algorithm;

[0113] The judgment unit 340 is used to determine whether the anti-double value can be inserted into the anti-double table;

[0114] Unit 350 determines that when it is determined that the anti-duplicate value can be inserted into the anti-duplicate table, the data to be checked is confirmed to have passed the anti-duplicate verification.

[0115] As an optional implementation, the construction unit 310 is specifically used to define the anti-duplicate table and set the general columns of the anti-duplicate table; wherein, the type of the anti-duplicate table is text; and the anti-duplicate table is set as a unique index.

[0116] As an optional implementation, the computing unit 330 includes:

[0117] The first acquisition subunit 331 is used to acquire unique data identification information based on the data to be deduplicated; wherein, the unique data identification information includes customer number, order number, product identification number and date;

[0118] The second acquisition subunit 332 is used to acquire the first anti-duplicate verification code and the second anti-duplicate verification code based on the unique identification information.

[0119] The calculation subunit 333 is used to calculate the first anti-duplicate value according to the preset anti-duplicate algorithm and the first anti-duplicate check code, and to calculate the second anti-duplicate value according to the anti-duplicate algorithm and the second anti-duplicate check code;

[0120] The summary sub-unit 334 is used to summarize the first and second anti-weight values ​​to obtain the anti-weight value.

[0121] As an optional implementation, the second acquisition subunit 332 includes:

[0122] The first module is used to select multiple character identifiers from the unique identifier information and concatenate them to obtain the first anti-duplicate check code.

[0123] The second concatenation module is used to select multiple character identifiers that are different from the first anti-duplicate check code from the unique identifier information and perform character concatenation processing to obtain the second anti-duplicate check code.

[0124] As an optional implementation, the judgment unit 340 is specifically used to determine whether both the first anti-duplicate check code and the second anti-duplicate check code can be inserted into the anti-duplicate table;

[0125] The determining unit 350 is specifically used to determine that the data to be checked has passed the anti-duplicate verification when both the first anti-duplicate verification code and the second anti-duplicate verification code can be inserted into the anti-duplicate table.

[0126] In this embodiment, the explanation of the data anti-duplicate verification device can be referred to the description in Embodiment 1 or Embodiment 2, and will not be repeated here.

[0127] As can be seen, the data deduplication prevention and verification device described in this embodiment can solve the problems of scalability and flexibility in deduplication programming by utilizing the strategy pattern programming method. Since the strategy is dynamic, different strategies can be used for deduplication prevention in different situations. Furthermore, for scenarios with multiple deduplication conditions, this method can complete data deduplication prevention with a single unique index, thus requiring only one unique index in the database table and eliminating the need for multiple indexes, thereby avoiding impacting the read and write performance of the data table.

[0128] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the data anti-duplicate verification method in embodiment 1 or embodiment 2 of this application.

[0129] This application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the data anti-duplicate verification method in embodiment 1 or embodiment 2 of this application is performed.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0131] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0132] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0134] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0135] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A data deduplication prevention verification method, characterized in that, include: Construct a deduplication prevention table for data deduplication verification; Obtain the data to be checked for plagiarism; The unique identifier information of the data to be deduplicated is obtained based on the data to be deduplicated; wherein, the unique identifier information of the data includes customer number, order number, product identifier number and date; The first anti-duplicate verification code and the second anti-duplicate verification code are obtained based on the unique identification information; The first anti-duplicate value is calculated based on the preset anti-duplicate algorithm and the first anti-duplicate check code, and the second anti-duplicate value is calculated based on the anti-duplicate algorithm and the second anti-duplicate check code; The weight protection value is obtained by summing the first weight protection value and the second weight protection value; Determine whether the anti-overload value can be inserted into the anti-overload table; If so, then the data to be checked for plagiarism is confirmed to have passed the anti-plagiarism verification; The construction of the anti-duplicate table for data anti-duplicate verification includes: Define a duplicate prevention table and set the general columns of the duplicate prevention table; wherein the type of the duplicate prevention table is text; Set the anti-duplicate table as a unique index; This includes defining an anti-duplicate table and setting the general columns of the anti-duplicate table, including: Build a separate anti-duplicate table in the database for data anti-duplicate verification; and set a general column of text type in the anti-duplicate table; the general column stores the string anti-duplicate key calculated by the GenKeyService interface implementation class; The first anti-repetition value is calculated by combining the order number with the product ID; The second anti-repetition value is calculated by combining the date, customer number, and product ID; The step of determining whether the anti-overlap value can be inserted into the anti-overlap table includes: Determine whether both the first anti-overload value and the second anti-overload value can be inserted into the anti-overload table; If both the first anti-duplicate value and the second anti-duplicate value can be inserted into the anti-duplicate table, then the process of determining that the data to be checked for duplicates has passed the anti-duplicate verification is performed.

2. The data anti-duplicate verification method according to claim 1, characterized in that, The step of obtaining the first anti-duplicate verification code and the second anti-duplicate verification code based on the unique identifier information includes: Multiple character identifiers are selected from the unique identifier information and concatenated to obtain the first anti-duplicate check code; Multiple character identifiers that are different from the first anti-duplicate check code are selected from the unique identifier information and concatenated to obtain the second anti-duplicate check code.

3. A data anti-duplicate verification device, characterized in that, The data anti-duplicate verification device includes: The building unit is used to build the anti-duplicate table for data anti-duplicate verification; The acquisition unit is used to acquire the data to be checked for plagiarism. The calculation unit is used to calculate the anti-duplicate value of the data to be checked according to a preset anti-duplicate algorithm; The judgment unit is used to determine whether the anti-overlap value can be inserted into the anti-overlap table; The determining unit determines that the data to be checked passes the anti-duplicate verification when it determines that the anti-duplicate value can be inserted into the anti-duplicate table; Specifically, the construction unit is used to define an anti-duplicate table and set the general columns of the anti-duplicate table; wherein the type of the anti-duplicate table is text; and the anti-duplicate table is set as a unique index; Specifically, the construction unit is used to build an independent anti-duplicate table in the database for data anti-duplicate verification; and to set a general column of text type in the anti-duplicate table; the general column stores the string anti-duplicate key calculated by the GenKeyService interface implementation class; The computing unit includes: The first acquisition subunit is used to acquire unique data identification information based on the data to be deduplicated; wherein, the unique data identification information includes customer number, order number, product identification number, and date; The second acquisition subunit is used to acquire the first anti-duplicate verification code and the second anti-duplicate verification code based on the unique identification information. The calculation subunit is used to calculate a first anti-duplicate value according to a preset anti-duplicate algorithm and the first anti-duplicate check code, and to calculate a second anti-duplicate value according to the anti-duplicate algorithm and the second anti-duplicate check code; The summarization subunit is used to summarize the first anti-weight value and the second anti-weight value to obtain the anti-weight value; The first anti-repetition value is calculated by combining the order number with the product ID; The second anti-repetition value is calculated by combining the date, customer number, and product ID; Specifically, the judgment unit is used to determine whether both the first anti-overload value and the second anti-overload value can be inserted into the anti-overload table; The determining unit is specifically used to determine that the data to be checked passes the anti-duplicate verification when both the first anti-duplicate value and the second anti-duplicate value can be inserted into the anti-duplicate table.

4. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the data anti-duplicate verification method according to any one of claims 1 to 2.

5. A readable storage medium, characterized in that, The readable storage medium stores computer program instructions, which, when read and executed by a processor, perform the data anti-duplicate verification method according to any one of claims 1 to 2.

Images