A data duplicate checking method, device and medium

Abstract

The application discloses a data duplication checking method and device and a medium, and is suitable for the technical field of computers. The target pair key data is generated from the data to be checked, and then is compared with the bitmap data of the redis list. If there is no duplication, it is determined that there is no duplication in the redis list, thereby saving a large amount of data comparison time. If there is duplication, it is further determined whether it is a misjudgment problem in the redis list, and then the secondary check of the stored data in the database is compared. When there is duplication in the database, it is determined that there is duplication in the redis list. If there is no duplication in the database, it is determined that there is no duplication. The secondary check is compared in the database, the time for comparing multiple data sources is saved, the operation time for data comparison and duplication checking is reduced, and the experience effect of users is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus and medium for data deduplication. Background Technology

[0002] In the Software-as-a-Service (SaaS) model, all the network infrastructure and software / hardware operating platforms needed for enterprise IT infrastructure are built, and the provider is responsible for all pre-implementation and post-maintenance services. Enterprises can use the information system via the internet without purchasing hardware and software, building data centers, or hiring IT personnel. Due to the large number of tenants and diverse data application scenarios, most systems require multi-dimensional, multi-source data and massive data comparisons.

[0003] Typically, when comparing massive amounts of data, the large volume of data compared due to multiple dimensions and data sources results in a long data request chain when comparing data one by one. This also increases the waiting time for users during the comparison process, leading to a poor user experience.

[0004] Therefore, how to reduce the operation time of data comparison and deduplication is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, and medium for data deduplication, thereby reducing the operation time for data comparison and deduplication.

[0006] To solve the above-mentioned technical problems, the present invention provides a method for data deduplication, comprising:

[0007] Obtain the data to be checked for plagiarism and determine the data type based on the request parameters corresponding to the data to be checked for plagiarism.

[0008] Based on the data type and request parameters, locate the target key data corresponding to the data to be deduplicated in the bitmap rules;

[0009] Determine whether the target bitmap data corresponding to the target key data is found in the Redis list;

[0010] If not, then it is determined that there is no duplicate data corresponding to the data to be deduplicated in the Redis list;

[0011] If so, the data is routed to the database corresponding to the target bitmap data, and it is determined whether there is duplicate data in the database corresponding to the data to be deduplicated.

[0012] If it exists, then it is determined that there is duplicate data in the Redis list;

[0013] If not, it is determined that there is no duplicate data in the redis list.

[0014] Preferably, the determination of the pair key data in the bitmap rule is obtained by the following way:

[0015] Obtaining the data to be written and determining the data type corresponding to the data to be written;

[0016] Determining the corresponding mapping rule in the bitmap rule according to the data type, wherein the mapping rule is the rule of the data type, the pair key data and the database storing the pair key data;

[0017] Determining the pair key data corresponding to the data to be written according to the relationship between the data type and the mapping rule.

[0018] Preferably, the determination of the bitmap data of the redis list is obtained by the following way:

[0019] Generating the corresponding bitmap data according to the pair key data in the bitmap rule and saving it to the redis list.

[0020] Preferably, after determining the bitmap data of the redis list, it further comprises:

[0021] Saving the data to be written to the database corresponding to the pair key data.

[0022] Preferably, the determination of the bitmap data of the redis list is also obtained by the following way:

[0023] Generating the corresponding subscription data according to the data to be written in the database and determining the subscription data type corresponding to the subscription data;

[0024] Determining the corresponding pair key data in the bitmap rule according to the subscription data type;

[0025] Generating the corresponding bitmap data according to the pair key data and saving it to the redis list.

[0026] Preferably, it further comprises:

[0027] When determining the query result in the redis list, a prompt information is outputted.

[0028] To solve the above technical problems, the application also provides a data duplication checking device, comprising:

[0029] A first determination module is configured to obtain data to be checked and determine the data type according to the request parameter corresponding to the data to be checked;

[0030] A query module is configured to find the target pair key data corresponding to the data to be checked in the bitmap rule according to the data type and the request parameter;

[0031] The first determining module is used for determining whether the target bitmap data corresponding to the target key data pair is queried from the redis list, if not, triggering the second determining module, and if yes, triggering the second determining module;

[0032] The second determining module is used for determining that the duplicate data corresponding to the data to be checked is not present in the redis list.

[0033] The second determining module is used for determining that the duplicate data corresponding to the data to be checked is not present in the redis list.

[0034] The third determining module is used for determining that the duplicate data is present in the redis list.

[0035] The fourth determining module is used for determining that the duplicate data is not present in the redis list.

[0036] To solve the above technical problems, the application further provides a data duplication checking device, which comprises:

[0037] The memory is used for storing the computer program.

[0038] The processor is used for executing the computer program to realize the steps of the data duplication checking method.

[0039] To solve the above technical problems, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the data duplication checking method.

[0040] The application provides a data duplication checking method, which comprises the following steps: obtaining to-be-checked data, and determining a data type according to a request parameter corresponding to the to-be-checked data; finding target pair key data corresponding to the to-be-checked data in a bitmap rule according to the data type and the request parameter; judging whether target bitmap data corresponding to the target pair key data is queried in a redis list; if not, determining that there is no duplicate data corresponding to the to-be-checked data in the redis list; if yes, routing to a database corresponding to the target bitmap data, and judging whether there is duplicate data corresponding to the to-be-checked data in the database; if yes, determining that there is duplicate data in the redis list; if not, determining that there is no duplicate data in the redis list. The method generates target pair key data through the to-be-checked data, and then compares the target pair key data with bitmap data in the redis list, if there is no duplicate data, it is determined that there is no duplicate data in the redis list to save a large amount of data comparison time, if there is duplicate data, it is further determined whether it is a false positive problem in the redis list, and then the stored data in the database is compared for secondary verification, when there is duplicate data in the database, it is determined that there is duplicate data in the redis list, if there is no duplicate data in the database, it is determined that there is no duplicate data. The secondary verification is compared in the database, the time for comparing multiple data sources is saved, the operation time for data comparison and duplication checking is reduced, and the experience effect of users is improved.

[0041] In addition, the application also provides a data duplication checking device and medium, which have the same beneficial effects as the data duplication checking method. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 A flowchart of a data duplication checking method provided by the embodiments of the application;

[0044] Figure 2 A structural diagram of a data duplication checking device provided by the embodiments of the application;

[0045] Figure 3 A structural diagram of another data duplication checking device provided by the embodiments of the application;

[0046] Figure 4 An application scenario schematic diagram of another data duplication checking device provided by the embodiments of the application. DETAILED DESCRIPTION

[0047] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0048] The core of the present application is to provide a data duplication checking method, device and medium, which reduces the operation time of data comparison and duplication checking.

[0049] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] It should be noted that the data duplication checking method provided by the present application is suitable for the application scenario of comparing multi-dimensional and multi-data source data with massive data in a SaaS scenario, and can also be suitable for other multi-dimensional and multi-data source data comparison scenarios. The present application does not make specific limitations, and is set according to the actual situation.

[0051] Figure 1 A flowchart of a data duplication checking method provided by an embodiment of the present application is shown in Figure 1 The method comprises the following steps:

[0052] S11: Obtain the data to be checked, and determine the data type according to the request parameter corresponding to the data to be checked.

[0053] It can be understood that before duplication checking, a duplication checking and comparison program needs to be established, and a corresponding database and data type need to be queried. Compared with the existing data comparison one by one, it saves a lot of comparison time according to the data type. The role of the data type is index data, which can be a Map data structure, and can also be a bitmap data structure, etc. The present application does not make specific limitations. For a bitmap data, each bit stores a certain state, which is suitable for large-scale data, but the data state is not many. It is usually used to judge whether a certain data exists or not, and is suitable for the existence of the data duplication checking and comparison of the present application. It is preferred that the bitmap rule is established with the generated data type.

[0054] After the duplication checking and comparison program is established, the data to be checked is obtained, and the data type is determined according to the request parameter corresponding to the data to be checked. In the SaaS scenario, the data type is determined in the SaaS data duplication checking module according to the data to be checked. For example, the data type of the data to be checked is lead data, and the request parameter is a telephone number. It can be understood that the data to be checked can be multiple data sources and multiple dimensions of multiple data for comparison and query at the same time.

[0055] S12: Based on the data type and request parameters, find the target key data corresponding to the data to be deduplicated in the bitmap rules.

[0056] Based on the data type and request parameters, the corresponding rule is found in the bitmap rules, along with the target key data (target key) corresponding to the data to be deduplicated, i.e., the corresponding target key is found in the bitmap rules module. Using the example above, if the data to be deduplicated is a clue data type and its request parameter is a phone number, then `bitmap_clue_phone` needs to be matched.

[0057] S13: Determine whether the target bitmap data corresponding to the target key data is found in the Redis list. If not, proceed to step S14; if yes, proceed to step S15.

[0058] The corresponding target key data is found using the bitmap rules module, and then the Redis list is checked to see if the target bitmap data corresponding to the target key data exists. It's understood that the Redis list is a simple list of strings, existing in the form of [key, value], and the target bitmap data is the value of the value. If the Redis list does not find the target bitmap data corresponding to the target key data, it means that the Redis list does not contain duplicate data with the data to be deduplicated, and the process proceeds to step S14. If the target bitmap data corresponding to the target key data is found in the Redis list, it means that the Redis list contains duplicate data with the data to be deduplicated. To further confirm whether there is duplicate data in the Redis list, and whether it is due to misjudgment, incorrect storage, or incorrect generation of bitmap data, the process proceeds to step S15.

[0059] S14: Determine that there is no duplicate data in the Redis list corresponding to the data to be deduplicated.

[0060] If the Redis list does not find the target bitmap data corresponding to the target key data, it means that the Redis list does not contain duplicate data with the data to be deduplicated, that is, it is determined that the Redis list does not contain duplicate data with the data to be deduplicated.

[0061] S15: Route to the database corresponding to the target bitmap data and determine whether there is duplicate data in the database corresponding to the data to be deduplicated. If there is, proceed to step S16; otherwise, proceed to step S17.

[0062] To further confirm whether there is duplicate data in the Redis list, since the Redis list is only arranged by bitmap key and bitmap data, and its routing database stores the corresponding bitmap data and the bitmap key's mapping rules, it is necessary to further check whether there is duplicate data in the routing database corresponding to the data to be deduplicated.

[0063] The routing data supports different data structure organizations of the routing table, and the data structure query involved can be realized by a hash table routing table or other ways. When the database has duplicate data, it is determined that the redis list has no misjudgment problem, and step S16 is entered, that is, it is determined that the redis list has duplicate data. When the database has no duplicate data, it is determined that the redis list has a misjudgment problem, and finally, the judgment of the database that there is no duplicate data is given priority, and step S17 is entered.

[0064] S16: Determine that the redis list has duplicate data.

[0065] When the database has duplicate data corresponding to the duplicate data to be checked, it is determined that the redis list has no misjudgment problem, and there is duplicate data, that is, the output data already exists, and the instruction with duplicate data.

[0066] S17: Determine that the redis list has no duplicate data.

[0067] When the database has no duplicate data corresponding to the duplicate data to be checked, it is determined that the redis list has a misjudgment problem, and the judgment of the database is used as a reference, that is, it is determined that the redis list has no duplicate data.

[0068] The method for checking duplicate data provided by the application comprises the following steps: obtaining duplicate data to be checked, and determining the data type according to the request parameter corresponding to the duplicate data to be checked; finding the target pair key data corresponding to the duplicate data to be checked in the bitmap rule according to the data type and the request parameter; judging whether the target bitmap data corresponding to the target pair key data is queried in the redis list; if not, it is determined that the redis list has no duplicate data corresponding to the duplicate data to be checked; if yes, the database corresponding to the target bitmap data is routed, and it is judged whether the database has duplicate data corresponding to the duplicate data to be checked; if yes, it is determined that the redis list has duplicate data; if not, it is determined that the redis list has no duplicate data. The method generates target pair key data through duplicate data to be checked, and then compares the target pair key data with bitmap data of the redis list. If there is no duplicate data, it is determined that the redis list has no duplicate data to save a lot of data comparison time. If there is duplicate data, it is further determined whether it is a misjudgment problem in the redis list, and then the storage data in the database is compared for secondary verification. When the database has duplicate data, it is determined that the redis list has duplicate data. If the database has no duplicate data, it is determined that there is no duplicate data. The secondary verification in the database saves the time for comparing multiple data sources, reduces the operation time for checking duplicate data, and improves the user experience.

[0069] On the basis of the above embodiment, the determination of the key data of the bitmap rule mentioned in step S12 is obtained by the following way:

[0070] Obtaining the to-be-written data and determining the data type corresponding to the to-be-written data;

[0071] Determining the corresponding mapping rule in the bitmap rule according to the data type, wherein the mapping rule is the data type and the rule of the key data and the database storing the key data;

[0072] Determining the key data corresponding to the to-be-written data according to the relationship between the data type and the mapping rule.

[0073] In establishing the bitmap rule, the to-be-written data needs to be obtained to determine the data type corresponding to the to-be-written data, and the mapping rule corresponding to the bitmap rule is determined according to the data type, wherein the mapping rule is the data type and the rule of the key data and the database storing the key data. Further, the key data corresponding to the to-be-written data is further determined according to the relationship between the data type and the mapping rule.

[0074] The bitmap rule in the bitmap method defines the mapping rule of the data type and the key of the bitmap in redis and the relationship between the actual data storage of the bitmap key and the database. For example, if it is a clue data, the bitmap_clue_phone rule (the data is stored in the mysql database under the condition that the phone number is not empty) and the bitmap_clue_cropname rule (the data is stored in the elasticsearch database under the condition that the company name is not empty) are used. The bitmap is a bit array to store a certain data structure. Specifically, the bitmap rule can be multiple rules, which are stored in a special database as a bitmap rule module.

[0075] The determination of the key data of the bitmap rule provided by the embodiment of the application when establishing the bitmap rule makes it convenient to query and compare by determining the corresponding bitmap data according to the data type when checking the duplicate data according to the different bitmap rules of the data type.

[0076] On the basis of the above embodiment, the determination of the bitmap data of the redis list in step S13 is obtained by the following way:

[0077] Generating the corresponding bitmap data according to the key data in the bitmap rule and saving it to the redis list.

[0078] It can be understood that the bitmap generation module needs to formulate different bitmap data according to different bitmap rules in the bitmap rule module, and the mapping relationship between the obtained bitmap data and the key data of the bitmap rule is saved in the redis list.

[0079] Bitmap generation is an operation for bitmap, for example, a data is added to the data type of clue, the phone number is 15811111111, and the bitmap data of 15811111111 is generated, and the bitmap data is added to the key bitmap_clue_phone in the redis list and saved to the redis list.

[0080] The bitmap generation module provided by the embodiment of the application determines the bitmap data corresponding to the redis list by establishing bitmap generation, so that when the data is checked for duplication, different bitmap rules are determined according to the data type, and then it is checked whether the corresponding bitmap data in the redis list is duplicated to realize data comparison, thereby avoiding comparison of a large amount of data one by one and reducing comparison query time.

[0081] On the basis of the above embodiment, after the bitmap data of the redis list is determined, the method further comprises:

[0082] The data to be written is saved to the database corresponding to the key data.

[0083] Specifically, when the data checking system is established, the data to be written is matched with different rules according to the data type and the bitmap rule, and then the data is stored in the database corresponding to the bitmap rule, for example, the data type of the clue data uses the bitmap_clue_phone rule (the data is stored in the mysql database when the phone number is not empty), and the bitmap_clue_cropname rule (the data is stored in the elasticsearch database when the company name is not empty).

[0084] The embodiment of the application provides that the data to be written is saved to the database corresponding to the key data, which facilitates secondary verification when subsequent data comparison and duplication checking are performed, and routes to a specific database to save time when a plurality of data sources are compared simultaneously.

[0085] On the basis of the above embodiment, since different bitmap rules generate corresponding bitmap generation data, due to different computing units and other irresistible factors, the bitmap rule does not generate corresponding bitmap data, and another way of generating bitmap data is opened, that is, the determination of the bitmap data of the redis list is also obtained by the following way:

[0086] Corresponding subscription data is generated according to the data to be written in the database, and the subscription data type corresponding to the subscription data is determined;

[0087] The corresponding key data is determined in the bitmap rule according to the subscription data type.

[0088] Corresponding bitmap data is generated according to the key data and saved to the redis list.

[0089] According to the data to be written into the specific database, the data to be written into the database generates corresponding subscription data. The subscription data determines the corresponding subscription data type according to the subscription field, and further determines the corresponding key data in the bitmap rule according to the subscription data type, and further generates corresponding bitmap data according to the key data and saves it to the redis list.

[0090] Specifically, when the bitmap data is saved to the redis list, when there is no duplicate data, due to the uniqueness of the bitmap generation, there is no duplicate bitmap data in the updated redis list.

[0091] The embodiment provided establishes the determination of the bitmap data of the redis list, establishes the bitmap generation module, so that when the duplicate data is checked, different bitmap rules are determined according to the data type, and then the corresponding bitmap data in the redis list is queried to realize data comparison, avoid a large amount of data comparison, and reduce the comparison query time. At the same time, avoid saving the data to be written into the database, and ensure that the redis list is also updated synchronously.

[0092] On the basis of the above-mentioned embodiment, it further comprises:

[0093] When determining the query result in the redis list, output a prompt information.

[0094] In combination with the above-mentioned embodiment, when it is determined that there is no duplicate data corresponding to the data to be checked in the redis list, a first prompt information is outputted, and when it is determined that there is duplicate data corresponding to the data to be checked in the redis list, a second prompt information is outputted. Specifically, the first prompt information and the second prompt information can be the same or different.

[0095] When it is a voice information prompt, different query results can be played; or the corresponding query instruction can be popped up for the display, or the specific query result of the query system can be prompted, which is not limited by the present application.

[0096] The embodiment provided determines the query result in the redis list, and outputs a prompt information, which is used for timely reminding the comparison result of the staff, so as to perform subsequent comparison data or other subsequent operation work processing.

[0097] The above detailed description of the data duplication checking method corresponds to each embodiment, and on this basis, the present application further discloses a data duplication checking device corresponding to the above-mentioned data duplication checking method, Figure 2 A structural diagram of a data duplication checking device provided by the embodiment of the present application is shown in FIG. 1. Figure 2 As shown in the figure, the data duplication checking device comprises:

[0098] The first determining module 11 is configured to acquire the data to be checked for duplication and determine the data type according to the request parameter corresponding to the data to be checked for duplication;

[0099] The querying module 12 is configured to find the target pair key data corresponding to the data to be checked for duplication in the bitmap rule according to the data type and the request parameter;

[0100] The first judging module 13 is configured to judge whether the target bitmap data corresponding to the target pair key data is queried in the redis list, if not, the second determining module 14 is triggered, and if yes, the second judging module 15 is triggered;

[0101] The second determining module 14 is configured to determine that the duplicate data corresponding to the data to be checked for duplication does not exist in the redis list;

[0102] The second judging module 15 is configured to route to the database corresponding to the target bitmap data and judge whether the database exists the duplicate data corresponding to the data to be checked for duplication, if yes, the third determining module 16 is triggered, and if not, the fourth determining module 17 is triggered;

[0103] The third determining module 16 is configured to determine that the duplicate data exists in the redis list;

[0104] The fourth determining module 17 is configured to determine that the duplicate data does not exist in the redis list.

[0105] Since the embodiments of the device part correspond to the above-mentioned embodiments, the embodiments of the device part are described with reference to the embodiments of the above-mentioned method part, and will not be repeated here.

[0106] The application provides a data duplication checking device, which comprises the following steps: obtaining to-be-checked data, determining the data type according to the request parameter corresponding to the to-be-checked data, finding the target pair key data corresponding to the to-be-checked data in a bitmap rule according to the data type and the request parameter, judging whether the target bitmap data corresponding to the target pair key data is queried in a redis list, if not, determining that there is no duplicate data corresponding to the to-be-checked data in the redis list, if yes, routing to the database corresponding to the target bitmap data, and judging whether the database has duplicate data corresponding to the to-be-checked data, if yes, determining that there is duplicate data in the redis list, and if not, determining that there is no duplicate data in the redis list. The device generates the target pair key data through the to-be-checked data, and then compares the target pair key data with the bitmap data in the redis list, if not, it is determined that there is no duplicate data in the redis list to save a large amount of data comparison time, if yes, it is further determined whether it is a false positive problem in the redis list, and then the stored data in the database is compared for secondary verification, when there is duplicate data in the database, it is determined that there is duplicate data in the redis list, if there is no duplicate data in the database, it is determined that there is no duplicate data. The secondary verification in the database saves the time for comparing multiple data sources, reduces the operation time for data comparison and duplication checking, and improves the user experience.

[0107] Figure 3 The structure diagram of another data duplication checking device provided by the embodiment of the application is shown in FIG. 2, which comprises the following components: Figure 3

[0108] The memory 21 is used for storing a computer program.

[0109] The processor 22 is used for executing the computer program to realize the steps of the data duplication checking method.

[0110] The data duplication checking device provided by the embodiment of the application can include but is not limited to a smart phone, a tablet computer, a notebook computer or a desktop computer.

[0111] ​The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 22 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 22 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 22 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0112] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 21 is used to store at least the following computer program 211, which, after being loaded and executed by the processor 22, is capable of implementing the relevant steps of the data deduplication method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. The operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, the data involved in the data deduplication method, etc.

[0113] In some embodiments, the data deduplication device may further include a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.

[0114] Those skilled in the field can understand, Figure 3 The structure shown does not constitute a limitation on the data deduplication device and may include more or fewer components than shown.

[0115] The processor 22 implements the data deduplication method provided in any of the above embodiments by calling instructions stored in the memory 21.

[0116] The application provides a data duplication checking device, which comprises obtaining to-be-checked data, determining a data type according to a request parameter corresponding to the to-be-checked data, finding target pair key data corresponding to the to-be-checked data in a bitmap rule according to the data type and the request parameter, judging whether target bitmap data corresponding to the target pair key data is queried in a redis list, determining that there is no duplicate data corresponding to the to-be-checked data in the redis list if the target bitmap data is not queried, routing to a database corresponding to the target bitmap data, judging whether there is duplicate data corresponding to the to-be-checked data in the database, determining that there is duplicate data in the redis list if there is duplicate data in the database, and determining that there is no duplicate data in the redis list if there is no duplicate data in the database. The device generates target pair key data from to-be-checked data, and then compares the target pair key data with bitmap data in a redis list. If there is no duplicate data, it is determined that there is no duplicate data in the redis list to save a large amount of data comparison time. If there is duplicate data, it is further determined whether the duplicate data is a false positive in the redis list, and then the stored data in the database is compared for secondary verification. If there is duplicate data in the database, it is determined that there is duplicate data in the redis list. If there is no duplicate data in the database, it is determined that there is no duplicate data. The secondary verification in the database saves the time for comparing multiple data sources, reduces the operation time for data comparison and duplication checking, and improves the user experience.

[0117] Further, the application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor 22 to realize the steps of the above data duplication checking method.

[0118] It can be understood that if the method in the above embodiment is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and executes all or part of the steps of the method described in each embodiment of the application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and various program code storage media.

[0119] For the computer readable storage medium provided by the application, refer to the above method embodiment. The application will not be described here again. It has the same beneficial effects as the above data duplication checking method.

[0120] In order to make the technical personnel in the art better understand the technical solutions of the present application, the application will be further described in detail below in combination with the drawings, the data duplication checking method provided by the embodiments of the application, the data duplication checking device provided by the embodiments of the application and the computer readable storage medium.

[0121] Figure 4 The application scenario diagram of another data duplication checking device provided by the embodiments of the application includes:

[0122] The data type of the data to be written is determined, and then the corresponding bitmap key in the bitmap rule 31 and the database 33 for actually storing data are found. The bitmap data is generated in the bitmap generator 32 according to the bitmap key in the bitmap rule 31, and the bitmap data is saved in the redis list 34.

[0123] Due to the force majeure factors of different computing units of the bitmap rule 31 corresponding to the bitmap generator 32, the data to be written in the database 33 is generated into corresponding subscription data. The subscription data is input into the bitmap rule 31 according to the subscription field to generate a corresponding bitmap key, and further corresponding bitmap data is obtained in the bitmap generator 32 and saved to the redis list 34.

[0124] When data duplication checking is performed, the data to be checked is input into the SaaS data duplication checker to assemble parameters and determine the data type according to the data request parameters, wherein the SaaS data duplication checker 35 realizes the assembly logic of the request parameters. The corresponding bitmap key is found in the bitmap rule, and the bitmap data is queried in the redis through the parameters and the key of the bitmap. If the bitmap data does not exist, the data does not exist and there is no duplicate data is directly returned; if the bitmap data exists, it is routed to the specific database 33 for secondary verification. If the data exists, it is returned that the data already exists and there is duplicate data; if the data does not exist, it is returned that the data does not exist and there is no duplicate data.

[0125] It should be noted that the present embodiment can be applied to the business scenario of de-duplicating phone numbers in clues, and the bitmap rule 31 saves the bitmap keys defined by the business scenario of de-duplicating phone numbers in clues: bitmap_clue_phone and the specific database 33. The bitmap generator 32 defines the logic of bitmap generation, generates bitmap data and saves it to the redis list 34.

[0126] The application scenario diagram of the data duplication checking device provided by the embodiments of the application is introduced above, which has the same beneficial effects as the above-mentioned data duplication checking method.

[0127] The above describes in detail the method for data duplication checking, the device for data duplication checking and the medium provided by the present application. Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be understood by referring to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be understood by referring to the method part. It should be pointed out that, for ordinary skilled in the art, without departing from the principle of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the claims of the present application.

[0128] It should also be noted that in this specification, relationship terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for data deduplication, characterized in that, include: Obtain the data to be checked for plagiarism and determine the data type based on the request parameters corresponding to the data to be checked for plagiarism; wherein, in the SaaS scenario, the data type is determined in the SaaS data plagiarism checking module based on the data to be checked for plagiarism; the data to be checked for plagiarism consists of multiple data from multiple data sources and multiple dimensions, which are compared and queried simultaneously; The target key data corresponding to the data to be deduplicated is found in the bitmap rules according to the data type and the request parameters; the bitmap rules define the mapping rules between data types and key pairs in the bitmap in Redis and the relationship between the actual data storage of the bitmap in the database; Determine whether the target bitmap data corresponding to the target key data is found in the Redis list; If not, then it is determined that there is no duplicate data corresponding to the data to be deduplicated in the Redis list; If so, the data is routed to the database corresponding to the target bitmap data, and it is determined whether the database contains duplicate data corresponding to the data to be deduplicated; wherein, the data structure is queried through a routing table in the form of a hash table. If it exists, then it is determined that the duplicate data exists in the Redis list; If it does not exist, then it is determined that the duplicate data does not exist in the Redis list; The bitmap data of the Redis list is determined in the following way: Generate corresponding bitmap data based on the key data within the bitmap rules and save it to the Redis list; Alternatively, generate corresponding subscription data based on the data to be written in the database and determine the subscription data type corresponding to the subscription data; The corresponding key data is determined in the bitmap rules based on the subscription data type. Generate the corresponding bitmap data based on the key data and save it to the Redis list; Correspondingly, the key data in the bitmap rules is determined in the following way: Obtain the data to be written and determine the data type corresponding to the data to be written; The mapping rule within the bitmap rule is determined based on the data type, wherein the mapping rule is the data type and the rules of the database where the key data and the key data are stored; The key data corresponding to the data to be written is determined based on the relationship between the data type and the mapping rule; Correspondingly, after determining the bitmap data of the Redis list, the process also includes: The data to be written is saved to the database corresponding to the key data; Correspondingly, it also includes: When the query result in the Redis list is determined, a prompt message is output.

2. A data deduplication device, characterized in that, include: The first determining module is used to obtain the data to be checked for plagiarism and determine the data type according to the request parameters corresponding to the data to be checked for plagiarism; wherein, in the SaaS scenario, the data type is determined in the SaaS data plagiarism checking module according to the data to be checked for plagiarism; the data to be checked for plagiarism is multiple data from multiple data sources and multiple dimensions that are compared and queried simultaneously. The query module is used to find the target key data corresponding to the data to be deduplicated in the bitmap rules according to the data type and the request parameters; the bitmap rules define the mapping rules between data types and key pairs in the bitmap in Redis and the relationship between the actual data storage of the bitmap in the database; The first judgment module is used to determine whether the target bitmap data corresponding to the target key data is found in the Redis list. If not, the second determination module is triggered; if yes, the second judgment module is triggered. The second determining module is used to determine that there is no duplicate data corresponding to the data to be deduplicated in the Redis list; The second judgment module is used to route to the database corresponding to the target bitmap data and determine whether there is duplicate data in the database corresponding to the data to be deduplicated. If there is, the third determination module is triggered; if there is no duplicate data, the fourth determination module is triggered. The data structure is queried through a routing table in the form of a hash table. The third determining module is used to determine that the duplicate data exists in the Redis list; The fourth determining module is used to determine that the duplicate data does not exist in the Redis list; The bitmap data of the Redis list is determined in the following way: Generate corresponding bitmap data based on the key data within the bitmap rules and save it to the Redis list; Alternatively, generate corresponding subscription data based on the data to be written in the database and determine the subscription data type corresponding to the subscription data; The corresponding key data is determined in the bitmap rules based on the subscription data type. Generate the corresponding bitmap data based on the key data and save it to the Redis list; Correspondingly, the key data in the bitmap rules is determined in the following way: Obtain the data to be written and determine the data type corresponding to the data to be written; The mapping rule within the bitmap rule is determined based on the data type, wherein the mapping rule is the data type and the rules of the database where the key data and the key data are stored; The key data corresponding to the data to be written is determined based on the relationship between the data type and the mapping rule; Correspondingly, after determining the bitmap data of the Redis list, the process also includes: The data to be written is saved to the database corresponding to the key data; Correspondingly, it also includes: When the query result in the Redis list is determined, a prompt message is output.

3. A data deduplication device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data deduplication method as described in claim 1 when executing the computer program.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data deduplication method as described in claim 1.

Images