Data deduplication method and device, storage medium and electronic equipment

Abstract

The invention discloses a data deduplication method and device, a storage medium and electronic equipment, and relates to the technical field of computers. The method comprises the steps that a to-be-deduplicated data set is obtained, the to-be-deduplicated data set comprises first data and second data, the generation time of the first data is earlier than the generation time of the second data, and the first data and the second data have the same user identification; calculating a difference value between the generation time of the second data and the generation time of the first data; according to the difference value and a threshold value, determining unduplicated data in the first data and the second data; according to the generation time of the non-duplicated data, under the condition that it is determined that duplicated data exist in the third data, the duplicated data are deleted, the generation time of the third data is later than the generation time of the second data, and the third data and the first data have the same user identifier. According to the method and the device, the problems of data redundancy and storage space waste caused by excessive repeated or similar data stored by a big data platform can be solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data deduplication method, device, storage medium and electronic device. Background Art

[0002] At present, big data technology has been widely used in various fields. In some big data application scenarios, data has a certain time continuity. For example, in web browsing data, when a user browses the browser in the computing device, the computing device will upload the log data generated by the browser to the big data platform. The log data generated by the browser has a certain time continuity. If the user stays on the same web page for a long time or the user frequently refreshes the same web page due to network delays and other issues, the big data platform will store a lot of duplicate or similar data. Therefore, the big data platform needs to deduplicate the received data.

[0003] In order to solve the above problem, the data deduplication method currently used is to determine a preset number of keywords from each data received in a deduplication cycle, and determine whether there is data containing these keywords in other data received in the deduplication cycle. If so, the data containing these keywords in the deduplication cycle will be deleted. However, simple keyword deduplication cannot eliminate similar data, and there is still a large amount of data redundancy after deduplication, which wastes a lot of storage space. Summary of the invention

[0004] In view of the above problems, the present application provides a data deduplication method, device, storage medium and electronic device.

[0005] In order to solve the above technical problems, this application proposes the following solutions:

[0006] In a first aspect, the present application provides a data deduplication method, the method comprising: obtaining a data set to be deduplicated, the data set to be deduplicated comprising first data and second data, the generation time of the first data being earlier than the generation time of the second data, and the first data and the second data having the same user identifier; calculating the difference between the generation time of the second data and the generation time of the first data; determining the non-duplicate data in the first data and the second data based on the size of the difference and a threshold, the non-duplicate data being the data determined to be non-duplicate during the comparison process; deleting the duplicate data based on the generation time of the non-duplicate data if it is determined that duplicate data exists in the third data, the generation time of the third data being later than the generation time of the second data, and the third data having the same user identifier as the first data.

[0007] In combination with the first aspect, in one possible implementation, when the difference between the generation time of the second data and the generation time of the first data is less than or equal to a threshold, the first data is retained and the second data is deleted, and it is determined whether the third data included in the data set to be deduplicated is duplicate data based on the generation time of the first data, and after determining that the third data is duplicate data, the third data is deleted.

[0008] In combination with the first aspect, in a possible implementation method, when the difference between the generation time of the second data and the generation time of the first data is greater than a threshold, a key-value pair is generated based on the first data and the second data; the key-value pair is inserted into the database, and whether the second data is duplicate data is determined based on the return result of the database for the insertion operation; if the second data is not duplicate data, the first data and the second data are retained, and whether the third data is duplicate data is determined based on the generation time of the second data; if the second data is duplicate data, the first data is retained, the second data is deleted, whether the third data is duplicate data is determined based on the generation time of the first data, and after determining that the third data is duplicate data, the third data is deleted.

[0009] In combination with the first aspect, in another possible implementation manner, the generation time of the first data is used as the key of the key-value pair, and the data string included in the second data is used as the key value of the key-value pair.

[0010] In combination with the first aspect, in another possible implementation, when the returned result is equal to the number of key values ​​included in the key-value pair, it indicates that the second data is not duplicate data; when the returned result is not equal to the number of key values ​​included in the key-value pair, it indicates that the second data is duplicate data.

[0011] In combination with the first aspect, in another possible implementation method, when the difference between the generation time of the third data and the generation time of the second data is less than or equal to a threshold, the second data is retained and the third data is deleted; when the difference between the generation time of the third data and the generation time of the second data is greater than the threshold, a key-value pair is generated based on the second data and the third data; the key-value pair is queried from the database, and whether the third data is duplicate data is determined based on the returned result; wherein, generating the key-value pair based on the second data and the third data includes: using the generation time of the second data as the key of the key-value pair, and using the data string included in the third data as the key value of the key-value pair.

[0012] In combination with the first aspect, in another possible implementation method, data of the same user at different times in the data set to be deduplicated are aggregated into one piece of data according to the user identifier, and one piece of data includes multiple rows of data at different times; the multiple rows of data of the same user at different times in the aggregated data set to be deduplicated are sorted in chronological order; and the data that is sorted first among the multiple rows of data with different times is used as the first data.

[0013] In a second aspect, the present application provides a data deduplication device, the data deduplication device comprising:

[0014] An acquisition module is used to acquire a data set to be deduplicated, the data set to be deduplicated includes first data and second data, the generation time of the first data is earlier than the generation time of the second data, and the first data and the second data have the same user identifier;

[0015] A calculation module, used for calculating the difference between the generation time of the second data and the generation time of the first data;

[0016] A determination module, used for determining non-duplicate data in the first data and the second data according to the difference calculated in the calculation module and the threshold, wherein the non-duplicate data is data determined to be non-duplicate in the comparison process;

[0017] The deletion module is used to delete duplicate data according to the generation time of the non-duplicate data in the determination module when it is determined that the third data has duplicate data, the generation time of the third data is later than the generation time of the second data, and the third data has the same user identifier as the first data.

[0018] In combination with the second aspect, in a possible implementation method, the deletion module is specifically used to: when the difference between the generation time of the second data and the generation time of the first data is less than or equal to a threshold, retain the first data, delete the second data, and determine whether the third data included in the deduplicated data set is duplicate data based on the generation time of the first data, the third data has the same user identifier as the first data, and the generation time of the third data is later than the generation time of the second data.

[0019] In combination with the second aspect, in a possible implementation method, the deletion module is specifically used to: when the difference between the generation time of the second data and the generation time of the first data is greater than a threshold, generate a key-value pair based on the first data and the second data; insert the key-value pair into the database, and determine whether the second data is duplicate data based on the return result of the database for the insertion operation; if the second data is not duplicate data, retain the first data and the second data, and determine whether the third data is duplicate data based on the generation time of the second data; if the second data is duplicate data, retain the first data, delete the second data, and determine whether the third data is duplicate data based on the generation time of the first data.

[0020] In combination with the second aspect, in another possible implementation manner, the deletion module is specifically used to: use the generation time of the first data as the key of the key-value pair, and use the data string included in the second data as the key value of the key-value pair.

[0021] In combination with the second aspect, in another possible implementation method, the deletion module is specifically used to: when the returned result is equal to the number of key values ​​included in the key-value pair, indicate that the second data is not duplicate data; when the returned result is not equal to the number of key values ​​included in the key-value pair, indicate that the second data is duplicate data.

[0022] In combination with the second aspect, in another possible implementation method, the deletion module is specifically used to: when the difference between the generation time of the third data and the generation time of the second data is less than or equal to a threshold, retain the second data and delete the third data; when the difference between the generation time of the third data and the generation time of the second data is greater than a threshold, generate a key-value pair based on the second data and the third data; query the key-value pair from the database, and determine whether the third data is duplicate data based on the returned result; wherein, generating the key-value pair based on the second data and the third data includes: using the generation time of the second data as the key of the key-value pair, and using the data string included in the third data as the key value of the key-value pair.

[0023] In combination with the second aspect, in another possible implementation method, the deletion module is also used to: aggregate data of the same user at different times in the data set to be deduplicated into one piece of data according to the user identifier, wherein one piece of data includes multiple rows of data with different times; sort the multiple rows of data of the same user with different times in the aggregated data set to be deduplicated in chronological order; and take the data that is sorted first among the multiple rows of data with different times as the first data.

[0024] In order to achieve the above-mentioned purpose, according to the third aspect of the present application, a storage medium is provided, the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the data deduplication method of the above-mentioned first aspect.

[0025] In order to achieve the above-mentioned purpose, according to the fourth aspect of the present application, an electronic device is provided, which includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call the program instructions in the memory to execute the data deduplication method of the first aspect mentioned above.

[0026] By means of the above technical solution, the technical solution provided by this application has at least the following advantages:

[0027] The present application determines whether the second data is duplicate data based on the difference between the generation time of the first data and the generation time of the second data in the data set to be deduplicated, and performs deduplication processing on the second data when the second data is duplicate data. The present application can solve the problem that when a user stays on the same web page for a long time or frequently refreshes the same web page due to network delays, the big data platform stores a lot of duplicate or similar data, which in turn causes data redundancy and wastes storage space.

[0028] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0030] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown;

[0031] Figure 2 A schematic diagram of a process flow of a data deduplication method provided in an embodiment of the present application is shown;

[0032] Figure 3 A schematic diagram of a process flow of another data deduplication method provided in an embodiment of the present application is shown;

[0033] Figure 4 A structural schematic diagram of a data deduplication device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0035] In the embodiments of the present application, the terms "first" and "second" do not have a logical or temporal dependency, nor do they limit the quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another element.

[0036] In the embodiments of the present application, the term "at least one" means one or more, and the term "plurality" means two or more.

[0037] It should also be understood that the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined that ..." or "if [a stated condition or event] is detected" may be interpreted as "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0038] The embodiment of the present application also provides a data deduplication device, which can be used to execute the above-mentioned data deduplication method. Optionally, the data deduplication device can be an electronic device with data processing capability, or a functional module in the electronic device, which is not limited.

[0039] For example, the electronic device may be a server, which may be a single server, or a server cluster consisting of multiple servers. For another example, the electronic device may be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and a cellular phone, a personal digital assistant (PDA), an augmented reality (AR), a virtual reality (VR) device and other terminal devices. For another example, the electronic device may also be a recording device, a video surveillance device and other devices. The present application does not impose any special restrictions on the specific form of the electronic device.

[0040] The following takes the data deduplication device as an example, Figure 1 As shown, Figure 1 The present application provides a hardware structure of an electronic device 100.

[0041] like Figure 1 As shown, the electronic device 100 includes a processor 110 , a communication circuit 120 and a communication interface 130 .

[0042] Optionally, the electronic device 100 may further include a memory 140 . The processor 110 , the memory 140 and the communication interface 130 may be connected via a communication line 120 .

[0043] The processor 110 may be a central processing unit (CPU), a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 110 may also be any other device having a processing function, such as a circuit, a device, or a software module, without limitation.

[0044] In one example, the processor 110 may include one or more CPUs, such as Figure 1 CPU0 and CPU1 in.

[0045] As an optional implementation, the electronic device 100 includes multiple processors, for example, in addition to the processor 110, it may also include a processor 170. The communication line 120 is used to transmit information between the components included in the electronic device 100.

[0046] The communication interface 130 is used to communicate with other devices or other communication networks. The other communication networks may be Ethernet, Radio Access Network (RAN), Wireless Local Area Networks (WLAN), etc. The communication interface 130 may be a module, a circuit, a transceiver or any device capable of achieving communication.

[0047] The memory 140 is used to store instructions, where the instructions may be computer programs.

[0048] Among them, the memory 140 can be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage devices, etc., without limitation.

[0049] It should be noted that the memory 140 may exist independently of the processor 110 or may be integrated with the processor 110. The memory 140 may be used to store instructions or program codes or some data, etc. The memory 140 may be located inside the electronic device 100 or outside the electronic device 100, without limitation.

[0050] The processor 110 is used to execute the instructions stored in the memory 140 to implement the communication method provided in the following embodiments of the present application. For example, when the electronic device 100 is a terminal or a chip in a terminal, the processor 110 can execute the instructions stored in the memory 140 to implement the steps performed by the transmitting end in the following embodiments of the present application.

[0051] As an optional implementation, the electronic device 100 further includes an output device 150 and an input device 160. The output device 150 may be a device such as a display screen, a speaker, etc. that can output data of the electronic device 100 to a user. The input device 160 may be a device such as a keyboard, a mouse, a microphone, or a joystick that can input data to the electronic device 100.

[0052] It should be pointed out that Figure 1 The structure shown in the figure does not constitute a limitation on the computing device, except Figure 1 In addition to the components shown, the computing device may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0053] The data deduplication device and application scenario described in the embodiments of the present application are intended to more clearly illustrate the technical solution of the embodiments of the present application, and do not constitute a limitation on the technical solution provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of data deduplication devices and the emergence of new business scenarios, the technical solution provided in the embodiments of the present application is also applicable to similar technical problems.

[0054] Next, the data deduplication method is described in detail with reference to the accompanying drawings.

[0055] This application takes the deduplication of data in the Hive database as an example. The Hive database is a data warehouse analysis system built on Hadoop. It provides a rich SQL query method to analyze data stored in the Hadoop distributed file system. It can map structured data files to a database table and provide complete SQL query functions. It can convert SQL statements into MapReduce tasks to run. Through their own SQL query analysis, users who are not familiar with MapReduce can easily use the SQL language to query, summarize and analyze data. MapReduce developers can use their own mappers and reducers as plug-ins to support Hive databases for more complex data analysis. The Hive database provides a series of tools for data extraction, transformation and loading, which are used to store, query and analyze large-scale data sets stored in Hadoop, and supports user-defined functions (User-Defined Function, UDF), user-defined aggregate functions (User-Defined Aggregate Function, UDAF) and table generation functions (User-Defined Table-Generating Function, UDTF). It can also realize the customization of map and reduce functions, providing good scalability and extensibility for data operations.

[0056] In order to achieve dynamic deduplication of data in the Hive database, that is, the data deduplication method provided by this application, the data deduplication method provided by this application is registered in the Hive database in jar format, so that the Hive database can deduplicate the data according to the data deduplication method provided by this application. It should be noted that the information, data and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions.

[0057] Figure 2 A flowchart of a data deduplication method provided in this application. The method is applied to Figure 1 The data deduplication device of the hardware structure shown specifically includes the following steps:

[0058] Step 210: Obtain the data set to be deduplicated.

[0059] Taking a user browsing a web page as an example, a computing device obtains a data set to be deduplicated, and the obtained data set to be deduplicated includes log data generated by multiple users browsing the same or different web pages at different times. For example, the log data includes generation time (generation time of the log data), IP address (IP address can be used to indicate the geographical location where the log data is generated, and is used for network security and analysis), user agent (identification of the browser or other client application used by the user, which can provide information about the user's operating system, browser version and device type, etc.), access time (specific time when the user accesses the web page, which can be used to analyze access patterns and web traffic peaks), page access path (the order and path of pages browsed by the user on the website, which can be used to analyze user behavior and improve website navigation), etc.

[0060] Since this application is to deduplicate data in the Hive database, this application first performs a series of preprocessing on the data set to be deduplicated after obtaining it. The preprocessing steps include but are not limited to data cleaning, data standardization, data screening, consistency verification, and data integration.

[0061] Specifically, in the data integration phase, the table generating function (TGF) in the Hive database is used for processing. For example, according to the user ID, the data with different generation times belonging to the same user in the data set to be deduplicated are aggregated into one record. Such a record contains multiple rows of data with different generation times, thus providing preparation for subsequent deduplication operations. Furthermore, the aggregated data is sorted in chronological order to ensure that the data rows with different generation times of the same user are arranged in chronological order. The data set to be deduplicated after aggregation and sorting is shown in Table 1.

[0062] Table 1 Dataset to be deduplicated

[0063]

[0064] Since the data to be deduplicated after the above processing, the data of the same user ranked first is the data with the earliest generation time, and there can be no data earlier than this data in the data set to be deduplicated, the data is used as non-repetitive data, so that the computing device uses the non-repetitive data as a judgment standard to judge whether other data in the data set to be deduplicated is duplicate data. In an embodiment of the present application, the data ranked first (i.e., the earliest generation time) in the data row with different generation times is used as the first data. Taking Table 1 as an example, the data in the row where the user is identified as 1 and the generation time is 2024.1.1 is used as the first data. This is because after the data is sorted, this row of data has the earliest generation time and can be regarded as a standard for non-repetitive data. Next, the data in the row where the user is identified as 1 and the generation time is 2024.1.2 is used as the second data, and the second data is judged to be duplicate data based on the generation time of the first data. If the second data is judged to be duplicate data, the first data is retained in the data set to be deduplicated, the second data is deleted, and the third data is continued to be judged to be duplicate data based on the generation time of the first data. Specifically, the third data refers to the data in the row where the user ID is 1 and the generation time is 2024.1.3. Only after the second data is determined to be duplicate data and deleted, the generation time of the first data is used to determine whether the third data is duplicate data. If the second data is not duplicate data, the first data and the second data are retained, and the generation time of the second data is used to determine whether the third data is duplicate data. And so on, continue to process subsequent data.

[0065] In summary, step 210 obtains the data set to be deduplicated that includes the user's web browsing log data, integrates and sorts the data, aggregates the different time data of the same user and arranges them in chronological order, and then determines that the data with the earliest generation time is non-duplicate data, and uses this as a standard to determine whether subsequent data is duplicate data by comparing the generation time, thereby efficiently and accurately completing the deduplication processing of large-scale data sets, providing a reliable data foundation for subsequent data analysis and application.

[0066] Step 220: Calculate the difference between the generation time of the second data and the generation time of the first data.

[0067] In this step, the main task is to calculate the difference between the generation time of the second data and the generation time of the first data. This difference is crucial for the subsequent data deduplication process because it provides a quantitative standard for evaluating whether two sets of data may constitute duplication.

[0068] In some embodiments, it is necessary to extract the generation timestamps of the first data and the second data from the data set to be deduplicated. These timestamps are usually stored in a standard time format (such as ISO8601 or UNIX timestamp) to ensure consistency between different systems and platforms.

[0069] Next, the two timestamps are converted to the same basis so that the difference can be calculated. This calculation process may involve converting the timestamps into different time units such as seconds, minutes, hours, or days, depending on the actual needs.

[0070] Before or after performing the difference calculation, a time threshold needs to be set. This threshold is a key parameter that determines the time range in which two sets of data are considered "close enough" to be considered as possible duplicates. When setting this threshold, you need to carefully consider the characteristics of the data and the need for deduplication.

[0071] Once you have the difference and the threshold, you can make a judgment. If the calculated difference is less than or equal to the threshold, you may think that the two sets of data are repeated. On the contrary, if the difference is greater than the threshold, you tend to think that they are independent and non-repeated data.

[0072] In summary, calculating the difference in data generation time is an important step in the data deduplication process. It provides a quantitative standard to help more accurately evaluate whether two sets of data may constitute duplication. Through this step, data deduplication can be performed more effectively, improving data quality and availability.

[0073] Step 230: Determine the non-duplicate data in the first data and the second data according to the difference and the threshold.

[0074] In this step, the time difference calculated in the previous step 220 needs to be compared with a preset threshold to determine the non-duplicate data in the first data and the second data. It should be noted that the non-duplicate data refers to the data determined to be non-repetitive during the comparison process.

[0075] Specifically, the time difference calculated in step 220 is compared with a preset threshold value, which is set according to the characteristics of the data and the deduplication requirements, and represents an upper limit of a time range within which two sets of data are considered to be duplicates.

[0076] Secondly, if the time difference is less than or equal to a threshold, then the first data and the second data may be considered to be close enough in time that they may constitute a duplicate.

[0077] If the time difference is greater than the threshold, it is tended to be considered that the first data and the second data are not close in time, so it is preliminarily judged that the two sets of data may represent independent behaviors or records of the user at different time points. In this case, the two sets of data are temporarily regarded as non-duplicate data, which is a preliminary judgment based on the current comparison result and the preset threshold. The reason why it is called "temporary" is that this judgment is not absolute, but an inference based on the information and rules currently mastered. In practical applications, the repeatability of data may be affected by many factors, including but not limited to user behavior patterns, system performance, network delay, etc. Therefore, even if the time difference is greater than the threshold, it cannot completely rule out the possibility that the two sets of data may be regarded as duplicate data in a certain specific situation. For example, by comparing the return result with the number of key values ​​in the key-value pair, it is determined whether the second data is duplicate data. Specifically, when the return result is equal to the number of key values ​​in the key-value pair, it indicates that the second data is not duplicate data; and when the return result is not equal to the number of key values ​​in the key-value pair, it indicates that the second data is duplicate data. Therefore, temporarily regarding the data with a time difference greater than the threshold as non-duplicate data is a preliminary judgment based on current information and rules, which aims to provide a starting point for subsequent data processing and analysis. In actual operations, it is also necessary to conduct a more in-depth evaluation and processing of the repeatability of the data in combination with specific application scenarios and business needs.

[0078] The embodiment of the present application also needs to record or mark the weightless data so that it can be easily identified and used in subsequent data processing or analysis. This can be achieved by adding tags to the data set, storing it in different database tables or files, etc.

[0079] In summary, step 230 is an important part of the data deduplication process, which determines the non-duplicate data by comparing the time difference with the threshold, providing a basis for subsequent data processing and analysis. Through this step, duplicate data can be removed more effectively, improving the quality and availability of data.

[0080] Step 240: If it is determined that duplicate data exists in the third data according to the generation time of the non-duplicate data, delete the duplicate data.

[0081] In this step, the non-repetitive data and time determined in step 230 are used to evaluate whether the third data constitutes duplicate data. If duplicate data exists, a deletion operation is performed. This step pays special attention to the relative relationship between the generation time of the third data and the generation time of the second data, and whether the third data and the first data have the same user identifier.

[0082] Specifically, it is checked whether the third data contains the same user identification as the first data. The user identification is a key field for identifying data ownership, which ensures that different records generated by the same user can be accurately associated when comparing data.

[0083] Next, it is necessary to compare the generation time of the third data with the generation time of the non-duplicate data determined in step 230. Since the generation time of the third data is later than the generation time of the second data (this is a prerequisite for this step), the comparison in this step will mainly focus on the time relationship between the third data and the second data (if the second data is confirmed to be non-duplicate, i.e., non-duplicate data). If the third data has the same user identifier as the first data, and its generation time is very close to the generation time of the second data, then the third data may be regarded as duplicate data. Of course, if the second data is determined to be duplicate data and deleted in step 220, it is necessary to compare the first data with the third data.

[0084] It should be noted that before making a final judgment, in addition to the generation time and user identification, other factors may need to be considered, such as data content, data format, data integrity, etc. These factors help to more comprehensively evaluate whether the third data is indeed duplicate data. Specifically:

[0085] Data content: Compare the content of the third data with the existing data (such as the first or second data). If the main fields of the two (such as key information such as user ID, generation time, etc.) are exactly the same, but there are slight differences in non-key fields (such as some additional information or secondary attributes), then the third data can be considered to be essentially the same as the existing data, and the third data is considered duplicate data. It should be noted that the difference in checking non-key fields can be ignored because if the main information of the two data records (such as the user's behavior or the core content of the event) is consistent, but some descriptive or auxiliary information is slightly different, these differences usually do not affect the judgment of the repeatability of the data.

[0086] Data format: When evaluating whether the third-party data is a duplicate, you need to check whether the format of the third-party data is consistent with that of the existing data, including verifying that the number, type, and order of fields are the same, and whether there are special tags or codes to identify the source or processing method of the data. If the third-party data and the existing data are inconsistent in these aspects, it means that the data format is different. Specifically, you need to ensure that the third-party data contains the same number of fields as the existing data, and the data type of each field is consistent. For example, if the existing data contains date fields, numeric fields, and text fields, then the third-party data should also contain these fields and the types match. In addition, you need to check whether the order of the fields in the third-party data is consistent with the existing data. Even if the field content is the same, if the order is different, it may mean that the source or recording method of the data is different. Sometimes the data contains special tags or codes to identify the source or processing method of the data, which also need to be compared with the existing data to confirm consistency.

[0087] Data integrity: To assess whether the third-party data is complete, it is necessary to check whether it contains all necessary key fields and whether these fields have no obvious errors or omissions. If the third-party data lacks any key field (such as user ID, generation time, etc.), or has obvious errors such as incorrect date format, empty numeric fields, etc., the data cannot be regarded as a valid record and should not be simply classified as duplicate data.

[0088] By comprehensively considering these factors, we can more comprehensively evaluate whether the third-party data is indeed duplicate data, thereby improving the accuracy and effectiveness of data deduplication.

[0089] When the third data is confirmed to be duplicate data, a deletion operation needs to be performed. This usually means removing the record from the database or marking it as "deleted" or "duplicate" in the data set so that it can be ignored in subsequent processing. After deleting duplicate data, this operation should be recorded and a report may be generated for auditing or to inform relevant parties of the deduplication results. This helps ensure transparency and traceability of data processing.

[0090] In summary, step 240 is a key step in the data deduplication process, which uses information such as non-duplicate data and generation time, combined with user identification and other factors, to evaluate and process the duplication of the third data. Through this step, the accuracy and consistency of the data set can be further ensured.

[0091] In the data deduplication method provided by the present application, through the data deduplication process provided by the above steps 210-240, the non-duplicate data in the first data and the second data with the same user identifier and a chronological order of generation time is first identified through the time difference calculation and threshold comparison mechanism. On this basis, the method further utilizes the generation time information of the determined non-duplicate data, combined with the third data having the same user identifier as the first data and the relatively later generation time of the third data, to accurately evaluate the repeatability of the third data. Once it is confirmed that the third data is duplicate data, a deletion operation is performed, thereby effectively removing redundant information in the data set to be deduplicated. This method not only improves the efficiency and accuracy of data processing, but also ensures the simplicity and consistency of the data set, providing a high-quality data foundation for subsequent data analysis and utilization.

[0092] Furthermore, in view of the fact that the operation of "determining whether the third data is duplicate data according to the generation time of the non-duplicate data" in step 240 is highly dependent on the accurate identification of the non-duplicate data in the first data and the second data, only by accurately determining the non-duplicate data can the accuracy of the repeatability judgment of the third data be ensured. Therefore, the following will elaborate on the specific process of how to screen and determine the non-duplicate data in the first data and the second data in the data set to be deduplicated, and then how to use these non-duplicate data to accurately determine whether the third data is duplicate data.

[0093] Case 1: when the difference between the generation time of the second data and the generation time of the first data is less than or equal to the threshold.

[0094] When a user browses a web page, there is a situation where he stays on a web page for a long time, or repeatedly refreshes a web page due to network delays and other reasons. At this time, the log data generated by the browser is duplicate data. Therefore, in the embodiment of the present application, setting the "threshold of generation time" is a flexible parameter. The threshold of generation time can be 2 seconds or 3 seconds, or it can be one minute or two minutes. For example, for applications that are sensitive to network delays, the threshold may be set to a few seconds; and for user behavior analysis, the threshold may be relaxed to a few minutes or even longer. In the implementation of the present application, the threshold of generation time is not specifically limited, and the above example is only used to explain the present application. Since the present application has pre-processed the deduplication data set in step 210, multiple rows of data with different generation times having the same user identifier (i.e., the same user) are sorted, and the data ranked first in the multiple rows of data with different generation times is used as the first data. That is, the data of the row where the same user has the earliest generation time is used as the first data (no duplicate data, i.e., data without duplicate data). The second data has the same user identifier as the first data, and the generation time of the second data is later than the generation time of the first data. Therefore, the present application determines whether the second data is duplicate data based on the relationship between the difference between the generation time of the second data and the generation time of the first data and the threshold of the generation time.

[0095] When the difference between the generation time of the second data and the generation time of the first data is less than or equal to a threshold, it indicates that the generation time of the second data and the generation time interval of the first data are extremely small. In this case, the present application believes that the user stays on a certain web page, or repeatedly refreshes a certain web page due to network delays, etc., and the second data is considered to be duplicate data.

[0096] When the second data is duplicate data, the first data is retained in the data set to be deduplicated, and the second data is deleted.

[0097] Further, whether the third data included in the to-be-deduplicated data set is duplicate data is determined according to the generation time of the first data. The third data has the same user identifier as the first data and the second data, and the generation time of the third data is later than the generation time of the second data.

[0098] Specifically, since the first data is the data with the earliest generation time of the same user (i.e., non-duplicate data, which represents the user's first behavior or record), the first data is chosen to be retained as a representative. Correspondingly, although the second data may contain information similar to the first data, it is regarded as duplicate data and deleted because its generation time is immediately after the first data and the difference is within the threshold range. After completing the deduplication processing of the first data and the second data, it is necessary to evaluate whether other data (such as the third data) in the data set to be deduplicated is also duplicate data. Here, the same method as that for judging the second data is adopted, that is, comparing whether the difference between the generation time of the third data and the generation time of the first data is less than or equal to the threshold. If the conditions are met, it is considered that the third data is also a duplicate behavior generated within the time period represented by the first data, so the third data should also be regarded as duplicate data and deleted.

[0099] This logic can be further extended to other data in the data set to be deduplicated, such as the fourth data. For each new set of data (such as the fourth data), the above steps will be repeated, that is, the difference between its generation time and the generation time of the previously retained non-duplicate data (in this case, the first data) will be compared to determine whether it is duplicate data. If the difference between the generation time of the fourth data and the generation time of the first data is also within the threshold range, the fourth data is also regarded as duplicate data and deleted.

[0100] In summary, in case one, by comparing the difference in data generation time with a preset threshold, effective identification and removal of duplicate data is achieved. Specifically, when the difference between the generation time of the second data and the generation time of the first data is less than or equal to the preset threshold, the system retains the earliest generated first data and deletes the subsequently generated duplicate data (such as the second data). Then, subsequent data (such as the third data) is evaluated according to the same principle, and data that meets the duplication conditions is continued to be judged and removed. This method not only effectively reduces redundant information in the data set, improves the purity of the data and the accuracy of the analysis results, but also simplifies the data preprocessing process, thereby ensuring the uniqueness and accuracy of the data in the data set.

[0101] Case 2: when the difference between the generation time of the second data and the generation time of the first data is greater than a threshold.

[0102] The following is combined with Figure 3 A method for processing the second data in the data set to be deduplicated when the difference between the generation time of the second data and the generation time of the first data is greater than a threshold is described in detail. Specifically, the following steps are included:

[0103] Step 310: Generate a key-value pair according to the first data and the second data.

[0104] Because when judging whether the second data is a duplicate of the existing data, it is assumed that the first data is unique (i.e., non-duplicate). Therefore, the generation time of the first data is used as the key of the key-value pair, and the relevant data string in the second data is used as the value of the key-value pair. This approach helps to efficiently identify duplicate items in the data set to be deduplicated. By using the unique generation time of the first data as the key, the relevant data record can be quickly located, and by comparing the data string of the second data, it can be determined whether it is duplicate data. This mechanism ensures that duplicate data can be quickly and accurately identified even in large-scale data sets, thereby improving the efficiency and accuracy of data processing while maintaining the consistency and integrity of the data set.

[0105] Specifically, when processing two sets of data (such as the first set of data and the second set of data) and checking whether there are duplications between them, it is usually assumed that all entries in the first set of data are unique (i.e., there are no duplications). In this scenario, the generation time of each record in the first set of data is used as a unique identifier (i.e., key), and then for each day's record in the second set of data, it is checked whether its generation time already exists in the record. If it exists, it means that this record in the second set of data may be a duplicate; if it does not exist, its related data (i.e., value) is stored under this generation time (i.e., key). Simply put: the generation time of the first set of data is used as the "key"; the related information of the second set of data is used as the "value". Taking "the first data is the data in the row where the user identifier is 1 and the generation time is 2024.1.1 in the above Table 1, and the second data is the data in the row where the user identifier is 1 and the generation time is 2024.1.2 in the above Table 1" as an example, the key of the key-value pair is 2024.1.1, and the key values ​​of the key-value pair are A2, B2, and C2, that is, the key-value pair is (2024.1.1, A2, B2, C2).

[0106] Step 320: insert the key-value pair into the database, and determine whether the second data is duplicate data according to the return result of the database for the insertion operation.

[0107] In one embodiment, a key-value pair is inserted into a Redis database. The Redis database is a high-performance key-value database. It supports storing multiple value types, such as string, list, set, zset, and hash.

[0108] Specifically, the key-value pairs generated according to the first data and the second data are inserted into the database through the insertion function SADD(). The characteristic of the insertion function SADD() is that when inserting one or more key-value pairs into the database, if the key-value pair already exists in the database, the inserted key-value pair will be considered as a duplicate key-value pair by the database, and the duplicate key-value pair will be ignored by the database, and the key-value pair will not be inserted into the database again. The insertion function SADD() only inserts key-value pairs that do not currently exist in the database. The return value of the insertion function SADD() is the number of key values ​​included in the key-value pairs inserted into the database. In an embodiment of the present application, the return value of the insertion function SADD() is the number of data strings included in the second data.

[0109] Based on this, the present application determines whether the second data is duplicate data according to whether the return value of the insertion function SADD() is equal to the number of key values ​​included in the key-value pair. When the return result of the database for the insertion operation (i.e., the return value of the insertion function SADD()) is equal to the number of key values ​​included in the key-value pair, it indicates that the second data is not duplicate data. When the return result of the database for the insertion operation is not equal to the number of key values ​​included in the key-value pair, it indicates that the second data is duplicate data.

[0110] For example, assuming that the key-value pair generated according to the first data and the second data is (2024.1.1, A2, B2, C2), then the insert operation SADD (2024.1.1, A2, B2, C2) is executed in the Redis database, that is, three data strings, namely A2, B2, and C2, are inserted into the data set with the time identifier of 2024.1.1 in the database. If these three data strings do not exist in the data set with the time identifier of 2024.1.1, all these three data strings are inserted into the data set with the time identifier of 2024.1.1, and a return result is generated, and the return result at this time is 3. Since the data string included in the key value of the key-value pair generated according to the first data and the second data is 3, the return result of the database for the insert operation is equal to the number of key values ​​included in the key-value pair. Therefore, it is determined that the second data is not duplicate data. If one or more of these three data strings already exist in the data set with the time identifier 2024.1.1, the duplicate data strings among the three data strings are discarded, and the non-duplicate data strings are inserted into the data set with the time identifier 2024.1.1, and a return result is generated. At this time, the return result of the database for the insertion operation is less than the number of key values ​​included in the key-value pair. Therefore, it can be determined that the second data is duplicate data based on the return result.

[0111] In general, whether the second data is duplicate data is determined by the return result of the database insert operation. When the result returned by the insert operation is equal to the number of key values ​​in the key-value pair, this means that all the data in the key-value pair is new and has never existed in the database, so it can be determined that the second data is not duplicate data. On the contrary, if the returned result is not equal to the number of key values ​​in the key-value pair, it means that at least one key value already exists in the database, so it can be determined that the second data is duplicate data. This method can efficiently verify the uniqueness of the data, avoid redundant information in the data set, improve the accuracy and efficiency of data processing, and also ensure the reliability of subsequent data analysis.

[0112] Regarding step 320, in addition to judging whether the second data is duplicate data according to the return result of the data for the insert operation (such as success or failure, the number of records returned, etc.), the following implementation methods may also be considered to implement:

[0113] Implementation method 1: Matching degree judgment based on database query results.

[0114] In this implementation, before inserting a key-value pair into the database, the database is first queried based on the generation time of the first data (as the key) to see if the same key exists. If the query result returns the same key as the generation time of the first data, but the corresponding value (i.e., the data string of the second data) is not exactly the same as the second data to be inserted, further comparison or matching calculation is required. A matching threshold is set to determine whether the second data is duplicate data by comparing the similarity of the data string of the second data with the existing data in the database (such as using a string similarity algorithm).

[0115] For example, suppose that a key-value pair (2024.1.1,"A1,B1,C1") already exists in the database, and now there is a key-value pair (2024.1.1,"A1,B2,C1") to be inserted. Although the keys are the same, the values ​​are not exactly the same. At this time, the string similarity algorithm can be used to calculate the similarity between "A1,B1,C1" and "A1,B2,C1". If the similarity exceeds the set threshold (such as 90%), the second data is considered to be duplicate data; otherwise, it is considered not to be duplicate data.

[0116] Implementation method 2: Hash value comparison based on data features.

[0117] In this embodiment, the data string of the second data is hashed to generate a unique hash value. Before inserting the key-value pair into the database, the database is first queried to see whether there is a record with the same hash value. If there is an identical hash value, the second data is likely to be duplicate data; if not, the second data is not duplicate data.

[0118] For example, assuming that the data string of the second data is "A2, B2, C2", a hash value "HASH_A2B2C2" is obtained by performing a hash process on it. Query the database to see whether there is a record with a hash value of "HASH_A2B2C2". If so, it indicates that the second data is duplicate data; if not, the second data is not duplicate data.

[0119] It should be noted that the above embodiments may need to be adjusted and optimized in actual applications in combination with specific data characteristics and deduplication requirements. At the same time, in order to ensure the accuracy and cancellation rate of data deduplication, factors such as database performance and concurrent processing need to be considered. In addition, in step 320, other types of databases (such as MySQL, PostgreSQL, etc.) or storage structures (such as NoSQL databases, distributed storage, etc.) can also be considered to store and query key-value pairs to adapt to different application scenarios and data scales.

[0120] Step 330: If the second data is duplicate data, retain the first data, delete the second data, and determine whether the third data is duplicate data based on the generation time of the first data.

[0121] Similar to the above situation, when the second data is duplicate data, the first data is retained in the data set to be deduplicated, and the second data is deleted.

[0122] Further, whether the third data included in the to-be-deduplicated data set is duplicate data is determined according to the generation time of the first data. The third data has the same user identifier as the first data and the second data, and the generation time of the third data is earlier than the generation time of the second data.

[0123] Specifically, determine whether the difference between the generation time of the third data and the generation time of the first data is less than or equal to a threshold value. If so, it indicates that the third data is also within the time range corresponding to the first data that is considered to be duplicate data, and the third data is also duplicate data. Determine whether the fourth data included in the data set to be deduplicated is duplicate data based on the generation time of the first data. The fourth data has the same user identifier as the first data, and the generation time of the fourth data is later than the generation time of the third data. The method for determining whether the fourth data is duplicate data is the same as that for the second data and the third data, and will not be repeated here.

[0124] If the difference between the generation time of the third data and the generation time of the first data is greater than the threshold, it indicates that the third data is not within the time range corresponding to the first data that is considered to be duplicate data, and it is necessary to further determine whether the third data is duplicate data. The specific method for further determining whether the third data is duplicate data is the same as the specific method for further determining whether the second data is duplicate data, and will not be repeated here.

[0125] Step 340: If the second data is not duplicate data, retain the first data and the second data, and determine whether the third data is duplicate data based on the generation time of the second data.

[0126] If the second data is not duplicate data, then both the first data and the second data in the deduplication data set need to be retained. At this time, the data in the deduplication data set that may be duplicated with the first data has been deleted within the preset time range. Therefore, when determining duplicate data for other data in the deduplication data set, the second data is used as a reference. That is, whether the third data is duplicate data is determined based on the generation time of the second data.

[0127] In one embodiment, when the difference between the generation time of the third data and the generation time of the second data is less than or equal to a threshold, the second data is retained and the third data is deleted. When the difference between the generation time of the third data and the generation time of the second data is greater than the threshold, further judgment is made on whether the third data is duplicate data. That is, a key-value pair is generated according to the second data and the third data, wherein the generation time of the second data is used as the key of the key-value pair, and the data string included in the third data is used as the key value of the key-value pair. The key-value pair is queried from the database, and whether the third data is duplicate data is determined according to the returned result. If the third data is duplicate data, the third data is deleted from the data set to be deduplicated, and whether the fourth data is duplicate data is determined according to the generation time of the second data. If the third data is not duplicate data, the third data is retained in the data set to be deduplicated, and whether the fourth data is duplicate data is determined according to the generation time of the third data, until all the data in the data set to be deduplicated is deduplicated. The fourth data has the same user identifier as the first data, and the generation time of the fourth data is later than the generation time of the third data. The method for determining whether the fourth data is duplicate data is the same as that of the second data and the third data, and will not be repeated here.

[0128] In summary, in this embodiment, the repeatability of data is distinguished by setting a time threshold. When the difference between the generation time of the third data and the generation time of the second data is less than or equal to the preset threshold, the system retains the second data and deletes the third data because the two are considered to be duplicates. On the contrary, when the difference is greater than the threshold, the system generates a key-value pair based on the second data and the third data, and queries the key-value pair through the database to determine whether the third data is duplicate data. This method can ensure the uniqueness and accuracy of data in the data set, reduce data redundancy, improve the efficiency and reliability of data processing, and provide a high-quality foundation for subsequent data analysis. By using the generation time of the second data as the key of the key-value pair and the data string of the third data as the key value of the key-value pair, the ability to accurately judge the repeatability of data is further enhanced.

[0129] In summary, in case 2, by setting a time threshold and combining database operations to verify whether the data is a duplicate record, it not only relies on the time difference to judge the data repeatability, but also further confirms it through the feedback of the database insertion operation, thereby avoiding misjudgment. When the difference between the generation time of the second data and the generation time of the first data is greater than the preset threshold, the system generates a key-value pair and attempts to insert it into the database. Only when the result returned by the database shows that the key-value pair already exists, the second data is determined to be duplicate data, thereby avoiding the risk of mistakenly deleting non-duplicate data. Through the above mechanism, the system can more accurately distinguish the repeatability of data, ensure the purity of the data set, and improve the accuracy of data deduplication and the efficiency of overall data management. Combining the generation time difference of the first data and the second data to judge the repeatability, and further confirming the data with a time difference greater than the threshold through the key-value pair, the accuracy of data deduplication is improved, data redundancy is effectively reduced, storage space is saved, and the quality and management efficiency of the data set are ensured.

[0130] The present application determines whether the second data is duplicate data based on the difference between the generation time of the first data and the generation time of the second data in the data set to be deduplicated, and deduplicates the second data when it is duplicate data. The present application can solve the problem that users stay on the same web page for a long time or users frequently refresh the same web page due to network delays, causing the big data platform to store a lot of duplicate or similar data, thereby causing data redundancy and wasting storage space. The present application further judges the data with a time difference greater than a threshold based on the key-value pair, which can further improve the accuracy of data deduplication.

[0131] It is understandable that in order to implement the functions in the above embodiments, the computer device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0132] Furthermore, as a response to the above Figure 2 The implementation of the method embodiment shown in the figure, the embodiment of the present application provides a data deduplication device, which is used to deduplicate data in a database. The embodiment of the device corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will no longer repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. Specifically, Figure 4 As shown, the data deduplication device 400 includes:

[0133] An acquisition module 410 is used to acquire a data set to be deduplicated, the data set to be deduplicated includes first data and second data, the generation time of the first data is earlier than the generation time of the second data, and the first data and the second data have the same user identifier;

[0134] A calculation module 420, configured to calculate a difference between a generation time of the second data and a generation time of the first data;

[0135] A determination module 430, configured to determine non-duplicate data in the first data and the second data according to the difference calculated in the calculation module 420 and the threshold, wherein the non-duplicate data is data determined to be non-duplicate in the comparison process;

[0136] The deletion module 440 is used to delete duplicate data according to the generation time of the non-duplicate data in the determination module 430 when it is determined that the third data has duplicate data, the generation time of the third data is later than the generation time of the second data, and the third data has the same user identifier as the first data.

[0137] Further, such as Figure 4 As shown, the deletion module 440 is specifically used for: when the difference between the generation time of the second data and the generation time of the first data is less than or equal to a threshold, retaining the first data, deleting the second data, and judging whether the third data included in the data set to be deduplicated is duplicate data according to the generation time of the first data, the third data has the same user identifier as the first data, and the generation time of the third data is later than the generation time of the second data.

[0138] Further, such as Figure 4 As shown, the deletion module 440 is specifically used for: when the difference between the generation time of the second data and the generation time of the first data is greater than a threshold, generating a key-value pair according to the first data and the second data; inserting the key-value pair into the database, and determining whether the second data is duplicate data according to the return result of the database for the insertion operation; if the second data is not duplicate data, retaining the first data and the second data, and judging whether the third data is duplicate data according to the generation time of the second data; if the second data is duplicate data, retaining the first data, deleting the second data, and judging whether the third data is duplicate data according to the generation time of the first data.

[0139] Further, such as Figure 4 As shown, the deletion module 440 is specifically used to: use the generation time of the first data as the key of the key-value pair, and use the data string included in the second data as the key value of the key-value pair.

[0140] Further, such as Figure 4As shown, the deletion module 440 is specifically used to: when the returned result is equal to the number of key values ​​included in the key-value pair, indicate that the second data is not duplicate data; when the returned result is not equal to the number of key values ​​included in the key-value pair, indicate that the second data is duplicate data.

[0141] Further, such as Figure 4 As shown, the deletion module 440 is specifically used for: when the difference between the generation time of the third data and the generation time of the second data is less than or equal to a threshold, retaining the second data and deleting the third data; when the difference between the generation time of the third data and the generation time of the second data is greater than a threshold, generating a key-value pair according to the second data and the third data; querying the key-value pair from the database, and determining whether the third data is duplicate data according to the returned result; wherein, generating the key-value pair according to the second data and the third data includes: using the generation time of the second data as the key of the key-value pair, and using the data string included in the third data as the key value of the key-value pair.

[0142] Further, such as Figure 4 As shown, the deletion module 440 is further used to aggregate data of the same user at different times in the data set to be deduplicated into one piece of data according to the user identifier, wherein one piece of data includes multiple rows of data at different times.

[0143] Further, such as Figure 4 As shown, the deletion module 440 is also used to: sort multiple rows of data with different times of the same user in the aggregated data set to be deduplicated in chronological order; and take the first-ranked data among the multiple rows of data with different times as the first data.

[0144] An embodiment of the present application provides a storage medium on which a program is stored, and when the program is executed by a processor, a data deduplication method is implemented.

[0145] An embodiment of the present application provides a processor, which is used to run a program, wherein a data deduplication method is executed when the program is running.

[0146] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that has the following method steps: obtaining a data set to be deduplicated, the data set to be deduplicated includes first data and second data, the generation time of the first data is earlier than the generation time of the second data, and the first data and the second data have the same user identifier; calculating the difference between the generation time of the second data and the generation time of the first data; determining the non-duplicate data in the first data and the second data according to the difference and the threshold, the non-duplicate data is the data determined to be non-duplicate in the comparison process; judging whether the third data is duplicate data according to the generation time of the non-duplicate data, and deleting the duplicate data after the duplicate data exists, the generation time of the third data is later than the generation time of the second data, and the third data has the same user identifier as the first data. Further, when the difference between the generation time of the second data and the generation time of the first data is less than or equal to the threshold, retain the first data, delete the second data, judge whether the third data included in the data set to be deduplicated is duplicate data according to the generation time of the first data, and delete the third data after determining that the third data is duplicate data.

[0147] Furthermore, when the difference between the generation time of the second data and the generation time of the first data is greater than a threshold, a key-value pair is generated based on the first data and the second data; the key-value pair is inserted into the database, and whether the second data is duplicate data is determined based on the return result of the database for the insertion operation; if the second data is not duplicate data, the first data and the second data are retained, and whether the third data is duplicate data is determined based on the generation time of the second data; if the second data is duplicate data, the first data is retained, the second data is deleted, and whether the third data is duplicate data is determined based on the generation time of the first data, and after determining that the third data is duplicate data, the third data is deleted.

[0148] Further, the generation time of the first data is used as the key of the key-value pair, and the data string included in the second data is used as the key value of the key-value pair.

[0149] Further, when the returned result is equal to the number of key values ​​included in the key-value pair, it indicates that the second data is not duplicate data; when the returned result is not equal to the number of key values ​​included in the key-value pair, it indicates that the second data is duplicate data.

[0150] Furthermore, when the difference between the generation time of the third data and the generation time of the second data is less than or equal to a threshold, the second data is retained and the third data is deleted; when the difference between the generation time of the third data and the generation time of the second data is greater than a threshold, a key-value pair is generated based on the second data and the third data; the key-value pair is queried from the database, and whether the third data is duplicate data is determined based on the returned result; wherein, generating the key-value pair based on the second data and the third data includes: using the generation time of the second data as the key of the key-value pair, and using the data string included in the third data as the key value of the key-value pair.

[0151] Furthermore, data at different times of the same user in the data set to be deduplicated are aggregated into one piece of data according to the user identifier, wherein one piece of data includes multiple rows of data at different times; the multiple rows of data at different times of the same user in the aggregated data set to be deduplicated are sorted in chronological order; and the data that is sorted first among the multiple rows of data with different times is used as the first data.

[0152] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] In a typical configuration, the device includes one or more processors (CPU), memory and bus. The device may also include input / output interface, network interface and the like.

[0154] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0155] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0156] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0157] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0158] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data deduplication method, characterized in that: The method comprises: Acquire a data set to be deduplicated, the data set to be deduplicated includes first data and second data, generation time of the first data is earlier than generation time of the second data, and the first data and the second data have the same user identifier; Calculating a difference between a generation time of the second data and a generation time of the first data; Determine, according to the difference and the threshold, non-duplicate data in the first data and the second data, wherein the non-duplicate data is data determined to be non-duplicate during the comparison process; According to the generation time of the non-duplicate data, when it is determined that the third data has duplicate data, the duplicate data is deleted, the generation time of the third data is later than the generation time of the second data, and the third data has the same user identifier as the first data.

2. The method according to claim 1, characterized in that The deleting the duplicate data when it is determined that the third data has duplicate data according to the generation time of the non-duplicate data includes: When the difference between the generation time of the second data and the generation time of the first data is less than or equal to a threshold, the first data is retained and the second data is deleted. It is determined whether the third data included in the to-be-deduplicated data set is duplicate data based on the generation time of the first data, and after determining that the third data is duplicate data, the third data is deleted.

3. The method according to claim 1, characterized in that The deleting the duplicate data when it is determined that the third data has duplicate data according to the generation time of the non-duplicate data includes: When the difference between the generation time of the second data and the generation time of the first data is greater than a threshold, generating a key-value pair according to the first data and the second data; Inserting the key-value pair into a database, and determining whether the second data is duplicate data according to a return result of the database for the insertion operation; If the second data is not duplicate data, retain the first data and the second data, and determine whether the third data is duplicate data according to the generation time of the second data; If the second data is duplicate data, retain the first data, delete the second data, determine whether the third data is duplicate data based on the generation time of the first data, and delete the third data after determining that the third data is duplicate data.

4. The method according to claim 3, characterized in that Generating a key-value pair according to the first data and the second data includes: The generation time of the first data is used as the key of the key-value pair, and the data character string included in the second data is used as the key value of the key-value pair.

5. The method according to claim 3, characterized in that: Determining whether the second data is duplicate data according to the returned result includes: When the returned result is equal to the number of key values ​​included in the key-value pair, it indicates that the second data is not duplicate data; When the returned result is not equal to the number of key values ​​included in the key-value pair, it indicates that the second data is duplicate data.

6. The method according to claim 3, characterized in that Judging whether the third data is duplicate data according to the generation time of the second data includes: When the difference between the generation time of the third data and the generation time of the second data is less than or equal to a threshold, retaining the second data and deleting the third data; When the difference between the generation time of the third data and the generation time of the second data is greater than a threshold, generating a key-value pair according to the second data and the third data; Query the key-value pair from the database, and determine whether the third data is duplicate data according to the returned result; The step of generating a key-value pair according to the second data and the third data includes: The generation time of the second data is used as the key of the key-value pair, and the data character string included in the third data is used as the key value of the key-value pair.

7. The method according to claim 1, characterized in that The method further comprises: aggregating data of the same user at different times in the to-be-deduplicated data set into one piece of data according to the user identifier, wherein the one piece of data includes multiple rows of data at different times; Sort multiple rows of data with different times for the same user in the aggregated data set to be deduplicated in chronological order; The data that is ranked first among the multiple rows of data with different times is used as the first data.

8. A data deduplication device, characterized in that: The device comprises: An acquisition module, configured to acquire a data set to be deduplicated, the data set to be deduplicated comprising first data and second data, the generation time of the first data is earlier than the generation time of the second data, and the first data and the second data have the same user identifier; a calculation module, used for calculating the difference between the generation time of the second data and the generation time of the first data; A determination module, configured to determine non-duplicate data in the first data and the second data according to the difference calculated in the calculation module and a threshold, wherein the non-duplicate data is data determined to be non-duplicate during the comparison process; A deletion module is used to delete the duplicate data according to the generation time of the non-duplicate data in the determination module when it is determined that the third data has duplicate data, the generation time of the third data is later than the generation time of the second data, and the third data has the same user identifier as the first data.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the data deduplication method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The device includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the data deduplication method as described in any one of claims 1-7.

Images