Data deduplication method and device, equipment, medium and product

Abstract

The invention relates to a data deduplication method and device, equipment, a medium and a product. The method comprises the steps of obtaining current batch data from a CDN log, wherein the current batch data comprises multiple pieces of original data; analyzing the original data to obtain analysis data corresponding to the original data, the analysis data including dimension data, measurement index data, processing time and position data; and under the condition that the processing interval time between the current batch of data and the previous batch of data is greater than the preset batch time granularity, performing de-duplication processing on the analysis data based on the dimension data, the measurement index data, the processing time and the position data to obtain de-duplication data. By adopting the method, the problem of repeated consumption data can be avoided.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data deduplication method, apparatus, device, medium and product. Background Technology

[0002] CDN (Content Delivery Network) is a technology that accelerates content delivery by deploying multiple servers across the internet. With a CDN, end users can quickly access various web pages, images, and video resources directly from cached edge nodes, avoiding direct access to the origin server. During the user's access process, CDN logs are generated on the server. These logs can be analyzed to reveal information such as user traffic, number of accesses, accessed domains, status codes, and ISP information.

[0003] In traditional technology, when processing CDN logs through a stream computing engine, Kafka is used to consume the raw data in the CDN logs in batches, obtain the processing data and consumption location of each batch, and record the processing data and consumption location in the database.

[0004] However, traditional data deduplication methods suffer from the problem of consuming data repeatedly. Summary of the Invention

[0005] Therefore, it is necessary to provide a data deduplication method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can avoid duplicate data consumption in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a data deduplication method, including:

[0007] Retrieve the current batch of data from CDN logs. The current batch of data includes multiple raw data sets.

[0008] The original data is parsed to obtain the parsed data corresponding to each original data. The parsed data includes dimensional data, metric data, processing time and location data.

[0009] If the processing interval between the current batch of data and the previous batch of data is greater than the preset batch time granularity, the parsed data is deduplicated based on dimensional data, metric data, processing time and location data to obtain deduplicated data.

[0010] In one embodiment, the method further includes:

[0011] Query the last processing time of the previous batch of data to get the query results;

[0012] If the query result is a query failure, find the earliest processing time from the multiple processing times corresponding to the current batch of data to obtain the first processing time, and find the latest processing time from the multiple processing times corresponding to the previous batch of data to obtain the second processing time.

[0013] The processing interval is obtained by performing calculations based on the first processing time and the second processing time.

[0014] In one embodiment, the method further includes:

[0015] If the query result is successful, the processing interval is calculated based on the first processing time and the last processing time.

[0016] In one embodiment, the parsed data is deduplicated based on dimensional data, metric data, processing time and location data to obtain deduplicated data, including:

[0017] The abnormal time interval is obtained based on the first processing time, the second processing time, and the final processing time;

[0018] Based on the abnormal time interval and location data, at least one duplicate parsed data is obtained from multiple parsed data.

[0019] The duplicate parsed data is deduplicated by using dimensional data and metric data to obtain duplicate data.

[0020] In one embodiment, duplicate parsed data is deduplicated using dimensional data and metric data to obtain deduplicated data, including:

[0021] For each duplicate parsed data, the metric data in the duplicate parsed data are inverted one by one to obtain the updated parsed data;

[0022] Multiple parsed data and the updated parsed data are merged according to their dimensions to obtain deduplicated data.

[0023] In one embodiment, the abnormal time interval includes the start time and stop time of the abnormal data;

[0024] Based on the first processing time, the second processing time, and the final processing time, the abnormal time interval is obtained, including:

[0025] The start time of the abnormal data is obtained by summing one of the processing times, the second processing time and the last processing time, with the batch time granularity.

[0026] The first processing time is used as the stop time for abnormal data.

[0027] Secondly, this application also provides a data deduplication device, comprising:

[0028] The data acquisition module is used to obtain the current batch of data from CDN logs. The current batch of data includes multiple raw data.

[0029] The data parsing module is used to parse and process the raw data to obtain the parsed data corresponding to each raw data. The parsed data includes dimensional data, metric data, processing time and location data.

[0030] The data deduplication module is used to deduplicatize parsed data based on dimensional data, metric data, processing time and location data when the processing interval between the current batch of data and the previous batch of data is greater than the preset batch time granularity.

[0031] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the data deduplication method as described in the first aspect.

[0032] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data deduplication method as described in the first aspect.

[0033] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the data deduplication method as described in the first aspect.

[0034] The aforementioned data deduplication methods, devices, equipment, media, and products utilize dimensional data, metric data, processing time, and location data obtained from parsing the original data for analysis. Compared to existing technologies that only record the latest consumption location, these methods effectively eliminate duplicate data caused by abnormal batches by comprehensively recording and processing dimensional data, metric data, processing time, and location data. This ensures the accuracy of statistical results during CDN log processing even when batches are discontinuous or abnormal compensation occurs, thus preventing duplicate data consumption. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1This is a flowchart illustrating a data deduplication method in one embodiment;

[0037] Figure 2 This is a flowchart illustrating the steps for obtaining the processing interval time in one embodiment;

[0038] Figure 3 This is a flowchart illustrating the steps for obtaining deduplicated data in one embodiment;

[0039] Figure 4 This is a flowchart illustrating the data deduplication method in another embodiment;

[0040] Figure 5 This is a structural block diagram of a data deduplication device in one embodiment;

[0041] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0044] In practical applications, statistical analysis of CDN logs can yield relevant billing metrics, such as bandwidth, traffic, and request count. These metrics are used by CDN providers to settle billing with their customers. Inaccurate metric values ​​can lead to discrepancies in billing and settlement between CDN providers and customers, potentially causing losses and disputes regarding financial income and expenses. To avoid the problem of duplicate metric values ​​in CDN log statistical analysis caused by stream computing engine malfunctions or program errors, this application provides a data deduplication method to ensure that data obtained from CDN logs is deduplicated and unique before being sent to the database.

[0045] In one exemplary embodiment, such as Figure 1As shown, a data deduplication method is provided. This embodiment illustrates the method applied to a server; however, it is understood that the method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0046] Step 102: Obtain the current batch data from the CDN logs. The current batch data includes multiple raw data sets.

[0047] The current batch of data extracted from CDN logs can be divided into two categories: dimensions and metrics. Dimensions are used to classify data, such as domain names, status codes, and carriers. Metrics are used to calculate specific numerical indicators and can perform mathematical operations, such as access traffic and access counts.

[0048] Step 104: Parse and process each original data to obtain the parsed data corresponding to each original data. The parsed data includes dimensional data, metric data, processing time and location data.

[0049] The parsing process includes extracting multiple dimensional fields from various raw data sets, such as customer identification ID, access domain name, request path, business line, and regional information, to form the dimensional data of the current data record. Subsequently, statistical calculations are performed on the measurable fields in the raw data sets, such as traffic values, request counts, and downlink byte counts, to obtain metric data. Furthermore, the processing time corresponding to these raw data sets is marked as the processing time of the current batch, and the data is written to a location record as location data.

[0050] In some embodiments, the processing time proc_time and location data offset_info corresponding to each piece of raw data are recorded in data table A.

[0051] For example, the structure of data table A can be represented as follows:

[0052] proc_time offset_info

[0053] procTime1 offsetInfo1

[0054] procTime2 offsetInfo1 ... ...

[0056] procTimeN offsetInfoN

[0057] In some embodiments, a data table B is created to record start and end times. Data table B contains three fields: batch identifier (e.g., UUID, Universally Unique Identifier), start time (start_time), and the end processing time (last_proc_time) for the current batch of data. When parsing and processing the raw data, a record is inserted into data table B to record the batch identifier and start time (start_time). After successfully completing the parsing of the current batch of data and stopping the operation, the last_proc_time field is updated in the record corresponding to the batch identifier in database table B. If the operation exits abnormally due to an error or other reason, the last_proc_time field is empty and no update is performed.

[0058] Step 106: If the processing interval between the current batch of data and the previous batch of data is greater than the preset batch time granularity, deduplication is performed on the parsed data based on dimensional data, metric data, processing time and location data to obtain deduplicated data.

[0059] The batch time granularity is used to specify the maximum allowable interval between the processing times of each batch under normal circumstances. If the comparison results show that the processing time difference between the current batch of data and the previous batch of data is greater than the preset batch time granularity, it indicates that the processing may have experienced abnormal restarts, data compensation, or batch processing blockages, leading to the risk of duplicate sending or processing. In this case, a deduplication process for parsing the current batch of data needs to be initiated. For example, the batch time granularity can be 1 minute.

[0060] In the deduplication process, based on the dimension data, metric data, processing time, and location data carried by each parsed data, the system judges and processes potentially duplicate data records. For duplicate data records, the system does not change their dimension fields, but instead performs specific deduplication processing logic on the metric data to obtain deduplicated data.

[0061] The above data deduplication method utilizes the dimensional data, metric data, processing time and location data obtained from parsing the original data for analysis. Compared with the existing technology that only records the latest consumption location, it can effectively eliminate duplicate data caused by abnormal batches by comprehensively recording and processing dimensional data, metric data, processing time and location data. This ensures that the statistical results remain correct even when batches are discontinuous or abnormal compensation occurs during the CDN log processing, and avoids duplicate data consumption.

[0062] In one exemplary embodiment, based on Figure 1 The illustrated embodiments, such as Figure 2As shown, the method also includes:

[0063] Step 202: Query the last processing time of the previous batch of data to obtain the query results.

[0064] In conjunction with the aforementioned embodiments, the last processing time of the previous batch of data is the last_proc_time corresponding to the previous batch of data in data table B. If the queried value is empty, the query result is a query failure; if a specific value is found, the result is a query success.

[0065] Step 204: If the query result is a query failure, find the earliest processing time from the multiple processing times corresponding to the current batch of data to obtain the first processing time, and find the latest processing time from the multiple processing times corresponding to the previous batch of data to obtain the second processing time.

[0066] The query failure indicates that the program that parsed and processed the previous batch of data did not exit normally. You can find the earliest processing time among the multiple processing times corresponding to the current batch of data in data table A to obtain the first processing time procTime1; and find the latest processing time among the multiple processing times corresponding to the previous batch of data in data table A to obtain the time difference of the second processing time procTime2.

[0067] It is understandable that the first processing time procTime1 and the second processing time procTime2 are two adjacent time records in data table A.

[0068] Step 206: Perform calculations based on the first processing time and the second processing time to obtain the processing interval time.

[0069] The processing interval is the time difference between the first and second processing times. If the time difference is greater than the batch time granularity, it indicates that the batch is not continuous and there may be duplicate data transmissions. In this case, deduplication processing is required on the parsed data corresponding to the current batch.

[0070] In one possible implementation, the method further includes:

[0071] Step 208: If the query result is successful, perform calculations based on the first processing time and the last processing time to obtain the processing interval time.

[0072] In this embodiment, the processing interval between processing the previous batch of data and the current batch of data is determined by querying the last processing time of the previous batch of data, thereby locating the interval where duplicate data may exist and effectively avoiding duplicate data consumption.

[0073] In one exemplary embodiment, such as Figure 3 As shown, the parsed data is deduplicated based on dimensional data, metric data, processing time, and location data, resulting in deduplicated data, including:

[0074] Step 302: Obtain the abnormal time interval based on the first processing time, the second processing time, and the final processing time.

[0075] In one possible implementation, the abnormal time interval includes the abnormal data start time (startTime) and the abnormal data stop time (endTime). Step 302 may further include: summing one of the processing times, the second processing time and the end processing time, with the batch time granularity to obtain the abnormal data start time (startTime); and using the first processing time as the abnormal data stop time (endTime).

[0076] In the process of calculating the abnormal data start time (startTime), if the query result for the last processing time of the previous batch of data is successful, the last processing time and the batch time granularity are summed to obtain the abnormal data start time (startTime); if the query result is unsuccessful, the second processing time and the batch time granularity are summed to obtain the abnormal data start time (startTime).

[0077] In some embodiments, the abnormal time interval can be represented as [abnormal data start time startTime, abnormal data stop time endTime].

[0078] Step 304: Based on the abnormal time interval and location data, at least one duplicate parsed data is obtained from multiple parsed data.

[0079] In order to accurately locate the starting consumption position of the duplicate segment, the position data corresponding to the startTime of the abnormal data in the current batch of data can be queried, and the data consumption can start from the data at that position, and the parsed data whose timestamp is within the abnormal time interval is regarded as the duplicate parsed data.

[0080] For example, Kafka's `offsetsForTimes` function can be used to calculate the closest position data for each partition near the `startTime` of the abnormal data start time. Then, parsing data is pulled from this position data, and the timestamp of each parsed data is checked. Since the timestamp parsing data is written to Kafka at the time of writing, it can be used to accurately determine whether a message belongs to a duplicate sending interval.

[0081] Step 306: Use dimensional data and metric data to deduplicate the repeated parsed data to obtain deduplicated data.

[0082] In one possible implementation, step 306 may further include: for each duplicate parsed data, inverting the metric data in the duplicate parsed data one by one to obtain updated parsed data; merging multiple parsed data and updated parsed data according to dimensional data to obtain deduplicated data.

[0083] For example, for parsed data T1, where W1, W2, W3, W4 are dimensional data, and D1, D2, D3 are metric data, parsed data T1 can be represented as:

[0084] W1 W2 W3 W4 D1 D2 D3

[0085] With the dimensional data remaining unchanged, and the metric data inverted one by one, the updated parsed data T2 is obtained, which can be represented as:

[0086] W1 W2 W3 W4 -D1 -D2 -D3

[0087] The updated parsed data T2 and parsed data T1 are aggregated according to their dimensions, and the same metrics are accumulated:

[0088] W1 W2 W3 W4 D1-D1 D2-D2 D3-D3 = W1 W2 W3 W4 0 0 0

[0089] Ultimately, this achieves the effect of eliminating duplicate data and ensuring message uniqueness.

[0090] In this embodiment, the interval where duplicate data may exist is located based on the batch processing time, and deduplication is performed by inverting the metric, thereby eliminating duplicate data, ensuring message uniqueness, realizing the self-recovery capability of the CDN log batch data processing link in the event of abnormal exit, thus ensuring the integrity, consistency and reliability of the data processing link, and significantly enhancing the fault tolerance and maintainability of abnormal scenarios.

[0091] In one exemplary embodiment, such as Figure 4 As shown, a data deduplication method is provided, which includes:

[0092] Step 401: Obtain the current batch data from the CDN logs. The current batch data includes multiple raw data sets.

[0093] Step 402: Parse and process each original data to obtain the parsed data corresponding to each original data. The parsed data includes dimensional data, metric data, processing time and location data.

[0094] Step 403: Query the last processing time of the previous batch of data to obtain the query results.

[0095] Step 404: If the query result is successful, perform calculations based on the first processing time and the last processing time to obtain the processing interval time.

[0096] Step 405: If the query result is a query failure, find the earliest processing time from the multiple processing times corresponding to the current batch of data to obtain the first processing time, and find the latest processing time from the multiple processing times corresponding to the previous batch of data to obtain the second processing time.

[0097] Step 406: Perform calculations based on the first processing time and the second processing time to obtain the processing interval time.

[0098] Step 407: If the processing interval between the current batch of data and the previous batch of data is greater than the preset batch time granularity, sum one of the processing times of the second processing time and the last processing time with the batch time granularity to obtain the start time of abnormal data in the abnormal time interval; and use the first processing time as the stop time of abnormal data in the abnormal time interval.

[0099] Step 408: Based on the abnormal time interval and location data, at least one duplicate parsed data is obtained from multiple parsed data.

[0100] Step 409: For each duplicate parsed data, invert the metric data in the duplicate parsed data one by one to obtain the updated parsed data.

[0101] Step 410: Merge multiple parsed data and the updated parsed data according to dimensional data to obtain deduplicated data.

[0102] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0103] Based on the same inventive concept, this application also provides a data deduplication apparatus for implementing the data deduplication method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data deduplication apparatus embodiments provided below can be found in the limitations of the data deduplication method described above, and will not be repeated here.

[0104] In one exemplary embodiment, such as Figure 5 As shown, a data deduplication device is provided, including: a data module 502, a data parsing module 504, and a data deduplication module 506, wherein:

[0105] The data acquisition module 502 is used to obtain the current batch of data from CDN logs. The current batch of data includes multiple raw data.

[0106] The data parsing module 504 is used to parse and process each piece of raw data to obtain the parsed data corresponding to each piece of raw data. The parsed data includes dimensional data, metric data, processing time and location data.

[0107] The data deduplication module 506 is used to deduplicatize parsed data based on dimensional data, metric data, processing time and location data when the processing interval between the current batch of data and the previous batch of data is greater than the preset batch time granularity.

[0108] In one embodiment, the data deduplication module 506 is further used to query the last processing time of the previous batch of data to obtain the query result; if the query result is a query failure, it searches for the earliest processing time from the multiple processing times corresponding to the current batch of data to obtain the first processing time, and searches for the latest processing time from the multiple processing times corresponding to the previous batch of data to obtain the second processing time; and performs calculations based on the first processing time and the second processing time to obtain the processing interval time.

[0109] In one embodiment, the data deduplication module 506 is further configured to perform calculations based on the first processing time and the last processing time to obtain the processing interval time when the query result is a successful query.

[0110] In one embodiment, the data deduplication module 506 is further configured to obtain an abnormal time interval based on the first processing time, the second processing time, and the end processing time; based on the abnormal time interval and location data, filter at least one duplicate parsed data from multiple parsed data; and use dimensional data and metric data to perform deduplication processing on the duplicate parsed data to obtain deduplicated data.

[0111] In one embodiment, the data deduplication module 506 is further configured to, for each duplicate parsed data, invert the metric data in the duplicate parsed data one by one to obtain updated parsed data; and merge multiple parsed data and updated parsed data according to dimensional data to obtain deduplicated data.

[0112] In one embodiment, the abnormal time interval includes the abnormal data start time and the abnormal data stop time; the data deduplication module 506 is also used to sum one of the processing times of the second processing time and the end processing time with the batch time granularity to obtain the abnormal data start time; and the first processing time is used as the abnormal data stop time.

[0113] Each module in the aforementioned data deduplication device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0114] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores batch data, parsed data, and deduplicated data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a data deduplication method.

[0115] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0116] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0117] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0118] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0119] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0120] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0122] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data deduplication method, characterized in that, The method includes: Obtain the current batch of data from CDN logs; the current batch of data includes multiple raw data sets. The original data are parsed to obtain parsed data corresponding to each original data. The parsed data includes dimensional data, metric data, processing time and location data. If the processing interval between the current batch of data and the previous batch of data is greater than the preset batch time granularity, the parsed data is deduplicated based on the dimension data, metric data, processing time and location data to obtain deduplicated data.

2. The method according to claim 1, characterized in that, The method further includes: Query the last processing time of the previous batch of data to obtain the query results; If the query result is a query failure, the earliest processing time is found from the multiple processing times corresponding to the current batch of data to obtain the first processing time, and the latest processing time is found from the multiple processing times corresponding to the previous batch of data to obtain the second processing time. The processing interval time is obtained by performing calculations based on the first processing time and the second processing time.

3. The method according to claim 2, characterized in that, The method further includes: If the query result is successful, the processing interval time is obtained by performing calculations based on the first processing time and the last processing time.

4. The method according to claim 3, characterized in that, The process of deduplicating the parsed data based on the dimensional data, metric data, processing time, and location data to obtain deduplicated data includes: The abnormal time interval is obtained based on the first processing time, the second processing time, and the end processing time; Based on the abnormal time interval and the location data, at least one duplicate parsed data is obtained by filtering from multiple parsed data. The duplicate parsed data is deduplicated using the dimensional data and the metric data to obtain the deduplicated data.

5. The method according to claim 4, characterized in that, The process of deduplicating the duplicate parsed data using the dimensional data and the metric data to obtain the deduplicated data includes: For each of the repeated parsed data, the metric data in the repeated parsed data are inverted one by one to obtain the updated parsed data; The multiple parsed data and the updated parsed data are merged according to the dimensional data to obtain the deduplicated data.

6. The method according to claim 4, characterized in that, The abnormal time interval includes the start time and stop time of the abnormal data; The step of obtaining the abnormal time interval based on the first processing time, the second processing time, and the end processing time includes: The start time of the abnormal data is obtained by summing one of the processing times, the second processing time and the last processing time, with the batch time granularity. The first processing time is taken as the stop time for the abnormal data.

7. A data deduplication device, characterized in that, The device includes: The data acquisition module is used to acquire the current batch of data from CDN logs, wherein the current batch of data includes multiple raw data. The data parsing module is used to parse and process the original data to obtain parsed data corresponding to each original data. The parsed data includes dimension data, metric data, processing time and location data. The data deduplication module is used to perform deduplication processing on the parsed data based on the dimension data, metric data, processing time and location data when the processing interval between the current batch of data and the previous batch of data is greater than the preset batch time granularity, so as to obtain deduplicated data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images