Two-stage data deduplication method and system without damaging data timeliness

Abstract

The invention provides a two-stage data deduplication method and system without data timeliness, and the method comprises the steps: obtaining a source end real-time data stream, dividing the source end real-time data stream according to a time window, a time identifier and an object identifier, and enabling the data of the same time window to be subjected to data deduplication; dividing the data with the same object identifier and the same space identifier in the same time window into the same group to obtain multiple groups of data to be subjected to duplicate removal; performing pre-de-duplication processing on multiple groups of data to be de-duplicated through two de-duplication paths, sending the data to a service calculation module for service calculation, and synchronizing the data subjected to pre-de-duplication to a real-time data stream at a target end; and performing post-merging processing on the multiple groups of data after the business calculation, and writing a result into a target end data table. According to the method, the data deduplication is divided into two stages, namely the pre-deduplication stage and the post-merging stage, and the pre-deduplication is carried out through two paths, so that the size of a deduplication time window is not limited on the premise of lossless data timeliness, and streaming computing resources and database storage resources can be effectively reduced.

Description

Technical Field

[0001] This application relates to the fields of data processing and computer technology, specifically to a two-stage data deduplication method and system for lossless data timeliness. Background Technology

[0002] In big data computing and storage scenarios, data deduplication is a crucial step. To simplify the data processing, deduplication is typically performed first to remove duplicate data before further processing. The quality of deduplication directly impacts the utilization of streaming computing and database storage resources. A typical streaming computing workflow is as follows: Figure 1 As shown, the input source real-time data stream 1 and the data deduplication step 2 not only provide users with a simpler data presentation in the business calculation step 3, but also reduce the investment of subsequent computing resources and reduce the data storage cost of the target data table 4. However, in the data deduplication step 2, a time window is required for data deduplication, which means that a certain amount of time needs to be waited before the deduplicated data can be handed over to the subsequent target real-time data stream 5, affecting the timeliness of subsequent streaming computing.

[0003] Existing data deduplication solutions often improve the deduplication rate by extending the deduplication time window. However, this approach comes at the cost of sacrificing the timeliness of the data; the longer the deduplication time window, the worse the timeliness of the data tends to be.

[0004] Patent document CN120030006A discloses a data deduplication method, apparatus, device and storage medium. Based on the Flink streaming computing framework, it calculates hash values ​​for real-time advertising behavior data to deduplicate the data. It improves the processing efficiency of data deduplication without the need to introduce external storage devices, but it does not specifically address the timeliness of the data.

[0005] Patent document CN120144572A discloses a method and system for deduplicating bidding information data based on multi-feature fusion. The focus is on effectively identifying and eliminating duplicate records in the bidding information data, but there is no specific consideration for the timeliness of the data.

[0006] Patent document CN119938655A discloses a data deduplication method, device and electronic device, which effectively reduces the amount of useless duplicate data during the training of large language models, without having to pay attention to the timeliness of the data;

[0007] Patent document CN119597737A discloses a data deduplication method, device, storage medium and electronic device, which aims to solve the problem of big data platforms storing too much duplicate or similar data, causing data redundancy and wasting storage space, without paying attention to the timeliness of data;

[0008] It is evident that current technologies rarely consider timeliness when deduplicating data.

[0009] Therefore, in order to improve the utilization of streaming computing resources and database storage resources, we need to perform data deduplication without compromising data timeliness. This application proposes a two-stage data deduplication method and system that does not compromise data timeliness. Summary of the Invention

[0010] This application addresses one or more technical deficiencies in the prior art by proposing the following technical solution.

[0011] Based on the first aspect of this application, a two-stage data deduplication method for lossless data timeliness is proposed, including:

[0012] S1: Obtain the real-time data stream from the source end, divide the real-time data stream from the source end according to the time window, time identifier, and object identifier, divide the data in the same time window, as well as the data with the same object identifier and spatial identifier in the same time window into the same group, and obtain multiple groups of data to be deduplicated;

[0013] S2: Record the count of the first data in each group of data to be deduplicated as 1 and the count of the last data as N(o,s,t). Perform deduplication processing on the multiple groups of data to be deduplicated and send them to the business calculation module for business calculation.

[0014]

[0015] Where N(o,s,t) represents the number of times the last data point is read, and t represents the time window [t start , t end Business hours within ], o and o i Represents object identifiers, s and s i I represents the spatial identifier, and I represents the indicator function.

[0016] S3: Perform post-merge processing on multiple sets of data after business calculations and write the results into the target data table.

[0017] This application divides the data deduplication process into two stages: pre-deduplication and post-merging. Without compromising the timeliness of the data, it achieves effective deduplication of real-time data streams, thereby reducing the resource consumption of streaming computing and database storage.

[0018] Furthermore, the process of performing pre-deduplication processing on the multiple sets of data to be deduplicated and sending them to the business computing module includes:

[0019] First deduplication path: Retain the first and last records of each group of data to be deduplicated, and send them to the business calculation module; or...

[0020] The second deduplication path: only the first data in each group of data to be deduplicated is kept to obtain the first deduplicated dataset and it is immediately sent to the business computing module. At the same time, each group of data to be deduplicated is copied and only the last data is kept. Other duplicate data is removed to obtain the second deduplicated dataset and it is then sent to the business computing module.

[0021] This application provides two forward deduplication paths, which enhance the flexibility and adaptability of the system. The first forward deduplication path is suitable for scenarios with high requirements for data integrity, while the second forward deduplication path further optimizes the allocation of computing resources through replication and distribution, and can ensure the timeliness of data.

[0022] Furthermore, in the first deduplication path, the first piece of data is immediately sent to the business calculation module for full business calculation, and the last piece of data is sent to the business calculation module before the next time window opens for incremental business calculation.

[0023] This step ensures data timeliness while also achieving reasonable distribution of computational tasks, thus improving the efficiency of streaming computing.

[0024] Furthermore, in the second deduplication path, the time window sizes of the first deduplication dataset and the second deduplication dataset are adjusted according to the computing resource usage.

[0025] This step can dynamically adjust the time windows of the two deduplicated datasets based on the usage of computing resources, realizing resource-aware adaptive scheduling and further optimizing the system's performance and stability.

[0026] Furthermore, after step S2 and before step S3, the process also includes synchronizing the first piece of data that has completed business calculations with the real-time data stream of the target end.

[0027] This step ensures that downstream systems obtain the data deduplication results as early as possible, enabling real-time decision-making and further improving the visibility and timeliness of data at the target end.

[0028] Furthermore, the post-merging process and writing the results into the target data table specifically includes removing the first data entry from each group and writing the last data entry from each group into the target data table.

[0029] This step effectively avoids data redundancy, reduces storage costs, and ensures the accuracy and consistency of deduplication results.

[0030] Based on the second aspect of this application, a two-stage data deduplication system for lossless data timeliness is also proposed, including a pre-deduplication module, a post-merging module, and a business calculation module;

[0031] The deduplication module is used to deduplicatize the data to be deduplicated, the merging module is used to merge the deduplicated data, and the business calculation module is used to perform business calculations.

[0032] The system acquires the real-time data stream from the source end and divides the real-time data stream from the source end according to the time window, time identifier, and object identifier. Data in the same time window, as well as data with the same object identifier and spatial identifier in the same time window, are grouped into the same group to obtain multiple groups of data to be deduplicated.

[0033] The data to be deduplicated is input into the deduplication module for deduplication, including retaining the first and last data entries of each group of data to be deduplicated, and then sent to the business calculation module; or...

[0034] Only the first data in each group of data to be deduplicated is kept to obtain the first deduplicated dataset and is immediately sent to the business computing module. At the same time, each group of data to be deduplicated is copied and only the last data is kept. Other duplicate data are removed to obtain the second deduplicated dataset and then sent to the business computing module.

[0035] After performing business calculations, the business calculation module synchronizes the data to the target end's real-time data stream and sends it to the post-merging module. The post-merging module removes the first data entry and writes the last data entry into the target end's data table.

[0036] Based on a third aspect of this application, a computer program product is also proposed, having one or more computer programs thereon that, when executed by a computer processor, perform the method as described in any of the above.

[0037] The technical advantages of this application are as follows: This application performs two-stage data deduplication without compromising data timeliness, without limiting the size of the deduplication time window, and reduces the amount of network data in the tens or hundreds of billions to 20%-25% of the original amount through deduplication. It is suitable for deduplication of massive data, and can effectively reduce streaming computing resources and database storage resources, thereby achieving cost reduction and efficiency improvement. Attached Figure Description

[0038] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0039] Figure 1 This is a flowchart of a data deduplication method for streaming computing based on existing technology.

[0040] Figure 2 This is a flowchart of a two-stage data deduplication method for lossless data timeliness provided according to an embodiment of this application.

[0041] Figure 3This is a block diagram of a two-stage data deduplication system for lossless data timeliness provided according to an embodiment of this application.

[0042] Figure 4 This is one of the embodiment diagrams of a two-stage data deduplication system for lossless data timeliness provided according to an embodiment of this application.

[0043] Figure 5 This is a second embodiment of a two-stage data deduplication system for lossless data timeliness provided according to an embodiment of this application.

[0044] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application. Detailed Implementation

[0045] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0046] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] Figure 2 This application illustrates a two-stage data deduplication method for lossless data timeliness, comprising:

[0048] S1: Obtain the real-time data stream from the source end, divide the real-time data stream from the source end according to the time window, time identifier, and object identifier, divide the data in the same time window, as well as the data with the same object identifier and spatial identifier in the same time window into the same group, and obtain multiple groups of data to be deduplicated;

[0049] S2: Record the count of the first data in each group of data to be deduplicated as 1 and the count of the last data as N(o,s,t). Perform deduplication processing on the multiple groups of data to be deduplicated and send them to the business calculation module for business calculation.

[0050]

[0051] Where N(o,s,t) represents the number of times the last data point is read, and t represents the time window [t start , t end Business hours within ], o and o i Represents object identifiers, s and s i I represents the spatial identifier, and I represents the indicator function.

[0052] S3: Perform post-merge processing on multiple sets of data after business calculations and write the results into the target data table.

[0053] It should be noted that this application divides the data deduplication process into two stages: pre-deduplication and post-merging. Without compromising the timeliness of the data, it achieves effective deduplication of real-time data streams, which can reduce the resource consumption of streaming computing and database storage.

[0054] It should be noted that the deduplication process before sending the multiple sets of data to be deduplicated to the business computing module includes:

[0055] First deduplication path: Retain the first and last records of each group of data to be deduplicated, and send them to the business calculation module; or...

[0056] The second deduplication path: only the first data in each group of data to be deduplicated is kept to obtain the first deduplicated dataset and it is immediately sent to the business computing module. At the same time, each group of data to be deduplicated is copied and only the last data is kept. Other duplicate data is removed to obtain the second deduplicated dataset and it is then sent to the business computing module.

[0057] It should be noted that this application provides two deduplication paths, which enhance the flexibility and adaptability of the system. The first deduplication path is suitable for scenarios with high requirements for data integrity, while the second deduplication path further optimizes the allocation of computing resources through replication and distribution, and can ensure the timeliness of data.

[0058] It should be noted that in the first deduplication path, the first piece of data is immediately sent to the business calculation module for full business calculation, and the last piece of data is sent to the business calculation module before the next time window opens for incremental business calculation.

[0059] It should be noted that this step, while ensuring data timeliness, also achieves reasonable distribution of computational tasks, thereby improving the efficiency of streaming computing.

[0060] It should be noted that in the second deduplication path, the time window sizes of the first deduplication dataset and the second deduplication dataset are adjusted according to the computing resource usage.

[0061] It should be noted that this step can dynamically adjust the time windows of the two deduplicated datasets based on the usage of computing resources, realizing resource-aware adaptive scheduling and further optimizing the performance and stability of the system.

[0062] It should be noted that after step S2 and before step S3, the process also includes synchronizing the first piece of data that has completed business calculations with the real-time data stream of the target end.

[0063] It should be noted that this step ensures that downstream systems obtain the data deduplication results as early as possible, enabling real-time decision-making and further improving the visibility and timeliness of the data at the target end.

[0064] It should be noted that the post-merging process and writing the results into the target data table specifically includes removing the first data entry of each group and writing the last data entry of each group into the target data table.

[0065] It should be noted that this step effectively avoids data redundancy, reduces storage costs, and ensures the accuracy and consistency of the deduplication results.

[0066] The following is for reference. Figure 3 It demonstrates a two-stage data deduplication system for lossless data timeliness, including a pre-deduplication module a, a post-merging module b, and a business calculation module c;

[0067] The deduplication module a is configured to perform deduplication on the data to be deduplicated, the merging module b is configured to merge the deduplicated data, and the business calculation module c is configured to perform business calculations.

[0068] The system acquires the real-time data stream from the source end and divides the real-time data stream from the source end according to the time window, time identifier, and object identifier. Data in the same time window, as well as data with the same object identifier and spatial identifier in the same time window, are grouped into the same group to obtain multiple groups of data to be deduplicated.

[0069] The data to be deduplicated is input into the deduplication module a for deduplication, including retaining the first and last data entries of each group of data to be deduplicated, and then sent to the business calculation module c; or...

[0070] Only the first data in each group of data to be deduplicated is kept to obtain the first deduplicated dataset and is immediately sent to the business computing module c. At the same time, each group of data to be deduplicated is copied and only the last data is kept. Other duplicate data are removed to obtain the second deduplicated dataset and then sent to the business computing module c.

[0071] After performing business calculations, the business calculation module c synchronizes the data to the target end's real-time data stream and sends it to the post-merging module b. The post-merging module b removes the first data entry and writes the last data entry into the target end's data table.

[0072] It should be noted that this application performs two-stage data deduplication without compromising data timeliness, without limiting the size of the deduplication time window. Through deduplication, the amount of network data in the tens or hundreds of billions can be reduced to 20%-25% of the original amount. It is suitable for deduplication of massive data and can effectively reduce streaming computing resources and database storage resources, thereby achieving cost reduction and efficiency improvement.

[0073] In a specific embodiment, the source-side real-time data stream (such as...) Figure 4 401 and Figure 5 Data is generated as shown in Table 1 (501). The deduplication time window is set to five minutes. The object identifier and spatial identifier of the data jointly determine whether a piece of data is unique. If the object identifier and spatial identifier of multiple pieces of data are the same, they are duplicate data. The first to fifth pieces of data in Table 1 have common object identifiers and spatial identifiers and are therefore duplicate data.

[0074] A new deduplication time window is opened every 5 minutes during the business hours. For example, the data with a business time of "8:06-8:08" in Table 1 belongs to a new time window. If the spatial identifier changes within the time window, the data will also be divided into another new time window for processing. For example, the data with a business time of "8:09" will also be processed in a new time window because the spatial identifier has changed. Therefore, four sets of data to be deduplicated are obtained: "8:01-9:05", "8:06-8:08", "8:09", and "8:10-8:14".

[0075] Table 1 Data Division Table

[0076]

[0077] In a specific embodiment, Figure 4 The first deduplication path of the deduplication stage of this system is shown. The first and last data of each group of data to be deduplicated are retained, and the number of the first and last data is recorded. If there is only one data in the group of data to be deduplicated, the first and last data are the same, as shown in Table 2. The results are shown in Table 3.

[0078] Complete the deduplication step (e.g.) Figure 4 After (as shown in 402), the first piece of data is immediately sent to the subsequent business calculation module for business calculation (such as...). Figure 4 As shown in 403), the last data item is sent before the next time window opens. The business calculation module performs business calculations on the first and last data items and then synchronizes the data to the target end's real-time data stream (e.g., ...). Figure 4 (shown as 406);

[0079] Table 2. Deduplication Process

[0080]

[0081] Table 3: Results of Deduplication

[0082]

[0083] If computing resources are scarce, the last piece of data is simplified, the business time and number are updated, and the remaining business calculation operations are completed when the first piece of data arrives at the business calculation module.

[0084] Post-merger stage (e.g.) Figure 4 (As shown in 404) Further remove the first data and send the last data to the target data table (e.g.) Figure 4 As shown in Table 4 (as shown in 405), when the target data table has abundant storage resources, the first data can be optionally not removed, which can avoid the situation where the last data in the subsequent merging stage is not in time due to the processing delay of the preceding steps. The writing status of the target data table is shown in Table 5.

[0085] Table 4 shows the merging process.

[0086]

[0087] Table 5 Target Data Table

[0088]

[0089] In a specific embodiment, Figure 5 This illustrates the second deduplication path in the initial deduplication stage of this system. After the data to be deduplicated is divided, as shown... Figure 5 As shown in Table 502, only the last data in the data group to be deduplicated is retained and the number of times is calculated. Other duplicate data are removed to obtain the first deduplicated dataset, as shown in Table 6.

[0090] Table 6 First Deduplication Dataset Table

[0091]

[0092] like Figure 5 As shown in 503, copy the data group to be deduplicated, keeping only the first record of the data group to obtain the second deduplicated dataset, as shown in Table 7, and immediately send it to subsequent business calculations (such as...). Figure 5 (504); The time windows for the two deduplicated datasets can be flexibly adjusted based on the usage of computing resources;

[0093] The business calculation module receives the first deduplicated dataset and the second deduplicated dataset, as shown in Table 8, and performs business calculation processing (e.g. Figure 5 As shown in 504), the data dataset after business calculation is synchronized to the target end's real-time data stream (such as...). Figure 5 (507), and simultaneously handed over to the post-merge module for post-merge processing (such as... Figure 5 (505), two identical "8:09" data points appear in this step, which will be processed in the later merging stage;

[0094] In the post-merge stage, the first record of each data group in Table 8 is removed, and the result is sent to the target data table (e.g., ...). Figure 5 The target data table is shown in Table 9. (506).

[0095] Table 7 Second Deduplication Dataset Table

[0096]

[0097] Table 8 Business Calculation Data Table

[0098]

[0099] Table 9 Target Data Table

[0100]

[0101] It should be noted that this application performs two-stage data deduplication without compromising data timeliness, without limiting the size of the deduplication time window. Through deduplication, the amount of network data in the tens or hundreds of billions can be reduced to 20%-25% of the original amount. It is suitable for deduplication of massive data and can effectively reduce streaming computing resources and database storage resources, thereby achieving cost reduction and efficiency improvement.

[0102] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0103] like Figure 6 As shown, the computer system includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for system operation. CPU 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0104] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a liquid crystal display (LCD) and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card and a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0105] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0106] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0108] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0109] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire a source-end real-time data stream; divide the source-end real-time data streams with the same object identifier and spatial identifier within the same time window to obtain multiple sets of deduplicated data; record the count of the first data item in each set of deduplicated data as 1 and the count of the last data item as N(o,s,t); perform pre-deduplication processing on the multiple sets of deduplicated data through two paths and send them to the business calculation module for business calculation; perform post-merging processing on the multiple sets of data after business calculation and write the results into the target-end data table.

[0110] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0111] Finally, it should be noted that the above embodiments are for illustration only and not for limiting the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention without departing from the spirit and scope of the present invention. Any modifications or partial substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A two-stage data deduplication method for lossless data timeliness, characterized in that, include: S1: Obtain the real-time data stream from the source end, divide the real-time data stream from the source end according to the time window, time identifier, and object identifier, divide the data in the same time window, as well as the data with the same object identifier and spatial identifier in the same time window into the same group, and obtain multiple groups of data to be deduplicated; S2: Record the count of the first data in each group of data to be deduplicated as 1 and the count of the last data as N(o,s,t). Perform deduplication processing on the multiple groups of data to be deduplicated and send them to the business calculation module for business calculation. Where N(o,s,t) represents the number of times the last data point is read, and t represents the time window [t start ,t end Business hours within ], o and o i Represents object identifiers, s and s i I represents the spatial identifier, and I represents the indicator function. S3: Perform post-merge processing on multiple sets of data after business calculations and write the results into the target data table.

2. The method according to claim 1, characterized in that, Performing pre-deduplication processing on the multiple sets of data to be deduplicated and sending them to the business computing module includes: First deduplication path: Retain the first and last records of each group of data to be deduplicated, and send them to the business calculation module; or... The second deduplication path: only the first data in each group of data to be deduplicated is kept to obtain the first deduplicated dataset and it is immediately sent to the business computing module. At the same time, each group of data to be deduplicated is copied and only the last data is kept. Other duplicate data is removed to obtain the second deduplicated dataset and it is then sent to the business computing module.

3. The method according to claim 2, characterized in that, In the first deduplication path, the first piece of data is immediately sent to the business calculation module for full business calculation, and the last piece of data is sent to the business calculation module before the next time window opens for incremental business calculation.

4. The method according to claim 2, characterized in that, In the second deduplication path, the time window sizes of the first deduplication dataset and the second deduplication dataset are adjusted according to the computing resource usage.

5. The method according to claim 1, characterized in that, Step S2 and step S3 includes synchronizing the first piece of data that has completed business calculations with the real-time data stream of the target end.

6. The method according to claim 1, characterized in that, The post-merging process and writing the results into the target data table specifically includes removing the first data entry from each group and writing the last data entry from each group into the target data table.

7. A system for data deduplication according to any one of claims 1-6, characterized in that, This includes a deduplication module, a merging module, and a business calculation module; The deduplication module is used to deduplicatize the data to be deduplicated, the merging module is used to merge the deduplicated data, and the business calculation module is used to perform business calculations. The system acquires the real-time data stream from the source end and divides the real-time data stream from the source end according to the time window, time identifier, and object identifier. Data in the same time window, as well as data with the same object identifier and spatial identifier in the same time window, are grouped into the same group to obtain multiple groups of data to be deduplicated. The data to be deduplicated is input into the deduplication module for deduplication, including retaining the first and last data entries of each group of data to be deduplicated, and then sent to the business calculation module; or... Only the first data in each group of data to be deduplicated is kept to obtain the first deduplicated dataset and is immediately sent to the business computing module. At the same time, each group of data to be deduplicated is copied and only the last data is kept. Other duplicate data are removed to obtain the second deduplicated dataset and then sent to the business computing module. After performing business calculations, the business calculation module synchronizes the data to the target end's real-time data stream and sends it to the post-merging module. The post-merging module removes the first data entry and writes the last data entry into the target end's data table.

8. A computer program product having one or more computer programs thereon, characterized in that, When the computer program is executed by a computer processor, the method described in any one of claims 1-6 is performed.

Images