Streaming data deduplication method and system based on dynamic association grouping

Abstract

The invention discloses a streaming data deduplication method and system based on dynamic association grouping, and relates to the technical field of computer data processing, and the method comprises the steps: obtaining streaming data from a mobile terminal, and carrying out the grouping and sorting; fusing the current data with historical context data obtained from the cache to form an extended data set; grouping the extended data set based on a dynamic association rule, and identifying repeated data describing the same physical target to obtain a data grouping result; selecting representative data according to a data grouping result, and performing secondary spatial deduplication on the representative data; and sending the final representative data to a downstream service system, storing redundant data into a duplicate removal result database, and updating time and space cache. According to the method, intelligent duplicate removal of the data is realized through a multi-stage context-aware processing flow.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to a method and system for deduplicating streaming data based on dynamic association grouping. Background Technology

[0002] In automated inspection scenarios for infrastructure such as roads, power lines, and pipelines, mobile terminals such as inspection vehicles or drones continuously collect large amounts of status data (such as road surface defects and damaged utility poles) at high frequency. Due to the continuous nature of the data collection, the same target may be collected multiple times in a short period of time, resulting in a large amount of redundant data.

[0003] Existing data deduplication methods typically employ a simple fixed threshold strategy, for example:

[0004] Fixed time window: Data within a specified time (e.g., 2 seconds) is considered duplicates.

[0005] Fixed spatial distance: Data with a physical distance less than a certain value (such as 5 meters) are considered duplicates.

[0006] These simple methods have significant drawbacks:

[0007] Missed reports (deduplication failure): When the mobile terminal is very fast, the physical distance of the collection point of the same target flying by in a short period of time may be far beyond the fixed threshold, causing data that should be deduplicated to be processed as new data.

[0008] Accidental deletion (excessive deduplication): When the mobile terminal is slow or stationary, different targets in the same location may be very close in physical distance, causing independent data that should not be deduplicated to be mistakenly merged and deleted.

[0009] Processing discontinuous data: Traditional batch processing mode cannot effectively handle duplicate data across batches, that is, two collections of data for a target are exactly split into two processing batches, resulting in deduplication failure.

[0010] Therefore, there is an urgent need for an advanced deduplication method that can adapt to dynamic acquisition conditions (such as speed changes), accurately identify and retain the most effective information. Summary of the Invention

[0011] This invention aims to solve the problems of low deduplication accuracy, inability to adapt to dynamic working conditions, and easy information distortion in existing technologies for streaming data. It provides a method and system that can efficiently and accurately deduplicate continuously acquired data while retaining the highest quality data.

[0012] On the one hand, to achieve the above objectives, the present invention provides a streaming data deduplication method based on dynamic association grouping, comprising:

[0013] Acquire streaming data from mobile devices and group and sort it;

[0014] The current data is merged with historical context data retrieved from the cache to form an expanded dataset;

[0015] The extended dataset is grouped based on dynamic association rules to identify duplicate data describing the same physical target and obtain data grouping results.

[0016] Based on the data grouping results, representative data is selected, and the representative data undergoes secondary spatial deduplication.

[0017] The final representative data is sent to the downstream business system, redundant data is stored in the deduplication result database, and the time and space caches are updated.

[0018] Preferably, the dynamic association rules include:

[0019] Temporal similarity: The difference between the collection timestamps of two data points is less than a fixed time threshold;

[0020] Spatial distance similarity: The physical distance between two data points is less than a dynamic distance threshold, wherein the dynamic distance threshold is dynamically adjusted according to the instantaneous speed of the mobile terminal and / or the data type;

[0021] Feature transformation similarity: The lateral position difference between two data points after bird's-eye view transformation is less than the dynamic BEV threshold, wherein the dynamic BEV threshold is dynamically adjusted according to the instantaneous speed of the mobile terminal.

[0022] Preferably, when grouping the extended dataset based on the dynamic association rules, all conditions in the dynamic association rules must be met.

[0023] Preferably, the dynamic distance threshold uses a first value when the mobile terminal speed is lower than a first speed threshold and a second value when the speed is higher than a second speed threshold, wherein the second value is greater than the first value.

[0024] Preferably, selecting representative data based on the data grouping results includes:

[0025] If the data grouping result contains both newly collected data and historical data from the cache, the newly collected data will be marked as redundant.

[0026] If the data grouping results contain only newly collected data, then a representative data point is selected from the corresponding group according to the preset optimal representative selection rule.

[0027] Preferably, the optimal representative selection rule is:

[0028] For a given type of disease data, select the data with the largest diseased area as the representative data;

[0029] For other data that is not of the specified type, the data with the latest collection time is selected as the representative data.

[0030] Preferably, the representative data undergoes secondary spatial deduplication, including:

[0031] Calculate the geohash value of the data point and match it with the location information already stored in the spatial cache for the day. If a match is found, it is considered a duplicate and is removed.

[0032] On the other hand, to achieve the above objectives, the present invention also provides a system for implementing a streaming data deduplication method based on dynamic association grouping, comprising:

[0033] The data acquisition and preprocessing module is used to acquire and group and sort streaming data from mobile terminals;

[0034] The context cache fusion module is used to merge the current data with historical context data retrieved from the cache to form an extended dataset;

[0035] The core deduplication module is used to group the extended dataset based on dynamic association rules and select representative data according to the grouping results;

[0036] The secondary deduplication module is used to perform a second deduplication on the representative data based on spatial location;

[0037] The data output and cache management module is used to send the final representative data to the downstream business system, store redundant data in the deduplication result database, and update the time and space cache.

[0038] Compared with the prior art, the present invention has the following advantages and technical effects:

[0039] (1) High precision and strong adaptability: By introducing dynamic thresholds that are linked to working conditions such as speed and data type, and combining multiple dimensions such as physical distance and BEV transformation, the accuracy of deduplication decision is greatly improved, and it can effectively cope with various complex working conditions.

[0040] (2) High data fidelity: The unique “optimal representative selection strategy” ensures that while removing redundancy, it retains the data with the largest amount of information and the highest quality, avoiding the loss of valuable information;

[0041] (3) Strong processing continuity: The "context cache fusion" mechanism seamlessly connects different processing batches, solves the problem of identifying duplicate data across batches, and ensures the continuity and integrity of data stream processing;

[0042] (4) High efficiency: The core grouping algorithm adopts disjoint set, and its near constant time query and merge operations enable this method to handle large-scale, high-concurrency real-time data stream processing tasks. Attached Figure Description

[0043] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0044] Figure 1 This is a flowchart of a streaming data deduplication method based on dynamic association grouping according to an embodiment of the present invention;

[0045] Figure 2 This is a flowchart illustrating the core deduplication function of an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of dynamic association rules in an embodiment of the present invention;

[0047] Figure 4 This is a logic diagram for grouping and representative selection in an embodiment of the present invention. Detailed Implementation

[0048] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0050] This embodiment proposes a streaming data deduplication method based on dynamic association grouping, such as... Figure 1 ,include:

[0051] Acquire streaming data from mobile devices and group and sort it;

[0052] The current data is merged with historical context data retrieved from the cache to form an expanded dataset;

[0053] The extended dataset is grouped based on dynamic association rules to identify duplicate data describing the same physical target and obtain data grouping results.

[0054] Based on the data grouping results, representative data is selected, and the representative data undergoes secondary spatial deduplication.

[0055] The final representative data is sent to the downstream business system, redundant data is stored in the deduplication result database, and the time and space caches are updated.

[0056] Specifically, including:

[0057] Data Acquisition and Preprocessing: Real-time reception of streaming data from mobile terminals. The data is first grouped by device ID, and then sorted in ascending order by acquisition timestamp within each device group.

[0058] Context cache fusion: To solve the problem of cross-batch deduplication, historical data adjacent to the time window of the current batch data (e.g., 5 minutes before the start time of the batch) is retrieved from the time cache (K) and merged with the current batch data to form a temporally continuous, unbounded extended dataset.

[0059] Core deduplication logic: Two core deduplication steps are performed on the extended dataset. The first is dynamic association grouping, which is used to handle duplicates generated by high-frequency continuous collection; the second is secondary spatial deduplication, which is used to handle duplicates generated by different tasks in the same location within a day.

[0060] Data distribution: After processing, the data is divided into two categories. Representative data identified as "reserved" is sent to the downstream business message queue; non-representative data identified as "redundant" is stored in the deduplication result database for analysis.

[0061] Cache management: Retained representative data is used to update time and space caches, providing accurate context for subsequent data processing. Caches are typically set with expiration times (e.g., 36 hours).

[0062] This technical solution is applicable to scenarios such as intelligent transportation, road inspection, and infrastructure maintenance that require high-precision online processing of continuously collected data.

[0063] Furthermore, such as Figure 2 Dynamic associative grouping is the core of this technical solution. It receives a list of device data and initializes a disjoint-set data structure. Then, it iterates through the data points in the list and uses... Figure 3 The dynamic association rules shown are used to determine whether there is a relationship between any two data points. If a relationship exists, they are merged into the same set using the union operation of a disjoint-set data structure. After traversal, the data is divided into several independent sets (equivalence classes), and the data in each set is considered to describe the same physical target.

[0064] Grouping and Representative Selection: Responsible for processing the various data sets generated in the previous step, and based on... Figure 4The logic shown is used to make decisions. The ultimate goal is to select a unique, highest-quality representative from each set and mark the other data in the set as to be deleted.

[0065] Furthermore, dynamic association rules include:

[0066] Temporal similarity: The difference between the collection timestamps of two data points is less than a fixed time threshold;

[0067] Spatial distance similarity: The physical distance between two data points is less than a dynamic distance threshold, wherein the dynamic distance threshold is dynamically adjusted according to the instantaneous speed of the mobile terminal and / or the data type;

[0068] Feature transformation similarity: The lateral position difference between two data points after bird's-eye view transformation is less than the dynamic BEV threshold, wherein the dynamic BEV threshold is dynamically adjusted according to the instantaneous speed of the mobile terminal.

[0069] Specifically, such as Figure 3 It illustrates the rule engine used to determine whether two data points (P1, P2) are related.

[0070] The rule engine includes the following multi-dimensional judgments, all of which must be satisfied for a relationship to be considered "related":

[0071] Temporal similarity: The difference between the timestamps of two data points must be less than a fixed short-time window T, such as 4 seconds. This is the basis for correlation.

[0072] Spatial distance similarity: The physical distance calculated from the GPS coordinates of two data points must be less than a dynamic distance threshold (D_thresh). This threshold is not fixed; it is dynamically adjusted based on the current vehicle speed (v) and the type of damage (type).

[0073] Furthermore, the dynamic distance threshold uses a first value when the mobile terminal speed is lower than a first speed threshold and a second value when the speed is higher than a second speed threshold, wherein the second value is greater than the first value.

[0074] Specifically, for example, the threshold may be relaxed to 10 meters when the speed exceeds 80 km / h, and tightened to 3.8 meters at low speeds. For certain special types of defects (such as signs), the threshold can be further relaxed.

[0075] Feature Transformation Similarity: To overcome GPS positioning errors, a Bird's-Eye (BEV) transformation is introduced. This transformation maps the location of defects in the image to the vehicle's own coordinate system, calculating its lateral position. The difference in the BEV lateral position between two data points must be less than a dynamic BEV threshold (B_thresh). This threshold is also related to speed (v); for example, a larger lateral deviation is allowed at high speeds.

[0076] Further, representative data is selected based on the data grouping results, including:

[0077] If the data grouping result contains both newly collected data and historical data from the cache, the newly collected data will be marked as redundant.

[0078] If the data grouping results contain only newly collected data, then a representative data point is selected from the corresponding group according to the preset optimal representative selection rule.

[0079] Specifically, such as Figure 4 It demonstrates how to process a dataset once it is formed.

[0080] Analyze set composition: First, examine the data composition within the set. Due to the "context cache fusion" step, a set may contain only new data from the current batch ("purely new data set"), or it may contain both new data and historical data from the cache ("mixed new and old data set").

[0081] Execution processing strategy:

[0082] For mixed groups: If historical cached data exists in the set, this indicates that the newly collected data is a duplicate of historical data that has already been entered into the database. In this case, there is no need to select a new representative. All new data within this group will be directly marked as pending deletion.

[0083] For a completely new set: if the set contains only new data, then an optimal representative needs to be selected from it.

[0084] Furthermore, the optimal representative selection rule is as follows:

[0085] For a given type of disease data, select the data with the largest diseased area as the representative data;

[0086] For other data that is not of the specified type, the data with the latest collection time is selected as the representative data.

[0087] Specifically, the optimal representative selection rule is a scalable set of rules designed to retain the most informative data. For example:

[0088] For specific disease types (such as "longitudinal cracks"), retain the data with the largest disease area.

[0089] For other specific disease types, the data with the largest area is also retained when the image resolution meets the requirements.

[0090] In general cases where special rules are not met, retain the data with the latest collection time.

[0091] Furthermore, secondary spatial deduplication is performed on the representative data, including:

[0092] Calculate the geohash value of the data point and match it with the location information already stored in the spatial cache for the day. If a match is found, it is considered a duplicate and is removed.

[0093] The core of this method lies in achieving intelligent deduplication of data through a multi-stage, context-aware processing flow. First, a context caching fusion mechanism breaks down the barriers between data processing batches. Then, a dynamic association grouping algorithm based on Union-Find is used to classify similar data according to a set of multi-dimensional rules linked to operating conditions (such as speed). Next, a sophisticated optimal representative selection strategy is used to select the data that best represents the real situation from each category. Finally, secondary space deduplication eliminates occasional duplications, and the results are distributed and cached for updates, forming an efficient processing closed loop.

[0094] This embodiment also provides a system for implementing a streaming data deduplication method based on dynamic association grouping, including:

[0095] The data acquisition and preprocessing module is used to acquire and group and sort streaming data from mobile terminals;

[0096] The context cache fusion module is used to merge the current data with historical context data retrieved from the cache to form an extended dataset;

[0097] The core deduplication module is used to group the extended dataset based on the disjoint-set data structure and dynamic association rules, and select representative data according to the grouping results;

[0098] The secondary deduplication module is used to perform a second deduplication on the representative data based on spatial location;

[0099] The data output and cache management module is used to send the final representative data to the downstream business system, store redundant data in the deduplication result database, and update the time and space cache.

[0100] To more clearly illustrate the technical solution of the present invention, specific embodiments are provided below for description:

[0101] This embodiment takes the "road inspection" scenario in the field of "intelligent transportation" as an example to explain in detail the complete processing flow of this technical solution.

[0102] Scene setting:

[0103] Equipment: A road inspection vehicle with ID "XJ-001" is traveling on the highway at different speeds and collecting road surface damage data in real time.

[0104] Data: Each data entry includes a timestamp, GPS coordinates, instantaneous speed (v), disease type (type), disease area (area), BEV lateral distance, etc.

[0105] System rules (example):

[0106] Time cache window: 5 minutes (for context fusion).

[0107] Dynamic association rules:

[0108] T_thresh(time difference) < 4 seconds;

[0109] D_thresh (physical distance):

[0110] When v < 40 km / h (low speed), D_thresh = 3.8 meters.

[0111] When v > 80 km / h (high speed), D_thresh = 10 meters.

[0112] B_thresh (BEV lateral distance) < 0.5 meters (linked to speed v).

[0113] Optimal representative selection strategy:

[0114] Type "Longitudinal Crack (136)": Select the area with the largest area.

[0115] By default: Select the one with the latest timestamp.

[0116] Assuming the current time is 10:00:00, the system begins processing a new batch of data.

[0117] Step 1: Integration of Data Acquisition and Context Caching

[0118] Acquiring streaming data: The system receives 3 new data entries (P2, P3, P4) from device "XJ-001" and sorts them by timestamp.

[0119] Integrated time cache: The system retrieves the data of “XJ-001” between 09:55:00 and 10:00:00 from the time cache (Redis).

[0120] Forming an expanded dataset: Suppose that one data P1 is retrieved from the cache (this data is the last data in the previous batch).

[0121] P1 (from cache): {ID: 101, Time: 09:59:58, GPS: (Lng1, Lat1), v: 30km / h, Type: 101 (pothole)};

[0122] P2 (New Data): {ID: 201, Time: 10:00:01, GPS: (Lng2, Lat2), v: 30km / h, Type: 101 (Pothole)};

[0123] P3 (New Data): {ID: 202, Time: 10:00:05, GPS: (Lng3, Lat3), v: 90km / h, Type: 136 (Longitudinal Crack), Area: 1.5, BEV: 0.3m};

[0124] P4 (New Data): {ID: 203, Time: 10:00:06, GPS: (Lng4, Lat4), v: 90km / h, Type: 136 (Longitudinal Crack), Area: 2.1, BEV: 0.4m};

[0125] Distance parameter description:

[0126] Dist(P1, P2) = 2.5 meters (physical distance is close);

[0127] Dist(P3, P4) = 9.0 meters (physical distance is far).

[0128] Step 2, Core Deduplication - Dynamic Grouping:

[0129] The system applies a disjoint-set data structure algorithm to the extended dataset {P1, P2, P3, P4} and uses dynamic association rules for judgment:

[0130] Compare P1 and P2:

[0131] Time difference: 3 seconds (<4 seconds). Satisfied.

[0132] Operating conditions: speed v=30km / h (low speed), trigger D_thresh = 3.8 meters.

[0133] Physical distance: 2.5 meters (<3.8 meters). Satisfied.

[0134] BEV: (Assuming it is satisfied). Satisfied.

[0135] Conclusion: P1 and P2 are determined to be "associated". Using the union(P1, P2) operation, a set Group A = {P1, P2} is formed.

[0136] This operation successfully linked duplicate data across batches, and because it was performed at a low speed, the threshold was tightened, preventing accidental deletion.

[0137] Compare P3 and P4:

[0138] Time difference: 1 second (<4 seconds). Satisfied.

[0139] Operating conditions: speed v = 90km / h (high speed), trigger D_thresh = 10 meters.

[0140] Physical distance: 9.0 meters (<10 meters). Satisfied.

[0141] BEV: Lateral distance difference 0.1m (<0.5m). Satisfied.

[0142] Conclusion: P3 and P4 are determined to be "associated". Union(P3, P4) forms the set Group B = {P3, P4}.

[0143] Under high-speed conditions, although the physical distance is 9 meters (far exceeding the traditional 5-meter threshold), the dynamic threshold is relaxed to 10 meters, which successfully deduplicates the data and avoids false negatives.

[0144] Step 3: Core Deduplication - Grouping and Representative Selection:

[0145] The system iterates through the collection formed in the previous step and applies... Figure 4 The logic:

[0146] 1. Process Group A={P1, P2}:

[0147] Analysis: The set contains cached data P1 and new data P2, and is determined to be a "mixed data set".

[0148] Strategy: Apply a hybrid group strategy.

[0149] Decision: All new data within the group (i.e., P2) is marked as "deleted". P1 is retained as a historical representation.

[0150] 2. Process Group B = {P3,P4}:

[0151] Analysis: The set contains only new data P3 and P4, so it is determined to be a "pure new data set".

[0152] Strategy: Implement the "optimal representative selection strategy".

[0153] Rule: Disease type 136 (longitudinal crack) triggers the "select the largest area" rule.

[0154] Decision: Compare Area(P3) = 1.5 with Area(P4) = 2.1. P4 wins.

[0155] Results: P4 is marked "Retained" (representative), and P3 is marked "Deleted" (redundant).

[0156] The system retains P4 because it has the largest area and high data fidelity, thus avoiding the loss of valuable information.

[0157] Step 4: Secondary space deduplication:

[0158] The system performs secondary spatial deduplication on all new data marked "reserved" (P4 in this example). This step is used to check if other tasks reported data at the same location that day. Assuming no match is found in the spatial cache for P4, P4 is ultimately confirmed as reserved.

[0159] Step 5: Data Distribution and Cache Management

[0160] 1. Data distribution:

[0161] Retained data (representative): P4 is sent to downstream business message queues (such as Kafka) for use by real-time alarms or ticketing systems.

[0162] Redundant data (deduplicated): P2 and P3 are stored in the deduplication result database (such as HBase) for subsequent analysis.

[0163] 2. Cache Management:

[0164] Update time cache: Write the representative data P4 to the time cache Redis and set the expiration time (e.g., 36 hours) as the context for subsequent batches.

[0165] Update the space cache: Write the space information of P4 (such as Geohash) to the space cache Redis for use in step four.

[0166] This embodiment uses "context cache fusion" to associate P1 and P2 across batches; it uses "dynamic association rules" to associate P3 and P4, which are 9 meters apart, at high speed, overcoming the defect of easy missed detection by fixed threshold; it uses "optimal representative selection" to retain P4 with the largest area, ensuring high data fidelity; the whole process is based on disjoint set, which is efficient and meets the requirements of real-time stream processing.

[0167] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for deduplicating streaming data based on dynamic association grouping, characterized in that, include: Acquire streaming data from mobile devices and group and sort it; The current data is merged with historical context data retrieved from the cache to form an expanded dataset; The extended dataset is grouped based on dynamic association rules to identify duplicate data describing the same physical target and obtain data grouping results. Based on the data grouping results, representative data is selected, and the representative data undergoes secondary spatial deduplication. The final representative data is sent to the downstream business system, redundant data is stored in the deduplication result database, and the time and space caches are updated.

2. The streaming data deduplication method based on dynamic association grouping according to claim 1, characterized in that, The dynamic association rules include: Temporal similarity: The difference between the collection timestamps of two data points is less than a fixed time threshold; Spatial distance similarity: The physical distance between two data points is less than a dynamic distance threshold, wherein the dynamic distance threshold is dynamically adjusted according to the instantaneous speed of the mobile terminal and / or the data type; Feature transformation similarity: The lateral position difference between two data points after bird's-eye view transformation is less than the dynamic BEV threshold, wherein the dynamic BEV threshold is dynamically adjusted according to the instantaneous speed of the mobile terminal.

3. The streaming data deduplication method based on dynamic association grouping according to claim 2, characterized in that, When grouping the extended dataset based on the dynamic association rules, all conditions in the dynamic association rules must be met.

4. The streaming data deduplication method based on dynamic association grouping according to claim 2, characterized in that, The dynamic distance threshold uses a first value when the mobile terminal speed is lower than a first speed threshold and a second value when the speed is higher than a second speed threshold, wherein the second value is greater than the first value.

5. The streaming data deduplication method based on dynamic association grouping according to claim 1, characterized in that, Representative data are selected based on the data grouping results, including: If the data grouping result contains both newly collected data and historical data from the cache, the newly collected data will be marked as redundant. If the data grouping results contain only newly collected data, then a representative data point is selected from the corresponding group according to the preset optimal representative selection rule.

6. The streaming data deduplication method based on dynamic association grouping according to claim 5, characterized in that, The optimal representative selection rule is as follows: For a given type of disease data, select the data with the largest diseased area as the representative data; For other data that is not of the specified type, the data with the latest collection time is selected as the representative data.

7. The streaming data deduplication method based on dynamic association grouping according to claim 1, characterized in that, Perform secondary spatial deduplication on the representative data, including: Calculate the geohash value of the data point and match it with the location information already stored in the spatial cache for the day. If a match is found, it is considered a duplicate and is removed.

8. A system for implementing the streaming data deduplication method based on dynamic association grouping as described in any one of claims 1-7, characterized in that, include: The data acquisition and preprocessing module is used to acquire and group and sort streaming data from mobile terminals; The context cache fusion module is used to merge the current data with historical context data retrieved from the cache to form an extended dataset; The core deduplication module is used to group the extended dataset based on dynamic association rules and select representative data according to the grouping results; The secondary deduplication module is used to perform a second deduplication on the representative data based on spatial location; The data output and cache management module is used to send the final representative data to the downstream business system, store redundant data in the deduplication result database, and update the time and space cache.

Images