Method and device for burying point deduplication, electronic equipment and computer readable storage medium

Abstract

The embodiment of the application discloses a kind of methods, devices, electronic equipment and computer readable storage medium of burying point deduplication, related to burying point technical field;The method comprises: obtaining the original event data of target event collected by burying point;The original event data is standardized coding, and the first event fingerprint of the target event is obtained;Check the fuse state, check the total number of reporting in the current period when not triggering the fuse;When the total number of reporting is lower than total number threshold, check the second event fingerprint of event that has been reported in the current period, when the first event fingerprint is included in the second event fingerprint, the target event is abandoned to report;When the first event fingerprint is not included in the second event fingerprint, the target event is reported, and the total number of reporting is updated.Such, the present scheme can realize burying point data deduplication in client.

Description

Technical Field

[0001] This application relates to the field of data embedding technology, specifically to a method, apparatus, electronic device, and computer-readable storage medium for deduplication of data embedding. Background Technology

[0002] In the field of mobile application event tracking data deduplication technology, mainstream solutions mainly rely on cloud-based hash verification or distributed Bloom filters to implement the core deduplication logic. These solutions have three key technical drawbacks: First, they are highly dependent on the network, requiring the final deduplication operation to be completed on the server side, leading to a large amount of invalid event tracking data being frequently uploaded to the cloud, wasting bandwidth and storage resources. Second, they consume excessive memory resources; in high-frequency event-triggered scenarios, the cloud needs to store massive amounts of event fingerprints for verification, consuming large amounts of memory resources and resulting in high operational costs. Third, they lack adaptability to dynamic scenarios; fixed-time-window verification mechanisms cannot effectively identify duplicate events in dynamic scenarios such as network retransmissions and short-term repeated user operations, easily leading to a high false positive rate. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and computer-readable storage medium for deduplication of embedded data, which can achieve deduplication of embedded data on the client side.

[0004] In a first aspect, embodiments of this application provide a method for deduplicating data entry points, applied to a client-side application, including: Obtain the raw event data of the target event collected by the tracking points; The original event data is standardized and encoded to obtain the first event fingerprint of the target event; Check the circuit breaker status; if the circuit breaker has not been triggered, check the total number of reports submitted in the current period. When the total number of reported events is lower than the total number threshold, check the second event fingerprint of the events already reported in the current period. If the second event fingerprint includes the first event fingerprint, abandon the reporting of the target event. If the first event fingerprint is not included in the second event fingerprint, the target event is reported and the total number of reports is updated.

[0005] In one embodiment, the step of standardizing and encoding the original event data to obtain a first event fingerprint of the target event includes: Extract the core field that uniquely identifies the target event from the original event data; The core fields are sorted according to their key names based on the ordered mapping table to obtain the sorted core fields; The keys and values ​​of the sorting core field are encoded, and special characters are converted into the encoding format to obtain the encoded keys and encoded values; The encoded key and the encoded value are concatenated using a delimiter to obtain the first event fingerprint of the target event.

[0006] In one embodiment, reporting the target event includes: Set a single-threaded timer; the initial delay and loop interval of the single-threaded timer are the same; Based on the single-threaded timer, the reporting task is automatically executed periodically to report events in batches; the events to be reported include the target events.

[0007] In one embodiment, the method further includes: When the reported task reports an error, the single-threaded timer is automatically restarted. When modifying the timing duration of the single-threaded timer, verify the validity of the new timing duration; When the new timeout period is valid, the old timeout task is stopped, and the new timeout task is started according to the new timeout period.

[0008] In one embodiment, reporting the target event includes: The original event data of the target event is decoded to obtain the original key-value pairs; The atomic accumulation of the reporting count of the target event within the current period is achieved by using the merging method of the ordered mapping table; The reported number statistics and the original key-value pairs are reported.

[0009] In one embodiment, the method further includes: Store the total number of reports in the current period and the statistical value of the number of reports of the target event in the current period into a local file; In the new cycle, the total number of reports and the report count statistics for each event stored in the previous cycle are cleared.

[0010] In one embodiment, the method further includes: When the circuit breaker is triggered, the reporting of the target event is abandoned; If the total number of reported events is not lower than the total number threshold, the reporting of the target event is abandoned.

[0011] Secondly, embodiments of this application provide a device for deduplicating embedded data points, applied to a client-side application, comprising: The data acquisition module is used to acquire the raw event data of the target events collected by the embedded points; The data encoding module is used to standardize and encode the original event data to obtain the first event fingerprint of the target event; The first inspection module is used to check the circuit breaker status. When the circuit breaker is not triggered, it checks the total number of reports in the current period. The second inspection module is used to check the second event fingerprint of the events already reported in the current period when the total number of reports is lower than the total number threshold, and abandon the reporting of the target event when the second event fingerprint includes the first event fingerprint. The event reporting module is used to report the target event and update the total number of reports when the second event fingerprint does not include the first event fingerprint.

[0012] Thirdly, embodiments of this application also provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the above-described method for removing duplicate data points.

[0013] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the above-described method for removing duplicate data points.

[0014] Fifthly, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in embodiments of this application.

[0015] The embodiments of this application have the following beneficial effects: The client generates and verifies event fingerprints using local standardized encoding, filtering duplicate events at the source and preventing invalid data uploads, significantly reducing network bandwidth consumption and server receiving pressure. Dual control via circuit breaker status and total threshold prevents sudden surges in reporting volume caused by high-frequency events and precisely controls the total reporting volume per cycle, avoiding excessive memory consumption on the server due to massive fingerprint storage and significantly reducing cloud operation and maintenance costs. Local real-time verification based on event fingerprints flexibly adapts to dynamic scenarios such as network retransmissions and short-term repeated user operations, effectively reducing the false positive rate of duplicate events and improving data accuracy compared to cloud solutions with fixed time windows. The total reporting volume is updated and reported only when the fingerprint verification indicates a new event, ensuring that the same event is reported only once per cycle, reducing invalid reporting while ensuring the integrity and validity of the data, balancing resource conservation and data quality. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the steps of a method for removing duplicate data points provided in an embodiment of this application; Figure 2 This is an interactive schematic diagram of a method for deduplicating data points provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the embedded point deduplication device provided in one embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0019] The current mainstream technical solutions for deduplication of event tracking data in mobile applications rely primarily on cloud-based hash verification, distributed Bloom filters, and traditional hash algorithms. The overall technical system is highly dependent on server-side computing power and network interaction, with the client only handling basic data collection and uploading functions, lacking independent deduplication filtering capabilities. This has exposed numerous technical problems in practical applications, specifically: Excessive reliance on cloud computing and high network overhead: Mainstream solutions all require hash comparison through a server-side fingerprint database to determine duplicate data, necessitating at least one network request to complete the deduplication operation. Some solutions even require multiple interactions with the server for verification, resulting in significant bandwidth waste and high processing latency. Especially in weak network environments, the timeliness of data processing is severely affected, and the lack of effective filtering mechanisms on the client side leads to the continuous uploading of invalid duplicate data, further exacerbating network resource consumption.

[0020] High memory consumption and poor adaptability: Bloom filter-based deduplication solutions require a large amount of memory space to be pre-allocated to store event fingerprints and do not support dynamic recycling, resulting in low resource utilization; existing client-side deduplication solutions require full caching of event fingerprints, with memory consumption exceeding 100MB in scenarios with tens of thousands of events. Even storing 100,000 event fingerprints requires approximately 12MB of memory, which not only causes a serious waste of memory resources but also makes it unsuitable for low-configuration mobile terminal devices due to excessive computational complexity and resource consumption.

[0021] Insufficient adaptability to dynamic scenarios and high false positive rate: Traditional solutions often use a fixed time window verification mechanism, which is difficult to effectively distinguish between real high-frequency user operations and duplicate events caused by network retransmission. The overall false positive rate exceeds 5%, which can easily lead to the loss of effective data. At the same time, traditional hash algorithms are highly sensitive to the field order when processing structured data points. The hash results will be different simply because the field order changes, which will lead to false positives for duplicate detection and seriously affect the accuracy of deduplication.

[0022] To solve or partially solve the above-mentioned technical problems, in one embodiment of this application, such as Figure 1 As shown, a method for deduplicating event tracking points is provided. Although the logical order is illustrated in the step diagram, in some cases, the steps shown or described can be performed in a different order than that shown in the diagram. Specifically, this method for deduplicating event tracking points can be applied to a client, which may include, but is not limited to, one or more of smartphones, tablets, laptops, desktop computers, and in-vehicle computers.

[0023] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0024] according to Figure 1 The method for removing duplicate data points shown includes at least steps S110 to S150, which are described in detail below: In step S110, the raw event data of the target event collected by the embedded points is obtained.

[0025] Data collection points are pre-set data collection points within an application, used to automatically capture user behavior, system status, and business events, providing raw event data for data analysis and statistical reporting. Target events can be any event such as user behavior, system operation, and business triggers. Raw event data can include, but is not limited to, one or more of the following: behavior type, module identifier, user ID, device information, network status, application version, trigger time, and context information.

[0026] During the operation of a mobile application, the preset event tracking points can collect all raw event data related to the target event. The raw event data is the basis for subsequent deduplication and verification. The collection process takes into account both real-time performance and completeness, ensuring that no core event features are missed, while avoiding the collection of redundant and invalid fields, thus reducing the pressure on subsequent standardization processing.

[0027] In one embodiment, raw event data can be received via synchronous (reportSync) or asynchronous (reportAsync) methods. The raw event data can be of type HashMap, which is an unordered key-value mapping collection implemented based on a hash table in the Java Collections Framework. Null value fields in the raw event data can be automatically filled with empty strings, and non-tracking fields (such as device timestamps) can be excluded.

[0028] In step S120, the original event data is standardized and encoded to obtain the first event fingerprint of the target event.

[0029] To address issues such as inconsistent raw event data formats, disordered field order, and interference from special characters, the collected raw event data can be standardized and encoded. First, core fields that uniquely identify the essence of the event can be selected. These core fields are then sorted according to fixed rules using an ordered mapping table (TreeMap) to eliminate differences in field order. Next, the keys and values ​​of the fields are URL-encoded, converting special characters such as "=" and spaces to a standard encoding format to avoid concatenation conflicts. Finally, the encoded fields are concatenated in a fixed format of "key1=value1|key2=value2" to generate the first event fingerprint. This ensures that events with identical content correspond to the same event fingerprint, providing a unified standard for subsequent deduplication verification.

[0030] In step S130, the circuit breaker status is checked. If the circuit breaker is not triggered, the total number of reports in the current period is checked.

[0031] The circuit breaker state is a pre-defined reporting disabling state in the client-side event tracking and deduplication process. If triggered, it directly returns "reporting not allowed"; otherwise, it continues to execute the verification logic for the total number of reports and event fingerprints. Therefore, we can first determine whether the circuit breaker has been triggered by the circuit breaker flag (mEnable) stored locally on the client. If the circuit breaker was triggered due to exceeding the reporting limit (mEnable=false), subsequent reporting requests are directly rejected, and the target event is abandoned, avoiding unnecessary resource consumption. If the circuit breaker has not been triggered (mEnable=true), the total number of reports in the current period is further read from the local statistics. This counter can record in real time the number of events that have passed deduplication verification and been successfully reported in the current period, providing data support for subsequent traffic management.

[0032] In step S140, when the total number of reports is lower than the total number threshold, the second event fingerprint of the events already reported in the current period is checked. If the second event fingerprint includes the first event fingerprint, the reporting of the target event is abandoned.

[0033] The total threshold is a pre-set upper limit for single-cycle reporting by the client, used to prevent a sudden surge in reporting volume due to high-frequency events. If the total number of reports in the current cycle is not lower than the pre-set total threshold but the circuit breaker is not triggered, the circuit breaker is triggered and the target event is abandoned.

[0034] If the circuit breaker is not triggered and the total number of reports in the current period is lower than the preset total threshold, the second event fingerprint set (mDayReportedSet) stored locally can be retrieved. The second event fingerprint set stores the fingerprints of all successfully reported events in the current period. By comparing the first event fingerprint with the second event fingerprint set in real time, if a match is found, it means that the target event is a duplicate event (such as repeated user operations in a short period of time or repeated triggering caused by network retransmission). In this case, the reporting is abandoned directly, and invalid data is filtered out from the source.

[0035] The cycle can be set according to actual needs; for example, it can be set to one cycle per natural day.

[0036] In step S150, if the first event fingerprint is not included in the second event fingerprint, the target event is reported and the total number of reports is updated.

[0037] If the first event fingerprint is not included in the second event fingerprint set, it indicates that the target event is a new event within the current period and meets the reporting requirements. In this case, the client-side reporting logic is triggered, reporting the target event data to the server. Simultaneously, the first event fingerprint is added to the second event fingerprint set to prevent duplicate reporting. The total number of reports is incremented by 1, and the local counter is updated to ensure the accuracy of the total report count.

[0038] The technical solution adopted in this application embodiment generates and verifies event fingerprints through local standardized encoding, filtering duplicate events from the source, avoiding invalid data uploads, and significantly reducing network bandwidth consumption and server receiving pressure. Dual control of circuit breaker status and total threshold not only prevents sudden increases in reporting volume caused by high-frequency events but also accurately controls the total reporting volume per cycle, avoiding excessive memory consumption on the server due to massive fingerprint storage, and significantly reducing cloud operation and maintenance costs. Local real-time verification based on event fingerprints can flexibly adapt to dynamic scenarios such as network retransmissions and short-term repeated user operations. Compared with cloud solutions with fixed time windows, it effectively reduces the false positive rate of duplicate events and improves data accuracy. The total reporting volume is only updated and reported when the fingerprint verification indicates a new event, ensuring that the same event is reported only once within a single cycle. This reduces invalid reporting behavior and ensures the integrity and validity of the data, balancing resource conservation and data quality.

[0039] Based on the above technical solution, as an embodiment, the standardization encoding of the original event data to obtain the first event fingerprint of the target event may include: extracting a core field that uniquely identifies the target event from the original event data; sorting the core field according to the key name based on an ordered mapping table to obtain a sorted core field; encoding the key and value of the sorted core field and converting special characters into an encoding format to obtain an encoded key and an encoded value; and concatenating the encoded key and the encoded value using a delimiter to obtain the first event fingerprint of the target event.

[0040] Filter the key information that can characterize the essential features of the event from the original event data (such as behavior type, module identifier, user ID, device ID, etc.), remove redundant fields that are volatile and do not affect the uniqueness of the event, such as random codes and temporary status values, and ensure that only the core fields that can uniquely identify the target event are retained.

[0041] The extracted core fields are stored in an ordered mapping table, which automatically arranges the core fields according to the preset character order of the key names, eliminating the fingerprint inconsistency problem caused by the disordered order of the original data fields, and outputting the sorted core fields with a unified order.

[0042] Using UTF-8 character encoding, URL encoding is performed on the key name and field value of each core field after sorting, converting special characters such as "|", "=", space, and "%" into their corresponding URL encoding formats (e.g., space is converted to %20) to avoid special characters interfering with subsequent concatenation logic and generate standardized encoded keys and values.

[0043] Using "=" as the separator for key-value pairs and "|" as the delimiter for different fields, all the encoded core fields are concatenated in a fixed format of "key1=value1|key2=value2" to generate a unique and stable string, which is the first event fingerprint that can accurately identify the target event.

[0044] The technical solution of this application can eliminate the difference in the order of key-value pairs by sorting the fields, avoid the generation of different fingerprints for the same event due to different field arrangements, solve the key order sensitivity problem of traditional hashing, and achieve zero false judgment; the standardized URL encoding process for special characters can avoid fingerprint disorder caused by separator conflict and ensure the uniqueness of the encoding; the generated string fingerprint is lightweight, easy to store and easy to compare, and can achieve real-time verification with O(1) complexity, and is adapted to efficient deduplication on the client's local machine.

[0045] Based on the above technical solution, as an embodiment, reporting the target event may include: setting a single-threaded timer; the initial delay and the loop interval of the single-threaded timer are the same; based on the single-threaded timer, automatically executing the reporting task periodically to report events in batches; the events to be reported include the target event.

[0046] Configure and start a single-threaded timer as the scheduling core for reporting tasks. The initial execution delay of this single-threaded timer is consistent with the loop interval of each execution, for example, it can be configured to a preset reporting cycle in minutes (e.g., 10 minutes), achieving uniformity and simplicity in reporting scheduling rules. Based on the periodic scheduling capability of this single-threaded timer, the system can automatically trigger and execute reporting tasks according to the set target cycle (e.g., 10 minutes). During each execution of a reporting task, all events to be reported that meet the reporting conditions within the current cycle (e.g., the most recent 10 minutes) are batch-summarized and uniformly reported. The events to be reported can include the target events that have completed deduplication verification, are determined to be non-duplicate, and meet the reporting rules. At the same time, the single-threaded execution mechanism can avoid multi-task concurrency conflicts, ensuring the orderliness of the reporting process and the accuracy of data.

[0047] The technical solution adopted in this application avoids multi-task concurrency conflicts through a single-threaded mechanism, ensuring that the reporting process is orderly and the data is accurate. The unified time configuration of the initial delay and the cycle interval makes the scheduling rules simple and easy to maintain, and there is no need to adjust the parameters separately when dynamically modifying the cycle. Periodic batch reporting of event data can significantly reduce the number of reporting requests, reduce network bandwidth consumption, and at the same time make the reporting traffic evenly distributed, avoiding sudden increases in server pressure. It can also efficiently summarize the data to be reported containing the target event, improving the overall reporting efficiency.

[0048] Based on the above technical solution, as an embodiment, the method for deduplicating data points may further include: automatically restarting the single-threaded timer when the reported task reports an error; verifying the validity of the new timer duration when modifying the timer duration; stopping the old timer task and starting a new timer task according to the new timer duration when the new timer duration is valid.

[0049] When a reporting task encounters an error due to program abnormalities, system resource fluctuations, or other reasons, the exception capture mechanism can automatically trigger a restart logic to restart the single-threaded timer, ensuring the continuity of the reporting scheduling service and preventing the subsequent reporting process from being interrupted due to the failure of a single task.

[0050] When it is necessary to modify the timing duration of a single-threaded timer to adapt to different business reporting requirements, the validity of the input new timing duration can be checked first. The check includes whether the new timing duration is a positive number and whether there is a valid difference from the current timing duration, to ensure that the newly configured duration conforms to the timer scheduling rules.

[0051] When the new timing duration is deemed valid after validity verification, the currently running old timing task needs to be terminated first to avoid scheduling conflicts caused by the parallel execution of the old and new tasks. Then, the new single-threaded timer is reconfigured and started according to the new timing duration that has passed the verification, so that the new reporting cycle rule takes effect immediately and achieves smooth and conflict-free dynamic adjustment of the timing duration.

[0052] The technical solution adopted in this application embodiment automatically restarts the timer when a reporting task fails, which can avoid reporting interruption caused by a single exception and ensure the continuity and integrity of the reported data. The validity check before modifying the timer duration can filter invalid configurations and prevent scheduling chaos caused by incorrect parameters. The method of stopping the old timer task before starting the new timer task can avoid the concurrent conflict between the old and new timer tasks, realize the smooth switching of the reporting cycle, adapt to the dynamic reporting needs of different business scenarios, and at the same time ensure the stability of the scheduling logic.

[0053] Based on the above technical solution, as an embodiment, reporting the target event may include: decoding the original event data of the target event to obtain the original key-value pairs; using the merging method of the ordered mapping table to achieve atomic accumulation of the reporting count statistics of the target event in the current period; and reporting the reporting count statistics and the original key-value pairs.

[0054] It can perform standardized decoding operations on the raw event data of the target event, reverse parse the URL-encoded content when the event fingerprint was generated, restore the original key-value pairs containing the core features of the event, ensure the integrity and readability of the reported data, and provide a standard and easy-to-parse raw data format for subsequent data analysis and statistics on the server side.

[0055] The atomic merge method of the ordered mapping table is called. Based on the event fingerprint as a unique identifier, the atomic accumulation operation is performed on the reporting count of the target event in the current period. This atomic operation can effectively avoid the counting error problem in multi-threaded concurrent scenarios, ensure the accuracy of the reporting count statistics, and at the same time, only the count is updated for duplicate events in the same period, without the need to repeatedly store event data, which greatly reduces memory usage.

[0056] The reported frequency statistics after atomic accumulation are associated and integrated with the restored original key-value pairs to form a complete reporting data packet containing basic event information and occurrence frequency. The reporting operation is executed uniformly, which not only realizes the frequency statistics of a single event, but also allows the server to obtain the original event information synchronously, thus balancing reporting efficiency and data dimension integrity. At the same time, the batch reporting method also reduces the number of network requests and reduces bandwidth consumption.

[0057] The technical solution of this application's embodiments decodes the encoded data to restore the original key-value pairs, enabling the server to directly parse and analyze them without secondary processing; atomic merging and accumulating counts can avoid concurrent counting errors and accurately count the frequency of event reporting within a period; associating the counts with the original data for reporting not only preserves the complete information of the events but also enables the aggregation and reporting of duplicate events, significantly reducing the number of reporting requests, lowering network bandwidth consumption, and making the reported data more valuable for analysis.

[0058] Based on the above technical solution, as an embodiment, the method for deduplicating data points may further include: storing the total number of reports in the current period and the statistical value of the number of reports of the target event in the current period into a local file; and clearing the total number of reports and the statistical value of the number of reports of each event stored in the previous period in the new period.

[0059] The total number of reports and the number of reports of the target event within the current period can be stored in a local file, using a lightweight local storage method for data persistence. During storage, a unique fingerprint of the target event can be bound to its corresponding number of reports, while the total number of reports for the current period is recorded separately, ensuring the relevance and independence of data storage. The local file storage uses a lightweight storage format, balancing data read / write efficiency with storage space usage, and is adapted to the local storage characteristics of mobile terminals. Even in cases of application process restarts or abnormal device exits, the reporting statistics for the period can be quickly restored, ensuring the continuity and accuracy of the count.

[0060] Within the new cycle, the total number of reports and the reporting count statistics for each event stored in the previous cycle are cleared. Through a cycle switching trigger mechanism, historical data cleanup is automatically performed when the new cycle starts, precisely locating and deleting the total number of reports and the reporting count statistics for all events from the previous cycle, without affecting the storage and statistics of data in the new cycle. The cleanup operation is completed automatically locally without manual intervention, avoiding redundant accumulation of historical statistical data, reducing local storage space usage, and achieving precise isolation of cycle data. This prevents statistical errors caused by cross-cycle data mixing, ensuring the independence and accuracy of the reported data statistics for each cycle.

[0061] The technical solution adopted in this application embodiment stores the total number of periodic reports and the number of event reports locally, and adopts a lightweight storage format adapted to mobile terminals. Even if the application restarts or the device malfunctions, the statistical data can be restored, ensuring continuous and accurate counting. The data of the previous period is automatically cleared in the new period without manual intervention. This avoids the accumulation of redundant historical data, reduces local storage occupation, and achieves precise isolation of periodic data to prevent cross-period statistical confusion, ensuring the independence and accuracy of the statistical data of each periodic tracking point.

[0062] Figure 2 This is an interactive diagram of the event tracking deduplication method provided in one embodiment of this application; wherein, the client uses reportAsync(map) as the asynchronous access entry point for event tracking, receives raw event tracking data of type HashMap, and after the data is accessed, first triggers the generateEventkey method of the data reporting manager (TDataReportManager), performs standardized encoding operations based on the raw event data, generates an event fingerprint that can uniquely identify the event, and provides a unified judgment basis for subsequent deduplication verification.

[0063] After the event fingerprint is generated, the `isAllowReport` deduplication verification step begins. This step retrieves the second event fingerprint set (mDayReportedSet) stored locally and checks whether the current event fingerprint already exists in this set to determine if the event is a duplicate event already reported that day. If the verification result indicates that the event has already been reported, it is directly determined that deduplication has been completed, and no further reporting is required. The event can be discarded, terminating the subsequent process. If the verification result indicates that reporting is allowed, the process proceeds to the next step, periodic merging and statistics.

[0064] Events that are allowed to be reported will be pushed into the mCurrentPeriodMap data structure. The merge(key,count) method is called to merge the counts. Using the event fingerprint as the unique key, the atomic accumulation operation is performed on the number of times the target event is triggered in the current reporting period, so as to realize the counting and merging of repeated events in the same period and reduce data redundancy.

[0065] After the merge count is completed, the event counting enters the timed scheduling phase. The scheduler uses a single-threaded timer to monitor the process periodically. The timer continuously checks whether the reporting time node has been reached according to a preset period. If the reporting period has not been reached, the event count merging continues in mCurrentPeriodMap; if the period has been reached, the periodic report is automatically triggered, and the periodReport reporting task is executed.

[0066] After executing the periodReport task, all event data to be reported stored in mCurrentPeriodMap can be read. Once the data reading is complete, the collection is immediately cleared to prepare for the event count merging in the next period. All event fingerprints read in this session are written to the mDayReportedSet collection to complete the fingerprint recording of events reported that day, avoiding duplicate reporting in the future. Simultaneously, these events to be reported are iterated and processed.

[0067] During the traversal, the core methods of the reporting execution module, reportHandle / mReporter, will be called uniformly to integrate the raw data of the event to be reported with the count information within the period and complete the final report. At this point, the entire process of client-side event deduplication and reporting is completed, and each module and data structure returns to its initial state, waiting for the access and processing of the next raw event.

[0068] The technical solution adopted in this application has the following beneficial effects: Completely decentralized: The deduplication logic is completed independently on the client side, without relying on the server and network interaction. The system's stability is not affected by network fluctuations, disconnections, or other scenarios.

[0069] Elastic resource management: Adopting event counting merging and lightweight storage architecture design, it greatly reduces memory and storage resource consumption and can be perfectly adapted to low-configuration mobile terminal devices.

[0070] Intelligent dynamic adaptation: It integrates event cycle merging and counting with reporting threshold circuit breaking mechanism to ensure complete collection of data points while achieving intelligent protection of system traffic, thus balancing data value and system security.

[0071] To facilitate better implementation of the embedding point deduplication method of this application, this application also provides an embedding point deduplication device based on the above-described embedding point deduplication method. The meanings of the terms used are the same as in the above-described embedding point deduplication method, and specific implementation details can be found in the description of the method embodiments.

[0072] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of the data embedding deduplication device provided in the embodiments of this application, wherein the data embedding deduplication device is applied to the client side, and the device includes: The data acquisition module 301 is used to acquire the raw event data of the target event collected by the embedded points; Data encoding module 302 is used to standardize and encode the original event data to obtain the first event fingerprint of the target event; The first inspection module 303 is used to check the circuit breaker status. When the circuit breaker is not triggered, it checks the total number of reports in the current period. The second inspection module 304 is used to check the second event fingerprint of the events already reported in the current period when the total number of reports is lower than the total number threshold, and abandon the reporting of the target event when the second event fingerprint includes the first event fingerprint. The event reporting module 305 is used to report the target event and update the total number of reports when the second event fingerprint does not include the first event fingerprint.

[0073] In one embodiment, the data encoding module 302 is specifically used to perform: Extract the core field that uniquely identifies the target event from the original event data; The core fields are sorted according to their key names based on the ordered mapping table to obtain the sorted core fields; The keys and values ​​of the sorting core field are encoded, and special characters are converted into the encoding format to obtain the encoded keys and encoded values; The encoded key and the encoded value are concatenated using a delimiter to obtain the first event fingerprint of the target event.

[0074] In one embodiment, the event reporting module 305 is specifically used to perform: Set a single-threaded timer; the initial delay and loop interval of the single-threaded timer are the same; Based on the single-threaded timer, the reporting task is automatically executed periodically to report events in batches; the events to be reported include the target events.

[0075] In one embodiment, the device further includes: The restart module is used to automatically restart the single-threaded timer when the reported task reports an error; The verification module is used to verify the validity of the new timer duration when the timer duration of the single-threaded timer is modified. The startup module is used to stop the old timed task and start the new timed task according to the new timed duration when the new timed duration is valid.

[0076] In one embodiment, the event reporting module 305 is specifically used to perform: The original event data of the target event is decoded to obtain the original key-value pairs; The atomic accumulation of the reporting count of the target event within the current period is achieved by using the merging method of the ordered mapping table; The reported number statistics and the original key-value pairs are reported.

[0077] In one embodiment, the device further includes: The storage module is used to store the total number of reports in the current period and the statistical value of the number of reports of the target event in the current period into a local file; The clearing module is used to clear the total number of reports and the report count statistics of each event stored in the previous cycle within the new cycle.

[0078] In one embodiment, the device further includes: The first abandon reporting module is used to abandon reporting the target event when the circuit breaker is triggered; The second abandon reporting module is used to abandon reporting the target event when the total number of reports is not lower than the total number threshold.

[0079] The technical solution adopted in this application embodiment generates and verifies event fingerprints through local standardized encoding, filtering duplicate events from the source, avoiding invalid data uploads, and significantly reducing network bandwidth consumption and server receiving pressure. Dual control of circuit breaker status and total threshold not only prevents sudden increases in reporting volume caused by high-frequency events but also accurately controls the total reporting volume per cycle, avoiding excessive memory consumption on the server due to massive fingerprint storage, and significantly reducing cloud operation and maintenance costs. Local real-time verification based on event fingerprints can flexibly adapt to dynamic scenarios such as network retransmissions and short-term repeated user operations. Compared with cloud solutions with fixed time windows, it effectively reduces the false positive rate of duplicate events and improves data accuracy. The total reporting volume is only updated and reported when the fingerprint verification indicates a new event, ensuring that the same event is reported only once within a single cycle. This reduces invalid reporting behavior and ensures the integrity and validity of the data, balancing resource conservation and data quality.

[0080] For specific limitations regarding the deduplication device, please refer to the limitations of the deduplication method mentioned above, which will not be repeated here. Each module in the aforementioned deduplication device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0081] In addition, this application also provides an electronic device, such as Figure 4 As shown, it illustrates the structural diagram of the electronic device involved in this application, specifically: The electronic device may include components such as a processor 401 with one or more processing cores and a memory 402 with one or more computer-readable storage media. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 401 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0082] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0083] In one embodiment, the electronic device further includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power equipment debugging circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0084] In one embodiment, the electronic device may further include an input unit 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0085] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402, thereby implementing the steps in any of the data collection and deduplication methods provided in this application embodiment.

[0086] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0087] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the methods described in any embodiment of this application.

[0088] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of this application.

[0089] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the methods described in any embodiment of this application.

[0090] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0091] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0092] Therefore, this application provides a computer-readable storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the data point deduplication methods provided in this application.

[0093] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0094] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0095] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the data point deduplication methods provided in this application, the beneficial effects that any of the data point deduplication methods provided in this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0096] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0097] The foregoing provides a detailed description of the embedding point deduplication method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for removing duplicate data points, characterized in that, Applied to the client side, including: Obtain the raw event data of the target event collected by the tracking points; The original event data is standardized and encoded to obtain the first event fingerprint of the target event; Check the circuit breaker status; if the circuit breaker has not been triggered, check the total number of reports submitted in the current period. When the total number of reported events is lower than the total number threshold, check the second event fingerprint of the events already reported in the current period. If the second event fingerprint includes the first event fingerprint, abandon the reporting of the target event. If the first event fingerprint is not included in the second event fingerprint, the target event is reported and the total number of reports is updated.

2. The method according to claim 1, characterized in that, The process of standardizing and encoding the original event data to obtain the first event fingerprint of the target event includes: Extract the core field that uniquely identifies the target event from the original event data; The core fields are sorted according to their key names based on the ordered mapping table to obtain the sorted core fields; The keys and values ​​of the sorting core field are encoded, and special characters are converted into the encoding format to obtain the encoded keys and encoded values; The encoded key and the encoded value are concatenated using a delimiter to obtain the first event fingerprint of the target event.

3. The method according to claim 1, characterized in that, The reported target event includes: Set a single-threaded timer; the initial delay and loop interval of the single-threaded timer are the same; Based on the single-threaded timer, the reporting task is automatically executed periodically to report events in batches; the events to be reported include the target events.

4. The method according to claim 3, characterized in that, The method further includes: When the reported task reports an error, the single-threaded timer is automatically restarted. When modifying the timing duration of the single-threaded timer, verify the validity of the new timing duration; When the new timeout period is valid, the old timeout task is stopped, and the new timeout task is started according to the new timeout period.

5. The method according to claim 1, characterized in that, The reported target event includes: The original event data of the target event is decoded to obtain the original key-value pairs; The atomic accumulation of the reporting count of the target event within the current period is achieved by using the merging method of the ordered mapping table; The reported number statistics and the original key-value pairs are reported.

6. The method according to claim 5, characterized in that, The method further includes: Store the total number of reports in the current period and the statistical value of the number of reports of the target event in the current period into a local file; In the new cycle, the total number of reports and the report count statistics for each event stored in the previous cycle are cleared.

7. The method according to claim 1, characterized in that, The method further includes: When the circuit breaker is triggered, the reporting of the target event is abandoned; If the total number of reported events is not lower than the total number threshold, the reporting of the target event is abandoned.

8. A device for removing weight at embedded points, characterized in that, Applied to the client side, including: The data acquisition module is used to acquire the raw event data of the target events collected by the embedded points; The data encoding module is used to standardize and encode the original event data to obtain the first event fingerprint of the target event; The first inspection module is used to check the circuit breaker status. When the circuit breaker is not triggered, it checks the total number of reports in the current period. The second inspection module is used to check the second event fingerprint of the events already reported in the current period when the total number of reports is lower than the total number threshold, and abandon the reporting of the target event when the second event fingerprint includes the first event fingerprint. The event reporting module is used to report the target event and update the total number of reports when the second event fingerprint does not include the first event fingerprint.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the embedding point deduplication method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps in the data embedding and deduplication method as described in any one of claims 1 to 7.

Images