Data acquisition method, system and device

Abstract

The invention relates to a data acquisition method, system and device, and relates to the technical field of data processing. The method comprises the following steps: acquiring collected data and calculating a corresponding data fingerprint; based on the data fingerprint and a data type corresponding to the collected data, filtering the collected data through a pre-deployed target filter to obtain candidate data; the data fingerprint of the candidate data does not exist in the target filter; different data types correspond to different filtering strategies of the target filter; sending the candidate data and the corresponding data fingerprints to a server; the server side is used for storing the target data in the candidate data into the database, and the data fingerprint of the target data is different from the data fingerprint of the stored data in the database. By adopting the method, the waste of resources can be reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data acquisition method, system and apparatus. Background Technology

[0002] In scenarios such as the Internet of Things (IoT), edge computing, and mobile applications, data collectors need to be deployed in resource-constrained environments for data acquisition. Related technologies primarily involve data collectors directly reporting the collected data to a server, which then performs deduplication. However, in edge environments with poor or unstable network quality, this approach leads to a large amount of identical, duplicate data being transmitted multiple times over the network, consuming bandwidth resources and causing unnecessary consumption of server computing and storage resources. Therefore, these data acquisition methods suffer from resource waste. Summary of the Invention

[0003] Therefore, it is necessary to provide a data acquisition method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can reduce resource waste in response to the above-mentioned technical problems.

[0004] Firstly, this application provides a data acquisition method, including:

[0005] Acquire the collected data and calculate the corresponding data fingerprint;

[0006] Based on the data fingerprint and the data type corresponding to the collected data, the collected data is filtered through a pre-deployed target filter to obtain candidate data; the data fingerprint of the candidate data does not exist in the target filter; different data types correspond to different filtering strategies of the target filter;

[0007] The candidate data and its corresponding data fingerprint are sent to the server; the server is used to store the target data in the candidate data into the database, and the data fingerprint of the target data is different from the data fingerprint of the data already saved in the database.

[0008] In conjunction with the first aspect, in one embodiment, the step of filtering the collected data based on the data fingerprint and the data type corresponding to the collected data using a pre-deployed target filter to obtain candidate data includes:

[0009] Based on the data fingerprint and the data type corresponding to the collected data, a filtering strategy for each of the collected data is determined; the filtering strategy includes filtering precision and filtering magnitude.

[0010] According to the filtering strategy of each of the collected data, the collected data is filtered through a pre-deployed target filter to obtain candidate data.

[0011] In conjunction with the first aspect, in one embodiment, the step of filtering the collected data according to the filtering strategy of each of the collected data, through a pre-deployed target filter, to obtain candidate data includes:

[0012] Based on the filtering strategy for each of the collected data, the target filter corresponding to each of the collected data is determined from a plurality of pre-deployed target filters;

[0013] According to the filtering strategy of each of the collected data, the collected data is filtered through the target filter corresponding to the collected data to obtain candidate data.

[0014] In conjunction with the first aspect, in one embodiment, sending the candidate data and its corresponding data fingerprint to the server includes:

[0015] Save each candidate data, the data fingerprint of each candidate data, and the sending status of each candidate data to a data file;

[0016] Based on the sending status in the data file, the candidate data and its corresponding data fingerprint are sent to the server.

[0017] In conjunction with the first aspect, in one embodiment, the method further includes:

[0018] In the event that the candidate data and its corresponding data fingerprint fail to be sent, the sending status corresponding to the failed candidate data in the data file is marked.

[0019] In response to the compensation mechanism, based on the data file, all the candidate data that failed to be sent and their corresponding data fingerprints are sent to the server.

[0020] In conjunction with the first aspect, in one embodiment, the method further includes:

[0021] Under the condition that the preset conditions are met, construct and store the image file of the target filter;

[0022] If the terminal restarts, load the image file to start the target filter;

[0023] Based on the data file, extract the data fingerprint after the timestamp of the image file, and update the target filter based on the data fingerprint.

[0024] In conjunction with the first aspect, in one embodiment, the step of constructing and storing the image file of the target filter under preset conditions includes:

[0025] If the amount of data processed by the target filter meets a preset threshold, or if the filtering time of the target filter meets a preset period, the bit array of the target filter is serialized to obtain the serialized target filter.

[0026] For the serialized target filter, an image file is constructed and stored.

[0027] In conjunction with the first aspect, in one embodiment, before acquiring the collected data and calculating the corresponding data fingerprint, the method further includes:

[0028] The data to be collected is processed to obtain multiple data points, and the data type of each data point is determined.

[0029] Based on the data type of each of the collected data, calculate the data fingerprint corresponding to each of the collected data for the full data or key business fields.

[0030] Secondly, this application provides a data acquisition system, comprising:

[0031] On the vehicle end, the terminal is used to implement the steps of the method described in any of the above embodiments;

[0032] The server is connected to the vehicle terminal and is used to receive candidate data and its corresponding data fingerprint sent by the vehicle terminal. Based on the local database and the data fingerprint, the server performs deduplication on the candidate data and stores the target data obtained after deduplication in the local database.

[0033] Thirdly, this application also provides a data acquisition device, comprising:

[0034] The data acquisition module is used to acquire collected data and calculate the corresponding data fingerprint;

[0035] The data filtering module is used to filter the collected data based on the data fingerprint and the data type corresponding to the collected data through a pre-deployed target filter to obtain candidate data; the data fingerprint of the candidate data does not exist in the target filter; different data types correspond to different filtering strategies of the target filter;

[0036] A data sending module is used to send the candidate data and its corresponding data fingerprint to the server; the server is used to store the target data in the candidate data into a database, wherein the data fingerprint of the target data is different from the data fingerprint of the data already stored in the database.

[0037] The aforementioned data acquisition methods, devices, computer equipment, computer-readable storage media, and computer program products involve a terminal acquiring collected data and calculating corresponding data fingerprints. Based on the data fingerprints and the data types corresponding to the collected data, a pre-deployed target filter is used to filter the collected data to obtain candidate data. The data fingerprints of the candidate data are not present in the target filter. Different data types correspond to different filtering strategies in the target filter. Managing the collected data through data fingerprints represents the content characteristics of the original data, avoiding the performance overhead of directly comparing the entire dataset. Furthermore, different filtering strategies are used locally on the terminal to perform preliminary deduplication on collected data of different data types, specifically reducing redundant data. Further, the terminal sends the candidate data and its corresponding candidate data fingerprints to the server. The server stores the target data from the candidate data in a database. The data fingerprints of the target data differ from the data fingerprints already stored in the database. The terminal sends the preliminary deduplicated candidate data and data fingerprints to the server, where the server performs secondary deduplication. This not only reduces network transmission resource consumption and server-side deduplication resource waste but also improves the accuracy of the final target data. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a diagram illustrating the application environment of a data acquisition method in one embodiment.

[0040] Figure 2 This is a flowchart illustrating a data acquisition method in one embodiment;

[0041] Figure 3 This is a flowchart illustrating the data acquisition method in another embodiment;

[0042] Figure 4 This is a structural block diagram of a data acquisition device in one embodiment;

[0043] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] As described in the background section, the data acquisition methods of related technologies suffer from resource waste. The inventors discovered that this problem arises because data collectors need to be deployed in resource-constrained environments in scenarios such as the Internet of Things (IoT), edge computing, and mobile applications. Traditional data acquisition schemes mainly fall into two categories. One involves the data collector directly reporting data, with the server handling deduplication. While the acquisition logic at the collector end is simple, it leads to a large amount of duplicate data consuming network bandwidth between the collector and the server, as well as server processing resources, resulting in low efficiency and significant waste of network bandwidth. In edge environments with poor or unstable network quality, data transmission failures and retry mechanisms are commonplace. This scheme causes a large amount of identical duplicate data to be transmitted multiple times over the network. This not only consumes already valuable bandwidth but can also lead to a vicious cycle of network congestion, severely impacting the real-time reporting of other critical data. Furthermore, it results in unnecessary consumption of server-side computing and storage resources, as the server needs to perform duplicate checks for each incoming data entry. This means that massive amounts of duplicate data, which could have been intercepted at the source, consume a significant amount of CPU (Central Processing Unit) computation cycles (for fingerprint comparison or database queries) and temporary storage space on the server side, leading to reduced efficiency in processing valid data and increased hardware costs. There is also the issue of system scalability bottlenecks; as the number of collection nodes and data volume increases, the resource consumption caused by duplicate data increases linearly or even exponentially. The server side can easily become the performance bottleneck of the entire system, making it difficult to support large-scale deployments. Furthermore, there is the problem of increased data latency; when processing large amounts of redundant data, the processing path for valid data is prolonged, leading to increased end-to-end latency from data generation to availability, failing to meet the needs of business scenarios with high real-time requirements.

[0046] Another approach involves using a built-in database for deduplication at the data acquisition end. While this reduces redundant data reported from the acquisition end to the server, the database's frequent writes, resource consumption, and operational complexity do not meet the lightweight deployment requirements of the acquisition end, and its performance is poor when the acquisition end crashes and needs to recover. This data acquisition method suffers from write performance bottlenecks and resource contention. Although embedded databases are lighter than large databases, their storage engines, based on B-trees (multi-way balanced search trees) or LSM trees (log-structured merge trees), still generate significant I / O (input / output) overhead and write amplification effects when dealing with the high-frequency, small-batch, and continuous writes characteristic of data acquisition scenarios. This can easily create write bottlenecks on low-performance edge devices and compete with the acquisition process for limited CPU and I / O resources. This method also presents operational complexity and reliability risks, as the database files need to maintain transaction consistency, logs, and indexes, which are susceptible to corruption. If the database file is corrupted due to an unexpected power outage, the entire data collector may be paralyzed. The recovery process is complex and does not meet the requirements of simplified operation and maintenance and high reliability, which contradicts the demand for unmanned operation and maintenance at the data collection end. This method also suffers from low efficiency in storage space reclamation and management. For time-series data, when old data needs to be cleaned up, the database deletion and space reclamation operations are usually heavyweight, causing service interruption or performance degradation during this period, affecting the real-time performance of data collection. In addition, this method also suffers from slow recovery speed. After the data collector restarts, the embedded database needs to load indexes and check logs to restore to a ready state. This process is much slower than simple file operations, prolonging the system failure recovery time. Therefore, existing technologies cannot meet the core requirements of lightweight deployment, simplified operation and maintenance, and high performance of the data collection end while ensuring data reliability and uniqueness. An innovative solution is urgently needed to balance resource consumption and data processing accuracy.

[0047] For the reasons mentioned above, this application provides a data acquisition method. The terminal acquires the collected data and calculates the corresponding data fingerprint. Based on the data fingerprint and data type, the terminal performs targeted filtering on the collected data using a pre-deployed target filter and a filtering strategy corresponding to the collected data to obtain candidate data. The candidate data and its corresponding data fingerprint are then sent to the server so that the server can store the target data in the candidate data in the database. This avoids the terminal transmitting all the collected data to the server, reducing the waste of network resources and server computing resources.

[0048] The data acquisition method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, the environment includes a terminal 101 and a server 102. The terminal 101 communicates with the server 102 via a network. A data storage system stores the data that the server 102 needs to process. This data storage system can be integrated into the server 102 or located in the cloud or on other network servers. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0049] based on Figure 1 In the application environment shown, terminal 101 acquires collected data and calculates the corresponding data fingerprint. Based on the data fingerprint and the data type of the collected data, it filters the collected data through a pre-deployed target filter to obtain candidate data. The data fingerprint of the candidate data does not exist in the target filter. Different data types correspond to different filtering strategies of the target filter. Further, terminal 101 sends the candidate data and its corresponding data fingerprint to server 102. Server 102 stores the target data in the candidate data in a database. The data fingerprint of the target data is different from the data fingerprint of the data already stored in the database.

[0050] In one exemplary embodiment, such as Figure 2 As shown, a data acquisition method is provided, which can be applied to... Figure 1 Taking terminal 101 as an example, the explanation includes the following steps S202 to S206. Wherein:

[0051] Step S202: Acquire the collected data and calculate the corresponding data fingerprint.

[0052] The collected data can refer to the raw business data that the terminal device actively or passively acquires during operation, and can come from vehicle sensors, user operation logs, system status information or other external input sources.

[0053] In this context, a data fingerprint can be a unique identifier generated from the collected data using a hash algorithm, used to quickly compare and determine whether the data is duplicated. In this embodiment, the data fingerprint is computationally efficient and can represent the content characteristics of the original data, avoiding the performance overhead caused by directly comparing the full amount of data.

[0054] Optionally, the terminal acquires the collected data from the data source and calculates the data fingerprint corresponding to each piece of collected data, laying the groundwork for subsequent filtering processing of the collected data. It is understood that the terminal is equipped with a data collector, which collects data from the data source to obtain the collected data.

[0055] Step S204: Based on the data fingerprint and the data type corresponding to the collected data, the collected data is filtered through a pre-deployed target filter to obtain candidate data.

[0056] In this case, the data fingerprint of the candidate data does not exist in the target filter.

[0057] Different data types correspond to different filtering strategies of the target filter; the data type can be the type of data collected, which is divided by dimensions such as business needs, importance, and data value.

[0058] The target filter can be a Bloom filter, which is a probabilistic data structure consisting of a bit array and multiple independent hash functions, used to determine whether an element may exist in the set or definitely does not exist.

[0059] Candidate data can be the collected data that has been filtered by the target filter and is retained. The data fingerprints corresponding to these data are not matched in the target filter, so they are considered as potential new data or non-duplicate data, and have the value of being uploaded to the server for further verification.

[0060] Optionally, the terminal determines the filtering strategy required by the target filter for the collected data based on the data fingerprint and data type of the collected data. The data fingerprint is input into multiple hash functions of the pre-deployed target filter, and the terminal checks whether all positions in the target filter's bit array are 1. Based on the determination result, the collected data is filtered to obtain candidate data. Specifically, if any bit in the bit array is 0, the data fingerprint is determined not to exist in the target filter's history, and the collected data corresponding to that data fingerprint is determined as candidate data. If all positions in the bit array are 1, the data fingerprint is determined to exist in the target filter's history, and the collected data corresponding to that data fingerprint is considered redundant data. It is understood that redundant data can be retained according to actual needs. In some embodiments, redundant data can be temporarily stored on the terminal. If the server needs to retrieve redundant data, the terminal can respond to the server's instruction and send the redundant data to the server.

[0061] Step S206: Send the candidate data and its corresponding data fingerprint to the server.

[0062] The server is used to store the target data from the candidate data into the database. The data fingerprint of the target data is different from the data fingerprint of the data already saved in the database. The server's database creates a unique index for the data fingerprint field and uses the "INSERT...ON DUPLICATE KEY UPDATE..." statement to automatically deduplicate or update the candidate data when inserting candidate data and its data fingerprint.

[0063] Optionally, candidate data obtained by initial deduplication can be processed locally on the terminal using a target filter. The candidate data and its corresponding data fingerprint can then be asynchronously sent to the server for secondary deduplication. The server can then obtain and store the target data, thereby reducing unnecessary network transmission and server load.

[0064] In the aforementioned data acquisition method, the terminal acquires the collected data and calculates the corresponding data fingerprint. Based on the data fingerprint and the data type of the collected data, a pre-deployed target filter is used to filter the collected data to obtain candidate data. The data fingerprint of the candidate data does not exist in the target filter. Different data types correspond to different filtering strategies of the target filter. Managing the collected data through data fingerprints can represent the content characteristics of the original data, avoiding the performance overhead of directly comparing the entire dataset. Furthermore, different filtering strategies are used locally on the terminal to perform preliminary deduplication on collected data of different data types, specifically reducing redundant data. Further, the terminal sends the candidate data and its corresponding candidate data fingerprint to the server. The server stores the target data from the candidate data in a database. The data fingerprint of the target data differs from the data fingerprint of the data already stored in the database. The terminal sends the preliminary deduplicated candidate data and data fingerprint to the server, where the server performs secondary deduplication. This not only reduces the resource consumption of network transmission and the wasted computing resources of server-side deduplication but also improves the accuracy of the final target data.

[0065] In an exemplary embodiment, step S204, based on the data fingerprint and the data type corresponding to the collected data, filters the collected data using a pre-deployed target filter to obtain candidate data, including:

[0066] Based on the data fingerprint and the data type corresponding to the collected data, a filtering strategy for each of the collected data is determined; according to the filtering strategy for each of the collected data, the collected data is filtered through a pre-deployed target filter to obtain candidate data.

[0067] The filtering strategy can refer to a set of data deduplication rules determined by the terminal based on the data type of the collected data. The filtering strategy includes filtering accuracy and filtering scale. Filtering accuracy refers to the false positive rate control level of the filter when judging whether data is duplicated, that is, the probability of a Bloom filter producing a false positive, usually expressed as a percentage (such as 1%, 0.1%, 0.001%). The lower the value, the higher the accuracy, but the greater the memory overhead. Filtering scale refers to the data range or abstraction level covered by the filtering operation, reflecting which dimensions are used for deduplication, and determining whether the filtering is based on single data, data fragments, or group features after semantic clustering.

[0068] Optionally, the terminal determines the filtering precision and filtering level of each collected data based on the data fingerprint, the data type corresponding to the collected data, and the business or scenario corresponding to the data type, and then determines the filtering strategy for each collected data. For example, the data type of the collected data represents the data value of the collected data. The higher the data value of the collected data, the higher the required filtering precision. If the data type belongs to the first value type, a low-precision layer filtering precision (1% false positive rate) and a first-level filtering level that can be loosely filtered are selected to increase the filtering granularity and perform high-speed judgment. If the data type belongs to the second value type, a medium-precision layer filtering precision (0.1% false positive rate) is selected, and a second-level filtering level with finer granularity is selected for balanced judgment. If the data type belongs to the third value type, a high-precision layer filtering precision (0.001% false positive rate) is selected, and a third-level filtering level with the finest granularity is selected for accurate judgment. The data values ​​of the first value type to the third value type are arranged in ascending order.

[0069] Furthermore, in some embodiments, when the collected data belongs to the vehicle-side data collection scenario, the data type is determined based on the driving scenario. The terminal dynamically adjusts the filtering strategy according to the driving scenario. For example, in a highway cruise scenario, a lenient filtering strategy is chosen, primarily focusing on filtering data collected from vehicles ahead. In a congested urban scenario with complex road conditions, strict filtering is applied to data collected from a large number of stationary or slow-moving objects in the scenario. If the collected data is safety-related and thus of high value, and the driving scenario involves emergency braking, the terminal can choose to disable filtering and upload all collected data to facilitate server-side assistance in assessing road conditions. Further, the terminal filters the collected data according to the filtering strategies for each type of data, using pre-deployed target filters to obtain candidate data.

[0070] In this embodiment, by combining the data type of the collected data with the corresponding business scenarios and data value, differentiated filtering precision and filtering scale are dynamically determined to form an intelligent and adaptive filtering strategy. High-value data is filtered with high precision and fine granularity or directly uploaded in full to ensure that key information is not lost; low-value data is filtered with low precision and coarse granularity to improve processing efficiency. At the same time, according to driving scenarios, such as highway cruising, urban congestion, and emergency braking, the deduplication logic is adjusted in real time to achieve an optimal balance between resource consumption and data integrity. This significantly improves the intelligence level of data collection, transmission efficiency, and system reliability, and is also applicable to complex and ever-changing in-vehicle edge computing environments.

[0071] In an exemplary embodiment, the step described in the above embodiment, filtering the collected data according to the filtering strategy of each of the collected data through a pre-deployed target filter to obtain candidate data, includes:

[0072] According to the filtering strategy for each of the collected data, a target filter corresponding to each of the pre-deployed target filters is determined; according to the filtering strategy for each of the collected data, the collected data is filtered through the target filter corresponding to the collected data to obtain candidate data.

[0073] Understandably, multiple target filters can be pre-deployed in the terminal, each focusing on different deduplication types. These target filters can be connected in series to gradually improve filtering accuracy, or they can be deployed in other ways without specific limitations. Among them, the target filters can filter data collected based on data fingerprints with similar content, data collected based on data fingerprints with similar semantics, and data collected based on data fingerprints with completely similar fingerprints, etc.

[0074] Optionally, the terminal determines a target filter that can satisfy the filtering strategy of each of the collected data from a plurality of pre-deployed target filters according to the filtering strategy of each of the collected data. Then, the terminal performs targeted filtering processing on the collected data through the target filter corresponding to the collected data according to the filtering strategy of each of the collected data to obtain candidate data.

[0075] In this embodiment, by pre-deploying multiple target filters targeting different deduplication types on the terminal, and supporting flexible selection or chaining based on the filtering strategy of the collected data, hierarchical filtering of multi-dimensional data, from complete duplication to content similarity and even semantic similarity, is achieved. Different target filters work collaboratively as needed, which can efficiently remove obviously redundant data and accurately identify potential duplication, significantly improving the comprehensiveness and intelligence of deduplication. In addition, by dynamically matching the filtering path with data value and scenario, resource utilization is optimized while ensuring the integrity of key data, making it suitable for lightweight and reliable deduplication needs of high-concurrency and diverse data collection.

[0076] In an exemplary embodiment, step S206 sends the candidate data and its corresponding data fingerprint to the server, including:

[0077] Save each candidate data, its data fingerprint, and its transmission status to a data file; based on the transmission status in the data file, send the candidate data and its corresponding data fingerprint to the server.

[0078] The sending status can represent the current status of a candidate data and its data fingerprint being sent to the server, such as sent, in the process of sending, or not sent.

[0079] The data file can be a structured file stored in the terminal's local persistent storage medium, used to temporarily store all candidate data and related metadata.

[0080] Optionally, the terminal saves each candidate data, the data fingerprint of each candidate data, and the sending status of each candidate data to a data file. It can be understood that the data file can be a local file that records data according to a preset structured format. The content stored may also include the ID (Identifier), timestamp, etc. of each candidate data. For example, the structured format of the data file is: unique ID | data fingerprint (MD5, a hash value) | data content | status (to be sent / sent) | timestamp. Further, when the data sending conditions are met (such as reaching a preset data sending cycle), the terminal sends the candidate data with a sending status of not sent and sending failure, along with their corresponding data fingerprints, to the server based on the sending status in the data file.

[0081] In this embodiment, by persistently storing candidate data, data fingerprints, and transmission status information in a structured format in a local data file, and conditionally uploading them to the server based on the transmission status, the reliability and integrity of data transmission are achieved. Even in scenarios such as network anomalies or terminal restarts, the data to be sent or retransmitted can still be accurately identified based on the data file, effectively avoiding data loss and duplicate uploads, and improving the fault tolerance and communication efficiency of the data acquisition process.

[0082] In one exemplary embodiment, the above data acquisition method further includes:

[0083] In the event that the candidate data and its corresponding data fingerprint fail to be sent, the sending status of the failed candidate data in the data file is marked; in response to the compensation mechanism, all failed candidate data and their corresponding data fingerprints are sent to the server according to the data file.

[0084] The compensation mechanism can be a fault-tolerant design mechanism used to automatically trigger a retransmission process after data transmission failure, network interruption, or system anomaly, ensuring that data is not lost.

[0085] Optionally, in the event of a failure to send candidate data and its corresponding data fingerprint, the terminal marks the sending status of the failed candidate data in the data file, such as updating the sending status of the failed candidate data to "sending failed." In response to the compensation mechanism, the terminal sends all candidate data in the data file with a sending status of "sending failed" and its corresponding data fingerprint to the server, thus preventing data loss. It is understood that the triggering condition for the compensation mechanism can be a preset compensation period, such as an independent thread scanning the data file every 60 seconds to identify failed candidate data and resend it to the server, or it can be set according to data sending requirements; no specific limitation is made in this regard.

[0086] In this embodiment, by marking the transmission status in the data file when transmission fails and automatically retransmitting the failed candidate data based on a compensation mechanism, the reliability and integrity of data transmission are achieved. Through periodic scanning or event-triggered compensation strategies, candidate data with abnormal interruptions is effectively identified and retransmitted, avoiding data loss due to network fluctuations or system failures.

[0087] In one exemplary embodiment, the above data acquisition method further includes:

[0088] Under the condition that the preset conditions are met, build and store the image file of the target filter; when the terminal restarts, load the image file to start the target filter; extract the data fingerprint after the timestamp of the image file according to the data file, and update the target filter according to the data fingerprint.

[0089] The image file can be a persistent snapshot of the current state of the target filter, containing the complete binary data of its bit array and additional metadata (such as timestamps, version numbers, hash function parameters, etc.), and is usually stored locally on the terminal in serialized form.

[0090] Optionally, under preset conditions, the terminal constructs and stores an image file of the target filter as a backup, facilitating rapid reconstruction of the target filter later. In the event of a terminal restart, whether normal or abnormal, the terminal loads the image file to launch the latest version of the target filter corresponding to the image file. Furthermore, based on the timestamps of each candidate data stored in the data file, the terminal extracts the data fingerprints after the timestamps in the image file and updates the target filter accordingly, achieving state synchronization.

[0091] In this embodiment, by generating an image file of the target filter when preset conditions are met, and loading the image after the terminal restarts to quickly rebuild the target filter, the problem of loss of the target filter's filtering records due to terminal crashes and restarts is avoided. In addition, by combining the timestamp information of each candidate data in the data file, the data fingerprints newly added after the restart are incrementally updated, thereby synchronizing the status of the rebuilt target filter with the local data processing progress and ensuring the continuity and consistency of the terminal's deduplication logic.

[0092] In an exemplary embodiment, the steps of the above embodiments, which involve constructing and storing a mirror file of the target filter under preset conditions, include:

[0093] If the amount of data processed by the target filter meets the preset threshold, or if the target filter's filtering time meets the preset period, the bit array of the target filter is serialized to obtain the serialized target filter; for the serialized target filter, an image file is built and stored.

[0094] The preset quantity threshold can be a data processing threshold used to trigger the generation of the target filter image file; the preset period can be a time interval set according to the actual backup needs of the target filter (such as every 60 seconds, every hour, etc.) to periodically trigger the generation operation of the target filter image file.

[0095] The bit array can be the core storage structure of the target filter. It is an array of several bits, initially all of which are 0. Whenever a data fingerprint is inserted, it is mapped to different positions in the bit array through multiple hash functions, and the corresponding bit is set to 1.

[0096] Optionally, when the number of data processed by the target filter meets a preset threshold, or when the target filter's filtering time meets a preset period, the terminal serializes the target filter's bit array, that is, converts the target filter's memory object into a byte stream to obtain the serialized target filter. Further, the terminal performs a snapshot operation on the serialized target filter to obtain an image file and stores it; for example, every time the target filter successfully processes 100 data entries or every 30 seconds, the terminal serializes the target filter's bit array and snapshots it to the bloom_filter.bf.img file.

[0097] In this embodiment, by automatically serializing the bit array of the target filter and generating a mirror file when the amount of data processed by the Bloom data reaches a preset threshold or a preset period has elapsed, the deduplication state is persisted at regular intervals. This avoids the performance overhead caused by frequent reads and writes, and prevents state loss due to power outages or abnormal restarts. It ensures that lightweight terminals can still maintain data deduplication capabilities efficiently and reliably under resource-constrained conditions, thereby improving the stability and recovery capabilities of the system.

[0098] In an exemplary embodiment, before step S202 acquires the collected data and calculates the corresponding data fingerprint, the method further includes:

[0099] The data to be collected is processed to obtain multiple data points, and the data type of each data point is determined. Based on the data type of each data point, the data fingerprint corresponding to each data point is calculated for the full data or key business fields of each data point.

[0100] The full data can refer to the complete original content of the collected data, including all fields and information. For example, a complete vehicle CAN (Controller Area Network) message includes all fields such as ID, data field, timestamp, and length.

[0101] Key business fields can refer to the most representative fields in the collected data that best reflect the essential differences in the data, rather than all of them. For example, in fault code records, only the fault code type and occurrence time are extracted as key fields.

[0102] Optionally, the terminal collects and processes the data to be collected from the data source to obtain multiple data entries. By evaluating the data value, business requirements, or scenarios of the collected data, the data type of each data entry is comprehensively determined. Furthermore, based on the data type of each data entry, a data fingerprint is calculated for the full data or key business fields of each data entry. It can be understood that the data fingerprint can be a hash or MD5 (Message-D Digest Algorithm 5, a hash function) of the full data entry, or a combined hash of key business fields, which can be flexibly selected according to the business definition of repetition.

[0103] In this embodiment, by combining the data value, business needs, and operational scenarios of the collected data, the data type is intelligently determined, and the data fingerprint is dynamically calculated based on either the full dataset or key business fields. This achieves refinement and differentiation in the deduplication process. For example, high-value collected data uses the full dataset to calculate the data fingerprint to ensure accuracy, while low-value or high-frequency collected data uses key field combination hashing to improve efficiency. This approach reduces computational and storage overhead while ensuring deduplication effectiveness, enhances the system's adaptability to diverse data types, improves the intelligence level and resource utilization efficiency of vehicle-side data collection, and is suitable for complex and dynamic in-vehicle edge computing environments.

[0104] In one exemplary embodiment, the above data acquisition method further includes:

[0105] The target filter is updated based on the data fingerprint of the candidate data so that the updated target filter contains the data fingerprint of the candidate data.

[0106] Optionally, after the terminal filters the collected data according to the target filter, the resulting candidate data is further updated based on the data fingerprint of the candidate data, so that the updated target filter contains the data fingerprint of the candidate data.

[0107] In this embodiment, after the collected data filtering is completed, the data fingerprints of the filtered candidate data are synchronously updated to the target filter to ensure that the processed non-duplicate data is recorded and to avoid subsequent duplicate reporting. By dynamically maintaining the state of the target filter, the consistency and continuity of the deduplication logic are effectively improved, preventing the same data from entering the candidate set multiple times, reducing redundant transmission and server pressure, and enhancing the deduplication accuracy and operating efficiency of the target filter in continuous data collection scenarios.

[0108] In one exemplary embodiment, a data acquisition system is provided, including a terminal and a server:

[0109] The terminal is used to implement the steps of any of the methods in the above embodiments; the server terminal communication connection is used to receive candidate data and its corresponding data fingerprint sent by the terminal, perform deduplication processing on the candidate data according to the local database and the data fingerprint, and store the target data obtained after deduplication processing in the local database.

[0110] The data collector comprises a fingerprint calculation module, a target filter module, a file storage module, a persistence and recovery module, and a communication module. The fingerprint calculation module calculates a unique data fingerprint for each piece of data collected. The target filter module performs existence determination based on the data fingerprint in memory. The file storage module uses a rolling file as the persistent storage medium for data, data fingerprints, and transmission status. The persistence and recovery module periodically serializes the target filter's state into a mirror file and saves it; upon collector restart, it quickly and accurately restores the target filter's state by loading the mirror file and replaying the data file. The communication module is responsible for asynchronously sending data and fingerprints to the server and managing the retransmission mechanism for failures.

[0111] The server-side component includes a receiving and deduplication module and a data persistence module. The receiving and deduplication module receives data and fingerprints, and performs precise deduplication queries based on database uniqueness constraints. The data persistence module persistently stores the unique data and its fingerprints in the database.

[0112] Understandably, the structural diagram of the data acquisition system can be consulted further. Figure 1 The server's local database indexes each data fingerprint. Therefore, when the server filters target data from candidate data, it queries the local database to see if the candidate data has been stored, thereby filtering out target data that has not yet been stored in the local database.

[0113] In one exemplary embodiment, such as Figure 3 As shown, another data acquisition method is provided, applied to the data acquisition system described in the above embodiments, including:

[0114] Step 1: The terminal's data collector, Agent, starts up and initializes or restores the target filter;

[0115] Step 2: The terminal's data collector collects data in batches in a loop and calculates the data fingerprint.

[0116] Step 3: The terminal queries the target filter based on the data fingerprint of the collected data. If the target filter indicates "may exist", the data is discarded. If the target filter indicates "definitely does not exist", the next step is continued.

[0117] Step 4: The terminal appends the filtered data, data fingerprint, and transmission status to a local file (data file) and adds the data fingerprint to the target filter in memory. If the terminal's data collector crashes or restarts for other reasons, the terminal checks and loads the target filter image file into memory. It then scans the data file and adds any new data fingerprints added after the specified time point in the image file to the target filter, completing the state synchronization.

[0118] Step 5: Periodically snapshot the state of the target filter to the image file for persistence, resulting in a local image file .bf.img.

[0119] Step 6: The terminal asynchronously sends the collected data to the server. If the attempt fails, it marks the data in the local file and initiates a retry.

[0120] Step 7: The server receives the data and fingerprint.

[0121] Step 8: The server queries the database. If the fingerprint already exists, it discards or updates the data; otherwise, it stores the new data.

[0122] This embodiment employs a collaborative architecture of file media and target filters, combined at the resource-constrained acquisition end, to achieve lightweight data deduplication and reliable temporary storage. At the resource-constrained acquisition end, simple files (such as log files) are used as the core persistent storage medium, replacing the embedded database that requires complex operation and maintenance. The introduction of the target filter, a probabilistic data structure, performs rapid existence judgment on data in memory based on data fingerprints, intercepting most duplicate data at the source; this overcomes the contradiction between lightweight deployment and data reliability; and solves the write performance bottleneck, operational complexity, and high resource consumption problems brought by databases at the acquisition end. Furthermore, a dual deduplication guarantee mechanism of end-to-cloud collaboration is adopted to construct a clearly defined two-level deduplication system. The acquisition end uses the target filter for fast, probabilistic pre-deduplication, filtering out most duplicate data. The server uses database unique constraints and other methods for precise, deterministic final deduplication, ensuring the global uniqueness of the data. This maximizes efficiency; the end-side coarse screening reduces the pressure on the network and server; the cloud-side fine filtering ensures final accuracy. Resource optimization achieves optimal configuration of overall system (end, network, cloud) resource consumption. Furthermore, the persistence and state recovery mechanism of the target filter periodically serializes the target filter bit array in memory and snapshots it to an image file, achieving persistence. When the collector restarts, the target filter state is quickly and accurately reconstructed by primarily loading the image file and secondarily replaying the data file log. This solves the fatal weakness of the target filter as a memory-volatile data structure in engineering practice. It ensures the continuity and correctness of the deduplication state after the collector experiences a crash and restart, giving the solution industrial-grade robustness. In addition, this embodiment also constructs a closed-loop data reliability mechanism based on files and fingerprints. Data fingerprints are written to a file along with the original data, and the sending status is recorded; the file serves as a reliable data source for failure retries; and the file also serves as a data log for target filter recovery. With a minimal file, four major functions—data persistence, state tracking, retry foundation, and recovery log—are simultaneously implemented, forming a highly reliable, self-contained, lightweight data management closed loop.

[0123] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0124] Based on the same inventive concept, this application also provides a data acquisition device for implementing the data acquisition method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data acquisition device embodiments provided below can be found in the limitations of the data acquisition method described above, and will not be repeated here.

[0125] In one exemplary embodiment, such as Figure 4 As shown, a data acquisition device 400 is provided, including: a data acquisition module 401, a data filtering module 402, and a data transmission module 403, wherein:

[0126] Data acquisition module 401 is used to acquire collected data and calculate the corresponding data fingerprint;

[0127] Data filtering module 402 is used to filter the collected data based on the data fingerprint and the data type corresponding to the collected data through a pre-deployed target filter to obtain candidate data; the data fingerprint of the candidate data does not exist in the target filter; different data types correspond to different filtering strategies of the target filter;

[0128] The data sending module 403 is used to send candidate data and its corresponding data fingerprint to the server; the server is used to store the target data in the candidate data into the database, and the data fingerprint of the target data is different from the data fingerprint of the data already saved in the database.

[0129] Furthermore, in one embodiment, the data filtering module 402 is also used to determine a filtering strategy for each of the collected data based on the data fingerprint and the data type corresponding to the collected data; the filtering strategy includes filtering precision and filtering magnitude; according to the filtering strategy for each of the collected data, the collected data is filtered by a pre-deployed target filter to obtain candidate data.

[0130] Furthermore, in one embodiment, the data filtering module 402 is further configured to determine the target filter corresponding to each of the collected data from a plurality of pre-deployed target filters according to the filtering strategy of each of the collected data; and to filter the collected data through the target filter corresponding to the collected data according to the filtering strategy of each of the collected data to obtain candidate data.

[0131] Furthermore, in one embodiment, the data sending module 403 is also used to save each candidate data, the data fingerprint of each candidate data, and the sending status of each candidate data to a data file; and send the candidate data and its corresponding data fingerprint to the server based on the sending status in the data file.

[0132] Furthermore, in one embodiment, the data sending module 403 is also used to mark the sending status of the failed candidate data in the data file when the candidate data and its corresponding data fingerprint fail to be sent; in response to the compensation mechanism, according to the data file, send all the failed candidate data and their corresponding data fingerprints to the server.

[0133] Furthermore, in one embodiment, the data acquisition device 400 further includes a filter update module, which is used to construct and store an image file of the target filter when preset conditions are met; load the image file to start the target filter when the terminal restarts; extract the data fingerprint after the timestamp of the image file according to the data file, and update the target filter according to the data fingerprint.

[0134] Furthermore, in one embodiment, the data filtering module 402 is also used to serialize the bit array of the target filter to obtain the serialized target filter when the amount of data processed by the target filter meets a preset quantity threshold or the filtering time of the target filter meets a preset period; and to build and store an image file for the serialized target filter.

[0135] Furthermore, in one embodiment, the data acquisition module 401 is also used to perform acquisition processing on the data to be collected, obtain multiple pieces of acquired data, and determine the data type of each piece of acquired data; according to the data type of each piece of acquired data, calculate the data fingerprint corresponding to each piece of acquired data for the full data or key business fields of each piece of acquired data respectively.

[0136] Each module in the aforementioned data acquisition device 400 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0137] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a data acquisition method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0138] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0140] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0141] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data acquisition method, characterized in that, Applied to a terminal, the method includes: Acquire the collected data and calculate the corresponding data fingerprint; Based on the data fingerprint and the data type corresponding to the collected data, the collected data is filtered through a pre-deployed target filter to obtain candidate data; the data fingerprint of the candidate data does not exist in the target filter; different data types correspond to different filtering strategies of the target filter; The candidate data and its corresponding data fingerprint are sent to the server; the server is used to store the target data in the candidate data into the database, and the data fingerprint of the target data is different from the data fingerprint of the data already saved in the database.

2. The method according to claim 1, characterized in that, Based on the data fingerprint and the data type corresponding to the collected data, the collected data is filtered using a pre-deployed target filter to obtain candidate data, including: Based on the data fingerprint and the data type corresponding to the collected data, a filtering strategy for each of the collected data is determined; the filtering strategy includes filtering precision and filtering magnitude. According to the filtering strategy of each of the collected data, the collected data is filtered through a pre-deployed target filter to obtain candidate data.

3. The method according to claim 2, characterized in that, The step of filtering the collected data according to the filtering strategy for each of the collected data, and obtaining candidate data through a pre-deployed target filter, includes: Based on the filtering strategy for each of the collected data, the target filter corresponding to each of the collected data is determined from a plurality of pre-deployed target filters; According to the filtering strategy of each of the collected data, the collected data is filtered through the target filter corresponding to the collected data to obtain candidate data.

4. The method according to claim 1, characterized in that, Sending the candidate data and its corresponding data fingerprint to the server includes: Save each candidate data, the data fingerprint of each candidate data, and the sending status of each candidate data to a data file; Based on the sending status in the data file, the candidate data and its corresponding data fingerprint are sent to the server.

5. The method according to claim 4, characterized in that, The method further includes: In the event that the candidate data and its corresponding data fingerprint fail to be sent, the sending status corresponding to the failed candidate data in the data file is marked. In response to the compensation mechanism, based on the data file, all the candidate data that failed to be sent and their corresponding data fingerprints are sent to the server.

6. The method according to claim 4, characterized in that, The method further includes: Under the condition that the preset conditions are met, construct and store the image file of the target filter; If the terminal restarts, load the image file to start the target filter; Based on the data file, extract the data fingerprint after the timestamp of the image file, and update the target filter based on the data fingerprint.

7. The method according to claim 6, characterized in that, The step of constructing and storing the image file of the target filter under preset conditions includes: If the amount of data processed by the target filter meets a preset threshold, or if the filtering time of the target filter meets a preset period, the bit array of the target filter is serialized to obtain the serialized target filter. For the serialized target filter, an image file is constructed and stored.

8. The method according to any one of claims 1 to 7, characterized in that, Acquire the collected data and calculate the corresponding data fingerprint, including: The data to be collected is processed to obtain multiple data points, and the data type of each data point is determined. Based on the data type of each of the collected data, calculate the data fingerprint corresponding to each of the collected data for the full data or key business fields.

9. A data acquisition system, characterized in that, include: A terminal, wherein the terminal is used to implement the steps of the method according to any one of claims 1 to 8; The server is connected to the terminal and is used to receive candidate data and its corresponding data fingerprint sent by the terminal. Based on the local database and the data fingerprint, the server performs deduplication on the candidate data and stores the target data obtained after deduplication in the local database.

10. A data acquisition device, characterized in that, Applied to a terminal, the device includes: The data acquisition module is used to acquire collected data and calculate the corresponding data fingerprint; The data filtering module is used to filter the collected data based on the data fingerprint and the data type corresponding to the collected data, and obtain candidate data through a pre-deployed target filter; the data fingerprint of the candidate data does not exist in the target filter; different data types correspond to different filtering strategies of the target filter; A data sending module is used to send the candidate data and its corresponding data fingerprint to the server; the server is used to store the target data in the candidate data into a database, wherein the data fingerprint of the target data is different from the data fingerprint of the data already stored in the database.

Images