Logistics trajectory data synchronization method, apparatus and device, and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a logistics track data synchronization method, device and equipment and a storage medium, and the method realizes real-time cleaning and updating of abnormal data through an incremental data processing mechanism, namely, real-time analysis and operation log generation based on a message queue, so that the real-time cleaning and updating of the abnormal data are realized. A data processing flow is advanced to a message queue stage, so that disconnection between a data cleaning link and a display link is avoided, and the external network data delay time is effectively shortened; a differential updating strategy is adopted, incremental data only updates local network point data, and full data is subjected to replacement operation after verification is passed, so that global data refreshing in a traditional full coverage mode can be avoided, data updating granularity is refined to a network point level, and the phenomenon of alternation of new and old data in a high-concurrency scene is effectively avoided; and invalid data cleaning is realized through a dual guarantee mechanism, so that the problem of dirty data residue caused by lack of a compensation mechanism in a traditional scheme is solved, and the validity of cached data is ensured.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and storage medium for synchronizing logistics trajectory data. Background Technology

[0002] In the logistics industry, to meet the real-time display requirements of multiple channels such as official websites, mini-programs, and push platforms, logistics tracking data must be efficiently synchronized from the internal network to the external network. However, the current technical solutions have shortcomings in practical applications, as follows: First, the process of cleaning abnormal data relies heavily on offline batch processing, which can easily lead to significant delays in the logistics tracking information displayed on the external network. This severely impacts the user's real-time query experience, preventing users from obtaining accurate logistics updates in a timely manner and reducing the user experience.

[0003] Secondly, regarding cache updates, existing technologies adopt a full-coverage update mode. In high-concurrency scenarios, since full-coverage requires replacing all cached data at once, it is very easy to cause the phenomenon of old and new data flashing alternately, resulting in users seeing inconsistent logistics information in a short period of time. In severe cases, data corruption may even occur, further reducing the user experience.

[0004] Furthermore, current technical solutions lack effective compensation mechanisms for deletion operations. Once certain branch information becomes invalid and needs to be deleted from the cache, without corresponding compensation measures, this invalid branch data often remains in the cache for a long time and cannot be cleaned up in time, gradually accumulating into dirty data. This dirty data not only occupies storage resources but also interferes with the normal display of logistics information, further exacerbating data inconsistency and inaccuracy, posing a serious threat to the overall operational efficiency and data quality of the logistics system. It is evident that existing technologies still need improvement and enhancement. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the purpose of this invention is to provide a logistics trajectory data synchronization method, which realizes real-time synchronization of logistics trajectory data, cache optimization and data consistency guarantee, and solves the technical problems of data delay, cache inconsistency and data residue in traditional solutions.

[0006] The first aspect of this invention provides a method for synchronizing logistics trajectory data, comprising: in a message queue, confirming the message type of trajectory data based on a header marker, wherein the message type includes incremental data and full data; receiving and parsing the trajectory data in the message queue; when the message type of the trajectory data is incremental data, first generating an operation log based on the incremental data, and then updating the cached network point data based on the incremental data; when the message type of the trajectory data is full data, first verifying the full data, and if the verification passes, replacing the baseline network point set in the cache with the full data; performing a heat assessment based on a sliding time window on the network point data in the cache, dynamically adjusting the lifespan of the network point data based on the heat assessment result, and periodically cleaning up invalid network point data in the cache based on a preset compensation mechanism; when a preset check time point is reached, performing a consistency check on the full data in the cache, and repairing the full data in the cache based on the consistency check result.

[0007] Optionally, in a first implementation of the first aspect of the present invention, the step of confirming the message type of trajectory data based on header tags in the message queue, wherein the message type includes incremental data and full data, includes: confirming the message type of trajectory data based on header tags in the message queue, wherein the message type includes incremental data and full data; when the message type of the trajectory data is full data, generating a version number of the trajectory data based on the full date stamp, and transmitting the trajectory data through a pre-created full channel; when the message type of the trajectory data is incremental data, assigning a priority tag to the trajectory data according to the operation type of the incremental data, generating a version number of the trajectory data based on the full date stamp and the daily incremental sequence, and transmitting the trajectory data through a pre-created incremental channel.

[0008] Optionally, in the second implementation of the first aspect of the present invention, when the message type of the trajectory data is incremental data, generating an operation log based on the incremental data and then updating the cached network point data based on the incremental data includes: obtaining a historical version traceability tree, wherein the historical version traceability tree takes the historical full data as the root node; when the message type of the trajectory data is incremental data, generating an operation log based on the operation type and operation content of the incremental data; confirming the modification of network point information based on the incremental data, wherein the modification of network point information includes the modified network point ID and the modification content; updating the network point data in the Hash structure of the Pika cache according to the confirmed modified network point information, and adding a branch node corresponding to the modified network point information at the root node corresponding to the modified network point ID in the historical version traceability tree.

[0009] Optionally, in the third implementation of the first aspect of the present invention, when the message type of the trajectory data is full data, the full data is first verified. If the verification passes, the full data is used to replace the reference point set in the cache. This includes: when the message type of the trajectory data is full data, calculating the CRC2 checksum of the full data; obtaining a preset reference checksum; if the calculated CRC2 checksum is consistent with the preset reference checksum, it indicates that the full data has passed the verification; covering the Hash structure of the Pika cache with the verified full data, wherein the Hash structure is the reference point set; replacing the root node of the historical version traceability tree with the verified full data; and calibrating the historical branch nodes of the historical version traceability tree based on the replaced root node to obtain the new version traceability tree.

[0010] Optionally, in the fourth implementation of the first aspect of the present invention, the step of performing a popularity assessment based on a sliding time window on the cached network point data, and dynamically adjusting the lifespan of the network point data based on the popularity assessment results, includes: deploying a sliding time window and counting the number of accesses to each network point data in the cache in real time; determining the popularity score of each network point data based on the number of accesses, wherein the popularity score includes high popularity, medium popularity, and low popularity; and dynamically adjusting the lifespan of the network point data and adjusting the storage method of the network point data based on the popularity score.

[0011] Optionally, in the fifth implementation of the first aspect of the present invention, the step of periodically cleaning up invalid node data in the cache based on a preset compensation mechanism includes: when a preset compensation time point is reached, scanning all node data in the cache using a Bloom filter to obtain suspected invalid node information; performing secondary invalidation confirmation on the suspected invalid node information based on the baseline node set in the cache and the generated operation log to obtain actual invalid node information; performing invalidation cleanup operation on the node data in the cache based on the actual invalidation node information, and generating an invalidation cleanup log based on the invalidation cleanup operation.

[0012] Optionally, in a sixth implementation of the first aspect of the present invention, the step of performing a consistency check on all data in the cache when a preset check time point is reached, and performing repair processing on all data in the cache based on the consistency check result, includes: when the preset check time point is reached, calculating a first CRC32 checksum of all data in the cache, and calculating a second CRC32 checksum of all data in the cold storage; comparing the first CRC32 checksum and the second CRC32 checksum; if the first CRC32 checksum and the second CRC32 checksum are consistent, then no repair processing is required on all data in the cache; if the first CRC32 checksum and the second CRC32 checksum are inconsistent, then a repair operation is performed on all data in the cache based on all data in the cold storage, and a repair log is generated based on the repair operation.

[0013] A second aspect of the present invention provides a logistics trajectory data synchronization device, comprising: a confirmation module, configured to confirm the message type of trajectory data in a message queue based on a header marker, wherein the message type includes incremental data and full data; a receiving module, configured to receive and parse the trajectory data in the message queue; a first processing module, configured to generate an operation log based on the incremental data and then update the cached network point data based on the incremental data when the message type of the trajectory data is incremental data; a second processing module, configured to first verify the full data when the message type of the trajectory data is full data, and if the verification passes, replace the baseline network point set in the cache with the full data; an adjustment module, configured to perform a heat assessment based on a sliding time window on the network point data in the cache, dynamically adjust the lifespan of the network point data based on the heat assessment result, and periodically clean up invalid network point data in the cache based on a preset compensation mechanism; and a repair module, configured to perform a consistency verification on the full data in the cache when a preset check time point is reached, and repair the full data in the cache based on the consistency verification result.

[0014] A third aspect of the present invention provides a logistics trajectory data synchronization device, the logistics trajectory data synchronization device comprising: a memory and at least one processor, the memory storing instructions; the at least one processor calling the instructions in the memory to cause the logistics trajectory data synchronization device to perform the various steps of the logistics trajectory data synchronization method described above.

[0015] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the logistics trajectory data synchronization method described in any of the preceding claims.

[0016] In the technical solution of this invention, an incremental data processing mechanism, namely real-time parsing and operation log generation based on message queues, is used to achieve real-time cleaning and updating of abnormal data. Compared with the traditional offline batch processing mode, this method advances the data processing flow to the message queue stage, avoiding the disconnect between the data cleaning and display stages and effectively shortening the latency of external network data. A differentiated update strategy is adopted, where incremental data only updates local network point data, and the full data is replaced after verification. This layered update mechanism avoids the global data refresh in the traditional full coverage mode, refines the data update granularity to the network point level, and effectively avoids the phenomenon of old and new data alternation in high-concurrency scenarios. Invalid data cleaning is achieved through a dual guarantee mechanism, which solves the problem of dirty data residue caused by the lack of compensation mechanism in traditional solutions, ensuring the validity of cached data. Furthermore, by periodically checking and repairing the full data, the occurrence rate of data anomalies is reduced, and the accuracy of data display is improved. Attached Figure Description

[0017] Figure 1 This is a first flowchart of a logistics trajectory data synchronization method provided in an embodiment of the present invention; Figure 2 This is a second flowchart of the logistics trajectory data synchronization method provided in an embodiment of the present invention; Figure 3 This is a third flowchart of the logistics trajectory data synchronization method provided in the embodiments of the present invention; Figure 4 This is a fourth flowchart of the logistics trajectory data synchronization method provided in the embodiments of the present invention; Figure 5 The fifth flowchart of the logistics trajectory data synchronization method provided in the embodiments of the present invention; Figure 6 The sixth flowchart of the logistics trajectory data synchronization method provided in the embodiments of the present invention; Figure 7 The seventh flowchart of the logistics trajectory data synchronization method provided in the embodiments of the present invention; Figure 8 A schematic diagram of a logistics trajectory data synchronization device provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of a logistics trajectory data synchronization device provided in an embodiment of the present invention. Detailed Implementation

[0018] This invention provides a method, apparatus, device, and storage medium for synchronizing logistics trajectory data. In this invention, the terms "first," "second," "third," "fourth," etc. (if applicable) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data used can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0019] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the logistics trajectory data synchronization method in this invention includes: 101. In the message queue, the message type for confirming trajectory data is based on the header marker, and the message type includes incremental data and full data; In this embodiment, the message queue is a Kafka message queue, which obtains trajectory data from the Baima system. The Baima system, as the data source, covers various information about logistics screening points, including basic information and operational status. Real-time incremental data refers to the immediate changes that occur at points during business operations, such as data generated by adding, modifying, or closing points. Scheduled full data is generated daily at a fixed time, such as 2 AM, and includes a complete list of all currently operational points. In actual operation, the Baima system accurately identifies each change in point information and converts it into incremental data conforming to the Kafka message format. The generation of full data must strictly adhere to the predetermined time, and the integrity and accuracy of the data must be guaranteed during the generation process to avoid data loss or errors. Simultaneously, the Kafka configuration parameters also need to be optimized based on factors such as data volume and data generation frequency to ensure efficient and stable data transmission.

[0020] 102. Receive and parse the trajectory data in the message queue; In this embodiment, after receiving a message transmitted from the Kafka message queue, the consumer's primary task is to parse the key information in the message header marker; by identifying the type field, it determines whether the message type is incremental or full.

[0021] 103. When the message type of the trajectory data is incremental data, first generate the operation log based on the incremental data, and then update the network point data in the cache based on the incremental data. 104. When the message type of the trajectory data is full data, first verify the full data. If the verification passes, replace the reference network point set in the cache with the full data. In this embodiment, differentiated processing methods are applied to incremental data and full data to efficiently handle different types of data changes; real-time processing of incremental data ensures the immediate updating of branch information and improves the real-time performance of business data.

[0022] 105. Perform a heat assessment based on a sliding time window on the cached branch data, dynamically adjust the lifespan of the branch data based on the heat assessment results, and periodically clean up invalid branch data in the cache based on a preset compensation mechanism. In this embodiment, a cache self-optimization mechanism is introduced to dynamically adjust the caching strategy based on the access frequency of branch data, i.e., the popularity of branch data. This mechanism improves the utilization rate of cache resources and reduces space redundancy by prioritizing the caching of high-frequency data. At the same time, it speeds up the reading speed of high-frequency data and significantly improves the response performance of the business system.

[0023] In this embodiment, the compensation task, through periodic scanning and cleaning, can promptly detect and handle the problem of residual invalid branch data caused by reasons such as missed message pushes and processing failures, ensuring the accuracy and cleanliness of cached and log data, avoiding invalid data from occupying system resources, and improving the system's operating efficiency.

[0024] 106. When the preset check time point is reached, perform a consistency check on all data in the cache, and repair the all data in the cache based on the consistency check result. In this embodiment, targeted repair is a key step in data quality control. By constructing a full data verification and repair mechanism, and based on a periodic comparison strategy between baseline data and cached data, a deep verification is performed at the end of each day's business. When the consistency verification fails, the full data in the cache is repaired, achieving high-precision alignment of the two-end data and significantly improving the reliability and effectiveness of data storage.

[0025] This application discloses a method for synchronizing logistics trajectory data. Through an incremental data processing mechanism, namely real-time parsing and operation log generation based on message queues, it achieves real-time cleaning and updating of abnormal data. Compared with the traditional offline batch processing mode, this method advances the data processing flow to the message queue stage, avoiding the disconnect between the data cleaning and display stages and effectively shortening the latency of external network data. It adopts a differentiated update strategy, where incremental data only updates local network point data, and the full data is replaced after verification. This layered update mechanism avoids the global data refresh in the traditional full coverage mode, refines the data update granularity to the network point level, and effectively avoids the phenomenon of old and new data alternation in high-concurrency scenarios. It achieves invalid data cleaning through a dual guarantee mechanism, solving the problem of dirty data residue caused by the lack of compensation mechanism in traditional solutions, ensuring the validity of cached data, and reducing the data anomaly rate and improving the accuracy of data display by regularly checking and repairing the full data.

[0026] Please see Figure 2 In this embodiment of the invention, the message type for confirming trajectory data based on header markers in the message queue includes incremental data and full data, including: 201. In the message queue, the message type for confirming trajectory data is based on the header marker, and the message type includes incremental data and full data; 202. When the message type of the trajectory data is full data, the version number of the trajectory data is generated based on the full date stamp, and the trajectory data is transmitted through the pre-created full channel; 203. When the message type of the trajectory data is incremental data, a priority label is assigned to the trajectory data according to the operation type of the incremental data, a version number of the trajectory data is generated based on the full date stamp and the incremental sequence of the day, and the trajectory data is transmitted through a pre-created incremental channel; In this embodiment, a unique version number is generated for each trajectory data point, following the rule: full data date + daily incremental sequence number, and written into the header tag of the Kafka message; for example, the 10th incremental data point on May 22, 2025, has a version number of 202505220010; while the version number of the full data is fixed as the date stamp of the day, such as 20250522, indicating the base version for that day; the introduction of a globally unique version number allows for clear tracing of the data source, generation time, and version change during data synchronization and processing, effectively avoiding errors caused by data version confusion, and greatly enhancing data manageability and maintainability; the generated version number and priority tag are written together into the header tag of the Kafka message and sent along with the Kafka message.

[0027] In this embodiment, the operation types for real-time incremental data include adding, modifying, and deleting. For deletion operations, if not processed in a timely manner, invalid branch information may remain for a long time, affecting the accuracy of business and the cleanliness of data. Therefore, deletion operations are marked as the highest priority. For adding operations, although the addition of new branch information has an impact on business, the short-term delay has a smaller impact on the overall business compared to deletion operations. Therefore, adding operations are marked as medium priority. Modification operations are marked as low priority. For example, minor adjustments to branch addresses have a low impact on business due to inconsistencies within a certain period of time. By assigning priority tags, resources can be scheduled according to priority during subsequent consumption processes to ensure the smoothness of critical operations and improve the overall business response speed.

[0028] In this embodiment, two independent topics are created in the message queue, including a channel dedicated to incremental data transmission and a channel dedicated to full data transmission. By using the dual-topic separate transmission method, the mixing of incremental and full data is avoided, enabling the data consumer to quickly and accurately identify and process different types of data, greatly improving the efficiency and accuracy of data processing, and laying a solid foundation for subsequent data synchronization and consistency assurance.

[0029] Please see Figure 3 In this embodiment of the invention, when the message type of the trajectory data is incremental data, first generating an operation log based on the incremental data, and then updating the cached network point data based on the incremental data, includes: 301. Obtain the historical version traceability tree, wherein the historical version traceability tree is rooted at the full historical data. 302. When the message type of the trajectory data is incremental data, an operation log is generated based on the operation type and operation content of the incremental data; 303. Confirm the modification of branch information based on incremental data, wherein the modified branch information includes the modified branch ID and the modified content; 304. Based on the confirmed modified site information, update the site data in the Pika cached Hash structure, and add a branch node corresponding to the modified site information at the root node corresponding to the modified site ID in the historical version traceability tree.

[0030] In this embodiment, the historical version traceability tree is either the version traceability tree of the previous day or the version traceability tree of the most recent modification. When the message type of the trajectory data is incremental data, a new branch is added to the corresponding branch version tree. Specifically, the version traceability tree uses the daily full data as the root node, and each incremental modification will extend a new branch from the root node or an existing branch node. For example, the root node of branch A is version 20250522 corresponding to the daily full data. When the 10th incremental modification occurs, a new branch node 202505220010 will be added. This branch node records key information such as modification time and modification content. Afterward, the incremental data is written into the Hash structure of the Pika cache. The Pika cache uses an efficient Hash storage method to store all branch data in the same Hash. Under the Key, the branch ID is used as the key, and the latest information of the branch is used as the value. Each time incremental data is input, only the value of the corresponding branch ID needs to be updated, without full overwriting, which effectively improves the efficiency and stability of caching operations in high-concurrency scenarios. In addition, to ensure the reliability of data operations, detailed information of incremental operations is appended to the WAL log, including version number, operation content, timestamp, etc. If the system fails during operation, the incomplete operation can be recovered by replaying the WAL log, ensuring that no data is lost. This ensures the recoverability of data operations and greatly improves the fault tolerance of the system.

[0031] Please see Figure 4 In this embodiment of the invention, when the message type of the trajectory data is full data, the full data is first verified. If the verification passes, the full data is used to replace the reference network point set in the cache. This includes: 401. When the message type of the trajectory data is full data, calculate the CRC2 checksum of the full data; 402. Obtain the preset baseline check code. If the calculated CRC2 check code is consistent with the preset baseline check code, it indicates that all data has passed the check. 403. Overwrite the hash structure of the Pika cache with the full amount of data that has passed the verification, wherein the hash structure is a base point set; 404. Replace the root node of the historical version traceability tree with the verified full data. Based on the replaced root node, calibrate the historical branch nodes of the historical version traceability tree to obtain the new version traceability tree.

[0032] In this embodiment, when the trajectory data message type is full data, the full data is first temporarily stored in a cold storage device, such as a low-cost SSD, to avoid occupying cache resources. Next, a CRC32 checksum is performed on the full data. The CRC32 checksum of the full data is calculated and compared with the baseline checksum sent by the White Horse system to verify whether the data is complete and untampered during transmission. If the checksum passes, the newly input full data is used as the new baseline version, replacing the root node of the historical version's traceability tree, while the original root node is archived. Since the full baseline has changed, i.e., the root node has changed, The incremental branch nodes need to be recalculated based on the new full baseline. For example, if a network point under the old full baseline has undergone multiple incremental modifications, after the new full baseline is determined, the incremental branch nodes of that network point need to be reorganized and generated according to the new full baseline, retaining only the modification records after the new full baseline to ensure the accuracy and consistency of the new version's traceability tree. Finally, the verified full data is used to overwrite the hash structure in the Pika cache, ensuring that the cached data is consistent with the latest full baseline of the day. By verifying the full data before replacement, the accuracy and integrity of the data are ensured, providing a reliable baseline for the processing of incremental data.

[0033] Please see Figure 5 In this embodiment of the invention, the step of performing a heat assessment based on a sliding time window on the cached network point data, and dynamically adjusting the lifespan of the network point data based on the heat assessment results, includes: 501. Deploy a sliding time window to count the number of accesses to each network point's data in the cache in real time; In this embodiment, a sliding time window mechanism is deployed, for example, a sliding window with a duration of 1 hour is set, and the sliding time window slides once every 5 minutes; within this window, the number of visits to each network point is counted in real time; for example, within a certain hour period, network point A is queried 20 times and network point B is queried 2 times.

[0034] 502. Determine the popularity score of each site's data based on the number of visits, wherein the popularity score includes high popularity, medium popularity, and low popularity; In this embodiment, a popularity score is generated based on the access frequency. The specific scoring rules are as follows: websites with more than 10 accesses within 1 hour are defined as high-frequency access websites, and their popularity score is assigned a value of 5, corresponding to a high-popularity state; websites with 3-10 accesses within 1 hour are defined as medium-frequency access websites, and their popularity score is assigned a value of 3, corresponding to a medium-popularity state; websites with less than 3 accesses within 1 hour are defined as low-frequency access websites, and their popularity score is assigned a value of 1, corresponding to a low-popularity state.

[0035] 503. Based on popularity scores, dynamically adjust the lifespan of branch data and adjust the storage method of branch data. In this embodiment, the cached data is optimized in two aspects based on the popularity score: Firstly, dynamically adjust the lifetime of cached data. For frequently accessed data, set its lifetime to 1 hour. Since this type of data is accessed frequently, extending its retention time in the cache can reduce the number of times data is repeatedly read from the backend storage, thereby improving system response speed. For moderately accessed data, set its lifetime to 30 minutes. This ensures a certain cache hit rate while balancing cache space usage. For infrequently accessed data, set its lifetime to 10 minutes. Since this type of data is accessed less frequently, its rapid expiration and removal from the cache can free up cache space to store more valuable data. Secondly, implement tiered storage. Store frequently accessed data in memory, such as using Redis memory caching and employing a lock-free read optimization structure to reduce lock contention during multi-threaded concurrent reads and further improve read speed. Store medium- and low-frequency data in SSD storage devices. SSDs are cheaper than memory and are suitable for storing data with low access frequency. At the same time, enable compression encoding for data stored in SSDs to reduce storage space usage and improve storage efficiency.

[0036] In this embodiment, by combining tiered storage with dynamic time-to-live adjustment, storage costs can be effectively reduced while ensuring system performance, achieving an optimal balance between performance and cost.

[0037] Please see Figure 6 In this embodiment of the invention, the periodic clearing of invalid branch data in the cache based on a preset compensation mechanism includes: 601. When the preset compensation time point is reached, a Bloom filter is used to scan all network point data in the cache to obtain information on suspected failed network points; In this embodiment, a Bloom filter is activated every 30 minutes to scan and output information on potentially invalid sites. A Bloom filter is an efficient probabilistic data structure used to quickly determine whether an element exists in a set. In this embodiment, the system pre-stores information on potentially invalid sites, such as site IDs that have been marked for deletion and site IDs that have not been updated for more than 7 days, into the Bloom filter. Every 30 minutes, the system automatically activates the Bloom filter to scan all site data in the Pika cache. The Bloom filter uses a specific hash algorithm to quickly filter out potentially invalid sites and outputs a list of potentially invalid sites. For example, if site C was marked for deletion in a previous deletion operation, but is still found to exist in the cache by a Bloom filter scan, then this site will be included in the list of potentially invalid sites.

[0038] 602. Based on the baseline network point set in the cache and the generated operation log, perform secondary failure confirmation on the suspected failed network point information to obtain the actual failed network point information; 603. Based on the actual failed branch information, perform invalid cleanup operations on the branch data in the cache, and generate invalid cleanup logs based on the invalid cleanup operations; In this embodiment, the network points in the suspected invalid network point list undergo secondary confirmation. There are two main confirmation methods: First, query the latest full baseline data in the database. If the network point information does not exist in the database, it can be confirmed that the network point has become invalid. Second, check the WAL log. If there is a clear record of deleting the network point in the WAL log and the record has been executed, the network point is confirmed to be invalid. After secondary confirmation, for network points that are determined to be invalid, the system will delete them from the Hash structure of the Pika cache and append a cleanup operation record for the network point to the WAL log. This makes the entire cleanup operation process traceable, providing strong support for system operation and maintenance and fault diagnosis, and further enhancing the data consistency and reliability of the system.

[0039] Please see Figure 7 In this embodiment of the invention, when a preset check time point is reached, a consistency check is performed on all data in the cache, and the full data in the cache is repaired based on the consistency check result, including: 701. When the preset check time point is reached, calculate the first CRC32 check code of the full data in the cache, and calculate the second CRC32 check code of the full data in the cold storage. 702. Compare the first CRC32 checksum and the second CRC32 checksum; 703. If the first CRC32 checksum and the second CRC32 checksum are the same, then there is no need to repair all the data in the cache. In this embodiment, when the preset inspection time point is reached, such as 11 PM every day, the system starts the end-of-day audit process. First, it calculates the CRC32 checksum of the full data in the cache, that is, the full data stored in the Hash structure of the Pika cache, and calculates the CRC32 checksum of the full baseline data of the White Horse system for the day, that is, the full data stored in the cold storage device. The CRC32 checksum is a fixed-length check value calculated by a specific algorithm for data, used to verify the integrity and consistency of the data. If the CRC32 checksums calculated by the two are consistent, it means that the full data in the cache is completely consistent with the baseline full data of the White Horse system, and no further processing is required.

[0040] 704. If the first CRC32 checksum and the second CRC32 checksum are inconsistent, then the full data in the cache is repaired based on the full data in the cold storage, and a repair log is generated based on the repair operation. In this embodiment, if the CRC32 checksums are inconsistent, it indicates a data discrepancy, requiring a full data repair operation. At this time, the system does not directly transmit the complete discrepancy data, but instead identifies the discrepancy by comparing the data fingerprints of the two full datasets. The data fingerprint refers to the 128-bit SHA3 hash value generated for each data point. By comparing the SHA3 hash value of the data point in the cache with the hash value of the corresponding data point in the baseline full dataset, the system can quickly locate the data points with inconsistent fingerprints, i.e., the data points with discrepancies. For example, if the SHA3 hash value of data point D in the cache is inconsistent with that in the baseline data, then data point D is the discrepancy point. For the located discrepancy point, the system uses the baseline full dataset from the White Horse system to overwrite the erroneous data point information in the cache, completing the data repair operation. Simultaneously, a consistency log, i.e., a repair log, is generated. The repair log includes information such as the data consistency rate for the day, the number of discrepancies, detailed repair records, and the amount of invalid data cleaned up by the compensation task on that day.

[0041] In this embodiment, hash fingerprint verification and difference point location technology are used to reduce network transmission redundancy through compression algorithms while ensuring data integrity, effectively improving data repair efficiency. In addition, by generating repair logs, key indicators and abnormal information in the data synchronization process are visualized, providing maintenance personnel with a systematic operational status analysis framework, which facilitates the rapid location of system performance bottlenecks and the implementation of targeted optimization strategies, thereby enhancing system stability and business continuity assurance capabilities.

[0042] The logistics trajectory data synchronization method in the embodiments of the present invention has been described above. The logistics trajectory data synchronization device in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 8 One embodiment of the logistics trajectory data synchronization device in this invention includes: The confirmation module 801 is used to confirm the message type of trajectory data in the message queue based on the header marker. The message type includes incremental data and full data. The receiving module 802 is used to receive and parse the trajectory data in the message queue; The first processing module 803 is used to generate an operation log based on the incremental data when the message type of the trajectory data is incremental data, and then update the network point data in the cache based on the incremental data. The second processing module 804 is used to first verify the full data when the message type of the trajectory data is full data. If the verification passes, the full data is used to replace the benchmark network point set in the cache. The adjustment module 805 is used to perform a heat assessment based on a sliding time window on the cached network point data, dynamically adjust the lifespan of the network point data based on the heat assessment results, and periodically clean up invalid network point data in the cache based on a preset compensation mechanism. The repair module 806 is used to perform a consistency check on all data in the cache when a preset check time point is reached, and to repair the all data in the cache based on the consistency check result.

[0043] Based on the same ideas as the methods in the above embodiments, the apparatus provided in this application can implement the methods in the above embodiments.

[0044] above Figure 8 The logistics trajectory data synchronization device in this embodiment of the invention is described in detail from the perspective of modular functional entities. The logistics trajectory data synchronization device in this embodiment of the invention is described in detail below from the perspective of hardware processing.

[0045] Figure 9This is a schematic diagram of the structure of a logistics trajectory data synchronization device 900 provided in an embodiment of the present invention. The logistics trajectory data synchronization device 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 910 (e.g., one or more processors) and a memory 920, and one or more storage media 930 (e.g., one or more mass storage devices) storing application programs 933 or data 932. The memory 920 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the logistics trajectory data synchronization device 900. Furthermore, the processor 910 may be configured to communicate with the storage media 930 and execute a series of instruction operations in the storage media 930 on the logistics trajectory data synchronization device 900 to implement the steps of the logistics trajectory data synchronization method provided in the above-described method embodiments.

[0046] The logistics trajectory data synchronization device 900 may also include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input / output interfaces 960, and / or one or more operating systems 931, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 9 The illustrated structure of the logistics trajectory data synchronization device does not constitute a limitation on the logistics trajectory data synchronization device. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0047] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a logistics trajectory data synchronization method.

[0048] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0049] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0050] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A logistics track data synchronization method, characterized in that, The method comprises the following steps: In the message queue, the message type of the trajectory data is determined based on the header mark, and the message type comprises incremental data and full data; Receiving and parsing the trajectory data in the message queue; When the message type of the trajectory data is incremental data, first, an operation log is generated based on the incremental data, and then the point data in the cache is updated based on the incremental data; When the message type of the trajectory data is full data, first, the full data is verified, and if the verification is passed, the full data is used to replace the reference point set in the cache; Performing heat evaluation on the point data in the cache based on a sliding time window, dynamically adjusting the survival time of the point data based on the heat evaluation result, and cleaning the invalid point data in the cache based on a preset compensation mechanism; When reaching a preset check time point, performing consistency verification on the full data in the cache, and performing repair processing on the full data in the cache based on the consistency verification result.

2. The logistics track data synchronization method according to claim 1, characterized in that, The method comprises the following steps: In the message queue, the message type of the trajectory data is determined based on the header mark, and the message type comprises incremental data and full data; When the message type of the trajectory data is full data, a version number of the trajectory data is generated based on a full data timestamp, and the trajectory data is transmitted through a pre-created full channel; When the message type of the trajectory data is incremental data, a priority label is assigned to the trajectory data according to the operation type of the incremental data, a version number of the trajectory data is generated based on a full data timestamp and a daily incremental sequence, and the trajectory data is transmitted through a pre-created incremental channel.

3. The logistics track data synchronization method according to claim 2, characterized in that, The method comprises the following steps: Obtaining a historical version trace tree, wherein the historical version trace tree takes historical full data as a root node; When the message type of the trajectory data is incremental data, an operation log is generated based on the operation type and operation content of the incremental data; Confirming modified point information based on the incremental data, wherein the modified point information comprises a modified point ID and modified content; According to the confirmed modified point information, the point data in the Hash structure of the Pika cache is updated, and a branch node corresponding to the modified point information is added at the root node corresponding to the modified point ID in the historical version trace tree.

4. The logistics track data synchronization method according to claim 3, characterized in that, The method comprises the following steps: When the message type of the trajectory data is full data, a CRC2 check code of the full data is calculated; A preset reference check code is obtained, and if the calculated CRC2 check code is consistent with the preset reference check code, it indicates that the full data passes the verification; The full data that passes the verification is used to cover the Hash structure of the Pika cache, and the Hash structure is a reference point set; The root node of the historical version traceability tree is replaced with the verified full data. Based on the replaced root node, the historical branch nodes of the historical version traceability tree are calibrated to obtain the new version traceability tree.

5. The logistics track data synchronization method according to claim 1, characterized in that, The step of performing a sliding time window-based popularity assessment on the cached branch data, and dynamically adjusting the lifespan of the branch data based on the popularity assessment results, includes: Deploy a sliding time window to count the number of accesses to data for each network point in the cache in real time; The popularity score of each site's data is determined based on the number of visits, and the popularity score includes high popularity, medium popularity, and low popularity; Based on popularity scores, the lifespan of branch data is dynamically adjusted, and the storage method of branch data is also adjusted.

6. The logistics track data synchronization method according to claim 1, characterized in that, The method of periodically clearing invalid branch data from the cache based on a preset compensation mechanism includes: When the preset compensation time point is reached, a Bloom filter is used to scan all network point data in the cache to obtain information on suspected failed network points. Based on the baseline network point set in the cache and the generated operation log, the suspected failed network point information is confirmed for a second time to obtain the actual failed network point information; Based on the actual failed branch information, invalidation cleanup operations are performed on the branch data in the cache, and invalidation cleanup logs are generated based on the invalidation cleanup operations.

7. The logistics track data synchronization method according to claim 1, characterized in that, When the preset check time point is reached, a consistency check is performed on all data in the cache, and the cached data is repaired based on the consistency check result, including: When the preset check time point is reached, calculate the first CRC32 check code of the full data in the cache, and calculate the second CRC32 check code of the full data in the cold storage. Compare the first CRC32 checksum and the second CRC32 checksum; If the first CRC32 checksum and the second CRC32 checksum match, then there is no need to repair all the data in the cache. If the first CRC32 checksum and the second CRC32 checksum are inconsistent, then the full data in the cache is repaired based on the full data in the cold storage, and a repair log is generated based on the repair operation.

8. A logistics track data synchronization apparatus, characterized by, include: The confirmation module is used to confirm the message type of trajectory data in the message queue based on the header marker. The message type includes incremental data and full data. The receiving module is used to receive and parse the trajectory data in the message queue; The first processing module is used to generate an operation log based on the incremental data when the message type of the trajectory data is incremental data, and then update the network point data in the cache based on the incremental data. The second processing module is used to first verify the full data when the message type of the trajectory data is full data. If the verification passes, the full data is used to replace the benchmark network set in the cache. The adjustment module is used to perform a heat assessment on the cached branch data based on a sliding time window, dynamically adjust the lifespan of the branch data based on the heat assessment results, and periodically clean up invalid branch data in the cache based on a preset compensation mechanism. The repair module is used to perform a consistency check on all data in the cache when a preset check time point is reached, and to repair the all data in the cache based on the consistency check result.

9. A logistics track data synchronization device, characterized by, The logistics track data synchronization device comprises a memory and at least one processor, and the memory stores instructions; The at least one processor invokes the instructions in the memory, so that the logistics track data synchronization device performs the steps of the logistics track data synchronization method according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon instructions, the instructions comprising, The instructions are executed by the processor to implement the steps of the logistics track data synchronization method according to any one of claims 1-7.

Images